Kernel Traffic
Latest | Archives | People | Topics
Wine
Latest | Archives | People | Topics
GNUe
Latest | Archives | People | Topics
Czech
Home | News | RSS Feeds | Mailing Lists | Authors Info | Mirrors | Stalled Traffic
 

KDE Traffic #61 For 11 Aug 2003

By Russell Miller

Table Of Contents

Introduction

Greetings, this is kde traffic number 61. This covers two weeks, and should hopefully bring everything up to date again.

I'd like to take a brief moment for a shameless plug. Please skip the next two paragraphs if you'd like to ignore it.

Although I provide this newsletter as a free service and I am happy to do it, it takes me up to two to three hours a week to prepare it (except for when I miss a week, as I occasionally do). This does translate directly to lost revenue on my part, as I run a business by which I charge hourly. There are, if you are so inclined, two ways by which you can show your appreciation for this newsletter, while at the same time giving me further incentive to continue to produce it and think of ways to increase its quality.

The first way is to send some work my way. I am a programmer with eight years of experience as a UNIX administrator, C programmer, but I can also write in perl, do web design, database administration, etc. Here is a link to my resume, and you may feel free to contact me for any further information. This is the way that I prefer. However, if you don't have any work to send my way, I also have a paypal account through which I'll accept donations. Any amount is welcome. I will put a button in this intro section from now on, but otherwise will generally keep quiet about it.

Also, to those of you who emailed me last week thanking me for the newsletter, you're welcome. I'm sorry that I haven't had a chance to reply personally, but know that your emails are always read and appreciated.

Mailing List Stats For This Week

We looked at 1409 posts in 5925K.

There were 391 different contributors. 179 posted more than once. 126 posted last week too.

The top posters of the week were:

1. where has kio_ftps gone?

17 Jul 2003 - 25 Jul 2003 (7 posts) Archive Link: "where has kio_ftps gone?"

People: Jens Benecke

Jens Benecke asked:

I seem to remember there was an IOslave for FTP via SSL (not SSH). We have a FTP-SSL server at work (that does not run SSH, or anything else... for "security reasons") and I would need that functionality.

sftp:// doesn't do any good because it only does SSH, not SSL

Any ideas?

The response was: no, there isn't one, there was never one, and one isn't being worked on. Jens asked if there was another solution he could use - there was no response.

2. What is the status of kghostview

18 Jul 2003 - 24 Jul 2003 (7 posts) Archive Link: "What is the status of kghostview"

Jorg asked what the status of kghostview was, as the last time he remembered seeing about it it didn't compile and he hadn't heard about it since. Apparently there is an ifdef that keeps it from compiling under QT 3.1.2, advice was to report it to the author, because those ifdefs aren't supposed to be there unless they are necessary. Another person suggested using kpdf. Vladimir Lazarenko reported that he tried to compile kpdf, and got an error. There was no response.

3. KAgenda L&F changes

20 Jul 2003 - 25 Jul 2003 (15 posts) Archive Link: "[Kde-pim] [PATCH] KAgenda L&F changes"

People: Tim JansenCornelius Schumacher

Tim Jansen started off by writing the following Holy Missive:

I'd like to commit the attached patch.

It does the following:

To make it short, here's a screenshot: http://www.tjansen.de/blog/pics/after2.png

For reference, here is the old renderer: http://www.tjansen.de/blog/pics/before.png

This shows item changes only, with old colors and without other KOAgenda changes: http://www.tjansen.de/blog/pics/after1.png

Responses were pretty favorable, everyone seemed to like the patch. Cornelius Schumacher, however, had the following feedback:

I have looked at the patch and I like it :-). This is a great improvement.

Some nitpicking and some more serious issues:

Tim accepted most of these suggestions. There was a short discussion on fixed vs. configurable color schemes, it seems the decision was to make them configurable, because the code was already there and the config option is easy to ignore.

4. CIA proposal (was: ClientInterface)

21 Jul 2003 - 29 Jul 2003 (28 posts) Archive Link: "CIA proposal (was: ClientInterface)"

People: Marc MutzDon SandersRussell MillerIngo Klöcker

There must not have been much light, because Marc Mutz came out of the woodwork, wiggled his ears and whiskers, and in a high, squeaky voice had this to say:

I was asked to provide a concept for a server-less solution to the CIA (Concurrent Index Access) problem. Here is it. It's based on the idea from LinuxTag that's described here as Step 3. It solves the memory load problems with index files by having them shared between KMail instances running on the same machine.

Step 1: Require Maildir
Rationale:

Step 2: Observe that the Index file format allows most changes (esp. all status updates) to be performed in-place.

Step 3: Don't create an in-memory representation of index entries ("KMMsgInfo::kd") anymore. Instead, let every entry just be a pointer to the beginning of it's mmaped region in the index file. For this to be efficient, use Don's idea of storing those offsets in a file of their own. Every access to information stored in the index accesses the index file data directly.

Rationale:

Step 4: Define a protocol by which a given Kmail instance can obtain write access to a given folder/index file.

Sketch of the CIA/4 protocol:

KMail instance A wants to obtain write access to folder F from B:

1. A tries to create folder-lockfile L_F
Success: Write own DCOP id into the lock file.
A has now (exclusive) write access to F.

2. A tries to read a DCOP ID of B from L_F
Success: A sends B the message
"Please release folder F"
Wait a random amount of time, then goto 1.
Failure: Wait a random amount of time, then goto 1.
Repeated failure: Tell user.

Observe how B is not required to close the folder or to unmmap the index file. It marks the folder read-only internally (or remaps the index to be ro).

An instance holding the write lock is required to

a. Release it as soon as practicable
b. Release it as soon as possible if the above DCOP message arrives
c. never alter existing index entries' length

If a change in entry length is necessary, mark the old one as deleted, rename the maildir file and add a new entry at the end. Other instances will see a delete and add through the dirwatching and can react accordingly.

CIA/4 can be extended with a DCOP broadcast call to all KMail instances to close F, so that A can perform (index file) compaction after all instances replied with a "folder F closed" message.

As is usual for the kmail list, this spawned off a great deal of discussion. Don Sanders had this to say:

What I like:

1) "the Index file format allows most changes (esp. all status updates) to be performed in-place.", deletion is the only exception I think. I'm all for changing deletion so that an index entry is marked as deleted only and then later a cleanup operation, 'compaction' is performed.

I'd prefer it if a timer was used to perform index file cleanup rather than waiting for exit, but that will add extra complexity, (I omit the details of that complexity).

2) The KMail instances are talking to each other via DCOP, so rather than a client only approach in my opinion it's a peer-to-peer network of servers approach. So I think there's some realization here that IPC is sensible.

3) I think this is a realistic suggestion for handling the problem of multiple clients accessing ~/Mail concurrently. Theoretically I think it might work, but I'm unsure about whether the mmap will hold up.

Slightly pro:

1) 'Step 3: Don't create an in-memory representation of index entries ("KMMsgInfo::kd") anymore. Instead, let every entry just be a pointer to the beginning of it's mmaped region in the index file.'

Well as every entry does have a pointer to the beginning of it's mmaped region I read this as simply drop the in-memory representation.

I'm uncertain of the performance implications of this change, I guess they will be implementation specific. Some fields like the status field of a message are always read. This is because when KMail opens a folder it looks at the status of all messages to sync the count of unread messages and to find the next unread messages. (IIRC 1/3 of the time to show a folder is spent searching for the next unread message).

So I think it makes sense to always cache the status field, given this for optimum performance it makes sense to cache it as the index file is read in sequentially, rather than later caching it by randomly accessing the file in some arbitrary order, (I hope that makes sense, I'm skipping over some details here).

Or better yet if you are going to index the index file, please consider caching the status in the index of the index.

There might also be performance implications to do with sorting and the caching of sort keys. Specifically kmheaders might be slow when a new message arrives or when the sort order is changed.

But since the index file is now mmap'd, and therefore cached, I think it makes sense to experiment with dropping/reducing the in-memory representation.

Neutral/Aside:

1) Be careful of the inbox folder, I think there is a design flaw in filtering. We currently use a inbox folder as a temporary storage location for incoming mail but I think it would make more sense to use a private folder. I say this as a private folder for incoming mail could be modified quickly without the need for every client to be informed of the changes.

2) I'm unsure how the index file mmap will hold up. If a client has to close the index file then i think the mmap will be lost, and I think losing the mmap and having no in-memory representation of index entries would be critically bad, performance wise. But if you can keep the mmap then effectively all the caching of index entries has been pushed into the operating system kernel via the mmap, with luck the os is designed in such a way that each client can share this ro mmap cache, and the operating system kernel itself is being used as the KMail server, which would be cool.

I do wonder if multiple processes ro mmap the same file whether the memory used to back the mmap is shared by all those processes.

My concerns:

1) This doesn't actually address the issue of external mail clients modifying ~/Mail does it? I mean not even for MailDir as this approach requires KMail to be running to detect changes in ~/Mail.

2) I'm concerned that this approach could be fragile. A client might die after deleting/inserting a Maildir message but before updating the index, or vice versa. Maybe this can be handled robustly so this is just a concern not a concrete criticism. But clearly this method does rely on mbox style locking rather than maildir style elimination of the need for locking.

3) If this approach works then it does address the criticism of having multiple clients modify index files but there's still the memory costs associated with this approach. Each client will need there own kmmsgdict, IIRC each entry in the kmmsgdict maps a sernum -> (folder , index ) which is an int -> (int, int) mapping or 12 bytes. I have >500K messages currently, so that's >6MB per client instance.

And then there's the full text index. A full text index is by it's nature a large object as it's space/time tradeoff. For me it's > 24MB.

Besides folder files there are also other files than need to be handled. The config file is a problem if it is desirable to have each client keep their config dialogs and general configuration info in sync.

Another set of files is the list of mails that are kept on the server for each account.

In summary:

I like the idea of having all index file operations non-destructive, specifically this means making deletion merely mark an index entry as deleted and then later performing an index compaction. I also like the idea of experimenting with dropping the in-memory caching of index entries, I'd like to see profiling stats on doing that. I think both these tasks make sense to do even if a client/server approach is ultimately taken.

I think this CIA proposal addresses (or at least attempts address) the key criticism I've made against the client only model. Which is having multiple clients work with the same index files. But I remain skeptical especially as to whether the mmap of index files can be retained, I think dropping both the mmap (or frequent re-mmaping) and dropping the in-memory cache would be a bad idea, (because of the performance implications).

I still think overall all a client/server approach makes the most sense. This is because I think there is a demand for KMail to provide database like services to other (KDE) apps. Namely I'm thinking of services to track messages (KMMsgDict) and search messages (KMMsgIndex). Implementing these services efficiently requires using substantial amounts of memory, and it makes sense to concentrate the cost of that memory use in one process rather than duplicate it across multiple processes.

In conclusion personally I'm ok with step 2 and are interested in the results of step 3 of this CIA proposal and think they can be performed in parallel with the Kernel/GUI separation of the client/server approach. Also perhaps similar to step 3 of the CIA proposal I hope to do more profiling work on my zero-copy display of messages idea (now that zero-copy parsing is implemented). I hope zero-copy display of messages will address what I expect to be the key bottle neck between the client/server, which is gross duplication of attachment data.

(ed. [Russell Miller] Whew! Thank Don for making this contentful! )

Marc responded:

I do wonder if multiple processes ro mmap the same file whether the memory used to back the mmap is shared by all those processes.

It is (if you use MAP_SHARED).

My concerns:

1) This doesn't actually address the issue of external mail clients modifying ~/Mail does it? I mean not even for MailDir as this approach requires KMail to be running to detect changes in ~/Mail.

New mail is delivered into the new/ subdir for maildir. The first instance of KMail to close the folder or have a user click on the new message will move the mails from new/ to cur/, updating the index along the way.

Of course, MUAs wishing to modify ~/Mail need to conform to CIA/4, but maildir-compliant MDAs work without modifications.

2) I'm concerned that this approach could be fragile. A client might die after deleting/inserting a Maildir message but before updating the index, or vice versa. Maybe this can be handled robustly so this is just a concern not a concrete criticism. But clearly this method does rely on mbox style locking rather than maildir style elimination of the need for locking.

Locking is only necessary for the index files, not the mails themselves. If a MUA dies while adding a message to the maildir, it will stay incomplete in tmp/ until cleaned up. tmp/ is not monitored, so nobody cares.

If a MUA dies moving a message from tmp/ to new/ or cur/, either the move happens completely or not at all. In the first case, another KMail instance can try to add the message after waiting a certain amount of time for the index to be updated. If the update doesn't arrive, the addition is performed as if it was delivered. For this to work, the sernum has to be added as a x-KMail header to the mail.

3) If this approach works then it does address the criticism of having multiple clients modify index files but there's still the memory costs associated with this approach. Each client will need there own kmmsgdict, IIRC each entry in the kmmsgdict maps a sernum -> (folder , index ) which is an int -> (int, int) mapping or 12 bytes. I have >500K messages currently, so that's >6MB per client instance.

You typically won't run that many KMail instances that these few MB make any difference. Once a lib is split off of KMail for other apps to use, I don't think most other apps will have a great need for the dictionary.

And then there's the full text index. A full text index is by it's nature a large object as it's space/time tradeoff. For me it's > 24MB.

See above. Can be loaded on demand and most clients won't want to use it anyway.

Besides folder files there are also other files than need to be handled. The config file is a problem if it is desirable to have each client keep their config dialogs and general configuration info in sync.

I'm cool with having that break. If a user is dumb enough to keep the config dialogs of two KMail's open at the same time, then he shouldn't complain if the second-to-be-closed one overwrites the settings of the first-to-be-closed one. I don't see a need to make that work.

Another set of files is the list of mails that are kept on the server for each account.

These are different. Locking is the std answer I'd give here ;-) You have to have locking for accounts anyway. Two instances concurrently checking mail will else break.

Ingo Klöcker agreed that most users won't have more than two KMail instances open at the same time, and also that there's no need to sync if there is more than one config window open at the same time. Which of course turned into a debate, to which there was no resolution.

Don's response also sparked off quite a bit of debate on other subjects, to which there was simply too much to report on here.

5. Kcalc: The killer application?

22 Jul 2003 - 2 Aug 2003 (21 posts) Archive Link: "Kcalc: The killer application?"

People: Klaus Niederkrueger

Disclaimer: I sent a non-thread-affecting post to this thread.

Klaus Niederkrueger had this interesting post:

What do you think about the follwing things:

Of course the question is how to do the latter things, and I would suggest using ginac (http://www.ginac.de).

I don't know what the policy is toward creating such dependancies, but I would argue in the following way:

Who uses Linux? I think mainly pupils and students, and I think this would be a really useful application for them. On the one hand, it should be possible to deactivate all functions so that kcalc looks like a normal pocket calculator that a secretary is able to use, but on the other hand instead of having to look into a book and copy all constants by hand, just press a few buttons and you are done (no need to convert eV to TNT-megatons anymore using tables and such)

Maybe some plugin-system would be cute. Imagine for example typing the price in Euros and converting it to $ (the plugin looks up the last conversion rate on the internet). Or, imagine a distance plugin, you want to have the distance from Bagdad to New York...

Reaction was generally favorable, and a few of these suggestions were unqualified hits, but there were questions about whether kcalc was the correct app to put these features into.

6. Qt 3.2 requirement

23 Jul 2003 - 31 Jul 2003 (130 posts) Archive Link: "QT 3.2 requirement"

People: Ian Reinhart GeiserCornelius SchumacherWaldo Bastian

Ian Reinhart Geiser started this whole fracas with:

Again what is the policy of Qt 3.2? There seems to be more and more Qt +3.2 commits creeping in. Is it time to cut over, or do we need to just try to be more carful of editing UI files and commiting code that is just not there yet in Qt 3.1.x

Waldo Bastian replied that QT3.2 had been released, so he thought it would be a good time to make the switch.

Cornelius Schumacker responded:

I would prefer, if we could keep compatibility with Qt 3.1 for as long as possible, at least until it is part of the major distributions. Requiring a developer to compile a new Qt in order to be able to test a patch for any KDE application in the KDE CVS is, in my opinion, too much. This will prevent some people from contributing to KDE and that's clearly not what we want.

And then it hit the fan. The gist of it is that moving to 3.2 is a good idea but there are some issues that need to be resolved - such as whether QT 3.2 is stable enough to use, whether there's a compelling reason to move to 3.2, etc. It seems the only thing that was not in dispute is that people were already checking in code that relies on 3.2, so the decision may have already been made.

7. Proposal: KProcess API enhancement and konsole

27 Jul 2003 (3 posts) Archive Link: "Proposal: KProcess API enhancement and konsole"

People: Malte Starostik

Malte Starostik said:

the attached KProcess patch adds some functions to check whether the process died with a signal. The konsole patch makes use of it in konsole to get rid display "Session blah exited with signal 2" etc. if applicable. On systems where ECOREDUMP is defined the user is also informed whether the session dumped core. Any objections/portability issues? BTW, is there some function available to translate signal numbers into their names (and maybe with a short textual description)?

Feedback said it was a good idea. A small modification was suggested, advice was to leave it as it is and put in a kde4 todo.

8. KRandr inclusion in KDE

30 Jul 2003 - 10 Aug 2003 (12 posts) Archive Link: "KRandr inclusion in KDE"

People: Hamish Rodda

Hamish Rodda mentioned:

The resize and rotate applications in kdenonbeta/kcmrandr are now imho ready for inclusion in KDE proper. They have significant installed bases already (as part of early adopting distros), and the few bugs filed to date have been resolved. KDE also for the most part behaves well with RANDR; as far as I know the only unresolved issue is that of font sizes in apps started before resolution changes (more to do with qt though).

This software has a run-time dependancy on qt 3.2 and a compile-time dependancy on XFree86 >= 4.3.

Comments? Where should it go...

Responses were either kdelibs or kdebase. Eventually kdebase was settled on, and the directory was moved over. We now get Randr support!

 

 

 

 

 

 

Sharon And Joy
 

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.