Kernel Traffic #31 For 19 Aug 1999

I just decided to merge what I had for last week into this week's issue. Some good threads and interesting sub-threads were missed or given short shrift. The upshot is that I'm learning how to organize my time better. I'll also be moving KT from Thursday morning to Saturday morning. So it'll be coming out Friday Midnight starting now.

BTW linux-kernel is under attack by a malicious user who is subscribing it to many unrelated mailing lists. The problem seems to be under control. Check back next week for the full details.

There were 386 different contributors. 151 posted more than once. 107 posted last week too.

Jeff Dike gave a pointer to http://www.mv.com/ipusers/karaya/ossf/ossf.html, which expanded on a number of ideas he felt might be worthy of implementation. He listed them briefly in his post.

Scalability Through Specialization

He suggested, "On an SMP system, when a critical section of code is being hit often enough that around one CPU equivalent is being lost to the locking overhead, then it might be worthwhile to have one processor stop running processes and start running a thread which does nothing but handle that critical section."

Stephen C. Tweedie objected, "If there is that much contention for the code then the resulting cross-CPU communication will be at least as expensive as the lock contention you were suffering in the first place!"

Making Idle Threads Earn Their Living

Under this heading, Jeff said, "This involves finding parallelism in the kernel and getting an otherwise idle processor to handle one of the parallel chunks of work." Stefan Monnier objected, "Hopeless. The cost of inter-CPU communication is way too high to try to share such fine-grained tasks." But Jamie Lokier took up the idea of making use of idle processors, saying:

I reckon a spare CPU could happily

pre-zero some pages, preferably spread over the range of page colours.
move pages around to make space for multi-page allocations.
checksum pages to find duplicates and merge them in the background

Elsewhere, Stephen C. Tweedie agreed, saying, "Yes, some of the non-Intel Linux porting projects have tried using the idle thread to pre-zero pages for get_free_page(), and have seen big performance improvements as a result."

Compressive Paging

Jeff suggested in his original email, "Before writing a page out to swap, compress it." Stefan Monnier replied that this had "already been tried in a few cases, and it has some definite advantages in a few cases. But it also has some nasty difficulties. Sadly every experiment I know of ended up stopping as soon as some code worked and showed little improvement."

To Jeff's original post, Stephen C. Tweedie replied, "Ouch, I'd hate to see the CPU cost. This is fine for a single-user box with one swapping task, but gets really expensive as soon as you are multitasking."

Elsewhere, Matthew Sayler said that people interested in the subject should check out http://www.cs.utexas.edu/users/oops/compressed-caching/index.html; he added, "There is quite a bit of research material there (some of which was presented at the most recent USENIX). I know that one of the authors was planning to hammer out an implementation for Linux this summer."

IPC Through Page Remapping

Jeff said in his original post, "Moving data from one process to another by remapping pages would be a lot more efficient than by copying the data."

Stefan Monnier replied, "Slow. Only worth it if the message is big enough (last time I heard, the cut-over point was around 8KB)." Jamie Lokier expanded, with, "Usually the overhead of TLB flushes or the need for inter-CPU TLB flush messages is given as the reason. But there are cases when those can be shown to be not necessary. Like when you read a page, and the pte is not marked "accessed" yet. No TLB flush required. Similarly when you write a page and its marked "dirty" but not "accessed" -- I'm not sure if this combination (written but not read) is flagged by CPUs though."

Stephen C. Tweedie replied to Jeff's original suggestion, "No, VM tricks can be very expensive (especially with threaded programs on SMP, where every VM update has to involve a cross-CPU interrupt to invalidate TLB caches on other CPUs). We actually tried this for pipes once and it wasn't that good." Jeff replied, asking if there were any references on the costs of interprocessor communication, and Stephen said there weren't (aside from the linux-kernel archives themselves).

Runaway Process Detection

Jeff also suggested originally, "Detecting runaway processes and putting them to sleep might be a nice little feature to add to the kernel. The web page contains what I think is a correct algorithm for determining that a process is a runaway."

Stephen C. Tweedie warned, "For every such definition you'll find a genuine program which triggers the killer. :)" and Stefan Monnier also replied to Jeff, "Excellent way to bloat the kernel with code that catches about 1% of the problematic cases, which happen to be the least annoying cases (your runaway processes will simply eat up some of your CPU time, but will have a very low priority, so it doesn't slow down the machine too much, is trivial to kill, ...)."

An Extra Channel For Kernel Messages

Jeff had also suggested, "/proc/<pid>/kmsg would be a good idea. This would contain messages pertaining to a process which are not important enough to go into the message log, which might not be useful in five minutes, but which the user might be interested in now."

Stephen C. Tweedie replied, "dmesg/syslog already offer this: you can specify messages to be of debugging priority and only log higher priority messages to syslog. "dmesg" will tell you all of the recent unlogged messages."

Jeff replied, "That works (on my system), but I don't think it should. /bin/dmesg is 755 and syslog(3, ...) is allowed for normal users, which allows them to see stuff that maybe they shouldn't." He added, " If this really is a bug, then it would be useful for normal users to see what the kernel has to say about their processes." There was no reply.

A Pseudo-Filesystem For Exposing Internal Process State

Jeff said, "This would be a uniform, generic way to attach real-time monitors to cooperating processes." Stephen C. Tweedie replied, "Ooh, like /proc fs?" Jeff said, "Should my sarcasm detector be firing here :-) /proc contains stuff that the kernel maintains. I was referring to very application-specific stuff like what function gcc is currently compiling or make's dependency tree and where it is in updating it." Stephen replied, "No, I'm serious. /proc is extensible by design. Don't do another filesystem when you can extend procfs."

Speculative I/O

Jeff said, "When a process does a read, the kernel could schedule the I/O and return success before it completes. This would allow the process to start work before the read has finished."

Most folks didn't see any difference between this and "asynchronous I/O", and it came out that some work on asynchronous I/O was in progress (by Arjan van de Ven).

Chris Smith tried to distinguish between Jeff's "Speculative I/O" and "Asynchronous I/O", saying he felt people were missing Jeff's point. He explained, "As I see it, the big difference between "speculative" I/O and asynchronous I/O is the interface. Speculative I/O would appear to be a blocking I/O call, but as soon as it queued the I/O request, it would return. The result page would be marked as unreadable and unwriteable, and the actual I/O blocking would be done on a page fault."

there's a good paper related to this published in the OSDI '99 proceedings. see:

Chang, F., Gibson, G., "Automatic I/O Hint Generation through Speculative Execution," Proceedings of the Third Symposium on Operating Systems Design and Implementation, pp. 1-14, February 1999.

the authors used speculative execution during I/O waits to predict future read requests instead of the method you described. their conclusion was that were good gains from speculative execution, but the technique requires significant O/S complexity.

Jeff said Chris' analysis was correct, saying, that had been "why I called it speculative rather than asynchronous. It is asynchronous I/O under a synchronous interface."

The discussion went on for quite awhile, and ranged over a number of operating system concepts.

This thread started off as a description of some filesystem corruption, but quickly veered off into general discussion of Linux, uptimes, and Microsoft. The (to me) most interesting part of the discussion was about Andre Hedrick's code for UPSes made by APC. It started with a brief discussion of whether to buy UPSes from APC or Best Power; it came out that Best supports Linux directly, while APC doesn't. Peter A. Castro pointed out that APC tech support did give him the URLs of Andre's work.

Yes, they are kind enough to supply you with the address of the code that will allow them to sell you a UPS. What they don't tell you was that they threatened Andre with legal action for his code, which caused a BIG rift on this mailing list last year (Oct-Nov 98). Despite the company trying to squash Andre's program, they surely didn't mind making money off selling UPS's and pointing out Andre's program's existence.

Finally after much discussion with APC, etc... Andre released the code back under GPL basically to "call their bluff" around April 7th I believe. They realized they had no legal footing and as far as I know they just shut up.

They are NOT a Linux friendly company however - despite what they may say on the phone. They'll say whatever you want to hear if you'll buy one of their products.

Paul Jakma added, "In contrast to APC and Andre's struggle with them, if you go to Best's website you can download their UPS software. The Unix version is available as source."

Andre came in with, "For the record, I never had any internal docs/specs on their product line. They were amazed that any idoit would lob characters at the device and see what happens." But he also said, "go easy on APC.......I still have high level access and open discussions to continue a convincing dialog on the merits of opensourse."

Ross Mikosh asked if there was a patch against 2.2.x that would allow accessing more than 2 gigs of ram. Benjamin C.R. LaHaise replied, "If you're talking about the 36 bit memory support, such a patch does not exist yet -- to the best of my knowledge it's still in the 'napkin' stage."

Ross replied, "But also, is there a patch which provides an unsigned 32 bit memory address space for each process? (as opposed to a signed 32 bit address, which I'm finding in all the 2.2.x kernel updates). So the maximum amount of addressable memory would be 4 GB."

Victor Khimenko gave a pointer to http://www.linux.eu.org/Linux-MM/ and said, "Unfortunatelly it'll need DEEP rework of linux kernel. And by DEEP I mean DEEP," and Ben explained:

That would be even harder to accomplish as you'd have to change page tables upon entering any exception handlers, copy_*_user would have to be kludged around... Not the sort of messiness likely to ever make it into the mainstream kernel (and less likely to be developed). The general consensus that keeps coming up is that if your needs are that great, you'd probably be better served by a 64 bit architecture where the fixes are much simpler (ie dealing with 32 bit pci dma issues).

The 36 bit support for ia32 should allow for something close to what you want -- it would be possible to make kernel address space smaller, reducing memory for caching, but increasing the virtual address space which could be filled with high memory.

I think you must be mistaken; by default Linux supports 3GB of virtual memory and 1GB (- epsilon = 960MB, iirc) of physical RAM. It's possible to change that to 2GB + 2GB, but it isn't possible to get 4GB of linear addressed space on x86. If you need that, get an Alpha.

Even the proposed 36-bit extensions won't allow this -- they merely allow the system to keep around many more anonymous pages which you can't use directly. Fortunately, this is exactly what big databases want.

Someone posted off-topic, and Chuck Mead suggested that linux-kernel be set up to prevent non-subscribers from posting. Nat Lanza pointed out, "While that would help prevent spam, it'd also keep non-subscribers from sending bug reports to the list, which wouldn't be such a good thing."

The user-mode kernel is now a 2.3.12 kernel, and I plan to keep it current with the latest development kernels. It is now possible to log in on the console, and there were also some improvements made to the serial line. All the details are available at http://www.mv.com/ipusers/karaya/uml/uml.html.

Below is a patch to ptrace that I would like to get into 2.3. It allows ptrace to change system call numbers. I need it for the user-mode kernel, and Michael Elizabeth Chastain needs it for his trace-and-replay debugger. I'd appreciate it if people would look at it and let me know about any problems.

Someone asked if there were man pages that covered kernel functions. Neil Moore gave a pointer to a man9 (kernel functions) snapshot, and added, "They haven't been updated in a long time (since November 1997), so they are vastly out-of-date. Also, they are very incomplete, and were so when they were written. I wrote a total three manpages, then ran out of free time approximately when I started uni. Apparently, everyone else ran out of free time or interest, as well. There are a total of around 10-20 man pages, not counting duplicates. I'm sure that many of them document interfaces that are now nonexistant or completely different. The project was run by Stephen Williams; you may wish to contact him for more information."

Dan Hollis found a documentation bug in linux/Documentation/mca.txt, which referenced http://glycerine.itsmm.uni.edu/mca/ as the MCA Linux home page. Dan had noticed that the hostname didn't seem to exist anymore. Alain Schroeder gave the new URL as http://www.dgmicro.com/mca/ and Riley Williams gave a pointer to the explanation.

Jeff Garzik gave a link to an article on PC-TEL's LinModems, and Mike A. Harris replied with biting sarcasm, trashing the whole idea of peripherals shunting their work onto a system's main CPU.

Jeremy Fitzhardinge disagreed, saying his laptop would be much more useful with a LinModem, and Steve Underwood pointed out that the lower cost-of-production would make a big difference to manufacturers of $200 Internet-access PCs.

Feature freeze in about two weeks is the current plan.

In short, people who think they have major requirements had better get their act together. That means that if ISDN people actually want to try to get into a real release one of these years, they don't have all that much time to futz around any more.

And it does mean that if it's not in working order already, it probably won't make it into 2.4.

I have a 2.3.12 running with the current isdn4linux cvs tree. I will get crasy if my work again will not find the way into the main stream kernel.

Why is it a problem to put this stuff into the kernel ?

What is the actual way to put changes into the actual kernel for isdn4linux? What can I do to fix that problems ? Should I sent a patch to 2.3.12 to you or someone else ?

I don't know why the ISDN people just do not ever GET IT.

I refuse to get huge patches every time two weeks before a feature freeze. How hard is that to understand?

I will continue to ignore the ISDN CVS tree. If the ISDN people cannot get their act together and actually start sending their fixes to the standard kernel in a timely manner, I cannot be bothered with it. This has continued for something like five years now, and every time I explain it to people my explanation gets lost or ignored.

It's not about sending "a patch".

It's about being a responsible programmer, and making sure I get MORE than just a patch every f*cing time I anounce a code freeze. I should have been getting patches for the last half year, and I have gotten ZERO. And I'm irritated at the ISDN peopl, because this is not the first time. In fact, I have never EVER gotten a single responsible ISDN developer that stands up and says "ok, I'll be the maintainer of this, and I'll make sure it gets fed to Linus in a timely manner".

You still wonder why ISDN is not there? Do you still wonder why I'm fed up with ISDN people who think they they can just force-feed me one huge patch and completely avoid any QA in the meantime?

WHY, oh why, is the ONLY time I ever hear from ISDN people when I have publically announced a code-freeze? Explain that.

Paul Slootman replied that the ISDN code was so big that maybe there was no alternative to at some point just accepting a big patch and moving on from there. He volunteered to be Linus' contact to the ISDN developers, and feed him regular patches after the initial mega-patch. About the lack of QA, Paul said, "I have to disagree there. QA is *not* being completely avoided. In fact, I think that the majority of people using ISDN in kernels later than 2.0.36 are using the CVS code. SuSE for example have it in their standard kernel; the rest find out they need it when the existing code simply doesn't work for them. Hence IMHO the "CVS" code (or at least certain snapshots of it) has been tested quite thoroughly. I use it at work to control a dial-on-demand link to the internet, and offer callback links from home to about 10 people. It's been up since I booted 2.2.9-ac4 + ISDN CVS code 59 days ago now, it's made 2674 calls with 302 hours of connect time since then. I'd say it's pretty stable :-)"

In response to Linus' "why oh why" lament, Paul added, "My opinion is: All the core ISDN developers are german, and are perhaps not that fluent in english. That may be the reason why they're hesistant to contact you directly; understandable, I have a mental block to overcome whenever I have to write a german email, for example. When you announce a code-freeze, urgency overcomes their hesitance. Don't see this as an attempt to excuse it, however. It should have been avoided."

Oh, I agree. That much is obvious. In fact, I've accepted some of them over time.

The reason I'm bitching is not just because I love biting peoples heads off in public, it's because I do not want to see the ISDN incompetence continue. I want people to be aware of the problem, and I want somebody to stand up and say "I'll do it".

I can accept large patches initially, but I want to do so in the solid knowledge that it's the last time EVER I have to wait for ISDN code until the last moment. There's obviously no way we can fix past problems, I just do not want them to repeat every single release cycle like they have so far.

It needs to be more than one person agreeing, but yes, I'd love to have a contact person. You need to make the other ISDN developers accept you as more than just the person who forwards patches to Linus. The problem is that sometimes I do not like a patch, and then there needs to be some kind of feedback cycle. It doesn't happen all that often when it comes to drivers (I just don't care enough - I mainly care about the fundamental interactions in the system rather than specific drivers), but it does happen.

I would certainly =love= to not have to hate ISDN support.

It's not getting any peer review, nobody looks at incremental patches, and nobody really seems to care.

In short, the ISDN code in Linux acts as if it was a traditional code project where the users might as well just have binaries.

The point of open development is that people see what's going on. You don't get that if people see just the end result after a year. You want to have random people just see small updates - because they will often catch silly mistakes.

Now, with huge mega-patches, people just go numb. They say "oh, an ISDN update", and skip it.

With the regular "let's release this as it is developed" support, there have been web-sites with commented patches, people who read the incremental stuff and comment on stupid things I and others do. No, it doesn't happen all the time, but it does happen. ISDN for Linux lacks that completely, and acts as a closed development environment.

Paul replied that he was CCing the ISDN list, and was hopeful about solidifying a position as laiason. Rogier Wolff pointed out that some folks might be trying to take some of the burden off Linus' shoulders, by not giving him so many patches, but saving them up into larger chunks. He added that he felt QA was being done for ISDN, but just on different mailing lists. Linus replied:

Common mistake: peer review does NOT mean that the code should be looked at by the same people who write it.

Peer review is _meaningless_ under those circumstances. The whole point of getting peer review is to find _different_ people who have a different background to look at your code.

It also came out that the ISDN mailing list was closed, at which point Rogier took back his argument.

Elsewhere, Hubert Mantel incurred Linus' wrath when he said, "So the huge ISDN update in proposed-2.2.11 will be removed from the final 2.2.11 release?"

[ Irritated mode: FULL BLAST ]

Are you being stupid on purpose, or were you born that way?

Go back and read the thread. Read why I'm irritated. READ. THINK.

[ Irritated mode: NORMAL ]

I'm really disgusted with ISDN development. I'm disgusted with the fact that I don't _ever_ see any development until after code freeze announcements, which is at a point where it is basically too late.

I'm used to that happening. It happens with other drivers too. The thing that makes ISDN stand out is that with other drivers it happens ONCE.

With ISDN, it has happened EVERY SINGLE DAMN RELEASE TIME! For the last FIVE years or so. I've complained before, and every time my complaints are apparently ignored, and result in stupid comments like the above.

It is over. I refuse to see this continue. The ISDN people need to open up, and stop the ridiculous practice of having closed lists and closed releases, and not feeding them back until it's basically too late. The ISDN people need to get on with the program, because it doesn't work the way it is handled now.

We had this exact discussion during the 2.2.x release thing, and yet people seem to not have woken up. I don't know what I have to do to make people understand. I'm considering asking SuSE to stop funding any development that doesn't find its way in a timely manner into the standard kernel, for example.

I know there _are_ people at SuSE who understand that the current ISDN development model is a disaster for releases. Maybe you should talk to them.

Rik van Riel said, "Unfortunately, the phone companies don't allow you to connect to the network with uncertified hardware/driver combo's. Opening up the ISDN development could bring all sorts of legal trouble with it :("

Not really true.

It's true in Germany, not in many other places. Germany is one of the few countries in the world who still have a rather strong monopoly on phones, although it finally seems to be crumbling there too, as even the Germans are growing tired of bad service and exorbitant fees ;)

And even in Germany, it's only true of drivers that do everything in software: most modern ISDN cards have the actual connection smarts in firmware and do not need the same certification (well, they do, but not on the OS driver side: now it's a hardware certification issue). So only a rather small subset of the ISDN code is actually affected by these rules, and its' fairly easy to say something like "for these cards, it is illegal to connect the machine to the phone line if this part of the code has been changed".

I think it's basically just one or two drivers, and a subset of the driver at that.

And quite frankly, let the people vote with their feet. Civil disobedience is not always a bad thing, as shown by people like Gandhi. Bringing down bad phone monopolies may not ever count as highly as getting the British Empire out of India, but let people decide on their own whether they should just bend over and take bad rules.

So, what do you suggest to remedy the situation? The ISDN updates are dearly needed by the masses, your argument (which I completely agree with) with the ISDN developers non withstanding.

What about if you both get over it: You _once_ accept the big patch so we can resolve the situation right now, and the i4ldevelopers promise to feed you smaller patches in the future so it never occurs again.

For 2.2.x we don't have much choice - either it dosn't get updated at all, or it gets updated in one large fell swoop.

For 2.3.x the choices are also fairly limited. Most of them basically involve one large initial sync patch, and then future patches in a more timely fashion. I don't think we can synch up reasonably any other way, although it might be nice to have the initial larger sync be done on a per-driver basis at least to some degree.

The thing is, that before I accept the large sync, I really want to get the issues out in the open, and I really do want to make sure that when the 2.5.x development tree opens, the ISDN code won't be in the same shape as it is now, with large pending patches.

For example, I really want ISDN developers to try to be as disciplined as other devlopers are during a feature freeze. Instead of thinking "Oh, the standard kernel has a feature freeze, so we'll just do all our development in the CVS tree and sych up in one large go", the ISDN people too should understand what the feature freeze is all about: testing a stable system, and making sure that the bugs at that time get ironed out and _nothing_ else. No secret CVS tree activity on the side etc. It's about hammering something out that you can live with for a year or two there-after WITHOUT telling people to upgrade to some completely new CVS tree version.

Now, "disciplined as the other developers" is probably an oxymoron. None of the Linux feature- or code-freezes have been all THAT disciplined, but at least people tried, and we basically succeeded reasonably well in the end.

And remember: feature-freeze and even code-freeze does not mean that bugs don't get fixed, it just means that people really ONLY concentrate on finding the bugs. It should be something ISDN developers need to do occasionally too, and this all really shouldn't be all that big of an issue. It's just that historically it has really not worked at all..

Frank van Maarseveen and Trond Myklebust discussed some potential NFS trouble. Frank posted the following exploit:

Run this on the client in a separate window:

        while usleep 100000; do ls notfound; done

Go to the server and create the "notfound" file. The client script will never see this file due to caching. When the usleep is increased towards 1 second then the file will be found in most cases

His solution was to disable negative dentry caching, forcing failed inode lookups to be redone rather than be fed data from a cache. Trond said that doing so many lookups would put an unreasonable burden on the network. He added that Linus Torvalds had rejected a patch for the 2.2 series on those grounds.

They went back and forth on the issue, and Kurt Garloff pointed out that the file was never being seen in Frank's exploit, which was clearly a bug. Trond agreed, and there was some confusion about whether the problem was in the stock 2.2.10 kernel or in Alan Cox's 2.2.11-* patches. Steven N. Hirsch asked if the discussion was specific to the -ac and/or the NFS patches.

Trond said it seemed to be only about the -ac patches, and that as far as he knew, the stock 2.2.10 and NFSv3 were both doing the right thing.

Johannes Erdfelt wanted to use a kmem_cache in the UHCI module, but felt that Andrea Arcangeli's comments in mm/slab.c were a little unclear. They seemed to say that modules should keep a pointer in unloaded memory (never a good idea).

I've been meaning to fix this, the way to really do it is:

Never require that a slab gets destroyed. Even after the module unloads, the cache will be shrunk naturally as the system trims SLAB caches all the time when memory gets tight.
Have kmem_cache_create copy the string over into a new buffer, so the reference to the string in the module doesn't exist. In fact make kmem_cache_t->name[64] or whatever so that it will still work while SLAB is setting itself up.
Have kmem_cache_create check to see if the slab exists already, by comparing the name and the parameters. If they match, just return the old slab cache pointer, don't make a new one.

This will solve all the problems in one go, and localize the changes to one spot.

Johannes Erdfelt volunteered to write the patch, if Dave would accept it, and Dave said he would.

Guys I implemented kmem_cache_destroy() long ago!! I posted a patch alone with explanation plus testing and testing kernel module as well!! Since it's been _ignored_ by _everybody_ out there (even from people like AV who asked for it) it stayed long time into my 2.2.x andrea-patches (as tons of other good stuff!!).

Luckily at that time I did a directory in my ftp area (unlike for the other good stuff). I am pretty sure the patch will apply cleanly on 2.3.x as well since the slab is been unchanged over the time.

        ftp://ftp.suse.com/pub/people/andrea/slab/slab-destroy-2.2.2
        ftp://ftp.suse.com/pub/people/andrea/slab/slab-test-0.tar.gz

andrea@penguin:/home/ftp/pub/andrea/slab$ ls -l
total 9
-rw-r--r--   1 andrea   andrea       6222 Feb 25 20:31 slab-destroy-2.2.2
                                          ^^^^^^
-rw-r--r--   1 andrea   andrea       1823 Feb 25 20:33 slab-test-0.tar.gz
                                          ^^^^^^

Please apply it!! (if you want a detailed description of the patch browse linux-kernel of february or ask me to browse linux-kernel and forward you my original email of february).

Dave said, "And as I described it's the wrong answer to this problem, instead kmem_cache_create needs to be made more intelligent. Adding a new interface for cache destruction requires changes to all users of SLAB, making SLAB creation more intelligent does not."

I've uploaded the 2.2.11pre6->7 patch to ftp.kernel.org:/pub/linux/kernel/alan

This one contains:

Second set of merges from DaveM
Alpha compile fix
SIS 900 PCI ethernet driver

This should mean 2.2.11pre7 is the final ready for 2.2.11. I've got various other patches from people queued because I want them to be in 2.2.12 so that 2.2.11 doesn't get too many non driver changes.

Please test this one hard folks.


Kernel Traffic Latest \| Archives \| People \| Topics	Wine Latest \| Archives \| People \| Topics	GNUe Latest \| Archives \| People \| Topics
Czech

1.	11 Jul 1999 - 6 Aug 1999	(85 posts)	Operating System Ideas Discussed
2.	20 Jul 1999 - 3 Aug 1999	(60 posts)	Uninterruptible Power Supplies
3.	30 Jul 1999 - 3 Aug 1999	(9 posts)	Large Memory Systems
4.	30 Jul 1999 - 1 Aug 1999	(4 posts)	Off Topic Postings
5.	3 Aug 1999 - 4 Aug 1999	(2 posts)	User-Mode Kernel Update
6.	4 Aug 1999	(4 posts)	Man Pages For Kernel Functions
7.	4 Aug 1999 - 5 Aug 1999	(4 posts)	Linux-MCA Homepage
8.	2 Aug 1999 - 13 Aug 1999	(35 posts)	LinModems
9.	3 Aug 1999 - 10 Aug 1999	(44 posts)	Code Freeze; ISDN Perennial Lateness
10.	5 Aug 1999 - 8 Aug 1999	(13 posts)	NFS Exploit
11.	6 Aug 1999 - 9 Jun 1999	(15 posts)	Development Conflicts
12.	8 Aug 1999 - 9 Aug 1999	(7 posts)	Linux 2.2.11pre7