Kernel Traffic #193 For 25 Nov 2002

There were 534 different contributors. 290 posted more than once. 220 posted last week too.

Here is a first shot at sysfs for the EISA bus. Nothing spectacular, just some basic probing/registration/naming. It also includes an EISA-ids database, which could also be useful for ISA-PNP devices, if it ever gets sysfs-ized.

I also ported two drivers to this infrastructure : 3c509 and 3c59x.

Tested on i386 SMP and alpha. Will try on parisc and mips later.

Andries Brouwer asked how complete Marc's EISA database was, and suggested putting such a list on a website instead of in the kernel sources. Marc said the database was pretty complete at approximately one thousand entries. He said the database would be required if folks wanted fancy naming instead of just ID numbers. But he really just wanted to get the core code in the kernel, and would be happy to sacrifice the naming database to achieve that. Alan Cox remarked, "I think a ".ids" file list is valuable. It can be used for things like EISA card identification obviously but it also has a big value for "lseisa" "lspnp" and friends (and hopefully when someone fixes the device model "lsdev"." (He later stated clearly that he supported the patch). Andries reiterated that the kernel sources were not the best place for such a list; and proceeded to offer some additions and corrections to the database. Marc thanked him, added some fixes of his own, and said he'd find a different home for the data.

Jeff Garzik said at this point, "I respectfully disagree with Andries. Until drivers/pci/pci.ids list is removed from the kernel source, I think we are best served by modelling EISA on PCI as much as is reasonable." Andries had no objection to that particular point, but did reply:

really, Jeff, these EISA IDs are a pile of junk. So many with the same ID describe different hardware.

I like a certain level of quality in the kernel source. When the kernel prints something it should not be random junk. "Never mind what the Linux kernel says - that is all just nonsense".

A user space utility with a long list is fine: 70% chance that it is right, 30% that it is wrong. A user space utility can print: "with that ID we have seen the following five hardware descriptions".

Since the kernel does not use these values - they are for informational purposes only - I would prefer to avoid the misinformation.

Alexander Viro decided to deliver another devfs anal probe. He reported his findings at length:

During the last couple of weeks I'd done a lot of digging in devfs-related code. Results are interesting, and not in a good sense.

a _lot_ of functions exported by devfs are never used. At all.
a bunch of functions is used only by SGI hwgraph "port". Moreover, a lot of codepaths in functions that *are* used outside of said port, is only exercised by hwgraph. (More on that below)
gratitious arguments (read: all callers pass the same value).
semantics of devfs_register() and devfs_unregister() is, er, suboptimal and leads to rather messy cleanup paths in drivers. (More on that below).
instead of using dev_t, devfs insists on keeping and passing majors/minors separately, which makes both callers _and_ devfs itself messier than necessary.
devfs_entry is a nightmare. It's a structure that contains union (by node type), one of the fields of said union is a structure that contains void *ops, flags, and a union of dev_t (stored as major/minor pair) and size_t. The reason for that (and charming expressions like de->u.fcb.u.device.major) is an attempt to use one field for regular files, character devices and block devices. The *only* thing really common to all of them is set of flags.
idea of "slave" entries (== unregistered when master is unregistered) hadn't worked out - it's easier to do explicit unregister in 3 or 4 places that use these animals.

Overall, I would expect ~50% size reduction in devfs/base.c simply from dropping unused code. The rest would be much easier to debug.

Note that devfs is *seriously* burdened by hwgraph. To the point where it would be better to give SGI folks a private copy (they want slightly different semantics anyway) and merge it with hcl.c and friends. And drop the unused stuff from devfs proper.

Situation when one obscure caller is responsible for ~30% of exported API and pretty much gets unrestrictred access to internals is Not Good(tm). Situation when ~40% of exported API is either not used at all or used only by devfs itself is also not pretty.

Note that there's a large part of devfs that is never used by hwgraph code. IOW, after such split *both* devfs and hwgraph copy would shrink a lot. Shared part is actually rather small and both sides would be better off from clear separation - e.g. locking and refcounting mechanisms should be different, judging by the tricks hcl.c tries to pull off.

Another problem is the semantics of devfs_register/devfs_unregister. rm -rf and install(1) do not match each other, even if they were suitable as primitives (which is a dubious idea, to start with).

First of all, devfs_unregister(devfs_register(...)) is _not_ a no-op. It may leave you with a bunch of new directories that have to be removed later. Moreover, you don't even know how many of them were there before devfs_register() and need to be removed.

What's more, after devfs_register() we are allowed to create objects in the intermediate directories that might appear (we can call mkdir(2) in there, to start with). However, devfs_unregister() wipes these out, which is arguably a wrong thing to do - they were not created by driver, so driver has no business to decide when they should be gone.

The following scheme would give saner behaviour (and deal with devfs_register() failing in the middle of the way, etc. more gracefully):

all entries get two new fields: integer R and boolen V.
VFS creation methods (mknod, symlink, etc.) set R on the new node to 0 and set V to true for that node and all its ancestors. Refcount of new node is set to 1.
register creates nodes with V set to false and increment R on the object we had created and all its ancestors. Refcount of created nodes is set to 1. If we fail to create a node (out of memory) we undo all increments of R we had done so far.
VFS removal methods (unlink and rmdir) fail if R is positive or V is false. Otherwise they set V to false.
unregister decrements R on the victim and all its ancestors.
node is detached from parent whenever (R == 0 && !V) becomes true. After that refcount of node is decremented.
node is freed when refcount reaches 0.
root is originally created with R set to 0 and V set to true.
places that currently grab/drop refcount still do that.

That will guarantee that

once userland creates an object, only userland can remove it or any of its ancestors.
object can't be removed when kernel holds it or some of its descendents registered.
unregister(register(...)) is a no-op
register failing mid-way cleans up after itself
objects are always removed by those who had created them.
as long as driver unregisters all objects it had registered, it doesn't have to worry about intermediate directories, etc.

The price of switching to that scheme is that we will need to switch drivers to explicit cleanups (i.e. instead of devfs_mk_dir "loop" + register <n> in that directory + remove "loop" upon the exit we would register loop/<n> when we initialize struct loop_device and unregister it when we clean struct loop_device - actually, that could be done as side effects of add_disk()/del_gendisk()).

Transition to explicit cleanups can be done before any changes in devfs proper - the sequence is

add explicit devfs_unregister() (or devfs_find_and_unregister()) where needed in drivers; everything keeps working.
add aforementioned fields to devfs_entry and modify devfs_register() and friends (see above). No changes in drivers.
drop a shitload of devfs_mk_dir()/corresponding directory removals from drivers; everything still works.
shift most of remaining calls in block device drivers into add_disk()/del_gendisk(), etc.

I'd estimate that sequence as about a week of work - devfs changes in it can be kept fairly local. And IMNSHO it is needed, since it will make devfs users much cleaner.

Aside of that, there is a bunch of obvious cleanups - e.g. the 6th argument of devfs_find_and_unregister() is (and should be) always 0; 3rd, 4th and 5th arguments are never looked at; the first one is NULL in almost all cases and getting 3 or 4 exceptions into that form is absolutely trivial. IOW, 4 arguments out of 6 are completely gratitious and reducing the thing to devfs_remove(pathname) is a matter of several one-liners.

There's a lot of such cases, but they definitely fall into "obvious cleanups" category. Really critical issues are getting sane model for register/unregister (doable in small steps and I'm ready to do the entire series) and separation of hwgraph - preferably giving it a filesystem of its own with the interface hwgraph wants.

Richard Gooch said he'd address Alexander's points as he found time in the next week or two. But he said that he was worried about breaking driver compatibility between 2.4 and 2.5 kernels. Alexander replied, "It's a bit late for that. Compatibility between 2.4 and 2.5 in the drivers is already broken and devfs won't be anywhere near the top of the list."

Elsewhere, Ryan Anderson asked Alexander if, after all the cleanups he suggested, Alexander himself would be willing to use devfs. Alexander said no, the cleanups wouldn't solve the internal races that still existed. Theodore Y. Ts'o suggested removing devfs altogether, and remarked that not very many folks used it. A number of people piped up, to say they did indeed use it, and liked it. Alexander also said, "If Linus decides to remove devfs, I certainly won't weep for it. However, I don't see any signs of that happening right now, and cleaned interface is less PITA than what we have in the current tree. Right now I'm mostly interested in making the glue in drivers simpler and less intrusive. The fact that it leads to less/simpler code in devfs proper is also a Good Thing(tm)..."

New patches are up on sf.net/projects/acpi. Non-Linux-specific releases will be available by tomorrow evening at http://developer.intel.com/technology/iapc/acpi/downloads.htm .

If you've been having problems with reading battery or other information being very slow, please try this release and report if problems persist.

Stephen Hemminger pointed out that ACPI currently interfered with interrupt request routing, and asked if Andrew's patches fixed that. He said, "On our SMP test machines, ACPI has to be disabled otherwise the SCSI disk controllers don't work. This is a major pain, and ACPI should be default off until it gets fixed." Andrew replied, "ACPI has not yet been adequately tested on machines with multiple IO-APICs. More assistance in this area would be gratefully accepted. In the meantime, "acpi=off" works pretty well to disable ACPI-related configuration problems."

what the hell is going on with the modules code in 2.5-CURRENT?

Rusty's monsterpatch breaks basically everything (and remember we're in feature freeze!) instead of doing one thing at a time, and it is doing three things that are absolutely separate issues.

We had an almost useable 2.5 and now exactly when we're feature freezing and people are expected to test it we break everything?

Linus, please backout that patch until we a) have modutils that support both the new and old code and b) support at least such basic features as parsing modules.conf and supporting parameters.

Rusty, the next time please submit stuff one feature at a time instead of a monster patch that is cool but breaks everything but looks cool.

The inkernel loader, generic boot-time option and your - umm - strange idea of module unload race reduction are absolute separate things.

Quite frankly, at this time a backout means that the thing doesn't go in _at_all_.

It came in before the feature freeze, but I decided that instead of having a totally hectic time I woul dmerge stuff that I got before the freeze at my own leisure, but backing it out now would be basically saying it's not going into 2.6.x. And I think it's worth it.

(There are some other patches I'm still thinking about, notably kprobes and posix timers, but other than that my plate is fairly empty froma feature standpoint. And the kexec stuff I want others to test, at least now it's palatable to me).

People who find the current module situation difficult can just compile in the stuff they need for now.

That makes driver debugging almost impossible. It also makes building a test kernel set for a lot of boxes impractical. The completely broken unload stuff is going to be a real pig, PCMCIA only works modular and doesn't work now the unloads are all broken. OTOH the module rewrite has some nice features and a combo modutils is going to sort some of the problem out fairly easily.

The biggest need though is documentation so people can actually fix all the drivers for this stuff.

Linus agreed that PCMCIA was a serious problem he'd forgotten about. But he added that he thought Alexander Viro had convinced Rusty to maintain compatibility. But Rusty said, "*You* convinced me not to break any driver source code: every time you dropped my patches I went back and implemented another compat macro 8)."

The bugzilla database we proposed earlier is now available for use, hosted by OSDL.

http://bugme.osdl.org

Feel free to go ahead and create yourself an account, and log bugs in there. This is only for 2.5 bugs currently, and the main intent is to help us drive 2.5 into a timely and stable 2.6 (or 3.0).

Please let me or the supplied mailto URLs know of any problems you encounter, but please be patient with any inital teething problems and don't tell slashdot just yet ;-) Apologies for this not being ready quite at Halloween ... and there's not much data in there right now, but we'll keep feeding it over the coming few days, including importing Thomas Molina's list of bugs.

The categories probably need some more work, they'll evolve as bugs get logged. The reason I own huge numbers of subsystems is not just because I'm a derranged megalomaniac, it's more to do with the fact that I don't have other people available with accounts created to own them. Brave volunteers who've created an account would be most welcome, ideally the code maintainers for those subsystems, but other people familiar with those areas would be a great substitute. ;-)

Jeff Garzik suggested a vendor-neutral URL, like bugzilla.kernel.org (later, with the blessings of all, he did set up an alias at kernel.org), and Pete Zaitcev replied:

OSDL is vendor neutral, by definition. Besides, we all know that Transmeta hosts ftp.kernel.org and Red Hat hosts vger (for varying definitions of "hosts", but you know what I mean). I do not see acceptance suffer, because we do not observe Transmeta or Red Hat pushing their agendas. Same with OSDL.

I'm more interested in contacting the admin to be a component owner for sparc, for instance. Someone is going to have a significant admin load, because Bugzilla is not going to be self-running. Who is that person?

Since people asked, I'd like to introduce myself as part of the "staff" that has volunteered some free time devoting to maintaining this Bugzilla; i.e., keeping the database well-groomed. My team actually consists of folks in different IBM locations and time zones: Austin, Texas; Poughkeepsie, New York; Beaverton, Oregon; Bangalore, India, so hopefully, we can keep an "eye" on the database around the clock. For the past two years, my team has been working Linux bugs and contributed bug fixes in support of our internal teams, and now, we have volunteered to help maintain this kernel bug database in our free time. However, we expect that the bug volume logged will be high, so the more people in the community volunteer to help us maintain the database, the better.

Please let us know if you like to volunteer and the Bugzilla administrator will give you enough "power" to do the job (e.g., assigning bugs, closing bugs, screening bugs for duplicates, invalid bugs, etc.).

Also if you have already used this kernel Bugzilla database, you might have noticed that many components are currently owned by Martin or myself. As Martin pointed out in his announcement, this is not because we are "egomaniacs", but rather because the rightful owners (or those who know enough about these components and want to volunteer to work bugs) have not been registered yet. Martin and I will try our best to turn over these components to their rightful "owners" as soon as we can. We are still learning the "ropes" on how to do this effectively, so it will take some time (not too long we hope). Thanks.

Elsewhere, on and off list, folks began carving up subsystems of responsibility. Jeff claimed net driver bugs, and David S. Miller claimed all other networking categories. Arnaldo Carvalho de Melo offered to take over all net/{ipx,llc,appletalk,x25,lapb} bugs, but Martin replied, "We didn't bother breaking those out as they're .... ummm ... obscure, and I wasn't desperately keen to end up with 10,000 categories ;-) They should get dumped into "networking, other" at the moment. These are just the default owners, so bugs can just get reassigned to somebody else if that suits ..." Arnaldo was fine with this, and offered to take over "networking, other", but also remarked, "I guessed that perhaps the reason for creating a category was if there was an active maintainer willing to actually look at the tickets for these a subsystem :-)." David said he was fine with Arnaldo taking over "networking, other".

Elsewhere, Paul Larson asked if bugs should still be reported to their respective mailing lists, or just registered with Bugzilla. Robert Love said bugs should definitely still be reported to their mailing lists for discussion. Pete pointed out that, "Requestors and anyone with an account can add a list to 'cc' field, so updates will get to the list." And Martin added, "Possibly we could create a tree hierarchy of email aliases for people who want to watch certain categories at some point in the future, then people can opt-in as they please." Nicolas Mailhot remarked close by, "at least the initial stage of bug submissions should be cc'd to the relevant lists, so that interested people can get on the problems. This is what apache does (with big fat warnings to not reply to the mail but fire bugzilla instead) and it seems to work well."

I want to express this now since the very first bug that hit my mailbox had this issue.

I DO NOT want to be working on bugs on anything other than Linus's actualy sources. The first bug I got was a networking bug with Andrew Morton's -mm patches applied.

This isn't going to work if that is what people are going to be allowed to do.

I want to suggest that all reported bug in the database must be reporducable with some release done by Linus or his BK sources. And also that we can automatically close any BUG submissions that have other patches applied.

Martin replied, "Hmmm ... I'm not sure that being that restrictive is going to help. Whilst bugs against any randomly patched version of the kernel probably aren't that interesting, things in major trees like -mm, -ac, -dj etc are likely going to end up in mainline sooner or later anyway ... wouldn't you rather know of the breakage sooner rather than later?"

Jeff and David said no, the database should definitely be only for mainline bugs. Eric Northup, close by, suggested simply having Bugzilla name the different trees, so people who wanted to deal with bugs for those trees could do so using the Bugzilla interface. Alan Cox replied, "That works for me. Create a 2.5-ac product that is assigned to me. I can then reassign them all to DaveM as appropriate." Martin said, "that makes sense ... I'll let Jon figure out the best way to acheive this inside bugzilla - Eric's suggestion of version would be nicer, but require some significant mods to bugzilla, I think. Failing that, your suggestion of a new product-type thing would be pretty easy to implement."

This may or may not help but it seems relevant. BK has uniq names for each changeset, we're fixing BK/Web so you can use the uniq names instead of the revs (which change out from under you).

So it should be possible to link the bug report with a changeset in Linus' tree if you want.

It's worth pointing out that if you can see the bug in a particular version of Linus' tree then *everyone* can see it by getting a copy of the tree as of that cset. BK guarentees that if you clone -r<rev> then you'll see exactly what anyone else saw as of that cset.

this is great! We can create a separate field in the bug reports to contain this unique names, so we can reference the cset directly from the bug reports. This allows us to link bug reports to csets -- great!

What format will these unique names be in? If we put them in the bug reports, can we click on them (as URLs) and get to the csets directly? Thanks.

That's the goal. I'm hacking on it it currently, we have some issues with how it works today, I'll try and get a bk-3.0.1 release out the door which fixes them.

The current format is may be seen with a

bk changes -k -r<rev>

where <rev> is the changeset revision you want. You'll get something like this:

torvalds@home.transmeta.com|ChangeSet|20021115061315|00914

That's sort of big and ugly, and it currently doesn't work as a name in BK/Web. I'm debugging an implementation of md5 sums of the above to see if we can use that instead. I'll let you know as soon as I have something which works.

Assuming that we get some format like dSD4okOiGmLGDcqOTpQPFQ== then you'll be able to view the cset with the following URL

http://linux.bkbits.net:8080/linux-2.5/cset@dSD4okOiGmLGDcqOTpQPFQ==

and that will always work and never get you different data.

Thanks. Back when we were setting up the OSDL kernel bugzilla, I thought about having a direct, unique URL link between the bugzilla bug reports and csets in BK, but could not find anything -- like you said, the revs changes as you move the changesets from one repository to another. So I am very happy that you are going to provide a unique URL link for each cset. Please let me know when you get this done and we will have a separate field (called "Changeset Link") in kernel bugzilla to hold this.

We expect that bug owners, after putting their patches into BK, will obtain the cset "name", and enter it into the "Changeset Link" field in the bug report. After hitting the "Commit" button, the kernel bugzilla will translate the cset "name" into a URL link like you indicated above.

This would allow people to view bugs, and if there are already fixes in BK for those bugs, they can get directly to the fixes (changesets).

Has my 2.5 Problem Report Status postings been useful? If so, when I discussed this with Martin one of the roles we agreed I would play was taking bug reports from the list and adding them to bugzilla. I'll also be a "filter" for some of the issues discussed in this thread, sort of a janitor if you will.

My question is how should compile failures figure into the bug database? Most of the compile failures are typos or thinkos that get quickly fixed. Should they get tracked, or dismissed quickly unless they linger on. I didn't track simple compile failures in my list.

Andrew Morton thought it would be great for Thomas to continue acting as a bug collector, but he added that compile failures probably didn't need to go into the database.

Yes it is unless you create toplevel categories for bugs that are occuring on non-official kernel trees.

My expeience so far with this bug database has been that I just immediately close every bug assigned to me, here are two examples:

TCP crash with -mm2 patches --> known error in Andrew's patches
tcp_MSS doesn't compile --> already fixed in current BK tree

the list goes on and on, and the simple fact of the matter is that I have yet to see a _REAL_ bonified bug. If this is how this bug database is going to continue to be used by people, it's going to be of only limited usefullness to me.

Look, if #1 and #2 would have been posted to linux-kernel instead, the fact is that before I woke up and hit my email box SOMEONE ELSE would have responded and even sent that person a patch.

In this sense it appears that linux-kernel is more effective than thus bug database, ESPECIALLY if we are going to allow people to report bugs against trees with random patches applied.

Martin replied that toplevel categories for different trees had already been created; and also remarked, "look at the flip side ... once that bug has been logged in bugzilla, the person who would have emailed lkml now has an easily searchable interface, and could have found the bug, found out what the patch for it was, and fixed it themselves, without ever bothering you, lkml, or anyone. I'm not saying that'll happen 100% of the time, but it should help overall ... will just take a short period whilst data builds up." But Larry replied:

Too much data and the data becomes noise.

I also think that if your goal is to make things progress more smoothly, adding work for people like dave is not a good plan.

This is not an easy problem space, on the one hand you want to have all bugs tracked, on the other hand, trivial bugs in the bug db just make the bug db unusable. No engineer is going to put up with 100,000 stupid bug reports. You need a plan to get rid of those or keep them out of the bugdb or it's unlikely to get used by the people who really need to use it.

Exactly.

It seems to me that only allowing one person to close a bug is going to be the big bottleneck in a project like this. There is no reason the community cannot close the bugs.

It isn't going to scale if it's just one person per category. That simply won't work.

So with that taken care of, basically the database begins to degenerate into a copy of linux-kernel with a nicer search engine :-)

mozilla handles it this way: the bug starts as unconfirmed. they have a volunteer group of pre screeners. Only when one of these people sets it to valid or similar then the owners of the module get mail.

I guess that could work for the linux kernel bugzilla too. Never hazzle a developer, until someone confirmed the bug in some way (this does not mean that he needs to reproduce it, just weed out obvious duplicates and bogus postings)

Currently in the kernel bugzilla, after a bug is filed, it is initially in the OPEN state -- this is similar to the Unconfirmed state mentioned above. The screeners (my team and others who volunteer) can get rid of many invalid bugs and dups. Only valid bugs then go to the ASSIGNED state with correct owners. Of course, we do not expect to get rid 100% of all the invalids and dups, but at least that should reduce the work of the owners who should only work with bugs in the ASSIGNED state.

Also, the bug owner can close MULTIPLE bugs at the same time on Bugzilla. A bug owner can query all of his bugs which will then be displayed in a list, click the option "Change several bugs at once" at the bottom of the list, select the bugs that he wants to close, and then hit Commit button. It's pretty simple. Besides closing the bugs, the owner can make similar changes to several bugs at the same time using the same mechanism.

Kallol Biswas asked, "Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port." Martin J. Bligh suggested, "A cheap terminal server might work ..." And Linus Torvalds replied:

Well, apart from the fact that a lot of machines don't even _have_ serial ports..

I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.

Soon even _desktops_ probably won't have serial lines any more, only USB.

The only real work which has ever been done on this was by San Mehat earlier this year. When he had the advantage of sharing an office with me and being repeatedly harangued to do it ;)

He did have it working - it was basically the same idea as Ingo's netconsole code, using a little polling stub in each driver. He extended the concept to support Rx as well as Tx. You provide a whole bunch of parameters to the kernel and to the debugger, right down to the MAC address.

But San became quite unwell some months ago and vanished. As far as I know, the code is lost.

He may have sent a copy to Amit?

But as far as integrating the stub goes: IMO it would need quite some cleanup work first. Great tool, but the implementation is quite straggly.

An ethernet version could never be as robust as something which spins on a uart port though. Anyone who seriously wants to use the facility would just need to get themselves a 16550.

A number of big guns like Alan Cox piled onto the problem and began going over possible implementation details.

This patch updates UML to 2.5.47.

Functionally, this is the same UML as the last working patch I had, which was for 2.5.44, which was up to UML 2.4.19-12. Since I'm now up to 2.4.19-31, I have a bunch of merging to do, and those patches will be forthcoming.

The patch is available at
http://uml-pub.ists.dartmouth.edu/uml/uml-patch-2.5.47-1.bz2

For the other UML mirrors and other downloads, see
http://user-mode-linux.sourceforge.net/dl-sf.html


Kernel Traffic Latest \| Archives \| People \| Topics	Wine Latest \| Archives \| People \| Topics	GNUe Latest \| Archives \| People \| Topics
Czech

1.	10 Nov 2002 - 17 Nov 2002	(16 posts)	Support For sysfs For The EISA Bus
2.	10 Nov 2002 - 14 Nov 2002	(15 posts)	Cleaning Up The devfs API
3.	12 Nov 2002 - 18 Nov 2002	(7 posts)	Status Of ACPI In 2.5
4.	13 Nov 2002 - 15 Nov 2002	(20 posts)	Status Of Module Support In 2.5
5.	13 Nov 2002 - 18 Nov 2002	(93 posts)	Bugzilla Bug Tracking Database For The Kernel
6.	15 Nov 2002 - 19 Nov 2002	(45 posts)	Kernel Debugger May Go Into The Main Tree
7.	16 Nov 2002	(1 post)	User-Mode Linux 2.5.47
8.	17 Nov 2002 - 19 Nov 2002	(17 posts)	Linux 2.5.48 Released
9.	20 Nov 2002	(1 post)	JFS 1.1.0 Released

Kernel Traffic #193 For 25 Nov 2002

By Zack Brown