Kernel Traffic #244 For 8�Dec�2003

The attached patch makes depmod to output modules.pnpmap file generated from the pnp device table.

The output format is not compatible with the old modules.isapnpmap. The new format shows the pnp id string (e.g. CTL0301) while the old format uses the hex numbers. I don't think it's worthy to keep the compatibility for this (since the new one is more intuitive), but it'd be easy to follow the old style.

That seems strange. If you don't worry about backwards compatibility, then the new scripts/file2alias.c approach is better, which generates aliases for each module (depmod then collects these into /lib/modules/`uname -r`/modules.alias for speed).

The tables generated by depmod are purely for backwards compatibility, although it does look like they will be required throughout 2.6 at this stage.

Does that clarify?

first i'll try to add the support of old isapnp format for compatibility, so that old programs can work as they are.

the file2alias format of (isa) pnp devices will need variable number of items, since a driver may require multiple ids. for example, snd-cs4236 driver supports the cards with three ids like

CSCe825:CSC0100:CSC0110

and four ids like

CSCd937:CSC0000:CSC0010:CSC0003

in each case, a matching card must include all ids listed there.

well, i'm not sure which identifier (separator) letter in which style should be used. something like

pnp:idXXXxxxxd0XXXxxxxd1XXXXxxxx

?? separators including a number might be a bad idea, though...

the attached is the patch to add pnp entries to file2alias.c. i moved the definitions of pnp_device_id and pnp_card_device_id into mod_devicetable.h as other devices do. if you don't like it, i'll try to revert them and put definitions inside file2alias.c to keep the changes minimum.

the format of pnp alias is:
pnp:dXXXYYYY
or
pnp:cXXXYYYYdXXXYYYY[dXXXYYYY...]
where XXXYYYY is the pnp id with 7 letters (e.g. CTL0031), c shows the card id, and d means the device id. multiple device ids will be listed depending on the driver.

for example,

alias pnp:dYMH0021* opl3sa2
alias pnp:cALS0001d@@@0001d@X@0001d@H@0001* snd_als100

Andrey, would it be feasible for hotplug stuff?

Will then every d be passed as separate parameter to hotplug? It means agent has to deal with unknown number of parameters or are there always fixed number of devs? apparently not as max is 8 and in your example only 3 are defined.

If number is variable I guess better would be

alias pnp:cXXXXXXdYYYYYYY[:YYYYYY...]

i.e. put all devs IDs in one field; actually may be even separator is redundant as IDs have strict format to my knowledge.

Then hotplug agent gets two parameters - PNPID and PNPDEVS - and it is quite easy to build alias.

I think it it high time to get the SIIMAGE.C 1.06 bugfree. It seems to have some problems with my HD+SATA converter mainly performance wise. I htink it is due to this error coming constanlty in dmesg unless I comment it out in the sources:

hde: sata_error = 0x00000000, watchdog = 0, siimage_mmio_ide_dma_test_irq

What is the problem? I think it may be due to the fact that I have an SiI3112A controller onboard and the driver detects it without the A revision (just as SiI3112). And/or it is due to the fact that I connected a PATA drive with a SATA converter to the controller.

Now with 2.6-test10 the performance got a bit better in comparison to test9 and prior 2.6 kernels. Before it was max 22MB/sec and now it is 25mb/sec. But it is still far away from 2.4.22-ac4 kernel which managed 37mb/s, which is still bad in comparison to windows which reaches 50mb/s.

It is NOT a problem of read-ahead. I tried various hdparm parameters and it didn't improve the situation. What makes the situation even worse:

When I try hdparm -d1 /dev/hde (though hdparm sates dma is already on) the drive stops working and I get some lines of erorrs like drive-seek error and some irq related stuff. So I have to push the button.

Someone else using a native SATA Maxtor on Sil3112 (dunno whether A or not) has no problems, hdparm -d works as well and he gets 40mb/sec with test10.

So what may be the problem, and how to get rid of it? (1. error message, 2. bad performance, 3. hdparm -d1 malfunctioning). 1 & 3 were also with 2.4.22-ac4 and 2 wasn't that bad, as stated above, so except 2 there is no regression, but also no fix yet. Changing max_kb_per_request didn't help either.

Holy Shit!

I just tried the libata driver and it ROCKSSSS! So far, at least.

I already wrote about the crappy SiI3112 ide driver, now with libata I get >60mb/sec!!!! More then I get with windows.

Also tests with dd. This rocks. Lets see whether it likes swsup, as well...

So folks, try libata, as well.

I dunno what all is actuall needed. I enabled scsi, scie disk, scsi generic, sata and its driver. In grub I appended "doataraid noraid".

YES!

I can't find the Silicon Image driver under

"SCSI low-level drivers" -> "Serial ATA (SATA) support"

under 2.6.0-test11. Just the following are there:

ServerWorks Frodo
Intel PIIX/ICH
Promisa SATA
VIA SATA

So, which kernel do I need?

Okay, stop bashing IDE driver... three mails is enough...

Apply this patch and you should get similar performance from IDE driver. You are probably seeing big improvements with libata driver because you are using Samsung and IBM/Hitachi drives only, for Seagate it probably sucks just like IDE driver...

IDE driver limits requests to 15kB for all SATA drives... libata driver limits requests to 15kB only for Seagata SATA drives...

Both drivers still need proper fix for Seagate drives...

for everybody who thought "stoned beaver" wasn't an appropriate name for a kernel (yeah, I'm sure IBM really minds my naming scheme, and is deathly afraid it will scare away customers), I'm happy to tell you that the beaver just went into detox, and I'm taking the Thanksgiving weekend off.

I give you "Beaver in Detox", aka linux-2.6.0-test11. This is mainly brought on by the fact that the old aic7xxx driver was broken in -test10, and Ingo found this really evil test program that showed an error case in do_fork() that we had never handled right. Well, duh!

While at it, this also pulls in some firewire fixes and a few potential skbuff leakage points.

Please don't even bother sending me patches, because I'll be stuffing my face away from email over the next few days. And after that it will be up to Andrew to say how to go on from here.

Mmmm. Turkey.

This is a forward port of Hugh Dickins' patch to implement ABI- preserving large software PAGE_SIZE support, effectively "large VM blocksize". It's also been called "subpages". "pgcl" is an abbreviation for "page clustering", after the historical but different BSD notion.

This is meant to make memory management more efficient by reducing the number of objects to manage, as well as establishing more physical contiguity of memory by keeping it pieces larger than what individual ptes map. This very noticeably reduces the space requirements for mem_map[]. I hope to eventually demonstrate further advantages like larger fs blocksize support and reduced sglist length requirements.

This release features a rewrite of all the fault handling logic, done to incorporate a forward port of hugh's original fault handlers. This should perform much better than prior releases as they efficiently implement COW unsharing's trivial case and have bounded search overhead, though there is clearly a lot of room for further optimization.

Tested for basic userspace functionality on a 256MB Thinkpad T21 and a 32GB NUMA-Q with 32KB PAGE_SIZE. It does break when you push it, though it does run init scripts, most programs, and some benchmarks here.

Available from: ftp://ftp.kernel.org/pub/linux/kernel/people/wli/vm/pgcl/

Incremental patches for all stages of the rewrite as well as cumulative diffs vs. 2.6.0-test5-bk3 and 2.6.0-test5 are also available there.

Errata:

1. the CONFIG_PAGE_CLUSTER==0/PAGE_MMUCOUNT==1 case is nonfunctional
2. some oopsen zwane found while running KDE
3. some mysterious preempt imbalance(s)
4. drivers and fs's are essentially totally unaudited, probably broken
5. non-i386 are all broken
6. CONFIG_DEBUG_HIGHMEM is nonfunctional
7. CONFIG_DEBUG_PAGEALLOC is nonfunctional
8. many, many more

TODO:

1. merge to current
2. sweep drivers/fs's
3. optimize and rework kmap_atomic_sg() API to not impact CONFIG_NOHIGHMEM
4. clean up potentially-removable code impacts (e.g. debug code)
5. rework pagetable allocation
6. rework/optimize rmap interaction
7. rework TLB invalidations
8. fix all bugs (as usual)
9. eventually play with ia64's 32-bit emulation

ftp://oss.sgi.com/projects/xfs/download/patches/2.4.23.

For some time the XFS group have been producing split patches for XFS, separating the core XFS changes from additional patches such as kdb, acl, dmapi. The split patches are released to the world with the hope that developers and distributors will find them useful.

Read the README in each directory very carefully, the split patch format has changed over a few kernel releases. Any questions that are covered by the README will be ignored. There is even a 2.4.24/README for the terminally impatient :).

Please reconsider -- the "core" kernel changes we need have existed for three+ years outside of the mainline tree, and each is a small and easily understood change that doesn't affect other filesystems. There is also a VFS fix in there from Ethan Benson, as we discussed during 2.4.23-pre, when you asked us to resend XFS for 2.4.24-pre!) Everything there is a backport from 2.6 in some form, there should be no surprises.

Not having XFS in 2.4 is extremely disadvantageous for us XFS folks (especially since every other journaled filesystem has been merged now). To our users it means some rescue disks simply don't support XFS, meaning you can't mount filesystems when you _really_ need to, etc, etc. Its also always extra work for distributors to merge XFS themselves, and hence a few just don't (and occasionally tell us that they are waiting for you to merge it) - which means some users don't even get the option of using XFS, despite our best efforts.

>From discussions with distributors, a stable 2.6 distribution will be many months after 2.6.0 is officially released, so these issues are not going to go away anytime soon.

So, please merge XFS this time round - its actively developed, has a large installed user base, and has been maintained outside of 2.4 for a long time. We have waited patiently as each release goes by for you to give us the nod, and have been knocked back on a number of occasions while various other merges are being done.

As for XFS, Christoph will review the patches and tell me what he thinks.

Also other people mailed me saying they reviewed the code.

That makes me more comfortable with merging the XFS modifications.

I've released the 008 version of udev. It can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-008.tar.gz

rpms are available at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-008-1.i386.rpm
with the source rpm at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-008-1.src.rpm

udev is a implementation of devfs in userspace using sysfs and /sbin/hotplug. It requires a 2.6 kernel to run. Please see the udev FAQ for any questions about it:
kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ

Note:

The way the configuration files for udev was structured has changed a lot. Previously there were two config files, /etc/udev/udev.config and /etc/udev/udev.permissions. The location of these files could be overridden in the build process, or at runtime with environment variables.

Now there is a single config file at /etc/udev/udev.conf. The contents of that file tell udev where its other config files are located at (for the rules and permissions) and specify other configurable things. For more details about this file, please see the man page, which covers this file and how it is set up. Also see the default udev.conf file provided in the release.

The major changes since the 007 release are:

• different structure of the config files (see Note above for more info.)
• proper handling of the BUS values now (previously they were just ignored.)
• a basic set of regular expression matching is now supported. See the man page for details about this.
• permission handling logic has been rewritten to actually work properly with rules that had regular expressions.
• new '%D' format modifier has been added to make it easier to create devfs like names.
• automated regression test script added (run the udev-test.pl script in the test/ directory to make sure any changes you make don't break anything else.)

Again, many thanks to Kay Sievers, for lots of great patches in this release. I'm really thankful for the automated test script and the regex matching he has provided. This release included patches from lots of other people, all detailed below.

There are a few features coming in the next version of libsysfs that should help udev out a bunch, but this release is showing how flexible udev is at in creating different types of device names. If you haven't tested udev out yet, thinking it was still to early, I suggest trying this release out, as it's quite full featured now.

I'm still looking for a devfs naming scheme config file. Can anyone verify that udev supports everything yet now or not? I know I've been saying this for a few releases now...

The full ChangeLog can be found below.

udev development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/udev

Daily snapshots of this tree can be found at:
http://www.codemonkey.org.uk/projects/bitkeeper/udev/ Many thanks to Dave Jones for managing this.

LSE Call minutes from Dec 3, 2003

Zwane Mwaikambo - OSDL experiences

Generally working with the OSDL has been great.

In order to use the OSDL machines you have to sign up to be an associate and register a project. Then wait for OSDL to review it. Zwane just wanted a large marchine to work on the irq subsystem. He needed access to the NUMA-Q machines to test for regressions.

Most of his interactions have been with Christine who is fairly responsive to email and has fixed the few problems that have popped up.

Zwane has had some trouble using the PLM (Patch Lifetime Manager). Cliff is going to talk to him about using it. Zwane has been doing something similar manually.

The STP enables the engineer to save time by not having to wait inbetween each step. It automates much of the testing process.

Marcelo Tosatti - Also uses the OSDL machines. He said it has been very easy to use. They are the only high mem systems he has available now. He uses them to stress test stuff and sometimes test 2.6 on it.

Zwane said he likes the cross compilation capabilities recently added. Cliff said that was a direct result of comments from OLS this year.

Zwane said one downside of STP and Tinderbox is the huge amount of data that comes out of it and it is hard to parse. He would prefer more minimal info saying if the kernel booted or not and the some of the errors. Cliff told him to try HackBench the most minimal of tests the STP runs.

The turn around time from patch submission to results on an idle system is 1.5 hours. Due to the reimaging of the entire system between runs. Cliff is going to look into making that faster.

Bill Irwin suggested using nfs for the root partition then the host system can feed it nfs and could cleanup and check using md5checksums which would be much faster than reimaging. Cliff said most users wont use nfs on root and it may effect the benchmark results. Bill said most benchmarks arent run on root anyway. Cliff said it is a good idea and worth looking into.

Cliff White - Kernel Tinderbox

History- Came from Brazil. Christian Reis of Asynch Open Source who worked on the Mozilla Tinderbox asked Marcelo about doing a Kernel Tinderbox. Marcelo told Christian to go talk to the OSDL people and the rest is history.

The Mozilla Tinderbox is based on CVS and can do fancy things with triggers. The kernel one is not as fancy because they are still working out issues with BK.

Basically, the client runs in a loop and every 15 minutes wakes up and checks bkbits. If there are any changes within thos 15 minutes the new kernel is downloaded and compiled with John Cherry's comp regress script, which is exhaustive. The best way to see the results is to go to http://developer.osdl.org and click on the tinderbox link. or here (http://tinderbox.osdl.org/showbuilds.pl?tree=linux2.5-bk)

Intel has contributed a 32 and 64 bit client. OSDL is looking to get other architectures added to it (hint hint nudge nudge, especially Power right now).

Marcelo asked if it also boots the kernel. Cliff, no it just compiles at this point. Working on a client that will boot but the STP is best for booting an unknown kernel generally. Cliff is looking at using STP as a client of Tinderbox.

They are not doing mm or other trees as they only work off bkbits right now.

Cliff asked if anyone has Power hardware they could really use one. It doesnt have to be onsite, just need access to a machine. Zwane said they could use cross compilation to test Power stuff instead since they dont boot.

If anyone has a desire to tweak the client is available off Cliffs page on osdl: http://developer.osdl.org/cliffw/

We've exchanged some email with support@bitmover.com, and they've been a great help. Really, there are two things.

The first is triggers. The Mozilla tinderbox is driven by triggers from CVS commits. I believe that triggers are resevered for the commercial version of BK.

However, unlike CVS, BK has a nice way of asking the remote repository if new changes exist, so we really don't need a trigger to tell us when to start a build. Instead, we run the timestamp column on a cron, so it is constantly incrementing. ( Mozilla only increments their timestamp column when a trigger is recieved )

The change from trigger-driven to time-driven timestamps meant we didn't try to create a link between timestamp->source commit. Again, since Mozilla is trigger driven, they have this link.

So, the second piece is linking either the time-stamp column or the build machine column directly to a Web-based view of the code. BkWeb already provides the view.

The main issue here is finding the proper syntax for the bkweb url so we get all of the changesets included in the commit. We've recieved a few examples from your support people, and we're using one currently.

ftp://oss.sgi.com/projects/kdb/download/v4.3/

This is alpha code. It has had minimal testing with a small set of config options. In particular it has not been tested with CONFIG_PREEMPT. It only works on vanilla ia64 boxes using serial console and PC keyboard/VT. Patches to port the kdb USB console support from 2.4 to 2.6 will be gratefuly accepted.

These 2.6.0-test11 patches must be applied in this order, against 2.6.0-test11 plus David Mosberger's linux-2.6.0-test11-ia64-031126 patch.

kdb-v4.3-2.6.0-test11-common-1.bz2
kdb-v4.3-2.6.0-test11-salinfo.bz2 (brings salinfo processing up to 2.4.23)
kdb-v4.3-2.6.0-test11-xfsidbg.bz2 (corrects out of date XFS debug code)
kdb-v4.3-2.6.0-test11-ia64-031126-1.bz2

This is a clone from kdb v4.3-2.4.23, with changes from several people for 2.6, thanks to Xavier Bru and Jim Houston for 2.6 changes to kdb.

The Linux Test Project test suite <http://www.linuxtestproject.org> has been released. The latest version of the testsuite contains 2000+ tests for the Linux OS. Our web site also contains other information such as: test results, a Linux test tools matrix, an area for keeping up with fixes for known blocking problems in the 2.5/2.6 kernel releases, technical papers and HowTos on Linux testing, and a code coverage analysis tool.

Highlights:

• Updated the LTP test driver to handle all real-time signals.
• Updated the command tests to ensure better portability.
• Added new NFS test, nfs_fsstress, to stress NFS using the fsstress test.
• Added new tests for low-level SCSI, virtual SCSI, Asynch IO, & ACL control and management
• Applied more bug fixes, patches, and code cleanups.

We encourage the community to post results, patches or new tests on our mailing list <ltp-list@lists.sf.net> and use the CVS bug tracking facility to report problems that you might encounter with the test suite.

I finally got tired of having to run "bk -r get" before doing a build, so I taught the kbuild system to get the needed files for me. I did most of the work before Sam added the KBUILD_OUTPUT option, so this doesn't work when you're building to a different directory. It could probably be added with a few tweaks to scripts/getfiles, but I'm lazy, and don't use KBUILD_OUTPUT myself. If that scratches your itch, then feel free to take this code and run with it. Same goes for CVS or subversion -- it should not be very difficult to get those working as well.

This isn't really meant for inclusion, just something that makes my life easier. Maybe it will make your's easier too. Though, I'll have to do many, many builds in order to make up for the lost time.... :)

Anyways, the patches follow, or BK users (the intended audience) can do a

bk pull http://typhoon.bkbits.net/autoget-2.5