Kernel Traffic #250 For 4 Feb 2004

(This text can be found at kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs for those who want to link to it. I'll also update it with info based on the thread I know is going to spawn from this post...)

Executive summary for those too lazy to read this whole thing:

I don't care about devfs, and I don't want to talk about it at all anymore. If you love devfs, fine, I'm not trying to tell anyone what to do. But you really should be looking into using udev instead. All further email messages sent to me about devfs will be gladly ignored.

First off, some background. For a description of udev, and what it's original design goals were, please see the OLS 2003 paper on udev, available at:
<http://www.kroah.com/linux/talks/ols_2003_udev_paper/Reprint-Kroah-Hartman-OLS2003.pdf>
and the slides for the talk, available at:
<http://www.kroah.com/linux/talks/ols_2003_udev_talk/>
The OLS paper can also be found in the docs/ directory of the udev tarball, available on kernel.org in the /pub/linux/utils/kernel/hotplug/ directory.

In that OLS paper, I described the current situation of a static /dev and the current problems that a number of people have with it. I also detailed how devfs tries to solve a number of these problems. In hindsight, I should have never mentioned the word, devfs, when talking about udev. I did so only because it seemed like a good place to start with. Most people understood what devfs is, and what it does. To compare udev against it, showing how udev was more powerful, and a more complete solution to the problems people were having, seemed like a natural comparison to me.

But no more. I hereby never want to compare devfs and udev again. With the exception of this message...

The Problems:

1. A static /dev is unwieldy and big. It would be nice to only show the /dev entries for the devices we actually have running in the system.
2. We are (well, were) running out of major and minor numbers for devices.
3. Users want a way to name devices in a persistent fashion (i.e. "This disk here, must _always_ be called "boot_disk" no matter where in the scsi tree I put it", or "This USB camera must always be called "camera" no matter if I have other USB scsi devices plugged in or not.")
4. Userspace programs want to know when devices are created or removed, and what /dev entry is associated with them.

The constraints:

1. No policy in the kernel!
2. Follow standards (like the LSB)
3. must be small so embedded devices will use it.

So, how does devfs stack up to the above problems and constraints:

Problems:

1. devfs only shows the dev entries for the devices in the system.
2. devfs does not handle the need for dynamic major/minor numbers
3. devfs does not provide a way to name devices in a persistent fashion.
4. devfs does provide a deamon that userspace programs can hook into to listen to see what devices are being created or removed.

Constraints:

1. devfs forces the devfs naming policy into the kernel. If you don't like this naming scheme, tough.
2. devfs does not follow the LSB device naming standard.
3. devfs is small, and embedded devices use it. However it is implemented in non-pagable memory.

Oh yeah, and there are the insolvable race conditions with the devfs implementation in the kernel, but I'm not going to talk about them right now, sorry. See the linux-kernel archives if you care about them (and if you use devfs, you should care...)

So devfs is 2 for 7, ignoring the kernel races.

And now for udev:

Problems:

1. using udev, the /dev tree only is populated for the devices that are currently present in the system.
2. udev does not care about the major/minor number schemes. If the kernel tomorrow switches to randomly assign major and minor numbers to different devices, it would work just fine (this is exactly what I am proposing to do in 2.7...)
3. This is the main reason udev is around. It provides the ability to name devices in a persistent manner. More on that below.
4. udev emits D-BUS messages so that any other userspace program (like HAL) can listen to see what devices are created or removed. It also allows userspace programs to query it's database to see what devices are present and what they are currently named as (providing a pointer into the sysfs tree for that specific device node.)

Constraints:

1. udev moves _all_ naming policies out of the kernel and into userspace.
2. udev defaults to using the LSB device naming standard. If users want to deviate away from this standard (for example when naming some devices in a persistent manner), it is easily possible to do so.
3. udev is small (49Kb binary) and is entirely in userspace, which is swapable, and doesn't have to be running at all times.

Nice, 7 out of 7 for udev. Makes you think the problems and constraints were picked by a udev developer, right? No, the problems and constraints are ones I've seen over the years and so udev, along with the kernel driver model and sysfs, were created to solve these real problems. I also have had the luxury to see the problems that the current devfs implementation has, and have taken the time to work out something that does not have those same problems.

So by just looking at the above descriptions, everyone should instantly realize that udev is far better than devfs and start helping out udev development, right? Oh, you want more info, ok...

Back in May 2003 I released a very tiny version of udev that implemented everything that devfs currently does, in about 6Kb of userspace code:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105003185331553

Yes, that's right, 6Kb. So, you are asking, why are you still working on udev if it did everything devfs did back in May 2003? That's because just managing static device nodes based on what the kernel calls the devices is _not_ the primary goal of udev. It's just a tiny side affect of it's primary goal, the ability to never worry about major/minor number assignments and provide the ability to achieve persistent device names if wanted.

All the people wanting to bring up the udev vs. devfs argument go back and read the previous paragraph. Yes, all Gentoo users who keep filling up my inbox with smoking emails, I mean you.

So, how well does udev solve it's goals:

• Prevent users from ever worrying about major/minor numbers

  And here you were, not knowing you ever needed to worry about major/minor numbers in the first place, right? Ah, I see you haven't plugged in 2 USB printers and tried to figure out which printer was which /dev entry? Or plugged in 4000 SCSI disks and tried to figure out how to access that 3642nd disk and what it was called in /dev. Or plugged in a USB camera and a USB flash drive and then tried to download the pictures off of the flash drive by accident?

  As the above scenarios show, both desktop users and big iron users both need to not worry about which device is assigned to what major/minor device.

  udev doesn't care what major/minor number is assigned to a device. It merely takes the numbers that the kernel says it assigned to the device and creates a device node based on it, which the user can then use (if you don't understand the whole major/minor to device node issue, or even what a device node is, trust me, you don't really want to, go install udev and don't worry about it...) As stated above, if the kernel decides to start randomly assigning major numbers to all devices, then udev will still work just fine.

• Provide a persistent device naming solution:

  Lots of people want to assign a specific name that they can talk to a device to, no matter where it is in the system, or what order they plugged the device in. USB printers, SCSI disks, PCI sound cards, Firewire disks, USB mice, and lots of other devices all need to be assigned a name in a consistent manner (udev doesn't handle network devices, naming them is already a solved solution, using nameif). udev allows users to create simple rules to describe what device to name. If users want to call a program running a large database half-way around the world, asking it what to name this device, it can. We don't put the naming database into the kernel (like other Unix variants have), everything is in userspace, and easily accessible. You can even run a perl script to name your device if you are that crazy...

  For more information on how to create udev rules to name devices, please see the udev man page, and look at the example udev rules that ship with the tarball.

So, convinced already why you should use udev instead of devfs? No. Ok, fine, I'm not forcing you to abandon your bloated, stifling policy, nonextensible, end of life feature if you don't want to. But please don't bother me about it either, I don't care about devfs, only about udev.

This is my last posting about this topic, all further emails sent to me about why devfs is wonderful, and why are you making fun of this wonderful, stable gift from the gods, will be gleefully ignored and possibly posted in a public place where others can see.

The ALSA 1.0.1 code for 2.6 kernels is available. I think that this update might be included into -mm or standard 2.6 kernels.

BitKeeper:

bk pull http://linux-sound.bkbits.net/linux-sound

The GNU patch is available at:

ftp://ftp.alsa-project.org/pub/kernel-patches/alsa-bk-2004-01-08.patch.gz

Summary:

• use pci_set_consistent_dma_mask()
• control midlevel - added user control elements, fixes
• timer midlevel - fixes
• OSS PCM emulation - fixes and updates
• joystick support selectable using module parameter
• AC'97 code - introduced ac97_bus_t and pcm support code
• USB audio driver updated
• VIA82xx driver updated
• EMU10K1 driver updated
• ICE1712 driver updated
• CMIPCI driver updated
• AD1848 driver updated
• Digigram VX library updated
• HDSP driver updated

When the CPU is under load, the buzzing disappears.
When something else is using /dev/dsp, the buzzing disappears.
When I reboot to 2.6.1-rc1-mm1, the buzzing disappears ;)

a new patch that adds hard real time support to the linux kernel (worst case interrupt response time below 5 microseconds):

The proposed "real time interrupt patch" enables the linux kernel for hard-real-time applications such as data aquisition and control loops by adding priorities to interrupts and spinlocks.

The following document will describe the patch in detail and how to install it:

http://home.t-online.de/home/Bernhard_Kuhn/rtirq/20040108/README

The patch and a demo application can be downloaded from:

http://home.t-online.de/home/Bernhard_Kuhn/rtirq/20040108/rtirq-20040108.tgz

I don't think that real time interrupts are usefull for a desktop environment: here, even interrupt latencies below one millisecond should be more than good enough for streaming/multimedia applications and games - if your specific application is not "fast" or "reactive" enough, i would tend to say that it is not the fault of the linux kernel and it's the implementation of the application itself or one of the related kernel drivers that needs improovment. If you recognize interrupt response times higher than a millisecond with a standard linux kernel, then there is definitly a bad device driver or something is broken with your hardware.

The real time interrupt patch is mostly intended to be used in control loop and data aquisition applications where higher latencies or jitters higher than a few mirocseconds could have catastrophic consequences. However, i can imagine that it makes sense to use this patch for some very special networking devices where the kernel might sometimes not be able to response quick enough because it is just processing an interrupt of type SA_INTERRUPT that takes too much time. But in these cases, i guess it's simpler to fix that other device driver (by moving parts of the interrupt handler to a tasklet) rather than introducing real time interrupts - the only problem with this variant is that for oftenly changing hardware configurations, you can never be sure if you catched all "longest execution paths" and you may end up spending most of your time improving other device drivers.

BTW.: having interrupt priorities is only the first step to make the kernel hard real time aware. The next step would be to make the kernel scheduler re-entrentable, so that a user space application related to a high priority interrupt could run at a higher priority than a low level interrupt service routine (low priority interrupts are disabled while the high priority application is running) - this scheme is pretty similar to LXRT, except that the kernel scheduler is used instead of an external dispatcher. But just as like as with LXRT, you only have a very limited set of system calls and you can easly lock up your system when an application takes 100% of the CPU.

Ok, the diffs from -rc3 are minimal, most noticeably the (very _very_ hard to trigger, but nasty if you ever did) fork() race that Ingo found.

I'm going to be in Australia (and on airplanes) for the week, but we're all in the capable hands of Andrew, so why worry? The fact that I'm fleeing the country should in no way be construed as anything sinister at all, no siree. Nope. I'm innocent, and nobody saw me do it.

The full changelog is getting uploaded right now along with the release, and the BK trees have already been pushed.

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.1/2.6.1-mm1/

• A big ALSA update.

• Added support for kgdb on the x86_64 platform. You'll need a couple of patches to gdb-6.0 itself to use it properly. They may be found at

  ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/gdb/gdb-6.0/

• Dropped the remap_file_pages() and prefault work - it is suspected of causing stability problems.
• The PCI IDE drivers should work as modules now.
• A significant update to the NFS client.

• A large s390 update. Various device drivers and IO layer changes there.

  Note that the 's390' patches actually include significant core MM changes in the area of pte writeback/invalidate, rmap tweaks, etc. Relevant patches are:

  s390-11-tlb-flush-optimisation.patch
  s390-12-dirty-referenced-bits.patch
  s390-13-tlb-flush-race-fix.patch
  s390-14-rmap-optimisation.patch
  s390-15-superfluous-flush_tlb_range-calls.patch
  s390-16-follow_page-lockup-fix.patch

• Merged the large SN2 update which Pat and Christoph have been discussing.

Note that the fb stuff is ancient because it's basically not maintained as far as I'm concerned.

I occasionally get huge drops from James, and they invariably break stuff. Which means that I often decide (espcially when trying to stabilize things) that I just can't _afford_ to apply the fr*gging patches. Because by past experience applying one of the big "everything changes" patches tends to break more things that it fixes.

I'm sorry, but this i show it is. The fbcon people have been changing interfaces faster than they have been fixing bugs in the code. Together with the fact that most of the development seems to happen in outside trees, and nobody ever sends me fixes relative to the released tree, this makes for a pretty bad situation.

I really think that development should happen in the regular tree, or at least be synched up in reasonable chunks THAT DO NOT BREAK everything.

I realize that some fb developers seem to disagree with me, but the fact is, the way things are done now, fb will _always_ be broken. Most people for whom the standard kernel works will never test the fb development trees, so those trees will never get any amount of reasonable testing. As a result, they WILL be buggy, and synching with them WILL be painful as hell.

There is a d*mn good reason for why development should happen incrementally, and in the standard trees, and not in some outside tree. For one: testing. For another: figuring out when things break in a timely manner.

The megaraid driver version 2.10.1 is released and can be downloaded from ftp://ftp.lsil.com/pub/linux-megaraid/drivers/version-2.10.1/

Following other components are also available at this location:

1. Patches for Red Hat and SuSE stock kernels
2. Driver update disks for Red Hat and SuSE
3. RPM packages to update drivers

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.1/2.6.1-mm2/

• Big update for the SuperH architecture
• x86_64 udpate
• SELinux update
• Many more NFS client patches. These mainly affect the experimental NFSv4 or RPCSEC_GSS, so they are unlikely to be noticed.
• The filesystem AIO patches were getting in the way of testing the normal readahead code and were making O_DIRECT testing tricky. Those patches have been dropped for now.

I've created a new version of the laptop-mode patch, this time against linux 2.6.1. It can be found here:

http://www.liacs.nl/~bsamwel/laptop_mode/laptop-mode-2.6.1-7.patch

It has no kernel code changes, only changes to the supporting documentation/scripts:

• Dax Kelson's ACPI integration script is included.
• Fixed a missing "esac" in the control script.
• Minor documentation improvements.

Especially Dax's addition should make it a lot more usable. I haven't been able to test this myself however, because I don't own a laptop. Dax probably does, so I'll trust him and assume that he has tested it. ;)

The 250+ patch queue was getting way too big to manage, so using BK I split things up locally into a number of buckets. This allowed me to much more easily digest Al Viro's latest patch flood, as well as get things into much better shape for submission to upstream (as soon as the tree re-opens)... some changes were definitely more experimental than others, and won't go to Andrew/upstream until bugs and interface issues are fully sorted out.

This split-up allowed me to easily create broken-out sets of patches, which are available at the URL below.

Note the new "netdev" moniker too, that's not a typo.

Summary of new changes:

• forcedeth update
• more fixes and cleanups from Al Viro
• netpoll, atmel, dgrs updates
• other bits

Summary of patchkit:

• new e100 driver (rewritten from scratch)
• new nVidia nForce NIC driver
• new pci200syn WAN driver

• r8169 major bug fixes
• e1000 minor updates / fixes
• sk98lin vendor updates / fixes
• many bonding updates and cleanups
• misc bug fixes

• 8139too NAPI support
• tulip NAPI support

• netconsole / netdump support
• net_device allocation and reference counting work

Patch:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/patchkits/2.6/2.6.1-bk1-netdev2.patch.bz2

Full changelog:
http://www.kernel.org/pub/linux/kernel/people/jgarzik/patchkits/2.6/2.6.1-bk1-netdev2.log

Broken-out patches (broken out into "buckets" not changesets):
http://www.kernel.org/pub/linux/kernel/people/jgarzik/patchkits/2.6/broken-out/

BK repo:
bk://gkernel.bkbits.net/netdev-2.6
or
http://gkernel.bkbits.net/netdev-2.6

kexec is a linux-reboots-linux kernel patch. It currently applies only to x86/ia32.

Patch, scripts, userspace tools, etc., are available at

http://developer.osdl.org/rddunlap/kexec/2.6.1/

Adaptec has been looking at the MD driver for a foundation for their Open-Source software RAID stack. This will help us provide full and open support for current and future Adaptec RAID products (as opposed to the limited support through closed drivers that we have now).

While MD is fairly functional and clean, there are a number of enhancements to it that we have been working on for a while and would like to push out to the community for review and integration. These include:

• partition support for md devices: MD does not support the concept of fdisk partitions; the only way to approximate this right now is by creating multiple arrays on the same media. Fixing this is required for not only feature-completeness, but to allow our BIOS to recognise the partitions on an array and properly boot them as it would boot a normal disk.
• generic device arrival notification mechanism: This is needed to support device hot-plug, and allow arrays to be automatically configured regardless of when the md module is loaded or initialized. RedHat EL3 has a scaled down version of this already, but it is specific to MD and only works if MD is statically compiled into the kernel. A general mechanism will benefit MD as well as any other storage system that wants hot-arrival notices.
• RAID-0 fixes: The MD RAID-0 personality is unable to perform I/O that spans a chunk boundary. Modifications are needed so that it can take a request and break it up into 1 or more per-disk requests.
• Metadata abstraction: We intend to support multiple on-disk metadata formats, along with the 'native MD' format. To do this, specific knowledge of MD on-disk structures must be abstracted out of the core and personalities modules.
• DDF Metadata support: Future products will use the 'DDF' on-disk metadata scheme. These products will be bootable by the BIOS, but must have DDF support in the OS. This will plug into the abstraction mentioned above.

I'm going to push these changes out in phases in order to keep the risk and churn to a minimum. The attached patch is for the partition support. It was originally from Ingo Molnar, but has changed quite a bit due to the radical changes in the disk/block layer in 2.6. The 2.4 version works quite well, while the 2.6 version is fairly fresh. One problem that I have with it is that the created partitions show up in /proc/partitions after running fdisk, but not after a reboot.

This patch updates UML to 2.6.0 and pulls in all the changes that have accumulated in my 2.4 tree.

The 2.6.0 UML patch is available at
http://www.user-mode-linux.org/mirror/uml-patch-2.6.0-1.bz2

BK users can pull my 2.5 repository from
http://www.user-mode-linux.org:5000/uml-2.5

For the other UML mirrors and other downloads, see
http://user-mode-linux.sourceforge.net/dl-sf.html

Other links of interest:

The UML project home page : http://user-mode-linux.sourceforge.net
The UML Community site : http://usermodelinux.org

UML has a need to free dirty pages in the middle of a file (which is described in more detail below). The obvious way to do this, and one which has come up before, is a sys_punch system call for making a hole in the middle of a file. Since I need something like this now, and sys_punch is not immediately forthcoming, I implemented a special device which implements the semantics that UML needs.

/dev/anon acts just like anonymous memory, except for not being anonymous. It implements the memory freeing semantics by keeping track of how many times each page is mapped, and freeing pages when the map count goes to zero.

I did it by mashing the map counting into tmpfs. I've attached the patch (against 2.4.23) below. The main additions are

• a munmap file_operation which is called from do_munmap
• map counts in a structure that's parallel to the swap entries in tmpfs
• a new misc device at minor 10

Comments on the implementation are welcome, as I'm not particularly proud of it, but nothing else came to mind quickly.

If this should be maintained out-of-tree, should I get an official minor for it anyway, or just unofficially use the first unused misc minor (10 in 2.4, 11 in 2.6)?

I'd also like opinions on whether this (or something like it) is sane enough to go into mainline, or whether I should just keep it as a out-of-tree patch until sys_punch shows up.

The immediate need for this would go away as soon as we got sys_punch.

On the other hand, a file backing anonymous memory seems superficially attractive because it can get rid of the vma->vm_file == NULL special cases scattered through the VM system. vm_file is expected to be the backing file for the data, which doesn't work so well for in-memory filesystems. tmpfs seems to know a lot about swapping pages out. It might be a good idea to resurrect the old swapfs idea and turn tmpfs into that. Then I guess vm_pgoff would become the equivalent of a swp_entry, which raises the question of what goes into a pte of a swapped-out page. Current convention says it becomes 0, which seems a bit wasteful. It's also not clear what vm_pgoff for an in-memory page would be. vmas like their data to be contiguous in the backing file, which suggests that there would be a swapfs file per process, with contiguous anonymous memory being contiguous in the swap file, if not on the actual device.

I've released the 013 version of udev. It can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-013.tar.gz

rpms built against Red Hat FC1 are available at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-013-1.i386.rpm
with the source rpm at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-013-1.src.rpm

udev allows users to have a dynamic /dev and provides the ability to have persistent device names. It uses sysfs and /sbin/hotplug and runs entirely in userspace. It requires a 2.6 kernel with CONFIG_HOTPLUG enabled to run. Please see the udev FAQ for any questions about it:
kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ

For any udev vs devfs questions anyone might have, please see:
kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs

NOTE: The udev.rules file format has changed! If you have a modified config file it MUST be changed in order to work properly. Here's what needs to be done:

• The "<METHOD>, " at the beginning of the line should be removed. <METHOD> is one of the following: LABEL, CALLOUT, NUMBER, TOPOLOGY, REPLACE.
• The result of the externel program is matched with RESULT= instead if ID=
• The PROGRAM= key is only valid if the program exits with zero (just exit with nozero in a script if the rule should not match)

Also note the following changes in the way the udev.rules file is processed:

• Rules are processed in order they appear in the file. There are no priorities anymore.
• if NAME="" is given, udev is instructed to ignore this device, no node or symlink will be created.

As a result of these changes, we can now create much more powerful rules by combining multiple key fields. Here's two new rules that show how you can do this:

# combined BUS, SYSFS and KERNEL
BUS="usb", KERNEL="video*", SYSFS_model="Creative Labs WebCam*", NAME="test/webcam%n"

# exec script only for the first ide drive (hda), all other will be skipped
BUS="ide", KERNEL="hda*", PROGRAM="/sbin/ide-devfs.sh_%k_%b_%n", RESULT="hd*", NAME="%1c", SYMLINK="%2c %3c"

Also, the result of the PROGRAM call is now cached accross multiple rules as long as a new PROGRAM key is not specified. As an example:

PROGRAM="/bin/echo_abc/index.html", RESULT="no_match", NAME="web-no-2"
KERNEL="video*", RESULT="123", NAME="web-no-3"
KERNEL="video*", RESULT="abc", NAME="web-yes"

The last rule would match properly.

Many thanks for these changes go to Kay Sievers. I really appreciate all of the development effort.

If anyone has any problems converting existing rules files, please see the example rules that are in the udev tarball, and feel free to ask questions on the linux-hotplug-devel mailing list.

Thanks again to everyone who has send me patches for this release, a full list of everyone, and their changes is below.

udev development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/udev

Daily snapshots of udev from the BitKeeper tree can be found at:
http://www.codemonkey.org.uk/projects/bitkeeper/udev/
If anyone ever wants a tarball of the current bk tree, just email me.

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.1/2.6.1-mm3/

• A big ppc64 update from Anton
• Added Nick's CPU scheduler patches for hyperthreaded/SMT CPUs. This work needs lots of testing and review from those who care about and work upon this feature, please.

• An I/O scheduler tuning patch. This is the 114th patch against the anticipatory scheduler and we're nearly finished, honest. Now would be a good time for people to run the appropriate benchmarks.

  We're still not as good as deadline on some seeky loads, and deep SCSI TCQ still hurts a lot. But it is looking good on average.

• Plenty of other random stuff

This version fixes the "memory filled with unfreeable inodes" problem which occurs on high memory machines in some workloads. That is the last big problem 2.4 VM had with highmem I believe.

It contains a few other important fixes:

• fixes a SMP deadlock introduced during 2.4.23 (which could hang the machine on high filesystem activity).
• fixes a memory allocation deadlock in USB

Amongst others.

Please test it!