Kernel Traffic #110 For 9 Mar 2001

I'd like to thank Seth David Schoen for designing some nifty data structures that ended up cutting about 45 minutes off of the publication process for these pages. You totally rock, Seth! Thanks!

There were 516 different contributors. 239 posted more than once. 178 posted last week too.

final:

sync up more with Alan
Urban Widmark: smbfs and HIGHMEM fix
Chris Mason: reiserfs tail unpacking fix ("null bytes in reiserfs files")
Adan Richter: new cpia usb ID
Hugh Dickins: misc small sysv ipc fixes
Andries Brouwer: remove overly restrictive sector size check for SCSI cd-roms

the patch looks huge (it's a meg and a half compressed, 6+ megs uncompressed), but most of the patch by far is S/390 updates and the new Cris architecture.

The biggest real changes that impact normal users are the two bugs that could corrupt your harddisk. The IDE driver bug that Russell found has, to my knowledge, never been shown to happen on anything but his ARM machine, but for all we know it could be quite bad even on x86. Similarly, the elevator bug could cause corruption, but probably has not actually bit people in practice. But both are definitely deadly at least in theory even on bog-standard common PC hardware.

The reiserfs fix should hopefully make the "null bytes in log-files" problem a non-issue, and along with the smbfs/HIGHMEM thing it is certainly important for those that it can affect.

Several folks ran into problems compiling this version, and at one point Andrzej Krzysztofowicz remarked, "It seems that Linus didn't fully merge some stuff from Alan patches." apparently 2.4.2 broke Minix partition support; Alan Cox and others posted patches, and Alan also suggested as an alternative, simply disabling Minix support in this kernel.

Elsewhere, John Heil asked, "Which -ac series patch does this match up with or superceed? ie should this be considered superior to -ac19 ?" Linus replied, "There is no 1:1 comparison to _any_ of the -ac patches, I'm afraid. The two series are fairly disparate, as they have different intentions. Alan accepts some stuff that I would be nervous about, and sometimes I say "to hell with it, we need to fix this" and make Alan nervous." Alan also added, "I'll be merging most of the 2.4.2 changes if there are any I dont have with Linus for the next -ac which will be a 2.4.2-ac1 and thus somewhat shorter." Also in reply to John, Peter Samuelson explained:

Neither "supercedes" the other -- they are different trees. The -ac series has some patches that Linus may never get because they are experimental, or still buggy.

If you want stability, run the real Linus 2.4. If you want all the really minor bug fixes and more of the experimental code, run -ac. If you want production quality, run your kernel on a test server before deploying. (As always!)

Regarding Peter's idea that Linus' tree had stability, Alan remarked, "I think the key word is actually probably 'predictability'. The Linus tree is conservative. (IMHO too conservative and probably in his not conservative enough 8))" Werner Almesberger asked, "given that there are several patches in your tree that never seem to make it to Linus' tree, would it make sense to flag patches that should go into 2.4 as "Not for Alan; Linus _please_ pick it up" (and to keep on pushing until Linus does) ?" Alan replied, "Im trying to push stuff to Linus in a reasonably controlled way. That tends to mean the less important or more obscure hardware comes later."

Folkert van Heusden announced a patch against 2.2.18 and 2.4.1, to cause the kernel to randomize process ID creation, instead of use the current incremental approach. Bert Hubert objected that the incremental approach tended to lower the chances that a new PID would be the same as one that had been used recently. In cases of reusing a PID very soon after its previous use, Bert said, the possibility existed that a signal intended for the earlier process would be received by the new one instead. Using an incremental PID assignment, the kernel would have to cycle very quickly through all available PIDs in order to trigger that potential race condition.

Bert went on to say that he approved of the the idea of making PIDs unpredictable, but suggested, "What would work is to have cryptographically randomly generated PIDs which would then guarantee not to return a previously returned number within 32000 tries, and also not be predictable" H. Peter Anvin replied, "It depends on the size of your number space. If you have a 31-bit pid_t (since it apparently must be sign-safe) then you can take random 16-bit numbers from the /dev/urandom code and add to the last used value instead of simple increment."

Folkert told Bert that his code already checked to make sure it didn't use any existing PID, but Bert explained that the code didn't check for recent PIDs, which he felt was the main problem. This made sense to Folkert, and he suggested something along H. Peter's idea, saying, "I think random increments (instead of last_pid+1) would be the best thing to do then?" Matt Johnston mentioned that OpenBSD had a working implementation of this, and that folks might want to check it out. At this point Sean Hunter pointed out, "I have already written a 2.2 implementation which does not suffer from these problems. It was rejected because Alan Cox (and others) felt it only provided security through obscurity." Folkert agreed with that assessment, though he felt that on any given system, taking those kind of precautions would improve security somewhat, or at least protect against the script-kiddies who lacked the knowledge to adjust their attacks to changing circumstances.

Yes, folks. We got per-process namespaces. Working. With proper behaviour on exit(), yodda, yodda. Enjoy. Help with testing would be more than welcome.

Current patch is on ftp.math.psu.edu/pub/viro/namespaces-S2.gz It's against 2.4.2.

Contents:

proper refcounting of struct super_block
GC for vfsmounts (finally)
fix for races between get_super() and umount()
SMP-safe lock_super()
general cleanup of fs/super.c
"lazy" option for umount() (detach from mountpoint now, do the rest when it will cease to be busy - use MNT_DETACH in 'flags' argument to get that behaviour).
Plan 9 per-process namespaces (sans unions so far)
large cleanup of boot process (ramdisk handling, etc.)

Variant without namespaces (they were the last part) is in the same directory, called s_lock-S2.gz.

rfork.c (in the same place) will copy a namespace and start shell in it. Use for testing... It's an equivalent of rfork(RFNAMEG) on Plan 9.

One detail - patch requires ramfs built into the kernel (boot process cleanup part needs that).

It works here (ran for about 12 hours with no problems). It's _NOT_ for inclusion into 2.4. Some pieces might go (get_super() races have to be fixed, after all), but most of this stuff is 2.5 fodder. However, it seems to be working. No doubt there are bugs and it's far from being a final version. I would call it _very_ early beta. Please, help with testing.

Comments on the code/design/amount of dope it took to write the thing (zero, actually) are welcome. I _will_ document it, but it's still not in the final form. Pretty close to it, hopefully, but...

I'm more than willing to answer questions on the design of the thing - just ask. So far that's the best I can do - all documentation is a pile of notes + CVS log.

Peter J. Braam was very impressed, and added that he and Ron Minnich had built a similar feature, though with slightly different semantics. In their implementation:

when you login, you get imounted into an environment where you have full priviliges (except mknod). The "/index.html" of your environment is not a directory in the Unix tree.
in this environment the system file systems are available to you on a copy on write private basis.
any files you change get out over a network file system to a server. We used InterMezzo backed by a ramfs cache.

When the user logs out, everything is gone, except possibly footprints in swap.

Christoph Hellwig replied, "These changes can be used separately, can't they? I'd really like to use them with Al's more generic namespaces concept. Once thing that worries is that his patch want special privilegs for creating a new namespace and I wonder if we really want that..."

Norbert Nemec reported system lockups, when a ramdisk filled to the point that the system would start swapping. Alan Cox replied, "RAMfs doesnt use swap. It also in 2.4.2 doesnt have limits. The -ac one uses limits so will stop you totally running the box out of ram. 2.4ac also has the true tmpfs with swap backing." Nicolas Pitre asked which of these kernel would be best under swapless systems, but there was no reply.

Russell King reported that running rsync over ssh would hang the process under 2.2.18 and 2.4.2; Rasmus Andersen gave a link to a similar report, in which rsync would complete but then fail to terminate. He added that he could reproduce this himself under 2.4 by giving rsync a big job, but that he couldn't reproduce it under 2.2; Bert Hubert remarked, "rsync problems of this type abound across several UNIX platforms - I had problems like this on FreeBSD, but never under Linux. I suspect that rsync bends or at least stretches certain limits."

Several folks offered possible explanations, but Russell felt he had shown all those possibilities to be incorrect. Eugene Crosser also confirmed the problem for 2.3 kernels, adding, "I had a feeling that it was triggered by certain data patterns because it often stopped at the same spot after restart (and therefore I attributed it to a bug in rsync)."

Elsewhere, under the Subject: Another rsync over ssh hang (repeatable, with 2.4.1 on both ends), Scott Laird reported a repeatable case of an rsync stall over ssh, between two machines running 2.4.1; running strace on the process would actually get it going again, although it would stall again shortly thereafter, with strace running.

He posted some strace output and various folks tried to get to the bottom of it, but the thread ended inconclusively. By the time it petered out, the general concensus was that there was indeed a kernel bug involved.

Zack Brown asked if the Mosix kernel patches would ever be included in the standard kernel, perhaps in 2.5; Christopher Chimelis mentioned that as far as he knew, Mosix hadn't been ported anywhere but the i386, and Zack replied that this didn't seem like a barrier to entry.

Elsewhere, David L. Nicol replied to Zack's initial post, saying that a better approach might be to create some general kernel clustering features that might be useful to more than one clustering system. Rik van Riel created the linux-cluster@nl.linux.org mailing list and said, "To subscribe to the list, send an email with the text "subscribe linux-cluster" to: majordomo@nl.linux.org" . He added, "I hope that we'll be able to split out some infrastructure stuff from the different cluster projects and we'll be able to put cluster support into the kernel in such a way that we won't have to make the choice which of the N+1 cluster projects should make it into the kernel..." Elsewhere, Alexander Viro offered to help with any filesystem work that might be needed by clustering implementations.

Daniel Ridge objected to the whole idea of putting clustering in the kernel. He said:

I have yet to hear a compelling argument about why any of them should go into the standard kernel -- let alone a particular one or a duck of a compromise.

The Scyld system is based on BProc -- which requires only a 1K patch to the kernel. This patch adds 339 net lines to the kernel, and changes 38 existing lines.

The Scyld 2-kernel-monte kernel inplace reboot facility is a 600-line module which doesn't require any patches whatsoever.

Compare this total volume to the thousands of lines of patches that RedHat or VA add to their kernel RPMS before shipping. I just don't see the value in fighting about what clustering should 'mean' or picking winners when it's just not a real problem.

Scyld is shipping a for-real commercial product based on BProc and 2-kernel-Monte and our better-than-stock implementation of LFS and and we're not losing any sleep over this issue.

I think we should instead focus our collective will on removing things from the kernel. For years, projects like ALSA, pcmcia-cs, and VMware have done an outstanding job sans 'inclusion' and we should more frequently have the courage to do the same. RedHat and other linux vendors have demonstrated ably that they know how to build and package systems that draw together these components in an essentially reasonable way.

Albert D. Cahalan replied, "You are basically suggesting the often-rejected "split up the kernel" idea. I think the linux-kernel FAQ covers this." Daniel replied:

No. I'm stipulating that I work with a really interesting piece of software that works with the Linux kernel and is available under the GPL and which we have never even bothered to 'submit' as a patch. I'm willing to suggest further that any responsible development community is suceptible to "race conditions" whereby the natural and studied development and evaluation process is end-run by attempts to 'win' and urinate on the kernel or ANSI or POSIX or whoever (with a facility or mechanism) directly.

These types of 'hacks' or 'denials-of-service' play on the adage that forgiveness is easier than permission. It's hard to dispute this point in an age when SourceForge makes it possible for anyone to maintain a driver or filesystem or whatever without needing to 'hitch a ride' on an existing project (ala linux) that already has a distribution and mirroring facility.

Someone had heard that it would be possible to port Wine into the kernel, thereby gaining a lot of speed. They asked about this, and Dan Kegel replied, "It's not ready for prime time yet. There is some discussion of the issues at news://news.winehq.com/wine.devel" . David Howells added:

I've been writing one to provide all the Windows kernel objects in Linux kernel space (the speed up appears as though it should be impressive). It is, however, not entirely complete yet. You can grab a copy by CVS from the wine repository:

export CVSROOT=:pserver:cvs@cvs.winehq.com:/home/wine
cvs login
(the password is cvs)
cvs -z3 checkout kernel-win32

Or you can browse it:

http://cvs.winehq.com/cvsweb/kernel-win32/

The numbers are looking good: the system call latency appears to be about half that of Win2000 on the same box! (however, use this number with caution).

The idea of putting Wine into the kernel was considered so ludicrous in Issue #10, Section #5 (6 Mar 1999: Linux As A Microkernel) , that the proposal itself was made as a joke demonstrating the flaws of trying to put the wrong things into the kernel. However, by Issue #87, Section #2 (7 Sep 2000: Wine In The Kernel; GPL Loopholes) , the subject was taken more seriously, and Linus Torvalds indicated that he'd be willing to create kernel calls that would help speed up Wine. Apparently work has continued since then.

Tim Walberg asked if anyone had succeeded in getting either a PCMCIA SmartMedia Adapter (specifically the Viking Components one) or a FlashPath floppy SmartMedia adapter, working under 2.2 or 2.4; Steffen Grunewald replied, "http://www.smartdisk.com has a driver (which includes a binary-only library) for FlashPath that you can compile for your kernel. Works fine here (2.2.16)" Andre Hedrick replied that he disapproved of the terms of agreement to access that driver, but invited anyone to send it to him for dissection. Elsewhere Andre added, "That is going to be a SDA device and will have another form of content protection like CPRM and Linux will not support that superset of features at this time or in the future. SMA's are on the hit list for music by the SDMI. If you want to use it as as standard ATA device cool, but the 0xD{0123} opt-codes are not public yet and fall under CFA."

The linux-fbdev mailing list is going to be moving over to sourceforge. The following lists have been created:

linux-fbdev-devel - for device driver developers and video subsystem
linux-fbdev-users - user related mailing list
linux-fbdev-announce - announcements (e.g. new software, drivers, etc.)

You may consult http://sourceforge.net/projects/linux-fbdev for subscribe information.

I have just released a new version of the WRR scheduler supporting the 2.4 kernels besides 2.2 as always . The WRR scheduler is an extension to the Traffic Control/network bandwidth management part of the Linux kernels. The scheduler was developed to support distributing bandwidth on a shared Internet connection fairly between local machines.

Further comments:

As a default all local machines will get equally much of the bandwidth if they have sufficient demand. This is obtained by doing so-called weighted round robin (wrr) scheduling.
It is possible to give machines transferring much data over a long or short period of time less bandwidth.
It can work on a bridge, a router or on a firewall.
Supports accounting locally generated masqgraded packets to the correct local machine.
On the WRR home page an extension is available which includes patches for Squid and the Nec socks5 proxy servers so that proxied packets can also be accounted to the correct local machine.
Includes a configuration file based set of scripts that will setup everything without changing your basic network setup. The scripts will allow you to shape both incoming and outgoing traffic.

Miles Lane gave a link to a NYT article on Unisys, which said that Unisys had no plans to port Linux to their cellular multiprocessor machines. Miles asked if anyone had been working on an independant port, and David Weinehall suggested that several folks might volunteer if Miles would give them the hardware to work on. Joanne Dow also replied to Miles, "if these babies are the 32 processor monsters that UniSys has been making recently there IS interest to get Linux on it. But the people I know who have mentioned "interest", mostly from a curiosity standpoint, have their hands neatly tied by Microsoft. Ya see, the developers at UniSys have NT source licenses so they can develop the HALs for the monsters. Microsoft insists that they spend a considerable time away from OS development before working on another OS. So, no Linux port is in the offing, I suspect. The people who KNOW the machine are not allowed to do it. And I can guarantee you that the machines are not well documented at the level a person making an NT port would need." There was no reply to this, but elsewhere, J Sloan replied to Miles, saying, "unisys is an example of brain damage and it's easiest to ignore/work around them rather than trying to get them out of bed with microsoft. Nature will eventually take it's course with unisys as it did with Dec." Joanne agreed with this assessment, and Gerhard Mack remarked, "Given Unisys' reputation you would think compaq and HP would leave them alone to avoid being dirtied. I think after the gif fiasco most people on the net hate that company."

I finally spent some time fixing up the maestro driver. lots of feature additions had backed up, and the source was rotting. Its still gross, but at least its cleaned up a bit. "It works for me" on my pentium with an ESS maestro2 engineering board, but laptops will be another story entirely. I'd love it if people could apply this patch to vanilla 2.2.18 and let me know how it goes.

The patch does a few things. Most interestingly for the user, it moves away from the model of having multiple /dev/dsp? files and instead allows /dev/dsp to be opened concurrently. It also adds some support for the hardware volume buttons on laptops, but not all vendors wire this the same way. As I don't have a maestro-bearing laptop, this is totally untested.

The code is butchered, so the diff is almost illegible. Perhaps I'll learn and do things in stages next time, but I was on a roll :) One of the more notable changes involves using the kernel's ac97_codec code rather than its own. Hopefully this will result in better mixer behaviour.

I'm particularly interested in hearing how suspend/resume functions, whether or not the multi-open stuff works, and I'd like to get subvendor IDs from people whose laptop's hardware volume buttons work. See the Documentation/sound/Maestro text for instructions on enabling multi-open (channels=2 or 4) and hardware volume support (hw_vol=1).

Its an awfully large diff, so it can be fetched from:

http://www.zabbo.net/maestro/patches/2.2.18-mega-1.diff.gz

if this works I'll officially submit it and make the same sorts of changes to 2.4.

Tom Sightler suggested, "I'd love to test this on my Dell 5000e (Maestro 2E) but it's pretty impractical for me to return to the 2.2.x kernels. I know you said you'll wait before making the changes to 2.4, but I think you'll get more testers if you turn it out sooner rather than later." There was no reply.


Kernel Traffic Latest \| Archives \| People \| Topics	Wine Latest \| Archives \| People \| Topics	GNUe Latest \| Archives \| People \| Topics
Czech

1.	21 Feb 2001 - 27 Feb 2001	(21 posts)	Minix Problem In 2.4.2; Some Discussion Of Development Pseudo-Policies
2.	22 Feb 2001 - 27 Feb 2001	(10 posts)	Addressing PID Prediction Attacks
3.	24 Feb 2001 - 27 Feb 2001	(29 posts)	Per-Process Namespaces For Linux
4.	26 Feb 2001	(3 posts)	System Lock With ramfs
5.	1 Mar 2001 - 4 Mar 2001	(10 posts)	Strange rsync Hang Over ssh In 2.2 And 2.4
6.	27 Feb 2001 - 2 Mar 2001	(19 posts)	Mosix In The Standard Kernel
7.	27 Feb 2001 - 28 Feb 2001	(3 posts)	Status Of Wine In The Kernel
8.	1 Mar 2001	(6 posts)	SmartMedia Adaptor Support
9.	2 Mar 2001	(1 post)	linux-fbdev List Moving To Sourceforge
10.	2 Mar 2001	(3 posts)	WRR Network Scheduler
11.	4 Mar 2001	(6 posts)	Linux On Unisys ES7000
12.	4 Mar 2001	(2 posts)	Maestro Sound Driver Update For 2.2.18

Kernel Traffic #110 For 9 Mar 2001

By Zack Brown