Kernel Traffic #52 For 24 Jan 2000

1. Multiwrite IDE breaks on a disk error
2. Poll on > 16000 file handles fails
3. Restore O_SYNC functionality
4. Merge the network fixes - there is a ton of backed up stuff to do asap
5. ISA DMA is no longer allocating correctly aligned data
6. vmalloc(GFP_DMA) is needed for DMA drivers
7. VM needs rebalancing
8. NFSD fixes for path walking to regenerate dentries
9. Fix eth= command line
10. Check O_APPEND atomicity bug fixing is complete
11. Protection on isize (sct)
12. Merge 2.2.13/14 changes
13. Get RAID 0.90 in
14. PAE36 failures
15. USB HID merge
16. Mikulas claims we need to fix the getblk/mark_buffer_uptodate thing gor 2.3.x as well
17. PIII/Athlon/MMX/etc acceleration merge from 2.2.x-ac
18. Merge arcnet update (DONE)
19. Fix SPX socket code
20. AHA152x isnt smp safe (FIXED)
21. NCR5380 isnt smp safe
22. isofs break on 4Gig disk (FIXED ?)
23. Finish 64bit vfs merges (stat64 etc) (DONE ??)
24. Make syncppp use new ppp code
25. Fbcon races
26. Fix all remaining PCI code to use new resources and enable_Device
27. Stackable fs ?? (Erez)
28. Get the Emu10K merged
29. Test PMC code on Athlon
30. Fix module remove race bug (-- not in open so why did I see crashes ??? --)
31. Per Process rtsigio
32. Maybe merge the ibcs emulation code
33. VFS?VM - mmap/write deadlock
34. initrd is bust
35. rw sempahores on page faults (mmap_sem)
36. kiobuf seperate lock functions/bounce/page_address fixes
37. per super block write_super needs an async flag
38. addres_space needs a VM pressure/flush callback
39. per file_op rw_kiovec
40. enhanced disk statistics
41. Fix routing by fwmark
42. put_user appears to be broken for i386 machines

While there is a lot of work to be done to do things right, I don't think there is a lot of work to put ext3 and ReiserFS into 2.3. We are working on the port right now for ReiserFS, and I don't think we are far away.

If we wait for everything to be right, software never ships....

I think the thing to consider is that we can put journaling into 2.4, and then niceties like allocate on flush can be done later in 2.5.

I have some concern that you are suggesting that there should be only one journaling coding for both filesystems, and that is not only far away but far from clear to me. Chris Mason (ReiserFS journaling code author) may disagree with this, and is encouraged to comment. I'll just note that we envision re-engineering journaling now that the current journaling is stable. Users should use the current journaling, it should go into 2.4 because lack of journaling is keeping many users away from Linux, but journaling is not ready for the ANSI standards process.:-) Nor is it ready for sharing between many filesystems.... We have more than one filesystem, why not more than one journaling system?

There are other problems as well: on smaller machines, we don't have any way for the "pinned" buffers associated with transactions to be flushed early in response to memory pressure, for example.

There are a few such VM issues which have to be addressed before we can get rock-solid journaling in the kernel.

1. some or all of ingo's latency patches
2. 32 bit uid/gid support
3. madvise
4. ibm's task struct cacheline fix for goodness()
5. appropriate dynamic sizing of kernel hash tables

When I tried it for a bit in 2.2.x-ac I couldnt measure any.

The reason I gave up adding it to -ac was that I cleaned it up , I fixed it for the Coding Style document and I fixed some bugs. I got an update from the author that simply ignored all that work and reverted to wrong formats.

Every annoyance I personally have with the LVM code comes down to two things

1. Not following the Coding Style
2. General poor readability - lots of complex loops, huge ioctl functions

The main thing it made me wonder was if the ioctl interface layer was perhaps structured badly and perhaps using different ways to pass the data would avoid a lot of the mess in the existing code.

The actual remapping code is fast, clean and works. It also has about zero impact if the LVM is disabled.

I think you took my message the wrong way; my impression from the LVM site was "LVM meets the technical criteria, but Linus wants an idea of who is interested in support for this" - maybe Linus replied to a message and didn't CC lkml. Maybe not. Still, LVM made it into a couple of -ac patches, so I figure it's technically sound.

I agree that the idea of petitioning for a patch that Linus has rejected based on his design judgement is, well, lame. 'Petition' was maybe the wrong word, but not my choosing.

1. memory detection is broken and may be causing fs corruption
2. IDE DMA failures

I tried a couple of times to sort the mess out. I never got the PCI scan code to work right, it kept not checking resources, registering I/O as memory and stuff. I at least don't think its ready. The drivers also seemed to have problems (not that the existing ones dont). Right now I'd rather 2.3.x picked up the fixes from 2.2.14 for tulip and acquired the eepro100 and other work people have been doing then the bug fixes from Donalds driver updates.

Its not that I don't think Don is on the right track, just I don't think he has the right implementation.

The thing to do is to create a

/proc/drivers/<drivername>/

directory. The /proc/drivers/ directory is already there, so you'd basically do something like

create_proc_info_entry("driver/mydriver/status", 0, NULL, mydriver_status_read);

to create a "status" file (etc etc).

Are you just blind to the neverending format/compatiblity/parsing/performance problems the whole idea behing /proc induces inherently? Oh yes they don't turn up that frequently anylonger, since everybody learned in the time between don't touching anything there like a heap of shit. Instead of changing something, one leaves the broken /proc interface where it is and adds just another new file (or even dir) there.

My favorite examples for how broken they are

/proc/stat	the information there is entierly *broken* misleading and incomplete. (leftover from early days.)
/proc/pci	static data continuously reconstructed on the fly. (binary to string and then back string to binray in userland...) And now (2.3.xx) it's event binary only...
/proc/cpuinfo	same here static data. uname is since the beginnging the proper interface for this stuff.
/proc/ksyms	entierly redundant and not used by the modutils.
/proc/modules	entierly redundant to the module syscalls. *Not* used by lsmod.
/proc/version	entierly static data with no apparent value
/proc/kmsg	entierly redundant to syslog.

One could continue with no end...

root:/proc# cat meminfo
total: used: free: shared: buffers: cached:
Mem: 64577536 62787584 1789952 20643840 1339392 17186816
Swap: 139821056 36478976 103342080
MemTotal: 63064 kB
MemFree: 1748 kB
MemShared: 20160 kB
Buffers: 1308 kB
Cached: 16784 kB
SwapTotal: 136544 kB
SwapFree: 100920 kB

Wonderfull!!!! The same data twice, albeit no one of them easly parsed! Easly parsed? By what? AWK? SED? or should the procps utilities beeing implemented in damn PERL? (Some loosers who don't know C would apreciate this, certainly) !!!!! The only thing I'm missing is adding floating point formats to this...

And then there is the phenomenon of proliferation of /proc items. Just an example...

root:/proc/ide# find /proc/ide
/proc/ide
/proc/ide/drivers
/proc/ide/hdd
/proc/ide/ide1
/proc/ide/ide1/hdd
/proc/ide/ide1/hdd/capacity
/proc/ide/ide1/hdd/settings
/proc/ide/ide1/hdd/model
/proc/ide/ide1/hdd/media
/proc/ide/ide1/hdd/identify
/proc/ide/ide1/hdd/driver
/proc/ide/ide1/model
/proc/ide/ide1/mate
/proc/ide/ide1/config
/proc/ide/ide1/channel
/proc/ide/hda
/proc/ide/ide0
/proc/ide/ide0/hda
/proc/ide/ide0/hda/smart_thresholds
/proc/ide/ide0/hda/smart_values
/proc/ide/ide0/hda/geometry
/proc/ide/ide0/hda/cache
/proc/ide/ide0/hda/capacity
/proc/ide/ide0/hda/settings
/proc/ide/ide0/hda/model
/proc/ide/ide0/hda/media
/proc/ide/ide0/hda/identify
/proc/ide/ide0/hda/driver
/proc/ide/ide0/model
/proc/ide/ide0/mate
/proc/ide/ide0/config
/proc/ide/ide0/channel

Hell only God know's what they are good for! And there is no userland tool for this. This is the last thing Mark Lord added before ditching ide developement.

root:/proc/sys# find /proc/sys | wc
208 208 7305

Don't tell me any sane admit will fiddle with ALL this... And in esp. any sane system doesn't need this degree of pseudo configuration flexibility.

And here my ABSOLUTE FAVORITE:

PID

USER

PRI

NI

SIZE

RSS

SHARE

STAT

LIB

%CPU

%MEM

TIME

COMMAND

21821

root

19

0

1032

1032

816

R

0

4.7

1.6

0:00

top

* *** ***** ******* ********* *** *** *** *** *** ***

Yes reading files, walking dirtrees and parsig them is indeed very very time consuming. I would like to know how well this design will scale to an enterprise server with 32 CPU and X*10000 concurrent processes:

user:~/mysweethome: Message from root@localhost to user@localhost resived... BLAH BLAH: "Please stop any intensive intermittient computational activity. Due to maintainance work I'm going to run ps auxw int 5 minutes. Thank's in advance for your understanding! You's sincerly: root@localhost"

Oh don't tell me procps could have been done better, there where years of time for this and apparently nobody managed to get it right for practical reaons..

I think you don't write enough user-land code... (just a guess) go and just compare for example the ps/netstat utlities from *BSD just too see WHY /proc as it is, is a BAD design :-).

Maybe it appears cute as an idea to have something like this, but in practice something like this is inevitable going to result in a coding mess in esp. in an such uncoordinated effort like Linux.

And I didn't even tell a word about the bloat/mess/races inside the kernel code caused by this all...

Really man sysctl *is* much much saner and what should be "depricated" is /proc

Oh, please! All we need is sysctlbyname(2) - _not_ a problem, and closes all problems with numbers. And it should not work through mounted procfs - we can traverse the tree doing comparisons by name just fine. The fact that sysctl(8) needs mounted procfs is an artificial misfeature, nothing more.

What _is_ bogus is the idea of sysctl() doing more than read/write access to constant-sized variables. Or procfs entries doing ioctl(), for that matter - just look at /proc/mtrr, for one specimen.

sysctl() is a perfectly reasonable subset of pseudofs-type stuff, with well-defined semantics (unlike the rest ;-/). The rest is pretty much a maze of twisted little formats, none alike. IMO dissolving the thing is _not_ a good idea. You have the final word, indeed, but I think that sysctlbyname() may remove most of the problems.

Because system calls are performance-sensitive.

And system calls are not clearly "hierarchical".

And system calls are supposed to be there regardless of what software and hardware configuration we have there.

In contrast, sysctl isn't all that performance-sensitive, AND they are extremely hierarchical, AND they depend on configuration and timing.

In short, sysctl NEEDS:

• "naming": you cannot name the sysctl space with a number: it is much too dynamic for that. How do you enumerate drivers? Give them random numbers?
• "listing": showing which sysctl's are there, in a hierarchical manner. Again, a listing is useless with a number.
• "hierarchy". You have different devices, but they have the same controls. Do they get the same name? Yes. But in different places in the hierarchy.

In short, you NEED a filesystem. You need to be able to "ls" the thing. You need to be able to search the thing. You need to be doing all the things you can do with a real filesystem.

And flattening it out and trying to number it does not work. Never has, never will. It's not an enumerated space.

1. New type (struct block_device) is defined. We have a cache of such objects, indexed by dev_t. struct block_device * is going to replace kdev_t for block devices. Handling of the cache is done in fs/block_dev.c
2. They have methods (struct block_device_operations). Currently the set is { open, release, ioctl, revalidate, check_media_change }. For now (and it's going to change) types are the same as in file_operations. However, in the near future they are going to become

int (*open)(struct block_device *bdev, mode_t mode, unsigned flags);
int (*release)(struct block_device *bdev);
int (*ioctl)(struct block_device *bdev, unsigned cmd, unsigned long arg);
int (*revalidate)(struct block_device *bdev);
int (*check_media_change)(struct block_device *bdev);

3. ->revalidate() and ->check_media_change() disappeared from file_operations.
4. register_blkdev() takes block_device_operations instead of file_operations now. For one thing, it means that block devices are more or less insulated from all future changes in file_operations (Good Thing(tm)). For another, it means that drivers should be modified. I did the change for all drivers in the main tree, see the patch for details. It's pretty easy.
5. blkdev_open() doesn't change ->f_op. def_blk_fops has all needed methods (open, release and ioctl call the methods from block_device_operations, indeed).
6. Inodes got a new field: i_bdev. Filesystems should not worry about it - just remember to call init_special_inode() when you are initializing device/fifo/socket in-core inode (in foo_read_inode() or in foo_mknod(); all filesystems in the tree are doing it now). Contents of this field: pointer to struct block_device if it is a block device inode, NULL otherwise.
7. Superblocks got a new field: s_bdev. Handled by code in fs/super.c, points to the struct block_device if the mount is device-backed, NULL otherwise (i.e. for NFS, CODA, procfs, etc.).
8. do_mount() first argument is struct block_device * now. It does the right thing for non-device mounts - just pass NULL and it will work (allocate the anonymous device, etc.)
9. Instead of calling get_blkfops(), use ->bd_op in struct block_device. Moreover, better use blkdev_get()/blkdev_put()/ioctl_by_bdev() (see examples in mm/swapfile.c, drivers/char/raw.c, fs/super.c, fs/isofs/inode.c, fs/udf/lowlevel.c).
10. Thing that is probably going to happen RSN: instead of struct gendisk per major we may want to go for struct gendisk per _disk_. It would mean that at some point near ->open() we will put the pointer to it into the struct block_device. One obvious consequence being that partitions-related ioctls() will become completely generic.

Notice that it is _not_ the same as devfs (and not a beginning of moving devfs into the main tree). It just provides the backplane - no namespace, no nothing. Inodes (either in normal filesystems or in devfs) point to such animals. That's it. Eventually things like ->b_dev, ->b_rdev, ->i_dev, ->rq_dev, etc. are going to become pointers to such objects, but it will be done step-by-step - otherwise we'll end up with a moby patch and moby breakage in bargain...

Character devices are not affected at all - IMO using the same type both for block and character device was a mistake. So their handling remains as-is. Probably something should be done for them too, but that's completely different story.

Good grief Charley Brown! You, in a few key-strokes, just blew away major portions of the work done over the past few years by software engineers who ported their drivers to Linux. Linux will never be accepted as a 'professional' operating system if this continues.

It's enough of a problem putting one's job on-the-line convincing management to risk new product development to Linux. Once these products are in Production, and bugs are discovered in the OS, we must be able to get the latest version of the OS and have our drivers compile. If this is not possible, you do not have an operating system that is anything other than an interesting experiment.

For instance, there was a simple new change in the type of an object passed to poll and friends. This just cost me two weeks of unpaid work! Unpaid because I had to hide it. If anyone in Production Engineering had learned about this, the stuff would have been thrown out, the MicroCreeps would have settled in with "I told you so..", and at least three of us would have lost our jobs.

Industry is at war. You can't do this stuff to the only weapons we have. Once you claim to have a "Professional Operating System", its development must be handled in a professional way. If major kernel interface components continue to change, Linux is in a heap of trouble as are most all of those who are trying to incorporate it into new designs.

The industrial use of Linux is not at the desktop. It involves writing drivers for obscure things like machine controllers (read telescope controllers), Digital signal processors (read medical imaging processors), and other stuff you can't buy at the computer store. It doesn't matter if you fix all of Donald Becker's drivers to interface with the new kernel internals. You have still broken most everything that counts.

We all know your position on compability. :) Many people, including myself, usually understand and agree with it.

However, you are going a little far on this one.

The change is going into 2.3.x, and that *IS* the approiate place to break interfaces. These kinds of changes should certantly not be introduced into 2.2.x.

This should cause you little difficulity, as your example of having to upgrade to fix a bug should not apply. When you upgade to fix a bug then you should just be increasing patchlevel. If there is not a patch for a bug in 2.2.x which is fixed in 2.4.x then there is a bug in the Linux development process.

In order to move forward, we *must* break things. To make up for this we continue to maintain old versions. There are still bugfixes being made against 2.0.x and there will be bugfixes against 2.2.x. RedHat even still issues updates against RH4.2..

So if this were to have occured within a stable kernel version, or if it had severly affected userspace, I would agree.

Industrial use of Linux usually doesn't involve the kernels which are marked as `development', ie. where the `middle' version number is odd and where major things are expected to change.

People venturing out on that terrain can know what they're heading into (see http://kt.zork.net/) and shouldn't come whining when some actual development happens in the development branch of the kernel. The should only whine when development stops, not when useful changes are taking place...

I have gotten a lot of mail on this so I will reply only once.

Many of the professional industrial uses of Unix were previously covered using Sun boards, boxes and SunOs. If you ever dial 10 before a long-distance number to get a cheaper rate, that's voice over IP and we make that stuff. This was developed on Suns, runs on them, but will soon be running on cheap Intel clones.

If you ever have to go to the hospital and have a CAT-Scan or a MRI, you are using equipment developed by us, even though the name on the box may be Phillips, General Electric, Toshiba, or various other companies. You can look http://www.analogic.com and see what we do for a living here.

The Sun driver interface has been constant. Unfortunately, you have to install it, meaning link it and reboot. When Installing a system, meaning the complete software package, the end-user's technician installs the OS from a CDROM. Then the application with its drivers are installed from another CDROM. This works on Suns and has been the De-facto standard way of doing things.

Linux was not suitable for the applications running on Suns until Linux provided the installable device driver. The ability to install a hardware-interface module into a kernel was my main selling point for using Linux to replace SunOs, and, indeed the whole Sun architecture.

Incidentally, the cost is the same. A CDROM for Solaris is essentially the same cost as a CDROM for Linux. Once you start distributing an operating system and supporting the distributors, a "free" operating system is no longer free.

By the time a decision was made to produce our new Exact Baggage Scanner, marketed by Lockheed-Martin, engineering management was dragging its feet on the use of Linux. They wanted something that was "everything to everybody", but didn't want the cost of using Suns. Further, it had to be completely under company control.

I was unable to convince anybody to use Linux so I had to write my own Operating System. It is called ARTOS (Analogic Realtime Operating System). Our Sky Computer Division, which produces the world's fastest (still) digital signal processor, made the high-speed stuff, a lowly Intel Pentium with my OS is used as the system controller, and an Alpha Workstation is used for the user interface.

When this was completed, we went on to producing our third generation CAT Scanner. This uses a Pentium as the main system controller and Linux as the operating system. The User Interface uses Windows-NT. It was felt that Linux was sufficiently well-hidden in the bowels of the machine so nobody would care.

The drivers in this machine comprise both block and character devices. One of major building blocks is the driver that interfaces to the Digital Signal Processor. This DSP board comprises up to 32 TMS-320C20 DSPs plus an i960 for interface. It is made by our CDA Division.

Completed data, available within a 32k window, a 512x512x16bit chunk, must be transferred to the User Interface within 1/4 second to make the specification. It does.

Now, our legal department has defined the criteria we must meet to use Linux. They presume that we will provide a "current distribution" of Linux to every end-user. They also defined that, since drivers may be deemed to modify the operating system, we have to provide driver source-code to the customer if they request it. Application code continues to be proprietary.

Changing the kernel interface to drivers is counter productive. In fact it makes the usual field installation impossible. The usual installation would automatically and transparently compile the interface modules, using the new Operating System. This is no longer possible because the compilation will fail.

Again, if Linux is to become other than an interesting experiment, one cannot change these interfaces without understanding the whole picture.

Distributors don't care. The more changes there are, the greater the obsolescence, the more money they make selling new boxes of CDROMs. Therefore there is no controlling negative feedback to be obtained from the distribution channel. You can reject what I say out-of-hand, and continue as an experiment, or you can listen and make a significant contribution to providing jobs worldwide.

It is, of course, possible to fragment Linux. A company could be started, called StableLinux that distributes only Linux n.n.n and performs bug-fixes and maintenance on that version only. This is not helpful to the greater Linux community. Instead, we need to minimize the changes that affect the interfaces to world-wide applications. Just as POSIX attempted to stabilize the API so that one could write "portable" code, the interface to hardware that hasn't even been invented yet has to be stable.

No. According to our Legal Department, to satisfy the GPL requirement that we provide source to the end-user, they required that we supply a "current" distribution of Linux if the end-user requests it.

This seemed, by them, to be an easy solution to possible problems. Unfortunately, for Engineering, this means that we have to keep everything "current" during development so that, by the time equipment is shipped, it will run with the "current" distribution (whatever this is).

The obvious solution, given these constraints, is that we just ignore all changes until shipping time, then attempt to compile with the latest distribution, fixing all the problems at once. However, we then end up shipping untested software which ends up being another problem. Checking to see if it "runs" isn't testing software in the cold cruel world of industry.

So, presently, I have 13 drivers I have to keep "current". Yesterday they all got broken again. A week before, half of them were broken because somebody didn't like a variable name!

That said, a major problem with changes that I see, is that the changes are made without the notion of a terminating condition. For instance, new parameters are being passed to existing interface functions.

If you are going to break an interface, you should plan on only breaking it once rather than opening the door for more changes and leaving it open. For instance, once you have to pass more than (depends upon the machine) about 3 parameters, it's best to put them all in a parameter- list (structure) and pass only the address of the parameter list (pointer).

From that time on, you only have to add structure members to the parameter list if you have to add changes. If I had seen these kinds of changes I would not have complained. It means I have to rework stuff only once.

So `read(f,.......)` should have been changed to `read(params *)` and you are done with it forever as long as you don't change structure member names and functions for kicks.

Oh, I didn't knew that. I know that this is no common malloc() bomb problem, and we haven't heard about it, so we want to make it public, even if it is known to the kernel developers. A bit pressure to the admins side could not be wrong to use resource limits.

Btw, any BSD we tried on doesn't suffer from this or similar problems.

my point was, I believe in both, full-non-disclosure (keeping something to just yourself or to a very small group) or full-disclosure (sharing it with everyone at the same time).

The point now is, that many Linux distributions ship with no resource limitations activated by default, and a lot of administrators don't know about them or how to enable them. By raising public attention to this problem you bring many administrators to raise the barrier by enforcing resource limits, which is good.

Creating this pressure is often seen in a bad way by both developers ("we developers want to be notified first to fix it") or by companies, of which some ignore security issues until enforced by customers to fix them (hi bill).

However, I think of this pressure as a necessary thing, and that was what I meant in the mail to Khimenko.

On the other hand, Unix wasn't build for DoS-users, and I'm sure Alan is able to crash mostly anything. But using resource limits anyway is a good thing and any admin should use them.

You'll notice that the second paragraph is the stock "restricted countries" list and the third paragraph is a "safe haven" clause for ftp/http posting.

This basically says that crypto source code which is unencumbered may be exported merely by notifying them of the URL (mailto URL's????) where it is available from. No review, no approval, no license, no key length silliness, and no inherited encumberances. :-)

I won't post the whole $#@$#@ thing (since you can read it at the CDT site anyways) but for things like "Idea" and "RSA", which ARE encumbered by patents, similar clauses exist at 740.17(a)(5) which say basically the same thing.

This is scheduled to become finalized on January 14. Everything I have heard indicates that there will be no significant changes at this point and these will be the new regulations and will be finalized on schedule.

If these regs get finalized and are in the form we now expect them to be in, can we get the paperwork filled and get IPSEC (and other crypto goodies like ppdd) into the 2.4 kernel? KLIPS (from IPSEC) would be a wonderful win! That would put us up with OpenBSD with integrated IPSEC (OK, IKE, aka pluto, still needs improvement - but that's not a kernel issue).

We can also begin to lobby the distro makers for bundling hardened crypto like PGP, GPG, CFS, TCFS, SSH, etc, etc, etc, as quickly as possible. The faster it's there and the faster it spreads the better we can seal this deal and make it done!