Kernel Traffic #70 For 5 Jun 2000

1. Fixed

   1. Tulip hang on rmmod (fixed in .51 ?)
   2. Incredibly slow loopback tcp bug (believed fixed about 2.3.48)
   3. COMX series WAN now merged
   4. VM needs rebalancing or we have a bad leak
   5. SHM works chroot
   6. SHM back compatibility
   7. Intel i960 problems with I2O
   8. Symbol clashes and other mess from _three_ copies of zlib!
   9. PCI buffer overruns
   10. Shared memory changes change the API breaking applications (eg gimp)
   11. Finish softnet driver port over and cleanups
   12. via rhine oopses under load ?
   13. SCSI generic driver crashes controllers (need to pass PCI_DIR_UNKNOWN..)
   14. UMSDOS fixups resync (not quite done)
   15. Make NTFS sort of work
   16. Any user can crash FAT fs code with ftruncate
   17. AFFS fixups
   18. Directory race fix for UFS
   19. Security holes in execve()
   20. Lan Media WAN update for 2.3
   21. Get the Emu10K merged
   22. Paride seems to need fixes for the block changes yet
   23. Kernel corrupts fs and gs in some situations (Ulrich has demo code)
   24. 1.07 AMI MegaRAID
   25. Merge 2.2.15 changes (Alan)
   26. Get RAID 0.90 in (Ingo)
   27. S/390 Merge
   28. NFS DoS fix (security)
   29. Merge the RIO driver
   30. Fix Space.c duplicate string/write to constants
   31. Elevator and block handling queue change errors are all sorted
   32. Fix all remaining PCI code to use new resources and enable_Device
   33. Make sure all drivers return 1 from their __setup functions (Done ?)
   34. Enhanced disk statistics

2. In Progress

   1. Merge the network fixes (DaveM)
   2. Finish I2O merge (Intel/Alan)
   3. Complete vfsmount merge (Al Viro)
   4. Merge removed-buf-open directory stuff into VFS (Al Viro)
   5. vmalloc(GFP_DMA) is needed for DMA drivers (Ingo)

3. Fix Exists In -AC Tree

   1. Signals leak kernel memory (security)

4. Fix Exists But Isnt Merged

   1. msync fails on NFS (probably fixed anyway)
   2. Semaphore races (fix in 2.2)
   3. Semaphore memory leak (fix in 2.2)
   4. Exploitable leak in file locking (Willy)
   5. Mark SGI VisWS obsolete
   6. 64bit lockf support
   7. TTY and N_HDLC layer called poll_wait twice per fd and corrupt memory
   8. ATM layer calls poll_wait twice per fd and corrupts memory
   9. Random calls poll_wait twice per fd and corrupts memory
   10. PCI sound calls poll_wait twice per fd and corrupts memory
   11. sbus audio calls poll_wait twice per fd and corrupts memory
   12. Support MP table above 1Gig (Ingo)
   13. Finish sorting out VM balancing (Rik Van Riel)
   14. access_process_mm oops/lockup if task->mm changes (Manfred)
   15. Dont panic on boot when meeting HP boxes with wacked APIC table numbering (AC)
   16. SMP affinity code creates multiple dirs with the same name (Ingo)
   17. Scheduler bugs in RT (Dimitris)
   18. TLB flush should use highest priority (Ingo)
   19. Set SMP affinity mask to actual cpu online mask (needed for some boards) (Ingo)
   20. Fix eth= command line
   21. HFS is still broken
   22. AIC7xxx doesnt work non PCI ? (Doug says OK, new version due anyway)
   23. 8139 + bridging fails

5. Security

   1. put_user is broken for i386 machines (security) - sem stuff may be wrong too
   2. Fix module remove race bug (mostly done - Al Viro)

6. To Do

   1. IDE fails on some VIA boards (eg the i-opener)
   2. Find out what has ruined disk I/O throughput.
   3. Restore O_SYNC functionality
   4. Trace numerous random crashes in the inode cache
   5. VM kswapd has some serious problems
   6. Test other file systems on write
   7. Audit all char and block drivers to ensure they are safe with the 2.3 locking - a lot of them are not especially on the open() path.
   8. Stick lock_kernel() calls around driver with issues to hard to fix nicely for 2.4 itself
   9. PCMCIA/Cardbus hangs, IRQ problems, Keyboard/mouse problem (may be fixed ?)
   10. Use PCI DMA by default in IDE is unsafe (must not do so on via VPx x<3)
   11. Use PCI DMA 'lost interrupt' problem with some hw [which ?]
   12. Crashes on boot on some Compaqs ? (may be fixed)
   13. pci_set_master forces a 64 latency on low latency setting devices.Some boards require all cards have latency <= 32
   14. Loopback fs hangs
   15. Problems with ip autoconfig according to Zaitcev
   16. pci_socket crash on unload
   17. truncate_inode_pages does unsafe page cache operations
   18. heavy swapping corrupts ptes
   19. Linux sends a 1K buffer with SCSI inquiries. The ANSI-SCSI limit is 255.
   20. Linux uses TEST_UNIT_READY to chck for device presence on a PUN/LUN. The INQUIRY is the only valid test allowed by the spec.

7. To Do But Non Showstopper

   1. Make syncppp use new ppp code
   2. Finish 64bit vfs merges (lockf64 and friends missing)
   3. NCR5380 isnt smp safe
   4. DMFE is not SMP safe
   5. ACPI hangs on boot for some systems
   6. Go through as 2.4pre kicks in and figure what we should mark obsolete for the final 2.4
   7. Per Process rtsigio limit
   8. RtSig limit handling bug
   9. Fix SPX socket code
   10. Boot hangs on a range of Dell docking stations (Latitude)
   11. iget abuse in knfsd
   12. Some people report 2.3.x serial problems
   13. USB hangs on APM suspend on some machines
   14. PCMCIA crashes on unloading pci_socket
   15. DEFXX driver appears broken
   16. ISAPnP IRQ handling failing on SB1000 + resource handling bug
   17. TB Multisound driver hasnt been updated for new isa I/O totally.
   18. Fix boards with different TSC per CPU and kill TSC use on them

8. Compatibility Errors

   1. Xterm broke in 2.3.99pre6 (FIONREAD/select loop)

9. Probably Post 2.4

   1. per super block write_super needs an async flag
   2. addres_space needs a VM pressure/flush callback (Ingo)
   3. per file_op rw_kiovec
   4. PIII FXSAVE/FXRESTORE support

10. Drivers In 2.2 not 2.4

11. To Check

    1. Check O_APPEND atomicity bug fixing is complete
    2. Protection on isize (sct) [Al Viro mostly done]
    3. Mikulas claims we need to fix the getblk/mark_buffer_uptodate thing for 2.3.x as well
    4. Network block device seems broken by block device changes
    5. Fbcon races
    6. VFS?VM - mmap/write deadlock (demo code seems to show lock is there)
    7. rw sempahores on page faults (mmap_sem)
    8. kiobuf seperate lock functions/bounce/page_address fixes
    9. Fix routing by fwmark
    10. Some FB drivers check the A000 area and find it busy then bomb out
    11. rw semaphores on inodes to fix read/truncate races ? [Probably fixed]
    12. Not all device drivers are safe now the write inode lock isnt taken on write
    13. File locking needs checking for races
    14. Multiwrite IDE breaks on a disk error [minor issue at best]
    15. ACPI/APM suspend issue - IDE related stuff ?
    16. NFS bugs are fixed
    17. BusLogic crashes when you cat /proc/scsi/BusLogic/0
    18. Floppy last block cache flush error
    19. NFS causes dup kmem_create on reload
    20. Chase reports of SMB not working
    21. exec loader permissions
    22. Locking on getcwd
    23. floppy fails on some machines
    24. IRDA calls get random bytes before random is set up
    25. Some AWE cards are not being found by ISAPnP ??

I have now completed and tested my second TSC patch. It is available at: http://hypocrite.org/linux/tsc.patch.new.tar.gz

This works with 2.3.99-pre8 at least, probably most others. This one disables use of the TSC by an option in the configuration (i.e. make config, make menuconfig, whatever) and therefore has NO performance impact on TSC-enabled kernels. The disadvantage is that this is another option to set in your configuration.

The old patch is available at: http://hypocrite.org/linux/tsc.patch.new.tar.gz

And works for at least 2.3.99-pre6 and pre8, probably most others. This one doesn't use any configuration but rather, determines the situation on bootup. There were some legitimate complaints about this.

I welcome comments and suggestions for improvement. I specifically would like to know how well it works for you and whether it works on a laptop which varies CPU speed (particularly an SMP laptop if such exists).

Just to note: Win2k *does* allow SMP machines with different speed CPUs, despite several people's claims to the contrary.

We'd love to, but we don't touch code like that. We could have made the old bogomips give infinite values if we _really_ wanted to, but code like that has one purpose, and one purpose only: delaying. So we never remove those kinds of loops.

(And in any case it would have been extra logic - empty loop removal is very much a special case, so whatever we did would have been an extra wart rather than anything that came really naturally to the way morphing works..)

There's a ton of funky stuff we _could_ do if we wanted to implement the x86 "right". Sadly, we have to be quite careful ;)

[ You _really_ don't want to know about taking page faults in the middle of a task switch that is the result of a task gate being used to handle another page fault, which in turn happened while we were loading the second part of a page-crossing 32-bit word. Question: do you mark the page table entry for the incomplete part that you already checked as accessed or not? ]

It actually depends on the CPU, and the type of access.

If it's a "atomic" 32-bit access, you don't mark the PTE accessed, because the page fault will have been noticed before any data was actually fetched. But that is not true for some data that has more "structure" to it that can be loaded as separate pieces and then a page fault part-way through can have already set some of the accessed bits.

The reason it is CPU-dependent seems to be that with speculative loads and out-of-order execution the whole thing becomes much more subtle, and "separate pieces" is no longer the firm thing it used to be. A Pentium core will act differently from a PPro core in many of these special cases.

Nobody sane obviously should ever care. It's hard to even test for, in many cases. We have some rather insane tests, and there's a number of differences between different chips.

Note that many of these things aren't even documented to work. Intel explicitly documents that if you use task switching hardware the TSS should all be on a single page etc. And nobody uses the hardware any more now that even Linux does it all in software. But it's often more painful to be different and have to convince everybody that those differences do not matter than to just try to look exactly like everybody else as far as humanly possible.

We've for the longest time occasionally considered just removing it. Th enotion of "sleep_on()" is a very simple notion, and people understand what it's all about, but most of the uses tend to be buggy.

It might be a good idea to remove sleep() altogether during early 2.5.x. Please remind me. It shouldn't be that painful for most people (but there are probably tons of drivers that need to be updated and tested - 2.5.x is definitely the right point to do this).

I suggest creating LKBS:

• includes test programs the kernel developpers have written
• runs somewhere early after booting (perhaps by starting it instead of /sbin/init, or in a seperate runlevel)
• records all relevant data from the machine (kernel version, config, cpu, memory, drives, etc.)
• puts all data together in easilly parsed (both human and machine ofcourse) file and sends it to a main benchmark repository
• should be free for download from a natural place (ftp.kernel.org, package for your $DISTRO) and easy to install/run.

I don't know what you think of it, or if there already are such things. If not, I'd happily volunteer to start this project.

I'd really love it if this effort got started. I have a huge backlog of lmbench submissions that I have not processed and put up on the web. I just don't have time these days but the data is there.

Furthermore, I've done a lot of work like what you are proposing and I'd like Linux to benefit from that, but I can't start a project to do that. On the other hand, if there are people who want to work on it, I can certainly help focus the efforts. I've literally spent years doing this sort of thing at SGI & Sun and it seems a waste that I'm not passing on that knowledge/experience. So I'll help if someone else wants to drive.

I've just checked out lmbench, that is indeed what I envisioned, except that it does some general benchmarks that apply to most Unices, and I think many developpers like to see their own Linux specific benchmark to go into that too, so they can gather information about the performance of their driver/patch. It was also a little hard to find (I had to use FTPsearch and follow some links), you should put it on Freshmeat IMO :).

This, combined with an automated repository that would gather all information and send digests on request to developpers, would certainly benefit development I think.

This definitely sounds like an excellent idea to me. I'd be happy to help out, so if you'd like a hand in organizing this, setting up the benchmarks, or creating the site for the benchmark results let me know.

IMO, there are two comparisons that would be really useful in helping to pinpoint performance problems or bugs in the kernel:

1. tracking how different kernels perform on a stationary hardware platform (ie fixing all hardware variables)
2. looking at the difference between different hardware platforms for a given kernel

It would also be very nice to link to lists of changes between versions, to help people figure out _why_ performance certain changes in performance occurr.

It might makes sense to move a discussion of this off the kernel list. I'd be happy to set up a discusion list for this - just let me know if people are interested.

When analyzing a kernel code change, it is sometimes not possible to deduce the true meaning of the change from the diff itself. When this happens, analysis of the surrounding code is required, sometimes accompanied by mailing list searches and such.

Occasionally, we even have bad patches which get applied and then un-applied, over the course of six months or a year. From many perspectives -- long term maintenance, short term change evaluation, general code understanding -- a small changelog entry for each change seems advantageous. I think GNOME, egcs, and other projects using changelogs have experience which bears this out.

Since the kernel doesn't have a canonical source code repository[1], we must current on the human (but still brilliant) minds of Linus, Alan, DaveM, sct, tytso, and others. Therefore I propose to have kernel changelogs.

Disadvantages exist too... Gotta get Linus, Alan, and other big contributors to agree. It takes more time, especially for people like Alan who feed lots of changes to Linus. And people will inevitably bicker on changelog entry format.

Three small technical notes:

• I prefer an egcs-style format (sample below), but don't feel strongly about the issue.
• ChangeLogs are generally not patch-friendly, due to the high possibility of conflicts. egcs'ers appear to cut-n-paste ChangeLog entries as necessary. If changelog entries are to be kept separate from patches (which seems reasonable if not necessary), this imposes an addition burden on Linus for each patch sent to him.
• ChangeLogs for drivers/{char,sound,etc} would be nice, but may be unreasonable. Documenting core and arch changes seems more important than doc'ing driver changes.

Overall I believe the advantages outweight the disadvantages. I've always felt that sharing knowledge around has a positive effect, and IMNSHO the Linux kernel community occasionally suffers a bit due to lack of documentation on changes.

Linux-2.2 is using the 'one-hand' but 2.3 uses a single LRU. Current fine BSD O/Ses seem to use the active/inactive queue algorithm that may well come from Mach (historians should confirm).

The 'one hand' algorithm made full sense at the time memory size were not that large. IIRC, the 'two hand' algorithm has been invented by BSD guys when memory size increased, for the reason it took too long to do a revolution of the map. This let me think that the 'two-hand' algorithm was rather an hack than a really superior algorithm. Btw, this happened may-be 10-15 years ago, so this may well be just off-topic nowadays. :)

No.

It doesn't really exist yet.

But I'll be gone for three weeks, and in the meantime there's a "2.4.0-test1" kernel here. It's not a real 2.4.0 release, but we should be getting closer. There's going to be other test-kernels after this one, and we'll find bugs. And bad behaviour. And wonderful features which we'll document some day.

Alan Cox is most likely to maintain a "2.4.0-ac" series while I'm gone. Not the same thing as 2.4.0 either, but by testing these test-kernels out and giving feedback, we'll get to that some day.

Have fun. And let's see how many people find this without it even being announced ;)

2.4.0-test1-ac1

• Statically init the i2o config semaphore (Juha Sievanen)
• Fix the eth= command line and extend it (Pekka Riikonen)
• Rip out the 2.2 GHOST hack for DVD's (Jens Axboe)
• Fix VCD playing on some Plextor drives (Jens Axboe)
• Intel 840 GART support (Tony Hoyle)
• Clean up initio resource handling (Thomas Graichen)
• Fix COMX compile problems (Francois Romieu)
• Fix incorrect altstk over exec inheritence (Bruno Haible)
• Support etherexpress 10 isa (82595FX) (Aristeu Filho)
• Fix proc reporting of md (Brian Kress)
• Reorder fs/locks.c but no function changes (Matthew Wilcox)
• Come out of APM suspend with interrupts off (Andy Henroid)
• Fix procfs sysctl to use structs (Tigran Aivazian)

Base point differences between Alan's tree and Linus

• Newer sk98 docs
• Ramdisk defaults to 8192 for S390 (needs cleanup)
• Debugging reminder on adbmouse
• Format changes only on atarimouse
• /proc fixes on kcapi
• /proc fixes on sg
• Maestro init cleanup
• Semaphore for ixj cards
• Version.h used in clgenfb
• PLX PCI ids
• Revert initrd change
• Ksym fixes for RAID
• Proc tidyup for ip_chains