Kernel Traffic #63 For 17�Apr�2000

1. Full Asynch IO Manager (SMP)
2. NetWare-ish LRU Mirrored Block Cache.
3. Handle Based Virtual Partition Mirroring and Hotfixing Engine (NWVP).
4. Full SMP Support (and we even tested it).

The Linux Buffer Cache does not present a logical block cache for NetWare's flavor of mirroring support, although basically what's there is implemented as a Logical Block Cache on top of the Linux Buffer Cache. You will notice that NWFS can use either it's own LRU or the linux buffer cache (there's an #if/#else/#endif for LINUX_BUFFER_CACHE in block.c -- this will make NWFS sit on top of buffer.c). You would want to crank down the MAX_BLOCK_BUFFERS count in nwfs.h to something smaller than 2000.

The Async processes bascially provides an elevator for remirroring and concurrent mirroring I/O so the file system doesn't get starved. It sits on top of the linux disk interface. These abstractions are also necessary for our clustered file system for Linux (M2FS). Look carefully at the layering in nwvp.c, and it will be clear why we did what we did.

This code base also runs on several other platforms, and some of what is there is not applicable to Linux, but is to NT and DOS. The tools use these interfaces and we have to have them as convinient abstractions since they are not available on every platform (not all OS's are as blessed as Linux).

What's here is a smaller version of the internal architecture of Native NetWare. Some of the services and what they do is a little different.

Wasn't trying to push anything, there are lots of people who want NetWare stuff, just trying to make Linux more attractive to Novell's 9,000,000 server installed base and get the NetWare file system on Linux out the door. You guys are always welcome to carve up NWFS like a christmas turkey and chunk the pieces that don't make sense for Linux moving forward. You'll find I'm not too religious about the sanctity of my own code, I'd rather be a contributor and team player with you guys. :-)

With the page cache, the file systems in Linux are more or less reduced to meta-data drivers (very similiar to where NT is at today). The extra layers are not really all that heavy, and are there to support our clustering stuff (which we will open source as well). Most of them use fast paths in the code (which you will see if you take the time to look at it). Unlike most of the File System code in Linux, NWFS has COMMENTS, NOTES, EXPLANATIONS, DETAILED TECHNICAL INFO, etc. :-)

You guys might learn something by looking at it. I am right at this moment staring at the NCPFS code in 2.3.99Pre3 since there is NO DOCUMENTATION ON THE PAGE CACHE, DCACHE, VFS or anything else here to help besides pouring over source code. :-)

Don't knock NetWare til you try it. :-)

Oh Yeah, and I almost forgot. Remember those emails I sent you 6 months ago. The one's asking for your help with creating an md driver for the NetWare mirroring so I didn't have to replicate a NetWare style LRU so I could tightly integrate NWFS with the Linux Buffer Cache and md (in fact, I even was willing to give you the nwvp code and sent it to you so you could put it into the current RAID driver)?

Did you ever respond. NO ......

How many emails did I send you? Was it seven or eight?

Did you answer any of them. NO .......

If you are still interested in doing what I suggested, unlike you, I will answer your emails and help you take over the code (and let you control it for Linux) so Linux will be able to enter Novell's market with a tighter implementation.

Think about it ......

:-) :-) :-).

Touche .....

As I understand it, your nwfs is probably the first filesystem to have been successfully "ported" to Linux. Pretty much everything else (with, perhaps, the exception of the abomination that is the NTFS driver) started off native.

The multiple times that I have written 30 to 70% of a filesystem, I found the romfs and minixfs code to be most instructive as a guide to the VFS interfaces. The buffer and page cache stuff is rather harder to track down canonical examples for, though again minixfs is pretty helpful, if rather simplistic.

Whenever Ion and I wanted to understand the VFS-f/s interaction, we often looked at the VFS code, and samples of three file systems:

• msdos/fat: it's a small block-level f/s. small enough that you can follow most of the code and understand it.
• ext2: does much more, in case msdos is too simple, but often we didn't need to look at ext2fs.
• nfs: b/c so much of the VFS was "hacked" to support NFS. This helped us understand things that we didn't need to worry about in our stacking templates, b/c they were specific to NFS.

The Open Source Code for NWFS 2.2.1 hss been posted to www.timpanogas.com and 207.109.151.240. This version corrects a reported mirroring coalescence problem and fixes a bug in the nwvp_vpartition_map_asynch_write() function.

NWFS 2.2.1 Sources are in the nwfs0328.zip and nwfs0328.tar.gz files on our FTP server at 207.109.151.240 or www.timpanogas.com. We will post NWFS 2.2 for Linux Kernel 2.4 support tommorrow morning at 7:00 a.m.

I am seeing an oops and message indicating that the VFS buffer free lists are getting corrupted under 2.2.15 when I use NWFS with multiple SMP threads with the Linux Buffer Cache. The NWFS LRU works just fine, however.

If I put a big lock over all accesses and make all the interactions with the buffer cache synchronous, the problem goes away.

Is it required that I hold the kernel lock with lock_kernel() over the Linux Buffer Cache when doing I/O to it in 2.2.15?

NetWare does not follow the Unix model of "one partition, one file system tree" (which is an ancient and limiting architecture for File Systems). The model linux is taking is almost identical to NT, a model where the I/O subsystem presents single partitions through a very restrictive I/O interface. (Doing this in NT is extremely nasty -- I have to build a disk and partition pointer map in user space, then pass it to the driver via an IOCTL to the Windows NT NetWare File System).

NetWare uses a multi-segmented architecture where several segments for a particular volume can be stripped across multiple drives (a single NetWare partition can host 8 volumes segments per partition, and these segments can be for the same volume or 8 other stripped volumes). I need to be able to scan the volume tables on each NetWare partition in order to build a global map of all NetWare volume segments on all drives (a single sector read per drive to locate the volume segment tables).

I also need it to locate all mirror groups and mirror members before bringing the mirroring and hotfixing engine on line. This is why. If there is a "cleaner" way to do this with a published API, please show me.

I don't see devfs creating anything for my usb printer when I plug it in, or the usbmouse either, for that matter. I've got 2.3.99pre3 and I've got devfsd running. If these things are supported, why isn't anything happening?

Patches for devfs support for the Computone boards were submitted weeks ago, but they're not in the kernel sources yet and I doubt their going to make 2.4.0. How many other drivers have no support? Somehow, I suspect that most of the intelligent multiport boards fall into that category and I don't see anything in the pcmcia stuff either. I agree that some things "fall out in the wash" when they register things like standard tty ports and drives. But that still leaves a lot out that haven't been done yet. Sure, I figured out that you aren't having any problems with this because you aren't using any of these things.

I also sent Richard (and the devfs list) a list of the ports and devices that the Computone boards use that are not in devfsd. Maybe one day they'll show up in devfsd and maybe they won't. I haven't heard a peep back on that front. I'm sure the patches will make it into the kernel eventually, I'm just not sure when. That's in the hands of the man in charge.

A number of people thought devfs was a bad idea. It went in, anyway, during feature freeze, because Linus was God.

The reason why I dislike devfs is precisely demonstrated by a lot of people kvetching with the namespace. It causes policy to be dictated in the kernel; in this case Linus single-handedly dictated the new naming convention, and now everyone has to live with it. Of course, it's *his* kernel, so he has the right to do this, even if it causes a lot of people pain and annoyance. Personally, it'll be interesting to see how many distributions actually ship with devfs turned on, or with devfs mounted somewhere else, like /devfs.

A number of user utilities, such as e2fsprogs, need modification to work well with a mandatory devfs with the new names. I don't plan to do that work right away, since I have other higher priority items to worry about. So if any distributions do ship with devfs, they *will* have to ship with devfsd. Perhaps some of them will simply decide to not ship with devfs at all. (It after all is still marked as experimental.)

Source code for Linux XFS now available!

A complete linux 2.3.99pre2 tree including the XFS filesystem is available for cvs checkout.

Please refer to: http://oss.sgi.com/projects/xfs/cvs_download.html for instructions.

A snapshot of the CVS tree is also availble: ftp://oss.sgi.com/projects/xfs/ftpdir/03302000linux-2.3-xfs.tgz This tar file will not be generated on a reqular basis. A "cvs update -d" should be performed once the tree is download and unpacked.

*************** PRELIMINARY WORK IN PROGRESS CODE *****************

While most of the basic functionality of the XFS file system is working, this code is still very unstable.

IT MAY CRASH OR HANG YOUR SYSTEM! sooner or later. THIS IS BLEEDING-EDGE CODE.

This release has only been tried on IA32 systems. People are certianly free to work on getting it running on other architectures.

Many of the more advanced XFS features are yet to be completed:

• Guaranteed Rate IO
• Direct IO (Raw I/O)
• Real time IO
• Delayed Allocation
• Integration with Volume Managers
• Access Control Lists.
• etc...

For a list of items currently being working on or soon to be worked on refer to: http://oss.sgi.com/projects/xfs/todos.html

This list will updated as new items are found.

A beta release is planned in a few months and at that time we will release an xfs rpm.

There is a linux-xfs@oss.sgi.com mail list that you can subscribe to and watch problems/issues as the get fixed and are found.

Please e-mail linux-xfs with any issues/problems/... that you find in the code or while running.

If you want to help with specific work items, please e-mail xfs-masters@oss.sgi.com.

In my own experience :

• reading and writing with one CPU is ok with 2.2.15pre15 and 2.3.51
• reading with 2 CPUs with 2.2.15pre14 and 2.3.51 doesn't work
• reading with 2 CPUs with 2.3.99pre3 is fine.

now, for the SCSI version of OnStream tape drives, Documentation/ide.txt mentions that no support is currently available. I havent' try though, and I don't know how updated this file is.

as far as realiability is concerned, I have written 2 tapes so far, about 10Gigs on a 15 actual gig tape :

• I had 1 crc error on the first
• I had 3 read/write error on the second

please remember, this is an IDE tape drive. SCSI tape drives perform better usually.

performance is as expected, no more no less.

Sorry, but I have to say that that has to be the dumbest idea I have ever seen.

Reading audio off of an audio CD is not a perfect process. If you read the Audio CD specification you'll notice that the best resolution you can get is 1/63 of a second (since every frame is 1/75 of a second). So to fix that problem you need things like jitter correction algorithms (unless your CD-ROM already does it) and algorithms to correctly get around scratches.

Trying to put all that crap in the kernel is pretty dumb. And if you just do a half-assed job and only let it work for perfect non-scratched CDs on CD-ROMs with built-in jitter correction, then we will have even more half-assed MP3s with pops and skips in them.

By far, the best thing to do is keep this crap OUT of the kernel. The cdparanoia program can do just about anything you want, and NEVER pops or skips even on shitty drives. If you are making MP3s of CDs I strongly suggest that you use cdparanoia and a good encoder.

As someone who has written an extremely sophisticated audio cd filesystem, I must say that I completely agree with Jens. It's a BAD IDEA. Actually, he yelled at me about this about a year ago :) This would be a great topic to add to the FAQ!

There are two schools of thought:

1. Things like jitter and scratch correction are not obvious kernel functionality. They bloat the kernel and can be done just as effectively in user space. Implementing corrections for drives that do it with hardware is a waste. Not implementing corrections is inadequate for the rest. The whole idea behind standards like ide and SCSI is that the drivers can be more generic.

2. The CD-ROM is a device, and doing corrections are the job of the driver. However, the audio sector size is bigger than the pagesize and this means doing translation if the filesystem is to use the normal blkdev approach.

   The inodes and superblock would all have to be virtual since the only directory structure on an audio cd is the TOC. I have no problem with this, personally.

   This means that the cd drivers (all of them) would need to be changed to translate between pagesized blocks and framesize blocks and then have a fairly intricate cache scheme. Otherwise, the audiofs implementation would need to make ioctl() calls. I, for one, don't think that the kernel should use ioctl's. Personally, I don't like them because the kernel has to turn off memory write verification using get_FS() and set_FS(). I'm not happy about that.

Now, honestly, do you think Option (2) is worth all that? Having implemented it all (for fun, mainly) I can tell you to stick with the userspace rippers/encoders. They are the right way to go.

Isn't this what makes Linux good? We don't have to have absolute, rock-solid, hard-and-fast rules about what goes where, or how it is done.

We can use what has gone before as precedent, but not be tied by it.

Do we always have to draw a line?

But it's not consistent. I'm not asking for dogma. If it's not written down somewhere (even if marked "in the opinion of the author"), then every new FS^H^Hpiece of code has to survive a trial by flamewar. And inevitably, the same old ground is covered each time. It's wasteful of the time of the people flaming, and it makes linux-kernel more voluminous, which leads to more people filtering it (or unsubscribing).

Something that might work is that opposing views are written up, by respected people (i.e. they should have contributed real code, rather than vocal hangers-on), and referred to in the FAQ. And they should have links to each other, as well.

IMHO this flamage/trial by fire is a strength, not a weakness. Yes, each "new" idea generates the same/similar arguments. However, it is through the trial of the arguments that the ideas are tested and retested to see if they are valid.

Certainly anything which has gone through the wringer more than once deserves mention in the FAQ. If it generates identical arguments then it deserves to be ignored. However, what about when someone says I understand that a, b, and c were extensively discussed before, but here is a new answer which I think invalidates some of the objections and here is the code which demonstrates that.

Good/Bad/Indifferent ideas are certainly floatin around all over the place. I've seen a number of them, and a percentage of those have matured in their seperateness and eventually got included in the kernel. vfat support and devfs are cases in point.

We also have the court of final authority in our benevolent dictator, Linus Torvalds -- even if sometimes he claims to wear a brown bag. He's said more than once, "I won't even consider X, so don't bother submitting patches for it." Let's use that; such things also belong in the FAQ. The system works well, even if it occasionally generates numerous messages. That is what procmail recipies and the 'n' key are for.

Trial by fire excludes some very important parts of the world. There is a cultural issue here. Consider world population and economic strength, then notice a really major country from which there are seldom any linux-kernel posts. (no, English is not the most serious problem - many can write it well enough)

Yes, really, there are places where flaming isn't accepted.

Not that I have an answer though, and don't bother asking me to single out the place that comes to mind.

1. Fixed
1. Tulip hang on rmmod (fixed in .51 ?)
2. Incredibly slow loopback tcp bug (believed fixed about 2.3.48)
3. COMX series WAN now merged
4. VM needs rebalancing or we have a bad leak
5. SHM works chroot
6. SHM back compatibility
7. Intel i960 problems with I2O
8. Symbol clashes and other mess from _three_ copies of zlib!
9. PCI buffer overruns
10. Shared memory changes change the API breaking applications (eg gimp)
11. Finish softnet driver port over and cleanups
12. via rhine oopses under load ?
13. SCSI generic driver crashes controllers (need to pass PCI_DIR_UNKNOWN..)
2. In Progress
1. Merge the network fixes (DaveM)
2. Merge 2.2.15 changes (Alan)
3. Get RAID 0.90 in (Ingo)
3. Fix Exists But Isnt Merged
1. Signals leak kernel memory (security)
2. msync fails on NFS
3. Semaphore races
4. Semaphore memory leak
5. Exploitable leak in file locking
6. Merge the RIO driver (probably do post 2.4.0 as it is large) (in AC tree)
7. S/390 Merge (merged in AC tree)
8. 1.07 AMI MegaRAID
9. Any user can crash FAT fs code with ftruncate
10. Mark SGI VisWS obsolete
4. To Do
1. Restore O_SYNC functionality
2. Fix eth= command line
3. Trace numerous random crashes in the inode cache
4. Fix Space.c duplicate string/write to constants
5. VM kswapd has some serious problems
6. vmalloc(GFP_DMA) is needed for DMA drivers
7. put_user appears to be broken for i386 machines
8. Fix module remove race bug (mostly done - Al Viro)
9. Test other file systems on write
10. Security holes in execve()
11. Directory race fix for UFS
12. Audit all char and block drivers to ensure they are safe with the 2.3 locking - a lot of them are not especially on the open() path.
13. Stick lock_kernel() calls around driver with issues to hard to fix nicely for 2.4 itself
14. IDE fails on some VIA boards (eg the i-opener)
15. PCMCIA/Cardbus hangs, IRQ problems, Keyboard/mouse problem (may be fixed ?)
16. Use PCI DMA by default in IDE is unsafe (must not do so on via VPx x<3)
17. Use PCI DMA 'lost interrupt' problem with some hw [which ?]
18. Crashes on boot on some Compaqs ?
19. pci_set_master forces a 64 latency on low latency setting devices.Some boards require all cards have latency <= 32
20. usbfs hangs on mount sometimes
21. Loopback fs hangs
22. Problems with ip autoconfig according to Zaitcev
23. Still some SHM bug reports
24. SMP affinity code creates multiple dirs wit the same name
25. TLB flush should use highest priority
26. Set SMP affinity mask to actual cpu online mask (needed for some boards)
27. pci_socket crash on unload
5. To Do But Non Showstopper
1. Make syncppp use new ppp code
2. Finish 64bit vfs merges (lockf64 and friends missing)
3. NCR5380 isnt smp safe
4. DMFE is not SMP safe
5. ACPI hangs on boot for some systems
6. Get the Emu10K merged
7. Finish I2O merge
8. Go through as 2.4pre kicks in and figure what we should mark obsolete for the final 2.4
9. Per Process rtsigio limit
10. Fix SPX socket code
11. Boot hangs on a range of Dell docking stations (Latitude)
12. HFS is still broken
13. iget abuse in knfsd
14. Mark NTFS as obsolete
15. Paride seems to need fixes for the block changes yet
16. PIII FXSAVE/FXRESTORE support
17. Some people report 2.3.x serial problems
18. AIC7xxx doesnt work non PCI ?
19. USB hangs on APM suspend on some machines
20. PCMCIA crashes on unloading pci_socket
21. DEFXX driver appears broken
22. ISAPnP IRQ handling failing on SB1000 + resource handling bug
6. Compatibility Errors
1. exec() returns wrong codes on a file not found
7. Probably Post 2.4
1. per super block write_super needs an async flag
2. addres_space needs a VM pressure/flush callback
3. per file_op rw_kiovec
4. enhanced disk statistics
5. AFFS fixups
6. UMSDOS fixups resync
8. Drivers In 2.2 not 2.4
1. Lan Media WAN
9. To Check
1. Truncate races (Debian apt shows it nicely) [done ? - all but Coda]
2. Elevator and block handling queue change errors are all sorted
3. Check O_APPEND atomicity bug fixing is complete
4. Make sure all drivers return 1 from their __setup functions
5. Protection on isize (sct) [Al Viro mostly done]
6. Mikulas claims we need to fix the getblk/mark_buffer_uptodate thing for 2.3.x as well
7. Network block device seems broken by block device changes
8. Fbcon races
9. Fix all remaining PCI code to use new resources and enable_Device
10. VFS?VM - mmap/write deadlock
11. rw sempahores on page faults (mmap_sem)
12. kiobuf seperate lock functions/bounce/page_address fixes
13. Fix routing by fwmark
14. Some FB drivers check the A000 area and find it busy then bomb out
15. rw semaphores on inodes to fix read/truncate races ? [Probably fixed]
16. Not all device drivers are safe now the write inode lock isnt taken on write
17. File locking needs checking for races
18. Multiwrite IDE breaks on a disk error
19. AFFS doesn't work on current page cache
20. ACPI/APM suspend issue

I've attached two patches for 2.2.15 (made against 2.2.2.15pre16).

The first prevents the RealTek 8139 NIC from wedging after a receive FIFO overflow. There maybe a better work around for this, but what I've got works here.

The second is not a high priority patch. In ide-pci.c/ide_match_hwif(), the code assumes that MAX_HWIFS is at least two. For those of us trying to build small kernels, this isn't always true.

If you have one bad sector you should write the other 7..

Also the error recovery code was wrong. I've not checked this for a while and much has changed.

Yes, but that is a "multi-mode" RW.

I need to think how to do a complete recovery.

A preferred solution in my mind would all for resetting the request cue and retry in the next sector and continue on; however, the need to update the badblock table that is written/created or needs to be similar to the "badblocks" call from "mke2fs". Do you agree?

Linus, Tim and I come to you hand-in-hand with a patch to Space.c.

It does the following:

• Arranges for the eight "eth%d" and "tr%d" strings to be generated separately, rather than commoned up by the compiler.
• Arranges for said strings to be placed in .data, not .rodata
• Correctly pads all these overwritable strings out to length IFNAMSIZ with trailing nulls.

A patch against 2.3.99-pre4-1 is attached.

Ok, I have an alternative approach that I think should be a lot cleaner and that looks obviously correct. How about this one-liner patch instead:

include/linux/netdevice.h:

struct net_device {
...
- char *name;
+ char name[16];
...

The above gets rid of _all_ of the problems, and gets rid of the need to use the silly PAD macros - both the writable-strings and the PAD macro are only needed because this thing was misdesigned in the first place. Putting the device name into the net_device thing just automagically makes both things work correctly.

Sure, some drivers would have to be changed to do

strcpy(dev->name, namelist[index]);

instead of doing the current

dev->name = namelist[index];

but looking at at least one of the drivers (3c503.c), the whole and only reason for the name games is _again_ that what 3c503.c really wanted in the first place was really to have the name be done as an array rather than as a pointer (so at least 3c503.c would be trivially cleaned up a bit by that simple change too).

Less obviously correct are the modifications to the network drivers :-)

I did start this (just to see how bad it really was), and I have 46 modified source files. This is only looking at "drivers/net/*.c".

I have checked it compiles with all network drivers as modules, but that is all. I may have goofed in moving from 2.3.99-pre2 to 2.3.99-pre3 (no compilation test performed).

If you want to have a look, point an HTTP client at:

http://www.alfie.demon.co.uk/dev_name-2.3.99-pre3.diff.gz

My own feeling is that this is rather a large upheaval close to 2.4. Then again, it isn't my call. In addition, many of the modified drivers should be using init_etherdev anyway.

One program uses mmap/mlock to accomplish the lockdown, and calls munmap to release the previously locked chunk.

The other locks down a malloc-ed buffer for each file, and uses read to move the data into user space.

I was very disheartened to find that on my system the mmap/mlock approach took *3 TIMES* as long as the read solution. It seemed to me that mmap/mlock should be at least as fast as read. Comments are invited.

People love mmap() and other ways to play with the page tables to optimize away a copy operation, and sometimes it is worth it.

HOWEVER, playing games with the virtual memory mapping is very expensive in itself. It has a number of quite real disadvantages that people tend to ignore because memory copying is seen as something very slow, and sometimes optimizing that copy away is seen as an obvious improvment.

Downsides to mmap:

• quite noticeable setup and teardown costs. And I mean _noticeable_. It's things like following the page tables to unmap everything cleanly. It's the book-keeping for maintaining a list of all the mappings. It's The TLB flush needed after unmapping stuff.
• page faulting is expensive. That's how the mapping gets populated, and it's quite slow.

Upsides of mmap:

• if the data gets re-used over and over again (within a single map operation), or if you can avoid a lot of other logic by just mapping something in, mmap() is just the greatest thing since sliced bread.

  This may be a file that you go over many times (the binary image of an executable is the obvious case here - the code jumps all around the place), or a setup where it's just so convenient to map the whole thing in without regard of the actual usage patterns that mmap() just wins. You may have random access patterns, and use mmap() as a way of keeping track of what data you actually needed.

• if the data is large, mmap() is a great way to let the system know what it can do with the data-set. The kernel can forget pages as memory pressure forces the system to page stuff out, and then just automatically re-fetch them again.

  And the automatic sharing is obviously a case of this..

But your test-suite (just copying the data once) is probably pessimal for mmap().

It's not the IO on the pages themselves, it's actually the act of populating the page tables that is quite costly. And doing that in the background is basically impossible.

You can do it synchronously, and that is basically what mlock() will do with "make_pages_present()". However, that path is not all that optimized (not worth it), and even if it was hugely optimized it would _still_ be quite slow. The page tables are just fairly complex data structures.

And on top of that you still have the actual CPU TLB miss costs etc. Which can often be avoided if you just re-read into the same area instead of being excessively clever with memory management just to avoid a copy.

memcpy() (ie "read()" in this case) is _always_ going to be faster in many cases, just because it avoids all the extra complexity. While mmap() is going to be faster in other cases.

lilo works, but you must mount the reiserfs partition with the "notail" option. reiserfs tries to be efficient with disk space by taking the excess portions of files that do not fill up a whole block and it takes these "tails" and packs them together. This leaves holes in the files which prevents lilo from finding a continuous kernel image.

As root something like:

# mount /dev/root / -o remount,rw,notail
# cp -a /boot /boot1
# rm -rf /boot
# mv /boot1 /boot
# lilo

The process of copying the files after the remount will cause the files to all be rewritten without any tails