Kernel Traffic #45 For 6 Dec 1999

People tell me that vfork() no longer is equivalent to fork() as the manpage states. Unfortunately, they are right, so I wrote a new page - see below.

I consider the introduction of vfork into Linux a very bad move (as will be clear from the text I wrote), but since there were people writing code and submitting patches there must be some positive side to this horrible kludge.

If nobody corrects me, this will be the vfork man page in man-pages-1.27.

While in general, I agree with you that vfork is not the nicest thing in the world, it's not necessarily much worse than some of the other things you can do with sys_clone(), which also allows two processes to share virtual memory space.

Despite BSD's man page warning off folks from depending on vfork's semantics, there were programs --- including BSD's csh, which did use vfork() so that the path hashing statistics could be accurately updated. So, there is some excuse for actually implementing vfork() in the traditional bad old way.

I think you were being a little bit too harsh, myself, but it is true that warning people that using it compromises the portability of their programs is perfectly fair.

Just describe it the way it works.

vfork() under Linux is actually just another case of clone(), and the old reasons why it was considered horrible are basically all gone. The Linux mm layer evolved to the point where it was trivial to implement, WITHOUT any of the special hacks that the original BSD implementation had, and that made people hate it in the BSD community.

So the ugly part about vfork() doesn't exist any more, yet the good attributes still do.

I always thought that was just because strace didn't understand about vfork().

On the other hand, maybe it's a generic problem with strace - if it gets the child PID from the return value of "fork()", and attaches to it that way, that has two problems:

• it won't work reliably even under normal fork(), as it's timing- dependent, and by the time strace has attached to the child the child might have already done several system calls.
• it would fail completely with vfork(), as vfork() only returns after the child has execve'd

Who's maintaining strace these days?

That pretty much sucks, as it makes it pretty much inpossible to strace a normal fork() on SMP machines.

Sounds like the correct fix is not vfork()-related at all, but rather a flag to clone() to set "trace child process", so that the new child starts out stopped and traced. And some way for strace to get at the child pid.

The man page is inaccurate with respect to the fork tracing mechanism. What happens is that strace uses ptrace(PTRACE_POKETEXT...) to insert an infinite loop following the trap instruction (so it becomes `int $0x80; jmp .'). So no matter when the child is spawned, it will get stuck in this loop. When the fork syscall returns in the parents, strace inspects the return value to find the PID of the child. It then attaches to this process, patches the original code back in, and sends it merrily on its way. (It also patches the original code back in to the parent.) So fork tracing should work regardless of the number of processors.

For vfork this won't work, as the parent won't return until the child has done something (exec or exit), and the child is looping. As a hack to work around this, when strace intercepts a vfork, it uses the ptrace(PTRACE_POKEUSR, ORIG_EAX...) mechanism (this is the "functionality not yet in the standard kernel", as that mechanism was disallowed until recent 2.3.x, and required a patch to enable. I will update the man page to reflect this and to fix the `fork' error.) to change the system call back into a plain old fork. I wrote that hack, and decided that it was reasonable on the grounds that vfork used to just call fork at the libc level, and anyone who depended on getting special vfork semantics was asking for trouble anyway.

In short, fork tracing works, and vfork tracing works with a new kernel or a patch.

Its such a brilliant idea that we already had it...

        if (!(clone_flags & CLONE_PTRACE))
                new_flags &= ~(PF_PTRACED|PF_TRACESYS);

That was added for the threaded-gdb folks a while back 8)

Hmm.. A sufficient fix for that might be kernel/fork.c:

        if ((clone_flags & CLONE_VFORK) || !(clone_flags & CLONE_PARENT))
                p->p_pptr = p->p_opptr = current;

would instead become

        p->p_opptr = current;
        if (!(clone_flags & CLONE_PARENT))
                p->p_pptr = current;

ie the original parent would _always_ be the "real" parent - which is what CLONE_VFORK has to use anyway, but CLONE_PARENT would make the "logical parent" be the same as the cloner. This should make debugging happy (the debugger stays as the logical parent), yet should be ok for pthreads like behaviour too, ie the reason for CLONE_PARENT in the first place.

You've lost:

Computer: K6-200, 128 MB Ram, Symbios 810 scsi controller, Fujitsu Magneto-Optical drive, 620 MB [I have no empty scsi disc left :(], 620,000,000 bytes test file, fat filesystem, the same disk is used for NT and Linux.

command: "./pagein fill 150000 #" where fill is the filename, 150000 means 150000 pages are trashed, and # is the number of threads.

Linux:
#	pages/sec
1	13
4	14
64	14
256	? [computer unresponsive]

NT:
#	pages/sec
1	18
4	20
64	28
256	31
512	33

Linux is slower, and it cannot use multiple threads to reorder the sector reads; NT gets faster if I add further threads.

source code is at http://colorfullife.com/~manfreds/pagein/pagein.cpp

I looked at the thing quickly, and there _is_ actually a reasonably quick fix, and one that I would like to have for other reasons anyway: having read-write semaphores.

We don't have them, but they should not be fundamentally any worse to implement than the current semaphores are, and they have occasionally come up as useful things to have. They would certainly fit the bill for this particular problem very well (a page fault would get a read lock, while a mmap() would get a write lock - multiple page faults can happen in parallel).

The one race with multiple page faults that we have is handled nicely by the page table spinlock, which we introduced for kswapd anyway..

Well, the more I look at a read-write semaphore, the more I like it: it looks like something that once the semaphore implementation itself was done, the MM side would be absolutely trivial. It does introduce a new issue (multiple threads updating the page tables at the same time), but that one doesn't look that horrible..

We don't ever export the page table handling to the low-level filesystems any more (we used to a long time ago: the nopage() function got to touch the page tables itself rather than just return the right page), so fixing up the new issue is actually a very local fix in mm/mmeory.c.

Is anybody willing to take a stab at creating a read-write semaphore?

Nope, not acceptable. The mm semaphore is one of the most timing-critical in the whole kernel. It usually has absolutely zero contention, but it needs to be FAST. Basically, a read-lock() must look something very very similar to the read-spinlock implementation, ie something like

lock ; incl (%ecx)
js fixup

for the successful fall-through case. Two instructions, no more. That's what the spinlocks do, and that's also what the semaphores do (although in the case of a semaphore, it's a "decl" in that case.

The "fixup" case is going to be more complex than for spinlocks: for spinlocks it's just a simple loop, while for semaphores you get all the complexity that you see in arch/i386/kernel/semaphore.c to handle the thing cleanly..

The read-write semaphore should be doable with the same skeleton as the normal semaphores, although it needs two counters (regular semaphores have just "sleepers", rw-semaphores need to have "read_sleeper" and "write_sleeper" counts etc).

I would go for something very similar to the current semaphore implementation - one global spinlock for all rw-semaphores, and only if that actually becomes a real contention point do we try to be more clever (starting with moving it to a per-semaphore thing, and only as a last thing doing separate wait-queues with separate spinlocks).

I doubt you'll get much contention. The current semaphores get very little contention - the test-case that triggered this discussion in the first place is probably the worst one by far, and that test-case will have no contention at all with the read-write version because 99% of everything is just readers.

The holy grail is "Make it as simple as possible. And no simpler"

I'll see if I can get a free afternoon some day and try to port the current x86 semaphore code over to a rw version too. The plan was something like this:

• read_down():
  

  lock ; incl mem
  js contention_rw

• read_up():
  

  lock ; decl mem
  js wake_up_writer

• write_down():
  

  lock ; btsl $31,mem
  jc contention_ww
  testl $0x7fffffff,mem
  jne contention_wr

• write_up():
  

  lock ; andl $0x7fffffff,mem
  jne wake_up_reader_or_writer

where all the three contention cases grab a "contention spinlock" before they then start sorting things out. The only interesting part is making sure that the contention case gets the wakeups, and the above counts on:

• if a writer is waiting for readers (contention_wr), then the writer will have already set the high bit, and a reader will know to wake it up because the rw-semaphore value will be negative when it does read_up().
• if a reader is waiting for a writer, then the reader will have incremented the semaphore, and the writer will know to wake it up becasue the semaphore value won't be zero after the "write_up()".
• if a writer is waiting for another writer (contention_ww case), it will have to increment the "reader" part of the semaphore value, in order to get the other writer to wake it up on "write_up()".

All other races should be trivially handled by just having the spinlock, so the only really hard cases are the fast-path stuff where we cannot get the semaphore because it is too expensive.

Does anybody see any holes in the above pseudo-implementation? Please take a look at the way the current x86 semaphores are implemented: they use exactly the above kinds of single-atomic-instruction-plus-condition-codes trickery to get the non-contention case without _any_ extra instructions.

This looks extremely good. I'll have to read it through a few times (and then a few more times just for luck), but on the face of it it looks solid and really clever. Me likee.

And here I thought _I_ was being subtle.

I wonder if you might be convinced to use the same approach for the rw-spinlocks too? I used to like my rw-lock implementation, but hey, I can recognize genius when I see it, and this sure looks like it.

(Having the same logic for both the spinlock and the sleeping version will make for less confusion, and I feel better about having _one_ clever trick used twice rather than having two different clever ways to do essentially the same thing).

What will it buy? You still have to write all this code. You still can't (and shouldn't) do it on a filesystem that is mounted r/w. This code can be equally easy placed into userspace (where it _is_ now). It doesn't give you any win in fsck(8), since the current fsck _already_ doesn't care for filesystem layout - it just calls an appropriate fsck.<something>.

All the difference you'll get will be that e2fsck will be moved into fs/ext2, fsck.minix - into fs/minix, etc. I.e. complexity is the same, except that the code will (a) permanently in core and (b) any bug will bcome more dangerous, since you are in ring 0.

So the question being: What For? We already have generic fsck. It sits in /sbin/fsck. We already have generic mkfs (/sbin/mkfs). What's the reason to move large chucks of code that feels perfectly OK in userland into the kernel? Automagical fsck upon mount? It's a bug, not a feature. That decision belongs to admin. Even if you want it (always want it, that is) you can trivially do it in mount(8) or in the script that calls mount.

If NT really does what you describe... Well, small wonder that it's so bloated.

Actually, from what I was just reading about Windows 2000, Microsoft is adding a new innovative feature that they single handedly came up with that allows you to "splice" a filesystem onto a directory point.

I read that the reason was to do away with drive letters. So, hats off to Microsoft for inventing this new concept of "splicing" filesystems onto directories.

The ITT XTRA MS-DOS 2.11 Reference had a chapter on Xenix compatibility which explained this stuff in detail; I've never seen it in other DOS manuals.

CP/M, and therefore DOS 1.x, used / to signal parameters --- and the CLI knew about this, such that "FOO/X" would be parsed as a command "FOO" with an argument "/X/index.html". To maintain compatibility with DOS 1.x while providing an upgrade path to Xenix compatibility, the SWITCHAR could be set via a DOS call or CONFIG.SYS; if set to anything but "/index.html", "Xenix-like" command parsing was used and DOS commands could be invoked with switches preceded by the specified SWITCHAR; 3rd party DOS programs were supposed to use another DOS call to get the SWITCHAR and use it appropriately. ("Xenix-like command parsing" meaning that "FOO-X" was not treated as command "FOO" with switch "-X" if the SWITCHAR was "-".) There was also AVAILDEV which, if set to NO, disabled "bare" device references such as "CON", and you had to use "\DEV\CON" or "/DEV/CON/index.html" instead; all the standard DOS 2.11 programs used the \DEV prefix internally so they would work with AVAILDEV=NO, and again 3rd party programs were supposed to query AVAILDEV and behave appropriately.

That chapter also stated that DOS 3.x would default to a SWITCHAR of "-" and would add limited multitasking features, and that DOS would gradually be migrated to full Xenix compatibility, followed by its being fully replaced by Xenix (!). I sometimes wonder what the computing world would be like if Microsoft had actually done this....

The point of /proc is to avoid direct poking into the kernel memory. Moreover, in its current form it is highly non-portable. And that includes portability between 2.0/2.2/2.4. Please, _stop_ abusing it. Basically, _nobody_ promises that any given file in /proc will remain there in 2.4. Most likely 2.4 will keep compatibility symlinks, but even that is not guaranteed. If k3wl krapplications will break - too bad for them.

The thing is in flux right now. Some things will become sysctls, some will move into more reasonable places and some will simply die. The only documented parts are /proc/<pid>/*, /proc/self and /proc/sys. FWIC the rest is fair game.

Let me put it that way: there is an area where nobody has decent (let alone portable) interfaces. Doing it _right_ would be nice and if we will get a clean filesystem-based solution it will not take much to take nullfs and roll the patches to *BSD. If some *BSD folks will fill the skeleton - fine, if not - their business. But *BSD lacks the thing just as we do. libkvm sucks too.

Decent namespace will fix most of our problems - we already have fs-based solution, but it's rather messy right now. Internal interfaces are getting more or less straight (we still have a crapload of interesting races, but they will be the next stage; most of intimate knowledge of procfs guts is gone from the rest of the kernel and that makes procfs fixes possible). But the namespace _is_ a mess and implementations of procfs methods are, should we say it, sometimes unorthodoxal (check what drivers/nubus/proc.c does. Or ISDN stuff. Or wanrouter). Ideally I'ld see more or less common format and a tree organised along the lines of buses hierarchy (kernel being the replacement for nexus ;-). But namespace stuff will go when we'll have the worst interface problems fixed and will have a list of animals that want to be there at all.

we had _no_ namespace policy in /proc. For many years. How would you like little gems like /proc/ip2mem or /proc/h8? No, the former has nothing to IP. And it has nothing to do anywhere near the root.

We have more or less regular parts of /proc - per-process stuff and /proc/sys (aka kernfs for BSD folks). The rest falls into several classes. There are "subsystem information" animals. They don't fit into sysctl() interface (albeit some of them might go there) and they are usually read-only. Not a big deal, except that we have 1001 format and _really_ messy namespace. Finding relevant thing may be very nontrivial. meminfo, loadavg, etc. would happily go there. There are oddballs that allow write() - usually it's a sysctl wanting to happen. And there are real horrors - come on, ioctl() on /proc file... (yes, we have such beasts; check /proc/mtrr for one). And there is kcore/ksyms/kmsg group. So far cleanup had been kernel-only, but now we are almost at the point where changes will become visible for userland. For the most of those files nobody will really care, but for some of them we will need compatibility links and they will stay for a while. As for the rest... Just about anything will be better than current ad-hockery.

Nobody talks about dropping the fs-based interface to this stuff and reverting to /dev/kmem games. But there is a serious need to reshuffle this bag of #@$! into coherent state.

zones are separate physical memory (RAM) areas. eg. zones in a 6GB box look this way:

• zone 0: 0-16MB [ZONE_DMA]
• zone 1: 16MB-1GB (roughly) [ZONE_NORMAL]
• zone 2: 1GB-6GB [ZONE_HIGHMEM]

each zone is a 'pool of pages', with separate freelists and separate buddy bitmaps. The 2.2 allocator had everything in one big zone.

Higher order requests 'steal' DMA pages only as a last resort - previously GFP_DMA had to search for DMA-able pages by looking through all pages in a given page-list, plus normal GFP_ requests took DMA pages. So the zone allocator alone already gives much better GFP_DMA behavior, even on smaller boxes. In the future there will be GFP_DMA32 too.

The top-level structure is the 'zonelists' array, which contains a NULL-delimited list of 'target zones', in priority order. Eg. for GFP_HIGHMEM (which now covers the majority of allocations done in a Linux system) is { zone2, zone1, zone0, NULL }. For GFP_BUFFER it's { zone1, zone0, NULL }. The 'gfp_mask' parameter of the allocation functions is now an index into this 'zonelists' array, this gets resolved at compile-time in 99% of the cases.

Some other checks have been moved into the inlined part as well and get eliminated at compile-time. The page allocation entry points have been reduced to the minimum of 2 (formerly we had separate free_pages() and __free_page(), now it's all interfacing into __free_pages_ok()). The result is a more streamlined and lightweight page allocator. (despite the additional code it has). __free_page() is partly inlined now as well, the 'put_page_testzero()' thing is inlined, which is triggered in 60-70% of the cases.

the 'zonelists' array is generated runtime (once at boot), so systems which do not have highmem do not have to go through the empty zone every time.

I can understand the point you're making, but would have to point out that Linux is NOT the main culprit here - that 'honour' belongs to a certain MacroHard range of products.

To be blunt, where a system needs to dual boot between LoseSleep and *ANY* decent operating system, and LoseSleep is set to honour DST, then the RTC *MUST* be run in localtime rather than UTC/GMT as if it isn't, LoseSleep will trash it for you. If you want to get this fixed, persuade MacroHard to fix it as they're the only ones who can...

I'm just revamped most of those pages, but here's a quick analysis of the kernels and patches that currently appear to exist:

1. There are lots of holes prior to 0.99.14 where released kernels appear to no longer exist. This includes most of the 0.99.13 subseries, of which only 0.99.13 and 0.99.13k now appear to exist.
2. For most of the kernels prior to 0.99.13, patches were apparently never released, so one would have to use the individual kernel tarballs as the necessary sources.
3. Although the actual kernel tarballs for 1.2.10 and 1.3.0 both still exist, the original patch between them does not. However, it would not be difficult to recreate it.
4. With the sole exception pointed out above, all kernel tarballs and patch files from 0.99.14 to date still exist.

Based on the above, it should be possible to build up a repository showing the complete history of all kernels from 0.99.14 to date, but it would probably be meaningless to include any earlier kernels.

Note that the current set of links refer exclusively to kernels and patches available on ftp.kernel.org and do *NOT* include any of the various author series (such as the -ac kernels). I will be adding other prime release kernels to that list as and when I find them on the Internet, but the pages include details of kernels and patches in my collection that are not on ftp.kernel.org (identifyable by there being no link for the relevant "Version" or "Patch From" number respectively).

Ok, I finally made a public release of 1.3.0, and it's available in the normal places (ftp.cs.helsinki.fi and ftp.funet.fi).

Only full-source versions of the kernel exist right now: any patches are likely to be huge, as lots of things have been moved around.

NOTE! 1.3.0 is a development kernel, and if things don't work perfectly don't be surprised. Lots of patches that didn't go into 1.2.x because they were too risky are in 1.3.0.

That said, I naturally run 1.3.0 at my own machine, and it seems to be pretty stable. Knock wood.

Anyway, a _very_ rough list of "what's new" for 1.3.0 (just a general overview):

• new networking setup: the net directory layout is different, and there's a lot more there. Expect this to change even more in the future, but now we're just supposed to pound out the current "undocumented features".
• Lots of portability patches: most of my earlier axp-patches are now in the standard kernel. Not all of them, though: expect the axp patches to be more-or-less fully integrated by 1.3.2 or so.
• three new system call entry points for x86 (two of which were actually motivated by axp-portability cleanups):
• new select() entrypoint (5-arg version, arguments in %ebx, %ecx, %edx, %esi, %edi instead of the old broken memory block setup)
• getdents() system call entrypoint to read multiple directory entries. The old "readdir()" system call entrypoint can handle only one entry at a time (and has a braindead interface).
• flock() system call entrypoint (the old flock() library emulation never really got all the BSD semantics quite right)

[Digression mode on: I'd personally like to see the new libraries be 1.3.0-specific and just leave the old libraries as a "stable" version together with 1.2.x, but I don't know if that is really practical, and it's up to hjl anyway.]

The axp patches change a lot of things all over the place, notably by splitting up the PCI handling into architecture-independent parts and moving include-files areound a lot to better fit a multi-architecture setup. The axp patches have also resulted in various cleanups, and doing things "right" to be able to handle it cleanly on different setups. Thus the directory reading code is now much cleaner, and the mmap() system call will follow normal unix semantics more closely by actually honouring the "where" argument for non-fixed mappings and trying to find an address that is close to the requested area.

One downside: the UMSDOS filesystem is disabled in 1.3.0, as the directory cleanups broke that temporarily. Expect this to be fixed in 1.3.1 or soon afterwards. If you rely on umsdos, don't get 1.3.0.

Alpha-people: as mentioned, don't expect this to compile cleanly on alpha's yet. The ext2fs 32/64-bit problem isn't cleanly resolved yet, and some other unclean axp-patches haven't been integrated. Others haven't been tested in their new incarnation (the PCI stuff, for example: David M-T, how does the new setup look to you?). And the latest alpha-diffs by others haven't been integrated at all.

So, there it is: comments and patches are welcome. I have been slightly overworked with all the mails/patches/updates after DECUS, so I may well have ignored your particular favourite patch. If so, keep on re-sending it if you think it really is needed. But remember: not all patches need to go into the very earliest 1.3.x releases, and we have about 95 patchlevels to go on this thing yet..

Warning... This is NOT spam. It is a cybermined hostile web site! Do not go there. It is designed to infect and damage windows systems. We've analyzed some of the pages on that site. They are designed to bypass security checks (pages build URL's to hostile pages by assembling Java Script variables) and run hostile scripts. The silly ape has one script which attempts to format all of your hard drives (but it starts out attempting to format c: - well duh!).

It should have little effect on our Linux systems, but I still would not go there with any active content (Java or JavaScript) enabled!

Uhh no offence, but instead of complaining to the list why doesn't somone document all of this and complain to the uplink?

14 sl-smartt-1-0-0-T3.sprintlink.net (144.228.135.6) 289.606 ms 218.194 ms 219.797 ms

15 blackhat-T1.austtx.ixcis.net (216.140.128.10) 359.599 ms 228.010 ms 270.004 ms

16 sex.interactwithme.com (216.140.158.119) 229.419 ms 238.185 ms 219.788 ms

Don't rant, protect the week and destroy the evil :) (if worst come to worst that's only a t1)

Well, the guy wasn't very bright. He sent it to an address that didn't exist, and the list got the bounce message from his relay.

From the headers:

|------------------------- Failed addresses follow: ---------------------|
<gjmichel@earthlink.net> ... transport smtp: 550
<gjmichel@earthlink.net>... User unknown
|------------------------- Message text follows: ------------------------|
Received: from ---(really [198.142.239.33]) by mailarray.mpx.com.au
via smtpd with smtp
id <m11nl4z-000JWeC@mailarray.mpx.com.au>
for <unknown>; Wed, 17 Nov 1999 03:03:37 +1100
(/\##/\ Smail3.1.30.13.Y2K #30.35 built 1-mar-01)

The address the relay received it from was 198.142.239.33, which nslookup says is:

Name: dialup-mdc23933.mpx.com.au
Address: 198.142.239.33

It's one of their own customers. Call them and complain. There's probably some charge you could bring against them, like stealing computer resources or some such thing, but they're in Australia, so good luck.

I asked adaptec about a month anda half ago and got the usual blah, blah and Win + Novell praise and they were unable to answer anything about support under linux, solaris and BSD.

After that I asked this question on linux-network on Oct 13 and the answer from Donald Becker was yes. Check the linux-network archive for the full thread... (I am not subscribed to linux-net).

The driver for the new ones is not tulip (as per the old web page instructions on what google returns for you) it is the starfire:

ftp://cesdis.gsfc.nasa.gov/pub/linux/drivers/test/starfire.c