While you're adding this feature, and it seems others are adding similar things, it is *highly* desirable to allow the build to put all the dumps in one place of desired (my first thought is /var/core) so that if you get a lot you won't run the system out of disk.

The directory name could be set in /proc/sys/coredir (or somesuch) with an initial value of "." of course.

Other than that I like the idea, although process "name" could get a lot of clashes on threads, and pid gets reused. There may be a better idea, but most of mine are cumbersome. This would really simplify certain kinds of dump analysis.

As requested, here are a number of tests of the latest VM patches. Tests are described in a previous post, archived here:

http://www.uwsg.indiana.edu/hypermail/linux/kernel/0109.3/0033.html

Results:

2.4.10 performance is great compared to 2.4.[7-9], but these tests still seem to point out some room for improvement in the 2.4.10 VM tree. 2.4.10 and 2.4.10(+00_vm-tweaks-1) performed similarly. The vm-tweaks patch improved the swap smoothness, but the number of pages swapped out didn't change measurably, nor did the large number of swap-ins. Clogging the system with dirty pages via 'dd' still causes XMMS to skip badly.

Let's push the aging/list-order code more by driving the system a bit harder in step d), namely adding mozilla to the common user application test. We will also stream mp3 audio throughout the entire test.

2.4.10(+00_vm-tweaks-1)
48 sec StarOffice load time
28 sec 2560x2560 GIMP image rotation
82400 KB swapped out, 92148 KB swapped back in

2.4.9-ac14 + aging
33 sec StarOffice load time
25 sec GIMP image rotation
30072 KB swapped out, 22252 KB swapped back in

2.4.9-ac15 + aging + launder
33 sec StarOffice load time
24 sec GIMP image rotation
57556 KB swapped out, 25900 KB swapped back in

'vmstat 1' sessions for these three cases are available at: http://loke.as.arizona.edu/~ckulesa/kernel/

2.4.10+ is clearly working a LOT harder to keep dentry and inode caches in memory, and is swapping out harder to compensate. The ac14/ac15 tree frees those caches more freely, and don't page application working sets out so readily.

Let's test this statement by not pre-filling the inode and dentry caches with 'slocate' and performing the same test:

2.4.10(+00_vm-tweaks)
26 sec StarOffice load time
24 sec GIMP image rotation
48332 KB swapped out, 33521 KB swapped back in

2.4.9-ac14 + aging
32 sec StarOffice load time
26 sec GIMP image rotation
37392 KB swapped out, 11952 KB swapped back in

2.4.9-ac15 + aging + launder
32 sec StarOffice load time
22 second GIMP image rotation
23884 KB swapped out, 10828 KB swapped back in

2.4.10 does much better this time; in particular the StarOffice loading that was so plagued by swapouts, pressured by dentry/inode caching last time, went smoothly. But there's still more paging than with 2.4.9-ac1[4-5].

Let's try one more aging/list-order experiment. Instead of creating a 2560x2560 GIMP image first, then loading StarOffice and many other applications after (to start swapping, and cause GIMP pages to be candidates for reaping) -- this time let's load StarOffice first and then create the GIMP image. This should keep the GIMP image at a 'younger' age and presumably shouldn't page back into memory (rotation should be faster). StarOffice may swap itself entirely out however.

2.4.10(+00_vm-tweaks)
25 sec StarOffice load time
29 sec GIMP image rotation
64427 KB swapped out, 77422 KB swapped back in

2.4.9-ac14 + aging
30 sec StarOffice load time
24 sec GIMP image rotation
22147 KB swapped out, 8922 swapped back in

2.4.9-ac15 + aging + launder
31 sec StarOffice load time
21 second GIMP image rotation
17204 KB swapped out, 8224 swapped back in

The 2.4.10 behavior surprised me. The GIMP pages are younger in memory, yet the rotation was slowed by swapin & swapout activity -- slower than before. Plus more StarOffice pages were swapped out, so it had to be paged back in order to close the application. I'm puzzled. The ac14/ac15 behavior was closer to what I expected; the GIMP pages were young and unswapped, only the earliest StarOffice pages had to be recalled.

These are samples of rather 'ordinary' loads which 2.4.10 needs some work handling; the ac15 tree is doing a better job with this particular set right now (ac15 tree also doesn't skip XMMS with the creation of lots of dirty pages via 'dd'). But all three kernels tested kept the user interface relatively responsive, which is an improvement over previous 2.4 releases. Very cool.

A note on page_launder(). ac14 has the smoothest swapping, with small chunks laundered at a time. ac14+aging and ac15+aging+launder both swap out huge (10-20 MB) chunks at a time. Admittedly, the user interface is responsive and XMMS doesn't skip a beat, but most of the 60 MB of actual swapout in the first test in ac15+stuff came from only THREE lines of 'vmstat 1' output. Otherwise there was no swapout activity.

Given the recent flurry of changes in the Linux kernel VM subsystems I decided to do a bit of benchmarking. The benchmark is a test of file server performance. I originally did this test about a year ago with fairly dismal results, so I thought I'd see how much things had improved.

The good news, things have improved. The bad news, they're still not good.

The test consists a linux server acting a file server (using netatalk) and 5 macintosh clients.The clients each write a 30 Meg file and read it back. Each client repeats this 10 times.Total amount of IO in the test 1.5 Gigs written, 1.5 Gigs read.

The tests were done with the following kernels

2.4.10: stock 2.4.10 kernel
2.4.10-aa1: 2.4.10 with Andreas patch aa1 including his vm-tweaks-1
2.4.10-p: 2.4.10 with Robert Loves preempt patch
2.4.9-ac15: Alans latest
2.4.9-ac15-al: 2.4.9-ac15 with Riks Aging+Launder patch
2.4.9-ac15 didnt fare too well, but Riks patch resolved these problems so I will leave 2.4.9-ac15 out of the discussion.

The hardware was a UP P-II 266 with 256 Megs of memory using SCSI disks on a Adaptec wide controller. The clients and server were all connected to a 100 Mbit switch. The hardware is nothing special, but disks and LAN are all capable of pushing 10 MB/s of bandwidth.

In the test, the clients are each accessing 30 Meg files. With 5 clients, thats a file working set of 150 Megs of disk space being accessed. With 256 Megs of memory, all the files can fit in memory. I don't consider this to be a realistic test of file server behaviour since if all your files on a file server can fit in memory you bought too much memory :-).

So for all the tests, the file server memory was limited to 128 Megs via LILO except for a baseline test with 256 Megs.

The features of a file server that I consider important are obviously file serving througput. But also fairness in that all clients should get an equal share of the bandwidth. So for the tests, I report the time that the last client finishes the run which indicates total throughput, and the time the first client finishes which ideally should be not too much before the last client.

Summary of the results
======================

In the baseline test with 256 Megs of memory, all the kernels performed flawlessly. Close to 10 MB/s of thoughput was achieved evenly spread between the clients.

In the real test with 128 Megs of memory, things didnt go as well. All the kernels performed similarly but none were satisfactory. The problem I saw was that all the clients would start out getting fairly bad throughput of only a few MB/sec total amongst all the machines. This is accompanied by heavy seeking of the disk (based on the sound). Then one of the clients would "get in the groove". The good client gets the full 10 MB/s of bandwidth and the rest are completely starved. The good client zooms through to the finish with the rest of the clients only just started. Once the good client finished, the disks seek madly for a while with poor throughput until another client "gets in the groove". Once you are down to 2 or 3 clients left, things settle down because the files all fit in memory again.

Overall, the total throughput is not that bad, but the fact that it achieves this by starving clients to let one client at a time proceed is completely unacceptable for a file server.

Note: this is not an accurate benchmark in that the run times are not highly repeatable. This means it can't be used for fine tuning kernels. But at the moment, I am not concerned with fine tuning but a huge gaping hole in linux file serving performance. And its probably true that the non repeatability indicates a problem in itself. With a well tuned kernel, results should be much more repeatable.

Detailed result
===============

Here are the timing runs for each kernel. Times are Minutes:seconds. I did two runs for each. Vmstat 5 outputs are available at http://tltsu.anu.edu.au/~robert/linux_logs/ But none of the vmstat output shows any obvious problems. None of the kernels used much swap. And I didnt see any problems with daemons like kswapd chewing time.

Baseline run with 256 Megs

Run 1     First finished 4:05       Last finished: 4:18

Notes: this indicates best case performance

linux-2.4.10:
Run 1     First finished 2:15       Last finished: 5:36
Run 2     First finished 1:41       Last finished: 6:36

Linux-2.4.10-aa1
Run 1     First finished 3:38       Last finished: 8:40
Run 2     First finished 1:35       Last finished: 7:07

Notes: slightly worse than straight 2.4.10

Linux-2.4.10-p
Run 1     First finished 1:39       Last finished: 8:33
Run 2     First finished 1:46       Last finished: 6:10

Notes: no better than 2.4.10, of course the preempt kernel is not advertised as a server OS but since the problems observed are primarily fairness problems, I hoped it might help.

Linux-2.4.9-ac15-al
Run 1     First finished 2:00       Last finished: 5:30
Run 2     First finished 1:45       Last finished: 5:07

Notes: this has slightly better behaviour than 2.4.10 in that 2 clients tend to "get in the groove" at a time and finish early and then another 2 etc.

Analysis
========

In the baseline test with 256 Megs, since all the files fit in page cache, there is no reading at all. Only writing. The VM seems to handle this flawlessly.

In the 128 Meg tests, reads start happening as well as writes since things get flushed out of the page cache. The VM doesnt cope with this as well. The symptom of heavy seeking with poor throughput that is seen in this test I associate with poor elevator performance. If the elevator doesnt group requests enough you get disk behaviour like "small read, seek, small read, seek" instead of grouping things into large reads or multiple reads between seeks.

The problem where one client gets all the bandwidth has to be some kind of livelock. Normally I might suspect that the locked out process have been swapped out, but in this case no swap is being used. I suspose their process pages could have been flushed to make space for page cache pages. But this would show up in an incease page cache size in vmstat. Which doesnt seem to be the case.

Ironically I believe this is associated with the elevator sorting requests too aggressively. All the file data for the processes that are locked out must be flushed out of page cache, and the locked process can't get enough reads scheduled to make any progress. Disk operations are coming in for the "good" process fast enough to keep the disk busy, these are sorted to the top by the elevator since they are near the current head position. And noone else gets to make any progress.

It has been suggested that the problems might be specific to netatalk. However I have been unable to find anything that would indicates that netatalk is doing anything odd. Stracing the file server processes shows that they are just doing 8k reads and writes. The files are not opened O_SYNC and the file server process arent doing any fsync calls. This is supported by the fact that the performance is fine with 256 Megs of memory.

I have been unable to find any non networked test that demonstates the same problems. Tests such as 5 simultaneous bonnie runs or a tiotest with 5 threads that are superficially doing the same things don't see the same problems.

What I believe is the cause is that since we have 5 clients fighting for network bandwidth, the packets from each client are coming in interleaved. So the granularity of operations that the server does is very fine. In a local test such as 5 bonnies, each process gets to have a full time slice accessing its file before the next file is accessed. Which leads to a much greater granularity. So I supposed a modified version of tiotest that does a sched_yeild after each read or write might see the same problems. But I havent tested this theory.

Linux 0.01 has a bug in disk request sorting - when interrupt happens while sorting is active, the interrupt routine won't clear do_hd - thus the disk will stay locked up forever.

Function add_request also lacks memory barriers - the compiler could reorder writes to variable sorting and writes to request queue - producing race conditions. Because gcc 1.40 does not have __asm__("":::"memory"), I had to use dummy function call as a memory barrier.

Well, IIRC, Alan and DaveM were essentially 1.2.x maintainers with various -ac and "ISS" patches (bonus points if you can remember what ISS stood for). I've probably got some of those 1.2.13 patches around somewhere...

As for 1.0.9, at one point some years ago I had updated it (cf. linux-lite) to compile with "new" gcc-2.7.2 when RAM was major $$$ - I don't imagine it has been touched since.

$ ls -l date

-rwxr-xr-x   1 root     root        13624 Sep  4  1992 date

        /lib/libc.so.4 (4.0)

Wheee! :) Now if I just had a Decwriter for a serial console...

You might want to read the mailing list entries from 1991 and early 1992:

http://www.kclug.org/old_archives/linux-activists/

I've put together a summary of some of the more interesting early posts from 1991 and early 1992 for the computer history book I'm writing...

http://penguicon.sourceforge.net/comphist/1991.html

http://penguicon.sourceforge.net/comphist/1992.html

The late 1993->95 list is archived at

http://www.linux.org.uk/Old-LK/Old-linux-kernel

I don't currently know if the rest of 1992->late 1993 exists

The kclug archives claim to run until the second week of october 1993. I just haven't sorted through them that far yet.

http://www.kclug.org/old_archives/linux-activists/

Please don't beat on them TOO hard, I think they're on an ISDN line or something...

(The reason for limiting the search to mid 1992 is that by they they were already up to 0.95, so 0.01 was no longer relevant...)

The biggest problem is to prevent the TCP server hogging all the threads when a client gets congested.

With the UDP code, we use non-blocking I/O and simply drop all replies that don't get through. For TCP dropping replies is not acceptable as the client will only resend requests every ~60seconds. Currently, the code therefore uses blocking I/O something which means that if the socket blocks, you run out of free nfsd threads...

There are 2 possible strategies:

1. Allocate 1 thread per TCP connection
2. Use non-blocking I/O, but allow TCP connections to defer sending the reply until the socket is available (and allow the thread to service other requests while the socket is busy).

I started work on (2) last autumn, but I haven't had time to get much done since then. It's on my list of priorities for 2.5.x though, so if nobody else wants to get their hands dirty I will get back to it...

I consider it extremely embarrassing that fundamental drivers like aic7xxx, emu10k1, tulip, etc. are breaking regularly in the mainline kernels. I haven't had any trouble with things like this in Windows for several years now... Sure the Windows drivers are probably a few percent slower, but as Nathan Myers once wrote, "It is meaningless to compare the efficiency of a running system against one which might have done some operations faster if it had not crashed."

I think we all owe major thanks to Alan Cox, who does his best to keep the house in order amidst the chaos of kernel development (kudos to Mr. Cox for holding on to Rik's VM design long enough for it to stabilize!)

it is illustrated by the following very simple program:

#include <unistd.h> int main() {

char buf[512]; while (read(0, buf, sizeof(buf)) == sizeof(buf))  
return 0;

}

The program should be reading a block device, but a big file probably does the trick as well.

./a.out < /dev/hde1

When the program is running, all cached pages pop up in the active list, and when the memory is full of active pages, the computer starts to page out stuff, becomes VERY unresponsive, and after half a minute or so it goes OOM and starts killing processes. There are lots and lots of free swap at this time. I also get a bunch of 0-order allocation failures in the log.