There was some linux-kernel activity in response to the Mindcraft study. Not as much as might be expected, considering the general outrage on the net at such an obviously rigged test. Now that linux-kernel's Mindcraft threads are all pretty much dead, it seems that their main upshot was that documentation on Linux tuning will probably be forthcoming. In other words, Microsoft helped us out by reporting a documentation bug. Thanks, Bill! ;-)

Under the Subject: NT reported faster than Linux 2.2.2 SMP, Tom Holroyd gave links to a NewsAlert article and the study itself. He pointed out that Mindcraft had tuned NT but not Linux. Alan Cox replied, they carefully went back to 2.2.2 which is the one with the known performance interactions with NT. I guess someone at MS did their research 8)

Robert G. 'Doc' Savage said, Mindcraft also used the v0.92 MegaRAID driver. An SMP race condition was fixed in v0.93 which was almost certainly available from the AMI web site long before the Mar 10-13 test.

Under the Subject: Linux vs. tNT ;-) Can sb explain this?, Carlos Costa Portela pointed to the Mindcraft study and a parody of it (as of KT press time, the page seems dead, but they give the location of a temporary copy) that he hadn't realized was a parody. Mitchell Blank Jr pointed that out to him, and pointed to the LWN page covering Mindcraft. Mitchell mistakenly called it the Netcraft study (a typo -- he corrected himself a few minutes later), Ingo Molnar also corrected him and, as an aside, added that the April Netcraft study shows that NT based server usage is starting to drop drastically after months of stagnation. [NT platform relative usage is down to 24.97%, thats a -0.66% drop from a month ago, pretty large considering the millions of hosts involved. He also pointed out that Microsoft's leaked Halloween II memo admitted that their own NT Performance Group found Linux _2.0_ to be on equal footing regarding Samba performance :) BTW, the Halloween documents live at http://www.opensource.org/halloween.html. If you haven't read them yet, go now.

Regarding the Netcraft study, Richard Gooch pointed out that MS market share has dipped and recovered before now. He pointed to the current Netcraft graphs and said, what I find interesting from the graphs is that it looks like the M$ installed base is growing because of the general growth in the industry. Their market share has been static for 6+ months and they've had no significant growth for over a year. They're just riding the wave. He added, If it was any other company, their market share would already have declined. The interesting question is when will they start to experience consistent declines, month after month?

Under the Subject: Linux Tuning, someone pointed out the ZDNet reaction to the Mindcraft study, which agreed that Linux tuning information is difficult to find. The poster agreed, and suggested that some kind of discussion on centralizing Linux tuning information would be a good idea. brent verner announced that he has started such a project at http://www.linux1.org/.

Tony Gale suggested that the document, Tuning Compaq Tru64 UNIX for Internet Servers, would be a good thing to imitate. Greg Lindahl added that a way to automate such tuning information would be a good thing, and volunteered to do it once the information was documented. Various folks offered to host these projects. Rik van Riel set up a CVS tree for the potential documentation and pointed to a manual for its use. There followed a bit of discussion about how the project should be organized and what sorts of things it should include.

Under the Subject: 2.2.5 optimizations for web benchmarks?, Someone wanted some tuning help and there was a bit of discussion. Then (a day later) Stephen C. Tweedie gave a link to The Linux Scalability Project, which is trying to tune the kernel for high server loads. He also reiterated one of the points in the prior discussion: that apache itself is where most of the benefits of tuning will be gained.

Apache developer Dean Gaudet replied, Uh I dunno. Unless by tuning you mean "replace apache with something that's actually fast" ;) and added Really, with the current multiprocess apache I've never really been able to see more than a handful of percentage improvement from all the tweaks. It really is a case of needing a different server architecture to reach the loads folks want to see in benchmarks.

He went on to give an interesting recent history of apache:

Stephen asked if there were any apache tuning info, and Dean pointed to http://www.apache.org/docs/misc/perf-tuning.html. Greg Lindahl replied that the document didn't give any tips on how to tune Linux, and Dean replied, "Tuning linux" is even more a black art and I wasn't about to write up everything. Plus it changes every couple kernel versions and libc versions anyhow. It's a nightmare to keep up to date any documentation surrounding linux internals. He went on, I'm saying this from the point of view of the person who answers the apache bugdb mail regarding linux problems. 90% of what I end up having to say to people is "well gee it works fine on all machines I have access to, maybe it's your distribution/kernel version/libc version/phase of the moon/colour of your hair/..." there are just too many variables that make linux inconsistent. Take it as a light flame. Greg gently reminded him that the point of the discussion was to get something written up, and Dean's input would be appreciated.

Jan Kara found and fixed a lot of bugs in the quota code. Chris Evans asked if the patches would make it into 2.2.7; Jan said he planned to submit them if there were no problems. He then posted a new patch, and Andre Couture replied with total success. Jan posted a new patch, which emptied his bug list and left only new features in his TODO file. Mike Galbraith pounded on it with sadistic fury, but his single processor machine stubbornly refused to crash. Jan said he'd wait a few more days for more bug reports before submitting the code to Linus Torvalds; and the thread was over.

Senko Rasic wrote a patch (against 2.2.x) to emulate a filesystem on audio CDs, showing each track as a separate file. He said it was slow and imperfect, but had no known bugs.

There followed some discussion of rounding out the features: Riley Williams asked about dual mode CDs: could the driver be made to handle both the data and audio tracks? He suggested that the 'mount' command could either show the audio tracks as existing in a separate directory, or be used once to mount the data tracks, and a second time to mount the audio.

Senko replied that his filesystem could currently not access data tracks, only display them. He dismissed the idea of using 'mount' twice, pointing out that VFS locks the mounted device. Bypassing VFS, he added, would be extremely ugly. As far as having mount show a separate directory for audio files, Senko wasn't too hopeful about that either: if it were even possible, he felt it would require a complete redesign of the isofs code.

Riley suggested just making isofs detect (rather than actually handle) the audio tracks; and then do a hidden loopback, using Senko's audiofs filesystem to access the tracks as files. Senko replied that the isofs implementation couldn't handle mounting audiofs on top of it via loopback. He did say it might be possible to mount isofs on top of audiofs.

There was a bit more discussion about naming conflicts and multi-session audio CDs, and the thread was over.

Pavel Machek wondered how CODA would handle files that were too big to fit on a single hard drive. His perusal of the code showed no implementation at all.

Matti Aarnio asked if this was only an implementation problem, or a fundamental design issue; and someone replied that it was a design issue: clients can connect and disconnect at will, so files are stored whole.

EOT.

In the course of discussion, someone asked if ACPI (Advanced Configuration and Power Interface) would be implemented for Linux. Alan Cox replied, I don't know. It is an open (but very hard to read spec).

In fact, there is an ACPI4Linux project in the works; but according to the web page, they could really use a hand.

Frank Bernard had a problem that turned out to be his fault, as he figured out eventually and explained to the list. In the course of the discussion, Dave Cinege gave a pointer to his initrd-archive patch, which allows ramdisk size to change dynamically. Richard Gooch was impressed, and asked if Dave had submitted the patch to Linus Torvalds. Dave replied, A few times since Jan 1998. Never heard back. I finally got a 'no' from Alan Cox for inclusion in 2.0.37 and 2.2.2 not long ago. (Code OK, but not interesting enough. : P)

Harvey J. Stein did a fairly repeatable test of several kernels under heavy load, and found that the 2.2.5arca12 performed worse than vanilla 2.2.5 (2.0.36 also performed poorly). Andrea Arcangeli was thrilled to have a test case, and said, I just know _exactly_ why arca12 has a worse worst-case and bad interactive feel while larging tons of data to disk. The culprit is buffer.c and the reason 2.2.5 is not stalling is due the set_writetime misfeature that on the other side is harming a lot performances in the stock kernel. But he replied to himself a couple hours later, saying in the last minutes I've found also some stupid design in my shrink_mmap() of "all" 2.2.5_arca*.bz2 that may lead to bad performances and maybe stalls. Unfortunately I was using the bad design of shrink_mmap() as a feature :(. So with the design fixed, the system runs less smoothly under swap, but if you are not swapping heavily this new patch should perform better than arca12. He posted a 2.2.6_arca1 patch.

At this point another thread started, with the Subject: [big performances boost for DataBases] Re: cache killer memory death test - 2.0 vs 2.2 vs arca - programs inside. Andrea summed up his own tests, and said, In your program you are killing the caching behaviour that my buffer code does on dirty buffers (the thing that doesn't work on the stock kernel due flushtime mess). The fact is that you are doing many gdbm_sync in the middle of your proggy. He posted a 2.2.6_andrea2 patch (sic).

While downloading 2.2.6_andrea2, Harvey replied with results for arca1 (it was faster than 2.2.6, but not by much), discussed his results and asked about the change from "arca" to "andrea". Andrea replied with much detail on the technical issues of the arca1 results, and explained the name change, saying, Because you all call me Andrea ;). All my friends instead call me `arca' since I was at mid school (from the surname obviously ;). And I also seen that there's just some software company called arca and so I preferred to call it andrea to avoid any kind of mess. But the name won't make differences at all, the patch-set is the same.

Harvey tried out 2.2.6_andrea2, and said, Smokin! 2.2.6andrea2 was so incredibly fast that I didn't really have much time to test it interactively, but it was great. Andrea replied, I am very happy to hear that ;). Now it would be also interesting to see a DBMS benchmark.

Stephen C. Tweedie came in, with, Do you have an idea of what exactly makes the difference? It looks as if the buffer write scheduling is making a large difference, and Andrea replied, It depends what you are benchmarking. If you are benchmarking a piece of code that rewrite in the same place many times, yes, definitely my redesign of the flushtime handling is the the big difference (it will completly avoids tons of not needed I/O). But all my other VM ideas/code are very connected with performances.

(Andrea also added, BTW, I have news about RB-trees. I did some benchmark and they are _definitely_ slower in the buffer cache for query (now find_buffer is far from the top of the profiling output). Now I am using the hash function of the stock kernel, but I'll move to the mul method shortly.)

In a different reply to Harvey's test of 2.2.6_andrea2, Andrea found some other problems, and I also discovered an interesting bug in my page-mapping handling in 2.2.6_andrea2 (I simply forgot to increase the map_count of the page at fork time... ;). The bug can't lead to corruption/crash or any kind of other harm because I designed the map_count handling in a safe way, but the bug was obviously harming performances :(. So if 2.2.6_andrea2 was fast 2.2.6_andrea3 can be an order of magnitude faster in shrink_mmap. He included a 2.2.6_andrea3 patch. Much later, he replied to himself, saying 2.2.6_andrea3 had a bug. He posted a small patch to fix it, and said he would release 2.2.6_andrea4 shortly.

Now this thread died, and two days later, the conversation continued on the original thread. Harvey had tried 2.2.6_andrea4, and found it to be slightly worse than 2.2.5arca12; but he still found 2.2.6_andrea2 to beat them all. He concluded, It seems like a2 (andrea2) is the best at both avoiding disk activity & wisely using the disk when necessary. A4 seems to be slightly more efficient when using memory but much worse at using the disk when necessary.

Andrea replied, With changes between A2 and A4 I did penalized the mmapped memory in favour of caching memory. This allowed the dbase program to alloc the buffer he needed, but this mean that it swapped out far more data to disk, and added, I'll revert to the andrea2 behaviour (thanks for the information).

He went on, I also released a 2.2.7_andrea1.bz2 but don't try it!!! because since I merged 2.2.7 my VM tree is unstable under high swap load. I don't think it's a 2.2.7 issue (since I overviewed all the patch and I couldn't find anything wrong in it) but maybe a missed bit in the merging or some more subtle bug. I hope to have a clue shortly. About 7 hours later he replied to himself, with, I fixed the bug. I forgot one of my lru_refile_unmapped_cache() in unuse_pte ;). So when delete_from_swap_cache() was running, it was doing a list_del(page->lru) it had page->lru pointing to the old ->lru not-valid-anymore values... He added, Now everything seems rock solid also after a swapoff -a, and posted a new 2.2.7_andrea2 patch.

Andre Couture said, It seem that using "make xconfig" in 2.2.6 does not allow to select/unselect the Mach64 Frame Buffer. The line is not shaded but also not selectable. Andrzej Krzysztofowicz replied, Xconfig is written basing on silent assumption that each variable should be defined in only one place. This assumption is broken in the drivers/video/Config.in file, however...For i386 architecture you can try my xconfig patch: ftp://rudy.mif.pg.gda.pl/pub/People/ankry/linux-patches/2.2/xconfig/patch-xconfig-990407.gz. It fixes this problem partially ignoring most of options which cannot be ever sellected for current architecture. However, for ppc/m68k/sparc/sparc64 the problem is more serious and cannot be removed in this way.

Illuminatus Primus said, After upgrading from 2.2.0 to 2.2.6, it seems that packets that arrive on an interface using the same address as another previously-existing interface get discarded. Christoph Lameter said to do an "echo 0 >/proc/net/ipv4/xxx/rp-filter" to switch the discarding off. Steven N. Hirsch couldn't find the problem in 2.2.6ac1, and asked if the default had changed in the vanilla kernels.

Illuminatus posted again, having tracked the culprit to a snippet in /etc/init.d/netbase in Debian Slink. Steven hadn't seen the problem in ac1 because he used Red Hat, which left rp_filter alone.

Jeff Garzik was exploring the docs, and found that in Documentation/IO-mapping.txt, it said that reading from IO space used readl(), while writing used ioremap(). He wanted to know why writing required ioremap(). Philip Blundell replied, 0xc0000 is in the ISA memory space. This area is special in that (on PC machines at least) you can access it without ioremap, and in fact ioremap will just return the original address. It's still good practice to use ioremap for this sort of memory though.

Alan Cox added, Using ioremap for all cases is almost certainly going to be required in 2.3.x. The fact its not always needed right now is one of the biggest obstacles to supporting a more sane range of memory sizes.

Ravi Wijayaratne asked if and how he could use a double or float in a kernel module. Richard B. Johnson said:

Alan Cox posted the latest 2.0 patch, and Patrick J. LoPresti said he was getting memory problems on both of his SMP machines after upgrading. He confirmed that the problem did not exist with 2.0.37pre10. Alan traced the problem to a patch from Stephen C. Tweedie that aimed at fixing an mm race condition, and said he'd back it out for the next release.

Ulrich Windl said, Viewing the USB patches, I get the impression that shell scripts like linux/drivers/usb/stopusb should go to the scripts (sub)directory... (Or include them in the documentation)

Linus Torvalds replied, Well, they should probably be removed, they're really only there to help early development..

EOT.

Oren Laadan wanted some comments on Out Of Band (OOB) data:

Andi Kleen replied, TCP has no real out-of-band data. It has urgent data. If you see it as urgent data the Linux behaviour makes sense. I don't think it is worth the effort to change it to be 100% BSD compatible. The bug is really that the sockets API maps urgent data to something called out-of-band data, and added, I'll document the behaviour in my network man pages.

Elsewhere, Craig Milo Rogers offered the following historical digression:

Alexander Viro posted a patch to fix a number of problems. David S. Miller replied, the architectures where you made the most changes don't even compile in 2.2.7 and haven't been updated in nearly half of a year :-) and Alan Cox added, and some of the bugs appear to be in the relevant CVS trees too.

Camm Maguire had some problems with the Sangoma wanpipe S508 cards on the routers he was using: the line would freeze after a day or so. Alan Cox replied, This is why the code is still in -ac and I've not sent it to Linus. You aren't the only people with this problem and the new code. Once I get updates from Sangoma that survive testing then it'll go on to Linus.

Andre Malafaya asked if and when the Alladin M1543 IDE chipset would be supported by Linux, and Andre Hedrick said: