Kernel Traffic #119 For 21 May 2001

This should put the accusations to rest that Linux got the outstandingly high SPECweb99 scores only because the webserver was in kernel-space. It's the 2.4 kernel's high performance that enabled those results, having the web-server in kernel-space didnt have much effect. TUX was and remains a testbed to test high-performance webserving (and FTP serving), without the API-exporting overhead of userspace.

[i suspect the small performance advantage of X15 is due to subtle differences in the SPECweb99 user-space module: eg. while the TUX code was written, tested and ready to use mmap()-enabled TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a mail how to enable it, perhaps he can do some tests to confirm this suspicion?]

doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time is spent in generic parts of the kernel, 12% of time is spent in the user-space SPECweb99 module, and only 2% of time is spent in TUX-specific kernel code.

doing the same test with the original TUX 1.0 code shows that more than 50% of CPU time was spent in TUX-specific code.

what does this mean? In the roughly 6 months since TUX 1.0 was released, we moved much of the TUX 1.0 -only improvements into the generic kernel (most of which was made available to user-space as well), and TUX itself became smaller and smaller (and used more and more generic parts of the kernel). So in effect X15 is executing 50% TUX code :-)

(there are still a number of performance improvement patches pending that are not integrated yet: the pagecache extreme-scalability patch and the smptimers patch. These patches speed both X15 and TUX up.)

(there is one thing though that can never be 'exported to user-space': to isolate possibly untrusted binary application code from the server itself, without performance degradation. So we always have to be mentally open to the validity of kernel-space services.)

Linux 2.4 is surely one of the most advanced OSs ever happened, especially from the optimization point of view and for the admirable economy of concepts on which it lies. I definitively hope that X15 helps reinforcing the success to this amazing system.

TUX has definitively been my performance yardstick for the development of X15, but I had many sources of inspiration for the X15 architecture. Maybe the most relevant are the Flash Web Server (Pai, Druschel, Zwaenepoel), several Linus observations on this list about (web) server architecture and kernnel services, and the reading of the Hennessy & Patterson architecture books. Last but not least, aside from some heated discussions, research in microkernel architecture has taught us many lessons on how to achieve an efficient model of interaction across separate addressing spaces.

If i have to make some sort of educated guess and point at where the current bottleneck lies for web server performance, I would say that it is somewhere between the memory subsystem and the PCI bus.

With zero-copy sendfile data movement is not an issue anymore, asynchronous network IO allows for really inexpensive thread scheduling, and system call invocation adds a very negligible overhead in Linux. What we are left with now is purely wait cycles, the CPUs and the NICs are contending for memory and bus bandwidth. It would be really interesting to see where the network shifts now that faster machines are becoming available.

On my whish list for future kernel developments I would definitively put disk asynchronous IO and a more decent file descriptor passing implementation. I'll detail this in subsequent messages.

I'll surely check out the impact of Ingo's patches on TUX performance sometime this week.

I'd also like to reiterate my request for help for testing X15 on higher end server architectures.

X15 is still very young alpha code and I can surely improve its performance in many ways.

Our intention is to release X15 with an open source license.

This will happen as soon as the codebase stabilizes a bit, that is when we go beta (in two - three weeks).

At the moment we just don't have the time...

The reason why I released the alpha binary version is that several people would not believe that a user-space server with this level of performance would be possible at all and several statements that I made on this list were challenged.

Besides I really appreciate the feedback that I received so far from Ingo and others, and I'd be very curious to know if anybody did any performance evaluation at all.

i noticed another RFC anomaly in X15. It ignores the "Connection: close" request header passed by a HTTP/1.1 client. This behavior is against RFC 2616, a server must not override the client's choice of non-persistent connection. (there might be HTTP/1.1 clients that do not support persistent connections and signal this via "Connection: close".)

the rule is this: a request is either keepalive or non-keepalive. HTTP/1.0 requests default to non-keepalive. HTTP/1.1 requests default to keepalive. The default can be overriden via the "Connection: Keep-Alive" or "Connection: close" header fields.

if you fix this, does it impact SPECweb99 performance in any way?

yet another anomaly i noticed. X15 does not appear to handle pipelined HTTP/1.1 requests properly, it ignores the second request if two requests arrive in the same packet.

SPECweb99 does not send pipelined requests, but a number of RL web clients do. (Mozilla, apt-get, etc.)

I have uploaded a new release of X15 that hopefully solves all the RFC bugs. I say hopefully because I haven't had the opportunity to fully test the request pipelining. Is there anything to automatize such tests?

From what I could measure X15 is still a good 5% faster than TUX.

You can find the file at: http://www.chromium.com/X15-Alpha-4.tgz

BTW: Next release (in a week or so) will be a beta and it will include source code!

I'm wary of using an Ali chipset. They are even less common than the VIA so just havent had the exposure to root out problems.

Also the main feature I'm looking for is a machine with 768 Meg or 1G ram at a reasonable price. Hence I want to use 256 Meg dimms. I can't use an i815 chipset as this tops out at 512 Meg. The apollo pro board is one of the few that has 4 dimm slots allowing 1 Gig of memory. The athlons boards only have 3 dimm slots so top out at 768 Meg.

I'm wary of using DDR dram. The chipsets havent been round long enough to have much of a track record. And the ram is too expensive. Also the A7M266 is using a VIA 686b southbridge anyway which I thought was the source of the problems. Anyway these boards only tend to have 2 DDR dimm slots and the biggest DDR dimm that crucial sells is 256 Meg. So I would be limited to 512 Meg.

Maybe I have to bite the bullet and go with 512 Meg dimms. They only appear to be available in registered with ECC which makes them cost about twice as much per meg and which I wasnt sure that all boards support. What motherboards/chipsets to people recommend for machines with 1Gig+ ram.

• Is it simply that it must compile separately from the kernel and load via insmod?
• May I #include kernel header files in my module?
• If so, which ones?
• What if these header files have inline functions? (Does it matter?)

If I don't hear from Linus directly, can someone point me to a document, file, or mailing list thread where this might be spelled out.

If you want to do binary only then it depends solely how your lawyers intend to interpret the concept of 'linking'. Linus comments on the matter have no impact since the kernel isnt all his copyright and he has linked in code by bodies who are most definitely opposed to binary modules.

The same applies for source code under 'additional restrictions' as the GPL calls things disallowing stuff it allows.

If you are releasing modules with source under terms that are at least as free as the GPL (eg BSD without advertising clause) then nobody has any cares. We probably wouldnt merge it with the mainstream kernel due to the lack of patent trap protection in the BSD license but I suspect you dont want that anyway.

It seems that in 2.4.4 suddenly the function "skb_cow" no longer returns the modified skb, but it retuns and integer for succes/failure.

This means that for networking modules requiring this function, there is no source code compatibilty between 2.4.3 and 2.4.4.

"when possible", we've made no such total souce level compat. guarentee. And more such changes are coming, for example the quota bugs can't be fixed without breaking source level compat. for the filesystems.

You may think and argue otherwise, but our ability to break source level compatibility is one of our strengths (see solaris rsh root owned socket bug of yesteryear for one example as to why).

User-mode Linux is now booting on PPC Linux - it can boot with a Debian root floppy image with init=/bin/sh and poke around. It mostly works, although there are still a few problems.

The current patch is available from http://www.tartarus.org/~chris/user-mode-linux/, made against recent UML CVS (see http://user-mode-linux.sourceforge.net).

First off, I'd like to thank Chris for volunteering to undertake the first port of UML and seeing it through to the point where it's basically working. It's a nice demonstration, if any were needed, that UML isn't i386-only.

Based on what I've learned from this port, I'm writing up what amounts to a UML porting guide. It will be found at http://user-mode-linux.sourceforge.net/arch-port.html when I have something ready. It will be incomplete at first - I'll be filling it in as I go through the existing code and as I finish integrating Chris's code into my pool.

So, if anyone wants to port UML to another arch, have a look at that page (and continue looking as I fill it in :-). You'll see that it's not a huge amount of work. UML is fairly portable.

Attached is a driver for the NCR Dual 700 Microchannel card. Since the chip engine of this card is the 53c700-66, which appeared in quite a few other SCSI cards as well, I've abstracted the chip function (in much the same way as the 8390 chip function is abstracted in network cards) so that it should be easy to link it into any other SCSI card using it. As you can see, the actual board specific code is about 150 lines.

The chip driver is full featured (sync (where supported), disconnects and tag command queueing). It will drive both single ended and differential interfaces and uses the new SCSI error handler.

I know we have two drivers that claim to do these chips (53c7xx and 53c7,8xx) but if you actually compile them for this chip, they are completely broken. The chip itself is extremely primitive (not having the table indirect mode, which is the backbone of most of the later drivers) so it makes much more sense to give it its own driver.

The chip driver is currently I/O mapped (because the only cards I know using the chip are I/O mapped), but could easily be made memory mapped as well, just let me know.

What is the on disk layout of Win2k's dynamic disk, i.e. Logical Disk Manager (LDM) database structures? - The article "Inside Storage Managment, Part 2" by Mark Russinovich in Windows 2000 magazine (full text available freely at: http://www.win2000mag.com/Articles/Index.cfm?ArticleID=8303) describes in detail the logical layout of the LDM database, but it doesn't cover enough detail to go off and implement it in Linux (without a certain amount of reverse engineering).

Linux needs to understand the LDM database in order to support dual-boot Win2k (or XP) and Linux configurations where there are one or more dynamic disks present in the system and the user wants to access their NTFS partitions residing on the dynamic disk(s) from Linux.

Just saying "Don't use dynamic disks if you want to use Linux" is IMHO a Bad Thing(TM) as a user might have bought a computer with Win2k preinstalled on a dynamic disk or, even worse, might have been using Win2k only previously, and then the user wants to also install Linux on it. In these cases the user would have to reformat the whole system and start from scratch unless Linux supports dynamic disks...