Kernel Traffic #247 For 31 Dec 2003

For examples of source code distribution in a patented area, look at MPEG4 projects, particularly MPEG4IP.

http://mpeg4ip.sourceforge.net/ has source code under a variety of licenses including GPL, and the front page has the following disclaimer: "This code is not intended for end users, and does not contain executables. Please read all the legal information to determine if it is suitable for you."

The project was started by Cisco, who likely did some legal research first. Cisco, however, have the resources to see off frivolous lawsuits, while individual developers do not.

Are you saying that there's code in the kernel that is subject to your patent?

If there is, then it should definitely be taken out. First, as Linus has stated recently (and as has been the policy for a while), the kernel should avoid having any patented code (http://www.ussg.iu.edu/hypermail/linux/kernel/0312.2/0624.html). Second, "RTLinux Free" is being ported over to Adeos (http://www2.fsmlabs.com/pipermail/rtl/2003-December/013178.html).

That's not true.

The kernel should have no patented code THAT DOESN'T HAVE A LICENSE.

There are several cases where this came up: RCU is one obvious one, but there were also issues with Intel's initial submissions of some of the networking drivers where they didn't want to originally release under the GPL because of worrying about patents they owned.

The email you quote expressly says "unless you can get the patent holder to grant a license". And the RTLinux patents were licensed to GPL'd projects. See the RTLinux "open patent license".

I don't understand why people continually complain about the RTLinux patents. I bet it's because Victor has all the easy charm of Larry McVoy, but I don't see why people still continue to spread obvious mis-information about the situation.

It's doubly discgusting with some of the people who were trying to spread all the FUD and mis-information were doing so because they were themselves doing a non-GPL microkernel, and they complained about how the patents were somehow against the GPL and wanted to get community support by trying to make out the situation to be somehow different from what it was.

I'm not a supporter of software patents, but while I dislike them, I don't dislike them _nearly_ as much as I dislike dishonest people.

In 2.6.x kernels you are also limited by the 2 TiB limit on block device sizes, limit which you can extend compiling a 2.6.x kernel with "Large Block Device" support (menu "Device Drivers" -> "Block devices", at the end).

You will also need filesystems that support that big sizes and can hold big files on them, have a look at the following URL for more information: http://www.suse.de/~aj/linux_lfs.html

Yes. Most glibc are distributed compiled with _LARGEFILE64 support.

See also http://www.gelato.unsw.edu.au/IA64wiki/LargeBlockDevices I haven't got around to updating for 2.6.0, but the 2.5 notes still apply.

Version 2.6.3 of perfctr, the Linux/x86 performance monitoring counters driver, is now available at the usual place: http://www.csd.uu.se/~mikpe/linux/perfctr/

Version 2.6.3, 2003-12-21

• Fixed a bug where a read of the global-mode counters could fail with EOVERFLOW due to an incorrect structure descriptor. The bug only existed in perfctr-2.6.2. (Thanks to Pavel Machek for reporting this problem.)
• AMD64 IA32 emulation code cleaned up for kernel 2.4.23.
• Updated kernel support: 2.6.0, 2.4.24-pre1, 2.4.23, 2.4.22-1.2129.nptl (FC1 update), 2.4.21-1.1931.2.393.ent (RHEL Taroon beta), and 2.4.20-24 (RH 7.x/8/9 update).

• User-space package rpm spec file fixes:

  • Don't remove /dev/perfctr on package uninstall.
  • Don't add alias to /etc/modules.conf if it's already there.

I spent half an hour tearing part of it apart for some journalists. No guarantees for the full accuracy of this write-up, and in particular I don't actually have "original UNIX" code to compare against, but the files I checked (ctype.[ch]) definitely do not have any UNIX history to them.

The rest of the files are mostly errno.h/signal.h/ioctl.h (and they are apparently the 2.4.x versions, before we moved some common constants into "asm-generic/errno.h"), and while I haven't analyzed them, I know for a fact that

• the original errno.h used different error numbers than "original UNIX"

  I know this because I cursed it later when it meant that doing things like binary emulation wasn't as trivial - you had to translate the error numbers.

• same goes for "signal.h": while a lot of the standard signals are well documented (ie "SIGKILL is 9"), historically we had lots of confusion (ie I think "real UNIX" has SIGBUS at 10, while Linux didn't originally have any SIGBUS at all, and later put it at 7 which was originally SIGUNUSED.

So to me it looks like

• yes, Linux obviously has the same signal names and error number names that UNIX has (so the files certainly have a lot of the same identifiers)
• but equally clearly they weren't copied from any "real UNIX".

(Later, non-x86 architectures have tried harder to be binary-compatible with their "real UNIX" counter-parts, and as a result we have different errno header files for different architectures - and on non-x86 architectures the numbers will usually match traditional UNIX).

For example, doing a "grep" for SIGBUS on the kernel shows that most architectures still have SIGBUS at 7 (original Linux value), while alpha, sparc, parisc and mips have it at 10 (to match "real UNIX").

What this tells me is that the original code never came from UNIX, but some architectures later were made to use the same values as UNIX for binary compatibility (I know this is true for alpha, for example: being compatible with OSF/1 was one of my very early goals in that port).

In other words, I think we can totally _demolish_ the SCO claim that these 65 files were somehow "copied". They clearly are not.

Which should come as no surprise to people. But I think it's nice to see just _how_ clearly we can show that SCO is - yet again - totally incorrect.

Linus

For example, SCO lists the files "include/linux/ctype.h" and "lib/ctype.h", and some trivial digging shows that those files are actually there in the original 0.01 distribution of Linux (ie September of 1991). And I can state

• I wrote them (and looking at the original ones, I'm a bit ashamed: the "toupper()" and "tolower()" macros are so horribly ugly that I wouldn't admit to writing them if it wasn't because somebody else claimed to have done so ;)
• writing them is no more than five minutes of work (you can verify that with any C programmer, so you don't have to take my word for it)
• the details in them aren't even the same as in the BSD/UNIX files (the approach is the same, but if you look at actual implementation details you will notice that it's not just that my original "tolower/toupper" were embarrassingly ugly, a number of other details differ too).

In short: for the files where I personally checked the history, I can definitely say that those files are trivially written by me personally, with no copying from any UNIX code _ever_.

So it's definitely not a question of "all derivative branches". It's a question of the fact that I can show (and SCO should have been able to see) that the list they show clearly shows original work, not "copied".

Analysis of "lib/ctype.c" and "include/linux/ctype.h".

First, some background: the "ctype" name comes "character type", and the whole point of "ctype.h" and "ctype.c" is to test what kind of character we're dealing with. In other words, those files implement tests for doing things like asking "is this character a digit" or "is this character an uppercase letter" etc. So you can write thing like

        if (isdigit(c)) {
                .. we do something with the digit ..

and the ctype files implement that logic.

Those files exist (in very similar form) in the original Linux-0.01 release under the names "lib/ctype.c" and "include/ctype.h". That kernel was released in September of 1991, and contains no code except for mine (and Lars Wirzenius, who co-wrote "kernel/vsprintf.c").

In fact, you can look at the files today and 12 years ago, and you can see clearly that they are largely the same: the modern files have been cleaned up and fix a number of really ugly things (tolower/toupper works properly), but they are clearly incremental improvement on the original one.

And the original one does NOT look like the unix source one. It has several similarities, but they are clearly due to:

• the "ctype" interfaces are defined by the C standard library.
• the C standard also specifies what kinds of names a system library interface can use internally. In particular, the C standard specifies that names that start with an underscore and a capital letter are "internal" to the library. This is important, because it explains why both the Linux implementation _and_ the UNIX implementation used a particular naming scheme for the flags.

• algorithmically, there aren't that many ways to test whether a character is a number or not. That's _especially_ true in C, where a macro must not use it's argument more than once. So for example, the "obvious" implementation of "isdigit()" (which tests for whether a character is a digit or not) would be

  #define isdigit(x) ((x) >= '0' && (x) <= '9')

  but this is not actually allowed by the C standard (because 'x' is used twice).

  This explains why both Linux and traditional UNIX use the "other" obvious implementation: having an array that describes what each of the possible 256 characters are, and testing the contents of that array (indexed by the character) instead. That way the macro argument is only used once.

The above things basically explain the similarities. There simply aren't that many ways to do a standard C "ctype" implementation, in other words.

Now, let's look at the _differences_ in Linux and traditional UNIX:

• both Linux and traditional unix use a naming scheme of "underscore and a capital letter" for the flag names. There are flags for "is upper case" (_U) and "is lower case" (_L), and surprise surprise, both UNIX and Linux use the same name. But think about it - if you wanted to use a short flag name, and you were limited by the C standard naming, what names _would_ you use? Maybe you'd select "U" for "Upper case" and "L" for "Lower case"?

  Looking at the other flags, Linux uses "_D" for "Digit", while traditional UNIX instead uses "_N" for "Number". Both make sense, but they are different. I personally think that the Linux naming makes more sense (the function that tests for a digit is called "isdigit()", not "isnumber()"), but on the other hand I can certainly understand why UNIX uses "_N" - the function that checs for whether a character is "alphanumeric" is called "isalnum()", and that checks whether the character is a upper case letter, a lower-case letter _or_ a digit (aka "number").

  In short: there aren't that many ways you can choose the names, and there is lots of overlap, but it's clearly not 100%.

• The original Linux ctype.h/ctype.c file has obvious deficiencies, which pretty much point to somebody new to C making mistakes (me) rather than any old and respected source. For example, the "toupper()/tolower()" macros are just totally broken, and nobody would write the "isascii()" and "toascii()" the way they were written in that original Linux. And you can see that they got fixed later on in Linux development, even though you can also see that the files otherwise didn't change.

  For example: remember how C macros must only use their argument once (never mind why - you really don't care, so just take it on faith, for now). So let's say that you wanted to change an upper case character into a lower case one, which is what "tolower()" does. Normal use is just a fairly obvious

          newchar = tolower(oldchar);

  and the original Linux code does

          extern char _ctmp;
          #define tolower(c) (_ctmp=c,isupper(_ctmp)?_ctmp+('a'+'A'):_ctmp)

  which is not very pretty, but notice how we have a "temporary character" _ctmp (remember that internal header names should start with an underscore and an upper case character - this is already slightly broken in itself). That's there so that we can use the argument "c" only once - to assign it to the new temporary - and then later on we use that temporary several times.

  Now, the reason this is broken is

  • it's not thread-safe (if two different threads try to do this at once, they will stomp on each others temporary variable)
  • the argument (c) might be a complex expression, and as such it should really be parenthesized. The above gets several valid (but unusual) expressions wrong.

Basically, the above is _exactly_ the kinds of mistakes a young programmer would make. It's classic.

And I bet it's _not_ what the UNIX code looked like, even in 1991. UNIX by then was 20 years old, and I _think_ that it uses a simple table lookup (which makes a lot more sense anyway and solves all problems). I'd be very susprised if it had those kinds of "beginner mistakes" in it, but I don't actually have access to the code, so what do I know? (I can look up some BSD code on the web, it definitely does _not_ do anythign like the above).

The lack of proper parenthesis exists in other places of the original Linux ctype.h file too: isascii() and toascii() are similarly broken.

In other words: there are _lots_ of indications that the code was not copied, but was written from scratch. Bugs and all.

Oh, another detail: try searching the web (google is your friend) for "_ctmp". It's unique enough that you'll notice that all the returned hits are all Linux-related. No UNIX hits anywhere. Doing a google for

_ctmp -linux

shows more Linux pages (that just don't happen to have "linux" in them), except for one which is the L4 microkernel, and that one shows that they used the Linux header file (it still says "_LINUX_CTYPE_H" in it).

So there is definitely a lot of proof that my ctype.h is original work.

there are problems with the loop driver in the kernel with block device backing.

In 2.6 we have a device-mapper which does such things in a much more generic way. I've already talked to a bunch of people working on loop and cryptoloop (also with Clemens Fruhwirth, the cryptoloop maintainer) and they all agreed that device-mapper is probably the most correct way to go, and would be happier if the loop driver was used for files only.

I've written a device-mapper target "dm-crypt" some month ago which uses cryptoapi and is compatible with cryptoloop. I never got much feedback, some people tested it and found it to work just fine while cryptoloop didn't, back then, when the test-series started or so.

Unfortunately I never got much public support on LKML itself and was mostly ignored.

I'm not a crypto-loop user, so I am not in a position to judge whether using dm for crypto-on-disk is feature-sufficient and adequate from an operational point of view.

It is good that Joe-and-co are OK with it and are prepared to help support it. If the people who _do_ use crypto-loop like the look of the feature set and the user interface then fine. Certainly the loop driver has a long history of not working very well.

So. If those-in-the-know like it then go wild. It would be useful to get an opinion from the distro guys too.

However I suspect that there will be a migration issue, and that we should continue to work to get crypto-loop functioning well, plan to remove it from 2.8, yes?

First of all, eventhou I'm the maintainer of cryptoloop, when Christophe posted the first time I immediately recognized that dm-crypt is vastly superior to cryptoloop for a number of reasons:

• It does not suffer from loop.c bugs (There are a lot, no maintainer)
• dm-crypt does not depend on special user space tool (util-linux)
• dm-crypt uses mempool, which makes it rock stable compared to cryptoloop.

From an operational point of view, patching util-linux has been the most troublesome point. Lot's of incompatible inofficial patches floating around in the net, while the last release of util-linux provides a broken and IMHO insecure why of setting up cryptokeys. Further: To fix the design flaw of unencrypted/unhashed IV vectors one has to add another setup parameter. That's where I really do like the flexible interface dm has.

version 0.20 of forcedeth (GPLed nvnet replacement for nForce on-board nics) for Linux 2.4 and 2.6 is available at http://www.hailfinger.org/carldani/linux/patches/forcedeth/. It is also integrated in current -mm patchset and the 2.6 experimental net driver queue.

I will make precompiled modules available at the above address as time permits.

Fixes in this release over 0.18:

• Work around bogus MAC addresses. Please report your exact hardware version/manufacturer if you hit this issue since NVidia has stated this should never happen and they are interested in any cases where it happens.
• Under extremely high network load on nForce3 systems, the nic won't lock up or slow down any more.

Known issues:

• Some nForce versions will report every received packet as 1500 bytes long and fool RX statistics. This is definitely a hardware bug and we intend to provide a workaround in v0.21
• nForce3 systems are programmed to an incredibly high interrupt rate. We still need to find out what value to write to the timer register and we intend to fix this in v0.21
• "eth0: received irq with unknown events 0x<something>. Please report" might show up in your logs. That means we didn't encounter such an event during development and can only guess about its meaning. Please tell us what you were doing when the message appeared, how often it appeared and if it affects behaviour of your machine negatively.

I've released the 010 version of udev. It can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-010.tar.gz

rpms built against Red Hat FC1 are available at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-010-1.i386.rpm
with the source rpm at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-010-1.src.rpm

udev is a implementation of devfs in userspace using sysfs and /sbin/hotplug. It requires a 2.6 kernel to run. Please see the udev FAQ for any questions about it:
kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ

Please read the FAQ if you have any questions about devfs and udev. It's been updated a lot in response to all of the confusion about this topic.

The major changes since the 008 release are:

• a number of bugs that were in the 009 release are now fixed. The extras/ subdir should now build properly (but I take no responsibility if it doesn't...)
• we now don't blow away old config files if you install this version on top of an older version. Sorry about that...
• udev now supports looking at multiple sysfs files at once for a single rule. This enables you to write such rules as: LABEL, BUS="usb", SYSFS_vendor="FUJIFILM", SYSFS_model="M100", NAME="camera%n" which helps in trying to identify a device in a unique way. Right now udev supports up to 5 different sysfs files. If you want to have more, the code can be changed, just let me know.
• new format modifier "%k" has been added to allow rules to use the kernel name of the device.
• an example shell script that handles an IDE devfs mapping has been added to the tarball. Thanks to Kay Sievers for this.

Thanks again to everyone who has send me patches for this release, a full list of everyone, and their changes is below.

udev development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/udev

Daily snapshots of this tree used to be found at:
http://www.codemonkey.org.uk/projects/bitkeeper/udev/
But that box seems to be down now. Hopefully it will be restored someday. If anyone ever wants a tarball of the current bk tree, just email me.

Oh, forgot to mention that udev runs a _lot_ slower now, taking at least 1 second for every device addition. This makes the initscript or the test scripts take a long time.

This was done to try to fix the race condition where udev beats the kernel before it has created all of the sysfs links and files. I need some libsysfs changes before this can be done properly, without the big speed hit (the libsysfs people are working on it.)

If anyone's interested in it, take a look at the FIXMEs in namedev.c. I tried messing around with just a simple stat() call, but still couldn't reliably get stuff to work for partitions on usb-storage devices (which are slow to enumerate.) Any help here would be apprecated.

This is a Memory hot-add prototype patch against linux-2.6.0.

My target is a machine which has multi-node like NUMA. So, I assume a situation that a node which has memory is hot-added. I use the idea of multi-node emulation from Iwamoto-san's hot-remove patch. (http://marc.theaimsgroup.com/?l=linux-kernel&m=107025023221904&w=2)

• Hot-added memory is used for highmem in this patch.
• A node has only 1 contiguous memory block.
• kswapd should be executed for added node, but not yet.

To hot-add memory, please execute following command.
$ echo "enable <number of node>" > /proc/memhotplug

TODO list is followings.

• Port to IA64. (It is real target for me. But IA32 code is good for investigation of generic way, and many person can try to use it.)
• Using SRAT table of ACPI to recognize multinode.
• There might be some useful parts in memory hotadd project's patch. (http://sourceforge.net/projects/linux-memhotadd/)
• The case that node has some discontiguous memory block. ( Should I use CONFIG_NONLINEAR? But, I don't have idea how to map the hole address of hot-removed memory by this configuration yet. If you have good idea, I'm glad.)
• etc. (it's a lot. ;-) )

I do not. devfs shall remain in 2.6 and shall continue to be supported.

Richard has disappeared but Andrey Borzenkov understands devfs well and performed valuable work on it during 2.5 development.

Nor would I recommend that devfs be removed early from 2.7.x. We should wait until the proposed udev/sysfs solutions have matured in 2.6 and have proven themselves in the field. Only then will we be in a position to confirm that devfs can be removed without causing some people unacceptable levels of grief. There is no rush.

And yes, there are architectural/cleanliness issues with devfs. In 2.5 Adam Richter totally reinventing devfs's internals, basing it around the ramfs infrastructure. If we elect to retain devfs in 2.8 then that effort should be resurrected.

Now would be a good time for someone to feed the whole thing through indent though.

Even though I don't own a laptop, I find it very irritating that my hard drive is active so much. Wanting to fix this, I found the Jens Axboe's "laptop-mode" patch. Unfortunately it hadn't been ported to Linux 2.6.0 yet, and I'm using that as my primary kernel now. I gave porting it a shot, and here is the result. I'm running it right now, and my hard drive has been spun down for the complete time I have been writing this message. Still, I'm not sure whether it really works as advertised. :) The reason is that my PC is also a mail server for my personal e-mail, and I receive e-mails more than once every 10 minutes (fscking spam!). Still, the tests that I've done seem to indicate that it works.

I've done some things differently than in Jens's patch (my port is based on v4 of the patch BTW):

• The block dirtying reporting is not there. As an alternative, you can write a number N higher than 1 into /proc/sys/vm/laptop_mode, which will give you reports on N-1 actual read/write operations, including the pid+command that caused them. This gives you a way to put a reasonable bound on the number of messages you're going to get. You're only interested in the first one (the one that spins up your disk) anyway. The messages look like this:

  ll_rw_block: (107 reports left) READ requested by imapd (pid 1137)

• When laptop mode is on, wb_kupdate() also performs all of the other actions mentioned in do_sync(). I don't know whether this is really necessary because I don't know the full implications of all those actions, but at least it REALLY makes sure that everything is synced.
• The script writes to different values in /proc/sys/vm, because the Linux 2.6 VM has different controls.
• The patch does not modify the ext3 journal commit timeout; you have to mount your filesystem with commit=600 (or whatever your preferred value is) to get the correct effect. The 2.4 patch modified the expiration of an ext3 transaction, I've removed this as I don't see the use if you're going to have to mount the fs with "commit=" anyway. The default expiration time of an ext3 transaction is equal to the commit interval, which is very reasonable. Usually you'd set your commit interval to the same as the dirty_writeback_centisecs value (in seconds, then), so it would boil down to the same thing.

Disclaimer: I'm not a very experienced kernel developer, and I may have made mistakes. Don't kill me if it doesn't work. :) I'd appreciate some feedback, so if it works or doesn't work for you, please let me know!

Thanks for getting this started. I'm not particularly fond of the behaviourial changes you made, I guess most are due to it being incomplete?

The block dirtying is the most interesting aspect of the dump functionality, reporting WRITEs don't give you the info you need to fix your setup.

This is the second release of the -tiny kernel tree. The aim of this tree is to collect patches that reduce kernel disk and memory footprint as well as tools for working on small systems. Target users are things like embedded systems, small or legacy desktop folks, and handhelds.

Latest release includes:

• sync against 2.6.0
• fix for CONFIG_SYSCTL compilation
• Jeff Garzik's netdrvr patchkit, along with latest netpoll/netconsole code
• kgdb and kgdb-over-ethernet debugging support
• "make checkstack" to find largest stack users
• script to estimate actual code size of inline functions (experimental)
• some other minor new tweaks

Thanks to Holger Shuring, Bill Irwin, and Zwane Mwaikamba for sending me various tweaks.

The patch, currently against 2.6.0, can be found at:

http://selenic.com/tiny/2.6.0-tiny1.patch.bz2
http://selenic.com/tiny/2.6.0-tiny1-broken-out.tar.bz2