Kernel Traffic #206 For 23�Feb�2003

Just a note that I have rediffed for 2.5.55 the patches that use the 64 bit jiffies value to avoid uptime and process start time wrap after 49.5 days. I will push them Linus-wards when he's back. They can be retrieved from

http://www.physik3.uni-rostock.de/tim/kernel/2.5/jiffies64-33a.patch.gz (1/3: infrastructure)
http://www.physik3.uni-rostock.de/tim/kernel/2.5/jiffies64-33b.patch.gz (2/3: fix uptime wrap)
http://www.physik3.uni-rostock.de/tim/kernel/2.5/jiffies64-33c.patch.gz (3/3: 64 bit process start time)

For discussion see http://www.uwsg.indiana.edu/hypermail/linux/kernel/0211.1/0471.html and http://www.uwsg.indiana.edu/hypermail/linux/kernel/0211.1/0847.html

You don't need to have 64-bit jiffy for things like internal timers, nor for uptime tracking.

Timers have well behaving constructs to use 32-bit jiffy quite successfully, and 64-bit values, especially atomicish, in 32-bit register-poor machines (i386) are damn difficult.

I do have a number of machines with 100 to 300 day uptimes, all with "mere" 32-bit jiffy. With 1000 Hz clock that means at least one full wrap-around of jiffy.

Two things:

• We're going to make a CVS archive of Linus tree available, automatically updated, and we'll rsync it to some public place like kernel.org so you can get at the data in a way you want with no BK involved at all.
• If you beat on our servers with that stupid script, we'll shut down the http access. Give us a break already.

If you're interested in checking out from the bitkeeper servers with an open source program (GPLv2 actually) you can do/try it now (almost, see below for the last trouble that I just discussed today on l-k).

http://www.us.kernel.org/pub/linux/kernel/people/andrea/openbkweb/openbkweb-0.0.tar.gz

I should thanks Aaron Swartz, since I included a GPLv2 html2text python library for him (google found it) that makes it trivial to extract the data.

For example to checkout bk head starting from a 2.5.59 tarball on kernel.org the command is:

        mkdir 2.5-pickle
        python bkweb.py -u http://linux.bkbits.net:8080/linux-2.5
--rev=1.947.3.4 -t linux-2.5.59 -c 2.5-pickle

The number 1.947.3.4 cames from this webpage:

http://linux.bkbits.net:8080/linux-2.5/ChangeSet@-4w?nav=index.html

you'll see that 1.947.3.4 matches the release number 2.5.59. The number 1.947.3.4 tells bkweb to apply all the changesets from 1.947.3.5 to the last one. So it's really simple.

bkweb will fetch the data, will save it in the cache directory (2.5-pickle) and it will apply it to the linux-2.5.59 directory.

Unfortunately it doesn't work reliably when Linus merges old changesets, the logic will be confused. The current logic thinks all changesets can be applied in order.

the problem is that to generate bk head it may have to apply also some changeset in the past, not only between 2.5.59 to the last one in cronological order. That still needs a solution. One solution could be also that nobody should send bk pull to Linus anymore, if everybody sends patches it won't risk to reject anymore ;) but I doubt it's an acceptable one for everybody, I assume somebody is confortable with it, if they're using it right now. I also suspect it will be necessary to use a different numbering and not to use the bk changeset numbers as index. The object is to never have to rebuild the database. Currently I don't even store persistent the ordering (not point to store it yet since it's not a 100% reliable ordering ;).

The -c parameter tells the directory where to generate the cache. the cache is truly the whole repository and it contains all the data and metadata in an open form. To get it just run:

changeset = self.unpickle(revision)

where revision can be '1.947.3.5'. dir(changeset) will show you what you can find there, it's a live class after being restored from disk (I put only data in there to save space on disk but in theory it could contain software too, not that I think it is useful but..).

the output looks like this

athlon:~/devel/bkweb> python bkweb.py -u http://linux.bkbits.net:8080/linux-2.5
--rev=1.947.3.4 -t ~/devel/kernel/x -c ~/devel/kernel/2.5-pickle >/tmp/x
Downloading (linux.bkbits.net:8080,/linux-2.5/) ... done.
Found 14 Changeset fallback pages.
Searching rev 1.947.3.4 ... found at CHS 6
Pending 294 revisions
Downloading changeset 1.947.3.5 ... done.
------ Changeset:       1.947.3.5
------ Author:          scott.feldman@intel.com
------ Date:            2003-01-16 23:06:26-05:00
------ Commentary:      [netdrvr e100] udelay a better way

Bug Fix: TCO workaround after hard reset of controller to wait for TCO traffic to settle. Workaround requires issuing a CU load base command after hard reset, followed by a wait for scb and finally a wait for TCO traffic bit to clear. Affects 82559s and above wired to SMBus.

------ Files:           ['drivers/net/e100/e100_main.c']
patching file drivers/net/e100/e100_main.c
make distclean ... done.
Pending 293 revisions
Downloading changeset 1.947.3.6 ... done.
------ Changeset:       1.947.3.6
------ Author:          scott.feldman@intel.com
------ Date:            2003-01-16 23:38:43-05:00
------ Commentary:      [netdrvr e100] fix TxDescriptor bit setting
------ Files:           ['drivers/net/e100/e100_main.c']
patching file drivers/net/e100/e100_main.c
make distclean ... done.
Pending 292 revisions
Downloading changeset 1.947.3.7 ... done.
------ Changeset:       1.947.3.7
------ Author:          scott.feldman@intel.com
------ Date:            2003-01-16 23:40:31-05:00
------ Commentary:      [netdrvr e100] standardize nic-specific stats
support

* Removed /proc/net/PRO_LAN_Adapters
* Added ethtool GSTATS support
------ Files:           ['drivers/net/e100/Makefile', 'drivers/net/e100/e100.h',
'drivers/net/e100/e100_main.c']
patching file drivers/net/e100/Makefile
patching file drivers/net/e100/e100.h
patching file drivers/net/e100/e100_main.c
[..]

at the first reject it stops and writes to stdout (/tmp/x in my example) the patch that rejected so you can analyze it. I think it never completed successfully yet but if it works correctly it should just exit with nothing in stdout (with stdout I really mean fd 1, not stderr of course).

So if you run the same command twice the second time it will find everything in the cache obviously, and the cache is trivial to manipulate using simple python scripts, so you can automate searches on files in the changesets or whatever (including feeding the data into CVS or whatever) So backing up the cache effectively you save the whole bitkeeper repository (only the main branch). Converting the cache to something python indipendent is trivial too, I wrote this thing fast so this was the easiest way.

My main object is to give access to data and metadata of the changesets generated during kernel development in a very manageable form to everybody, reliably, without proprietary licence restrictions, and to store it somewhere (possibly mirrored) in a open standard. This had to be the case since the first place IMHO. the cset/ directory doesn't allow anything like that in terms of regenerating the whole tree or managing the metadata, and it's even less reliable than bkweb. The fact I've to fetch from the web and not from the cset/ directories on the mirrors should tell the whole story about the completeness of the cset/ directory. cset/ is better than nothing but it doesn't obviate the need of bkweb or bk.

I wrote the first line of this program yesterday (it's not that I've that much time for this thing but eventually I wanted to checkout through the bitkeeper database too), clearly it's not well tested and I don't have much time for it.

I guess the bitkeeper network protocol could be also implemented on the longer run, it should be much faster to fetch all the database that way, but clearly decoding the html is been much easier and it is supposed to run in background for me, so I don't care about the checkout speed at the moment.

Reverse engineering the protocol is probably allowed, as long as you don't create an alternative implementation yourself.

I can't see how Larry's license would stop people from writing that alternative implementation, as long as those people don't use bitkeeper themselves.

This is mostly because the license doesn't forbid creating an alternative to bitkeeper (I doubt Larry would even want that, even if the law granted that much power). All it does is not grant the free use of bitkeeper to people working on an alternative.

We'd view reverse engineering the protocol as falling under the "you're working on a competing implementation".

The general message is that you are free to use BK but you aren't free to use BK in any way which could hurt the business which produces BK.

Please show me the case law which says we have to give you our technology, for free, and we do not have the right to say "no way unless you agree to not reverse engineer".

Lots of law says "if you paid for this product, the seller may not impose the following restrictions" with reverse engineering being amongst those.

I do not have any data which says that the same law applies in the case of a no charge copy of the software, do you?

Correct. You do not have to give your software for free. (In European patent terminology it is not called technology, by the way).

Correct. You have the right to say "no way unless you agree not to reverse engineer".

Someone may copy and use your software _without_ agreeing to the license.

Then you can sue them for breach of copyright.

You will win, unless their copying was fair use.

Reverse engineering for interoperability is a form of fair use in many countries, including Germany and the UK.

Draw your own conclusion :)

Larry's touched an open source nerve by creating what is in effect a closed protocol, and getting it into the heart of the kernel development process.

Blame has nothing to do with it. We _greatly_ respect Larry's accomplishments in this area (and others), and as for the kernel development process we might just as well say it's Linus fault for choosing to use featureful Word 7 when unusable Abiword 0.01 was available :)

Quite a lot of kernel trees are maintained using Bitkeeper now, btw, which is testament to how well that vision is implemented.

It would be very *impolite* to ignore Larry's license and use the software he developed against his express wishes. (It does _not_ give me a good feeling even to talk about it).

On the other hand, it's widely accepted that making a program which _interoperates_ with someone elses closed source program is socially acceptable, and often desirable. This is so widely understood that it's expressly written into copyright law in virtually every country which has copyright law.

Thats how important interoperation is considered to be - it's not just a geek thing, it's a _principle_ - something which is widely held to be the right thing to do for the sake of the big picture.

That's why Larry gets so much flak. He is generous with a caveat which touches a nerve. Conditional love. (Slippery slope... the GPL is like this too :)

He gets flak precisely _because_ his software is so good that so many people choose to use it, and accept the attached strings. (It's a compliment, see?)

Imagine if all your friends started talking a different language, called Binglish say. You'd want to talk to them in that language so you could socialise and work with them. Now if they told you you must sign a contract and join a private society, or pay significant cash, otherwise you couldn't talk the language? If it were a few people you'd ignore them, but if it seemed like all the really powerful people who affect your life would just ignore you unless you talked it, you'd be pissed off. You'd feel the playing field had become unlevel and needed correction. You'd be tempted to either (1) cave in and pay or join the private society, (2) learn the language and use it anyway.

In other words, we see shades of glass ceiling for non-Bitkeeper users. Not intentionally, it is simply an effect that occurs when something good is limited to a subset of people - whether by choice or not.

And nobody thinks glass ceilings should be sustained, do they? Do they?

Now, if you accept that writing a program to communicate with users who do their daily work using the BitKeeper product - in their choice of language - is a good idea. How are you going to do that? Larry's created a situation where the only way to do that is to analyse his software against his express wishes.

Which is more important? Being polite to Larry, or being able to write programs that communicate with important people in their preferred language - levelling the playing field somewhat?

It's harsh reality, but if you create something that benefits many people and attach conditions to your generosity, you'll get a lot of complaints. It's true of the GPL, it's true of the BKL, it's true of the World Bank etc.

Sometimes those complaints are based on sound principles. Sometimes they're not.

You decide.

my short term concern is not to speak Binglish but just to translate from Binglish to English. We need access to the data with an open protocol and to backup the data in a open format. so we can use it too. And Larry is now going to provide the data in the open, IMHO only if that didn't happen we had to research into the possibility of legally reverse enegeneering the bitkeeper protocol. the fact he is now providing the data out in the open avoids us to waste time.

After we can reach the data we can use any version control system we want to manage it, I'm going to write MORE STUPID scripts to do that. I'm been told of several giga archives with dozen thousand revisions under subversion for istance (I know Al Viro blamed subversion code but if the design it's good it may be a good start). subversion may not have all the features of bitkeeper but we can certainly add them over time, the only thing it matters to me is that we get rid of being forced to use a proprietary protocol to fetch the data.

The kernel CVS in more than enough for my/our needs and I thank Larry for seeing it was necessary to allow the kernel data to be open. Now there's no reason to argue anymore with Larry or Linus, they can choose what they can legally use and we can choose what we can legally use and what we find more productive in the long run. I really believe in open protocols and open source software being superior and a necessary thing in the long run, it's not that I advocate people to use open source products and then I change my mind and I run proprietary apps to develop the kernel (I don't put a smile here because clearly this isn't an obvious thought).

the open source community has a choice. We've granted people the free use of our tools, unlike any other company. If the response is going to be "hey, cool, now it's even easier to copy these tools" then (a) we're idiots and (b) we're not going to follow that up with more free services.

Make up your mind as to what you want as a community and let me know. I know what you want is a GPLed BK. That's not on the table. You don't have one and you can't produce one any time soon, it would take years, ask the Subversion guys or the arch guys or anyone you like, this isn't a matter of some perl scripts and a weekend hack. So until someone produces a viable alternative, you can have or not have BK to use, depending on your behaviour. I'd love to reach a compromise which leaves all parties happy. You want free access to the data in real time, I want you to leave us a chance to grow our business. If you want to have access to the data in CVS and you want to be free to go clone BK and you want to be free to go use BK without charge, that's asking way too much.

One answer, maybe the only viable answer, is to use patents to protect our technology. To date we've been very sparing in where we have done that, there are substantial chunks of BK without patent protection. Some leaders in the kernel group have privately told me to not ship BK without patent protection. That slows down how fast you get a better BK, it's not something we can just wave our hands and make be so, it's lots of time and money. A single patent costs more than enough bandwidth to keep all of you happy for a year. Whatever. If you guys can't come to some sort of consensus, then patents are the route we'll choose, even German law respects patents.

Why can't we have a mailing list that sends out a diff for each update?

That way, all you need is:

:0
* ^X-Mailing-List: up-to-the-second-linux-patches
| /usr/local/bin/apply_patches

and

#!/bin/sh
cd /usr/src/linux-current
cat | patch -p1

I just wanted to confirm that bk-commit-head is actually:

1. Complete
2. Realtime

If so, anybody can track Linus' tree, in real time, without using BitKeeper.

Therefore, mailing lists for every single Linux-related BK repository would allow anybody to track all of the trees in real time, without using BK.

It should be complete -- but bear in mind that you may receive the mails in a different order to the order in which they were sent, so just applying them from a mail filter isn't necessarily sensible.

Also note that the dates on them are the date of the changeset itself, not the date of application to Linus' tree (or indeed the date of the cron job which creates the mail).

Almost -- it's run from an hourly cron job, which is more 'real time' than Linus actually pushing from his own box to master.kernel.org and quite enough of a demand on resources already.

It's not done with triggers on Linus' tree because I suspect that would actually make Linus _wait_ while the mail is generated for every changeset he's pushing to master.kernel.org. I do it with a cron job which pulls from Linus' tree to another, and I don't do it with triggers in my own tree because I suspect that would keep Linus' tree locked while it generated the mails too. I do need to investigate possible improvements to the way it's generated, though.

Ok, this contains a lot of patches all over the map, since there were two weeks of merging to be done (and the merging took almost a week).

Networking, USB, SCSI, ALSA, ACPI, and various architecture updates (UML, Sparc, ia64, ppc64 and ARM).

And obviously part of the merge from Andrew Morton.

And Roland McGrath has been fixing POSIX thread signal handling, and that in turn ended up causing a lot of downstream fixes in affected areas.

Currently, my "hack" to get ARM working is as follows, and is not the best thing to view on a full stomach.

Generalising the signal handling might have made sense, but this amount of duplication _just_ to be able to handle non-hardware breakpoints is getting rather rediculous.

I will be looking into the possibility of carving up the generic signal handling into a saner structure so we don't have this mess.

Obviously, this patch is not for applying. 8)

Just for throwing out suggestions, UML can easily avoid using glibc altogether as it is already intimate with the system call layer.

Or it can use the linker to play games with symbol names to move the kernel off into it's own separate name space.

This sounds like a good opportunity to figure out which makes most sense and future proof UML.

This falls apart for a couple of reasons -

• I get a duplicate definition of sscanf which I can't get rid of (libc vs. lib/vsprintf.c)
• More seriously, this assumes a static link and breaks the dynamic build of UML

In order for this to work, references from userspace.o which can be resolved in libc must somehow survive the link against the kernel code unresolved. That is, references to things not in libc must be resolved from kernel code and references to things in libc must be resolved from libc even if there is a symbol with the same name in the kernel. I don't see any way of doing this.

The only other solution I see is to rename the kernel sigprocmask. Oleg Drokin has done with this with a

-Dsigprocmask=__sigprocmask

on kernel code compiles, and I've done it at link time with objcopy. This sucks, but it does have the great advantage that it actually works.

Anyway, if there's something better, I really want to know about it.

ipv6 should be basically identical between 2.4.x and 2.5.x

The only difference is IPSEC, and even for that the ipv6 portions are still only in their most basic stages.

I bet that all their 2.4.x patches apply pretty cleanly to 2.5.x

Finally, after too many delays, I am pleased to announce the release of my driver for 3Com's 3cr990 family of network cards.

This initial release is endian clean, tested on x86 and sparc64, and does

• NAPI
• Zerocopy Tx
• TCP segmentation
• Hardware VLAN accel

It does *not* do IPSEC crypto offload as of yet. I have a version for the 2.4 series of kernels, but it is an ugly, ugly kludge, and would be shot if seen in public. I will be working on integrating the hardware with the IPSEC support in 2.5.x.

There are a few issues with the firmware -- DMA to a 2 byte aligned address hangs the firmware, so we cannot easily align the IP header, and the firmware will always strip the VLAN tags on packet reception, regardless of our desires. I hope to work with 3Com to resolve these issues.

The code is available via BK at
http://typhoon.bkbits.net/typhoon-2.4
http://typhoon.bkbits.net/typhoon-2.5

Since I work for a company who works on competing version control products I'm not allowed to use bitkeeper. Since his patches are only in bitkeeper format I can't read them.

I'd like to read them so I asked for them in a open format.

Since Red Hat works on CVS I wonder if they are breaking the license or not. However thats not relevant right now. If I get stuff in .doc format that I can't read I ask people to resend it in a format I can, ditto bitkeeper or any other randomly weird VC file.

And no I don't want to use bitkeeper any more, your ever changing licenses and removals of more and more rights have reminded me why many proprietary vendors just cannot be trusted and why format lockin is so dangerous.

You are turning a simple "can I have that in a format I can read" into yet another epic bitkeeper flame war.

Linus stated in public that he was/is unhappy with CVS. Without Bitkeeper he might use Subversion today. But by using Bitkeeper he made it possible that you and your company started using him as your posterboy for the "SCM good enough for Linus Torvalds to use". This is IMHO not correct. BK is just "the first SCM which came along and was good enough for Linus Torvalds to use it".

I do remember Linus saying that he wants to try out BitKeeper for the 2.5 development tree and if it does not work out, switch to something else in the 2.6, 2.7... cycle.

The rift that the whole BitKeeper/BitMover stuff has opened in the kernel developer community IMHO justifies such a step forward . I'd like to see SVN to be used as an alternative tool. Not because it is better (it probably is not, but I haven't had a chance to try out BK because I don't qualify for the free license) than BK but because it has no strings attached to its usage.

http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.61/2.5.61-mm1/

• Jens has fixed the request queue aliasing problem and we are no longer able to break the IO scheduler. This was preventing the OSDL team from running dbt2 against recent kernels, so hopefully that is all fixed up now.
• The anticipatory scheduler is performing well. I've included that now.
• Also included the CFQ I/O scheduler. The kernel defaults to using the deadline/anticipatory scheduler. Select CFQ by adding "elevator=cfq" to the kernel command line. Do "dmesg|grep elevator" to see which one you are using.

• There is an updated version of the reiserfs_file_write patch here. This patch addresses CPU efficiency when performing appending writes (ie: the usual sort).

  To do this, it requires that userspace pass "large" amounts of data into the write() system call. So the filesystem returns a value of 128k in the stat.st_blksize field from the stat(2) system call. In the hope that some applications are using that kernel-provided hint.

  Turns out that some parts of KDE (kmail, at least) were indeed using this hint, and it triggers a nasty bug in (at least) kmail: it is reading the same 128k of the file again and again and again. It runs like a dog. Ed Tomlinson upgraded his KDE/kmail version and this problem went away.

  So that is something for reiserfs users to keep an eye on.

Thanks again for doing all this, really appreciate it.

Well, we're closer....

The showstopper for us is still the flock() issue. We have Mathew Wilcox's patch from 2.5.52, which we have been applying to all recent kernels. The patch is in PLM as patch id # 1061. The issue is in BugMe as bug #94 .

Without proper flock() we cannot stop and restart the database, which means we can't run the test. We've tried applying Wilcox's flock patch to -mm1, but it's doesn't go clean, and frankly we're not smart enough to do the merge by hand - lock code scares us.

We just tested 2.5.61 vanilla, and 2.5.61-mm1.

The patch applies cleanly to stock 2.5.61, and we can cycle the database. We can't run dbt2 on stock 2.5.61, because of the scheduler bug. We believe the scheduler fix in -mm1 will be the ticket, but we can't try it because of the flock() issue. So we're wedged. Can someone smarter than us maybe do a merge?

Most PC superio chips have supported up to 460800 or 921600 baud for years.

This patch adds kernel support for the two most common ways of achieving this -- the SMSC chips' "magic multipliers" where a value of 0x8001 or 0x8002 as divisor actually means _multiply_ by 2 or 4 respectively, and the National Semiconductor version where you can set the baud_base to any of three values, but whenever you touch the original 16550-compatible divisor registers, it reverts baud_base to 115200 for compatibility.

Although both of these modes are designed for backward-compatibility, there's _also_ a bit in the superio chip which can disable them and force _complete_ 16550A compatibility. If your BIOS doesn't enable these, you may need Shsmod <http://www.devdrv.com/shsmod/> to do so for you.

If you have an SMSC superio chip, the driver can't autodetect this -- you need the (also-attached) patch to setserial, and to run 'setserial /dev/ttySx magic_multiplier' for each port. This will make it support baud rates of up to 460800.

If you have a NatSemi chip and it's set to enable high-speed before the 8250 driver probes it, then its high-speed capability should be automatically detected and you'll get up to 921600 baud from it without having to do anything special. If your BIOS doesn't enable the high-speed mode, then after using Shsmod to do so, run 'setserial /dev/ttyS0 autoconfig'.

I've also had a hacked-up superio probe enabling high-speed mode on these ports automatically so you can boot with 'console=ttyS0,460800' etc., but that's an ugly hack and adds a _third_ set of superio probe code to the kernel, so doing that can come later. Eventually we can do the superio probe centrally and remove it from the parport, IrDA, SH board setup, and other code where it's currently duplicated. That's probably a 2.7 task though.

This is a patch to support IPv6 IPsec on linux-2.5.62. It work well.

I assume that skb->h.raw points properly on the skb in both inbound and outbound.

IPv6 has some extension headers. It is not as simple as IPv4 options. AH however needs to fill zero on mutable options. skb->h.raw is used as an end of processing mutable options.

There is crude trick at IPsec on Neighbor Discovery, because kernel needs dst to do IPsec but there is no dst at doing ND.

I have no idea to avoid this issue except making dummy route at this moment.