Kernel Traffic #207 For 2 Mar 2003

Hmm.. Mostly lots of small updates, although the merge with Andrew included the RCU dcache patches from IBM that he has carried along for a while (ie fairly fundamnetal, but also very well tested).

ARM, PPC, PPC64, alpha, kbuild.

Oh, and as a sign that 2.6.x really _is_ approaching, people have started sending me spelling fixes. Kernel coders are apparently all atrocious spellers, and for some reason the spelling police always comes out of the woodwork when stable releases get closer.

After much testing, which is still in progress it would seem that *maybe* mjb4 does have the problem too, although it's much harder to hit. Please note that this is a single data point where for other kernels I have two or more occurrences of spontaneous reboots.

I've been checking older kernels... it would seem the problem first occurs in 2.5.53 (that is 2.5.53 through 2.5.62-bk all reboot for me). 2.5.51 doesn't appear to and thus far neither does 2.5.52.

I say thus far, because the problem usually appears after about 15 minutes of compiling, but it sometimes takes a little longer. I'm running 2.5.52 now and after 45 minutes it's still going.

As to what difference it might be between '52 and '53 I have no idea. I had a quick look and the changes there are considerable.

I've tried different compiles, with and without preempt, and and without IO-APIC and trimming down the kernel...

Btw, this is particularly true if it takes you potentially hours to test something like 2.5.51 for stability, but you can reboot 2.5.59 at will in ten minutes.

In that case, you can test several vrsions of "2.5.59 + partial -mjb patches" much more quickly than you can walk backwards in 2.5.x, and try to pinpoint the "this part of -mjb makes it much less likely to reboot".

Also, with the -mjb patch there are some new configuration options. For example, CONFIG_100HZ on -mjb has very different behaviour than a plain 2.5.59 kernel that defaults to 1kHz timer clock, and maybe the reason -mjb seems more stable is that you may have selected a configuration option that made -mjb act differently.

Regardless, it would be very interesting to hear what the -mjb split-down results would be. Even if the answer might be "at 1kHz timer it is unstable, at 100Hz it is stable" (and if that were to be it, then you'd have to walk backwards to 2.5.24 to find the old 2.5.x kernel that had a slow tick rate).

• Completely arch-independent.
• No longer touches atkbd (as suggested by Vojtech Pavlik).
• With minimal changes to existing code.
• Introducing morse code ops -- a good hack value (MOST IMPORTANT!!).
• Useful to people who spend most of their time in X.

Based on a suggestion and several people saying they would find it useful, I'm keeping a kernel errata page on the Linux Test Project website at http://ltp.sourceforge.net/errata

I'll try to maintain a list of known fixes for blocking problems with the most recent kernel on this page. By "blocking", I mean anything severe enough to keep you from testing the kernel (can't compile, panic on boot, catches your hair on fire, etc). This is _not_ a replacement for the bug tracking system. I won't put anything here that doesn't have a fix or a workaround. So if you're looking for an exhaustive list of problems for a given kernel, look at bugme, but if all you want is a list of known fixes to get you up and running quickly, the errata list should give you a quick and easy answer.

If you have any suggestions to make this more useful or if you see anything I've missed, please let me know.

With attached patch to CSSC (www.sf.net/projects/cssc), and

rsync -zav --delete nl.linux.org::kernel/linux-2.5 linux-2.5

you can download local copy of whole linux-2.5 repository, and you can access it, too.

I've put some extensive documentation for the latest version of my Kernel Bug Database online at:

http://grabjohn.com/kernelbugdatabase/documentation/

The latest version of the Kernel Bug Database is at the usual location:

http://grabjohn.com/kernelbugdatabase/

Main new features since version 2.0:

• A slightly redesigned layout that loads more quickly
• Kernel versions are now sorted numerically by version number order
• The list of all bug reports or confirmed bugs on the search screens has been replaced with a list of the five most recent
• Search results now include status information for the latest tested kernel versions, not the latest versions.
• Automatic archiving of bug reports after four weeks
• An active/archive flag for bug reports and confirmed bugs
• A count of the number of returned bug reports, confirmed bugs or patches for each search
• A count of the number of non-returned due to being archived bug reports or confirmed bugs for each relevant search
• The searching of confirmed bugs page now replaces the initial main menu
• An 'include status information for a particular kernel version' toggle in bug reports and confirmed bug search pages

I've written the code for automatic testing of patches to see whether they will apply to various kernel versions, but it will not be enabled until the machine that hosts http://grabjohn.com/ is replaced with a faster one with more disk space :-).

I found a few stupid bugs in the IPMI driver dealing with certain error cases, and this also contains the documentation updates that I have not sent to Linus enough times to be included yet :-).

The 2.5 version is attached. The 2.4 version is at: http://sourceforge.net/projects/openipmi/

FUSE lets you write your very own filesystem, as an ordinary program. It has a simple yet comprehensive interface, and provides an easy way to create a virtual filesystem for just about any application.

Example applications include: automatic CD changer fs, remote filesystems for handhelds, filesystem view for databases, etc...

FUSE currently works on all 2.4.x kernels (up to 2.4.20 and possibly later). Installation is simple, no kernel patching or recompilation is needed. Documentation for the interface and example programs are provided in the package.

You can download the latest version from:

http://sourceforge.net/projects/avf

Lots of people have requested that threads should show up in /proc again, to be able to look at per-thread CPU usage, activity, and generally, to ease the debugging of threaded apps.

the main problem with threads in /proc is that there's a big slowdown when using lots of threads. Here are some runtime numbers in seconds, under 2.5.62, on a 525MHz PIII box:

   # of threads (*):  1000    2000    4000    8000    16000
   --------------------------------------------------------   
   ps                 0.77    1.52    3.13    6.44    13.81
   ps -axm            0.93    1.85    3.78    7.73    18.37
   top -d 0 -n 0      0.75    1.53    3.23    7.60    22.12

[ (*): the system is completely idle, all threads are sleeping. Overhead is combined system and userspace overhead measured via 'time'.]

ie. the overhead is really massive, eg. with 16K threads running, procps is basically unusable for any administration or debugging work. And there are users that want 50K or more threads. So clearly, this state of procfs and procps is unacceptable - who would use 'top' to check a system's state if a single screen-refresh takes 22 seconds?!

in the above timings, only 'ps -axm' is actually displaying every thread, all other commands produce only a few lines of output. The reason of the overhead is two-fold:

1) there's significant kernel overhead in reading large /proc directories, the overhead of many readdir()'s is O(N^2). The main overhead is in get_pid_list(), which has to loop over an increasing number of threads to find the next intended batch of PIDs.

to fix this overhead i've introduced a 'lookup cursor' cookie, which is cached in filp->private_data, across readdir() [getdents64()] calls. If the cursor matches then we skip all the overhead of skipping threads. If the cursor is not available then we fall back to the old-style skipping algorithm.

2) procps is forced to parse every thread in /proc to build up accurate 'process CPU usage' counters. The parsing and accessing of every /proc/PID/stat file is necessary because CPU statistics are scattered across all threads.

the fix for this is two-fold. First, it must be possible for procps to separate 'threads' from 'processes' without having to go into 16 thousand directories. I solved this by prefixing 'threads' (ie. non-group-leader threads) with a dot ('.') character in the /proc listing:

 $ ls -a /proc
 .      16994   .17078  412  7          execdomains  locks       stat
 ..     16995   .17079  460  8          filesystems  meminfo     swaps
 1      17031   .17080  469  9          fs           misc        sys
 16864  17033   .17081  5    92         ide          mounts      sysvipc
 16866  17034   .17082  515  buddyinfo  interrupts   mtrr        tty
 16867  17072   17113   516  bus        iomem        net         uptime
 16946  .17073  2       517  cmdline    ioports      partitions  version
 16948  .17074  3       518  cpuinfo    irq          profile     vmstat
 16949  .17075  390     519  devices    kcore        scsi
 16989  .17076  4       520  dma        kmsg         self
 16992  .17077  400     6    driver     loadavg      slabinfo

the .17073 ... .17082 entries belong to the thread-group 17072.

The key here is for procps to be able to parse threads without having to call into the kernel 16K times. The dot-approach also has the added benefit of 'hiding' threads in the default 'ls /proc' listing.

the other change needed was the ability to read comulative CPU usage statistics from the thread group leader. I've introduced 4 new fields in /proc/PID/stat for that purpose, the kernel keeps those uptodate across fork/exit and in the timer interrupt - it's very low-overhead.

the attached patch, against 2.5.62-BK, implements these kernel features.

Alex Larsson has modified procps for these new kernel capabilities, the new procps package (or the patch against upstream procps) can be downloaded from:

http://people.redhat.com/alexl/procps/

here are the performance measurements (with stock procps+procfs numbers in paranthesis)

   # of threads:       1000     2000     4000      8000      16000
   ----------------------------------------------------------------
   ps                  0.02     0.03     0.03      0.03       0.04
                      (0.77)   (1.52)   (3.13)    (6.44)    (13.81)

   ps -axm             0.89     1.72     3.40      6.87      15.57
                      (0.93)   (1.85)   (3.78)    (7.73)    (18.37)

   top -d 0 -n 0       0.11     0.12     0.12      0.13       0.16
                      (0.75)   (1.53)   (3.23)    (7.60)    (22.12)

eg. with 16K threads running in the background, a single 'top' screen-refresh got more than 130 times faster. A simple 'ps' got more than 340 times faster ... Even the 'ps -axm' (which displays all threads) command got faster, due to the pid-cursor. But even with just 1000 threads running a simple 'ps' is 30 times faster, and top refresh is 6 times faster.

another advantage of this approach is that old procps is fully compatible with the new kernel, and new procps is fully compatible with old kernels. Plus everything is still encoded in the ASCII namespace, no binary interfaces are used.

the patch works just fine on my boxes. (the patch is also included in the threading backport, in the rawhide 2.4 kernel, and has been in use for a couple of weeks already.)

Well, part of the problem (I think) is that you added all the threads to the same main directory.

Putting a "." in front of the name doesn't fix the /proc level directory scalability issues, it only means that you can avoid some of the user- level scalability ones.

So to offset that bad design, you then add other cruft, like the lookup cursor and the "." marker. Which is not a bad idea in itself, but I claim that if you'd made the directory structure saner you wouldn't have needed it in the first place.

It would just be _so_ much nicer if the threads would show up as subdirectories ie /proc/<tgid>/<tid>/xxx. More scalable, more readable, and just generally more sane.

Currently, 32-bit emulation in kernel has *5* copies, and its >1000 lines each. Plus, locking of all but x86-64 architectures is broken (I'm told by andi ;-).

So, here's patch that starts sharing sys32_ioctl() [as a first step], which should rmove locking problems.

I've done the work for x86-64 and sparc64; if it looks good I'll attempt to do other architectures. [Unless maintainers prefer to do it themselves: I don't have easy access to 64-bit machines besides hammer.]

Yes :/ Consolidating all these copies into a single layer has been a "project to be" for quite some time.

I do not know if it is too late in 2.5.x to begin this work, however. We _are_ in a feature freeze... I suppose it is up to the consensus of arch maintainers, because it [obviously] does not affect ia32.

For s390, I'd love to see progress in the consolidation. Feel free to submit changes for arch/s390x/kernel/ioctl32.c directly, like Stephen Rothwell does for the syscall32 consolidation. Of course, Martin has the last word here, but I'm rather sure he agress with me in this.

If you want access to an s390x system, you can probably get access to one at http://www-1.ibm.com/servers/eserver/zseries/os/linux/lcds/ or install the hercules emulator. I try to keep working kernel tree at http://linux-390.bkbits.net/, but the 32 bit emulation has been broken for most of 2.5.

Note that for any ioctls that pass pointers, you will need special massaging for the high order bit of the user space pointer, because s390 only has 31 bit pointers, not 32 bit.

These problems are being resolved. Most of them have already been resolved. The cvs version for 2.4, which you can find at

http://www.linux-usb.org/SpeedTouch/

is quite stable. In theory it can still crash (due to various micro races), but in practice it does not. In any case, these micro races will be fixed soon. The 2.5 version, which is essentially identical to 2.4 cvs, doesn't work very well in the current 2.5 kernel. I don't know why. I am working on it.

I have nothing against the user space version, which I used for many moons. The kernel version is certainly much lighter weight - less CPU, less memory. Whether this matters for you depends on your machine/needs. My machine is slow, and I need all the CPU time I can get!

The main disadvantages of the kernel mode driver were:

1. unstable, and very unstable on SMP/preempt boxes
2. required running the closed source speedmgmt program
3. required compiling your own kernel

The driver is in 2.5, and is heading for inclusion in 2.4, so I expect that in the future most distributions will ship with the speedtch module compiled. Thus (3) is going away.

The cvs version of the user space driver contains a patch for modem_run which enables it to be used with the kernel driver in place of speedmgmt (use the -k flag). Thus (2) has already gone away.

As I mentioned, (1) is (almost) dealt with.

I am just wondering if anyone thinks the same way. I am thinking about going to BSD for good because of these things.

Linux is starting to include code written under non-disclosure agreements and other nasties, and for me this kills the magic of Linux. Doesn't stop anyone from reading the code, but it's the principle that counts.

I don't know. I feel Linux has lost the plot. It's supposed to be getting bigger and better (well, at least bigger -- to borrow words from Andy Tanenbaum. He was referring to the onslaught of "improvements" to Minix being submitted, but now this applies to Linux as well.)

True, there are lots of nice improvements and binary only drivers cannot use ksysms, so I've read somewhere. (True? Nice.) But NDAs? Come on, where do you get off on that?

OpenBSD contains drivers _obviously_ written under NDA, just like every other free BSD. Theo probably imported these drivers from FreeBSD, I would guess.

I pay attention to net drivers in all the BSD OS's, so if you know net drivers and their vendors, these things are obvious ;-)

http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.62/2.5.62-mm3/

• Included Dave McCracken's "Object-based RMAP" patch.

  What this dopey name actually means is that when page reclaim tries to unmap a file-backed page it walks the VMAs attached to the address_space and the pagetables attached thereto. So there is no need for pte_chains for these pages.

  The patch is simple, but potentially has search complexity problems with weird workloads which have high sharing levels. Allegedly. Work is ongoing.

• Several more anticipatory scheduler tweaks. This has been an exercise in hunting down situations in which the scheduler does the wrong thing, and plugging those up.

  The only known problem at this time is a ~20% falloff in threaded OLTP-style database activity. This is really complex, involving interactions between reads, O_SYNC writes and i_sem contention.

  The problem will recede to less than 10% when we retune the anticipation timer (it is currently set too high just so we can discover these things).

  But we do not see a sane way of fixing this for real. At present it will need to be manually tuned away with

  echo 0 > /sys/block/hdXX/antic_expire

  Later, we will probably have to perform this disabling automatically.

• There are some performance fixes which will help OLTP-style workloads which are using regular old files through the pagecache. It seems to be running maybe 60-70% faster than 2.4.x now, but it varies.
• A few performance patches to reduce the amount of work we do in update_atime() and __mark_inode_dirty() should pull back some of the regressions which have been observed in there.

NTFS 2.1.1a is now released for kernel 2.4.20. This fixes both the reported hangs and improves the handling of compressed files so that the warning message people keep reporting is now gone. (Note the hangs were specific to the 2.4.x kernel ntfs versions. 2.5.x kernel ntfs versions are not affected.)

Download the patch from:

http://linux-ntfs.sf.net/downloads.html

Or get from our BK repository (which is at the current BK linux-2.4 version, i.e. 2.4.21-pre4-bk):

bk://linux-ntfs.bkbits.net/ntfs-2.4

I'd like to annouce that there are two new interesting vendor kernels available on the kernelnewbies vendor kernels page (http://www.kernelnewbies.org/kernels/). Both unfortunately don't use the traditional organization of multiple pathes agains ta base kernel release but were tarballs that I had to diff against known kernel release.

In detail they are:

1. The NEC kernel for their IA64 machines.

   Interestng here are their VM changes and a NFS extension for shared storage called GFS (it's different from Sistina's filesystem of the same name)

2. The kernel from TimeSys 3.1 demo release

   This one is really interesting, it features a completly new scheduler architectured around hooks to their propritary real time kernel and heavyweight mutexes (e.g. with priority inheritance) that replace Linux spinlocks. These architecture probably violates the GPL, but I'd like to hear some more opinions on people who actually read the code before bothering TimeSys about this issue.

Overall statistics
------------------
First message was written at: 2003/02/18 13:09:03
Last message was written at: 2003/02/19 23:54:45
Total number of messages: 1595
Total size: 8704KB
Total number of writers: 369
Number of people who wrote >1 message: 196
Total number of lines: 141544
Average lines per message: 88
Total header length (lines): 76324
Average header length (lines): 47
The header is on average 53.92% of the message (lines).
The header is 46.05% bytes in size of the total.
Average number of bits information per byte: 0.7501
Total number of unique user-agents: 142
Total number of unique organisations: 44
Total number of unique top-level domains: 51

Importance
----------
Low   : 0.00%
Normal: 1.38%
High  : 0.00%
(the rest is unspecified)

Top writers
   | # msgs|av size| total|time| e-mail address
---+-------+-------+------+----+--------------------------------
  1]     76|   4785| 355KB|0854| Alan Cox <alan@lxorguk.ukuu.org.uk>
  2]     59|   5692| 328KB|1428| "Martin J. Bligh" <mbligh@aracnet.com>
  3]     57|   3410| 189KB|1124| "David S. Miller" <davem@redhat.com>
  4]     53|  16844| 871KB|1937| Osamu Tomita <tomita@cinet.co.jp>
  5]     42|   5531| 226KB|1133| William Lee Irwin III <wli@holomorphy.com>
  6]     42|   5055| 207KB|1415| Andrew Morton <akpm@digeo.com>
  7]     35|   4743| 162KB|1334| Linus Torvalds <torvalds@transmeta.com>
  8]     32|   3391| 105KB|1441| Jeff Garzik <jgarzik@pobox.com>
  9]     28|   5374| 146KB|1437| Andrea Arcangeli <andrea@suse.de>
 10]     26|   3938|  99KB|1540| Ingo Molnar <mingo@elte.hu>

Top subjects
   | # msgs|av size| total|time| subject
---+-------+-------+------+----+--------------------------------
  1]    150|   4476| 655KB|1255| Minutes from Feb 21 LSE Call
  2]     34|   4474| 148KB|1339| doublefault debugging (was Re: Linux v2.5.62 --- spontaneous
  3]     28|   4249| 116KB|1154| [patch] procfs/procps threading performance speedup, 2.5.62
  4]     23|   3795|  85KB|0840| Longstanding networking / SMP issue? (duplextest)
  5]     23|   3507|  78KB|1609| Linux kernel rant
  6]     21|  10297| 211KB|1216| AGP backport from 2.5 to 2.4.21-pre4
  7]     21|   3944|  80KB|1026| Linux v2.5.62
  8]     19|   4189|  77KB|0925| [PATCH] add new DMA_ADDR_T_SIZE define
  9]     17|   4824|  80KB|1129| [RFC] Is an alternative module interface needed/possible?
 10]     16|   4265|  66KB|1355| RFC3168, section 6.1.1.1 - ECN and retransmit of SYN

Top receivers
   | # msgs|av size| total|time| e-mail address
---+-------+-------+------+----+--------------------------------
  1]    424|   6563|2717KB|1428| linux-kernel@vger.kernel.org
  2]    149|   7420|1079KB|1409| Linus Torvalds <torvalds@transmeta.com>
  3]     58|   6021| 341KB|1316| Andrew Morton <akpm@digeo.com>
  4]     48|   8122| 380KB|1334| "David S. Miller" <davem@redhat.com>
  5]     47|   5845| 268KB|1509| "Martin J. Bligh" <mbligh@aracnet.com>
  6]     36|   5781| 203KB|1359| Jeff Garzik <jgarzik@pobox.com>
  7]     29|   8334| 236KB|1052| Alan Cox <alan@lxorguk.ukuu.org.uk>
  8]     26|   4200| 106KB|1042| wli@holomorphy.com
  9]     23|   4752| 106KB|1134| Larry McVoy <lm@bitmover.com>
 10]     22|   5346| 114KB|0959| Ingo Molnar <mingo@elte.hu>

Top CC'ers
   | # msgs|av size| total|time| e-mail address
---+-------+-------+------+----+--------------------------------
  1]    752|   5396|3962KB|1242| <linux-kernel@vger.kernel.org>
  2]    115|   4061| 456KB|1334| torvalds@transmeta.com
  3]     56|   5549| 303KB|1249| Andrew Morton <akpm@digeo.com>
  4]     54|  14239| 750KB|1747| Alan Cox <alan@lxorguk.ukuu.org.uk>
  5]     39|   6608| 251KB|1102| "David S. Miller" <davem@redhat.com>
  6]     35|   9217| 315KB|1301| kuznet@ms2.inr.ac.ru
  7]     32|   5170| 161KB|1419| Chris Wedgwood <cw@f00f.org>
  8]     32|   4268| 133KB|0844| "Martin J. Bligh" <mbligh@aracnet.com>
  9]     26|   5165| 131KB|1459| Zwane Mwaikambo <zwane@holomorphy.com>
 10]     24|   8095| 189KB|1313| Jeff Garzik <jgarzik@pobox.com>

Top of top-level-domain
----------------------------------------------------------
 1]  com 668
 2]  org 200
 3]   de 118
 4]   uk 117
 5]  net 90
 6]   jp 55
 7]   au 49
 8]   cz 32
 9]   hu 30
10]  edu 27

Top organisations
----------------------------------------------------------
 1]   81
 2]   42 The Domain of Holomorphy
 3]   11 Working Overloaded Linux Kernel
 4]    8 none
 5]    8 Nuix
 6]    7 USAGI Project
 7]    7 ith Kommunikationstechnik GmbH
 8]    3 Open Source Devlopment Lab
 9]    3 My House
10]    3 daimi.au.dk

Top user-agents
----------------------------------------------------------
 1]  207 Mutt/1.4i
 2]   78 Mutt/1.3.28i
 3]   70 Mutt/1.2.5.1i
 4]   70 Mutt/1.5.3i
 5]   60 ELM [version 2.5 PL6]
 6]   55 Mulberry/2.2.1 (Linux/x86)
 7]   51 Mutt/1.3.25i
 8]   42 Sylpheed version 0.8.9 (GTK+ 1.2.10; i586-pc-linux-gnu)
 9]   42 Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI)
10]   41 Ximian Evolution 1.2.1 (1.2.1-4)

Messages per day
----------------------------------------------------------
   Sunday   257 ************************************
   Monday   133 *******************
  Tuesday   146 ********************
Wednesday   215 ******************************
 Thursday   300 *****************************************
   Friday   239 *********************************
 Saturday   136 *******************

Messages per Month
----------------------------------------------------------
Jan     0
Feb  1427 *************************************************
Mar     0
Apr     0
May     0
Jun     0
Jul     0
Aug     0
Sep     0
Oct     0
Nov     0
Dec     0

Messages per day-of-the-month
----------------------------------------------------------
 1     0
 2     0
 3     0
 4     0
 5     0
 6     0
 7     0
 8     0
 9     0
10     0
11     0
12     0
13     0
14     0
15     0
16     2 *
17     0
18   146 ************************
19   215 ***********************************
20   301 *************************************************
21   239 ***************************************
22   136 **********************
23   255 *****************************************
24   133 **********************
25     0
26     0
27     0
28     0
29     0
30     0
31     0

Messages per hour
----------------------------------------------------------
 1    45 *******************
 2    17 ********
 3    13 ******
 4    10 *****
 5     4 **
 6    12 ******
 7    21 *********
 8    41 ******************
 9    63 ***************************
10    83 ***********************************
11    72 *******************************
12    83 ***********************************
13    86 *************************************
14    93 ***************************************
15   100 ******************************************
16   101 *******************************************
17    84 ************************************
18   117 **************************************************
19    52 **********************
20    72 *******************************
21    84 ************************************
22    61 **************************
23    63 ***************************

Created with mboxstats; written by folkert@vanheusden.com

And it wouldn't hurt to aggregate versions. This is really just a top-6 list.

 1]  476 Mutt (most common versions versions)
 2]   60 ELM [version 2.5 PL6]
 3]   55 Mulberry/2.2.1 (Linux/x86)
 4]   42 Sylpheed version 0.8.9 (GTK+ 1.2.10; i586-pc-linux-gnu)
 5]   42 Mew version 2.1 on Emacs 21.1 / Mule 5.0 (SAKAKI)
 6]   41 Ximian Evolution 1.2.1 (1.2.1-4)

Wonder where "unknown" would land if counted.

Problem is: what part is version-information and what is name? For Mutt 1.0, ELM 3.6 it's clear; it's the number part.

But what about:
FlokEdit peanutbutterrelease
FlokEdit RMDrelease

Hmm.. Nothing really fundamental here - various updates all over (architecture updates, networking, usb, acpi, bluetooth, the usual suspects).

The task structure reference counting seems to have broken alpha, Richard is still chasing that one down.

Compile statistics: 2.5.63

Note that gcc 3.2 was used for all of these statistics.

                               2.5.62               2.5.63
                       --------------------    -----------------
bzImage (defconfig)         18 warnings          15 warnings
                             0 errors             0 errors

bzImage (allmodconfig)      33 warnings          29 warnings
                             9 errors             9 errors

modules (allmodconfig)    2514 warnings        2426 warnings
                           105 errors           128 errors

Compile statistics have been for kernel releases from 2.5.46 to 2.5.63 at: www.osdl.org/archive/cherry/stability

I am also compiling nightly views of Linus' linux-2.5 bitkeeper tree. Results can be found at:

www.osdl.org/archive/cherry/stability/linus-tree/running.txt