Kernel Traffic #299 For 6 Mar 2005

Version 0.1.5 of the Intel Sofware RAID driver (iswraid) is now available for the 2.4 series kernels at http://prdownloads.sourceforge.net/iswraid/2.4.29-iswraid.patch.gz?download

It is an ataraid "subdriver" but uses the SCSI subsystem to find the RAID member disks. It depends on the libata library, particularly on either the ata_piix or the ahci driver, that enable the Serial ATA capabilities in ICH5/ICH6/ICH7 chipsets. More information is available at the project's home page at http://iswraid.sourceforge.net/.

Driver documentation is included in Documentation/iswraid.txt, which is part of the patch. The license is GPL.

The changes WRT version 0.1.4.3 are the following:

• Resource deallocation bug fixed for failed initializations.
• Read IO resubmission to mirror bug fixed.
• RAID1E (covers 4-disk RAID10) code added.
• More aggressive marking disks as bad in metadata.
• Claiming disks for RAID "feature" removed.
• Option defaults now customizable from the build configuration.
• iswraid_never_fail "feature" watered down into iswraid_resist_failing.
• iswraid_halt_degraded now prevents degraded volumes from being registered.
• Debug printouts more customizable.
• Some code cleanup and optimization.
• Documentation changes.

Please consider this driver for inclusion in the 2.4 kernel tree.

It sorts sucks for users with that hardware. The typical complaint comes from trying to share data between Windows and Linux, where "just use md" isn't a solution.

Without device mapper (another new feature) to enable dmraid, these users are just sorta S.O.L.

I consider it not a new feature, but a missing feature, since otherwise user data cannot be accessed in the RAID setups.

I personally dislike and discourage the addition of ANY new drivers to v2.4 at this point, and I sincerely appreciate every argument against iswraid, but I have no problems with it because it looks like a valid special case since it allows users to access their ICH5/6 RAID partitions, as Jeff mentions.

Moreover the driver is going to die with v2.4 anyway, its not like any future compatibility problem is being introduced.

So I understand the argument against having it in the tree: the elegant way of doing it is to use dmraid.

But I dont buy it as an argument against merging it in a dying v2.4.x tree which purpose is to serve existing users.

You are mistaken in arguing that "oh, since this driver can be merged, its likely that any v2.6 HW support/driver will be accepted in v2.4".

So, its up to Jeff, and he seems to be OK with it.

I've played lately a bit with Subversion and used it for managing the kernel sources, using Larry McVoy's bk2cvs bridge and Ben Collins' bkcvs2svn conversion script.

Since there is little information on the web on how to properly set up a SVN repository and use it for tracking the latest kernel tree, I wrote a small howto (modeled after the bk kernel howto) in case it can be useful for other people too.

Feel free to comment on it (but let's not start a new BK flamewar or SVN bashing session please). If there is enough interest I'll submit a patch to include this in the kernel Documentation/ directory.

I've put it also on my web page along with the necessary scripts: http://popies.net/svn-kernel/

This is a release of Preempt Real-time for ARM . It includes everything up to CONFIG_PREEMPT_RT , and all of the latency tracing except interrupts off timing. The timing also excludes syscalls. This patch includes only a port to OMAP boards. However, it should be straight forward to get it working on other boards.

The biggest point of discussion relates to the interrupts in threads implementation. It is largely identical to what is implemented in the generic irq handling. However, ARM doesn't not implement generic irq handling, and will not support it in the near future. I am not in support of two different threaded interrupt implementations.

I recently made a proposal to separate the threaded interrupt handling from the generic irq handling, but I'm open to other ideas.

Well, I remain unconvinced about the generic irq handling.

Back in 2.4 times, ARM used to use the x86 way to handle IRQs, and it caused lots of dropped IRQs for CF cards and the like, particularly in mixed level/edge triggered interrupt environments (where a mixture of level and edge based outputs are connected to edge triggered inputs.)

The ARM IRQ code got completely rewritten during 2.5 with a clean design, generated from the requirements of the machines.

This caused major changes throughout all the machine support files, and I'm _NOT_, repeat _NOT_ going to consider going back to some half baked approach which doesn't really fit the needs of the ARM architecture, just because "oh, it's generic." If it doesn't work reliably, I'm not interested.

This is especially so when it impacts so many machines in ways specific to each machine, and there's no way to get them tested in one go.

If this is to be done, doing it in the middle of a stable kernel series is NOT the time or place to do it. I have recently had people complaining about the "stability" of 2.6, particularly in relation to changes made by other people affecting drivers.

Consider these questions in relation to the generic IRQ code:

1. Does it know the difference between handling level, edge-based and "simple" IRQs? ("simple" IRQs are those which are cascaded, but don't have their own individual interrupt mask controls.)
2. Does the generic autoprobe code know which IRQs can be autoprobed and which can't? (cascade interrupts are just one example of interrupts which must not be autoprobed. There may be other reasons you wish to avoid probing other interrupts on a particular machine, which the machine support code knows about.)
3. Does the generic IRQ code know which IRQs can be claimed and which can't? (IRQs 0 to NR_IRQS aren't always claimable, even when they appear to be available - iow, desc->handler != &no_irq_type.)
4. Does it allow per "hw type" retriggering of interrupts, even if the hardware itself is not capable of such an action? (and running these interrupts at the next hardware interrupt?)
5. Does it allow control of interrupt wakeup sources?
6. Does it allow architectures to define their own irq_desc_t so that all the data for a particular IRQ is localised and contained within one data structure?

What you'll find is that the ARM interrupt structure is designed to efficiently meet the requirements of our wide range of hardware interrupt controllers, with chained interrupt controllers, with as low latency as possible.

In essence, I'm opposed to completely rewriting the ARM interrupt handling at this stage.

I was looking at some of the code we generate, and happened to notice that we have this strange situation where the x86 "get_user()" macros generate out-of-line code to do all the address verification etc, but the "put_user()" ones do not, and do everything inline.

I also noticed that (probably as a result of this), our "put_user()" on old i386 machines does not do the full magic manual page-following. Which means that copy-on-write doesn't necessarily work right due to the broken paging hw on the original 386 core.

I didn't fix the second part, but at least making things out-of-line makes it possible. And making "put_user()" be out-of-line seemed quite doable.

I no longer use x86 as my main machine, so this patch is totally untested. I've compiled it to see that things look somewhat sane, but that doesn't mean much. If I forgot some register or screwed something else up, this will result in a totally nonworking kernel, but I thought that maybe somebody else would be interested in looking at whether this (a) works, (b) migth even shrink the kernel and (c) might make us able to DTRT wrt the page table following crud (old i386 cores may be hard to find these days, so maybe people don't care).

I've written a driver for probably the most common touchscreen type - the serial Elo touchscreen.

The driver should handle all generations of serial Elos, as it handles Elo 10-byte, 6-byte, 4-byte and 3-byte protocols.

I sometimes feel that we should have a "generic" touch screen driver from looking at the code for the different brands.

Almost all touch screen data goes something like this:

• every packet is fixed size, N bytes long
• the "header" bytes can be described by <data> AND <mask> == something expected
• the X,Y,pressure,touch data can be described as "starting at bit B, length N, multiply by Z". An array of these tokens should be able to handle coordinates broken into several.

If this information could be passed as a module parameter, new touchscreens could be supported without any kernel modification.

We could parse a definition "string", like this:

"SIZE:10,SYNC:0:8:85,SYNC:8:8:54,X:24:8:1,X:32:8:256,Y:40:8:1,Y:48:8:256,T:16:2:1"

This string defines the touch driver for elotouch, 10 bytes packet (I didn't include the pressure reading, for simplification).

I currently have 6 different "drivers" that would all fit into this model. The same goes for all 3 elotouch protocols that you implemented.

Does this sound like a good idea?

Here's the latest relayfs patch, incorporating the previous round of suggestions. Thanks to everyone who sent comments. Here's a list of the major changes:

• replaced remap_page_range() and reserved bits with a nopage() handler.
• buffers are now allocated/freed as part of inode alloc/destroy.
• got rid of the automatic creation/teardown of directory hierarchies, and exported create/remove dir functions instead.
• replaced the fileop callback with explicit map/unmap callbacks - these were the only fileops currently being used by anything, so I got rid of the other cases.
• relay_write() and __relay_write() no longer return a value - it doesn't really make sense for clients to check a return value in the fast path anyway.
• added a buf_full() callback to notify users that a buffer has become full and therefore events might be lost.
• got rid of the 'helper' macros, which weren't helping much.
• various other bits of code and comment cleanup.

Also, there was some question as to whether or not the memcpy in relay_write() was being inlined properly - I looked at the generated assembly code, and it seems to be, but I'll be taking a closer look later.

This is what the API now looks like:

API functions:

rchan *relay_open(base_filename, parent, subbuf_size, n_subbufs, flags, callbacks);
void relay_close(chan);
dentry *relayfs_create_dir(name, parent);
int relayfs_remove_dir(dentry);
void relay_reset(chan);

void relay_write(chan, data, length);
void __relay_write(chan, data, length);
void *relay_reserve(chan, length);

void relay_subbufs_consumed(chan, subbufs_consumed, cpu);
void relay_commit(buf, subbuf_idx, count);

callbacks

int subbuf_start(buf, subbuf, prev_subbuf_idx);
int deliver(buffer, subbuf, subbuf_idx);
void buf_mapped(buf, filp);
void buf_unmapped(buf, filp);
void buf_full(buf);

As before, I've tested this code on a single proc machine using a hacked version of the kprobes network packet tracing module, which can be found here:

http://prdownloads.sourceforge.net/dprobes/plog.tar.gz?download

If people are more or less happy with the current version, I'll do some SMP testing and write some Documentation.

Here goes v2.4.30-pre1.

It contains, amongst others, a SATA update, series of networking bug fixes, and v2.6 hardening backports.

this is hopefully the last -rc kernel before the real 2.6.11, so please give it a whirl, and complain loudly about anything broken.

As can be seen from the shortlog, most of the changes are pretty trivial. I think the biggest change is the radeon updates, and some of the NLS codepage things caused big diffs even if the changes themselves are pretty trivial (oh, and moving the ia64 "shubio.h" file accounts for about seven thousand lines of diffs, but no real changes ;)

In short: some driver updates, some arm/uml/sparc updates, and various random (mostly) one-liners all over. The most noticeable of the one-liners is hopefully that raid5/6 should work again.

I have written an introduction to the Linux 2.6.8.1 CPU scheduler implementation. It should help people to understand what is going on in the scheduler code faster than they would be able to by just reading through the code. The paper can be downloaded in PDF or LyX form from here:

http://josh.trancesoftware.com/linux/

This paper will never be "done," as I'd like to keep improving it over time, and updating it to newer versions of the kernel as time allows. If you have comments, suggestions, or corrections you'd like to make, please email me. Technical corrections in particular would be appreciated. Hopefully this can be as accurate and helpful as possible, and will inspire more people to look into the Linux scheduler.

My employer, SGI, did not ask me to write this paper - it was done as part of a school project last semester. While SGI owns the copyright to the paper, they have allowed me to release it under the GNU FDL.

Here is the next release of the genetic library based against 2.6.10 kernel.

There were numerous changes from the first release, but the major change in this version is the introduction of phenotypes. A phenotype is a set of genes the affect an observable property. In genetic-library terms, it is a set of genes that will affect a particular fitness measurement. Each phenotype will have a set of children that contain genes that affect a fitness measure.

Now multiple fitness routines can be ran for each genetic library user. Then depending on the results of a particular fitness measure, the specific genes that directly affect that fitness measure can be modified. This introduces a finer granularity that was missing in the first release of the genetic-library.

I would like to thank Peter Williams for reworking the Zaphod Scheduler and help designing the phenotypes.

Some of the other features introduced is shifting the number of mutations depending on how well a phenotype is performing. If the current generation outperformed the previous generation, then the rate of mutation will go down. Conversely if the current generation performed worst then the previous generation, the mutation rate will go up. This mutation rate shift will do two things. When generations are improving, it will reduce the number unnecessary mutations and hone in on the optimal tunables. When a workload drastically changes, the fitness should go way down, and the mutation rate will increase in order to test a greater space of better values quicker. This should decrease the time it takes to adjust to a new workload. There is a limit at 45% of the genes being mutated every generation in order to prevent the mutation rate spiralling out of control.

SpecJBB and UnixBench are still yielding a 1-3% performance improvement, however (though it's subjective) the interactiveness has had noticeable improvements.

I have not broke the Anticipatory IO Scheduler down to a fine granularity in phenotypes yet. Any assistance would be greatly appreciated.

Currently I am hosting this project off of:

http://kernel.jakem.net

after a short discussion with Marcelo, we quickly agreed that a hotfix tree would be a good thing for kernel 2.4, since a few months can separate two stable releases. I offered to help in this area because I already have to pick random patches from the BK changesets anyway, so the only additional work will be to pack them in a more presentable way than what I can do just for me. Marcelo offered to help me by telling me when he thinks that a particular patch needs to be merged or excluded.

Overall, the patches included are classified in 6 categories:

• security : fixes for security bugs in general
• critical : fixes for problems leading to panic, or data corruption
• major : fixes for general system stability (oopses, memory leaks, ...)
• minor : fixes for erroneous behaviours in general.
• build : missing includes, Makefile bugs, etc... which prevents certain configurations from being built (here, we should often encounter Adrian Bunk's numerous patches :-))
• documentation : configure.help and friends. Will most probably be fed by Adrian's contributions too.

As much as possible, I will avoid changes in drivers because we all know that in this area, a fix for one user breaks another one.

Anyway, all of those patches will be extracted from -BK. I'm not building a parallel tree (I already have another one for that :-)). The main goal is that the diff between this kernel and the next release should be smaller than the diff between previous and next release.

Marcelo and I agreed on the "-hf" suffix (for "Hot Fix"). Two patches are systematically provided, one with the version in the Makefile, and one without. The goal is to ease both people using vanilla kernels in production (who need to check the real version) and people who want a virgin kernel base to apply external patches while limiting the number of rejects.

A tarball is also included with all the individual patches. A makefile relies on the "CONTENTS" file itself to rebuild the patch against the kernel of your choice. It is very easy to add/remove patches in the CONTENTS file, so I hope it will be convenient enough for people who don't want to include minor or documentation fixes for example. It is documented anyway.

I've started with 2.4.29 and released 2.4.29-hf2 a few days ago. In fact, I was waiting for the site to migrate from home to my company (EXOSEC) to benefit from more outgoing bandwidth, before sending this announce. Despite this, I will try to stay up to date within the shortest time, and release a new hotfix as soon as either something weird gets fixed, or Marcelo asks me to do so. In any case, I'll do my best to send an announce here for new releases.

The patches and tarballs are hosted here :

http://linux.exosec.net/kernel/2.4-hf/

I'm open to comments, advices, critics, etc, but flames such as "you lose your time" or "2.4 is dead" will feed /dev/null. One possible improvement I have already identified would be to publish the work in progress so that people could get fixes in real time, but if they really need so, they should download from BK instead.

If this branch gets enough demand, I will maintain several previous versions in parallel (eg: go back to 2.4.27).

In the mean time, please find here the CONTENTS file which details what patches have been included and basically what they do.