ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.11-rc1/2.6.11-rc1-mm1/

• Added bk-xfs to the -mm "external trees" lineup.

• Added the Linux Trace Toolkit (and hence relayfs). Mainly because I haven't yet taken as close a look at LTT as I should have. Probably neither have you.

  It needs a bit of work on the kernel<->user periphery, which is not a big deal.

  As does relayfs, IMO. It seems to need some regularised way in which a userspace relayfs client can tell relayfs what file(s) to use. LTT is currently using some ghastly stick-a-pathname-in-/proc thing. Relayfs should provide this service.

  relayfs needs a closer look too. A lot of advanced instrumentation projects seem to require it, but none of them have been merged. Lots of people say "use netlink instead" and lots of other people say "err, we think relayfs is better". This is a discussion which needs to be had.

• The 2.6.10-mm3 announcement was munched by the vger filters, sorry. One of the uml patches had an inopportune substring in its name (oh pee tee hyphen oh you tee). Nice trick if you meant it ;)
• Big update to the ext3 extended attribute support. agruen, tridge and sct have been cooking this up for a while. samba4 proved to be a good stress test.
• davej's "2.6 post-Halloween features" document has been added to -mm as Documentation/feature-list-2.6.txt in the hope that someone will review it and help keep it up-to-date.
• Added FUSE (filesystem in userspace) for people to play with. Am agnostic as to whether it should be merged (haven't read it at all closely yet, either), but I am impressed by the amount of care which has obviously gone into it. Opinions sought.

As a long time user of KDE's kio-slaves, I was always missing the kio-slave functionality on the command line and in non-kde programs. FUSE provides a kio-slave interface, but hopefully the inclusion of FUSE in the mm-kernel will cause more "fuse native" filesystems to come out which provide the functionality of the various kio-slaves.

Some things I'd like to see (as I am currently using the KIO equivalent) implemented as FUSE fs:

• "fish", virtual file access over ssh
• "audiocd", virtual audio cd filesystem which copies and encodes audio tracks on the fly
• "ftp", virtual file system ftp server access
• etc..

Imagination is the limit, and since it can be implemented in userspace pretty easily with FUSE, I am looking forward to see what people can come up with and hope that FUSE is here to stay.

relayfs and netlink are for completely problem spaces. relayfs is for relaying a lot of data quickly (e.g. for kernel instrumentation). There it fills a niche that printk doesn't fill (since it's too slow). netlink is quite slow (allocates data for each event, does lots of other gunk), but an useful extensible format for low frequency events.

For the problems that relayfs solves netlink is totally unusable due to low efficiency (you could as well use printk, but that is also to slow). I think a low overhead logging mechanism is very much needed, because I find myself reinventing it quite often when I need to debug some timing sensitive problem. Trying to tackle these with printk is hopeless because it changes timing too much.

The problem relayfs has IMHO is that it is too complicated. It seems to either suffer from a overfull specification or second system effect. There are lots of different options to do everything, instead of a nice simple fast path that does one thing efficiently. IMHO before merging it should go through a diet and only keep the paths that are actually needed and dropping a lot of the current baggage.

Preferably that would be only the fastest options (extremly simple per CPU buffer with inlined fast path that drop data on buffer overflow), with leaving out anything more complicated. My ideal is something like the old SGI ktrace which was an extremly simple mechanism to do lockless per CPU logging of binary data efficiently and reading that from a user daemon.

Granted tracing is not free, but please avoid spreading FUD without actually carrying out proper testing. We've done quite a large number of tests and we've demonstrated over and over that LTT, and ltt-over- relayfs, is actually very efficient. If you're interested in actual test data, then you may want to check out the following:

http://www.opersys.com/ftp/pub/LTT/Documentation/ltt-usenix.ps.gz

http://lwn.net/Articles/13870/

We are aware of the cost of the various tracing components, as you can see by my earlier posting about early-checking to minimize the cost of the tracing hooks for kernel compiled with them, and are open for any optimization. If you have any concrete suggestions, save the scrap-everything-I-know-better (which is really unproductive as you would anyway have to go down the same path we have), we are more than willing to entertain them.

loop-AES changes since previous release:

• Fixed externally compiled module version multi-key-v3 ioctl incompatibility with boxes running 64 bit kernel and 32 bit userland. Kernel patch versions were not affected (2.4 and 2.6 kernels).
• Fixed bug that made v3 on-disk format always use file backed code path on some 2.6 kernels that did not have LO_FLAGS_DO_BMAP defined. No data loss, but file backed code path is not journaled file system safe. Same bug also had cosmetic side effect of "losetup -a" status query always displaying file backed v2 on-disk format as v3 on-disk format.

bzip2 compressed tarball is here:

http://loop-aes.sourceforge.net/loop-AES/loop-AES-v3.0b.tar.bz2
md5sum b295ff982cd4503603b38fdc54e604cc

http://loop-aes.sourceforge.net/loop-AES/loop-AES-v3.0b.tar.bz2.sign

This is an update of the earlier version of plugsched (previously released by Con Kolivas) and has a considerably modified scheduler interface that is intended to reduce the amount of code duplication required when adding a new scheduler. It also contains a sysfs interface based on work submitted by Chris Han.

This version of plugsched contains 4 schedulers:

1. "ingosched" which is the standard active/expired array O(1) scheduler created by Ingo Molnar,
2. "staircase" which is Con Kolivas's version 10.5 O(1) staircase scheduler,
3. "spa_no_frills" which is a single priority array O(1) scheduler without any interactive response enhancements, etc., and
4. "zaphod" which is a single priority array O(1) scheduler with interactive response bonuses, throughput bonuses and a choice of priority based or entitlement based interpretation of "nice".

Schedulers 3 and 4 also offer unprivileged real time tasks and hard/soft per task CPU rate caps.

The required scheduler can be selected at boot time by supplying a string of the form "cpusched=<name>" where <name> is one of the names listed above.

The default scheduler (that will be used in the absence of a "cpusched" boot argument) can be configured at build time and is set to "ingosched" by default.

The file /proc/scheduler contains a string describing the current scheduler.

The directory /sys/cpusched/<current scheduler name>/ contains any scheduler configuration control files that may apply to the current scheduler.

Peter, thank you for maintaining Con's plugsched code in light of Linus' and Ingo's prior objections to this idea. On the one hand, I partially agree with Linus&Ingo's prior views that when there is only one scheduler that the rest of the world + dog will focus on making it better. On the other hand, having a clean framework that lets developers in a clean way plug in new schedulers is quite useful.

Linus & Ingo, it would be good to have an indepth discussion on this topic. I'd argue that the Linux kernel NEEDS a clean pluggable scheduling framework.

Let me make a case for this NEED by example. Ingo's scheduler belongs to the egalitarian regime of schedulers that do a poor job of isolating workloads from each other in multiprogrammed environments such as those found on Enterprise servers and in my case on PlanetLab (www.planet-lab.org) nodes. This has been rectified by HP-UX, Solaris, and AIX through the use of fair share schedulers that use O(1) schedulers within a share. Currently PlanetLab uses a CKRM modified version of Ingo's scheduler. Similarly, the linux-vserver project also modifies Ingo's scheduler to construct an entitlement based scheduling regime. These are not just variants of O(1) schedulers in the sense of Con's staircase O(1). Nor is it clear what the best type of scheduler is for these environments (i.e., HP-UX, Solaris and AIX don't have it fully solved yet either). The ability to dynamically swap out schedulers on a production system like PlanetLab would help in determining what type of scheduler is the most appropriate. This is because it is non-trivial, if not impossible, to recreate the multiprogrammed workloads that we see in a lab.

For these reasons, it would be useful for plugsched (or something like it) to make its way into the mainline kernel as a framework to plug in different schedulers. Alternatively, it would be useful to consider in what way Ingo's scheduler needs to support plugins such as the CKRM and Vserver types of changes.

Thats true. The current CKRM CPU scheduler is not a standalone component...if it were made one, it would need a non-CKRM interface to define classes, set their shares etc.

However, we have not investigated the possibility of making our CPU scheduler a pluggable one that could be loaded into a kernel equipped with the plugsched patches AND the CKRM framework. This should be possible but not a high priority until there is more consensus for having CPU schedulers pluggable at all (we have more basic stuff to fix in our scheduler such as load balancing).

Of course, we're more than happpy to work with someone willing to chip in and make our scheduler pluggable.

Another issue (than indirection) that I think needs to be addressed at some stage is freeing up the memory occupied by the code of the schedulers that were unlucky not to be picked. Something like what __init offers only more selective.

And the option of allowing more than one CPU per run queue is another direction that needs addressing. This could allow a better balance between the good scheduling fairness that is obtained by using a single run queue with the better scalability obtained by using separate run queues.

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.11-rc1/2.6.11-rc1-mm2/

• There are a bunch of ioctl() and compat_ioctl() changes in here which seem to be of dubious maturity. Could people involved in this area please review, test and let me know?
• A revamp of the kexec and crashdump patches. Anyone who is interested in this work, please help to get this ball rolling a little faster?
• This kernel isn't particularly well-tested, sorry. I've been a bit tied up with other stuff.

I've reworked the relayfs patch extensively. The API and internals have been heavily purged. The patch is in fact almost HALF! its original size, loosing 90KB and going from 200KB to 110KB.

Here's a summary of changes:

• Dropped the lockless scheme (complaints from Christoph, Arjan, Ingo, Andrew, etc. regarding the debatable efficiency of cmpxchng over traditional locks.)
• Dropped the resizing capabilities (Andi, Christoph, etc.)
• Dropped all user-space read/write capabilities (Andi, Christoph, Roman, Tim, etc.) Basically this is about making relayfs do one thing and do it good: transfer high volumes of data to user-space as efficiently as possible.
• Implemented the ad-hoc mode. This mode contrasts with the managed mode (old locking scheme) in that it manages only one ring buffer, doesn't care about outside synchronization, and is basically the most basic buffer scheme one could have. (ongoing discussion with Roman on this issue.)
• Dropped the klog client (Greg).
• Modified API to use pointers instead of IDs (Roman).
• Miscallaneous cleanups (Domen).
• Miscallaneous rearrangement of code (me).

I've tested this with a hacked ltt patch and I can get it to collect data in the managed mode without a problem. Reading the data though is another story, I'll update the LTT patch once I know where the relayfs stuff is heading. Beware: don't try to use the ltt code in Andrew's tree with this relayfs, it's a completely different beast.

So without further ado, here's the code. I've removed the Documentation/filesystems/relayfs.txt file for now (no, don't worry that's not part of the 90KB that went away ;). It's probably going to need some rewriting before we're done, so let's not get distracted by it for now.

Karim

The full patch is here:
http://www.opersys.com/ftp/pub/relayfs/patch-relayfs-redux-2.6.10-050120-real
http://www.opersys.com/ftp/pub/relayfs/patch-relayfs-redux-2.6.10-050120-real.bz2

Hm, how about this idea for cutting about 500 more lines from the code:

Why not drop the "fs" part of relayfs and just make the code a set of struct file_operations. That way you could have "relayfs-like" files in any ram based file system that is being used. Then, a user could use these fops and assorted interface to create debugfs or even procfs files using this type of interface.

As relayfs really is almost the same (conceptually wise) as debugfs as far as concept of what kinds of files will be in there (nothing anyone would ever rely on for normal operations, but for debugging only) this keeps users and developers from having to spread their debugging and instrumenting files from accross two different file systems.

However this assumes that the users of relayfs are not going to want it during normal system operation. This is an assumption that fails with at least LTT as it is targeted at sysadmins, application developers and power users who need to be able to trace their systems at any time.

I don't mind piggy-backing off another fs, if it makes sense, but unlike debugfs, relayfs is meant for general use, and all files in there are of the same type: relay channels for dumping huge amounts of data to user-space. It seems to me the target audience and basic idea (relay channels only in the fs) are different, but let me know if there's a compeling argument for doing this in another way without making it too confusing for users of those special "files" (IOW, when this starts being used in distros, it'll be more straightforward for users to understand if all files in a mounted fs behave a certain way than if they have certain "odd" files in certain directories, even if it's /proc.)

There is a user requirement for overcommit mode, you know.

Saying "hey, there's no more overcommit mode in future v2.6 releases, you run out of memory and get -ENOMEM" is not really an option is it?

You propose to remove the OOM killer and do what? Lockup solid?

It is _WAY_ off right now: look at the amount of free pages:

DMA free:4536kB min:60kB low:72kB high:88kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0
Normal free:524648kB min:4028kB low:5032kB high:6040kB active:76508kB inactive:81760kB present:1031360kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0
HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0
DMA: 556*4kB 155*8kB 65*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4536kB
Normal: 29800*4kB 25115*8kB 6953*16kB 1251*32kB 326*64kB 103*128kB 31*256kB 12*512kB 3*1024kB 1*2048kB 0*4096kB = 524648kB
HighMem: empty

v2.4 gets it pretty much right for most cases, and its obviously screwed up right now in v2.6.

Andrea/Thomas were working on getting it fixed ??

Right now we have three overcommit modes. They are specified by:

0: overcommit, but keep it reasonable (the current default)
1: overcommit, always say yes
2: keep track of all our obligations, do not overcommit

So, one has the right to expect that no OOM situation can occur in overcommit mode 2. But in 2.6.10 it can. That is a bug. The conclusion must be that bookkeeping is done incorrectly. Perhaps also mode 0 is affected by that same bug.

Now you ask what I propose. There is no hurry worrying about that - the first thing should be to fix the bookkeeping problem.

But assume that fixed. Then everybody can run in mode 2 and never have any problems. That is what I do.

Yes, you say, but that is an inefficient use of memory. Perhaps. That is the price I am willing to pay for the guarantee that my processes are not killed at some random moment.

But if someone else does not do anything of importance and doesnt care if his processes die at arbitrary moments if only things go as fast as possible and use as much of his precious memory as possible, then also for him overcommit mode 2 can be useful. It is accompanied by the variable overcommit_ratio R - the amount of memory that can be used is Swap + Memory*(R/100). Here R can be larger than 100, so in overcommit mode 2 one can specify very precisely what amount of overcommitment is considered acceptable.

Very few people run overcommit mode 2, and lots of things are badly tested. It cannot become the default today. But I would like to see it the default at some future moment.

I'm working on fixing it, not just tuning it. The bugs in mainline aren't about the selection algorithm (which is normally what people calls oom killer). The bugs in mainline are about being able to kill a task reliably, regardless of which task we pick, and every linux kernel out there has always killed some task when it was oom. So the bugs are just obvious regressions of 2.6 if compared to 2.4.

But this is all fixed now, I'm starting sending the first patches to Andrew very shortly (last week there was still the oracle stuff going on). Now I can fix the rejects.

I will guarantee nothing about which task will be picked (that's the old code at works, I changed not a bit in what normally people calls "the oom killer", plus the recent improvement from Thomas), but I guarantee the VM won't kill tasks right and left like it does now (i.e. by invoking the oom killer multiple times).

After cleaning up a bit df suddenly showed interesting results:

Filesystem            Size  Used Avail Use% Mounted on
/dev/md4             1019M  -64Z  1.1G 101% /tmp

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md4               1043168 -73786976294838127736   1068904 101% /tmp

This is on a ext3 filesystem on a 2.6.10-ac10 kernel.

The Used column is total-free, so free was 2^66 + 964440. That 2^66 no doubt was 2^64 in a computation counting 4K-blocks, and arose at some point where a negative number was considered unsigned.

But having available=1068904 larger than free=964440 is strange.

I assume this was produced by statfs or statfs64 or so. You can check using "strace -e statfs64 df /dev/md4" that these really are the values returned by the kernel, so that we can partition the blame between df and the kernel.

The values are computed by

buf->f_blocks = es->s_blocks_count - overhead;
buf->f_bfree = ext3_count_free_blocks (sb);
buf->f_bavail = buf->f_bfree - es->s_r_blocks_count;

that is: blocks = total - overhead, and available = free - reserved. strace shows three values, and I expect tune2fs or so will show 2 more.

More available than free sounds like a negative count of reserved blocks. Are you still able to examine the situation?

I do have some more information. A e2fsck run on that filesystem was just as interesting:

/dev/md4: clean, 16/132480 files, -15514/264960 blocks

Forcing an e2fsck revelated a few groups with incorrect block counts:

Free blocks count wrong for group #2 (34308, counted=32306).
Free blocks count wrong for group #6 (45805, counted=32306).
Free blocks count wrong for group #8 (14741, counted=2354).
Free blocks count wrong (280474, counted=252586).

After fixing those everything returned to normal. I did run dumpe2fs on the filesystem, if that is interesting I can retrieve and post that.

Because the store method doesn't have an offset argument, it must assume that all writes are based from the beginning of the buffer.

So if your sysfs file contains "123" and you do

echo "45" >> mysysfsfile

instead of the expected "12345", you end up with "45" in the file with no errors. Opening the file, seeking, and writing gives the same type of behavior, with no errors.

This patch just sets a few flags to make sure that errors are returned when this behavior is seen. Logically then, the two "features" do the same thing (set flags), and prevent the same behavior (writing wrong contents without error).

Here is a new release of my time of day proposal, which include ppc64 support as well as suspend/resume and cpufreq hooks. For basic summary of my ideas, you can follow this link: http://lwn.net/Articles/100665/

This patch implements the architecture independent portion of the time of day subsystem. Included is timeofday.c (which includes all the time of day management and accessor functions), ntp.c (which includes the ntp scaling code, leapsecond processing, and ntp kernel state machine code), timesource.c (for timesource specific management functions), interface definition .h files, the example jiffies timesource (lowest common denominator time source, mainly for use as example code) and minimal hooks into arch independent code.

The patch does not function without minimal architecture specific hooks (i386, x86-64, and ppc64 examples to follow), and it can be applied to a tree without affecting the code.

New in this version:

• TIMESOURCE_CYCLES type
• __iomem labes and proper mmio access
• write_seqlock_irq -> writeseqlock_irqsave
• arch generic interface for for get_cmos_time() equivalents
• suspend/resume hooks for sleep/hibernate (lightly tested)
• timesource adjust_callback hook

SoftwareSuspend 2.1.5.7B for the 2.4.28 kernel is now available from softwaresuspend.berlios.de.

Bug fixes and forward ports to 2.4.29 and later kernels notwithstanding, it is intended to be the last release of SoftwareSuspend for the 2.4 series kernels.

The 2.4 version of Suspend is generally pretty easily to get going, but if you have any questions or problems, you will find lots of resources at softwaresuspend.berlios.de. In particular, there are HOWTOs, FAQs, and a Wiki that you can consult before asking on the mailing lists you'll also find there.

Fuller instructions regarding applying the package can be found in the README file, included in the package.