Andrew Morton announced Linux 2.6.11-rc3-mm2, saying:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.11-rc3/2.6.11-rc3-mm2/
- Added the mlock and !SCHED_OTHER Linux Security Module for the audio guys. It seems that nothing else is going to come along and this is completely encapsulated.
Various other stuff. If anyone has a patch in here which they think should be in 2.6.11, please let me know. I'm intending to merge the following into 2.6.11:
alpha-add-missing-dma_mapping_error.patch
fix-compat-shmget-overflow.patch
fix-shmget-for-ppc64-s390-64-sparc64.patch
binfmt_elf-clearing-bss-may-fail.patch
qlogic-warning-fixes.patch
oprofile-exittext-referenced-in-inittext.patch
force-read-implies-exec-for-all-32bit-processes-in-x86-64.patch
oprofile-arm-xscale1-pmu-support-fix.patch
Regarding the security module patches, Christoph Hellwig said, Even if we accept a module that grants capabilities to
groups this isn't fine yet because it only supports two specific capabilities
(and even those two in different ways!) instead of adding generic support
to bind capabilities to groups.
Olaf Dietsche remarked:
Unless I misunderstood the code, this one is available for quite some time: <http://www.olafdietsche.de/linux/accessfs/> or a newer, self-contained version <http://lkml.org/lkml/2005/1/11/221>
Or you could use a real solution - filesystem capabilities: <http://www.olafdietsche.de/linux/capability/> and if you don't like this one :-), there's also an alternative existing here: <http://www.stanford.edu/~luto/linux-fscap/>
Andrew also said to Christoph, I'm sure
that got discussed somewhere in the 1000 emails which flew past last time.
Jack?
And Jack O'Quin replied:
Most people felt that a more general capabilities module would be nice to have. But, no one offered any code, or volunteered to work on it.
I have no objection to that approach, but am not willing or able to do it myself. My opinion is that expanding the scope of the LSM would significantly increase its security risk. That job needs to be done very carefully, by someone with a deep understanding of the kernel's internal use of capabilities.
Perhaps, Christoph's suggestion could become part of a more general module, which might replace the RT-LSM in the 2.8 timeframe. Our LSM is a modest solution aimed at solving the immediate needs of audio developers and users with minimal impact on kernel security or correctness.
Matt Mackall asked what happened to the RT rlimit code from Chris Wright,
and Chris replied, I still have it, but I had the
impression Ingo didn't like it as a long term solution/hack (albeit small) to
the scheduler. Whereas the rt-lsm patch is wholly self-contained.
It turned out there were many opinions about the various
implementations. Chris was right that his rlimit patch was considered too
invasive. Actually, the main reason Andrew chose the RT LSM solution was
because none of the alternatives were self-contained enough. And although,
as Matt pointed out, Chris' patches were more straightforward than the other
althernatives considered; there was still the problem that, as Ingo Molnar
pointed out, it didnt solve the problem (unprivileged
user can lock up the system) in any way. So after it became visible that all
the existing 'dont allow users to lock up' solutions are too invasive, we went
to recommend the solution that introduces the least architectural problems:
RT-LSM.
Elsewhere in the thread, Ingo made another point about Chris'
rlimit patch. He said, if it turns out to be a
mistake then it's already codified into the ABI, while RT-LSM is much less
'persistent' and could be replaced much easier.
Christoph and Matt,
however, were very dubious about the possibility of removing RT-LSM later;
they said once RT-LSM was in the kernel, people would begin to rely on it,
and removing it would not be just a simple technical issue. Ingo replied:
i somewhat share that view. (despite all the promises from the audio folks - if they are just half as agressive resisting removal as they were pushing integration then it will never be removed ;-)
but i'm not sure how rlimits will contain the whole problem - can rlimits be restricted to a single app (jackd)? The most canonical use of rlimits is per-user (per-group), so the rlimit could end up _widening_ the effects of the hack ...
Matt confirmed that rlimits could indeed be restricted to a single application. Ingo asked how this could be accomplished, and Matt replied that the application had to be invoked by a setuid launcher; it was not possible to use rlimits to elevate the rlimits of a running process. Ingo argued that if there was already a requirement for a setuid application launcher, there was no need for anything else; the launcher itself could give the needed privileges. Matt argued that the rlimits patch would do a lot more than just this; but the discussion ended abrubtly, with no resolution.
Greg KH said:
I'd like to announce, yet-another-hotplug based userspace project: hotplug-ng. This collection of code replaces the existing linux-hotplug package with very tiny, compiled executable programs, instead of the existing bash scripts.
It currently provides the following:
- a /sbin/hotplug multiplexer. Works identical to the existing bash /sbin/hotplug.
- autoload programs for usb, scsi, and pci modules. These programs determine what module needs to be loaded when the kernel emits a hotplug event for these types of devices. This works just like the existing linux-hotplug scripts, with a few exceptions.
But why redo this all in .c code? What's wrong with shell scripts? Nothing is wrong with shell scripts, unless you don't want to have an interpreter in your initramfs/initrd and you want to provide /sbin/hotplug and autoload module functionality. Or if you have a huge box that spawns a zillion hotplug events all at once, and you need to be able to handle all of that with the minimum amount of processing time and memory.
So, how small are these programs? Take a look:
text data bss dec hex filename 4669 32 124 4825 12d9 hotplug 5077 8 348 5433 1539 module_pci 4925 8 412 5345 14e1 module_scsi 5349 8 348 5705 1649 module_usbThose are all static binaries, linked with klibc (which is included in the hotplug-ng package, just like udev.)
This compares to the following bash scripts:
-rwxr-xr-x 1 root root 4412 Feb 10 15:28 /sbin/hotplug -rw-r--r-- 1 root root 702 Sep 24 08:04 /etc/hotplug/blacklist -rw-r--r-- 1 root root 5293 Sep 24 08:04 /etc/hotplug/hotplug.functions -rwxr-xr-x 1 root root 3739 Sep 24 08:04 /etc/hotplug/pci.agent -rwxr-xr-x 1 root root 1459 Sep 24 08:04 /etc/hotplug/scsi.agent -rwxr-xr-x 1 root root 13466 Sep 24 08:04 /etc/hotplug/usb.agent -rw-r--r-- 1 root root 39306 Sep 24 08:04 /etc/hotplug/usb.distmap -rw-r--r-- 1 root root 4364 Sep 24 08:04 /etc/hotplug/usb.handmap -rw-r--r-- 1 root root 189 Sep 24 08:04 /etc/hotplug/usb.usermapAll of which are loaded into memory for each hotplug event (for specific hotplug events, only that bus type of file is loaded.)
But what about speed? With a completely unscientific measurement on my old, slow laptop, it takes about 2 seconds from the time I plug a usb device into the machine, for the proper module to be loaded. With the hotplug-ng program, it takes less than a second.
And for those of you who might remember the old dietHotplug program that also did the same thing in a tiny amount of space, this project obsoletes that one. dietHotplug had to be rebuilt for every kernel that was used on the system, hotplug-ng uses the ability for modprobe to determine the module that needs to be loaded based on the module aliases (modprobe as it currently works stops loading modules when it finds an alias that matches. This does not work for drivers that claim to support "all devices" and then later on fail on devices that they really don't support. For that, all matching drivers need to be loaded for the system to work properly. The linux-hotplug scripts handle this correctly, so if you rely on this functionality, please stick with that package for now. I'll be modifying modprobe to add this feature in the near future.).
The code can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/hotplug-ng-001.tar.gz
for those who wish to poke around in it.I still have a few more programs to write to get it up to the same functionality as the existing hotplug scripts (firmware, ieee1392, etc.) but those will be done soon. I'd like to get people's comments on the idea, and welcome suggestions and even patches :)
hotplug-ng development is done in a BitKeeper repository located at:
bk://linuxusb.bkbits.net/hotplug-ng
Patrick McFarland replied:
Wow, thats pretty awesome. Just the other day I said, "Why is hotplug written in sh? Isn't that horribly inefficient way of handling something that needs to be done quickly using the least amount of resources possible?" It seems you were reading my mind.
Please, continue this project and encourage distros to switch to it (when it exceeds hotplug in functionality and stability). Ubuntu currently is trying to reduce boot time, and I bet something like this would factor in (even a few seconds helps).
Greg was grateful for the praise, and he did confirm that he'd been
talking to the Ubuntu folks; but he didn't think his code would really have
an impact on boot-time. Vojtech Pavlik, however, said, Hotplug scripts were identified as one of the major culprits of slow
boot when we did the analysis for SuSE 9.2. They took 40+ seconds from the
total boot time.
At this point the debate veered off, into which distributions put in how much effort to reduce boot time.
Larry McVoy said:
This is a heads up that the BK tree for the kernel is currently at 59,000 changesets give or take a few. The BK that you are using uses unsigned shorts for the internal names of each delta which means you folks are about 100 days away from things no longer working.
The good news is that the openlogging tree for the kernel has 135,000 changesets so we've obviously long since fixed this problem.
The bad news is that you will need to upgrade your BK binary in order to pass over the 64K changeset boundary. The data is stored on disk in ascii so it doesn't matter if you upgrade until you hit the problem but sooner or later you will need to upgrade.
We'll get the fix into bk-3.2.4 which should be out by the end of the month. When we release that we'll send out notice and it would be good if people gave it a try and let us know if they hit issues because in a couple of months everyone is going to have to upgrade.
It's possible that we'll be changing the BK license to conform more with our commercial license but we won't do that without running it by Linus & Co to make sure that it's acceptable. One change we'd like to make there is to clarify the non-compete stuff. We've had some people who have indicated that they believed that if they used BK they were agreeing that they would never work on another SCM system. We can see how it is possible that people would interpret the license that way but that wasn't our intent. What we would like to do is change the language to say that if you use BK you are agreeing that you won't work on another SCM for 1 year after you stop using BK. But after that you would be able to hack on anything that you wanted. That was more of what we had in mind in the first place but we didn't make it clear. If you all thought that using BK meant you had no right to hack on SCM ever again, that's just not fair. We need to protect our IP but you should have the right to choose to go hack SCM if that's what you (our first choice is that you came and worked here, we really like kernel hackers, but if you don't want to that's cool too).
Various folks protested that it was absurd to have a license that
prohibited working on a competing project for a year; and some folks said it
was unenforceable, or that it had no standing in many parts of the world. A big
discussion started up, in which some folks accused Larry of misrepresenting his
objectives; while others pleaded with him to use a more sensible license. At
one point Alan Cox remarked, The real fix is to replace
BK with something free and better, but thats *not* a trivial task.
Itsuro Oda said:
Attached is an implementation of /proc/cpumem. /proc/cpumem shows the valid physical memory ranges.
- i386 and x86_64
- implement valid_phys_addr_range() and use it. (the first argument of the i386 version is little uncomfortable.)
- /dev/mem of the i386 version should be mofified. but not yet.
example: amd64 8GB Mem
# cat /proc/cpumem 0000000000000000 000000000009b800 0000000000100000 00000000fbe70000 0000000100000000 0000000100000000 #start address and size. hex digit.
Eric W. Biederman replied, Interesting.
My imagination when I proposed this was something based on struct resource
that works like /proc/iomem on x86 but can be meaningfully be used on systems
where ram lives in a separate address space from io device memory.
He still felt that adding a type field to /proc/cpumem would be a more
appropriate solution, though he didn't feel it would be necessary for x86
machines, as the information was available elsewhere.
Erik van Konijnenburg said:
OK, time to stop polishing and start publishing.
This is to announce yaird, Yet Another mkInitRD, a rewrite of mkinitrd based on hotplug algorithms.
MOTIVATION
Why a rewrite? The versions of mkinitrd that I studied, Debian sid and Fedora FC3, have some problems: they capture a lot of knowledge about the boot process, but don't always understand when a new kernel uses a different module name than the old kernel, may rely on modules compiled into the kernel, and don't always catch errors at the earliest opportunity.
Assumption: there are three issues that cause most of these problems.
- Backward compatibility: sysfs provides a wealth of information about hardware, but if you have to support 2.4 kernels, you have to limit yourself in the use you can make of that information.
- Originality: hotplug does a pretty good job of finding the appropriate modules for a device, basically because it closely follows the algorithms the kernel uses for matching hardware with modules. Deviating from those algorithms is unlikely to produce a more robust result.
- Shell scripting: beyond a certain level of complexity, the shell syntax makes it difficult to focus on the data structures you're trying to process and makes it difficult to do rigorous error checking.
Yaird is intended to find out whether that assumption is correct: if so, a program to build initrd images will be more reliable if it's written in perl, if it uses sysfs to determine what hardware needs to be supported, and if it closely follows the methods hotplug uses find the modules needed to support some hardware.
STATUS
There is working code: yaird booted the machine this note is written on. Code is available online, there also is a paper that discusses the workings of the application in detail.
http://www.xs4all.nl/~ekonijn/yaird/
So far, the program works correctly on every machine it's been tested on, but with only two test boxes that does not mean much.
Basic creature comforts are in place: "configure; make" but no RPM or deb files, a README and help option but no manual page.
Features:
- understands SATA and IDE. USB sticks and SCSI should work, but are untested.
- understands MDADM and LVM2, activates only necessary devices, understands stacks like LVM on top of stripe on top of mirror.
- handles both initrd and initramfs.
- understands USB keyboards.
- understands hotplug blacklist.
- knows that a module compiled into the kernel does not need insmod.
- understands /etc/fstab, including niceties such as labels, uuids and octal escapes.
- image generation understands symbolic links and shared libraries. (should support 32bit emulation on 64bit kernels; untested).
- templating system to simplify tuning the initrd image to the distribution.
- avoids hard-coded device numbers; does not require devfs.
- testing mode gives detailed overview of the systems hardware and the modules needed to support that.
There are rough edges in practically every feature: this is suitable for testing, but not for production use.
TODO
- Testing so far is 100% successful, but with just two boxes to play with, that's not saying much. If you can find space to test the code on your system, your results are highly appreciated. At this point, hardware testing is most valuable: I already know that dm-crypt is unsupported for now, but whether this stuff can boot a powerbook, sparc, or just about anything else is an open question.
- Feedback. This may be an interesting idea, but how does it relate to other new stuff floating about? In particular, what about the work that's going on putting udev on initramfs: are these approaches complementary or alternatives? Different perspectives on where this stuff fits in would help.
- Support more configurations. dm-crypt is unsupported for now, and so are multipath, swsusp, EVMS, NFS and loopback mounts. Implementing this stuff becomes interesting once there are some tests results that the basic ideas work in practice.
Jeff Garzik replied:
Having a mkinitrd that's not a shell script is a godsend. I would endorse yaird on that fact alone :)
I've long wanted a "mkinitfoo" that would create .cpio.gz for initramfs by default. So, good job there, too.
Eventually your script will need pull in, into the initramfs, klibc and programs linked against klibc.
Lorenzo Hernández García-Hierro said:
I've been working in implementing extended attributes support in the JFFS2 filesystem.
During the (short) time I worked on it, I just decided to try to bring back the thread on SELinux in hand-held and embedded devices and see if there's someone interested in contributing to it and collaborating to make something out of it.
The current work is just a draft, I've started with the standard Vanilla kernel sources plus mtd JFFS2 sources, used to patch the vanilla ones for latest code (I'm confused on which one has the most updated tree or if there are special differences between mtd's trees and vanilla's), and implemented the skeleton using the reiserfs xattr code base.
Then Russell Coker commented to me that I should use the handhelds.org kernel, so, I have the doubt on which one use, even if porting the changes made to JFFS2 code base to vanilla sources wouldn't be difficult, at least I suppose.
I've opened a few wiki pages for discussion and documenting until it gets further developed:
http://wiki.tuxedo-es.org/tuxedo/JFFS2xattr
http://wiki.tuxedo-es.org/tuxedo/SELinuxEmbeddedIn addition, having someone experienced with xattr API could be great, as development documentation seems inexistent, among James Morris' merged xattr consolidation code.
Josh Boyer suggested talking to the JFFS2 developers; he said xattr support would probably be added to JFFS3. After an IRC discussion between the two of them, Lorenzo migrated his code to JFFS3, and posted a new patch.
Marcelo Tosatti announced Linux 2.4.30-pre2, saying:
Here goes the second pre of v2.4.30.
It contains a bunch of important networking fixes, most noticeably the brlocks rework.
Plus USB fixes, megaraid2 driver update, JFS update, amongst others.
Erik van Konijnenburg said:
Version 0.0.4 of yaird is now available at:
http://www.xs4all.nl/~ekonijn/yaird/yaird-0.0.4.tar.gz
Yaird is a perl rewrite of mkinitrd. It aims to reliably identify the necessary modules by using the same algorithms as hotplug, and comes with a template system to to tune the tool for different distributions and experiment with different image layouts. It requires a 2.6 kernel with hotplug. There is a paper discussing it at:
http://www.xs4all.nl/~ekonijn/yaird/yaird.html
There are rough edges in practically every feature, and numerous features still need to be added: this is suitable for testing, but not for production use.
Thanks to all who gave feedback, sent patches and were brave enough to test this stuff. Below are highlights from the change log and todo list.
Summary of user visible changes for version 0.0.4:
- Process kernel command line options: init=, ro, rw.
- Boot into single user mode supported
- Support modules outside /lib/modules
- Support modules with extension other than .ko
- Warn about duplicates in modules.dep
- Generated image now waits for device to become visible in /sys, and gives error message if it doesn't
- Support 2.6.10 sysfs layout: SCSI now has a new subdirectory 'target'. (Thanks to Harald Dunkel for testing)
- Warn about unrecognised paths in /sys
- Empty lines in /etc/fstab are valid. (Patch Goffredo Baroncelli)
On top of the todo list are now:
- add command line option (--root=/dev/hdb) to simplify testing.
- add tree copying to the templates, to allow all of firmware to be copied to the image. Or all of /lib/modules, if you want to have hotplug on the image.
- get klibc run_init working. Test by switching the Debian template to initramfs. This should make Debian and Fedora templates more similar, it is also groundwork for possible hotplug-ng support.
- any patches you may wish to send :)