Kernel Traffic #239 For 1 Nov 2003

We're proud to announce the availability of a _proof of concept_ file system, called srfs. ( http://www.cs.bgu.ac.il/~srfs/ ). a quick overview: [from the home page]

srfs is a global file system designed to be distributed geographicly over multiple locations and provide a consistent, high available and durable infrastructure for information.

Started as a research project into file systems and self-stabilization in Ben Gurion University of the Negev Department of Computer Science, the project aims to integrate self-stabilization methods and algorithms into the file (and operation) systems to provide a system with a desired behavior in the presence of transient faults.

Based on layered self-stabilizing algorithms, provide a tree replication structure based on auto-discovery of servers using local and global IP multicasting. The tree structure is providing the command and timing infrastructure required for a distributed file system.

The project is basically divided into two components:

1. a kernel module, which provides the low level functionality, and disk management.
2. a user space caching daemon, which provide the stabilization and replication properties of the file system.

these two components communicate via a character device.

more info on the system architecture can be find on the web page, and here: http://www.cs.bgu.ac.il/~tzachar/srfs.pdf

We hope some will find this interesting enough to take for a test drive, and wont mind the latencies ( currently, the caching daemon is a bit slow. hopefully, we will improve it in the future. ) anyway, please keep in mind this is a very early version that only works, and keeps the stabilization properties. no posix compliance whatsoever...

Suppose I install srfs on both my laptop and my server. I then move the CVS repository for my pet project onto the new srfs filesystem and I take off for the weekend with my laptop. Over the weekend I commit several changes to file X. Over the weekend my friend also commits several changes to file X.

When I get home and plug in my laptop, presumably the caching daemon will try to stabalize the system by deciding which version of file X was changed last and replicating that latest version.

Who's work will the caching daemon overwrite? My work, or my friends work?

Of course, this need not involve anything so extreme as days of disconnected independent operation. A rebooting router between two previously syncd srfs peers seems sufficient to trigger this kind of data loss, unless you make the logging daemon fail all writes when disconnected.

i want to apologize if my explanation was not clear enough: self stabilization (original idea by Dijkstra) - A self stabilizing system is a system that can automatically recover following the occurrence of ( transient ) faults. The idea is to design a system which can be started in an arbitrary state and still converge to a desired behavior.

Our file system behaves like this: lets say you have several servers, with different file system trees on them. If (and when ...) you connect these file systems with an srfs framework, all servers will display the same file system tree, which is somewhat of a union between them all. if you wish to talk in coda terms, you can say all servers operated disconnectedly, and then were connected at the same time. the conflict resolving mechanism we use, is by majority.

We differ from coda in the sense we don't have a main server, which pushes Volumes to sub-servers (im not sure what the coda terminology is... ), and data is served in a load-balanced way. In Srfs, all the data resides on all servers (hosts) and is replicated between them. replication takes place at two levels: tree view (plus meta data) and the actual data.

• tree view - the tree view on all hosts is the same. an `ls` on a dir on any host will produce the same output.
• data - data will be replicated to all hosts upon a successful write, and upon each access to a dirty file on each host.

all replication is lazy, and happens only on access to dirs / files (and on successful writes - when the file is being closed.)

Thus, the following behavior can be achieved: lets say you have 2N+1 hosts, all with coherent file system trees. now, take N of them offline, change the tree, put those N back online, and their tree will be the same as the other N+1 other hosts.

The main goal of the file system is self stabilization, over long periods of time and long distances. you can use it as a SAN, or as a data farm, using system like LinuxVirtualServer to balance the load between nodes.

I wrote a simple setuid-root wrapper which sets some capabilities, gives up all other privs, and and then execs a shell. I was hoping to use this wrapper as a login shell so that I could have a user log in interactively with a small subset of elevated privileges.

Unfortunately after looking over the capabilities code in the 2.4 kernel, it would appear that this is not currently possible, and my wrapper cannot work without filesystem support for capabilities. And even then, I'd have to set each file's inheritable flag for the capabilities I want on every executable that I am likely to run, including the shell. Am I mising something, or is this an accurate description?

I think I understand the rationale behind this behavior; the draft posix 1003.1e specification states:

The purpose of assigning capability states to files is to provide the exec() function with information regarding the capabilities that any process image created with the program in the file is capable of dealing with and have been granted by some authority to use.

So, the lack of an inheritable flag on a file can serve to prevent that file from executing with the corresponding capability enabled.

Fine, but what about my semi-superuser shell situation? How can I force the retention of a capability set across exec() for all executables? It would seem that neither the spec nor the current implementation in the 2.4 kernel allow for this, but it strikes me as a pretty reasonable and useful thing to do in some cases.

As an interim workaround, how about assuming all capabilities are inheritable in fs/exec.c:prepare_binprm, i.e. instead of cap_clear(bprm->cap_inheritable), call cap_set_full() ??? I don't think this would break anything, and it would make capabilities a lot more useful until we get fs support merged in.

The people who wrote the code were working from two different drafts of the spec. I think some people used draft 16, while others used draft 17. (or 15 and 16, or 17 and 18 -- a difference of 1) Between these two drafts there had been BIG changes. Well, a critical equation changed.

People at SGI, mindlessly cloning the IRIX code, stuck us with the half-ass set of capability bits we have today. They ignored the DG-UX implementation using 256 bits and slightly different equations. They ignored the fact that the security model will be terribly inconsistent if you still have apps making UID-based decisions -- that is, you need to allocate bits for glibc, XFree86, Linux vendors, admin tools, various databases, and local site usage. Yes it's yucky, but it's required. Covering ears and burying the head won't make this go away.

Nobody thought to have half the bits default to "on" for stuff currently allowed for regular users. For example, the right to listen for incoming network connections could be limited if this had been given a default-enabled bit.

Then there's the emergency hack done to patch a security hole that the capability bits introduced. I think that was back in the early 2.4.x days.

People like to ignore the fact that apps tend to answer "Do I need setuid-style precautions?" by examining UID.

People like to ignore the fact that privileged code, written with setuid in mind, can lead to all sorts of mayhem if 42% of the privileged operations are prohibited. Yeah, you'd hope that a setuid app has great error checking and can cope... but "hope" shouldn't satisfy you. We really need a way for app authors to mark a binary as "always block rights P, Q, and R" and "block all rights unless given V, W, and X", with the assumption that an unmarked app requires an all-or-none situation.

Probably there should be two worlds on the system. Apps with "funny" rights should be kept away from UID 0 and setuid apps, while apps with UID 0 or setuid should be kept away from "funny" rights. Give the init process a special ability to cross worlds.

The authors of our code seem to have given up and moved on. Nobody cleaned up the mess. Is it any wonder the POSIX draft didn't ever make it beyond the draft state?

(and damn, WTF is with !capable(...) meaning that you are capable of performing something?)

One final horror: just imagine trying to write up some sane documentation for the average admin. Poorly-understood security mechanisms are a hazard. BTW, don't forget to imagine documenting all the interactions with UID, filesystems, etc.

Face it: admins will think in terms of assigning rights to users, never minding that there are some weird equations, UID interactions, and perhaps per-executable bits.

In general, we're moving towards a BK version where keys (internal revisions, sort of like mail message id's) are useable anywhere a rev is useable.

One place we'll be using this is on BK/Web so that you guys can have URLs that don't change out from underneath you.

We should fix that at the same time that we turn on the GNU patch server so you can get any changeset as a patch. The dual T1's are due in at the end of this month.

There may be some delay, I'm away dealing with family stuff that is way higher in priority than this but I'll try and get someone else to do it if it takes longer than the end of the month before I'm back.

This patch updates UML to 2.6.0-test8.

The 2.6.0-test5 UML patch is available at
http://jdike.stearns.org/mirror/uml-patch-2.6.0-test8.bz2

BK users can pull my 2.5 repository from
http://jdike.stearns.org:5000/uml-2.5

For the other UML mirrors and other downloads, see
http://user-mode-linux.sourceforge.net/dl-sf.html

Other links of interest:

The UML project home page : http://user-mode-linux.sourceforge.net
The UML Community site : http://usermodelinux.org

This release is done in advance of a talk about it for the CGL meeting tomorrow at OSDL.

I've released the 005 version of udev. It can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-005.tar.gz

rpms are available at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-005-1.i386.rpm
with the source rpm at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-005-1.src.rpm

udev is a implementation of devfs in userspace using sysfs and /sbin/hotplug. It requires a 2.6 kernel to run.

The major changes since the 004 release are:

• klibc is now included in the udev tarball. If you want to build udev with klibc, please see the README file for how to do this.
• LABEL for the device symlink now works again
• if the 'dev' file is not present (like the device was yanked out before udev started looking at it), udev will now timeout properly.
• the namedev.permission and namedev.config files are renamed to udev.permission and udev.config. Make sure to realize this if you have customized your rules in the past.
• man file updates

The biggest stuff is the klibc integration. If you build with klibc, the 453K binary shrinks to 45K. Nothing like a power of ten decrease :)

The rpms are still built with debugging enabled, using glibc, so they do not get any size savings yet...

Again, many thanks to Dan Stekloff, Kay Sievers, and Robert Love for their help with patches for this release. I really appreciate it.

The full ChangeLog can be found below.

The udev FAQ can be found at:
kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ

Development of udev is done in a BitKeeper tree available at:
bk://kernel.bkbits.net/gregkh/udev/

I have the initial framework of some regression tests in the bk tree, but there is a libsysfs bug that is keeping these tests from working properly right now. The libsysfs people are working on fixing this.

If anyone ever wants a snapshot of the current tree, due to not using BitKeeper, or other reasons, is always available at any time by asking.

Here's a patch for 2.4.23-pre8 kernel for MPT Fusion driver, coming from LSI Logic.

This patch is large, so I have placed it on the LSI ftp site at: ftp://ftp.lsil.com/HostAdapterDrivers/linux/Fusion-MPT/2.05.10/mptlinux-2.05.10.patch

A new email address is setup for directing any MPT Fusion questions: mpt_linux_developer@lsil.com

I'm clear on the policy, however the 2.05.00.03 MPT driver in 2.6 kernel is *NOT* compatible with what is shipping in 2.4 kernel which is 2.05.05+ driver. The driver in 2.6 has had most of the backward compatibility stripped out, such as Old Error Handling, and many other changes to make it work with new kernel structures and functions, however doesn't make it backward compatible to 2.4 kernel.

Our focus has been 2.4 kernel version of the driver as that is what is shipping in all Linux distributions, and our customers have been asking for RPM driver updates to the latest driver fix bugs and enhancements for their shipping systems out in the field. One major OEM player has requested we update Kernel.org as to reduce their dependency on LSI for RPM driver updates. I wish that these updates make their way into the 2.4 kernel. I will begin porting these changes over the driver in 2.6 immediately. Also one thing is that there have been change on Maintainership of this driver from Pam Delaney to myself and Larry Stephens, so things are about getting back to normal.

I'm pleased to be able to announce the first test release of a port of the current 2.0 pre-release Software Suspend code to 2.6. This is now available from www.sourceforge.net/projects/swsusp and bk://swsusp25.bkbits.net/main.

Release notes:

• The patch is prepared against current bk. It should apply against test8 with minimal and perhaps no fuss.
• Breaks current Software Suspend implementations in the kernel. Apologies to Patrick and Pavel. I won't be leaving it this way for long, I promise!
• I/O is slow and jerky. I need to investigate the cause for this further; something in the hooks to the new bio code is not quite right.

Apart from the above, and the normal problems with incomplete driver support will continue. In addition, you may see freezing failures. If the process hangs at 'Freezing processes: Waiting for activity to finish' or 'Syncing remaining I/O', try pressing escape once. If the process doesn't abort, try a second time (which tries harder to restart things). All going well, you should be able to cancel the suspend. A log of what went wrong will be stored in /var/log/messages. Run it through ksymoops if necessary and send it to me, and I should be able to address the issue.

Please send feedback via the Software Suspend mailing list on Sourceforge. See http://swsusp.sf.net for FAQs, mailing list details and so on. Because the code is essentially the same as the 2.4 version, many of the solutions to issues will be the same.

Initial availability of User-Space System Device Enumeration (uSDE) package, version 0.74, can be found at http://sourceforge.net/projects/usde

The uSDE provides an open framework for the enumeration (specification) of system devices in a dynamic environment. Device handling is implemented via plug-in programs known as policy methods. Policy methods are free to handle their devices in any way, from trivial to complex - anything from providing LSB device nodes to persistent device name handling with replacement and relocation strategies.

The uSDE depends on /sbin/hotplug (for dynamic insertions and removals), sysfs (for device information) and /proc (various pieces of information). It is not dependent on initrd - it explicitly scans sysfs upon system startup to determine the initial device ensemble.

Part of the uSDE release is a collection of sample polices:

disk-ide-policy - handles IDE, EIDE, SATA and USB-EIDE disks. Implements persistent device naming, automatic device replacement and automatic device relocation features.

disk-scsi-policy - handles SCSI, IEEE-1394, FibreChannel and USB-SCSI disks including multiported devices. Iplements persistent device naming, automatic device replacement and automatic device relocation features.

multipath-policy - handles the automatic provisioning of multipathing for multiported storage devices.

ethernet-policy - handles ethernet interefaces. Implements persistent interface naming, interface anchoring, automatic device replacement and automatic device relocation features.

floppy-policy - handles internal floppy disks.

simple-device-policy - a "catch all" policy for block and character devices.

devfs-policy - provides devfs device names.

lsb-policy - provides LSB device names.

Location: http://sourceforge.net/projects/usde
Mailing list: usde-general@lists.sourceforge.net

How does uSDE relate to udev? You do not mention it in your email, though it claims to implement similar, if not identical functionality. Is it related? Is it built on top of it?

If not, are you planning on merging your efforts with udev in the future?

Are you using the libsysfs library for accessing sysfs data? If not, I highly recommend it.

I would also recommend sending email to the linux-hotplug list, as most of the hotplug-related applications are discussed and developed via that list.

The uSDE is not built on top of udev.

The uSDE and udev are similar in some respects. They both create device nodes. There is a lot more to handling devices than managing device nodes.

Some differences between uSDE and udev that come to mind as I type (a good deal of this is part of the INTRO in the uSDE tarball):

Devices are classified and an explicit, ordered list of policies are invoked on behalf of the devices based on that classification.

Policies are implemented as open plug-ins that have complete control (e.g. naming, configuration, special needs) over a device.

Multiple policies can be executed concurrently; they can be independent or cooperative.

All device types are embraced - ethernet, disks, cdroms, floppies, MD, LVM and so on. Policies can analyze data and handle complex situations such as ethernet interface anchoring, multiported disk handling and automatic multipath device management.

The concept of service agents who provide critical information to the enumeration framework allowing policies to handle extremely diverse hardware situations such as multiple chassis and geographical addressing.

The uSDE sample policies implement basic device replacement and relocation strategies, something that the community has been asking about for some time.

If you want to learn more about that differences, download the tarball and try it out...

The uSDE was built in response to a set of telco and embedded community requirements. We found it difficult to express our ideas. Everyone wanted to see code and documentation. Here is the code and the initial documentation. This is a starting point...

The two packages take philosophically different approaches and arrive with (largely) overlapping and some non-overlapping capabilities - after all they are both trying to do "the same thing". The uSDE has strengths and weaknesses just as udev or any program does. It is certainly possible to discuss changes (and make patches) to udev to incorporate the key issues addressed in the uSDE implementation.

The uSDE is an encapsulation of ideas and techniques. It is "complete" enough for those ideas to be discussed in a community setting and we can see how/what to move things together. Think of it as the projects "resting place" from which to confidently discuss techniques and implementions.

The requirements were collected from the OSDL CGL requirements specification version 1.0 and 1.1 ratified September 2002. They come from extensive discussions with the OSDL members as part of the definition of these requirements, expounding on them:

• The embracing of all device types with no specialization or limitation.
• The ability to have total control over the handling a device via external policy programs. Policy programs are invoked with a formal command line and description of the event that caused there invocation.
• The "service container" concept. A device is classified (or recognized by a pattern match) and this raises an (queued) event which is caught by a configurable "service container". The container is an ordered list of handlers that process the device.
• Event queuing and aggregation. Minimizing the number of program invocations (fork/exec) is critical in embedded environments - small processors.
• Aggressive device enumeration. Multiple concurrent policy execution and management.

• Device information persistence is a function of device policies, not the enumeration framework.

  There are many situation where persistence is not an issue at all or only in specific cases (like disks). Why always pay for the memory/disk, for persistence, when it is not (always) necessary?

• Transactional protection of multiple configuration files is necessary. Multiple configuration files must often be modified in unison and insurance is necessary that an accurate and correct set of data is used when processing devices.