Kernel Traffic #237 For 26 Oct 2003

By Zack Brown

Table Of Contents

Mailing List Stats For This Week

We looked at 1552 posts in 6957K.

There were 475 different contributors. 226 posted more than once. 177 posted last week too.

The top posters of the week were:

 

1. Linux 2.6.0-test6 Released
27 Sep 2003 - 12 Oct 2003 (143 posts) Subject: "Linux 2.6.0-test6"
Topics: I2C, POSIX, Profiling, Sound, USB, Virtual Memory
People: Linus TorvaldsCon KolivasRoger LuethiVojtech PavlikGeert UytterhoevenRussell KingMikael PetterssonAlbert D. CahalanRobert WhiteSam RavnborgUlrich DrepperJamie LokierAndrew MortonBernardo Innocenti

Linus Torvalds put out 2.6.0-test6, saying:

Ok, too long between test5 and test6 again, so the patch is pretty big. Lots of driver updates and architectures fixed, but also lots of merges from Andrew Morton. Most notably perhaps Con's scheduler changes that have been discussed extensively and made it into the -mm tree for testing.

This also finally gets one of the last "must-fix" things for 2.6.0: the extended 32-bit dev_t support. Courtesy of Al Viro (with a lot of prodding and input over the years from Andries).

arm, s390, ia64, x86-64, and ppc64 updates. USB, pcmcia and i2c stuff. And a fair amount of janitorial.

Regarding his scheduler patches, Con Kolivas replied:

For those who are trying this for the first time, please note that the scheduler has been tuned to tell the difference between tasks of the _same_ nice level. This means do NOT renice X or it will make audio skip unless you also renice your audio application by the same amount. Lots of distributions have done this for the old 2.4 scheduler which could not treat equal "nice" levels as differently as the new scheduler does and 2.6 shouldn't need special treatment.

So for testing note the following points:

Make sure X is NOT reniced to -10 as many distributions are doing. Some shells spawn processes at nice +5 by default and this will make audio apps suffer.

Make sure your hard disk, graphics card and audio card are performing at equal standard to your 2.4 kernel (ie dma is working, graphics is fully accelerated etc). before commenting on audio performance.

Elsewhere, Roger Luethi remarked:

With test6, keyboard repeat takes very noticably longer to kick in after X has been started (for both X and console). In test5, starting X makes no difference.

Also, if you move your test5 .config forward and lose sound, you may find that you now have to enable gameport in input devices to be able to select (and thus, compile) your sound card driver.

On the up side, the scheduler changes make the infamous xmms skips go away (for my purposes).

Vojtech Pavlik identified the keyboard slowdown as a bug in the repeater code, and posted a patch to fix it. Someone replied, saying that this made the behavior more like recent 2.4 kernels, instead of recent 2.6 kernels, and Vojtech explained, "This is because it is the same as on the latest 2.4 kernel. 2.6 used software autorepeat up to test6. Now, because of hardware bugs, it was necessary to switch back to hardware autorepeat, like 2.4 uses."

Elsewhere, Geert Uytterhoeven noticed that a changelog entry by Bernardo Innocenti referred to a "GCC 3.3.x/3.4 compatiblity fix in include/linux/init.h". Geert reported, "This change breaks 2.95 for some source files, because <linux/init.h> doesn't include <linux/compiler.h>. Do you want to have the missing include added to <linux/init.h>, or to the individual source files that need it?" Russell King confirmed that the change broke GCC 3.2.2 and 3.3 as well. Linus said:

Interesting. I'm pretty sure I did a "make allyesconfig" just before the test6 release, so apparently x86 includes it indirectly through some path, and so it only shows up on m68k and arm?

This, btw, is a pretty common thing. I wonder what we could do to make sure that different architectures wouldn't have so different include file structures. It's happened _way_ too often.

Russell King replied, "The two files that it showed up in on ARM are fairly simple in nature and don't include may headers. Making the ARM include structure identical to x86 wouldn't have removed the problem from ARM." And Geert added, "Same for m68k. The offender was a m68k-specific file (arch/m68k/sun3/sbus.c), which just included <linux/types.h> and <linux/init.h>, and uses subsys_initcall()."

Replying to Linus' post, Sam Ravnborg suggested requiring the most used header files to include all the headers needed for their own compilation. This would have the benefit of consistency, even if it was overkill in most cases. Linus thought this was fine, saying, "sure, we could just require that the header files compile cleanly, and for extra points verify that the end result is an empty object file (ie no bad declarations anywhere..)." His one concern was whether all versions of GCC would optimize out redundant header includes; but Jamie Lokier said this had been standard with GCC for at least a decade.

Elsewhere, Mikael Pettersson reported:

Linus' 2.6.0-test6 announcement doesn't seem to mention the fact that 2.6.0-test5-bk9 fundamentally changed the semantics of /proc/self and the /proc/<pid> name space. These used to map to actual (kernel) tasks, now they map to what I assume are Posixly-correct processes (groups of tasks). In particular, /proc/self is no longer an alias for `current'.

I don't actually disagree with the change, but it took me by surprise since neither the 2.6.0-test6 annoucement nor the diff between the t5-bk8 and t5-bk9 logs seem to mention it.

(It broke the perfctr driver, but I'm handling that by making an already planned API switch now instead of later.)

Linus was somewhat taken aback by this report, saying, "the semantics weren't _supposed_ to change. The new semantics were meant to be a superset of the old behaviour, with just the added "task" subdirectory that lists the actual threads. However, you're right that "/proc/self/index.html" should likely point into the _thread_, and not into the task. But it's debatable." He asked Albert D. Cahalan (who wrote that code) what he thought about it; and Albert replied, "That seems likely to break other stuff, as new-style threads become more common. Right now, many tools are unaware of new-style threads. Pointing at the tgid directory (POSIX PID directory) lets tools ignore threads." He went on:

This is an interesting problem for sure. The link was pointing to a directory that didn't get listed, except that CLONE_THREAD wasn't exactly a popular feature yet. So it was very seldom that the distinction mattered.

Currently, I rely on checking for /proc/self/task to see if threads can be examined. Like this:

task_dir_missing = stat("/proc/self/task/index.html", &sbuf);

That wouldn't work if /proc/self pointed at the task.

It certainly seems to me that the intent of /proc/self is to point to a "process", which is a tgid in kernel terms. Back in the 2.4.xx days, that was a pid in kernel terms. Now that we have CLONE_THREAD and the tgid, a "process" is represented by the tgid. Pointing to the tgid matches what other OSes do.

I think there is something clearly defective about having the /proc/self link point to a hidden directory. It could be pointed at /proc/42/task/58 and such I guess, but I think tools could break as new-style threads begin to get used in the real world.

A lively technical debate sprang up, mainly between Albert and Ulrich Drepper, and at one point Robert White remarked, "I can say that (virtually) any programmer who does a lot of threads work is going to presume that he may pass file handles between threads safely. IMHO it would be exceptionally bad to break this assumption." Linus explained:

It's true that if you use the pthreads model, you'll pass fd's between threads freely.

But there are lots of other valid thread usages. Many of the original uses of Linux threading were for special-case apps which used the clone() interface directly. Some were games, where the native threading stuff was doing things like sound etc in the background.

And when you have _that_ kind of model (with assymetric specialized threads), it makes perfect sense for the threads to have independent file descriptors.

Robert had added, "At the purist level, when I pass an abstraction (data structure etc) around between my threads, having done my due-diligence WRT locking and such, I expect that when the abstraction gets there it will still be valid." And in his same reply, Linus said:

And that is what you get with pthreads.

But the native Linux threading has never been pthreads. It's been about a much more generic thing, where the user is in control of what he does.

And that, btw, implies that thread nazis that only use pthreads do _not_ get to determine what the rest of the world does.

A couple posts down the line, Linus added:

The reason people use threads is that sharing the VM space has real advantages: it makes context switching much cheaper (fewer hw resources in the form of TLB usages) and it allows for much faster synchronization through a shared address space.

But the same isn't true of file descriptors or a lot of other software- level abstractions. There are no inherent advantages to sharing, and in fact sharing just gives more opportunity for race conditions, bad interaction etc.

For example, one reason _not_ to share is that the subthread may want to be as invisible to the "main thread" as possible. That's just good programming practice - trying to isolate and encapsulate as much data as possible.

The same way you shouldn't make all your variables global, you shouldn't make all your data structures global unless you have a reason.

At one point Robert asked, "If all the CLONE_THREAD members of a process (automatically) have the same signal handling code/context but not the same list of file descriptors, what happens when a file descriptor posts SIGPIPE or SIGIO (etc.) to a process?" Linus replied:

You have to explicitly _ask_ for SIGIO. If you do so, and you don't share file descriptors, that's _your_ problem.

But it does indeed have perfectly valid semantics - the signal may well just wake up a thread: and in fact, as most IO is illegal in signal handler context anyway, it usually has to.

Clearly, if you have per-thread file descriptors, you have to keep track of which thread is doing what.

 

2. Linksys And The GPL: The Saga Continues
6 Oct 2003 - 13 Oct 2003 (16 posts) Subject: "Re: Linksys/Cisco GPL Violations"
Topics: FS: ramfs, Networking, Patents
People: David WoodhouseFlorian SchirmerSasa OstrouskaLuite Stegeman

Continuing from Issue #235, Section #8  (28 Sep 2003: Possible Linksys GPL Violations: The Saga Continues) , David Woodhouse responded to the ongoing "negotiations" with Linksys and Cisco:

I fail to comprehend. In the United Kingdom, Linksys-Cisco B.V. are committing a criminal offence under the provisions of Section 107(1) of the Copyright, Designs and Patents Act 1988, by knowingly distributing software without the licence of the copyright owner.

Even if the source were to appear on their web site today, their packaging remains a problem since it contains neither said source nor a written offer to provide it. In order to comply, they _must_ recall this product and amend the packaging. So even if they are already working on putting the source on their web site, they still need to recall the product -- and should already have done so.

Once they have withdrawn these products from the market and ceased to commit this criminal offence, _then_ there is scope for negotiation regarding the ways in which they can re-release the products.

Until then, I see none. By continuing to unlawfully sell this product, Cisco are clearly demonstrating that they are not acting in good faith.

I would like to know if this is also a criminal matter in the United States, and if you have also discussed this with the appropriate authorities, rather than only with Linksys/Cisco?

I would also like to know if in the US, as in the UK, it is also an offence for the _retailers_ and _distributors_ of these products to continue to sell them after being informed of the problem. And have they each been informed in a manner which allows it to be proven in a criminal court that they had 'reason to believe' (to use the phrase from the 1988 Act) that there was a problem?

Florian Schirmer replied:

good news. Broadcom/Linksys finally decided to provide all sources. I haven't checked everything but it looks like everything is there. Don't know if they did it by accident but they even donated more ... including the bootloader, ethernet driver and lots of other non-gpl'ed stuff. Interesting change of attitude. I'm sure one day i'll ask them for the source of the (modified) toolchain which they distribute binary only inside this package... but for now i'm happy :-)

Thanks for all the people that supported us. Thanks Broadcom/Linksys for finally come to the conclusion that working together with the community is much more effective than working against us. Thanks!

James Stevenson and others asked for a URL to the code, and Sasa Ostrouska said:

This is the link

http://linksys.com/support/gpl.asp

Download the wrt54g.tar.gz it is 37MB but I have not find anything interesting in it. Maybe I have not looked well.

Florian suggested looking into the WAP54G, WAP54AG or WRT54AG files, and Luite Stegeman said:

There seems to be much more in the package, more developer tools, source code for the broadcom reference design etc. Good for the linksys router hackers, but unfortunately there is still no source code for the driver.

I think the driver files are in /release/src/wl. This directory contains only object files, no source code. The wl driver directory in the kernel tree ( release/src/linux/linux/drivers/net/wl ) is empty.

I've checked the WAP54G, WAP55AG and WRT55AG archives and none of these contain the source code for the driver.

Elsewhere, David also confirmed that he could find only object files for the wireless driver, not source code. Florian replied, "The ethernet and wireless driver where never linked into the kernel. So it should be okay if they only distribute the module. They decided to provide object code. Which is far better than a linked the module." ... "The main Linksys case was about the GPL violation by linking stuff into the kernel. _That_ is resolved now. The wireless driver is a completely different story. IMHO." David replied:

That is true, according to the GPL, _only_ if the modules are distributed as separate works. If they are part of a collective work which is based on the kernel (note, not a _derived_ work but a _collective_ work) then they must be released under the terms of the GPL.

This is a _different_ issue to the question of whether a module is indeed a derived work, and it's _far_ more clear-cut.

Ask yourself the following questions:

  1. The wireless and Ethernet driver modules are distributed within a cramfs file system in a flash image on a chip soldered to the board of the device.

    Are they being distributed 'as separate works'?

  2. The fundamental mode of operation of these devices is to receive network packets from one of the drivers, pass them through the Linux kernel routing or bridging code, and then back out through another of the network interfaces. All three parts of this are indispensable and the product is useless without any one part.

    1. Does this form a whole which is a derived work based on the Linux kernel?
    2. Does this form a whole which is a collective work?
    3. Is this collective work based, in part, on the Linux kernel?

  3. Refer back to the facts in question 1. Is this 'mere aggregation of a work not based on the [kernel] on a volume of a storage or distribution medium'?

Now, having answered those questions, reread the final three paragraphs of ??2 of the GPL.

A couple posts later, he quoted the GPL:

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections WHEN YOU DISTRIBUTE THEM AS SEPARATE WORKS. BUT WHEN YOU DISTRIBUTE THE SAME SECTIONS AS PART OF A WHOLE WHICH IS A WORK BASED ON THE PROGRAM, THE DISTRIBUTION OF THE WHOLE MUST BE ON THE TERMS OF THIS LICENSE, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative OR COLLECTIVE works based on the Program.

 

3. Status Of RelayFS
7 Oct 2003 - 15 Oct 2003 (15 posts) Subject: "[PATCH][RFC] relayfs (1/4) (Documentation)"
People: Tom ZanussiKarim YaghmourDavid S. MillerStephen HemmingerHubertus FrankeJames Morris

Tom Zanussi announced:

This 4-part patch contains code for an interim version of relayfs (see Documentation below for a description of relayfs). This version still needs more testing and cleanup, but it contains most of the API changes prompted by user comments, which has resulted in a somewhat simpler API as well as some code simplification.

Here's a summary of the major changes:

  • added support for poll()
  • moved some of what once was exposed in the API into an opaque reader object, which also simplifies relay_read() (see Reader objects in Documentation)
  • removed buffers_full() callback (the function this performed is now specified as a channel attribute (see RELAY_MODE_CONTINUOUS in Documentation)
  • removed relay_resume() (clients need to check return value of relay_write() to figure out if a write failed)
  • removed offsets_changed() callback (now hidden in reader)
  • added relay_reset() which some kernel clients sometimes may need
  • changed locking scheme to use n-buffers instead of only 2 (which simplifies relay_open() a bit)
  • VFS readers made auto-consuming if a relay file is opened using O_EXCL (and a hopefully better description of what 'consuming' means in Documentation)
  • various bug fixes

Still todo -

  • resizing cleanup
  • resizing of mmapped buffers
  • code to allow channels to start out with static buffers

Thanks to Stephen Hemminger, Marco Cova, and Hubertus Franke for contributing patches and pointing out problems, and to Hubertus Franke for a lot of good ideas and for persevering through a couple of limitations (now overcome) in the previous versions that caused some headaches when trying to use relayfs from deep within the scheduler.

James Morris asked if the same functionality wasn't provided with Netlink sockets, and Tom replied:

One thing you can do with relayfs files is mmap() them. That combined with the kernel-side API, designed to make writing data into buffers and transferring it as large blocks to user-space efficient and flexible, allows for high-speed, high-volume applications which I'm not sure Netlink was designed for.

relayfs can also be used in 'packet' mode, using read(2) to read data as it becomes available, so it can be used for low-speed, low-volume applications as well. Also, some people might find the file-based approach more natural to deal with. Personal preference, I suppose.

James said it was possible to make Netlink sockets mmappable, and Karim Yaghmour replied:

So would you consider running printk on Netlink sockets? Do you think Netlink could accomodate something as intensive as tracing? etc.

While I am aware that a lot of people are using Netlink sockets to exchange data from the kernel to user-space, I don't think Netlink sockets can handle the type of throughput relayfs can handle. Netlink and other communication mechanisms (pipes, shared memory pages, etc.) were not designed to handle the type of throughput relayfs was designed for. If nothing else, the use of netlink also drags with it lots of networking code (netlink_sendmsg-> alloc_skb->kmalloc->etc. and then memcpy) With relayfs, you get direct access to the buffer: relay_write->relay_write_direct (which is actually a macro for memcpy()).

So yes, as you say, "It should be possible to make Netlink sockets mmapable", but in that case you might as well port the netlink sockets API to relayfs and you'll probably get better results.

David S. Miller came in at this point, saying:

Look, netlink is used on routers to transfer hundreds of thousands of routing table entries in one fell swoop between a user process and the kernel every time the next hop Cisco has a BGP routing flap.

If you must have "enterprise wide client server" performance, we can add mmap() support to netlink sockets just like AF_PACKET sockets support such a thing. But I _really_ doubt you need this and unlike netlink sockets relayfs has no queueing model, whereas not only does netlink have one it's been tested in real life.

You guys are really out of your mind if you don't just take the netlink printk thing I did months ago and just run with it. When someone first told showed me this relayfs thing, I nearly passed out in disbelief that people are still even considering non-netlink solutions.

 

4. Status Of kexec
8 Oct 2003 - 11 Oct 2003 (9 posts) Subject: "kexec update (2.6.0-test7)"
Topics: Kexec
People: Randy DunlapCherry George MathewEric W. BiedermanBill Davidsen

Randy Dunlap announced:

I've updated the kexec patch for 2.6.0-test7. It can be found at http://developer.osdl.org/rddunlap/kexec/2.6.0-test7/kexec-260t7.patch

A slightly different version of it can also be found in the -osdl patchset at http://developer.osdl.org/shemminger/patches/2.6/2.6.0-test7/

The userspace tools are at http://www.xmission.com/~ebiederm/files/kexec/. You'll need to update the kexec-syscall.c file for the correct kexec syscall number (274). I intend to try to automate this (somehow).

Cherry George Mathew asked, "Is there a consensus about what the syscall number will finally be ? We've jumped from 256 to 274 over the 2.5.x+ series kernels. Or is it the law the Jungle ?" Eric W. Biederman replied, "So far the law of the jungle. Regardless of the rest it looks like it is time to submit a place keeping patch." Bill Davidsen asked, "Forgive me if the politics of this have changed, but will a place keeping patch be accepted for a feature which has not?" Randy said placekeepers had already been accepted for other features like vserver, so there was every reason to think a kexec placeholder would be accepted as well. But he added, "But I don't think that it's quite time for a placeholder syscall number (IMO of course). Eric can submit one though." Bill said that he'd really meant, wasn't it too late in the game to see this feature in 2.6, and so wouldn't Linus reject it until 2.7? Randy replied, "Unless 2.6 is much different from past kernel versions, new features can be added after 2.6.0-final is out, usually if they are well-contained, like a new driver or filesystem. I don't see this as a big hurdle."

 

5. Linux 2.4.23-pre7 Released
9 Oct 2003 - 11 Oct 2003 (9 posts) Subject: "Linux 2.4.23-pre7"
Topics: USB
People: Marcelo Tosatti

Marcelo Tosatti announced, "Here goes -pre7... It adds the laptop "mode" functionality already present in recent SuSE/RH kernels, adds the megaraid2 (improved, faster, but not so extensively tested as the old megaraid) driver, adds BIOS EDD (enhanced disk detection) support, contains a USB update, network update, amongst other fixes. I hope we enter -rc stage in more or less one month."

 

6. Possible GPL Violation By EasyRDP
10 Oct 2003 - 12 Oct 2003 (4 posts) Subject: "Linux kernel GPL violation (EasyRDP)"
People: Ivo PalliLars Marowsky-BreePavel Machek

Ivo Palli reported:

I ran into a product called EasyRDP that uses Linux. However they license their product under a very restrictive license that completely violates the GPL license.

A complete description and dissection of the product can be found on my website: http://www.palli.nl/~ivo/rdp/

Lars Marowsky-Bree replied, "You are right technically, but hey, you informed them of a copyright violation which may mean a lot of internal hassles. Without giving them any slack because what they are doing _IS_ stupid, calm down a little. 1 day is a hardly a timeframe in which they can respond adequately, in particular at the end of the week. You should have allowed them a week or so..." Pavel Machek replied, "Well, it looks to me like 1 day is 1 day more than they deserve. All they have seems to be modified rdesktop; I do believe that wireless routers are honest mistake, EasyRDP looks *way* worse."

 

7. New VST "Variable Scheduling Timeouts" Code
10 Oct 2003 - 14 Oct 2003 (3 posts) Subject: "[ANNOUNCE] VST (tick elimination) is now available"
Topics: SMP
People: George Anzinger

George Anzinger announced:

The first release of the VST package is now available. VST or Variable Scheduling Timeouts (or if you prefer, Variable Sleep Times) contains code that, from the idle task, scans the timer list and, if no timer is near, skips the timer interrupts that would otherwise be generated. The patch name is hrtimers-vst-*

The net result is that a quite system will use far less power as it does not need to wake up ever 1/HZ timer tick.

This patch depends on the high-res-timers patch version hrtimers-2.4.20-3.0 which must be applied first.

Both of these patchs are on sourceforge at:

http://sourceforge.net/projects/high-res-timers/

Some details:

There is a proc directory (/proc/sys/kernel/vst/) dedicated to vst controls. Root may turn VST on or off here with "echo "0">enable. There is also a log option (danger Will Robinson) which, if turned on causes " x" where x is the number of ticks being skipped, to be printed on the log console. This will quickly swamp the console, but it does show things happening during quite times and no such output when the system is busy. It is also possible to set the threshold here. This is in units of milliseconds and is the time the next timer must away from now to trigger a VST sleep. Currently this is set rather large considering that we can sleep only 50ms, but you can change it here.

The file .../include/linux/vst.h contains details on the arch interface. The core code is all in .../kernel/timer.c with enabling bits in various places. The arch code for the x86 version is all located in .../include/asm-i386/hrtime.h

This is, of course, the first cut at this stuff. There is a lot left to do....

Things left to do:

Currently the PIT is used to wake the system for the next timer. If that timer is further away than about 50ms, the PITs limitations force us to use the max PIT time of about 50ms. We plan to use the rtc hardware to do this wake up, thus eliminating this restriction.

SMP imposes further restrictions and, as it currently stands, may cause late timers in some cases. When we port to the 2.6 kernel this will change, for the better, we hope :) This will also allow us to use the APIC timers in these systems to do the wake up, thus allowing longer sleep times.

 

8. Job Postings on linux-kernel
10 Oct 2003 - 14 Oct 2003 (14 posts) Subject: "kernel developers:two openings"
Topics: Spam
People: David S. Miller

Hotjobs posted some job opportunities on the list, and David S. Miller (list admin) asked them not to do this, as it was against list policy. When they posted another job opportunity, David said more strongly:

By spamming job opennings to our kernel development lists where such things are not considered allowed, you fucknuts are basically guarenteeing that no kernel programmer with a brain is going to respect your company enough to apply for these jobs.

Please stop posting this crap now.

Stephen Satchell said that job postings might well be welcome in the current economic climate; and that maybe David should let them go through. David replied:

Posting job offerings here has always been and will always be verboten here on these lists. And in particular technical people who join this list get a very bad taste in their mouth when someone advertises here be it for jobs or products.

If there is no good place to look for Linux kernel development jobs, that isn't my problem. What is my problem is to enforce the rules of these lists at vger.kernel.org.

 

9. BitKeeper Statistics
13 Oct 2003 - 14 Oct 2003 (4 posts) Subject: "Silly BK statistics"
Topics: BitKeeper, Version Control
People: Larry McVoy

Larry McVoy said:

The BK openlogging tree, which has all changesets ever made by anyone in the Linux kernel, was getting big. Really big. The nodes in the graph have internal "serial numbers" which are currently 16 bits, i.e., there can't be more than 64K nodes in the graph.

I sent mail to some of my engineers this morning saying "hey, I suspect the Linux openlogging tree is about overflow, we need to go to 32 bit ser_t's." I had no idea how close we were, I just knew it was a problem we needed to solve.

I just got mail from one of the team which reads: "With 199 serials shy of overflowing , the 32 bit version is now installed".

What that means is that in about a year, you've managed to create 65,337 changesets. That's 179 per day, 7.4/hour, 24x7. You guys are busy.

To put that in perspective, the most active project on sourceforge today, Gaim, has 805 commits to its changelog. Over 3.5 years. That means you are changing your source base 284x more often than they are. And that's just the BK users, that doesn't count the people not using BK, which are a substantial fraction.

No matter how you slice it, it is pretty amazing rate of change. If change is good, you guys rock, I've never seen anything like it.

 

10. New frandom Random Number Generator Module
16 Oct 2003 (32 posts) Subject: "[RFC] frandom - fast random generator module"
Topics: Random Number Generation
People: Eli BillauerNick Piggin

Eli Billauer announced:

Frandom is the faster version of the well-known /dev/urandom random number generator. Not instead of, but rather as a supplement, when pseudorandom data is needed at high rate. Few tests so far show that frandom is 10-50 times faster than urandom.

The project's home page: http://frandom.sourceforge.net.

The module works on 2.2, 2.4 and 2.6 kernels. A few straightforward #ifdef's handle compatability (easy to remove to match common coding style).

Purpose
=======

  1. Frandom is a handy source of bulk random data.
  2. It is *not* intended for encryption and security-related applications.
  3. frandom is intended for (scientific) simulations, wiping the disk, stress tests on algorithms and so on.
  4. It is more of an /dev/zero than /dev/random

Quality of random numbers
=========================

  1. The module has been tested for random number quality with the "diehard" set of tests, and passed them all. This indicates that the bytes are random enough for most scientific purposes.
  2. Additional tests results are welcomed.
  3. The core of frandom is based upon RC4. frandom is exactly RC4, minus the XOR operation with the data. So if frandom doesn't generate good random numbers, I would wonder why RC4 is considered safe.
  4. The random generator is seeded with 256 bytes of the kernel's get_random_bytes() for every file opened on /dev/frandom. This is equivalent to a 2048-bit random key on RC4.
  5. I don't see frandom fit for crypto purposes, mainly because the module was naively written. I won't fall off my chair if it turns out to be crypto-safe, but I wouldn't trust it either. Not yet, anyhow.
  6. Those who read the source and feel that such a simple algorithm can't create good random: That's exactly the beauty of RC4: It's simple and it works.

frandom and the linux kernel tree
=================================

  1. Occasionally, people complain that /dev/urandom is too slow, wishing for something faster.
  2. Other argue that a random generator can be written in user space.
  3. I agree with both. And I use /dev/zero a lot. I know how to write a zero-generating application in user space.
  4. The module is small: 6kB of source code as a standalone module, and 2.3 kB of kernel memory.

Nick Piggin and others felt that there was no compelling reason to put frandom in the kernel instead of user-space. There was a bit of a debate over what constituted kernel-worthy features, but nothing conclusive came out of it.

 

11. New iSCSI Target Implementation
16 Oct 2003 (2 posts) Subject: "[ANNOUNCE] iSCSI target implementation"
Topics: Disks: SCSI, FS: procfs, FS: ramfs, FS: sysfs, Ioctls
People: Roman ZippelJeff Garzik

Roman Zippel announced:

I'm proud to announce the first public release of this iSCSI target implementation. It's already usable, but it doesn't implement everything yet what is required by the iSCSI spec. The missing parts shouldn't be that difficult to add (I just didn't need them so far). Other more interesting parts of the spec (e.g. session recovery) are not implemented because I had no client to test it with (although recent UNH releases seem to support this now). It was mostly tested with Cisco iSCSI initiator driver, but also with with the MS initiator driver. You can download the driver from http://www.ardistech.com/iscsi/ , the README should have all the information to configure and build the driver.

This target driver is partly implemented in the kernel, the user space daemon takes care of session startup, cleanup and discovery and the kernel module takes care of the iSCSI requests and disk IO. There are a few reasons to put part of it into the kernel, the main reason is to have direct access to the page cache, this is also a main difference to other available target implementations, which don't use the page cache directly. User space has not that fine grained control (yet) and had to use a lot of threads (the kernel module currently uses 2 threads per target and 1 per device). Part of the problem could be solved with async IO, another part is to be able to control what is in the cache (this is somewhat related to the recent direct IO discussion). So this also an experiment of what is needed if it should be ever moved to user space.

Anyway, while it's in the kernel, there are also some interesting issues for the kernel-user space communication. Right now it's a proc only interface (no sysfs as it's 2.4 only, no evil ioctl, but someone will kill me for the macro abuse :) ), but it should be rather easy to replace it with something else. Perfomance isn't not that important at the moment (but not completely unimportant, if a lot of information has to be read from the kernel, e.g. at startup). More important issues are error reporting and event handling, which could benefit from a better integrated interface. So this driver is also intended to experiment with a few things, if anyone is interested I can explain it further.

Further development will depend a bit on the feed back. This is an older project and we thought about for a while and since we have no immediate need and not the resources to develop it alone, we decided it would be a good idea to release it completely under the GPL. So I updated it to the latest spec and fixed some of the worst embarrasments. If there is enough interest, the development will continue, but without some help it will be rather slow. :-)

Jeff Garzik didn't like the use of the /proc filesystem as the iSCSI interface. He said:

Adding to procfs is only slightly less "ewww" than ioctls :) A dedicated chrdev would be IMO preferred to procfs.

Al Viro also had even even more interesting suggestion for [zerocopy] kernel/userspace communication: ramfs.

But no more was said.

 

 

 

 

 

 

We Hope You Enjoy Kernel Traffic
 

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License, version 2.0.