Kernel Traffic #83 For 5 Sep 2000

One of the things that bothers me is the feeling of eliteness that Linux people have. Alan does not :) at least IMHO. He has replied to several programming questions across several platforms. However, the LACK of a common path for basic questions disturbs me.

People send email to Kernel for not know where else to send it. They get flamed. It then turns off others from sending email and learning things. Many people can be turned off from the Process this way.

With the 2.4 kernel nearing release, what is the perceived level of scalability that can usefully be achieved now? How many processors until you max out performance? If it varies from architecture to architecture, please indicate the differences.

Please feel free to point me to URL's discussing this. I'm in a SMP scalability discussion of sorts, and people believe Linux can't usefully go past 4 processors. I figured that was true for 2.2.x but believe 2.4.x scales much further and just want to share the facts - whatever they may be - with people.

As long as the software used to create the GIF was licenced to create GIF's, owning and distributing actual GIF images is not in violation of the law. If it were, every web site in existance would be sued. Distributing web pages is not much different than distributing tarred up kernels.

The real reason to convert it to a .png, is simply to boycott gif images entirely, which I totally agree with. IMHO, PNG should be the standard format for open source, used in place of .gif everywhere. I believe GNU advocates png usage and boycotts gif..

My guess is the only reason for logo.gif is that nobody has bothered to notice it or care. Anyone care to convert it and submit a png? Sending a diff of a png wouldn't be a good idea I don't think. ;O)

I can do the conversion if someone else hasn't allready..

GIF and PNG have approximately the same functionality, with mostly these differences IIRC:

• GIF has thumbnail and animation support
• PNG is much better thought out and has a far more regular structure
• PNG has *far* better documentation (RFC 2083, for starters).

If anything, I'd say PNG is simpler.

Still, for uuencoding, there are far too many implementations of uuencode around (with/out checksum, with/out boundary, using blanks (0x20) or backticks (0x60)), and finally, uuencode is trashed when quoted-printable encoded.

These considerations are overcome by MIME which is well-defined. Wrong implementations of a well-defined standard may not be excused, flaky implementations of a not-so-well "standard" are to be avoided.

Note that particularly, 8 years after RFC 1521/1522 (which have not really changed in 2045..2049), there is absolutely no excuse for a lack of MIME support.

I have yet to understand WHY all of this is done.

Why?

I personally think this is a major step backwards. Moving things to be in the same directory just because it made some configuration easier. Nobody has explained to me why you couldn't just add one new configuration option, and make the affected drivers (in their regular places) dependent on that configuration option.

Are all the SCSI drivers going to be under drivers/scsi/? No. The "normal" ones that don't have any better place for them are, but nobody has really suggested moving drivers/usb/storage around to another place just because it uses the SCSI layer.

This patch is not going in until somebody can explain to me (in words of one syllable) why it makes any sense at all. As far as I can tell, it would be a ton more cleaner to do just

mv drivers/usb/input.c drivers/char/

and add a config option for it that the real drivers (wherever they may be - be they USB, firewire, joysticks, what-not) can know to disable themselves or not.

I object to moving files around in ways that makes the tree _less_ clear. I want the kernel tree to be a nice hierarchy of drivers, filesystems, etc. I don't see the point in lumping everything together in one subdirectory just because they happen to be "input" devices, regardless of what kind of device they really are.

I have to admit I don't quite understand how drivers/ is organized right now. There seem to be at least four criteria for subdirectories of drivers/:

drivers/<function> (sound, net)
drivers/<interface> (char, block) (this isn't the same as function IMHO)
drivers/<bus-the-device-is-on> (usb, sbus)
drivers/<architecture> (s390, sgi)

then there are some hybrids (usb seems to be both "devices that are on USB" and "USB controllers") and strange things like drivers/char/sysrq.c (which isn't a character device at all, and isn't really a driver either).

I don't see any particular preference, and my impression is that new directories get added pretty much at random.

I think I'm not the only one to be a bit confused about this.

Basically, the thing originally was just "drivers/char" and "drivers/block". Nothing more.

It became quite unmanageable fairly quickly, because by the time you have a few hundred files in a directory, "ls" no longer gives the directory listing, it just gives tons of lines scolling by too fast to make much sense of it. And configuration etc is painful.

Right now, drivers/net is pretty horrible, for example, and we've started separating things out a bit (ie drivers/net/tulip, drivers/net/pcmcia etc to get a better hierarchy). Same thing happened to drivers/block - although in that case the splitup happened to drivers/ide, drivers/cdrom.

drivers/char is messy, and _should_ probably contain only "fundamental char drivers" (ie things like basic tty layer stuff, ptys, /dev/zero, etc). It has too many "misc" drivers in it. Stuff like bttv, zr36120 etc. And the low-level serial stuff should probably go into drivers/serial or something.

I'm eager to add new directories if they help make for finer-granularity hierarchy.pThe reason I dislike drivers/input is that it seems to collapse many drivers under one, and split up something that conceptually is clearly one tree (usb). So it fails both the "split things up for clarity" and the "organize things after something simple" idea.

Which is not to say that the drivers/ hierarchy is necessarily all that wonderful as-is.

because we cannot reasonably do a good job for serial mice, we'll always be in the situation that the X server needs to be able to handle different mice. Which in my opinion makes it redundant in the kernel: while I know that things like GPM etc use mouse devices directly, I'm personally convinced that in the bigger picture the only thing that really matters is X.

And X handles multiple input devices quite nicely, these days. That didn't use to be the case. So the advantage of having an input layer in the kernel is greatly diminished - and basically nonexistent if X still has to be able to do different mice.

It's a stupid lie, and one we should ignore at all times it comes up.

It is NOT true that "shared code means less bugs". People, wake up!

It's true that "simple and tested code means less bugs". And quite often, code sharing will imply that the code gets (a) more complex and (b) has cases that people who make changes will never be able to test.

No, I'm not saying that sharing is bad. Sharing is good. But anybody who makes a blanket statement like "sharing reduces bugs" is full of crap.

Good and clear interfaces reduce bugs. Avoiding complexity reduces bugs. Using your brain before you code something up reduces bugs.

But making a very complex driver that handles ten different variations on the same hardware? No. That does _not_ reduce bugs. It's quite often more efficient to have two people work on two drivers that are independent of each other, than to have two persons working on the same driver. Alan put it well at some point: the way to create working open source software is to minimize the amount of communication needed.

Make good interfaces. THAT will reduce bugs.

Note that this too can be beneficial.

Not for technical reasons, but for purely psychological ones. Which can often be as important as the technical ones in the end.

The reason? People feel uncomfortable modifying other peoples drivers. This is often true even if the original author hasn't even been a very active maintainer for the last few years. So you find somebody new come in, who cares about the new cards, but is not willing to maintain the old stuff.

So he creates a "new" driver that works for him, and in the end he may end up maintaining it a lot better than the non-existent maintenance that the old driver gets. The old driver ends up working with the old cards, and the new one gets the new ones. And in some cases this eventually gets the old maintainer involved again and things work out.

Yes, this has happened.

And yes, it can go the other way too. It can become a source of painful and unnecessary duplication.

On the whole, people tend to _want_ to share, because it ends up being the easier "quick hack" in many cases. So I'm not worried about that part overmuch. I'm worried about people who share even when it doesn't make sense. And I'm worried about people having bad interfaces, which makes even sensible sharing end up as a experiment in horror.

That's why I'm so un-interested in the "let's share" argument. I don't think that is where the problems are.

Sharing code is a way to increase the test population for the shared parts, which increases expected test coverage on the shared parts, which is good. It's especially good because the shared parts tend to have subtler bug signatures than the unshared parts (data structure corruption is harder to diagnose than a device lockup), so the better test coverage happens exactly where it's most needed.

Clean APIs also reduce bugs, of course, but that effect is orthogonal to the one Rogier was trying to describe. IMHO, we should strive for both.

Testing only gets you so far.

And having a driver that handles everything under the sun can easily get too complex - to the point where testing only shows that something is wrong, and actually _fixing_ the bugs is getting to be the problem.

See Windows.

Or see the IDE driver. It's actually on it's second generation, and people have talked about a third one. Supporting every IDE controller, whether it is PCI or not, whether it can do DMA or not, is just fairly painful. To the point where trying to fix one case tends to break another.

No, the serial driver is not there yet. Probably never will be. There just isn't enough incentive for hardware people to do new things.

But the "common code helps" thing is WRONG. Face it. It can hurt. A lot. And people shouldn't think it is the God of CS.

I think you're mistaken about this. Or, to be more precise, I think your remarkable native talent as an implementor has given you some unhelpful design biases that are only going to be cured by more experience.

I'm fairly sure that experience will prove survivable for the Linux kernel and the rest of us -- but when you say things like the above, I start worrying a bit...

I'll give you a rule of thumb, and I can back it up with historical fact. You can back up yours with nada.

Simple rule of thumb:

• sharing is good when it is true 100% sharing, no special cases, and there is nothing that has reason to avoid the sharing.
  Example: the VM layer. TCP/IP. The VFS layer.
• sharing is bad when it's done to 90%, and people work at trying to avoid using the functionality that some other people want.
  Example: the SCSI layer. Parts of the IDE driver. Glibc vs kernel header files.

Quite frankly, the mentality that "we must share all common problems, whether it makes sense or not" has resulted in untold woes. Usually it starts out small. You add one more device. It obviously makes sense to just tweak the code a bit. You add one more. You'll add some special case code that only really matters for that one, and you hope that you got the other cases right.

Eventually, you'll have code that spans 5 generations of hardware, where the first generation and the last one basically share _no_ commonality except for the common heritage. And they share a lot of the code, because they have been written to do things the same way, even if it really wouldn't make much sense any more.

End result: bad organization for the new hardware, inability to sanely take advantage of the full features of the new hardware. Forcing people to leave old code in place, because it is work-arounds for bugs in hardware three generations gone. People end up having to thread very lightly because the developer probably has access to all the new stuff, but doesn't even have a way to test the old stuff any more - except by having users holler when he broke it by mistake when he added code to handle new cases.

Face it. Every once in a while, you have to start afresh. Tell people that "Ok, we can't share this code any more, it's getting to be a major disaster".

We've done this quite successfully several times. Each time it resulted in a better product. Which is exactly the reverse of what you advocate.

Example 1: "hd.c" was left alone, "ide.c" was created. Some day, we'll hopefully leave "ide.c" alone, and create a "ata100.c" that only handles modern hardware.

Example 2: 53c8xx driver. Which broke from the 53c7xx driver and just said "Screw it. That driver is too limited by the hardware it supports. I don't want to deal with it". See also the u14-34f driver vs ultrastor.c.

Example 3: ne2k driver. Nope, the ne.c file was not just expanded to handle yet another card type. Yes, it shares the basic 8390 code anyway.

This goes on and on and on. According to your logic, none of the above improvements should ever have happened. You're wrong.

I'm not arguing that splitting a driver is always wrong -- I can easily imagine making that call myself in your shoes, and for the exact reasons you give. I'm arguing that the perspective from which you approach this issue causes you to underweight the benefits of sharing code, and to not look for ways to do it as carefully and systematically as you ought.

When you were in college, did you ever meet bright kids who graduated top of their class in high-school and then floundered freshman year in college because they had never learned how to study? It's a common trap. A friend of mine calls it "the curse of the gifted" -- a tendency to lean on your native ability too much, because you've always been rewarded for doing that and self-discipline would take actual work.

You are a brilliant implementor, more able than me and possibly (I say this after consideration, and in all seriousness) the best one in the Unix tradition since Ken Thompson himself. As a consequence, you suffer the curse of the gifted programmer -- you lean on your ability so much that you've never learned to value certain kinds of coding self-discipline and design craftsmanship that lesser mortals *must* develop in order to handle the kind of problem complexity you eat for breakfast.

Your tendency to undervalue modularization and code-sharing is one symptom. Another is your refusal to use systematic version-control or release-engineering practices. To you, these things seem mostly like overhead and a way of needlessly complicating your life. And so far, your strategy has worked; your natural if relatively undisciplined ability has proved more than equal to the problems you have set it. That success predisposes you to relatively sloppy tactics like splitting drivers before you ought to and using your inbox as a patch queue.

But you make some of your more senior colleagues nervous. See, we've seen the curse of the gifted before. Some of us were those kids in college. We learned the hard way that the bill always comes due -- the scale of the problems always increases to a point where your native talent alone doesn't cut it any more. The smarter you are, the longer it takes to hit that crunch point -- and the harder the adjustment when you finally do. And we can see that *you*, poor damn genius that you are, are cruising for a serious bruising.

As Linux grows, there will come a time when your raw talent is not enough. What happens then will depend on how much discipline about coding and release practices and fastidiousness about clean design you developed *before* you needed it, back when your talent was sufficient to let you get away without. The code-sharing issue -- more specifically, your tendency to abandon modularization and re-use before you probably ought to -- is part of this. Andy Tanenbaum's charge against you was not entirely without justice.

The larger problem is a chronic topic of face-to-face conversation whenever two or more senior lkml people get together and you aren't around. You're our chosen benevolent dictator and maybe the second coming of Ken, and we respect you and like you, but that doesn't mean we're willing to close our eyes. And when you react to cogent and well-founded arguments like Rogier Wolff's as you have -- well, it makes us more nervous.

I used to worry about what would happen if Linus got hit by a truck. With all respect, I still worry about what will happen if the complexity of the kernel exceeds the scope of your astonishing native talent before you grow up.

Ok, getting close to the real test7.

The patch looks bigger than it is due to the drivers re-organization (discussed to death on this very list ;).

pre7:

• block_all_signals()/unblock_all_signals() interface to allow drivers to react nicely to signals that happen to a process that owns driver data. Read: direct rendering lock handling.
• ThunderLAN update (timer fixes, full-duplex, activity-led)
• Fix NFS oops on removing negative dentry. Honour rsize for directory read.
• usb updates
• scheduler wakeup race fix.
• move radio/tv cards to drivers/media, cleaning up drivers/char
• move "input" layer to drivers/input, cleaning up drivers/usb
• Cirrus SoundFusion CS4280/461x sound driver.
• proper camera locking in usb/dc2xx.c
• USB printer driver update (Printer Protocol 3 and timeout handling)

I took some time to update the code I found at http://hp.vector.co.jp/authors/VA008030/bfs/. This patche applies to a clean 2.2.16 kernel tree and adds support for bos (name changed to prevent conflict with bfs in 2.4). I hope to mod it for use in 2.4 as time permits.

It provides for read-only, however it will allow you to mount read write. Not certain that it won't do something nasty, so mount with -o ro.

Basically, I fixed one typo and updated the nls stuff for long filename support.

Ehrmm...

Let's see (this is from the v2.4-test tree, not v2.2.xx):

• ntfs
• hpfs
• ext2(fs)
• qnx4(fs)
• ramfs
• romfs
• umsdos
• vfat
• sysv(fs)
• coda
• adfs
• affs
• cramfs
• isofs
• jffs
• ncpfs

Not to be picky, but I'd say there are more 4-letter abbrevations than three letter abbr's.

befs (not beos; the fs should be there to stress that it is a filesystem) is a good name I think. beosfs might be a good name too.

No. No flags. That's the mistake SCSI did, and it is a fundamental mistake. It makes it basically impossible to "shift out" old code.

I'd much rather have two separate functions for this - sure, they can share most of the code by using common inline functions and what not, but I'd be a lot happier with drivers just using the function that they are most comfortable with directly.

Anyway, this is not a change I'd like to have at this point. This is definitely a 2.5.x thing.

This is the version waiting Linus. Its in the queue for holy penguin pee so either it will get peed on or I will get abuse from Linus depending whether he likes it or not 8)

In the mean time I'm taking a break rescuing a rather buggy but otherwise neat user space app. I'll start 2.2.18 in a couple of weeks.

James Cloos is making diffs between 2.2.17pre versions are available at http://jhcloos.com/pub/linux-2.2.17-pre ftp://jhcloos.com/pub/linux-2.2.17-pre

2.2.17pre20

1. Fix EIP/ESP printk thinko (Willy Tarreau)
2. Final small DAC960 adjustments for 2.2.17 (Leonard Zubkoff)
3. Improve AARP handling (Alistair Riddell)
4. Fix bug in the appletalk code (Marcelo Tosatti)
5. Last minute fix to the CS4281 (Tom Woller)
6. Mention CS4280 and use __initdata for data (me)
7. Turn on the EAPD bit on the AD1885 (me)
8. Don't honour the status bit for audio on a 440MX - it appears it doesnt work (Marcus Sundberg)
9. Fix high cpu usage on i810 audio (Marcus Sundberg)
10. Apply the same fix to the cs46xx (Bill Nottingham)
11. Change the power/CD algorithm on the 46xx (me)

Now, after the last MWDMA timing issue was resolved, I can do that. Yes, while I can't say my code is 100% perfect, because I'd have to verify that with an IDE analyzer which unfortunately I don't have access to, after some time of extensive testing there are no more failure cases on all the machines it was tested on. (One case is still unclear, but that user is now using 2.2 and couldn't do more testing.)

If you need it said explicitely - yes, now I can stand behind that code and I do accept the responsibility for it if it breaks anywhere. (Of course the legal no-warranty statement still applies).

Building the kernel uses two compilers. CC is the compiler for actually building the kernel you are going to boot, and HOSTCC is the compiler for building auxiliary programs *while you are building the kernel*.

So you don't need to change HOSTCC.

The host side programs compiled with HOSTCC include the configuration programs and other programs in scripts/, as well as some table generation programs in places like drivers/char and drivers/sound.

during 2.3 we got rid of 99% of lock_kernel()s within the core kernel. IMHO the time has arrived to get rid of the big kernel spinlock forever - by changing it to a ordinary semaphore. Most lock_kernel() code paths are holding the kernel lock for a long time and they do it rarely - so a semaphore is more suited to do this than a spinlock. It's even a speedup (and a debugging help), because lock_kernel() will not spin wasting CPU cycles anymore.

the attached biglock-2.4.0-test7-A6 patch implements this and does a couple of related cleanups:

• the impact of the 'big IRQ lock' is smaller as well, thus schedule() now enforces that it is to be called with IRQs turned on. sleep_on() enables IRQs, and entry.S's reschedule as well. (the later is a code path taken by returning IRQ handlers which might not always disable IRQs.)

  As a result neither release_kernel_lock(), nor the tq_scheduler path needs to do a sti() - which speeds up the common schedule() path. Nor the performance, nor the semantics of sleep_on() are impacted.

• new sema_locked() added to semaphore.h - similar to is_spin_locked().
• got rid of asm/smp_lock.h - linux/smplock.h has the generic code, now that no global-IRQ locking logic is present in schedule(). This simplifies things.
• irqs_enabled() added to asm/hardirq.h.

the only place that still needs to spin actively is reacquire_kernel_lock() - because semaphore.c calls schedule() itself, so we cannot call down() at that point. Solving this is not impossible, but needs more changes than justified i think.

the patch has the IMHO nice side-effect of 'speeding up races' - more processes can run when the kernel lock is taken because there is no active spinning - thus SMP, kernel-lock related races are triggered with higher probability.

the patch compiles/boots/works just fine on SMP and UP x86 systems. Other architectures have to add the (trivial and generic) sema_locked() and irqs_enabled() functions. The patch should be an invariant and does not intend to change semantics of any kernel functionality otherwise.