Kernel Traffic #103 For 19 Jan 2001

Welllllll... crashes tend to produce different effects from sudden power interruptions. In the first case parts of the system keep running, and bizarre results are possible. An even bigger difference is the matter of intent.

Tux2 is explicitly designed to legitimize pulling the plug as a valid way of shutting down. Metadata-only journalling filesystems are not designed to be used this way, and even with full-data journalling you should bear in mind that your on-disk filesystem image remains in an invalid state until the journal recovery program has run successfully. You would not want to upgrade your OS with your filesystem in this state, nor would you want to remove a disk drive that didn't have the journal file on it.

Being able to shut down by hitting the power switch is a little luxury for which I've been willing to invest more than a year of my life to attain. Clueless newbies don't know why it should be any other way, and it's essential for embedded devices.

I don't doubt that if the 'power switch' method of shutdown becomes popular we will discover some applications that have windows where they can be hurt by sudden shutdown, even will full filesystem data state being preserved. Such applications are arguably broken because they will behave badly in the event of accidental shutdown anyway, and we should fix them. Well-designed applications are explicitly 'serially reuseable', in other words, you can interrupt at any point and start again from the beginning with valid and expected results.

All valid ways to shut down the system involve sending SIGTERM to running applications -- only broken ones would live long enough after that to be killed by subsequent SIGKILL.

A lot of applications always rely on their file i/o being done in some manner that has atomic (from the application's point of view) operations other than system calls -- heck, even make(1) does that.

Depends on the error. If the disk has gone hard-readonly, then we need to recover in core, and that's something which is not yet implemented but is a known todo item. Otherwise, it's not worse than an error on ext2: you don't have a guaranteed safe state to return to so you fall back to recovering what you can from the journal and then running an e2fsck pass. e2fsck groks the journal already.

And yes, a badly faulty drive can mess this up, but it can mess it for ext2 just as badly.

The code currently checks if the underlying media is write-protected. If it is, it fails the mount; if not, it does the replay (so that mounting a root fs readonly works correctly).

I will be adding support for virtual replay for root filesystems to act as a last-chance way of recovering if you really cannot write to the root, but journaling filesystems really do expect to be able to write to the media so I am not sure whether it makes sense to support this on non-root filesystems too.

The problem is: where did you give the explicit instruction? Just that you define "read-only" as "the medium should not be written" does not mean everybody else thinks the same.

actually, I regard "ro" mainly as a "hey kernel, I won't handle writes now, so please don't try it", like for cd-roms or other non-writeale media, and please filesystem stay in a clean state.

That ro means "the medium is never written" is an assumption that does not hold for most disks anyway and is, in the case of journlaing filesystems, often impossible to implement. You simply can't salvage data without a log reply. Sure, you can do virtual log replays, but for example the reiserfs log is currently 32mb. Pinning down that much memory for a virtual log reply is not possible on low-memory machines.

So the first thing would be to precisely define the meaning of the "ro" flag. Before this has happened it is ansolutely senseless to argue about what it means, as it doesn't mean anything at the moment, except (man mount):

ro Mount the file system read-only.

Which it does even with journaling filesystems...

in an enbedded device you can

1. setup the power switch so it doesn't actually turn things off (it issues the shutdown command instead)
2. run from read-only media almost exclusivly so that power event's don't bother you much
3. you can add extra power inside the device so that if someone does pull the plug, you have a few seconds of power to do the clean shutdown
4. you can run out of ram and force the user to do an extra step to save any changes to non-volitile storage (and if they power off during the save the results are undefined)

I have seen all of these approaches used in different devices (that are not running linux). This is not a new problem and the people working in this space have a bunch of answers.

an improved filesystem that tolorates bad shutdowns reasonably well will be welcomed for other reasons, but should not be viewed as a fix for people pulling the plug on you.

Linux 2.4 is now out, it is also what people should be concentrating on first when issuing production drivers and driver updates. Effective from this point 2.2 driver submissions or major driver updates will only be accepted if the same code is also available for 2.4.

Someone has to do the merging otherwise, and it isnt going to be me...

egads! how can there be "development" on a *stable* kernel line?

maybe this is the time to reconsider terminology/policy: does "stable" mean "bugfixes only"? or does it mean "development kernel for conservatives"?

It means development kernel for those who don't have enough time to debug the main kernel as well as their own project. The stable branch tends to be *far* better documented than the bleeding edge branch. Try to find documentation on the all-important page cache, for example. It makes a whole lot of sense to develop in the stable branch, especially for new kernel developers, providing, of course, that the stable branch has the basic capabilities you need for your project.

Alan isn't telling anybody which branch to develop in - he's telling people what they have to do if they want their code in his tree. This means that when you develop in the stable branch you've got an extra step to do at the end of your project: port to the unstable branch. This only has to be done once and your code *will* get cleaned up a lot in the process. (It's amazing how the prospect of merging 500 lines of rejected patch tends to concentrate the mind.) I'd even suggest another step after that: port your unstable version back to the stable branch, and both versions will be cleaned up.

here are some comments in Alan's favor:

He did not say people can not release 2.2 patches without 2.4 patches. He only said they will not be integrated into the kernel distribution without 2.4 patches.

If people continue to develop for 2.2 and have someone else, who is probably less familiar with the hardware, port to 2.4 for them, how soon would you trust the drivers over the 2.2 drivers?

In short, I agree with Alan completely. This is the correct move forward to cause 2.4 to become the stable release that everyone will be willing to adopt.

This is *exactly* why Alan's policy change makes sense.

If somebody submits a driver bugfix or update for 2.2, but not for 2.4, it'll take FOREVER for 2.4 to become as "trustable" as 2.2...

This change, however, will make sure that 2.4 will be as reliable as 2.2 much faster. Unlike 2.2, the core kernel of 2.4 is reliable ... only the peripheral stuff like drivers may be out of date or missing.

You don't understand people, I think.

No amount of publicity will matter all that much in the end: yes, it will result in many people who are not afraid of a compiler to try it out. And we've had that for over six months now, realistically.

But that's very different from having somebody like RedHat, SuSE or Debian make such a kernel part of their standard package. No, I don't expect that they'll switch over completely immediately: that would show a lack of good judgement. The prudent approach has always been to have both a 2.2.19 and a 2.4.0 kernel on there, and ask the user if he wants to test the new kernel first.

That way you get a completely different kind of user that tests it.

The other thing is that even if something like 2.4.0-test8 gets rave reviews, that doesn't _matter_ to people who crave stability. The fact is that 2.4.0 has been getting quite a lot of testing: people haven't even seen how the big vendors have all done testing in their labs etc.

And to the people who really want to have stability, none of that matters. They will basically "start fresh" at the 2.4.0 release, and give it a few months just to follow the kernel list etc to see what the problems will be. They'll have people starting to ramp up 2.4.0 kernels in their own internal test environment, moving it first to machines they feel more comfortable with etc etc.

None of which would happen if you just try to make the beta testing cycle much bigger.

Which is why to _me_ the most important thing is that I'm happy with the core infrastructure - because once you've tested it to a certain degree, it's not going to improve without a real public release.

Bad plan, considering packages rely on some infrastructure that is not in the rpm (update-modules). I tend to be pretty quick with making and uploading the deb anyway.

Having said that, I won't package 2.4.0 and will wait for 2.4.1 instead.

here is a TODO list for the memory management area of the Linux kernel, with both trivial things that could be done for later 2.4 releases and more complex things that really have to be 2.5 things.

Most of these can be found on http://linux24.sourceforge.net/ too

Trivial stuff:

• VM: better IO clustering for swap (and filesystem) IO

  • Marcelo's swapin/out clustering code
  • ->writepage() IO clustering support
  • page_launder()/->writepage() working together in avoiding low-yield (small cluster) IO at first, ...

• VM: include Ben LaHaise's code, which moves readahead to the VMA level, this way we can do streaming swap IO, complete with drop_behind()
• VM: enforce RSS ulimit

Probably 2.5 era:

• VM: physical->virtual reverse mapping, so we can do much better page aging with less CPU usage spikes
• VM: move all the global VM variables, lists, etc. into the pgdat struct for better NUMA scalability
• VM: per-node kswapd for NUMA
• VM: thrashing control, maybe process suspension with some forced swapping ? (trivial only in theory)
• VM: experiment with different active lists / aging pages of different ages at different rates + other page replacement improvements
• VM: Quality of Service / fairness / ... improvements

Additions to this list are always welcome, I'll put it online on the Linux-MM pages (http://www.linux.eu.org/Linux-MM/) soon.

A couple of thoughts:

1. A full kernel with everything compiled in might not fit on boot media such as floppies, while modules allows you to not load stuff that isn't needed to until after the main booting is accomplished.
2. There are several devices that have multiple drivers (such as tulip, and old_tulip for example). Which particular driver works depends on your exact particular hardware. If both of these drivers are linked into the kernel, whatever the kernel chooses to initialize first will talk to the device.
3. Having drivers as modules means that you can remove them and reload them. When I was working in an office, I had one scsi controller that was a different brand (Adaptec) than the main scsi controller (TekRam), and I hung a disk in a removable chasis on the scsi chain in addition to a tape driver and cd-rom. When I was about to go home, I would copy all of the data to the disk, unmount it, and then unload the scsi device driver. I would take the disk out, and reload the scsi device driver to get the tape/cd-rom. I would then take the disk to my home computer. I would reverse the process when I came in the morning.
4. If you have multiple scsi controllers of different brands, building on into the kernel and the other brand(s) as modules allows you to control which scsi controller is the first controller in terms of where the disks are.

why not include a script which takes care of ALL the leg work? All of the files it asks the reporter to include are o+r...

I can whip up a bug_report script to walk the user though all of the steps in REPORTING-BUGS, if the list isn't averse to 'dumbing down' the process to the point where maybe some people who shouldn't be submiting bugs (two words: 'user error') end up not being scared off by the process.

Is perl allowed for kernel scripts intended for users, or am I stuck with sh?

It can be done in sh, surely. I only tried to promote my perl version because I've done it in perl and nobody told me earlier that perl is not liked in the kernel tree - and I've seen some perl scripts there.

I guess I'll have to convert the script to sh.

I thought I'd mention the policy for 2.4.x patches so that nobody gets confused about these things. In some cases people seem to think that "since 2.4.x is out now, we can relax, go party, and generally goof off".

Not so.

The linux kernel has had an interesting release pattern: usually the .0 release was actually fairly good (there's almost always _something_ stupid, but on the whole not really horrible). And every single time so far, .1 has been worse. It usually takes until something like .5 until it has caught up and surpassed the stability of .0 again.

Why? Because there are a lot of pent-up patches waiting for inclusion, that didn't get through the "we need to get a release out, that patch can wait" filter. So early on in the stable tree, some of those patches make it. And it turns out to be a bad idea.

In an effort to avoid this mess this time, I have two guidelines:

• I've basically thrown away all patches sent to me so far, and I will continue to do so at least over the weekend. I'm not going to bother thinking about patches for a few days.
• In order for a patch to be accepted, it needs to be accompanied by some pretty strong arguments for the fact that not only is it really fixing bugs, but that those bugs are _serious_ and can cause real problems.
• Obviously, the size of the patch matters too: if you can make an obvious fix in 5 lines, do it. Don't try to make a clean fix that fixes the problem the clever way in 150 lines.

In short, releasing 2.4.0 does not open up the floor to just about anything. In fact, to some degree it will probably make patches _less_ likely to be accepted than before, at least for a while. I want to be absolutely convicned that the basic 2.4.x infrastructure is solid as a rock before starting to accept more involved patches.

Right now my ChangeLog looks like this:

• Don't drop a megabyte off the old-style memory size detection
• remember to UnlockPage() in ramfs_writepage()
• 3c59x driver update from Andrew Morton

The first two are true one-liners that have already bitten some people (not what I'd call a showstopper, but trivially fixable stuff that are just thinkos). The third one looks like a real fix for some rather common hardware that could do bad things without it.

Now, I'm sure that ChangeLog will grow. There's the apparent fbcon bug with MTRR handling that looks like a prime candidate already, and I'll have people asking me for many many more. But basically what I'm asking people for is that before you send me a patch, ask yourself whether it absolutely HAS to happen now, or whether it could wait another month.

Another way of putting it: if you have a patch, ask yourself what would happen if it got left off the next RedHat/SuSE/Debian/Turbo/whatever distribution CD. Would it really be a big problem? If not, then I'd rather spend the time _really_ beating on the patches that _would_ be a big issue. Things like security (_especially_ remote attacks), outright crashes, or just totally unusable systems because it can't see the harddisk.

We'll all be happier if my ChangeLog is short and sweet, and if a 2.4.1 (tomorrow, in a week, in two, in a month, depending on what comes up) actually ends up being _better_ than 2.4.0. That would be a good new tradition to start.

And before you even bother asking about 2.5.x: it won't be opened until I feel happy to pass on 2.4.x to somebody else (hopefully Alan Cox doesn't feel burnt out and wants to continue to carry the torch and feels ok with leaving 2.2.x behind by then).

Historically, that's been at least a few months. In the 2.2.x series, 2.3.0 was the same as 2.2.8 with just the version changed - and it came out in May, almost four months after 2.2.0. In the 2.0.x series, 2.1.x was based off 2.0.21, four and a half months after 2.0.0.

Yes, I know this is boring, and all I'm asking is for people to not make it any harder for me than they have to. Think twice before sending me a patch, and when you _do_ send me a patch, try to think like a release manager and explain to me why the patch really makes sense to apply now.

In short, I'm hoping for a fairly boring next few months. The more boring, the better.

You are right. I have seen this bug before with the kernel moving unswappable pages from the active list to the inactive_dirty list and back.

We need a check in deactivate_page() to prevent the kernel from moving pages from locked shared memory segments to the inactive_dirty list.

The only solution I see is something like a "active_immobile" list, and add entries to that list whenever "writepage()" returns 1 - instead of just moving them to the active list.

Seems to be a simple enough change. The main worry would be getting the pages _off_ such a list: anything that unlocks a shared memory segment (can you even do that? If the only way to unlock is to remove, we have no problems) would have to have a special function to move all pages from the immobile list back to the active list (and then they'd get moved back if they were for another segment that is still locked).

Note that this would be solved very cleanly if the SHM code would use the "VM_LOCKED" flag, and actually lock the pages in the VM, instead of trying to lock them down for writepage().

That would mean that such a segment would still get swapped out when it is not mapped anywhere, but I wonder if that semantic difference really matters.

If the vma is marked VM_LOCKED, the VM subsystem will do the right thing (the page will never get removed from the page tables, so it won't ever make it into that back-and-forth bounce between the active and the inactive lists).

I'd really like an opinion on whether this is truly legal or not? After all, it does change the behaviour to mean "pages are locked only if they have been mapped into virtual memory". Which is not what it used to mean.

Arguably the new semantics are perfectly valid semantics on their own, but I'm not sure they are acceptable.

In contrast, the PG_realdirty approach would give the old behaviour of truly locked-down shm segments, with not significantly different complexity behaviour.

What do other UNIXes do for shm_lock()?

The Linux man-page explicitly states for SHM_LOCK that

The user must fault in any pages that are required to be present after locking is enabled.

which kind of implies to me that the VM_LOCKED implementation is ok. HOWEVER, looking at the HP-UX man-page, for example, certainly implies that the PG_realdirty approach is the correct one. The IRIX man-pages in contrast say

Locking occurs per address space; multiple processes or sprocs mapping the area at different addresses each need to issue the lock (this is primarily an issue with the per-process page tables).

which again implies that they've done something akin to a VM_LOCKED implementation.

Does anybody have any better pointers, ideas, or opinions?

It appears that the fine-detail semantics vary across the board. DYNIX/ptx supports two forms of SysV shm locking - soft and hard. Soft-locking (the default) merely makes the pages sticky, so if you fault them in, they stay in your resident set, but don't count against it. If, however the process swaps, they're all evicted, and when the process is swapped back in, you get to fault the back in all over again. Hard locking pins the segment into physical memory until such time as it's destroyed. It stays there even if there are currently no attaches. Again, such pages are not counted against the process RSS.

SVR4 only support one form. It faults all the pages in and locks them into memory, but doesn't treat the especially wrt rss/paging, which seems none too clever - if they're locked into memory, you might as well use them :-)

While I really like the idea with this patch, I'm 100% certain that Linus would not, under any circumstances, accept this patch.

I suggest that we instead force everyone to program with:

syntax on
let c_space_errors=1

(Or equivalent Emacs/[insert favourite editor here]-setting instead)

While at it, force people to read linux/Documentation/CodingStyle and make them adhere to it.

Of course, I guess this is a free world (yeah, right) and everyone should have the right to code in their own way, but I'd wish that people at least could be consistent when indenting/spacing/bracing/whatever, and when patching other people's code, also follow the already set standard of that file instead of introducing a new one...

Everyone laughs, I guess. The 2.0.39final didn't became the final release (could've told you so...) The good thing? Well, some bugs were found and removed. But this is it. Enjoy!

Changelog for v2.0.39

• Fix memory-leak in af_unix (Jon Nelson, Alan Cox, me)
• Added headerfiles for devfs to simplify backports of drivers (Richard Gooch)
• Fix a bug involving syncronous writes and -ENOSPC that could cause file-corruption (Jari Ruusu)
• Added new versions of PCI-2000 (Mark Ebersole)
• Added new versions of PCI-2220i (Mark Ebersole)
• Fixed a few typos in PCI-2000, PCI-2220i, PSI-240i and related files (me)
• Removed unused variable in xd.c (me)
• Renamed the initfunctions in pi2.c and pt.c, as their names clashed with paride-names (obviously, noone uses paride together with hamradio) (me)
• Changed most references to vger.rutgers.edu to vger.kernel.org (me)
• Fix the few vger.rutgers.edu references that I missed (Daniel Roesen)
• Fix a bug in af_unix that wrote to a socket after freeing it (aka the Win9x-related oops) (Michael Deutschmann)
• Fixed typo in Documentation (Martin Douda)
• IDE-patches (Andre Hedrick)
• Fixes for the IDE-patches (Andries Brouwier,
• Move memory-offset for dynamic executables (Michael Deutschsmann)
• Fixes to the Cyclades-driver (Ivan Passos)
• Fix for a bug in ext2 (Stephen C. Tweedie)
• Added marketing-names for 3Com NICs in drivers/net/Config.in (Yann Dirson, me)
• Fix for a buf in smbfs (Rick Bressier)
• Large-disk fixes (Andries Brouwer)
• Wavelan-driver cleanup & bugfixes (Jean Tourrilhes)
• Security-fixes (Solar Designer)
• Quota-fixes (Jan Kara)
• Fixed GPF using IPsec Masquerade (Rudolf Lippan)
• Fixed Config.in bugs in drivers/net and drivers/isdn (Marc Martinez)
• Added IPX-routing of NetBIOS packages (Jan Rafaj)
• Fix for a bug in paride (Wolfram Gloger)
• Fix an erroneous printk in ip_fw.c (Todd Sabin)
• Fix for IP multicast on WAN-adapters (Matthew Grant)
• Big updates to MAINTAINERS (me)
• Big updates to CREDITS (me, others)
• Various updates in Documentation/* (me)
• Styled up all Configuration-files in a similar manner to newer v2.3 kernels, and various other cleanups (me)
• Updated CodingStyle to the one used in recent v2.3 kernels (me)
• Backported nls_8859-14 (me)
• Added support for sparse superblocks (Theodore T'so)
• Fix for the ping -s 65468 exploit (Andrea Arcangeli, others)

There *should* be a patch for 2.4 final:

linux-2.4.0-ia64-010109.diff.(bz2|gz)

If not, your mirror isn't up to date.

oprofile is a low-overhead statistical profiler capable of instruction-grain profiling of the kernel (including interrupt handlers), modules, and user-space libraries and binaries.

It uses the Intel P6 performance counters as a source of interrupts to trigger the accounting handler in a manner similar to that of Digital's DCPI. All running processes, and the kernel, are profiled by default. The profiles can be extracted at any time with a simple utility. The system consists of a kernel module and a simple background daemon.

Typical overhead is around 3 or 4 percent. Worst case overhead on a Pentium II 350 UP system is around 10-15%

You can read a little more about oprofile, and download a very alpha version at :

http://oprofile.sourceforge.net/

oprofile is released under the GNU GPL.

This is really interesting. Great stuff.

As Alan had once suggested, it would be very interesting to have this information correlated with the content of the traces collected using the Linux Trace Toolkit (http://www.opersys.com/LTT). For instance, you could see how many cache faults the read() or write() operation of your application generated and other unique info. It would also be possible to enhance the post-mortem analysis done by LTT to take in account this data. You could also use LTT's dynamic event creation mechanism to log the profiling data as part of the trace.

There are definitely opportunities for interfacing/integrating here.

Let me know what you think.

There is a patch to the LVM kernel code which should help: ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.0ac4/lvm-fix-2

You should also get the LVM user tools from CVS (with TAG LVM_0-9-patches) to solve this problem. There will hopefully be a new LVM release soon.

This shows the following problems with khttpd:

1. Connect times are on average longer than boa. Why???

2. Transfers also take longer,

What is wrong here?

I applied the persistent khttpd patch + my vhost patch and now khttpd beats boa!!! (patch against 2.4.0 follows at the end of the message)

The connection times of boa are still better but khttpd wins in transfers.

I've never seen anything like it before, which I'm happy for. The system had been running a standard RedHat 7 kernel for days without any problems, but who wants to run a 2.2 kernel? I compiled 2.4.0 for it, rebooted, and blam! The RedHat init stripts got to the "remounting root read-write" point, and just froze solid.

Rebooting into RH7 failed, becauce inittab could not be found. In fact the filesystem was completely messed up, with /dev empty, lots of device nodes in /etc, and files missing all over the place. I had to reinstall RH7 from scratch.

I do not understand how this could happen during a remounting root rw. Is the filesystem really that unstable?

Am I right in suspecting DMA, which was enabled at the time? Any other ideas? Is it a known problem?

This is on a 450 MHz AMD-K6 with the following IDE controller:

00:07.1 IDE interface: VIA Technologies, Inc. VT82C586 IDE [Apollo] (rev 06)

Grab a tree from http://www.fsmlabs.com/linuxppcbk.html. Those always compile and are up-to-date.

I send patches, but they don't always make it into the main tree. In the mean time, you have a consistent source of kernels with the above web site.