Kernel Traffic #97 For 11 Dec 2000

Thanks go out to the many folks who wrote in to report that the printer-friendly version broke last week. There was not one flame in the entire bunch :-)! Sorry for not replying personally to every email, but I did read them all. The problem, finally, was a disk space shortage on one of our CVS servers, and it took a little while to straighten out. Thanks for the reports, folks!

There were 369 different contributors. 158 posted more than once. 147 posted last week too.

Rik van Riel said to Alan Cox, "here's a fix for a blindingly stupid bug that's been in 2.2 for ages (and which I've warned you about a few times in the last 6 months, and which I've even sent some patches for)." He explained, "This patch should make 2.2 VM a bit more stable and should also fix the complaints from people who's system gets flooded by "VM: do_try_to_free_pages failed for process XXX"" Ville Herva confirmed the problem, and described, "2.2.18pre19, uptime 8 days, machine had been idle for hours. Then, all of a sudden, I get 30 times "VM: do_try_to_free_pages failed for kswapd...", then 15 "VM: do_try_to_free_pages failed for xmms...", then "VM: killing process xmms" and that repeats for ~10 processes including X." But he also asked, "I saw Andrea's VM-global patch being recommended as a solution for this problem, and I already compiled it in (although I haven't booted into it yet). Should I use Rik's or Andrea's patch?" And also wanted to know if either of them would be going into 2.2.18. Rik replied, "On whether any of these improvements are going into the next 2.2, don't bother asking me since I have no intention paying much attention to 2.2" . There followed some technical discussion about Rik's patch and its interaction with ext3.

Mohammad A. Haque reported ext2 errors "Freeing blocks not in datazone" during normal use on 2.4.0-test11. Neil Brown replied, "Oh, good. It's not just me and Tigran then. I was at first blaming my raid5 code for this, but if you get it and Tigran gets it (reported http://boudicca.tux.org/hypermail/linux-kernel/2000week48/0257.html) then it's probably not me." He added, "Now if only we had a reliable way to reproduce it, we could start a binary search for the offending patch... but I can only reproduce it on a patched kernel after several hours of performance testing." Anries Brouwer replied, "You have it all backwards. It would be good if it were just you and Tigran. Unfortunately it also hits me." Alexander Viro started posting patches, though Mohammad and Tigran Aivazian had trouble reproducing the errors even on their initial setups. At one point, Neil remarked, "I ran my test script, which builds a variety of raid5 arrays with varying numbers of drives and chunk sizes, and runs mkfs/bonnie/dbench on each array, and it got through about 8 file systems but choked on the 9th by trying to allocate lots of blocks in the system zone (after running for about an hour)." Alexander replied, "Bloody interesting. I don't see anything recent that could affect the areas in question. Intersting versions to check: 11-pre5 and 11-pre6. It smells like buffer cache corruption, but I don't see anything relevant." Later he added, "Error messages would be interesting... So far we have _both_ 2.95 and 2.91 involved, raid and non-raid alike. Just fscking peachy..."

At one point Neil discovered that his report was unrelated to the others, saying to Alexander, "Turns out my data is a false alarm. It was a bug in my raid5 code - and not a recent bug either - that was causing my filesystem corruption. So if your earlier patches work for everybody else then they look like a good way to go. I have fixed my fatal flaw and I cannot reproduce the problems any more. Patch has gone to Alan."

Other folks also reported corruption, but no one managed to isolate it during the thread. For more on filesystem corruption in the developer series, see Issue #91, Section #7 (17 Oct 2000: Filesystem Corruption Bug Revisited)

Jeff V. Merkey posted a patch to modutils, to make the 'depmod' program compatible with Red Hat's scripts and programs such as Anaconda. In the course of discussion, he explained that his patch changed very little, while the alternative was to change all the hundreds of scripts that required it. David Ford said, "It's still a bad precedent. Anaconda should have been written correctly in the first place." Mohammad A. Haque added, "I'd rather have Anaconda changed rather than special casing standard utils to account for distro handling." Jeff argued, referring to Red Hat's alternate command line switch handling in depmod, "if RedHat has standardized on this set of switches, why not add them as alias commands? It's a trivial patch." David Ford replied, "Then let RedHat maintain their version of modutils. RedHat isn't the standard, nor should RedHat dictate to authors, nor should other distributions and persons be affected by RedHat's methods. If you don't like it, replace your depmod with a script that strips that flag before calling the original depmod."

Elsewhere, Keith Owens (modutils maintainer) said, "I have a big problem with Redhat. They make incompatible changes to utilities, do not feed patches back to maintainers then expect the rest of the world to follow their lead. The -i and -m flags to modutils are not the only example, I recently found IA64 and Sparc patches they had added to modutils code and not bothered to tell me. Other distributors are much better about sending me patches, Debian and SuSe in particular do the right thing." He added, "the "-m -i" patch is unnecessary. Consider this my protest against bad habits by distributors, they created the mess with their lack of communication and they have to fix it." Jes Sorensen replied:

I think you are pointing out a very valid problem. The same problem exists within the kernel, I see it every so often that someone decides to hack one a drivers and send the patch to Linus without bothering to even Cc the author a copy. Sometimes this is 'just' to make it compliant with the latest development kernel but Cc'ing the maintainer is not too much to expect.

I would also like to encourage people to contact a maintainer if they want to make extensive changes to a bit of code someone else maintains. It is not uncommon that the maintainer already has an idea about how to do something and might even be working on it. It is a waste of his/her (and other peoples') time to have two conflicting development like this going.

Kernel Traffic first covered this in Issue #95, Section #2 (13 Nov 2000: Serious Dell 5000e Laptop Power Management Problems) , when no one seemed very hopeful about getting APM working properly on Dell's 5000e laptop. By Issue #96, Section #14 (22 Nov 2000: Dell 5000e: The Saga Continues) , it seemed as though the manufacturers were working on an updated BIOS that would solve the problem. This week, Brad Douglas reported:

Alan, here's the DMI info you requested. Sorry about the delay.

The BIOS listed is a new test BIOS that has a *corrected* APM that I received this morning. I really want to take a second to thank the people at Compal (BizCom) for the short turnaround once we figured out who the right people were to talk to.

Once I get the OK from Compal (and finish testing), I'll post it to the Tuxtops support site for all to download.

Linus Torvalds announced 2.4.0-test12-pre2 and said, "Due to the birth of my third daughter last week (yes, I got /.'ed), if you sent me patches that aren't in pre2, you can pretty much consider them lost." A lot of people offered their congratulations, and Neil Brown asked kernel-wise, "What happens about the stuff that went in to 2.4.0test11-ac{1,2,3,4}? Are you going to "sync-up" with Alan, or should we send bits directly to you?" Alan Cox replied:

When Linus puts out pre3 I will start sending him stuff from my tree which proves workable. Stuff that seems suspect and needs more work I'll keep in the -ac tree and continue to release it against current Linus code.

It doesnt cause me any problem if you send Linus a copy, I'll just drop it from my patches as it appears in his tree.

Alan feeds me his patches in small chunks anyway, and does a good job of keeping stuff separate. Re-sending directly to me means that Alan would just drop that part of the patch - or that I'd get the patch twice. Both of which work ok, as long as it's the _same_ patch.

If you've made modifications since sending the stuff to Alan, you should synchronize with Alan too - just to make sure that I don't en dup applying the old stuff through Alan.

Kurt Garloff reported an infinite loop in modutils-2.3.21, where PPP over ethernet recursed endlessly in the build_stack() function. He included the 'modules.dep' file to produce the behavior, and added, "There is a circular dependency of pppoe on pppox on pppoe on .... modprobe has code to detect this in build_stack(), but it seems to not work. The old dep is thrown away and the new one is taken. And checked for dependencies again :-("

Keith Owens (the modutils maintainer) replied, "The kernel code is broken. Circular dependencies make no sense, the pppoe maintainer agrees and I thought that bug was fixed." Rod Stewart replied, "It is fixed in test10/11." Regarding the modutils code, Keith had also added in his same post:

Circular dependencies are not supported, nor are they correrctly detected. The existing code to walk the inter module relationships, including dependencies, alias, probe, probeall, before and after statements is a mess. It just grew over the years with special cases being added and is not robust.

In modutils 2.5 the entire code will be discarded and replaced with a standard graph walking algorithm with loop detection and back tracking instead of special case code. That might change some modutils behaviour in rare cases and I do not want to change its behaviour just before kernel 2.4 is released. I have a list of changes that might break backwards compatibility waiting for modutils 2.5, this is just one of them.

Kurt said he was looking forward to modutils 2.5, but he felt that even before that release, "the current behaviour is not acceptable, as it can kill the machine by just being invoked by kmod. I will try to make sense out of the code and make sure that modprobe will not go crazy, by either detecting loops (if I can do that in a way wihtout breaking things) or by limiting the recursion depth. I'll send you a patch." He posted a patch the next afternoon, adding, "As the dependency generation looked indeed rather cumbersome to me, I didn't really touch it. I just did implement the recursion limit to prevent the modprobe process grabbing all the memory of the system ..."

Tigran Aivazian reported happily, "Just to let people know that 2.4.0-test12-pre3 behaves much better than earlier versions on my 6G RAM machine. Not only /proc/mtrr is correctly showing all 6G cached for write-back but also I so far I never had to up/down one of the eepro100 interfaces to get it to work -- something I hda to do in all previous versions. (without david-mtrr.patch)" And Boszormenyi Zoltan replied:

Excellent! :-))))

BTW what test12-pre2/3 contains is David Wragg's work, updated to HPA's CPUID code that is in test11. Linus incorrectly attributed to me the whole patch in test12.log.

John Fremlin reported that the Yamaha OPL3-SAx sound card "is currently broken for people whose BIOS used to activate it with ISA PnP, as the kernel now decides to deactivate it." He'd sent a patch to the maintainer a month before, with no reply. He posted the patch (against 2.4.0-test10-pre6 and test12-pre3) again to linux-kernel, and explained, "This patch implements ISA PnP probe and activate/deactivate for the OPL3-SAx. As I don't have the specs for this card, I only know that it works for me; nevertheless, it should not break any configurations as the PnP probe only kicks in if the resource parameters are not given as module arguments. It should now be possible to compile the driver directly into the kernel." Scott Murray replied, "As the maintainer in question, I apologize. I've been remiss in getting a patch into 2.4 due to being focused on a new job. My current plan is to take Friday off and work on getting all of the various fixes people have sent me glued together into one patch. I must admit, though, that I've been running 2.4.0-test kernels for quite some time and have not had any problems activating my OPL3-SA3 card the old-fashioned way with isapnp." He found some problems with John's patch, and reiterated that he'd try to get a comprehensive patch going soon.

Richard Pries posted a patch, and explained, "Currently, mmc_ioctl() in cdrom.c is passed a cdrom_msf structure when ioctl() is called with CDROMREADRAW, CDROMREADMODE1, or CDROMREADMODE2 as its second argument. This structure does not provide the required buffer for reading the data, and it does not correspond to the structure that cdrom.h says to use with these ioctl() calls. This patch replaces the cdrom_msf structure with a cdrom_read structure (as specified in cdrom.h). Preliminary tests indicate that this patch works for both IDE and SCSI drives." But Jens Axboe replied, "Sure I bet it works, but you just broke all the programs that currently use any of the above ioctls. I've known about this for years... You can do all that you want with cdrom_msf, it's just more hassle. For 2.5 I'll introduce newer variants of the above ioctls and keep them as-is for compatability, tossing them out is not an option." He added, "I will take a patch that corrects the comment though!" Elsewhere, under the Subject: [Patch] Correct cdrom.h comments., Richard posted a patch to correct the misleading comments in cdrom.h that had led to his earlier patch. Jens thanked him and applied it.

Minutes after slashdot ran their article saying that the Transmeta recall was limited to about 300 Fujitsu computers, I ran to Fry's and bought a Sony PictureBook PCG-C1VN. Thank heavens for those extended Christmas hours I thought, while praying that the statements about the Crusoe problems being that limited would turn out to be true.

This device is the only commercially available computer in the world that uses a processor made by Transmeta (a 600MHz TMS5600, stepping 03). I thought surely that there would be a little subculture of Linux PictureBook users at transmeta making sure that this particular combination would work.

Well, alas, it appears that linux-2.4.0-test12-pre3 freezes hard while reading the base address registers of the first PCI device (the "host bridge"). Actually, I think the problem is some kind of system management interrupt occuring at about this time, since the exact point where the printk's stop gets earlier as I add more printk's. With few printk's the printk's stop while the 6th base address configuration register is being read; with more printk's it stops at the second one, and it will stop in different places with different boots, at least with the not-quite-stock kernels that I usually use. Also, turning off interrupts during this code has no effect, so I do not think it is directly caused by the something in the PictureBook pepperring the processor with unexpected interrupts (I thought it might have to do with the USB-based floppy disk).

Although the results of the debugging printk's that I added from a somewhat modified linux-2.4.0-tset12-pre3 built for CONFIG_M386, I also tried "pristine" linux-2.4.0-test12-pre3. When built with CONFIG_M386 (which has historically been the way to get a kernel that runs on all x86 processors), I get no output or other apparent activity after the boot loader jumps to it. When buid with CONFIG_MCRUSOE, it hangs after printing "PCI: Probing PCI Hardware", just like our kernels (which, oddly, do work up this point even though they are build with CONFIG_M386). In case anyone is curious, I have put the .config file from the pristine CONFIG_MCRUOSOE build in ftp://ftp.yggdrasil.com/private/adam/linux-crusoe/.config.

My initial attempts to find a processor manual on the tms5600 on the web and on Transmeta's web site have no yet turned up anything, and I understand that the tms5600 includes the north bridge. So, I assume that that would be the first place to look for ideas about any weirdness that occurs during PCI initialization of the PCI host bridge.

One sin that I am committing in these builds is that I am bulding them under gcc-2.95.2, although I do not think this is the sort of behavior that an optimizer bug is likely to produce.

If anyone out there is using Linux 2.4.0-test on a Sony PictureBook PCG-C1VN (the Transmeta version), I would be interested in at least trying to build from your .config file.

H. Peter Anvin explained, "It's a slight bug in the Linux PCI probing code that triggers when there is ongoing DMA activity during PCI probing. Linus already have a fix for it; I expect that it will be in the next prepatch." And Linus Torvalds also replied to Adam:

This is due to a Linux bug, where we disable the northbridge while we do the PCI window probes.

[ I actually suspected for a while that we'd messed up at Transmeta, but after talking with and double-checking the PCI specs, it turns out that Linux really was at fault. Oh, well. Whichever way I turn, I'm always to blame ;) ]

What happens is that the Sony notebook has Legacy USB support on in the BIOS, which causes USB DMA events several thousand times a second. When Linux does PCI probing, Linux will turn off the MEM and IO bits in the PCI command register of the device it probes. It so happens that according to the PCI spec, turning off the MEM bit of the host bridge (aka "northbridge") disconnects the host from the PCI bus.

A few microseconds later a USB legacy DMA event comes in, but now the host bridge no longer forwards the DMA between the PCI bus and memory, and the machine hangs. Oops.

The simplest (working) solution is to remove the jiggering with the PCI command register IO and MEM bits in drivers/pci/pci.c: pci_read_bases().

The proper fix (which we discussed with Martin Mares and Richard Henderson) is actually to do the full bus enumeration first, _without_ doing any window probes (and thus without having to muck with the IO and MEM bits in the command register), and when we find the offending USB controller that the BIOS has left active, we kill it off first (we already have this in the PCI quirks section, it's just that the PCI quirks get executed too late to fix this problem as it is now).

Adam had also suggested that Transmeta get Linus one of the PictureBooks, and Linus replied:

Actually, I have one, and have had one for about two weeks, but because of the newest (human) addition to the Torvalds family I didn't have any time to debug this until the day before yesterday.

NOTE! Getting the 2.4.x kernel up and running is the easy part. The machine also has a very recent ATI Rage Mobility chip in it, and you need the newest XFree86 CVS snapshot to make it work (along with a one-liner patch from me, unless that has already made it into the CVS tree by now).

Even then XFree86 does something bad with DPMS, and will lock up the graphics chipset when it tries to shut down the flat panel display. Solution: don't enable DPMS is XF86Config. That's an XFree86 problem, but happily easily worked around.

Oh, and there's a UHCI driver bug that will bite you (again because the machine has legacy USB enabled by default - and unlike almost every other laptop out there, Sony didn't allow USB legacy code to be turned off in the BIOS setup), so unless you've applied the USB patches from the USB mailing list you'll just hang there instead.

Anyway, I do have this machine working now, although not everything is to my liking. Unlike older picture-books, for example, this one has a WinModem. Ugh. And the sound chip is supported, but only by the ALSA driver (the OSS version is too broken to be used).

But the camera is cool, and works beautifully (once you get XFree86 happy) thanks to Andrew Tridgell. (If I could just coax the X server into giving my a YUV overlay I could play DVD's with this thing).

Several folks gave Linus some tips on how to get stuff working, and by the thread's end he seemed to be having good success with it.


Kernel Traffic Latest \| Archives \| People \| Topics	Wine Latest \| Archives \| People \| Topics	GNUe Latest \| Archives \| People \| Topics
Czech

1.	18 Nov 2000 - 30 Nov 2000	(12 posts)	Fix For Longtime 2.2 Virtual Memory Bug
2.	22 Nov 2000 - 28 Nov 2000	(42 posts)	More 2.4 Filesystem Corruption
3.	26 Nov 2000 - 27 Nov 2000	(18 posts)	Linux Distros Making Incompatible Changes To System Tools
4.	27 Nov 2000 - 28 Nov 2000	(4 posts)	New BIOS For Dell 5000e APM Problems: The Saga Continues
5.	27 Nov 2000 - 3 Dec 2000	(13 posts)	Linus' Daughter
6.	28 Nov 2000	(5 posts)	'modprobe' Infinite Loop On Buggy Drivers
7.	29 Nov 2000	(2 posts)	Success Report For Big Memory Machines
8.	29 Nov 2000	(2 posts)	Slow Maintenance Of Yamaha OPL3-SAx Sound Driver
9.	30 Nov 2000	(4 posts)	Longtime struct Weirdness And Doc Bug; True Fix Planned For 2.5
10.	1 Dec 2000 - 2 Dec 2000	(14 posts)	Linux On Transmeta Chips

Kernel Traffic #97 For 11 Dec 2000

By Zack Brown