Kernel Traffic #38 For 11 Oct 1999

Abit BP-6 dual 366 celerons, 128MB PC100 RAM, redhat 6.0, 12 ide disks attached (to primary PIIX4 and three PCI PDC20246's... note, I am NOT using the onboard HPT-366), 3com 3c905b 100mbit ethernet

I have two other very similar systems, both Abit BP-6 dual celeron 366, 128MB RAM, 3c905b's, redhat 6.0, 2.12.12 but using Tekram 390F controllers and UW SCSI disks + raid, and they are both 100% stable.

Asus P2B-D
+ 2 * Promise UDMA/66 controllers
+ 1 * Adaptec 2940UW
+ 1 * 3com 3c905b
and dual p2-450 / 512mb ram
+ 1 * 9.1u2w ibm scsi hd
+ 5 * 25gb ibm ide hd (1 on the onboard controller
and 4 on the promise controllers)

Dual P2 350
mixed IDE + aic7xxx
2 x 4.5GiB IBM SCSI
1 x 2GiB IDE

I know about this, but not a clue to resolve the issue. I have brought up the issue with Promise and they are working with me. There is a duplicate issue with M$'s NT, I hope to get the method of resolution to this problem and not "miniport" code.

First test to verify the problem, build another kernel that is UP. Include raid and rag the hell out of the box. This should not fail, based upon other test reports.

If the box throttles in UP, the driver core is RAID UP Stable.

Second, disable RAID under SMP and push a sequencal access on the last two channels.

What is the irq routing table for this board? Did you remove all the Promise BIOS chips execpt for the one that registers hde/f/g/h ?

Have you patched with "ide.2.2.12.19990921.patch.bz2"? New Promise OEM support.

You are required to set these for more than two cards.
CONFIG_BLK_DEV_PDC202XX=y
PDC202XX_FORCE_BURST_BIT=y

Ok, so what have I been doing since I first reported this?

The box that I reported these lockups against is basically a production box. It's an 80 gig RAID array, so you know we must be using it for something ;) To make things worse, it is colocated at an ISP. After I reported the problems, I pretty much had to set it to UP mode and let it run.

However, I have two other abit BP6 based development systems, and my intent is to try to duplicate these results with one of these. They are local to me, and can be fucked with without worries.

They are both Tekram based SCSI system, however, so some modifications were needed. I have temporarily claimed two 14G IBM 7200 udma33 drives for testing, and I ordered two PDC20246's to put in to round out the picture.

So far, I haven't been able to get this configuration to crash. But I fear it's for lack of volume running through the controllers. I hooked up one IBM drive each to the two PDC controllers, and ran it with equal load on the hard disks, but no crashes.

I am going to break apart the rest of my computers here in an attempt to borrow three more IDE drives (two maxtors, and a quantum). I'm going to hook these up so that I have a disk as hda, and one disk on each of the promise channels. I'll then test with the vanilla kernels and with the ide.2.2.12.19990921.patch.bz2 patch and see if I can get it to crash. My guess is that 5 drives, working most of the channels... will be able to produce the same results.

I'll be doing these tests in the next few days, so I will relay my results.

As an update, I have succeeded in getting both of the boxes to crash very repeatedly. The trick turns out to be running drives on both interfaces on a PDC20246. With SMP enabled and concurrent access to both disks, the machine locks up almost immediately. It doesn't have to have more than one PDC20246, or more than two ide drives attached.

Though the normal kernel doesn't produce an oops, I am able to get an NMI OOPS with 2.2.12 + ikd + NMI oopser. However, I must obtain a null modem cable to capture the OOPS before I can decode it an forward the results. Hopefully that will be tomorrow.

kernels that crash this way: (all smp) 2.2.12 vanilla, 2.2.12 + ide, 2.2.13pre12, 2.2.10

kernels that don't crash: (any UP), anything pre 2.2.6 (I will also test 2.2.7-9 for my full report)

I write prematurely just in case you are working on tracking this down. Since I've done it on three machines now, it might be reproducible on your hardware Andre. I think SMP + PDC20246 with two drives attached will cause it reliably.

Like I said, I will write as soon as possible with more kernel tests and the decoded OOPS that ikd generates.

I just now have a way to produce the crash as of tonight. What I need to know is what parallel races are we referring/hunting and against which kernels?

Alan just pointed out one location last night. It is now 10:00pm here and I have about 4 hours of brain left. I will check back in about 15 minutes. I am off to grab AC's latest and see what "Andrea A." has to offer about finding the races.

I have seen one race. At this point it appears to me that any time you have concurrent access to two ide channels sharing the same interrupt while running SMP. As you see below, I have duplicated with the hpt-366 controller as well.

I have tried:

2.2.6 (with or without ide patch): works in SMP mode

2.2.7 (same): works in SMP (as I remember, haven't tested recently)

2.2.[89] : haven't tested

2.2.10 and later: crashes in SMP mode

2.2.12 +

2.3.18: crashes in SMP mode

It would seem that is actually a generic ide/smp bug, and not one that is promise controller specific. I was able to cause the same behavior tonight with the onboard hpt-366 controller.

kernel: 2.2.12 + ide.2.2.12.19990925.patch.gz + 2.2.12-ikd1.gz

Tested normal crashing setup, one drive on each channel of pdc20246. Got normal ikd NMI oopser oops. Looked like the other one I reported. oops is attached as text file.

Then I moved the drives, one per channel to the onboard hpt-366. I had previously commented out the two lines in ide-pci.c at line 630 that say:

if (dev2 && hpt363_shared_irq)
   return;

so that I could enable the second channel for the test.

I did my standard crashme (which I have simplified to 'dd if=/dev/hdi of=/dev/null & dd if=/dev/hdk of=/dev/null' you only need one block each to trigger the crash) and got what looks like to me the same lockup as the pdc20246 with the hpt-366. This oops is also attached.

I retested this configuration in UP mode and found it completely stable even with the 2nd channel disabling removed. My abit bp-6 bios is the original LP revision, I have never flashed. Board is a "newer" board, bought about 8/1... it has plastic cpu handles as opposed to metal, like my first one bought in early june. I cannot see any silk screening indicating revision on the board.

I was thinking that this might have caused your impression that there is a buggy chipset/revision out there that needs this 2nd channel disabled on the hpt-366. Or is it two bugs? One is the motherboard, and if that's OK you still have the multichannel + smp bug?

Ok whats happened is this

                CPU0                            CPU1
          take hwgroup spinlock
                                        take an irq
          disable_irq
          [wait for IRQ completion]
                                        try to take hwgroup spinlock

There is a 0.01 release which came out 1-2 weeks ago. It was against 2.2.2, and has bugs which have since been fixed. I imagine that Stephen will be releasing a new version fairly shortly.

So the good news is that most of the code has been written but we still need to do some bugfixing (and bug finding) before it will be completely production ready.

0.0.2 over the weekend is the plan. I've fixed all of the known bugs except for one, but that one requires a bit of a reworking of how I track committed buffers which are still needed by the transaction code. Once that is done I expect to have a version which is pretty stable against the 2.2.2 kernel. I already have a 2.2.12 version of the 0.0.1 code done so merging the 2.2.2 fixes into the 2.2.12 stream should not be hard.

Currently the released journaling code journals everything, including data. That is deliberate: journaling metadata only turns out to be _lots_ more complex (hint---what happens if you delete a block of metadata and reallocate it as data or vice-versa), so although all of the necessary support is there in the journaling layers, the released ext3 code does not do metadata-only journaling yet.

Once people have had a chance to thrash out any bugs left in the 0.0.2 code, I'll enable the metadata-only journaling and we can all watch the pretty fireworks as filesystems all over the planet explode in a curious and interesting manner...

That isn't finalised. I _might_ make the ext3 journal use a reserved inode, but right now there isn't actually a difference between ext2 and ext3. You just create a journal file on the ext2 filesystem and tell the kernel where to find it. Then when you mount it as ext3, everything is automatically journaled to that file.

If you uncleanly shutdown, then the ext3 filesystem will have a compatibility flag set in the superblock to prevent you from remounting it as ext2 without doing a recovery step first. Once that is done, it becomes mountable as ext2 again.

The problem is that different, already established filesystems: AFS, Coda, NTFS, etc., all have different ACL semantics. For example, AFS only has an ACL on a per-directory basis. I'm not sure about Coda, but it may be the same as AFS. NTFS uses 128 bit UUID's in its ACL's to name users and groups. The POSIX acl interface uses uid_t and gid_t for user and group id's.

So it would be *nice* to do this, but there's quite a lot of design work to make the interfaces similar enough that a single interface could be used at both the UI and system call level. I won't say that it's impossible, but it's definitely non-trivial.

I've got insomnia so you get a long answer.

First, on a literal level, leaving out a "then" keyword is not a "little irrelevant change". Documentation/kbuild/config-language.txt is the specification for Config Language. Based on legacy "bash" syntax, the Config Language syntax specifies that an "if" statement must have a "then" keyword.

So I think the real question is "how come Menuconfig doesn't print out informative syntax errors?" Because if it did, then the people who write Config.in scripts with syntax errors would get error reports before submitting their patches to Alan or Linus.

The problem goes back a long, long ways to the use of ad hoc scripts for kernel configuration.

First there was some old 'Configure' shell script, which I've never seen. Then sometime in 1993, or even earlier, Raymond Chen of Microsoft wrote the current 'Configure' script.

Next, Eric Youngdale wrote a very kludgy Config.in->tk translator, which works OK most of the time. So you have a choice: you can run the reliable but awkward text-mode script written by a Microsoft engineer, or you can run the flashy, crashy windowing version from somebody else. (Ironic, isn't it?)

After that, William Roadcap (not from Microsoft) wrote Menuconfig, which is a super kludge of a language interpreter, but wow does it have a nice curses interfaces. Mr. Roadcap disappeared, I took responsibility, I fixed a lot of bugs in Menuconfig, and then I semi-disappeared too.

These ad hoc script interpreters simply do not scale up to the size of the development community. Some poor guy trying to add something to the network drivers omits a required keyword and instead of a nice syntax error, dozens of people get a mysterious message. As you noticed, this happens several times a year.

Here is the second Microsoft-ish irony:

<flame-bait>

Linux developers have high standards for the work they produce, but when it comes to the tools they use to do their *own* work, they value features and familiarity and speed more than correctness.

</flame-bait>

Don't believe me? Read scripts/Menuconfig, figure out what the hell it's doing (including the magic awk script in the middle), and then ask yourself if Linus would knowingly accept a driver or a filesystem that behaves so randomly in the presence of errors. But the kbuild procedure is full of this kind of shit: CONFIG_MODVERSIONS, "make modules_install", three different configuration parsers, configuration parameters tucked into the Makefiles.

I believe the right fix is a real grammar, written in bison, with real syntax checking and real error messages. I have one in progress with a generic back-end and a curses front-end. I've actually configured and built kernels with it. Jim Bray is working on a gtk front-end for it.

The code's been available for eight months now and I've gotten some interest in it. Menuconfig is not well-loved among its captive audience (more Microsoft parallels).

My new mconfig runs in about 5% of the CPU time of Menuconfig. (I developed it on a 486, a *slow* 486, so I made it fast, *very* fast). But forget about the speed. The nicest part of mconfig is that when I ran it on 2.3.18ac9 tonight, it said:

    net/Config.in: 70: parse error

What I really need now is a block of hacking time, about 2-4 weeks long, to bring it to a usable level. I expect to use my winter vacation to do this.

Then I am going to try to put Menuconfig out of business. Also Configure and Xconfig, too. Not by taking them away from people -- but by offering something that they prefer to run so that the old ones die from lack of users.

Work in progress:

    ftp://ftp.shout.net/pub/users/mec/experimental/mconfig-0.15.tar.gz

I actually *don't* want much feedback on this yet. I'm just putting it up so people can see the direction I'm going.

they are fixing the symptoms. Windows NT apparently has problems with high contention spinlocks. So instead of reducing the number of spinlocks and reducing lock collisions, they decided to optimize the 'contention case'. This is a step in the wrong direction, as it makes the 'no contention' case slower. No amount of trickery is going to avoid the spinning CPU to waste precious cycles on pure spinning! The 'no contention case' in Linux is highly optimized, it's only 2 inlined assembly instructions to aquire, and 1 inlined assembly instruction to release. Windows NT is digging itself into a deeper and deeper architectural hole me thinks ...

Linux is doing another trick here to reduce bus traffic if lock contention happens: once we go into the slow path we do not do interlocked atomic instructions to poll the state of the spinlock flag, but normal memory access instructions - this also ensures that we generate cross-cache bus traffic only when the spinlock is released. This way we basically get the kind of 'good' (nonintrusive) bus traffic what queued spinlocks are supposed to do primarily - without the overhead of queued spinlocks.

(btw. NT doesnt do the kind of off-line spinlock slow-path thing Linux does, no wonder they see high spinlock overhead. NT calls functions to aquire/release spinlocks, yuck!)

NT designers i believe also have made a mistake with the 'processor-local area' thing which they currently implement through a special segment and %fs. Not only is it slower on x86 (%fs access is a bigger opcode and doesnt optimize as well within the CPU), but they'll also have to waste a whole register on that in IA64 ... Apparently that David Cutler guy has already left the building? ;)

in the semaphore case the picture is completely different - semaphores (mutexes) naturally want to have the kind of queueing characteristics described above. Linux 2.3 does this and more (we do wake-one on semaphores). Linux semaphores are still very lightweight in the no contention case: 2 inlined assembly instructions to aquire and 2 inlined assembly instructions to release.

anyway, Linux tries to have the highest quality SMP core architecture possible physically - if you can poke holes into it, feel free!