KDB is putrid. Can it debug double faults? NO. Can it debug complex register and numeric evaluation statements like IF ((EAX == 1) && [ESP-4] == 0x3000)? NO. Can it debug nested task gate exceptions? NO. Can it debug SMP locks races? NO. Can it debug priority inversion problems in sleep locks? NO.

Can the Kernel debugger in NetWare? YES. Can the Kernel Debugger in NT? YES.

Remote gdb on Linux - yes and I can do my debugging source level. Unfortunately Linus seems to have a one man campaign against putting sensible debugging into his kernel.

The tools exist and they should be in the x86 tree as well as sparc etc where with other maintainers it is present

And I bet the lessons learned and the issues involved in those nights with printk will never leave your brain, you will remember precisely in the future next time you see the same types of symptoms what kinds of things to look for and where.

This is what a debugger does not do for you. The debugger allows you to be lazy, step around, "oh yeah check for NULL" and never have to _think_ about what you're doing or the changes you're making or even if the same bug might be elsewhere.

This is why Linus does not allow a debugging facility like this into the kernel, so people spend time _thinking_ when they go hunting down bugs.

It takes longer to nail a bug, yes, but the resulting fix is always far superior. And the person who discovers the bug leaves with a much larger amount of knowledge about how that area of the kernel works.

There are two basic types of 'debugging behaviors':

A) 'Collect enough data to understand the local context and fix the bug.'

B) 'See the point of the immediate crash, think about that code, think about code that called this code, think about the intent and implementation of that code and find the bug based on understanding the code.'

A) results in people being able to debug easy and moderate kernel bugs. The same people are lost if faced with bugs that are not apparent in the 'state of the system around the bug' represented by the debugger.

B) results in people with the ability to identify bugs based on the source code - and the ability to fix the really mindblowing tough bugs. We had about 10 such bugs during the cycle of the 2.3 kernel.

I do claim that the real mainsteam-kernel development 'bottleneck' is the ability to fix the tough bugs. It's always these tough bugs that keep up the addition of new features in devel kernels. We should thus do everything to teach people the proper debugging methodology.

A) takes less time for the easy bugs - it might also speed up driver development. But it is utterly useless for the really tough problems. We had really tough bugs in 2.3 that one couldnt find even with the heaviest hitting debugging equipment. Debugging code in the kernel also blurs the intention of the code, which makes B) harder.

B) is a slower process to both learn and practice, but will also lead to more (unrelated) code improvements (based on better understanding of various kernel subsystems) and ultimately leads to better kernel developers.

one problem is developer laziness ;-) We could as well include debugging code so that experienced people like you can fix easy / moderate bugs quicker. But the problem is that in this case people are not forced to understand the code as much as if the debugging tools were 'minimalistic' like today.

i have nothing against (in fact contributed to) independent debugging functionality like IKD / KDB. It's slightly more pain to keep it uptodate with the devel kernel, but the devel kernel itself must stay focused on the strategic goals, not the tactical ones.

I don't like debuggers. Never have, probably never will. I use gdb all the time, but I tend to use it not as a debugger, but as a disassembler on steroids that you can program.

None of the arguments for a kernel debugger has touched me in the least. And trust me, over the years I've heard quite a lot of them. In the end, they tend to boil down to basically:

• it would be so much easier to do development, and we'd be able to add new things faster.

And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug. I do not think that extra visibility into the system is necessarily a good thing.

Apparently, if you follow the arguments, not having a kernel debugger leads to various maladies:

• you crash when something goes wrong, and you fsck and it takes forever and you get frustrated.
• people have given up on Linux kernel programming because it's too hard and too time-consuming
• it takes longer to create new features.

And nobody has explained to me why these are _bad_ things.

To me, it's not a bug, it's a feature. Not only is it documented, but it's _good_, so it obviously cannot be a bug.

"Takes longer to create new features" - this one in particular is not a very strong argument for having a debugger. It's not as if lack of features or new code would be a problem for Linux, or, in fact, for the software industry as a whole. Quite the reverse. My biggest job is to say "no" to new features, not trying to find them.

Oh. And sure, when things crash and you fsck and you didn't even get a clue about what went wrong, you get frustrated. Tough. There are two kinds of reactions to that: you start being careful, or you start whining about a kernel debugger.

Quite frankly, I'd rather weed out the people who don't start being careful early rather than late. That sounds callous, and by God, it _is_ callous. But it's not the kind of "if you can't stand the heat, get out the the kitchen" kind of remark that some people take it for. No, it's something much more deeper: I'd rather not work with people who aren't careful. It's darwinism in software development.

It's a cold, callous argument that says that there are two kinds of people, and I'd rather not work with the second kind. Live with it.

I'm a bastard. I have absolutely no clue why people can ever think otherwise. Yet they do. People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system.

And I'm not just saying that. I'm really not a very nice person. I can say "I don't care" with a straight face, and really mean it.

I happen to believe that not having a kernel debugger forces people to think about their problem on a different level than with a debugger. I think that without a debugger, you don't get into that mindset where you know how it behaves, and then you fix it from there. Without a debugger, you tend to think about problems another way. You want to understand things on a different _level_.

It's partly "source vs binary", but it's more than that. It's not that you have to look at the sources (of course you have to - and any good debugger will make that _easy_). It's that you have to look at the level _above_ sources. At the meaning of things. Without a debugger, you basically have to go the next step: understand what the program does. Not just that particular line.

And quite frankly, for most of the real problems (as opposed to the stupid bugs - of which there are many, as the latest crap with "truncate()" has shown us) a debugger doesn't much help. And the real problems are what I worry about. The rest is just details. It will get fixed eventually.

I do realize that others disagree. And I'm not your Mom. You can use a kernel debugger if you want to, and I won't give you the cold shoulder because you have "sullied" yourself. But I'm not going to help you use one, and I wuld frankly prefer people not to use kernel debuggers that much. So I don't make it part of the standard distribution, and if the existing debuggers aren't very well known I won't shed a tear over it.

Because I'm a bastard, and proud of it!

I've done an implementation of some of the Win32 "system calls" in a kernel module in an attempt to speed up Wine.

The preliminary benchmarks that I made, while not very real-world since I don't think I have managed to implement enough for that yet, seem to indicate that in some tests, I can beat Win2000 by 20% on syscall latency, and Wine by 900%.

Wine is this slow, I think, because it uses an extra process (the wineserver) to implement synchronisation and handle<->fd mapping, and involves the use of sendmsg and recvmsg to communicate.

I'm interested in your comments on the code I've written (attached to the bottom of this document). I'm also interested in ideas as to how a large speed up might be implemented by some other user-space mechanism or by less intrusive kernelspace intervention.

I'm also interested in finding a better way of getting to kernel space from user space... Currently, this involves the client process opening a proc file and doing ioctl's on it to request Win32 operations (easy to do from a kernel module).

However, the ioctl syscall seems to incur a fairly large penalty in the number of standard I/O commands it checks for first, before calling the supplied service routine.

Other methods that have been mentioned include persuading Linus to reserve a syscall number specifically for this purpose, and having the module supply a handler for it.

Please by no way don't include this patch into the official tree. It's insane due to the following:

1. Linux is UNIX not NT... (in terms of API)
2. WINE in itself is barely usefull - even in fact non existant, since there is no official stable release out there.

Hmm.. I have this feeling that it would be much nicer to just implement the NT system calls directly.

We used to have the iBCS2 project, and I was actually considering making it part of the standard kernel when it started becoming a non-issue simply because there started to be more native Linux programs than iBCS2 programs.

And we've already had the vm86 mode and the BIOS trap stuff speedups for DOSEMU for a long time.

It looks like if you want to do this, it would be better to just try to do it outright, and have the same kind of "emulate the ones we know about and care about performance" in the kernel that we already have with the vm86 mode emulation.

I wouldn't be adverse to supporting Wine better - as long as it's fairly cleanly separated. This doesn't look too bad.

The problem is that dynamic system calls are not going to happen.

Why?

License issues. I will not allow system calls to be added from modules. Because I do not think that adding a system call is a valid thing for a module to do. It's that easy.

It's the old thing about "hooks". You must not sidestep the GPL by just putting a hook in place. And dynamic system calls are the ultimate hook.

I met with Cort for a couple of hours on Tuesday. I thought he was going to be back in NM sometime today (Wednesday), and I was hoping he would have responded to the messages of confusion by now.

We discussed how to better manage the kernel source trees, and it was pretty clear to me he intended to keep working on this. Like all of us, he has other jobs to do, and sometimes he just can't respond as quickly as we like. This kind of work has a multiplier effect as it moves upstream.

This note puts the case for including a kernel debugger in the master tarballs. These points do not only apply to kdb, they apply to any kernel debugger. Comments about the perceived deficiencies of kdb, kgdb, xmon or any other debugger are not relevant here, nor are questions about how or when debuggers should be activated. I want to concentrate of whether the kernel should have *any* standard debugger at all.

If Linus still says "no" to including any debugger in the master tarball then that will be the end of this thread as far as I am concerned. I will then talk to distributors about getting a debugger included in their kernels as a patch. Hopefully the distributors who want a kernel debugger can then agree on a standard one.

Disclaimer: Part of my paying job is to maintain kdb. SGI want kdb to be used more widely to benefit from GPL support. More eyes and hands means better code for everybody.

1. Random kernel errors are easier to document and report with a debugger. Oops alone is not always enough to find a problem, sometimes you need to look at parameters and control blocks. This is particularly true for hardware problems.
2. Support of Linux in commercial environments will benefit from a standard kernel debugger. The last thing we want is each commercial support contract including a different debugger and supplying different bug reports. Bug reports on supported systems should go to the support contractor but some will filter through to the main linux lists.
3. Architecture consistency. Sparc, mips, mips64, ppc, m68k, superh, s390 already have remote debugger support in the standard kernel. i386, alpha, sparc64, arm, ia64 do not have standard debuggers, they have to apply extra patches. Some architectures have extra debugger code in addition to the remote gdb support.
4. Debugger consistency. Back in 1997 there were a lot of individual kernel debugging patches going around for memory leaks, stack overflow, lockups etc. These patches conflicted with each other so they were difficult for people to use. I built the original set of Integrated Kernel Debugging (IKD) patches because I contend that having a standard debugging patch instead of lots of separate ones is far easier for everybody to use. The same is true of a kernel debugger, having one standard debugger that all kernels use will improve the productivity of everyone who has to support kernel code, no need to learn the semantics of multiple separate debuggers.

5. Easier for kernel beginners to learn the kernel internals. Having worked on 10+ operating systems over the years, I can testify that some form of kernel/OS tracing facility is extremely useful to get people started. I agree with Linus when he said

   "'Use the Source, Luke, use the Source. Be one with the code.'. Think of Luke Skywalker discarding the automatic firing system when closing on the deathstar, and firing the proton torpedo (or whatever) manually. _Then_ do you have the right mindset for fixing kernel bugs."

   But Linus has also said "The main trick is having 5 years of experience with those pesky oops messages ;-)". Beginners need some way of getting that experience. Reading the source from a cold start is an horrendous learning curve, debuggers help to see what the source is really doing. Always remember that 90%+ of kernel users are beginners, anything that helps to convert somebody from kernel beginner to kernel expert cannot be bad.

6. I contend that kernel debuggers result in better patches, most of the time. Sometimes people misuse a debugger, as Linus said

   "I'm afraid that I've seen too many people fix bugs by looking at debugger output, and that almost inevitably leads to fixing the symptoms rather than the underlying problems."

   Does that occur? Of course it does, I have been guilty of that myself over the years. Is it inevitable? IMNSHO, no. Seven of the twelve architectures in the standard kernel already have built in debuggers. Where is the evidence that these architectures have more bad patches because of the presence of the debuggers?

   Even if somebody does submit a patch to fix the symptom instead of the problem, that alone is valuable information. Fixing the symptom focuses attention and the associated information helps to fix the real problem. We get problem patches even without debuggers (let's not mention the recent truncate problems ;) but there are enough eyes on the kernel to find problem patches and remove them. Adding a standard debugger will improve the quality of some of those eyes at a faster rate.

I think the case for the kernel debugger is better stated as the case for RAS (Reliability, Serviceability and Availability) in the kernel, in other words, there is a case for having the right diagnostic, reporting and recovery tools in the right place at the right time. A kdb does not fulfil all diagnostic RAS needs. IMHO it's an extremely powerful developement tool, but hang on so is a logic analyser and a source-level debugger. It can also be a real pain if trying to debug HLL source using an assembler based debugger. The point is, one generally needs a debugger that matches the semantics that the programmer is dealing with. If its, assembler code the so be it, use a kdb. If you're poking around with H/W specific interfaces and system busses the you make need a lower level tool.

But should a kdb be a standard part of the kernel for use in production/commercial/enterprise environments? I don't believe so. Looking back at the techniques we've deployed over the years to debug system problems in commercial environments, we only ever had the luxury of using a kdb with OS/2. Just about every other OS we supported did not have a kdb. The OS/2 case is interesting because initially, we had only the kdb for debugging, and it was the worst platform for serviceability that we ever supported. We couldn't debug those typically obscure problems that occurred only in production environments and which could never be readily re-created in the lab. We took an enormous amount of pain over this from our customers over poor serviceability. They hated every minute of production time we took from them when a developer took control of their systems in order to debug, or in many cases not debug the problem. Of course we had created a rod for our own backs. Customers knew we never sent developers on site to debug OS/390 (or MVS as it was called in those days). They also knew that we never rejected a problem because it was not re-creatable and we never even asked for a re-creation scenario. The reason for this was that we had appropriate RAS capability in MVS which allowed data to be captured automatically at fault time combined with a certain amount of self-healing capability and automated recovery. What we did to OS/2 to make it approach this level of RAS capability was to implement a system dump capability - similar to SGI's kernel crash dump, + a comprehensive system tracing facility that could be dynamically customised to tracing events in any code path without any code recompilation - IBM's Dynamic Probes for Linux is an initial port of the capability + a comprehensive and customisable virtual storage based dump, a bit like core dump, except that it could dump process trees if required and memory from not only from user space, but from system space based upon kernel sub-component, for example file-system structures etc..

That capability completely transformed our ability to debug serious and obscure problems, with minimal disruption. It's true that we weren't immediately successful when we implemented this stuff. There's a major learning curve and mind-set change required to work with captured data as opposed to interactive debugging.

We didn't throw away the kdb, it's still very useful:

1. as a didactic tool.
2. for the final stages of problem determination - every problem is re-creatable once you know the triggers. And when you do, which you can get from dumps and traces, then you can set up a lab-based experiment where you use a debugger to solve the final mystery.
3. in production for those exceedingly rare cases where we needed to know what the underlying hardware was up to - it's a cheaper option than using a logic analyser.

One big argument against RAS of any sort is that it bloats the kernel and not every one wants it (until they have a problem). A further argument with Linux is that you may have to do quite a bit of hard work to get the subset of RAS you need to co-exist, if it exists at all. Something we're working on which may help resolve this, and will be made available with the next drop of Dynamic Probes is Generalised Kernel Hooks Interface (GKHI). The idea here is to make all our RAS function the option of being dynamically loadable kernel modules. In most cases we don't need to modify kernel function, just get control at the right time. So we place hooks in kernel source, which remain dormant until activated by the GKHI when a RAS module asks it to. Maybe this will provide a way out of the difficulty.

So Linus is right. His main argument is about people selection. Keep the ones who don't want to do it the hard way out of the door. You're just saying that his strategy is succeeding.

Let me ask: if you want to learn how the kernel works, and you think you need a debugger, why don't you apply the patches yourself? What Linus does with his own tree has little impact on you. You won't buy Linus with the 'learning' argument, I think.

I don't think Linus' main point is "do I the hard way or don't". The point is that "doing it with a debugger is not the easy way". It seems to be easier but you don't get the Knowledge. So in the end it's 'the right way' vs. 'the wrong way'. not 'hard' vs. 'easy'. In this, I must say I agree with him. I'd also add that source code is (should be) a language to express ideas (at the same time to implement them). With a well written source, you don't need a debugger at all to understand 'how it works'. And if you find you need it, we've quite a problem with the source in the beginning. Source code should be more human readable than debugger output, *expecially* for beginners. In this Linus is doing a great job.

My points againts Linus' arguments:

1. you should give people freedom to do what they do. If they want to spend thier time with a debugger let them do. This 'social engineering' thing is crap. You don't make people better by filtering them.
2. apply tight filters on what people produce (patches, features) not on how they produce. If Linus is right (and in the end I believe it), 'debugger people' will produce low quality patches (the ones that fix the syntoms not the problems), and those patches won't be blessed by Linus. I think Alan made the point that those patches sometimes may be useful to others who are looking for the real problems.

Please explain why it does it twice for compiled-in, and only once for modules. There must be some other logic that does the extra initializations (and that does not trigger for modules), and I'd much rather just get rid of THAT thing instead.

I don't like it that modules behave differently from built-in. You're suggesting I make them even _more_ different, so that the end result would be similar. I'd like to know why we cannot just make them the exact same logic, and not break it more.

Basically, there has to be some other place that does the equivalent of

if (!module) init_subsystems();

and I don't see why we can't just remove that one and make modules and compiled-in behave the same.

That's another case where the SCSI layer is module dependent. If it's a module, we use the "init_module()" in scsi/scsi.c, and if not, we instead use "scsi_dev_init()". They do some of the same things (well, they obviously would have to, otherwise there wouldn't be any point to having a init routine at all), but in general do it in very different ways for no good reason I can see.

Torben, would you mind terribly expanding on your previous patch a bit, and also cleaning this part up? As far as I can tell, we should just remove scsi_dev_init() completely, and use the module init code with an initcall(). Two less regions of #ifdef MODULE, and one less different code-path to worry about..

Why was this done this way anyway? I've never seen this kind of setup in any of the other drivers that have been de-liced of module dependencies..

As you probably saw, it really started out from fixing this silly bug that was introduced by mistake some time ago - which was due to both the module init and the "built-in" init code kicking in. The fact that it wasn't clear which happened where is really for me the driving force here - I'd like to avoid the same bug cropping up in half a year when somebody cleans up some low-level driver init.

Oh, and getting rid of the init list in hosts.c is a nice bonus. It just goes away automatically if you look at the module init path instead ;)

It was intentional. Writing to this file hasn't worked well since 2.3.50 or so, when the zoned VM was merged.

Also, the fact that the new VM keeps a list of directly reclaimable inactive pages around that varies according to the amount of VM activity should make tweaking this value no longer needed...

My opinion is that 2.4 bug fixes are less common than unnecessary rewrites of working code that brings about even more instability and delays for a real and stable 2.4. Bitterness leaks out...

I have a start to the 2.5 tree for PPC. We're keeping up-to-date with 2.4 changes so when the official 2.5 does come out we can merge our changes into it without too much trouble. I'm only allowing bugfixes into our 2.4 tree now that we have some place to play.

If anyone else wants access to the 2.5 tree we have as a place to keep experimental changes I'm happy to open it up to the outside.

The _whole reason_ 2.5.x isn't started is so that people concentrate on stabilizing 2.4.x instead of working on new stuff. Why not just tell these people "why are you working on experimental stuff, put together PPC stress test and kernel regression suites if you are bored, because we know 2.4.x isn't read for prime time"

You cannot complain about 2.4.x not being timely if you are doing things which directly encourage folks to not work on 2.4.x at all. Right?

My PPC guys want to change things. I can't stop them, but I can prevent them from screwing with 2.4 which needs to stay stable. If they don't wan to fix bugs in 2.4 then I can't force them to. They don't work for me, and I don't work for anyone who cares about PPC. So I'm not going to put a great deal of effort into forcing them to them to do something they don't want to. I've effectively kept them from making 2.4 unstable on PPC. That's a good thing, I think.

I've also given them a place to do their experimentation before it becomes safe. Once it does, I can move it into 2.4 or wait for 2.5 if the changes don't become stable.

I /thought/ I had fixed this, since the system runs fine on my (SMP, SCSI) test machine when I boot it with mem=8m.

Somebody on IRC suggested to me that this may be an UP-only bug ... I'm looking into this and hope to fix it soon, but I have to admit some help would be welcome ;)

(I'm still at Linux Kongress and won't be back in the office for about a week)

A year or two ago I had a peek after I got reports that it didn't work together with ZMailer -- turned out that some baseline PYTHON library for SMTP does something stupid in its protocol line construction.. The system experiencing problems was running ZMailer with ultra-strict RFC 821 (plus amendaments) mode.

Ergo, the MAILMAN might be ok, but underneath it there may be buggy language library.. (may STILL be, that is.)

Another point, DaveM, was that MAILMAN is written in Python, and neither of us "speaks" that language...

There is a Majordomo 2 in development. I'm using it for all of my lists now. The upgrade from Majordomo 1 wasn't terribly painful.

Note that it's not really a new version, but rather a total rewrite. MIME, archives, digests, and everything else you would expect to "just work" does. Of course, the catch is that there hasn't been an official, stable release yet.

I've found and fixed the deadlocks in the new VM. They turned out to be single-cpu only bugs, which explains why they didn't crash my SMP tesnt box ;)

They have to do with the fact that processes schedule away while holding IO locks after waking up kswapd. At that point kswapd spends its time spinning on the IO locks and single-cpu systems will die...

Due to bad connectivity I'm not attaching this patch but have only put it online on my home page:

http://www.surriel.com/patches/2.4.0-t9p2-vmpatch

(yes, I'm at a conference now ... the worst beating this patch has had is a full night in 'make bzImage' with mem=8m)

I've been away on a conference last week, so I haven't had much chance to take a look at the code after you integrated it and the test base got increased ;(

One thing I discovered are some UP-only deadlocks and the page ping-pong thing, which I am fixing right now.

If I had a choice, I'd have chosen /next/ week as the time to integrate the code ... doing this while I'm away at a conference was really inconvenient ;)

I'm looking into the email backlog and the bug reports right now (today, tuesday and wednesday I'm at /another/ conferenc and thursday will be the next opportunity).

It looks like ther are no fundamental issues left, just a bunch of small thinkos that can be fixed in a (few?) week(s).