Kernel Traffic #85 For 18 Sep 2000

On my laptop, suspending to disk is slow. It takes a whole minute, and another minute to unsuspend--if all of core is in use. But it is much faster when I do it not too long after the system was booted, when it has not yet managed to use up most of core.

This suggests that it could be nice to make the kernel empty its caches, and mark all that core "unused", when a suspend is about to happen. I don't know the details of how suspension works, so I am not sure this is possible, but I think there are hooks for doing something in the kernel before a suspend.

there are 4 major issues left in the VM area:

1. system hangs under load with 0 lowmem free (but still some high memory free) [not much details on this one yet]

2. dirty bits can get lost, try_to_swap_out() and other places have a race with the hardware

   [from mm/vmscan.c, line 60 has a race with the /hardware/]

   
   55  if (pte_young(pte)) {
   56          /*
   57           * Transfer the "accessed" bit from the page
   58           * tables to the global page map.
   59           */
   60          set_pte(page_table, pte_mkold(pte));
   61          SetPageReferenced(page);
   62          goto out_failed;
   63  }
   
   

3. it appears something can corrupt page->count or delete a page from the cache while the page is locked [tripped up by my VM patch?]
4. the innd data corruption bug

Fun... OK, folks. There was a bug in ext2 (and damn next to every other local fs) since 2.3.7. Sometimes (depending on phase of moon, pagecache contents and a bunch of other shit) truncate() leaves the tail of last block not zeroed. Looks like it surfaced from time to time, but had been mostly unnoticed. Something in the last kernels made it show up more often.

I dearly hope that patch I've posted today will fix the bloody thing for good. Because if it will not we've got something else to deal with.

As for the bug itself - look at the truncate_inode_pages() (part that does zero-out of the partial page) and ask yourself: WTF will happen if pagecache has nothing for that file? Had been there for more than a year...

Could people who have seen the innd active list file corruption thing please try out linux-2.4.0-test8-pre4? Despite some reports, it was _not_ fixed in pre2, and pre3 was a internal-only test to fix the remaining issue with Al Viro.

pre4 finally passes all my truncate() tests, and the code looks good. The only thing left to verify is whether innd really is happy, or whether the inactive file corruption may have been due to something else altogether. I'm personally fairly optimistic, but this needs to be verified so that everybody can either breathe a sigh of relief or go looking for "the final bug".

Note that reiserfs is reported to have the same bug, and pre4 won't have fixed that - only ext2. The infrastructure is all in place so that the reiserfs fix is probably a one-liner, but for now I just want to hear from people who have their news-spools and active file on an ext2 filesystem.

Hell knows. Let me put it that way: one long-standing bug definitely had kicked the bucket. The question being: had any other crap survived?

In other words, if you can reproduce fs corruption on clean ext2 (after forced fsck if you've used it with earlier kernels) - tell.

This entry in the changelog says it all:

- truncate. Guess what? We threw away the key to the clue-box

Most of the other stuff is cleanups or reasonably straightforward fixes. The truncate thread that's been going through the last few pre-releases is the big thing, and the one that has caused heartache.

Oh, well. It really is fixed now. Everybody involved has been forced to wear the brown paper bag for the next month or so, but we cut out small holes so that we can see where we are going and not bump into things and make even more of a spectacle of ourselves.

And the really is fixed this time. Cross my heart and hope to die.?I can only say that "truncate" has always been the nastiest operation of them all.

The truncate bug, btw, has been there for roughly 14 months. We just first made it easier to trigger, and then messed up about three times when we tried to fix it. Oh, well.

Read my lips. No more fscks. And thanks to everybody who did.

• pre5
  1. truncate. Guess what? We threw away the key to the clue-box.
  2. simplify signal notification. And remember the spinlock.
  3. VIA ide driver update (well, rewrite - the old one was buggy and broken)
  4. network driver fixes (not checking for oom etc)
  5. USB serial driver SMP locking fixes
  6. fix memory leak on failed USB configuration queries
  7. USB initialization using proper "init()" calls.
  8. dvd capacity bug fix and other cdrom driver cleanups
  9. sis5513 IDE chipset update
  10. do_fork() - add "stack-top" for ia64 (and potentially other architectures that may care)
  11. devfs support for LVM
  12. quota transfer miscount fix
  13. x86 checksum/copy prefetch
  14. NFS sillydelete fix
  15. mark_buffer_dirty() doesn't actually use the second argument. Delete it.
  16. SCSI communications device - no need to complain about it
  17. SCSI WP test fix (all pages, not just the first one)

How about this patch?

NOTE NOTE NOTE! I'm on my way home now to be a family man, so I've not actually tested it AT ALL. You have been warned.

The basic approach should be ok, and it avoids using the write path at all because it isn't actually needed. The truncate() case is, in the end, much simpler than writing, exactly because we don't need to allocate any new blocks etc.

We just grab the page, populate it with buffers if required, and find the one buffer that we need to clear out. We clear it out and mark it dirty. End of story.

NOTE: Udo, because I haven't actually tested this (it may not actually compile etc small details), you probably shouldn't actually test this out as-is unless you are _really_ daring and don't mind fixing up after me. It's more a "this is how it should work" kind of thing.

Al? Mind giving it a quick look?

This one I cannot explain.

It's a bh that is NULL, but it's a new case completely. It looks like you have a 1kB blocksize, no? It furthermore looks like the page only had two buffers on it, and accessing the fourth one blows up (accessing the third one gets NULL, which is why it only blows up on the fourth one).

But it _has_ to have four buffers on it. Two buffers just aren't enough to cover a 4kB page.

Uh.

DUH!

I found the bug.

It shouldn't be "bh->b_next". That's just the next buffer on the hash chain.

It should be "bh->b_this_page".

Just change block_truncate_page() to use b_this_page instead of b_next.

The "good news" is that me testing this would neve rhave found it, because all my filesystems are 4kB blocks, so the "next bh" case never triggers at all.

Thanks, and THIS time it really is fixed. I mean, how many times can we get it wrong? At some point, we just have to run out of really bad ideas..

Yeah. Maybe we fixed truncate, and maybe we didn't. I've thought that we fixed it now several times, and I was always wrong. Time for some reverse phychology:

I'm sure this one doesn't fix the truncate bug either.

But I have this ugly feeling that I'm coming down with the same flu that everybody else in my family had the last week, so I'd better release this before I start puking on my keyboard.

• pre6:
  1. trunate - the never-ending story. Makes me feel like a long Kurosawa movie. But in this one the hero _will_ survive, or my name isn't Maxwell.
  2. SCSI tape driver potential memory leak.
  3. XMM FP handler bug fix: we really must not change the FP error mask on exceptions. People care.

My company is currently working on a linux network driver (I'm sorry, but I can't disclose which company or the nature of the driver right now). However, recent discussions on this list have made me grow concerned about licensing problems with the GPL.

The source code for the driver _is_ going to be available, but it will not be GPL'd. There are no patches to the kernel involved. I understand that there should be no problems, but the use of inline functions in the kernel header files makes the situation a bit more complicated. The compiled binary code then does contain GPL'd code, so would it not then be considered a "derivative work"?

If indeed this is a violation of the GPL, is there any way around this by releasing only the source code (even though it isn't GPL'd)? I mean, the compiled binary code does contain GPL'd code, but the source code does not. Is it OK to distribute this?

So, I'm asking the experts here for opinions or insights.

Note that whenever it's not GPL'd, all the module restrictions kick in. So it's going to be "legal" the same way any binary only module is "legal" - assuming all the nasty requirements are met. For something as simple (from a conceptual standpoint, not necessarily an implementation standpoint) as a network driver, doing that is not likely to be a big problem.

It obviously cannot be linked into the kernel, but as a loadable module it's ok as long as it uses the standard interfaces and nothing more.

And sure, having source available might make it easier for people to help you: it can't become part of the standard kernel, and as such it will never be supported, but that's true of binary-only modules too.

I wouldn't recommend it, but I don't see that it would be an insurmountable problem.

Back in May I wrote a quite estensive documentation about all the possible/best ways to debug the Linux Kernel for a talk/tranining that I did in San Jose in May. I find now the time to clean it up and to upload since I think it could result useful to everybody dealing with kernel developement.

ftp://ftp.suse.com/pub/people/andrea/talks/english/2000/kdebug-may-2000-20000907.tar.gz

It addresses MCORE/LKCD/KDB/KGDB/NMI-watchdog/TRACER/IKD/PRINTK/BUG/OOPS/KMSGDUMP and many other issues (simptom/realbug as well).

They were the digital slides for the talk, so while writing them I expected to ingegrate them with speach, but they should be readable also standalone.

They're written in MGP (MagiPoint, not that I like it too much but kpresenter wasn't that powerful at that time). A postscript is included into the tarball as well (they should be easily readable with `gv` with antialiasing enabled).

I'd say it's a _must_ read for any kernel developer (feel free to announce it on other places as well if you think it's good idea of course).

Hope you find them useful, have fun.

If you have patches for the document send them to me and I'll integrate them. Kurt just sent me a 1000 lines patch to correct my english errors in the original version :))

The only one of any note that I'd like to point out directly is the clarification in the COPYING file, making it clear that it's only _that_ particular version of the GPL that is valid for the kernel. This should not come as any surprise, as that's the same license that has been there since 0.12 or so, but I thought I'd make that explicit.

Why? There's been some discussions of a GPL v3 which would limit licensing to certain "well-behaved" parties, and I'm not sure I'd agree with such restrictions - and the GPL itself allows for "any version" so I wanted to make this part unambigious as far as my personal code is concerned.

The reason I wanted to mention that particular issue here explicitly (rather than as just a one-liner in the changelog) is that code written by others is obviously under _their_ discretion, and not limited by my personal foibles, fears and misgivings.

If anybody wants to explicitly state that their code will be valid under any version of the GPL (current or future - whatever they may look like), please send patches to say so for the code in question. If you've used the FSF boiler-place copyright notice, you already have this in place (it says "v2 or later" - the FSF itself doesn't recommend v1 any more).

(Me, I'm taking the careful "wait and see" approach. I don't know if a GPL v3 is imminent, and I don't know if the issues discussed will even _become_ real issues, so you might as well consider me a paranoid, if careful, bastard).

Linus, nobody can ever force GPLv3 upon you. If you don't like GPLv3 when you actually see one, then you can restrict the kernel to GPLv2 only and refuse contributions that don't honour that.

It would be a right pain to be unable to cut & paste between kernel code and GPLv3 user space code, except for kernel versions prior to test8. This may well affect you.

I wouldn't be surprised if GPLv3 simply clarifies things. Clearer legal language, clearer on dynamic linking etc.

If you end up happy with GPLv3 after all, (and you might; it may suit you better than the current one), it will be very difficult to retroactively change post-test8 kernels to allow GPLv3 after all.

Remember, the cabal aren't writing any new license, RMS and his lawyer are. You may ignore speculation about the contents on slashdot, gnu.misc.discuss etc.

... But these are the most important reasons not to use "GPLv2 only" yet:

1. If you accept a patch now to your "GPLv2 only" kernel, *you* won't have permission to retroactively permit "GPLv2 or later" on the patched code. You'd have to contact all the major contributors or remove the patched code.
2. If the GPLv3 is accepted in the long term and is applied to some good user space code, or code from another OS, *you* won't be able to incorporate that code into your "GPLv2 only" kernel. Even if you like GPLv3.
3. Similarly, *others* won't be able to take parts of the kernel and incorporate them into their GPLv3 user space code, or code for another OS.
4. You didn't write most of the device drivers. I think they should be treated as a public resource, available to other OS designers as well as user space, under the terms of their original authors. Which, we can best assume, is GPLv2 _or later_. Many of the device drivers may well outlive the Linux kernel that we know today.

For these reasons I think you should hold off the "GPLv2 only" declaration until such time as you actually see an officially published GPLv3. All speculation you've header about GPLv3's contents so far is just hot air.

It's good that you raised your concerns though.

As I already explained to Alan in a private email, it's not about me accepting other peoples GPLv3 code.

It's about the fact that when I chose the GPL, I did it because I wanted the source-code to be free and unencumbered. Forever. Whether I maintained that code or not. I didn't want my code to have any extra rules and regulations - the GPLv2 is already quite complex enough, but it is, in my opinion the "minimum required" complexity. So it suited and continues to suit my needs and opinions admirably.

The fact that I still maintain my own code and can choose to ignore other peoples code doesn't change that feeling. I want _my_ code to be out there, freely available, and unencumbered by any extra restrictions, forever and ever. Regardless of whether it is I who maintain it or not. I obviously won't be able to maintain it _forever_, after all.

And I'd hate to see my code become part of something I don't like.

Thus the (current) limitation to v2. And only, obviously, for code _I_ wrote and hold the copyright to.

I hope so. And yes, in the end I _believe_ so. It's just that I used to believe so without even thinking about it. Last week some people opened my eyes to the fact that I can't just take it for granted.

In the likely case that the GPL v3 is fine, I will license all my code under "v2-v3". Maybe it even clarifies the issue of "similar in spirit", so that I wouldn't need to worry at all about what the FSF considers "similar" ever again, and I can stop worrying altogether.

I'm not against a new version of the GPL - I'm just spooked by some of the things that have been discussed that _might_ be part of a new version of the GPL. Things that would mean that if I didn't limit my code to the v2, future code maintainers might use my code with restrictions or other things that I never agreed to.

Quite frankly, I'm probably just jumpy. And part of this is very much pre-emptive: making sure that the FSF knows that whatever they do, they have to take other peoples feelings into account too, and not just make a new version of the GPL on their own whims.

So don't read _too_ much into this. It just means that the FSF does not have a blanket permission to expand upon the GPL as far as my personal code is concerned. Nothing more.