Kernel Traffic #161 For 7 Apr 2002

Here is a kernel patch to support multithreaded coredumps being worked on by Mark Gross (Intel, mgross@unix-os.sc.intel.com) and Vamsi Krishna (IBM, vamsi_krishna@in.ibm.com).

Multi-threaded core dump patch for 2.4.17:

• multithreaded coredump functionality is enabled by a new sysctl core_dumps_threads. (0 = off, 1 = on).
• Core dump is started by the first thread which gets the signal
• Threads are located by walking the entire task list looking for tasks with matching mm as that seems to be the only reliable way to locate other threads of a given task. In fact, IMO this is the only way until all user space libraries migrate to using thread groups provides by linux kernel (CLONE_THREAD).
• Other threads are prevented from executing while core dump is in progress to improve the accuracy of the dumps. This is done without changing the state of the task. We set cpus_allowed in task struct to be 0 to stop a task from being scheduled and reset it to -1 for resume execution. This has the advantage to not depending on user space at all for correct functioning. IMO sending SIGSTOP to stop other threads does not work if the process is being run under a debugger. The only possible issue with using cpus_allowed is that we could lose task affinities once a core dump is taken. However, this is not a big deal as the task is going to die anyway fairly soon, which is why the dump was taken in the first place.
• Support of SSE registers in the core dump
• Code cleanups/reorg - breakup into smaller functions. Main function elf_core_dump() reorganized/cleaned up by moving filling up of elfhdr, prstatus, psinfo and notes to separate functions to make this very long function a little bit more readable and to reuse some code with the function dumping status of other threads.
• Easy to port to other architectures. It just needs
  ELF_CORE_COPY_TASK_REG - to copy task specific registers
  ELF_CORE_COPY_FPREGS - to copy floating point registers and
  ELF_CORE_COPY_XFPREGS - to copy extended fp registers(SSE) if present
  ELF_CORE_SYNC - to sync up fpu status of other processors in SMP systems if needed by a particular architecture. Read the patch for more details.
• We started with the tcore patches by John Jones and Jason Villarreal as base which were then heavily reworked. This patch is entirely different from theirs in pretty much all aspects.

Current TODO list:

• May be remove reschedule_other_cpus in suspend_other_threads and do this as part of ELF_CORE_SYNC. Rescheduling the other CPUs the way the current patch work may be over kill for accurate core files. Any thoughts?
• Port to 2.5.x, specially the logic to stop other threads from executing while dumping is in progress.
• Make the loop looking for other threads a little shorter by counting the number of tasks found and breaking out of for_each_task loop when it is equal to current->mm->mm_users.

Some usage notes on this patch:

GDB 5.1 works with the core files produced, but only for Red Hat 7.2, and only if the /lib/i686/libpthread.so library is hidden. It turns out that for IA32 RedHat, that there exists 2 libpthread.so files. If the /lib/i686/libpthread.so is loaded then the gdb post mortem debug will not work. We don't understand what's going on here, but its real. Hide the /lib/i686/libpthread.so such that the /lib/libpthread.so gets loaded at debug time, and then debugger will work with the core file. Any insights into this is very much welcome. This behavior is very mysterious to us.

Thanks to Bharata B Rao(IBM) for helping with capturing FPU registers and testing and Suparna Bhattacharya(IBM) for design discussions.

Thanks to Tony Luck (Intel) and Jun Nakajima (Intel) for helping with the review and design of the suspend_other_threads implementation.

This is currently i386 only, it has been unit tested on 1P-P4, 2P-P4, and 4P-PIII systems. I haven't seen any failures so far, YMMV.

The patch is against kernel version 2.4.17. We will port this to latest versions of the kernel if there is any interest.

It looks like we have a bad disk, I'm checking them now to figure out if it is the the primary or backup data drive. I'll run checks in all the repositories if fsck doesn't find the problem so it may take a couple of hours before we are back up.

In the not so distant future, we're moving the backup drive to a different machine such that we can just flip machines when this happens but for now you'll have to wait for a bit.

We did indeed lose the primary disk (IBM 40GB, I am starting to lose all the respect I had for IBM drives, this is one of many that has failed on me personally). I have restored from the backup disk, and in the process redone hardlinks across all the linux kernel trees, which saved about 5GB (nice). All trees which are now on bkbits.net check clean, which means BK thinks all the files are there and that the checksums are correct, a fairly reasonable indication that we are in good shape.

I wouldn't be a bit surprised if we have some permissions problems, mail support@bitmover.com if you hit any and we'll fix things as we become aware of them. In fact, I know we have permissions problems but given that I've been working on this for 12 hours straight, I'll get to it tomorrow.

There are a couple of trees which are missing files, both in Rik's linuxvm.bkbits.net, I suspect an interrupted clone. They are:
bk://linuxvm.bkits.net/linux-2.5-vmtidbits
bk://linuxvm.bkits.net/linux-2.5-writethrot
Rik, ping me if you need help cleaning these up.

The ppc tree seems to be missing linuxppc_2_4, Paul/Tom/Troy/Cort, where is this tree? You'll want to get a copy back here, I suspect, so if you are a PPC person and you have a recently updated version of linuxppc_2_4, hang on to it. We'll sort it out on the ppc mailing list.

We have lost a number of ssh keys. We backed these up a while back but we did not catch all of these. The list is below, send me mail with your ssh key / project name and I will restore them by hand. We already had plans in place for dealing with this problem so that it doesn't reoccur.

Sorry about the long downtime, we are struggling with the economic downturn like everyone else and hadn't put a hot spare in place yet. We bought them, in fact, I bought ten spare boxes for this sort of thing, but I have been too busy to put them in place. We'll get on it, we're aware that people depend on this.

Here's the list of projects missing ssh keys:

bcrlbits
freebsd-dvb (probably not Linux, eh? :)
lia64
linux-mtd
linux-srn
linux24 (I think this one is dead, right Marcelo?)
ltr
misc
nonblock
palinux
test1

If you are the admin for any of these projects, drop me a mail with your ssh key and I'll add it back in.

Leaving the drive off overnight "fixed it" enough that I am able to get some of the data off. It will be a couple hours before I know how much, but I did manage to get all the ssh keys, project descriptions, and project statistics. I'm now working on the actual data just in case there is one of the trees, such as the ppc trees, that we can't find again.

The drive has bad blocks and when it hits them it goes into retry la la land, so I won't know which data is bad until I hit the bad blocks.

You obviously didn't read that thread. Both in the context of BitKeeper and in the context of normal data, you would have seen that we have backups, we just have backups that we can verify are correct. The repositories on bkbits.net are automirrored after each incoming event. There were a few ppc ones which were not and we're still trying to figure out why, and things like the .ssh keys were not completely backed up; we're fixing that by putting that information into a BK repository so it will just automirror like everything else.

I'm not sure why you yanking my chain, it's counter productive and flat out rude after I just spent two days doing nothing but putting things back together for kernel developers. What, exactly, did you hope to accomplish?

Awareness on your side that there are people presenting valid arguments to you even if you don't agree. I did read the first posts of the thread up until it degraded to "our application does every integrity check possible to verify that the data is correct. So even if it gets corrupted, we will know. That's why we better than the rest" and you shot down everyone presenting you with other solutions with this arguments. Well, in this case, you obviously knew that the data is incorrect (because the disk died, I'm really feeling with you here, from my eight IBM DTLA disks, three have died, too and I fear that the remaining five will also die) but all your checks couldn't help you where just a few up-to-date backups would have.

Actually I'm a bit disappointed too, that you with all the professionalism that you sprikle over this mailing list in every of your posts, run such a showcase part of your business as bkbits.net on IDE disks without RAID. And without clustering in case of emergency.

I think we are back in action. We put all the ssh stuff back. As well as the download statistics, take a peek at www.bkbits.net, your stuff should be there.

Let me know if your project is missing anything, I know about the ppc tree, we have that data, that's next. But other than that, everything should be back, let me know if that is not true.

For past 5+ days VGER has been unable to connect any MX server of your domain. Why ? I have no idea. Diagnostics says: "connection timed out".

Somebody has tweaked a firewall there, and is now rejecting connections with TCP/ECN ?

Use http://vger.kernel.org/mxverify.html tool to see how things are working.

See Documentation/filesystems/porting

BKL had been shifted inside several methods, so that filesystem code itself had the same locking as it used to (i.e. code that used to be under BKL stayed under it). If your code doesn't need BKL - feel free to shrink the area, but keep in mind that it used to be under BKL.

I didn't _add_ BKL - neither in devfs nor anywhere else. lock_kernel() is the boundary of the protected area and all that had happened is that this area had slightly shrunken, so its boundaries are inside the method instead of being around its caller.

Again, further shrinking is up to maintainers of the filesystems.

A week ago, I posted this: http://groups.google.com/groups?selm=linux.kernel.3CA20C9B.20309%40us.ibm.com

Nobody had anything to sayabout it, so here's a patch. It moves the disassociate_ctty(1) up, and releases the BKl after it gets done. Is this a sane thing to do, or do some of those exit_*() functions still need the tty?

The patch reduces hold times of the BKL in do_exit() by a factor of 100. They were on the order of 200us, now they're about 1.5us. However, those numbers were on Martin Bligh's NUMA-Q box, so they represent a serious worst-case scenario.

I'd prefer to have the BKL just moved into the functions that need it, and removed altogether from do_exit().

That's especially true as I don't know if sem_exit() actually needs the BKL any more at all - so that if it doesn't, we can just remove it from there (at which point it is a local implementation issue, rather than a cross-module thing).

The disassociate_tty thing falls under a similar heading - we're going to have to fix up the tty layer some day anyway, let's make the BKL detail a tty layer internal thing.