Kernel Traffic
Latest | Archives | People | Topics
Wine
Latest | Archives | People | Topics
GNUe
Latest | Archives | People | Topics
Czech
Home | News | RSS Feeds | Mailing Lists | Authors Info | Mirrors | Stalled Traffic
 

Hurd Traffic #104 For 20 Aug 2001

By Paul Emsley

Mach 4 | Hurd Servers | Debian Hurd Home | Debian Hurd FAQ | debian-hurd List Archives | bug-hurd List Archives | help-hurd List Archive | Hurd Reference Manual | Hurd Installation Guide | Cross-Compiling GNUMach | Hurd Hardware Compatibility Guide

Table Of Contents

1. Concerning stat()

11 Aug 2001 - 13 Aug 2001 (8 posts) Archive Link: "stat() and its consequences (was: Re: Strange behavior while copying files)"

People: Igor KhavkineNiels MöllerNeal Walfield

Igor Khavkine has been concerned for a while about stat(). Recently he got around to saying:

Although stat() has to wake up the underlying translator on the node it's being passed as argument, but we should consider a supplementary call like stat() but which does not wake up the translator. If programs like mc, ls, cp... where changed to use this new call instead of stat on some occasions it can eliminate some slowness and unexpected behavior on their part.

What I expected to happen when I ran cp -a was to get verbatim copy of one directory's contents in another. By verbatim I mean, same permissions, same owners, same all other attributes, a symlink if the orginal file was a symlink and and the same translator setting if it there was a translator on the original file. All of the above are done except for copying the same translator setting. And I was certainly not expecting device files appearing in my / directory.

Although using stat() on a node and creating a "verbatim" copy of a file are not directly related, it illustrates the same point. Although the concept of translators requires them to be transparent, we do not always want them to be so.

Niels Möller thought that "cp" should be updated to know about translators. Neils also suggested "translators should be started with a current directory which is the place where they were installed in the file system."

However Neal Walfield did not like this second suggestion because it "is impossible because we never know where the cwd of a translator will be; this is up to the underlying filesystem"

When asked to explain this point further, Neil said: "

Passive translator are connected to the inode. Thus, if we have two hard links to a translator, what is the current working directory? We just do not know.

You could (and do) argue that we should handle this case. So, what does this involve. Well, let us take a look at libdiskfs/dir-lookup.c and libdiskfs/fsys-getroot.c. Here we see that we start passive translators using the fshelp_fetch_root function. Now, let us look in libfshelp/fetch-root.c and we see that when fshelp_fetch_root goes to invoke the translator, it needs to set the standard port and it uses the current working directory of the parent translator. This usually means that the current working directory will be set to /.

So, in conclusion to fix this, you need to change the interface to fshelp_fetch_root to pass a port to the current working directory and change a few callers. Personally, I would welcome this change.

"

2. Alpha Port Thoughts

11 Aug 2001 - 12 Aug 2001 (6 posts) Archive Link: "A REAL installation opportunity!"

People: Roland McGrathNiels Möller

Ted Rolle wanted to install GNU/Hurd on his Alpha box and asked how to go about it. Unfortunately, there is no working Alpha port yet.

This lead Niels Möller to ask if porting oskit to alpha would be useful for other non-x86 platforms. Roland McGrath answered:

Sure, it's a step. It's also helpful to anyone else to whom the oskit is of use on whatever hardware you port it to.

The necessary step per se is porting the microkernel. The work of porting the microkernel to a particular hardware platform can be divided into three essential parts: booting support, device drivers, and MMU support. The first two parts go into the OSKit. The latter requires some work specifically for Mach on that platform (though at some point the OSKit may have interfaces for this stuff too).

In the case of Alpha, there was a port done of CMU Mach a long time back. That code can be found somewhere and used as a starting point.

3. Concurrent Hurd and Linux

12 Aug 2001 (7 posts) Archive Link: "running hurd in linux"

Topics: Emulators: VMWare, Emulators: bochs, Emulators: plex86

People: Douglas HiltonFarid HajjiPatrick StrasserNeil Walfield

Mohanaraj asked if it was possible to run GNU/Hurd concurrently with GNU/Linux. Neil Walfield replied that others have tried using VMWare, but it was buggy. Plex86 and Bochs were offered as alternatives.

Douglas Hilton interpretted the question differently, saying: "I think the idea is rather to run Linux with Hurd. For instance you could boot something like a MkLinux single server under gnuMach and connect it to ethernet1, com1, tty1, etc, and then also have a Hurd login on the console. "

Farid Hajji expanded on this: " as you suggested (MkLinux server on top of GNUmach besides the Hurd servers) is possible. One big problem here is how to coordinate access to Mach devices. This is exactly the same problem than you face when you run a sub-hurd parallel to the Hurd. As long as you use different devices, it should be okay, but as soon as you need to access shared devices, things get hairy. Devices with likely contention are hd's (even if you use different partitions), com? and I/O. "

Finishing up, Patrick Strasser suggested using user-mode-linux, saying "They run a patched kernel as an user process like a sub-hurd. This could be a point to start and look for experiencies. "

4. Woody, security and architectures

13 Aug 2001 (2 posts) Archive Link: "Woody, security and architectures"

People: Wichert Akkerman

In a message shared with other debian ports, Wichert Akkerman said:

As people have probably noticed we've done a number of security advisories over the last few days, and doing those has made it clear that the way we currently do those will not scale with future release.

The problem is the number of architectures we have to support. For potato we have to recompile on 4 architectures, which is a hassle but doable. Besides having to wait for recompiles to finish we also frequently run into missing build-depends and have to wait for someone from DSA to install those for us, which means it might (worst case) take days before we have all the needed packages.

With the extra architectures that woody will add that is simply not going to work anymore. Now it so happens there is an excellent solution for this: rbuilder. With rbuilder one can simply send an email to a build machine and it will install the build-depends and build the package automatically. This means we can spent our time a lot more effectively.

This brings me to my point: with the release of woody the security team can no longer handle doing all the recompiles manually, and as a result of that we will only support architectures which have a working rbuilder setup that we can use.

I realize that setting up rbuilder is not trivial, but given that woody won't be released soon there should be plenty of time for people to set those up.

5. APT Offline

14 Aug 2001 - 15 Aug 2001 (4 posts) Archive Link: "How to upgrade without ppp"

Topics: Apt

People: Moritz Schulte

Tristan had installed GNU/Hurd from the F2 CDs and since PPP is not working routinely, wondered how to arrange the upgrading. Moritz Schulte suggested APT Offline: "There's an APT Offline Howto (included in the Debian package...), which explains how to upgrade a system via APT, which itself doesn't have internet access. That means, you let APT on your GNU/Linux system fetch all the packages your GNU/Hurd system needs onto a shared store. Then you can install them from GNU/Hurd providing that store as an APT cache. "

6. Approaching 2000 (Packages)

16 Aug 2001 (2 posts) Archive Link: "More statistics"

People: Philip CharlesGlenn Alexander

Philip Charles said: "There are 1934 Hurd specific binaries 116 of these did not make the main CD. " and listed the 116 packages. Glenn Alexander noticed that this was approaching 2000. "Do we get some sort of party when we hit the new millenuim?" he asked.

There was no reply. Hurd developers are too busy to party, I suppose.

7. OSKit-Mach Partition Checking

14 Aug 2001 - 15 Aug 2001 (8 posts) Archive Link: "oskit-mach: partition detecting hangs"

People: Jeroen DekkersJeroen Dekkers Igor KhavkineRoland McGrathDaniel Wagner

Daniel Wagner sent a list of problems he encountered while compiling oskit mach. To get the configuration smoother, Roland McGrath suggested using CC=x86-oskit-gcc configure and using CFLAGS=-g for adding debugging.

Igor Khavkine agreed that the oskit-mach Makefile is weird and Jeroen Dekkers added: "this is Roland's configuration thing. If I'm right, you can do: "make kernel-ide+foo+bar". This will build oskit-mach with the ide drivers and driver foo and bar. AFAIK it's nowhere documented, I think it should be in the readme. The problem here is that it can't find oskit/multiboot.o, when you change OSKIT_LIBDIR to the right path it works. "

Daniel's kernel hung when detecting the partition tables, and to this Jeroen said "that's the oskit code if you're going to read from a partition, it only supports "one level of nesting" according to the documentation. With partfs the partition code isn't in mach anymore and the problem should be fixed. However, the partition check is in the linux driver code, which should always display the right results. I think this oskit-mach is linked against the newest oskit, it should be linked against oskit 20000202. This should be in the readme too. "

8. Fixing The proc Bug

14 Aug 2001 - 15 Aug 2001 (9 posts) Archive Link: "more about proc"

People: Marcus BrinkmannRoland McGrath

Over the past several months, Marcus has been mentioning "the proc bug" - it has been eluding him for a while. This week he gets to fix it and thus stablise the Hurd considerably. He started by analysing a fork memory leak:

What I described earlier as a memory leak in fork handling doesn't seem to be a real leak, rather the hash tables (esp pidhash) seem to grow more quickly than necessary. I am still investigating this. With mtrace I could verify that no memory is actually leaked (the trick is to call mtrace in some rpc that is called after startup, and not in main, which will crash proc).

Now to the tough issue. Igor and I were able to reproduce the crash in a gdb session and investigate the stack.

Marcus posted the stack showing where "The Bad Guy" had scribbled over the stack and went on to ask how to identify it.

Roland McGrath responded saying "It's a good bet the bug is in libihash/primes.c, since that creates a large array of chars on the stack and sets them all to 1 or 0. "

Marcus explained libihash/primes.c "it's the sieve used for finding the next prime number. That's the only char array I could find, and it meets our requirements (every second entry is ignored, so it is not set to 1. It seems we can save some stack usage here to generate even higher primes.) In fact, I am quite confident we could show that the other zeros represent prime numbers."

Marcus went on to say:

I am just thinking, if the stack size of a Mach thread is 16 pages = 65 kb (is it?), and some other bug causes libihash to grow the hash tables very fast (in the order of the number of processes ever registered, rather than the number of living processes), as I observed it, then:

In this case, with about 60000 processes ever, and libihash on the lookout for the next prime, the sieve allocated on the stack would be as large as the stack or larger, wouldn't it?

So my current working hypothesis is that there are two bugs:

However, the real bug is the first one, as 30000 processes and more running at the same time would cause other resource shortages as well.

Roland replied with: "The large stack is suspect in the general case. It can use malloc."

This enigmatic (to your summariser) statement was however the key for Marcus. He then posted:

Which makes all the difference!

 
  PID TT STAT     TIME COMMAND
 4119 co S     0:00.09 screen
 4124 p0 R     3:04.67 forks 100000 1000
87394 p1 S     0:00.01 ps

  PID  UID TH  Vmem   RSS     User   System Args
    0    0 15  132M 2.09M  2:50.25  1:39.59 /hurd/proc

The forks 100000 test did crash much earlier (at around 60000) in all tests before, and we know why now. And while I am writing this, it is at 95000 and still crunching happily.

ulysses:~# forks 100000 1000
Time: 892 seconds.
ulysses:~# echo What a relief\!
What a relief!

9. Kernel Stabilises But Still Leaky

15 Aug 2001 - 16 Aug 2001 (3 posts) Archive Link: "kernel panic, zone kalloc.8192 exhaustion"

People: Marcus BrinkmannIgor KhavkinePaul Emsley

Marcus didn't stop with fixing the proc bug, he then went onto discussing zone limits (zones are fixed-sized blocks that can be allocated/deallocated quickly for use in kernel routine data strctures):

with the proc bug fix, the Hurd core is very stable. In fact, I had an uptime of 16 hours, and handled over 100000 processes over all.

Then it crashed with

panic: zalloc: zone kalloc.8192 exhausted

The question that arises is, if there is some sort of leak (and which type of leak could this be), or if the zone limits are simply too small for todays machine. Maybe even both is true. The reason why I find a leak is possible is that I was not adding a lot of load at the time I got the panic, but maybe it is also some other, more subtle problem.

I have almost no feeling for how much is allocated in these zones and what objects come where, and if the number of objects remains small and more or less constant under the same load, or if they can queue up etc... maybe we will only find out with some serious tracing/debugging.

Marcus then posted the (initial) zone limit table.

It was felt that the initial zone limits were too small for todays machines (16 years after the code was written) but that there were Hurd leak(s) too. Igor Khavkine finished up for now with "zones are supposed to grow dynamically if the initial size is not sufficient. kalloc zones definitely should grow dynamically since they don't have the ZONE_FIXED flag. So if zalloc panics, that means it's ran out of virtual memory (RAM+swap). And that's probably because there's a leak somewhere in the kernel. "

(ed. [Paul Emsley]

You may be interested to know that the code discussed in this and the previous thread was written by Avadis "Avie" Tevanian, who along with his then-colleague, Rich Rashid went on to better (ok, well... more lucrative, at least) things, becoming Senior Vice Presidents at Apple Computers and Microsoft Research respectively.

)

10. OSKit-Mach and Net Access

16 Aug 2001 (3 posts) Archive Link: "oskit-mach: net access causes crash"

People: Daniel WagnerRoland McGrathIgor Khavkine

Daniel Wagner has been trying to boot using oskit-mach recently. He posted his most recent problem: "It crashes immediately when the net device is accessed. Unfortunalely I can't debug it till the end, because I get invalids results over the serial line. What I can say is, that ds_net_get_status is called exactly twice before it crashes. What should I try to collect more information about these bug. BTW the pingreply example kernel from oskit-20000202 works fine with my Realtek 8029 ethernet card."

Roland McGrath advised him: "Use "set remotedebug 1" in gdb before you start. It should then show you everything it sends and receives on the serial line. You can see if there is something else coming out on the serial line confusing gdb. Are you using one serial port or two? I always recommend using one serial port for the serial console and the other port for gdb. "

And Igor Khavkine had seen this before:

I remember that I ran into the same problem previously. You are compiling agains oskit headers from the debian package (newer) but you're linking agains oskit-20000202 (older). This is a big no-no.

They changed the oskit_etherdev binary interface, now there is a `rxpoll' element in the oskit_etherdev_t structure just before `getaddr', that's why `getaddr' was being assigned 0 by default (hence the jump to %eip 0). As for the screwup with GDB, it might hav something do with the kernel going into `kernel_trap()' which is an interrupt handler. And funky things can happen if you mix GDB with interrupt handlers. However there still might be something wrong because usually you should fall on your breakpoint in `panic()'.

So the solution is, unistall your debian oskit package to make sure that its headers are not being used, and check your include path when compiling oskit-mach.

11. /hurd/mouse And Libchannel Integration

9 Aug 2001 - 12 Aug 2001 (4 posts) Archive Link: "Where can I find /hurd/mouse ?"

People: Marcus BrinkmannJohan Rydberg

Johan Rydberg wanted the source for /hurd/mouse. Marcus Brinkmann told him that it is in the Debian specific patch of the Debian source package. Marcus added that, later, he wants to have mouse and keyboard integrated into streamio/libchannel, and went on to say "libchannel should be to character streams what libstore is to block oriented data media. The idea was to have loadable modules to provide the necessary ioctls etc for a device (as character devices usually live and die not with their basic function of reading and writing data, but the additional interface that controls the behvaiour and data stream). "

 

 

 

 

 

 

Sharon And Joy
 

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.