Kernel Traffic #213 For 13 Apr 2003

Why did you move the driver includes to include/video ? What is the reasoning here ?

For example, drivers/video/radeon.h moved to include/video/radeon.h

Is this to be able to share register definitions with the DRM drivers ? (I doubt this will ever happen as the DRM is rather self contained)

I would have preferred those includes to stay next to their respective drivers (though renaming radeon.h to radeonfb.h might have made some sense).

here is a further attempt at fixing the smp problems in the linux-atm stack. things have been moved around in net/atm:

• atm sk's are reference counted. this makes the vcc_release (renamed from atm_release_vcc for consistency) and svc_release easier to write since you dont need to explicity free the sk
• vcc's are stored as a list of sockets, like other protocols
• atm drivers should use vcc_hold/vcc_put to modify reference counts. the driver should read_lock(vcc_sklist_lock) before doing a vcc_hold() if they are in their 'bottom half'. i converted eni, idt77252 (untested -- no hardware), fore200e (untest -- lazy) and nicstar. i also included the he driver. the eni driver is special, since the bottom half is a tasklet you can simply stop the bottom half and be sure the bottom half isnt holding a reference. the he just grabs the vcc_sklist_lock and processes all the vcc's needing serviced. a vcc cant disappear while the lock is held (since the vcc_release needs to do a write_lock to unhash the socket/vcc before dropping the refcnt to 0).
• the nodev list has been removed and bind_vcc/unlink have been changed to just insert/remove vcc list operations. this makes things a bit clearer.
• driver's check_ci() routine have been removed in favor of the one in atm_misc.c. very few were actually any faster, and this functionality is rarely used.
• there is a new function, vcc_lookup(), used mostly by the fore200e and he driver. vcc's returned by this function will need a vcc_put() or the ref count will be wrong. in the future, most drivers should probably use this function to find the vcc for their newly arrived data. there possibly could be a routine atmif_rx(atmdev, skb, vpi, vci) which just does the right thing. ideally, atm would use netif_rx() but one would need to dummy up a ethernet header on each of the pdu's. comments on that idea?
• all the stuff from the previous version: atmdev ref counting, atm and clip can now build as modules.

ftp://ftp.cmf.nrl.navy.mil/pub/chas/linux-atm/2_5_64_atm_dev_lock.patch

[i will try to get a 2.5.65 version available this weekend]

i have made an equivalent version of these patches for 2.4.20 in hopes of getting more feedback.

ftp://ftp.cmf.nrl.navy.mil/pub/chas/linux-atm/2_4_20_atm_dev_lock.patch

(only the nicstar, fore200e, eni and he (included) driver support the new smp 'safe' locking)

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.66/2.5.66-mm2/

• Just lots of little fixes, cleanups, late-but-promised minor features, etc.
• There is a small patch from Ingo here against the CPU scheduler which we hope will fix the new starvation problems which people have been reporting. I this is you, please test and report.
• It turns out that a recent change to the anticipatory scheduler accidentally made it quite ineffective on SCSI (performance is similar to deadline). The patch which did that has been disabled, but we expect that the seeky OLTP loads will suffer until this is fixed up for real.

A while ago, there was some discussion on various mechanisms for inserting taps for debuggers, tracing, security monitors, etc. into the kernel.

Here's a pretty light-weight approach I call "reliable markers":

• they provide only the code locations, the rest is up to the debugger (or such)
• gcc won't move them out of the code path (unlike C labels)
• they force arguments to be where gcc says they are (in the DWARF2 information)
• markers in inline functions get replicated, so one can easily set breakpoints
• zero overhead if disabled at compile-time
• relatively small overhead if enabled at compile time (uses a memory clobber which constrains optimizations, no other code or run-time data is added)

I'm using them for umlsim, http://umlsim.sourceforge.net/ The umlsim control process reads the usual DWARF2 debugging information, and can then also do things like:

• set breakpoints, a bit like kprobes
• call functions in the kernel (usually from the idle task)
• force a function to return
• read and write arguments and variables

Below is a patch with the markers and two usage examples in the kernel. On the umlsim side (uses a Perl/C-ish script language), they would be used like this:

run/ping-peek.umlsim:

...
$brk_a_out = break($a.daemon_write.entry);      <--- breakpoint
...
        $pkt = $grab_outgoing_packet();         (see below)
        $enqueue($$,$pkt,$delay);
        $$.return (*skb)->len;                  <--- return from function
...

include/nettap.umlsim:

global $grab_outgoing_packet = function
{
    return (unsigned char [(*skb)->len]) (*skb)->data; <-- access arg & var
};

umlsim is still in its infancy, but this should illustrate how much can be done with such simple breakpoint support, plus the debugging information gcc already provides.

Very interesting.

We've already got a small program that does this for LTT statements which we plan to use in the future: genevent.
http://www.listserv.shafik.org/pipermail/ltt-dev/2003-January/000408.html

genevent is pretty much straight forward. You type:

$ ./genevent default.event
$ ls default*
-rw-rw-r--    1 karim    karim         392 Apr  5 11:58 default.c
-rw-rw-r--    1 karim    karim        6892 Apr  5 11:56 default.event
-rw-rw-r--    1 karim    karim       18328 Apr  5 11:58 default.h

The default.event file contains declarations about each event. Here's an example:

//TRACE_EV_SYSCALL_ENTRY
event(TRACE_EV_SYSCALL_ENTRY, "Entry in a given system call",
      field(syscall_id, "Syscall entry number in entry.S", uint(1)),
      field(address, "Address from which call was made", uint(4))
     );

And genevent creates the following entries in the default.h for it:

/****  structure and trace function for event: TRACE_EV_SYSCALL_ENTRY  ****/

__attribute__((packed)) struct TRACE_EV_SYSCALL_ENTRY_default_1{
  uint8_t syscall_id; /* Syscall entry number in entry.S */
  uint32_t address; /* Address from which call was made */
};

static inline void trace_default_TRACE_EV_SYSCALL_ENTRY(uint8_t syscall_id,
uint32_t address){
  int bufLength = sizeof(struct TRACE_EV_SYSCALL_ENTRY_default_1);
  char buff[bufLength];
  struct TRACE_EV_SYSCALL_ENTRY_default_1 * __1 = (struct
TRACE_EV_SYSCALL_ENTRY_default_1 *)buff;

  //initialize structs
  __1->syscall_id =  syscall_id;
  __1->address =  address;


  //call trace function
  trace_new(facility_default, TRACE_EV_SYSCALL_ENTRY, bufLength, buff);
};

Also, genevent automatically generates an enum for all the events
described in the .event file:
enum default_event {
  TRACE_EV_START,
  TRACE_EV_SYSCALL_ENTRY,
  TRACE_EV_SYSCALL_EXIT,
  TRACE_EV_TRAP_ENTRY,
...

Obviously the word "trace" is used throughout, but it can easily be replaced by "marker" if this mechanism were generalized.

The intent is to have a .event file in every directory where there are traced events in files (which are the equivalent of "markers" in your scheme). During the build, the various headers would be created and all the code would be generated on the fly.

Of course this is just a begining. We're open to suggestions and contributions.

This looks like a clear improvement, and the markup needed in the kernel is also pretty straightforward.

But my approach actually goes a bit further: I don't even need to know types, because I can extract them from debugging information. Clearly, there is a limit on how useful this is. E.g. if a variable changes its type from "int" to "struct something *", anything using it will need to know. But at least changes like "int" -> "unsigned long" or -> "whatever_t" don't need to be synchronized.

Furthermore, the markers are for desperate cases, where the debugging information does not allow us to find arguments or variables, or even the location for placing the breakpoint. On function entry, it should be possible to access arguments without any markup in the code. I'll examine that part soon.

Generally, my idea is to gather as much useful information from debugging data as possible. A few observations so far (based on using DWARF2 information; all this is on the kernel, with the usual optimizations):

• gcc provides accurate type and name information
• location of static or extern functions is accurate, but there doesn't seem to be a reliable means for determining where the function prologue ends [10000]
• the location of inlined functions is less accurate than for non-inlined functions [10003]
• labels (as in "goto foo;") are completely erratic (that's one reason for markers) [10001, 10002, 10003]
• variable locations information is frequently only useful if we're after the prologue, and it does not accurately reflect register copies, and such (that's another reason for markers) [10005]
• call frame information seems to be almost sufficient for emulating a function return. "Almost" because I also need to analyze the add N,%esp instruction following the call. I'm not sure yet if there isn't a better way, though.

The numbers in square brackets are the gcc PRs I've submitted. (http://gcc.gnu.org/cgi-bin/gnatsweb.pl)

Without optimization, some things look better, of course.

I'm not sure how much help we can expect from the gcc side. Some of these things may just be too hard to fix. And others can't be usefully fixed at all (e.g. the markers tell gcc where the focal points are, where it has to relax optimization for accessibility).

Some thing I haven't looked at yet:

• using line numbers to specify locations. With optimization, I expect total disaster here. Also, line numbers are too volatile for few things but manual debugging.
• passing structures as arguments, or returning them from functions
• unusual (i.e. large) alignment settings. Perhaps I'll eventually need to know CFLAGS in order to reconstruct some things.

My goal is to reduce the markup that has to be done on the kernel to the bare minimum. The "reliable markers" are the least intrusive way I could think of for handling those cases where nothing else works (e.g. in the middle of a function).

It would be nice to have a "global" clobber, though. I.e. one that flushes and invalidates all values cached in registers, and that forces all evaluations happening prior in the nominal execution flow to be carried out, including initialization of function-local variables. That way, reliable markers wouldn't need the list of things that might be looked at.

This is a new release of the IPMI driver that fixes some performance problems. Some vendors implement firmware updates over IPMI, and this speeds up that process quite a bit.

Linus, please apply.

Alan, I will send you the 2.4 patch, so you don't have to fetch it from SF.

This release:

• Improves the "send - wait for response - send -wait for response - etc" performance when using high-res timers. Before, an ~10ms delay would be added to each message, because it didn't restart the timer if nothing was happing when a new message was started.
• Adds some checking for leaked messages.

The patch is a 2.5 version against what is current in Bitkeeper (well, what is current in bk-CVS, thanks for the tool, Larry). The 2.5.66 and 2.4.20 versions are at: http://sourceforge.net/projects/openipmi/

This patch removes the long deprecated flush_page_to_ram. We have two different schemes for doing this cache flushing stuff, the old flush_page_to_ram way and the not so old flush_dcache_page etc. way: see DaveM's Documentation/cachetlb.txt. Keeping flush_page_to_ram around is confusing, and makes it harder to get this done right.

All architectures are updated, but the only ones where it amounts to more than deleting a line or two are m68k, mips, mips64 and v850.

I followed a prescription from DaveM (though not to the letter), that those arches with non-nop flush_page_to_ram need to do what it did in their clear_user_page and copy_user_page and flush_dcache_page.

Dave is consterned that, in the v850 nb85e case, this patch leaves its flush_dcache_page as was, uses it in clear_user_page and copy_user_page, instead of making them all flush icache as well. That may be wrong: I'm just hesitant to add cruft blindly, changing a flush_dcache macro to flush icache too; and naively hope that the necessary flush_icache calls are already in place. Miles, please let us know which way is right for v850 nb85e - thanks.

I recently changed the v850's nb85e_cache.h to make flush_page_to_ram a nop anyway (however since Linus hasn't applied my patch, I guess you won't have seen that change ... HEY LINUS, PLEASE APPLY MY PATCHES! Ahem.).

What's in the old version of nb85e_cache.h is incorrect anyway; until recently I didn't have any working hardware, so it's basically all just random cruft.

[To be honest, I'm not sure my new version is correct either -- just _exactly_* what's expected from these macros is very hard to guess from the documentation, and only a little more clear from perusing the source -- however it does seem to work in practice, unlike the old version. * I could just make all of them flush everything, but that's very * inefficient.

I made a patch (against current Marcelo bk) that contains a bunch of fixes to the radeonfb driver. Among others, it includes new PCI IDs, fix problem with some TFT panels, fix 800x600 mode acceleration, and so on... It also brings the PowerMac PM code up to date.

Please, test it and let me know, especially if I broke an existing working setup. If it didn't work for you before and still doesn't let me know of course too, but I don't expect that to delay a merge with Marcelo.

This driver have been diffing for too long, with patches too scattered, this is an attempt to consolidate all. Without news from Ani in the upcoming week(s), I'll probably just consider myself as the new maintainer for this driver.

Patch can be found at http://penguinppc.org/~benh/radeonfb_ben.diff

Please, send me comments asap, as I intend to ask Marcelo to merge this version soon. I'll send those patches to James directly for the 2.5 version since the port in there seem to already be the one I did a few monthes ago.

The question is why the utterly beautiful generator was dropped?

Its good side was in its inability to get an invalid kernel configuration.

One of the obvious problems is that it was python-based, and thus being slow and requiring python to be installed.

So what if it was written in C?

Is it possible for it to get in before 2.6?

It required a complete mangling of the config files
It didn't include automated tools to do it
It required python2
It didn't support alternative config tools
And a few other things

A new 2.5 port of the 2.4 version is available on www.sourceforge.net/projects/swsusp.

The changes from the last version are not huge; perhaps most significant is that the dynamically allocated bitmaps (in place of pageflags) are implemented. This won't mean anything to most people, but Andrew Morton, Pavel and Patrick should be happy. Note for Patrick: The driver code was using KERN_EMERGENCY to print when devices were suspended/resumed. Can we lower this to INFO (I did something alone these lines in this patch). It mucks up the nice display :>

Hopefully I'll soon find time to start bombarding Pavel, Patrick and Linus with incremental patches for merging :>. In the meantime, please give it a go.

Under 2.4, we use scripts that unload problematic drivers, switch to a text console and so on before starting the process. Hopefully this will end up being unneeded in 2.5 once the driver model is complete. In the meantime, however (especially if you get problems!), you might like to try first with a minimal load, and slowly add things. It does work fine on my Omnibook XE3! YMMV.

Brief howto:

(Assuming you've applied the patch against 2.5.66 - other versions may work too).

1. Patch, compile as normal.
2. Ensure your kernel commandline includes resume=/dev/hdaX where X is a swap partition you will be using. More than one swap partition can be used. This just specifies which will have its header used to indicate that the system is suspended and where to find the data.
3. If you want debugging info: echo 1 20 15 31 > /proc/sys/kernel/swsusp (See line 201ff of kernel/suspend.c for what the numbers mean).
4. echo 4 > /proc/acpi/sleep to start the process. The system should save the image and powerdown. If you choose the debugging info, you'll have to press SHIFT at the end of each step. If you press SHIFT at other times, you can toggle this pausing on and off. Whether you selected the debugging info or not, you can press ALT to cancel the process.
5. To resume, reboot with the same kernel. The image should be detected and reloaded without you needing to do anything special. Pausing or not is state-persistent from when you suspended.
6. You should (DV) get your system back to where it was when you started the process.

Please report success/failure/questions on the swsusp mailing list. See Florent's home page for details.

While we're on the subject of framebuffers, one area which needs to be looked into is the pixel layout for all of:

• little endian byte, little endian pixel
  1bpp: word 0 bit 31..0 = pixel 31..0)
  16bpp: word 0 bit 31..0 = pixel1 bits 15..0 pixel0 bits 15..0)
• little endian byte, big endian pixel
  1bpp: word 0 bit 31..0 = pixel 24..31, 16..23, 8..15, 0..7)
  16bpp: word 0 bit 31..0 = pixel1 bits 15..0 pixel0 bits 15..0)
• big endian byte, big endian pixel
  1bpp: word 0 bit 31..0 = pixel 0..31)
  16bpp: word 0 bit 31..0 = pixel0 bits 15..0 pixel1 bits 15..0)

We currently do not support all these combinations, and so far I haven't looked into it. It is on my (great long) to do list.

Yes, this is still a (hard) problem.

BTW, more combinations are possible. E.g. swapped bytes in 16-bit words or swapped 16-bit words in 32-bit words, due to hardware swapping of data bus lines. And things get really fishy on such a system for 24-bit wide pixels.

The order within a pixel can already be specified using fb_bitfield.msb_right. But how pixels are laid out in the frame buffer is a different thing.

I thought about adding the following FB_TYPE_* flags a few years ago, but I'm not sure they'll allow us to handle all cases:

• FB_TYPE_SWAP_8_IN_16: individual bytes within a 16-bit word are swapped
• FB_TYPE_SWAP_16_IN_32: individual 16-bit words within a 32-bit word are swapped
• FB_TYPE_SWAP_32_IN_64: individual 32-bit words within a 64-bit word are swapped

These are supposed to be OR'ed with the current FB_TYPE_* definitions, e.g. FB_TYPE_SWAP_32_IN_64 | FB_TYPE_SWAP_16_IN_32 | FB_TYPE_SWAP_8_IN_16 | FB_TYPE_PACKED_PIXELS would indicate a completely swapped video bus.

An alternative solution moves more processing into the kernel. Since the actual fbdev driver knows about all this (it has to provide fb_ops), it can provide the following operations to userspace:

• fillrect()
• copyrect()
• put_image()
• get_image()

This would allow a user space application to perform all simple drawing operations without having to care at all about the actual frame buffer layout.

For more complex operations, these can be performed by the application on a shadow frame buffer in a simple packed format, while letting the kernel take care of the actual drawing, including necessary chunky-to-planar conversions and bit mangling.

For performance reasons, the driver should report optimal positional and size alignments for {put,get}_image().

VGER runs email processing with two layers of filters. That we need any such thing is due to the sorry state of email (all manner of spamming all around).

VGER has web-pages where various aspects of the system are shown, _including_ present filter-rules in Majordomo. ( http://vger.kernel.org/ and onwards.. )

We have added also some synchronous filters into VGER's MTA, so that incoming email gets rejected VERBOSELY to its sender, when couple common cases are encountered.

How do these filters work, then ?

Our filters are line-based one-match keyword trigger thingies. Majordomo 1.x does not have any sort of scoring system. Nor have we had much interest in integrating something else, like SpamAssassin, into our MTA environment to make scorings.

We are treating things like messages of TEXT/PLAIN type with BASE64 encoded content, or messages with HTML in them as obfuscated and potentially spam. Our rather simple filters don't decode BASE64 (nor QP, but our MTA decodes that).

I recall that I have myself tried to use Hotmail, and found quite easily the setups so that my outgoing email will never have HTML in them. -- Current version of HM does not appear to send HTML, nor did I find any settings for it.

Current Yahoo does not send HTML attachments either, unless poster WANTS to send HTML by activating "Allow HTML tags" thingie at right underside of the message body entry box. Turning that off will not send HTML. Plain and simple.

(Making these tests took me about an hour, most of the time to get thru all those foobar verifiers.)

With Yahoo I had at first immense problems to get any email from them, as their SMTP email sender uses INVALID protocol:

<<- MAIL FROM: <yahoo-dev-null@yahoo-inc.com>
->> 501 5.1.7 strangeness between ':' and '<': <yahoo-dev-null@yahoo-inc.com>

When you read really carefully RFC 821 / 2821 syntax about that, you will see that it does not allow space in that place. Sendmail does, and that has forced others to extend the syntax alike.

That happens only during the registration if alternate address is given. Actual web-mail sending works as it should.

Yahoo is the only legitimate email source doing that of what I have seen. (Tons of spammers do it, of course.)

I am trying to do IO on 4000 disks simulataneously. Since I don't really have 4000 disks, this is what I did.

I faked disks using scsi_debug. scsi_debug shares a single 8 MB chunk for all disks. So, I created a filesystem and made a 6MB file on it. Then I mounted all the filesystems.

I did "read" forever on all the mounted filesystems. Machine hosed up. I was able to start only 300 processes (each process doing IO on a single filesystem).

I ran out of lowmem. I am wondering why I have so much ext3_inode_cache. All 4000 filesystems are ext2.

Okay !! I just made all my filesystem except root ext2. (I couldn't boot when I configed out ext3 - mounting root failed).

Anyway, problem seem to have gone away.. Machine is really slow while trying to do IO on all 4000 filesystems. It is slowly creating processes. System is 100% busy. (so far, it created 461 processes out of 4000). But we are not running out of lowmem and inode caches seems to be reasonable..

So, it must be some leak in ext3..

The Linux Test Project test suite ltp-full-20030404.tgz has been released. Visit our website (http://ltp.sourceforge.net) to download the latest version of the testsuite that contains 1000+ tests for the Linux OS. Our site also contains other information such as: test results, a Linux test tools matrix, an area for keeping up with fixes for known blocking problems in the 2.5 kernel releases, technical papers and HowTos on Linux testing, and a code coverage analysis tool.

Lists of test cases that are expected to fail for specific architectures and kernels are located at: http://ltp.sourceforge.net/expected-errors.php . These lists also contain expected LTP compiler warnings for each architecture and kernel.

Highlights:

• Stability, stability, stability! Most compiler warnings have been resolved for i386, ppc32, ppc64, s390, and s390x architectures.
• Thanks to Andreas Jaeger, Dan Kegel, and Doug Ramir for their dedication to and help in increasing the stability of the LTP.

A revised and updated LTP HowTo is in its final stages of review and should be available later this month. Plus, an ncurses-type GUI is also in the works for inclusion into the May release.

We encourage the community to post results, patches, or new tests on our mailing list, and to use the CVS bug tracking facility to report problems that you might encounter. More details available at our web-site.

I've had numerous requests for a backport of the interactivity changes to the O(1) scheduler for the -ck* kernels. I have resisted posting my backport because people had described real problems with these patches. However it seems most, if not all of the problems are related to one patch.

I've posted a special split out patch (001_o1_int_pe_ll_030407_ck_2.4.20.patch) for ck that includes the new interactivity changes, with the one patch responsible for problems backed out. No desktop tuning patch is supposed to be necessary for this so I've removed it from the site.Note that the full -ck4 patch does not include this update. I would like some feedback from people using it before I make a more substantial update to bring out a -ck5. The patches must be applied manually in order as they're desired. I've been using them for a little while without any problems.

Get them here:

http://kernel.kolivas.org

Have you looked at Stellation at all? I know the code itself is Java but they have some neat ideas about being able to take 'slices' across the repository and treat the slice as a single file for things like revision tracking. In some ways this is like a changeset; I could see wanting to track revisions this way, for example:

1. Changeset 'fix broken ptrace' gets applied
2. Other stuff gets changed
3. Changeset 'fix fix broken ptrace' is applied

I would want to be able to treat 1 and 3 as a single changeset with two revision levels.

Since 0.8.0, OpenSSI has been booted on more than 67 IA-32 nodes. It includes a first cut of HA-CFS, which allows the cluster to lose its CFS root filesystem server node and continue running, if another node is attached to the same disk. The Lustre and NFS clients were integrated (non-root). LVS now automatically registers any socket that does a listen(). Unix domain sockets, SysV shared memory, and SysV semaphores are all clusterwide.

A /proc interface can now be used to migrate a process, rather than SIGMIGRATE. An ssidevfs mount is now done for every node, and /dev is a context symlink into the ssidevfs for a process' node. CFS now supports file locking via fcntl(). The SSI version of util-linux has been merged with 2.11z. The xinetd server is now run on all nodes by default.

Various bugs have been fixed, including races, hangs, panics, and problems with strace on IA64 and Alpha.

The latest version of OpenSSI is 0.9.6. Both binary RPMs and source code can be found on http://openssi.org/.