Kernel Traffic #203 For 31 Jan 2003

http://eastbay.bizjournals.com/eastbay/stories/2003/01/20/story1.html

The hardcopy edition is better. It has sweet little TUX snacking on a Windows Logo! Go to http://eastbay.bizjournals.com/eastbay/ next saturday and see this weeks print cover on the web!

Linux will NEVER move into the desktop market!!! Linux has found it niche in the server market and some aspects of the embedded market. Well it is struggling to keep alive in the embedded space. Why is this?

Number one reason it will never move into the desktop market is the free beer mentality. Alot of people expect something for nothing or next to it. I not just talking users. Even multi-billion dollar companies. I had a large company tell me "You are charging us? That is not very open source of you!!" As for end users the same problem exist. Plus companies toke note that it would cost them money to hire some to port their software.

The only reason linux toke off the last few years is because companies thought it could make pure profits by using free stuff. Well they are discovering linux does have a cost. You have to actually hire programmers and you need people to actually understand linux. We will see linux adaption slow down if not come to a halt in all markets except the server and none GUI interface embedded devices such as telecom devcies.

Some at this point might stand up and shout what about PDAs. First this is a very vertical market. In the real world you see lots of Palm Pilots and a few iPAQs here and there. I never seen a linux PDA in large use. iPAQs can run linux but they will never ship will linux. I worked with several experimental PDAs. Several that never made it to market and even more the companies deceided linux was to immature compared to Windows CE so moved to using a M$ product.

What is the immature? The bare basics is stable and fine but people want more than just to login in via a serial console. I seen alot of nice development of new types of GUI. Out of the few dozen vendors all but one decided not to go with X windows. BTW that one moved over to windows CE later. These companies felt X was a hinderance. So they went to other GUIs like microwindows or embedded Qt. Still there is a lack of apps and a even greater lack of comapnies wanting to write apps for linux PDAs.

Now for the issue of the desktop itself. We have the basic two problems above. The biggest issue with X is the long developement cycle. The good news is since NVIDIA, which makes there own X server and drivers, is the dominate graphics card we don't feel it so much. If we had 20 to 30 graphics cards with equal market space we would notice. Especially when the graphics cards were have 6 month cycles before they become obsolete.

So what is my PROOF of all this. First take a look at the linux jobs out there. You will notices System Admin jobs. Several of those in fact. Then the development jobs are iSCSI or network card or some other aspect of network programming. Now look for a GNOME or KDE programming job outside of a distro looking to hire someone. I seen only one in Austalia. Now try a search in flipdog.com, CareerBuilder.com, or HotJobs.com for a GNOME or KDE jobs. Well what do you know. No jobs avaible. So no company is looking to either port there software to linux nor create new linux software. Mind you a few companies tried like lokigames. Now they are gone. Next level is graphics and multimedia programming in linux. Again nothing really avaiable.

Now the next question is what companies invest in non sever related matterial for linux i.e mulitmedia, GNOME, KDE, X outside of the distros. I know people there are people on the list from IBM, HP etc who are reading this email. Speak up if this in not the case. The only one I knew of was VA linux. They hired several of the DRI/X windows developers. To my knowledge they no longer work there. So the only companies pursing non server related are the distros. Now the question is hwo many will be left soon. One of them filed for a form of bankruptcy a few days ago. Very few remain. Also we are seeing the strongs one move to where the money is. The server market. So I wouldn't count on any R&D from anyone to much to move linux to the desktop.

I'm working on running Linux as a guest OS inside a lightweight cut-down plex86 environment. My goal is to run a stock Linux kernel, which can be slimmed down to the essentials via kernel configuration, since a guest OS doesn't need to drive much hardware.

For this, there's a few critical but simple diffs to macro'ize the use of the PUSHF and POPF instructions, due to broken semantics of running stuff using PVI (protected mode virtual interrupts). The rest of the stuff I believe can be monitored effectively by the VM monitor.

Would you please consider integrating these diffs before 2.6? There's only one new header file, and macro substitution for a few cases where these instructions are used. For a normal compile, there are zero logic changes. Just 1:1 macros.

I gutted the old plex86 code, eliminated all the fancy stuff and it now does nothing except provide a VM container to run user-privilege code only. X86 is reasonably VM'able at user priviliege, but not so at kernel privilege.

Plex86 is a kernel module which runs on the host OS. It provides a separate set of page tables, segment registers and other stuff so that there is no interference with the host structures, nor dependence on them whatsover.

The thrust behind these simple mods is to be able to "push" Linux kernel code (when running as a guest OS) down to user level. In this case, it can also be run in what is now an extremely light-weight VM. To do this, proper maintenance of the interrupt flag (x86) is necessary, since behaviour of this flag in the eflags register is different at user-level. The x86 architecture provides a mechanism for this, called PVI (protected mode virtual interrupts), although the logic for this was not carried over to 2 instructions (PUSHF/POPF). Thus my patches...

Other than that, plex86 "shadows" the guest page tables and discovers the guest page tables on-demand. So nothing special needs to be done. Access to system registers trap from user code, so the VM monitor can handle those properly.

About 99% of the work of a full x86 VM is on handling less than 1% of the cases. So the new plex86 angle is, forget doing all the fancy work for 1%. If you're running a VM friendly OS (like Linux with my small patches), you end up with a potentially high performance and Open Source VM, with very little work.

As well, plex86 can bolt on to bochs for accelerating user code, reverting back to bochs for emulation of kernel code - perhaps good for running non-Linux, and for debugging Linux kernels.

But for the normal case of running Linux VMs, plex86 will be a standalone. Because the non-essential IO stuff can be configured out of Linux, and the remaining essential IO can be monitored very lightweight style in plex86 - even right in the VM monitor for high performance.

One of the first implementations of VM was by IBM, called CP/67. It eventually evolved into VM/370 and its follow-ons.

The initial design reason for CP/67 was to allow 2 or more MVS development teams to share a system for testing, so the other team could keep working while the first team debugged a system crash with tools better than the lights-n-switches at the console.

It turns out that the 99% of the work to cover the 1% of the cases is really important. The usual reason for doing VM is to isolate images from each other - and if you don't cover that last 1%, a programming error in one of the images can nuke your supervisor code into oblivion. It may be a "VM friendly OS like Linux", but it can still oops in strange ways. For starters, what happens if you run a Linux *without* your patches under plex86? ;)

Now if you think about it, and not covering the 1% case is deemed acceptable, that's OK too. But it's something that needs to be considered.

Plex86 can 100% isolate guests from each other. What I'm saying is, it takes 99% of the work to do a full x86 VM which doesn't need those 2 macros for PUSHF/POPF. (my oversimplified, but yet useful explanation of the state of affairs)

You have to do a lot of work to "get under the hood" of an OS, to fix up a few cases where if you let them run native, they'll get the wrong information or make the wrong thing happen. Not to the other guests, but to themselves. So if you don't need to do those things, you can let them run without all the black magic.

One thing that seems to have been alluded to but not explicity stated is just where is this patch going, and what affect will happen when running a non 'VM friendly' OS under the 'new plex86'.

One thing that I'm curious about is how say thing'd work when running a linux host, with a VM-friendly linux client, and say a Win-2k client.

I assume that the Win-2k client woudl end up having to trap to a simulator (bochs) for it's ring 0 stuff. But would things in the above scenario work nicely, with proper isolation between the two (or more) clients?

it sounds like you are saying that with the plex86 you have two ways to load a client OS.

1. 'normal', full isolation of VMs no modification of the client OS needed.
2. 'nice VM'. modification of the client OS required, but with the exception of the kernel level commands being eliminated in the modification full VM isolation is still provided. Much better performance then case #1

if you load a client OS and tell the system that it's a #2 when it's really a #1 then you don't have valid isolation, but that's a sysadmin error that you will allow to happen in order to make #2 possible.

is this correct?

No, there's always full isolation. If a guest is run without mods similar to the ones I submitted for Linux, the interrupt behaviour will not work correctly for the guest. Neither the host nor other guests will be affected. Nor do I care, because this is not for running arbitrary guest OSes.

x86 does not have pure VMability. So, rather than trying real hard to get under the hood to make it VM'able with heavy software techniques, just forget about running all guest OSes and run ones that can work, like Linux.

If you look, you'll notice my patches do nothing except macroize the pushf/popf instructions to un-break the handling of eflags.IF inside PVI mode (since x86 breaks it). This has nothing to do with isolation of the guest OS. Nothing to do with running Windows. Nothing to do with anything except running Linux as a guest.

GNU patch relative to 2.5.59:

http://people.FreeBSD.org/~gibbs/linux/SRC/aic79xx-linux-2.5.59-20030122-gnupatch.gz

BK send and tarball distributions:

http://people.FreeBSD.org/~gibbs/linux/SRC/aic79xx-linux-2.4-20030122.bksend.gz
http://people.FreeBSD.org/~gibbs/linux/SRC/aic79xx-linux-2.5-20030122.bksend.gz
http://people.FreeBSD.org/~gibbs/linux/SRC/aic79xx-linux-2.4-20030122-tar.gz
http://people.FreeBSD.org/~gibbs/linux/SRC/aic79xx-linux-2.5-20030122-tar.gz

Driver update diskettes for most distributions:

http://people.FreeBSD.org/~gibbs/linux/DUD/aic7xxx/
http://people.FreeBSD.org/~gibbs/linux/DUD/aic79xx/

RPMs for most distributions:

http://people.FreeBSD.org/~gibbs/linux/RPM/aic7xxx/
http://people.FreeBSD.org/~gibbs/linux/RPM/aic79xx/

Changes since the driver versions incorperated in 2.5.59:

ChangeSet@1.961, 2003-01-22 15:09:24-07:00, gibbs@overdrive.btc.adaptec.com Bump aic79xx driver version number to 1.3.0, now that it has passed functional test.

ChangeSet@1.960, 2003-01-22 14:44:51-07:00, gibbs@overdrive.btc.adaptec.com
  Update Aic7xxx and Aic79xx driver documentation.

ChangeSet@1.959, 2003-01-20 16:46:37-07:00, gibbs@overdrive.btc.adaptec.com
  Aic7xxx Driver Update 6.2.28
        o Add some more DV diagnostic code
        o Fix bug that caused sequencer debug code to be
          downloaded always.

  Aic79xx Driver Update 1.3.0.RC2
        o Correct a regression in RC1 that effectively limited DV to just ID 0.
        o Add some more DV diagnostic code
        o Misc code cleanups.

ChangeSet@1.958, 2003-01-17 14:49:42-07:00, gibbs@overdrive.btc.adaptec.com
  Aic7xxx and Aic79xx Driver Update
        Force an SDTR after a rejected WDTR if the syncrate is unkonwn.

ChangeSet@1.957, 2003-01-17 13:20:53-07:00, gibbs@overdrive.btc.adaptec.com
  Bump aic7xxx driver version to 6.2.27.

ChangeSet@1.956, 2003-01-17 13:17:49-07:00, gibbs@overdrive.btc.adaptec.com
  Aic7xxx Driver Update:
    o Determine more conclusively that a BIOS has initialized the
      adapter before using "left over BIOS settings".
    o Adapt to upcoming removal of cmd->target/channel/lun/host in 2.5.X
    o Fix a memory leak on driver unload.
    o Enable the pci_parity command line option and default to pci parity
      error detection *disabled*.  There are just too many broken VIA
      chipsets out there.
    o Move more functionality into aiclib to share with the aic79xx driver.
    o Correct a few negotiation regressions.
    o Don't bother doing full DV on devices that only support async transfers.
      This should fix a few more of the reported problems with DV.

  Aic79xx Driver Update
    o Add abort and bus device reset handlers.
    o Fix a memory leak on driver unload.
    o Adapt to upcoming removal of cmd->target/channel/lun/host in 2.5.X.
    o Correct a few negotiation regressions.

ChangeSet@1.955, 2003-01-17 12:18:22-07:00, gibbs@overdrive.btc.adaptec.com
  Aic79xx Driver Update
        Enable abort and bus device reset handlers for both legacy
        and packetized connections.

ChangeSet@1.954, 2003-01-17 12:10:23-07:00, gibbs@overdrive.btc.adaptec.com
  Aic7xxx and Aic79xx DV Fix:
        Don't bother with DV if the device can only do async

I'm glad that we've established that we both provide endless amounts of comedy for each other.

Look Justin, the fact remains that you get paid top dollar to maintain the Linux Adaptec driver. If you can't be bothered to reliably integrate fixes that show up in Linus's and Marcelo's tree, then that's regretfully sad given your circumstances.

Now that is what makes me laugh!

David,

this is not how distributed development can work. The communication clearly is broken here.

Regardless of whether Linus' tree is broken or no, ALWAYS Cc: the fixes -- even if trivial -- to the driver maintainer. It's as simple as that.

Same about complaints. If a tree breaks, complaining to the maintainer in addition to Linus/Marcelo/Alan may yield a "Linus' merge is incomplete, here's the missing bit" message from the maintainer.

It's all about communication. If maintainers drown in messages, they'll tell this (Linus for example is notorious for dropping messages).

I don't mean to offend anyone, but what you expect looks like clairvoyance to me, regardless of whether Justin gets paid or not, this is simply not reasonable to expect.

Unless someone comes up with a "watchmydriver" script that checks the ChangeSet figures of a set of files after every bk pull and complains if Linus' tree complains unauthorized ChangeSets. I'm not sure if there is an invariant tag that remains across getting bk patches applied or if real diffs are needed. Larry or other BK experts might know more.

That is not how things works out. Doing trivial changes to 10+ Makefiles does not require to bother 10+ arch maintaners. Same goes for trivial fixes for any subsystem.

Linus pointed out what to do:
Keep a copy of the kernel used for last sync.
Take a copy of latest kernel.
Do a diff of all relevant files, and apply that before submitting.

When it was brought up last time, someone came with a small script to do so. No bk magic or other stuff needed, and I see that used by many arch maintainers - otherwise they would loose to many trivial changes.

This is absolutely NOT WHAT YOU SHOULD DO.

You should INSTEAD, send a mail to postmaster@vger.kernel.org asking why you were removed.

People who continually keep resubscribing eventually get black listed. This means DO NOT DO IT. Ask why you are being removed so that the problems at your site can be fixed.

When an address bounces, it puts a major burdon on both vger and the postmasters here.

No, infact what you are suggesting is absolutely not what you should do - the postmasters are already overworked, and don't need to be troubled as a first resort.

Please, read the FAQ. If you wish to embarrase yourself on this mailing list, that is up to you, but please do not make me look stupid, AND THERE IS NO NEED TO SHOUT YOUR INCORRECT ADVICE.

The relevant section of the FAQ is:

Section 3, subsection 14

"I am not getting any mail anymore from the list! Is it down or what?"

Note that there is NO problem with any of the original poster's mailservers, they are all accessible, so it is not a case of mail getting bounced from some of them.

The FAQ actually says, "Just resubscribe. Majordomo will get you a nice note saying you're still subscribed if suddenly everybody went dumb".

It also says, "Asking for help from postmaster@vger.kernel.org could expedite the issue.", but common sense suggests that it's best to try re-subscribing at least once before contacting the already overworked postmasters.

my paper from linux.conf.au is available from

ftp://ftp.uk.linux.org/pub/linux/willy/lca/

in both TeX and pdf formats. I'd like to turn this into documentation that's more suitable for linux/Documentation, but that's for a later time. I'll probably not check my email for the next week, so don't expect an immediate response.

Here's a patch which adds module symbol versioning to 2.5. The old implementation had been broken with rusty's module rewrite and been disabled in the kernel config.

The idea behind the patch is the same as in the old implementation, i.e. add checksums to exported symbols which change as the ABI changes. Modules also record the version checksums of the ABI they are built against. When loading a module into the kernel, those two checksums (per symbol) are checked against each other to make sure that the ABIs are compatible.

The implementation is basically completely different from the old code (which was well-known for "do make mrproper when something goes wrong"), the idea used partially dates back to discussions on linux-kbuild/Keith Owens/Rusty Russell, and the current code is based in part on a patch by rusty.

A kernel built with CONFIG_MODVERSIONING will continue to work fine with unpatched module-init-tools, however, forcing the load of a module with mismatched symbol versions will need a small patch to module-init-tools.

Version 0.8.2 of the RivaTV package has finally been released.

!!Users of a recent CVS version of RivaTV are advised to keep using CVS!!

0.8.2 is already greatly outdated by CVS, expect 0.8.3 soon.

http://rivatv.sourceforge.net/

http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.59/2.5.59-mm6/

• Some rework and restructuring of the anticipatory scheduling code.

  The reported slowdown in RAID1 rebuild _may_ have been fixed. At least, it doesn't happen for me with this patchset.

• The request aliasing problem hasn't been fixed yet, so this kernel (and 2.5.59) will still fail under heavy direct-IO load.

• The mysterious "machine hangs late in boot" problem has been narrowed down thanks to some great work by Andres Salomon. The machine is stuck waiting on I/O completion when performing the initial lookup for /sbin/devfs_helper:

          Thread 11 (Thread 10):
           #0  io_schedule () at include/asm/atomic.h:122
           #1  0xc014cd0a in __wait_on_buffer (bh=0xd3fe45b0) at fs/buffer.c:132
           #2  0xc014dfa6 in __bread_slow (bh=0xd3fe45b0)
               at include/linux/buffer_head.h:260
           #3  0xc014e1c8 in __bread (bdev=0x0, block=0, size=0) at fs/buffer.c:1385
           #4  0xc0181774 in ext3_get_inode_loc (inode=0xd3d697bc, iloc=0xd3d13ce0)
               at include/linux/buffer_head.h:235
           #5  0xc0181841 in ext3_read_inode (inode=0xd3d697bc) at fs/ext3/inode.c:2205
           #6  0xc0183db4 in ext3_lookup (dir=0x0, dentry=0xd3d4cae0)
               at include/linux/fs.h:1199
           #7  0xc01585fb in real_lookup (parent=0xd3d4cce0, name=0xd3d13d94, flags=0)
               at fs/namei.c:372
           #8  0xc0158849 in do_lookup (nd=0xd3d4cae0, name=0xd3d13d94, path=0xd3d13d84,
               cached_path=0xd3d13d8c, flags=-1071144428) at fs/namei.c:537
           #9  0xc01589ef in link_path_walk (name=0x0, nd=0xd3d13dc8) at fs/namei.c:651
           #10 0xc01558c1 in open_exec (name=0x0) at fs/exec.c:454
           #11 0xc0156200 in do_execve (filename=0xd3d6d000 "/sbin/devfs_helper/index.html",
               argv=0xc133bd08, envp=0xd3d13dc8, regs=0x0) at fs/exec.c:1032
           #12 0xc0107e0d in sys_execve (regs=
                 {ebx = -1071125472, ecx = -1053573880, edx = -1071125308, esi =
           -740448672, edi = 0, ebp = -741261356, eax = 11, xds = -1072562053,

  Which _looks_ like a request queueing problem, but Andres says it goes away when devfs is disabled in config. So I've dropped the smalldevfs patch for now - would be appreciated if devfs users could retest this patch, with CONFIG_DEVFS=y.

• There appears to be a CPU utilisation problem with reiserfs_file_write.patch - but it doesn't oops or corrupt data so I've left that in for now while Oleg scratches his head over that one.

syscalltrack-0.81, the 13th alpha release of the Linux kernel system call tracker, is now available. syscalltrack supports version 2.4.x of the Linux kernel on the i386 platform.

This release containes several important bug fixes and new features.

* What is syscalltrack?

syscalltrack is made of a pair of Linux kernel modules and supporting user space environment which allow interception, logging and possibly taking action upon system calls that match user defined criteria. syscalltrack can operate either in "tweezers mode", where only very specific operations are tracked, such as "only track and log attempts to delete /etc/passwd", or in strace(1) compatible mode, where all of the supported system calls are traced. syscalltrack can do things that are impossible to do with the ptrace mechanism, because its core operates in kernel space.

* Where can I get it?

Information on syscalltrack is available on the project's homepage: http://syscalltrack.sourceforge.net, and in the project's file release.

The source for the latest version can be downloaded directly from: http://osdn.dl.sourceforge.net/sourceforge/syscalltrack/syscalltrack-0.81.tar.gz or any of the other sourceforge mirrors.

* Call for developers:

The syscalltrack project is looking for developers, both for kernel space and user space. If you want to join in on the fun, get in touch with us on the syscalltrack-hackers mailing list (http://lists.sourceforge.net/lists/listinfo/syscalltrack-hackers).

* License and NO Warranty

syscalltrack is Free Software, licensed under the GNU General Public License (GPL) version 2. The 'sct_ctrl_lib' library is licensed under the GNU Lesser General Public License (LGPL).

syscalltrack is in _alpha_ stages and comes with NO warranty. We put it through extensive testing and routinely run it on our systems, but if it breaks something, you get to keep all of the pieces.

* PGP Signature

All syscalltrack releases from now on will be signed. This release is signed with my pgp public key, which you can get from http://www.mulix.org/pubkey.asc or via 'gpg --keyserver wwwkeys.pgp.net --recv-keys 0xBFD537CB'