Kernel Traffic #32 For 27 Aug 1999

There were 585 different contributors. 235 posted more than once. 165 posted last week too.

Riley Williams posted a patch to allow kernels to be larger than a meg. To explain the 1 meg limitation, he said, "the current kernel loader uses the value of a 16-bit unsigned int in the boot sector as the number of 16-byte paragraphs of kernel source to load, thus only supporting kernels of not more than 0xFFFF0 (1,048,560) bytes in size. Worse still, the value stored in this word is the lower 16 bits of the calculated value, so if the kernel was, say, 0x107630 (1,078,832) bytes in size, the stored value would be 0x00763 paragraphs, or about 30k !!!" He explained that his patch would allow kernels to be up to 64 terabytes in size (he figured we wouldn't hit that ceiling for awhile). However, he also pointed out that his patch only created the necessary data structures; more patches would be needed, in order to actually create larger kernels. He proposed some ways of approaching the work that remained, and there followed some discussion of the different ways kernels might be loaded.

Mike A. Harris asked if a 10Mbit wireless net connection was possible with Linux, and there was a bit of discussion. Erik Walthinsen pointed out that Lucent's WaveLAN was supported by Linux (and FreeBSD) though it was only 2MBps. Thomas Davis said that WebGear's cards had Linux drivers, with about 1.6MBps. Nat Lanza said that Aironet made some 11MBps PCMCIA cards with Linux drivers. Robert G. Brown gave a pointer to a March Network Computing article comparing wireless LAN products.

Pavel Machek added that he had written a Linux driver for the Diamondmm Homefree (100KB/sec TCP) which would be released under the GPL in December 1999 (but not before, because he wrote it for money).

Christopher E. Brown reported a problem with the tulip driver under 2.2.11, where large transfers would stall. There were some confirmations of the problem, and some non-confirmations from people with similar configurations who were not experiencing the problem.

I did a pile of testing in pre11 with all sorts of people who bothered to test it

0.89H hangs occasionally

0.89H + fix wasn't hanging for those folks 0.89H worked for

0.91 broke for numerous users and worked for numerous others - not acceptable

0.91g has "testing" on it. So even Donald doesn't trust it. You want me to put that in a stable kernel. Not unless Donald says so.

Donald replied, "v0.91 works for far more chips and boards than any previous version. (I thought I had sent Alan mail about this.)" and added:

v0.91g has the following known limitations:

It doesn't do autoegotiation/autoselection correctly with the 21041
I don't know if the ASIX AX88140 no-rx-broadcast work-around functions. (A lack of bug reports doesn't mean it's fixed.)

The following changes *will not* be put into my version.

The horrible hacks to work with the Xircom chip. Most of the changes have no reason.
The Rx-FIFO-replay work-around for the PNIC chip from the BSD people. The Lite-On people claim that the problem only occurs in chained descriptor mode, which is documented not to work."

David S. Miller added, "remember that we can't merge in any of Donald's latest Tulip drivers until someone does the fast routing support additions. It is unacceptable to lose this feature which several people depend upon now in 2.2.x with Tulip cards." But Alan disagreed, saying, "Why not? More people need a working Tulip than need fastroute. A lot more people need a working tulip driver. Also btw unless Im mistaken 2.2.x doesnt include a fastroute tulip driver anyway"

The current Tulip driver uses I/O space accesses to registers. I've known for a while that using memory space would produce better code on machines like the Alpha and PPC that don't have native I/O space instructions.

It turns out that using memory space results in dramatically better performance on certain x86 SMP machines as well. The trivial single processor performance improvement is multiplied because the tiny number of I/O writes occur in locked critical regions. Using memory space allows the writes to "complete" into the write buffer immediately, instead of waiting until the PCI bus transaction has finished.

I have a test version of the Tulip driver that uses memory operations. This might fix the problem above, and will likely avoid the need for udelay(2) on the PNIC to work around the write-buffer-delay bug.

This is a major change, and I'll have to test a lot of boards. I expect the biggest issue to be timing changes for the serial bit streams used to read the EEPROM and MII registers. I initially validated the original code by probing the traces on the circuit board and checking that actual bit rate was in-spec. Since then I've used a software approach (reading a real-time counter) to verify that the bit rate hasn't changed. The software method may no longer be accurate with the write buffers in the memory subsystem that arbitrarily changes the timing of writes.

A second issue is checking that all of the chips are reliable with memory operations. Some chips, notably the VIA Rhine, were only design-tested with I/O operations.

2.2.12pre:

ftp://ftp.uk.kernel.org/pub/linux/alan/2.2

Fix

Fix the TCP memory leak (Dave Miller)
Fix the 6Mb kernel bug on PPC (Paul Mackerras)
Fix shared IRQ on ISI serial (me)
| Still not SMP safe it seems
Sb1000 compiles built in (me)
Sb1000 works (me)
Fix warning in eth16i driver
Sis900 driver update (Jim Huang SIS)
PAS16 takes module arguments (Bob Tracey)
53C875 model 37 detection (Pekka Pietikanen)
Autofs deadlock with nfs
Require RAWIO cap for kcore (Matthew Kirkwood)
munlock isnt priviledged (Matthew Kirkwood)
RLIM_INFINITY handling (Kanoj Sarcar)

Add

Enhanced Raid (Ingo Molnar)
Computone Serial Driver (Computone)
Minor documentation update
IP virtual server masquerade (Wensong Zhang and others)
Iomega Buz capture driver (Rainer Johanni, Dave Perks, et al)
Large file array support (Stephen Tweedie)
FPU emulation enhancements for Alpha (Richard Henderson)
Universal Serial Bus (Assorted, backport by Paul Mackerras)
Detect AMD Athlon and report (me)
Honour Bios EBDA (??)
PPC updates
Celtic language NLS (Rhys Jones)
IPH5526 FC Driver (Vineet M Abraham)

Hirsimeki Mika asked if anyone had tried getting the programming specifications for RealMagic MPEG decoding boards from Sigma Designs. Pasi Kärkkäinen replied that he'd mailed them and been told they were "researching" it. Hirsimeki replied that he had also tried mailing them and been told the same thing. He had also asked about the availability of specs, and been told they were classified. He suggested starting a petition, and Jonathan Masters suggested getting some help organizing it from veiled.net, who had been successful with an ATI petition.

Elsewhere, B.W. McAdams said he'd recently gotten a DVD player that came with a Sigma Hollywood Plus, and asked if anyone was working on a software-based DVD player. Aaron Holtzman gave a pointer to the Linux Video and DVD (LiVid) project and the Linux DVD project, and added that there were a few other projects working along similar lines.

Alan Cox added, "its hard. In order to protect their habit of selling the same movie in different countries at different times and different prices the DVD vendors use crypto and country coding. People are breaking this stuff but its non trivial." Elsewhere he added that for decoding specs, "You can sign NDA's for the info for a fee (about $15,000 I seem to remember) but then can't release the info itself. Also if you do a software decoder and your decoder is 'cracked' to open the spec you agree to pay them a huge fine."

Gentlemen and Gentlewomen,

I wish to share with you all my latest discovery which I just made a few minutes ago, but I must explain the background first.

For quite a while, since I returned from the States in Feb, I had been the victim of a recurring but random lock-ups with Linux in all incarnations of the 2.2.x series kernels, and have in fact reported several. The strange thing was that I never had these lock-ups in the States.

All that was a mystery until tonight. I had left the keyboard for a short period of time, and returned to find one of my six cats SLEEPING on the keyboard! I shooed the cat away and sat down to continue my editing. I was completely thunderstruck when I realised that the machine was locked up tighter than a chastity belt on a virgin, X11 was still on the display, just completely frozen. Then a light went on in my tiny brain, somehow my cats must have discovered quite by accident the best way to annoy me; that is, pressing keys at random until the reboot/sync/meminfo key sequences (i.e alt-r, alt-s I think) kicks in and machine locks up as a result.

I propose four solutions to this problem.

Keep cats out of room.
Disable the keyboard with a sequence of key-presses, and re-enable by pressing a different sequence of key-presses. Of course, it should be cat-proof.
Unplug the keyboard.
Put cat-repellent on keyboard.

I hope this helps those of you with cats in the house and experiencing strange happenings.

There is apparently much bad blood between the BSD folks and the Linux folks. Linux folks are in a stronger position because of the hype Linux has been getting lately from commercial interests, so they can always say "ha ha we won"; while BSD folks appear jealous when they criticise Linux and tout the good features of their own system.

I will state my own personal opinion here, briefly. It's silly for non-Linux Open Source people to attack Linux, and it's silly for Linux people to retaliate. Stallman's prediction of the '80s has come true: competition has been taken out of the Operating System arena. Everyone can now have a fully functional system for free, without being tied to inscrutible file formats, bloated software, and black-box dependencies. It makes no difference at all if someone elects to use Linux, FreeBSD, or any other free OS that meets their needs. What does make a difference, however, is when people who should be rejoicing together, instead have in common only bitterness and antagonism. If anything can destroy the Open Source movement, this is it.

In this particular case, the dispute was over Linux's IP stack: was it original, or was it 'stolen' from BSD. The mere fact that anyone could take an angry position on either side of that issue, in my opinion indicates an unpleasant rupture in people's understanding of Open Source. And to say, "but I am only trying to set the record straight regarding the facts," only feeds the flames.

H.J. Lu announced, "I have put my NFS server presentation at LinuxWorld Expo on my knfsd ftp site. The filename is nfs.doc.tar.gz. It has both PostScript and html."

He also announced the latest version of his knfsd patch and gave a long description (see his post for the details).

Aaron Tiensivu asked if the newer I2C code, used by the lm_sensors2 package, would make it into 2.3 before the feature freeze. Jeff Garzik gave a pointer to the Hardware Monitoring By lm_sensors homepage, and quoted from the page, "August 13th, 1999 -- We are working on the kernel patches. It doesn't look like it will be included in time for kernel 2.4.x, but things are coming together. Work on 2.3.x is basically done, but we may have a few patches to keep it working under the newer experimental kernels. Look for a new release of Lm-sensors soon which will be in the form of a kernel patch and a user-app directory. [...]"

David Olofson gave a pointer to the first release of the Driver Programming Interface (dpi) layer v. 0.1.0 for Audiality, a real-time audio processing engine in the early stages of development. See his post for the details.

Ulrich Windl announced version 0.7.1 of PPSkit, the kernel time patch for NTP (RFC1305) and PPS support (RFC1589) against Linux 2.2.11, available at http://www.us.kernel.org/pub/linux/daemons/ntp/PPS. See his post for details.

This first came up on Kernel Traffic in Issue #10, Section #10 (7 Mar 1999: Big Memory Machines) , and again in Issue #31, Section #3 (30 Jul 1999: Large Memory Systems) . This time, Andrea Arcangeli posted a patch and announced that he and Gerhard Wichert had co-developed the patch to allow nearly 4 gigs of memory on 32-bit systems. A big discussion followed.

Andrea claimed the performance hit after applying the patch would be virtually zero, and Matthew Wilcox asked for some corroboration. Andrea posted some code that represented the worst-case scenario, and Gerhard posted a benchmark that showed only a 2% overhead. He added that in real-world applications the degradation would probably be invisible.

In a different sub-thread, Kanoj Sarcar pointed out that there might be some conflict with Stephen C. Tweedie's raw IO code. This turned out to be a serious concern, and some interesting consequences emerged. Andrea posted some incremental patches, trying unsuccessfully to make bigmem work with Stephen's rawio patches.

At one point in the midst of this, Linus Torvalds said the bigmem patches looked perfect for 2.3.x, if Andrea would put all his #ifdefs in a .h file. He added that #ifdefs should never appear in actual code.

Kanoj continued his point, saying that rawio was not the only thing which broke with bigmem. He said, "I will give you one example of the type of cases that I am talking about. In drivers/char/bttv.c, VIDIOCSFBUF ioctl seems to be setting the "vidadr" to a kernel virtual address from the physical address present in the user's pte. This will not work for bigmem pages."

Linus replied, "This is exactly why I have always been adamant that people should NOT do direct IO and try to walk the page tables. But people have ignored me, and quite frankly, those drivers should just be broken. The painful part is finding out which of them do it, but once done they should just be broken wrt bigmem, no questions asked."

Kanoj was in favor of finding and fixing all such drivers, but Linus said, "The code in question cannot be "fixed". It's doing something wrong in the first place."

To expand on the above:

If you write a driver and you want to give direct DMA access to some program, the way to do it is NOT by using some magic ioctl number and doing stupid things like some drivers do (ie notably bttv).

The way to do it is to just be up-front about the fact that the user process wants direct access to the buffers that the IO is done from, and use an explicit mmap() on the file descriptor. The driver can then allocate a contiguous chunk of memory of the right type, and with the right restrictions, and then let the nopage() function page it into the user process space.

Suddenly, such a _wellwritten_ driver no longer needs to play games with the page tables. And such a well written driver wouldn't have any problems at all with the BIGMEM patches.

Btw, this is not somehting new. Quite a number of sound drivers do exactly this, and have been doing it for several years. I don't know why the bttv driver has to be so broken, but as far as I can tell it's one of two (the other one being some completely obscure planb driver for power macs).

Oh, and I notice that the USB cpia driver does bad things too, although it seems to be limited to vmalloc'ed memory so it's not nearly as horrible. It seems to have copied the bug from the bttv sources. Johannes, could you look at that a bit, it really _is_ going to break horribly at some point, and I hadn't noticed until after I did a quick grep.. You can use __get_free_pages() to grab a larger area than just a single page.

Peter Desnoyers came in at this point, saying he was writing a driver for a native Virtual Interface Architecture device. Although his driver walked the page tables, he felt that with some work it could be bigmem-safe. He gave an explanation.

I will NOT allow anything that walks page tables. That's pretty much completely out of the question. The rawio stuff gives access to the page tables, and no device will be accepted that does more than that. Page table walking has always been a complete disaster, and it's much simpler to just set up the array of physical addresses separately (ie either at mmap time or through the factilities that rawio does offer).

The intel DMA scatter-gather engine is very useful for doing various scatter-gather things (and for 32<->64-bit translations), but you should NOT see it as an excuse for walking the page tables. I will not accept drivers that do that - it's a maintenance nightmare, and is just basically a bad idea.

Peter took a look at the rawio patches and got a better understanding of the difficulties involved. In particular he realized his own code had several bugs regarding walking the page tables, that the rawio patches handled correctly.

That's why I hate the notion of drives trying to do their own page table walking. Especially with SMP there are some rather subtle issues, and it's just hard to get right. The rawio interface also has the added advantage of forcing some organization onto the driver - you can't do page table walking from interrupts, for example, even though some driver writers have tried..

So yes, something using the rawio interfaces is ok, at least that way when something in the memory management changes I won't have to worry about random drivers breaking..

There was an interesting discussion of Linux security problems, and several interesting URLs were published, detailing some Linux exploits. Sven Koch gave a pointer to http://www.phrack.com/search.phtml?view&article=p52-18, which gives in-depth explanations (including code) of how to maintain control of a Linux system once it's been cracked. Elsewhere, Herve MORILLON gave a pointer to http://www.geog.ubc.ca/snag/bugtraq/msg00788.html, which goes into depth on ways an attacker can totally cover their traces once they've cracked a Linux system.

At a job interview, the director told Carlos Costa Portela that Linux had no kernel-level thread implementation, and Carlos asked on the list if this were true. Rik van Riel gave a pointer to a page listing Multithreaded Programs On Linux, which in turn had a link to The LinuxThreads Library.

Elsewhere, Richard Gooch pointed out that the clone() system call appeared between kernel 1.3.10 and 1.3.20, which puts it (according to the Linux Kernel Version History) right around July or August of 1995.

At some point, Carlos posted again, thanking everyone for their help, and adding that the director of the company has subsequently told him, "my interest in Linux has grown a 50%"

A couple of very low risk items dropped in my mailbox today so seeing as you've not replied about 2.2.12pre7 as a 2.2.12 final I've done an update. (2.2.12pre8). It fixes some documentation items for the ESS Solo sound card, updates the megaraid driver to the one AMI now ship and fixes a small table leak in the masquerade as well as a compile failure for a weird ipvs option combination

If you read this after you release 2.2.12 final no big deal.

I've merged most of the 2.2.11/12 changes into my 2.2.14pre1 tree.

The stuff I'm not going to try and merge is:

Non x86 ports
Net/ipv4 and Net/ipv6
ISDN
drivers/video
Drivers with obvious active maintainers (eg NCR53c8xx scsi)

So if your work is in that area its between you and the relevant maintainers to sort out. Otherwise drivers and stuff will start to be pushed into 2.3.x as I can test them (if possible) or if they look ok (if not). Please check your drivers work on 2.3.x as they appear in that tree.

If you don't want me pushing your stuff into 2.3.x and its in 2.2.11/12 drop me a note.

Benjamin C.R. LaHaise posted a patch and gave a URL for it, to implement lazy context switching for the FPU state on x86. He added, "Actual testing of the patch has thus far involved the mmx rc5 client, plus mpg123 on a dual P233MMX and a K6-2/350. Comments and bugs reports appreciated. Soon to follow: use of the FPU for large memcpy."

About a year ago, I posted a patch to extend the existing x86 lazy floating point context support to multiprocessor kernels (in resopnse to a Alan Cox's arguing against having CONFIG_SMP on for all kernels, because it deactivated lazy FPU support). Shortly thereafter, x86 lazy FPU support was removed from the kernel entirely, with some comment about how it did not make much different on recent x86 processors. I have not been paying attention lately, but, from your patches, Ben, it looks like some lazy FPU support has returned.

I am kind of curious: what is the latest thinking on lazy FPU context loading for x86 these days? Is the pendulum swinging the other way now? Has anyone done any benchmarks one way or the other?

Alan answered, "With Ingo's scheduler and with large boxes running tuned simulations set to run over the right number of processors it is a win. With FXSAVE we dump 512 odd bytes of data out so its an even bigger win. With the K7 we want to use MMX aggressively so having FPU saving that can save once when the kernel first uses MMX/3Dnow is going to help too." Benjamin also explained, "My reason for implementing lazy FPU saving is so that I can make use of the FPU (or MMX) for memcpy and related activities in the kernel without having to pay the price of saving state every single time the kernel is entered/exited, even in interrupts. At least on my box (P233MMX), an FPU memcpy is about 45% faster on a large block copy than rep movs when you don't have to save the FPU's state. Even on the K7 it's worth using MMX for zeropage/memcpy."

linux-kernel comes under attack from time to time. Kernel Traffic has covered such cases in Issue #11, Section #5 (17 Mar 1999: Hunt For A linux-kernel Mailbomber) , where someone bombed the list with many 80K posts; and Issue #14, Section #1 (5 Apr 1999: linux-kernel Troubles) , where someone subscribed linux-kernel to a Japanese mailing list, resulting in some confusion.

This time, linux-kernel was subscribed to many mailing lists of various kinds, ranging from sex discussion groups, to news wires, to gaming groups. Under the Subject: You are now subscribed to the WEB-SERVICES-DISPATCH list, David S. Miller replied to one of the welcome messages (CCing the cnet.com administrator), with:

Note to linux-kernel members: I'm working on this, don't mailbomb me about it, I am aware of it.

To cnet.com administrator: please remove these subscriptions of linux-kernel@vger.rutgers.edu from your lists at cnet.com immediately. Your prompt attention is appreciated, thanks.

3 root@asdf:~# nslookup 4.16.70.237

Server: gw.capslock.lan

Address: 192.168.1.1

Name: PPPa34-ResaleSanAngelo1-5R1035.saturn.bbn.com

Address: 4.16.70.237

So, our friend here is on bbn.com. Roast his ass. I see he's subscribed l-k up to about 50 mailing lists. Mommy must have left home for the weekend and left the computer turned on.

Until Matti and I clean up the crazy spammage, use "x-linux-kernel@vger.rutgers.edu" for your postings for the next day or so.

I have the normal linux-kernel going into a log file so I can get rid of the crap when this idiot's pipe is closed....

Please bear with me on this one...

There followed a medium-sized discussion about moderation and other ways of preventing unwanted posts. At a certain point Matti Aarnio said the discussion should be ended; no mailing list policies would be overhauled, although some long-overdue updates (e.g. from Majordomo 1.x to 2.0) would be done. He added, "The perpetrator was/is US resident, and that "prank" got Rutgers University people quite upset also. I suppose "frying" perpetrators internet access is the least that will happen..."

Much later, under the Subject: "linux-kernel" is back on..., Matti said there was no need to use the x-linux-kernel address anymore.

Arjan van de Ven posted a one-line patch, and announced, "Andrea just found a bug in kHTTPd in 2.3.14, of the brown-paperbag-kind. I don't know how the h*ll this ever worked. (It did work when I coded it, I can't trace when it stopped working). If you are using kHTTPd, you should apply this patch."

Thorsten Kranzkowski found a typo in io.c and io.h in 2.3.14 and posted a patch, and Linus Torvalds replied, "Indeed. There are more of them - there's a few "writew"s that incorrectly got turned into "writeb"s, and the two you found are just the ones that cause the compiler to complain (the others cause silent bugs)." He added, "I'm making a pre-patch-2.3.15-1 in another few minutes, that should fix it."

I've just released the 19990820 version of the Linux-MTD code, which contains a preliminary attempt at write support for the NFTL flash filing system used on the DiskOnChip devices.

It appears to work if you're nice to it, although it gets confused if the NFTL doesn't _exactly_ meet the specs (or my understanding of them).

Anyway, there's also an nftl_format program included with it, so if you use that to create your NFTL filesystem on the flash, you shouldn't have any problems. My contact in M-Systems is on vacation at the moment, so I'll work on interoperability with their code when he gets back in a couple of weeks' time.

In the meantime, it needs error correction to be implemented, it needs some proper arbitration of access to the flash other than sti(), and some other cleanups. My request for clever people who know about Reed-Solomon ECC codes to package up their brain and send it to me still remains.

I'd like to offer my thanks to M-Systems for seeing the light and helping me out with specifications for the beast, even though it took me two years to get them to do so :)

Andre Hedrick was skeptical about whether the code would be GPLed if it was connected with M-Systems, but David assured him the code was all his own, and all GPLed. He also added, "I was hoping to be able to submit it to Linus before the 2.3 feature freeze, but I'm not sure I'll be ready in time. We'll see - as it's only a driver addition he might let me get away with it."


Kernel Traffic Latest \| Archives \| People \| Topics	Wine Latest \| Archives \| People \| Topics	GNUe Latest \| Archives \| People \| Topics
Czech

1.	7 Aug 1999 - 25 Aug 1999	(69 posts)	Maximum Kernel Size
2.	8 Aug 1999 - 17 Aug 1999	(18 posts)	Wireless Net Connectivity
3.	11 Aug 1999 - 20 Aug 1999	(25 posts)	Tulip And 2.2.11
4.	12 Aug 1999 - 23 Aug 1999	(37 posts)	Linux 2.2.12pre Announcement
5.	12 Aug 1999 - 16 Aug 1999	(16 posts)	DVD Players Under Linux
6.	14 Aug 1999	(1 post)	Denial Of Service (DoS) Exploit Works Against All Kernels
7.	15 Aug 1999 - 18 Aug 1999	(28 posts)	Editorial: Competition Between FreeBSD And Linux
8.	15 Aug 1999	(1 post)	NFS Docs And New knfsd 1.4.7 Announced
9.	15 Aug 1999	(2 posts)	Newer I2C Code Probably Not Going Into 2.4
10.	16 Aug 1999	(1 post)	First Component For Audiality Available
11.	16 Aug 1999	(1 post)	PPSkit v. 0.7.1 Available
12.	16 Aug 1999 - 21 Aug 1999	(60 posts)	Using 4 Gigs Of Ram On 32-Bit Systems
13.	17 Aug 1999 - 20 Aug 1999	(33 posts)	Linux Cracking Exploits
14.	17 Aug 1999 - 23 Aug 1999	(23 posts)	Threads In Linux
15.	17 Aug 1999 - 23 Aug 1999	(2 posts)	Low-risk Fixes Possibly For 2.2.12-final
16.	17 Aug 1999	(1 post)	Merging 2.2.x Features Into 2.3.x
17.	17 Aug 1999	(1 post)	X86 SMP Lazy FPU Context Switching
18.	17 Aug 1999	(2 posts)	linux-kernel Under Attack
19.	19 Aug 1999	(1 post)	Important kHTTPd Bugfix
20.	19 Aug 1999	(2 posts)	Typos In The Source
21.	20 Aug 1999 - 21 Aug 1999	(4 posts)	Write Support For DiskOnChip 2000

Kernel Traffic #32 For 27 Aug 1999

By Zack Brown