There should be a lot of compile fixes here, along with updates for ia64, and the (painful) move of the 'name' entry out of the "struct device" that helps avoid unnecessary memory waste.

It's a lot of small stuff all over: nothing really stands out in diffstat, except the big update of the Zoran video capture driver, and the blkmtd driver - both updated from their respective development trees (and the ips scsi driver, but that was due to massive whitespace fixing).

Normal merges with Andrew and arch maintainers (x86-64, ia64, sparc64, arm), and AGP updates (notably the merging of the ATI IGP). And network driver updates, ACPI and power management infrastructure.

In both 2.4 and in 2.6, error handling for bad cdrom media is wrong. And it is my fault I'm afraid, since I botched an earlier fix for the problem by putting the fix in the wrong spot.

My kids have a "Jumpstart Toddlers" cd they have long since completely killed, which makes a great test disc. Without this fix, the best time projection I can get for completing a dd type sector copy is about 2 years... Most of that is spent thrashing about in kernel space trying to re-read sectors we already know are not correctable.... After the fix, I was able to rip a copy the CD (or rather muddle through it getting lots of EIO errors) in about 15 minutes.

I have just released another reiser4 snapshot that I hope all interested parties will try. It is released against 2.6.0-test4. You can find it at http://namesys.com/snapshots/2003.08.26. I include release notes below.

Reiser4 snapshot for 2003.08.26

WARNING!!! This code is experimental! WE ARE NOT KIDDING! DO NOT PUT ANY VALUABLE DATA ON REISER4 YET!

Fixed some bugs. And finally reiser4 should compile on 64bit boxes (hm. somebody try it, as I am unable to build any 2.6 kernel for ia64). Also reiser4 should now build without debug enabled. Important SMP bug was fixed (only was in effect for SMP kernels on boxes with less than 3 CPUs). There are still some OOM problems sometimes that we are working on, but generally I hope problems reported by various people about compile failures should be fixed now. Readonly mounts (and hence - reiser4 as rootfs) are not supported too.

reiser4progs update includes some 64 bit fixes too along with other stuff. fsck still does not work, so don't even try to run it.

Netplug is a daemon that responds to network cables being plugged in or out by bringing a network interface up or down. This is extremely useful for DHCP-managed systems that move around a lot, such as laptops and systems in cluster environments.

For more details and download instructions, see the netplug homepage: http://www.red-bean.com/~bos/

• reorganized initialization code to minimize indentation changes to existing code path.
• removed proc version of efivars driver - this driver has been rewritten and will be sent via a separate patch shortly.
• reserve memory for memmap using bootmem allocator to ensure that EFI call set_virtual_address_map() functions properly.

I've been able to successfully boot kernels on EFI systems with this patch using version 3.4 of the ELILO boot loader released last week by Stephane Eranian as well as using GRUB on ia32 systems with legacy BIOS.

Special thanks to Bjorn for providing valuable feedback on the initial patch.

Just for my edification: why does EFI exist?

"The EFI specification defines a new model for the interface between operating systems and platform firmware. The interface consists of data tables that contain platform-related information, plus boot and runtime service calls that are available to the operating system and its loader. Together, these provide a standard environment for booting an operating system and running pre-boot applications.

"The EFI specification is primarily intended for the next generation of IA-32 and Itanium Architecture-based computers, and is an outgrowth of the "Intel Boot Initiative" (IBI) program that began in 1998."

It sounds like it's filling in some gaps in ACPI?

As I have heard the story.

The guys at Intel were having problems getting a traditional PC style BIOS to run on the first Itaniums, realized they had a opportunity to come up with a cleaner firmware interface and came up with EFI. Open Firmware was considered but dropped because it was not compatible with ACPI, and they did not want to dilute the momentum that had built up for ACPI.

And now since Intel has something moderately portable, they intend to back port it to x86 and start using/shipping it sometime early next year.

What I find interesting is that I don't see it addressed how the 16bit BIOS calls in setup.S can be bypassed on x86. And currently while it works to enter at the kernels 32bit entry point if you know what you are doing it is still officially not supported.

No. It would be a total nightmare.

Vendor-supplied drivers without source are going to be BUGGY.

They are going to be doubly buggy if they are run with a compiler that has a buggy back-end.

And that back-end is going to be buggy if it's for some random bytecode that isn't widely used except for some silly EFI thing and is tested exclusively with just a few versions of Windows and _maybe_ occasionally on Linux.

Face it: firmware bytecode is a total braindamage. The only thing that works is _source_code_ that can be fixed, and lacking that, we're better off with a well-defined ISA that people are used to and that has stable simple compilers.

In other words: x86 object code is a better choice than some random new bytecode. It's a "bytecode" too, after all. And it's one that is stable and runs fast on most hardware. But as long as it's some kind of binary (and byte code is binary, don't make any mistake about it), it's going to always be broken.

EFI is doing all the wrong things. Trying to fix BIOSes by being "more generic". It's going to be a total nightmare if you go down that path.

What will work is:

• standard hardware interfaces. Instead of working on bytecode interpreters, make the f*cking hardware definition instead, and make it SANE and PUBLIC! So that we can write drivers that work, and that come with source so that we can fix them when somebody has buggy hardware.

  DO NOT MAKE ANOTHER FRIGGING BYTECODE INTERPRETER!

  Didn't Intel learn anything from past mistakes? ACPI was supposed to be "simple". Codswallop.

  PCI works, because it had standard, and documented, hardware interfaces. The interfaces aren't well specified enough to write a PCI disk driver, of course, but they _are_ good enough to do discovery and a lot of things.

  Intel _could_ make a "PCI disk controller interface definition", and it will work. The way USB does actually work, and UHCI was actually a fair standard, even if it left _way_ too much to software.

• Source code. LinuxBIOS works today, and is a lot more flexible than EFI will _ever_ be.
• Compatibility. Make hardware that works with old drivers and old BIOSes. This works. The fact that Intel forgot about that with ia-64 is not an excuse to make _more_ mistakes.

Don't screw this up. EFI is not going in the right direction.

As one of the people responsible for the EFI Specification and our industry enabling efforts around that spec, I'd like to offer some background that I hope will illuminate some of the issues discussed in this thread. This is going to be a bit long...let me apologize in advance for that but I think there's quite a bit of context here and sharing that may help people understand why EFI works the way it does for Option ROMs.

In 1999 when we were first working on the EFI spec in draft form, a number of the OEMs and IHV companies that we talked to told us that an EFI spec without a solution for the "option ROM" problem would not be accepted in the industry.

At that time, I tried to make the case that instead of propagating the problem into the future we should focus on moving the industry to "architectural hardware" that wouldn't even need option ROMs. What I meant by that was add-in cards with common register-level hardware interfaces to allow operating systems code to carry driver and boot loader code that would be able to work across a range of vendors' products. Perhaps the UNDI network card interface that Intel developed would be a good model for a start at this approach as an example; both in terms of how to do it and the level of traction (or lack thereof) one can expect taking this approach.

The trouble with the "architectural hardware" argument proved to be that PCI is already well established and there is a vibrant industry churning out innovative PCI cards on a regular basis. The idea of a single interface definition for all cards of each of the network, storage or video classes is viewed as simply too limiting and the argument was made to us that to force such a model would be to stifle innovation in peripherals. So effectively the feedback we got on "architectural hardware" was therefore along the lines of "good idea but not practical..."

Faced with that and what amounts to a demand for a solution, we tried to scope the problem. Today's IA-32 Option ROMs are typically 16-bit, IA-32 real mode code, they must live in a magic 128k (192 on some boxes) window below 1MB, and there are no hard and fast rules about what resources on the machine they may or may not touch. The reason the OEM folks asked us to look at solving this issue set in the context of EFI is to try and improve the real nightmares that they face every day. The kind of thing where you plug in an adapter card and suddenly the floppy doesn't work anymore. The kind of thing where you plug in four SCSI controllers and it's highly likely that you can't reach a perfectly good OS install on a drive connected to one of those controllers because the BIOS can't shadow that much ROM code. The kind of thing where ROM code uses I/O reads in lieu of calibrated delays causing controllers to fail on newer, faster systems.

EFI's origins come from the 64-bit side of the house. It was originally conceived in the context of a need for a means to handle programmatic transfer of control from the platform code (BIOS/firmware) to the OS; in other words an abstraction for the platform to support booting shrink-wrap OSes, installed right off the distribution CDs. However, we also worked hard at building a C language binding for those interfaces that would work just as well for IA-32 or even XScale or perhaps even for non-Intel Family processors in fact. The idea being a piece of code written to consume EFI services can compile unmodified and without gratuitous #ifdef's for 32-bit or 64-bit system merely by choice of compiler.

In the context of option ROMs then, this approach would say that you can write a single chunk of C language EFI driver code, your option ROM equivalent. This code can load anywhere in the address space of the machine (EFI uses protected mode, virtual equals physical addressing model), it can use the full address width of the machine for data references and you can compile it for your target machine architecture of choice. So far so good.

However there are some other practical deployment issues that add-in cards bring to bear that we also had to address.

These cards have a habit of traveling from machine to machine. Customers have a reasonable expectation that cards just work when you move them from one system and plug them in to another. Since the receiving system motherboard might have no knowledge of the card you just added, how does it present devices connected to that card as candidates for booting?? Motherboards cannot reasonably carry code for every device a customer might choose to plug in. Addressing that problem is what the Option ROM does for you.

We also tried to advocate for having the drivers/Op ROM images be separately distributed. Some of the IHVs liked that: just ship a floppy with the card, much cheaper than putting NVRAM memory on card. The OEM folks however point out that the floppy gets lost and now the card is useless to the customer...support calls ensue. Thus the code needs to travel as part of the card.

If a card can travel from system to system, that also means it can cross processor architecture boundaries too - there are Itanium Family and IA-32 family machines with PCI slots that are electrically compatible and the expectation is that the cards work equally well in both system types. For the Option ROM content though that presents a dilemma - what do you carry in the ROM?? Native compiled IA-32 code and also native compiled Itanium family code perhaps. Well that works, the PCI spec says a ROM container can have multiple images; we take advantage of that now to build cards that carry a 16-bit conventional ROM and an EFI driver together and there are also Forth images out there for SPARC and Power systems.

As a practical matter carrying multiple instruction set versions of the same code gets expensive in FLASH memory terms. Consider an EFI compiled driver for IA-32 as the index, size: one unit. With code size expansion, an Itanium compiled driver is going to be three to four times that size. Total ROM container requirement: one unit for the legacy ROM image plus one for an EFI IA-32 driver plus three to four units for an Itanium compiled driver image; to make the card "just work" when you plug it into a variety of systems is starting to require a lot of FLASH on the card. More than the IHVs were willing to countenance in most cases for cost reasons.

EFI Byte Code was born of this challenge. Its goals are pretty straightforward: architecture neutral image, small foot print in the add-in card ROM container and of course small footprint in the motherboard which will have to carry an interpreter. We also insisted that the C source for a driver should be the same regardless of whether you build it for a native machine instruction set or EBC.

We did some other things with EBC's definition too; like not including direct I/O instructions. That may sound odd for an environment specifically designed for I/O devices but if you think about it, it's the motherboard code that knows what is and is not "safe" to do by way of I/O more than the device itself that could find itself in pretty much any old machine design. This we believe will significantly improve the reliability of ROM code...it relieves the add-in card Op ROM writer of any attempts to guess and assume what the I/O environment is that the card will encounter out in the field.

You may ask why we didn't just use an existing definition as opposed to making a new one. We did actually spend quite a bit of time on that very question. Most alternatives would have significantly swelled the ROM container size requirement or the motherboard support overhead requirement or had licensing, IP or other impediments to deployment into the wider industry that we had no practical means to resolve. With specific reference to why we chose not to use the IA-32 instruction set for this purpose, it was all about the size of an interpreter for that instruction set. To provide compatibility 100% for the universe of real mode option ROM binaries out there would require a comprehensive treatment of a very rich instruction set architecture. We could see no practical way to persuade OEMs building systems using processors other than IA-32 to carry along that much interpreter code in their motherboard ROM.

Consider the model of Alpha and FX32 as an example; FX32 would be impractical to carry on the motherboard outside the scope of a running OS. At one point we did have an EFI draft that included a processor binding for Alpha at Compaq's request. That material isn't in the final spec for reasons that don't really relate to EFI. Nevertheless, making an Option ROM solution that could plausibly work on multiple CPU architectures (including ones from outside the IA family) and before there is an OS on the box to support an expansive interpreter loomed large in our thinking at the time. [By the by, we remain open to adding other CPU bindings into the EFI spec should anyone approach us with such a proposal in hand.]

By contrast EBC requires a small interpreter with no libraries (roughly 18k uncompressed total on IA-32 for example) and the average add-in card ROM image size is 1.5 units relative to native IA-32 code. And keep in mind that using byte code for this purpose is in widespread, long time use on other CPU architectures so we felt the technique in general was viable based on industry experience with it. Yes, it's a compromise but the best balance point we have been able find to date.

I agree that the compiler back end for EBC will be used for small chunks of code and relatively few of them at that. That compiler and its back end will by definition end up with less code-mileage on it, if you will. I can only say that Intel is supporting the compiler as a commercial product and we stand behind it just as much as we do the native IA-32 and Itanium compilers. Find a bug, let us know - we'll fix it. We run the same tests on the EBC compiler as we do on the native compilers and a few more besides that do EBC torture exercises. The compiler has been in testing for more than a year and in release for nearly than long now. At any rate, feedback we've received so far doesn't seem to indicate stability problems in the compiler; if your experience varies from that please let me know - I'd like to help fix it for you! Incidentally, nothing prevents someone from retargeting GCC for this application.

It is with no small trepidation, given the assembled company, that I turn to the question of Open Source as it relates to EFI and Option ROM code. However...

There is nothing about the definition of the EFI spec or the driver model associated that prevents vendors from making add-in card drivers and presenting them in Open Source form to the community. In fact we've specifically included the ability to "late bind" a driver into a system that speaks EFI. In practice that late binding means that code that uses EFI services and that is GPL code can be used on systems that also include EFI code that is not open source.

The decision on whether to make any given driver Open Source or not therefore lies with the creator of that code. In the case of ROM content for an add-in card that will usually be the IHV that makes the card.

Now, we observe that the high-end add-in card makers often preserve their intellectual property behind proprietary code (via binary drivers) and/or "object models" that they implement in the ROM. In today's lexicon that means an INT13 service for a SCSI card or an INT10 service for a video card. Even for Linux OS-present drivers I understand that some open source drivers for such cards don't actually touch the metal directly for all operations they perform - they use some abstraction between the driver and the actual hardware, something that is carried around in the ROM. I suspect that this paradigm is one that will continue for some time to come. Any change in this approach will have to be worked with the vendors who feel commercial pressure to protect their IP with these kinds of mechanisms.

The EFI spec itself is published with a simple copyright statement and not one of Intel's "colored" NDA covers. The sample code that you can download from our web site is free to you and comes with what amounts to a patent license grant so that you can implement EFI, perhaps using our code in derivative fashion, without royalty or other concern. We also have support tools for folks building EFI code, like Option ROM drivers, that is distributed under the FreeBSD license.

Actually no, there are no patents that I'm aware of that read on EFI. We made a point of not filing any on the spec. We told everyone we talked to during the spec's development that we wouldn't file any so that the spec would end up free of any IP considerations when complete. This was a deliberate effort to support the goal of minimizing any potential barriers to adoption of EFI as much as possible.

The patent license grant is thus in some sense a double coverage approach...you don't really need a patent license grant since there aren't any patents that read but to reinforce that you don't need to worry about patents we give you the grant anyway. This helped make some corporate entities more comfortable about implementing support for EFI.

In practice we have required that any feedback or contribution to the EFI spec or code from third parties that is given to us also comes without any IP encumbrances. There are a couple of things that I've been offered that I would like to have included but couldn't in the end because it would not have been possible to continue telling folks using the EFI spec that they can do so without concern for IP issues.

Actually, I do that all the time. In fact, it was I who asked Larry to add the "bk comment" command in to make it easy to do so.

The thing is, it's hard to do after the message has already gone out into the public - but I fix up peoples email commentary by hand both in the email and often later after it has hit my BK tree too. I try to fix obvious typos, and just generally make the things more readable.

And if the comment was wrong, then it should be fixed. Not because of any "censorship", but because it's misleading if the comment says it fixes something it doesn't fix - and that might make people overlook the _real_ thing the change does.

I have updated dontdiff in the usual place:

http://www.moses.uklinux.net/patches/dontdiff

for the 2.6 kernels. Obviously this was only tested on my configuration(s) so any additions are welcome. Just email them to me and I will add them.

For those who don't know what "dontdiff" is --- grep the file:

/usr/src/linux/Documentation/SubmittingPatches

No, you have to understand that I don't want to call software_suspend() at all. You've made the choice not to accept the swsusp changes, so we're forking the code. We will have competing implementations of suspend-to-disk in the kernel.

You may keep the interfaces that you had to reach software_suspend(), but you may not modify the semantics of my code to call it. At some point, you may choose to add hooks to swsusp that abide by the calling semantics of the PM core, so that you may use the same infrastructure.

Please send a patch that only removes the calls to swsusp_* from pm_{suspend,resume}. That would be a minimal patch.

That's fine. Do what you want at your own pace, with your own code.

I don't think you understood my assertion of not working with you, though. I'm not going to wait around for you to merge my patches, or take more abuse from you. I have better things to do, and a stringent time frame in which to do them.

I recommend either a) accepting my changes and fixes, and help merge Nigel's 2.4 changes into the base or b) accepting the fork, merging Nigel's changes, and later trying to merge the two source bases.

Is it correct to say of ext3 that it guarantees and only guarantees atomicity of writes that do not cross page boundaries?

I am trying to define the difference between "Atomic Reiser4" and ext3, as it seems to be a frequently asked question, and I am thinking of saying something like:

Reiser4 allows you to define a set of up to A separate arbitrary filesystem operations (where A by default is not allowed to exceed 64) that are to be committed to disk atomically. Every individual filesystem operation is atomic without the need to specify it.

By contrast, ext3 only guarantees the atomicity of a single write that does not span a page boundary, and it guarantees that its internal metadata will not be corrupted even if your applications data is corrupted after the crash.

Happenstance.

It's semi-trivial to do this in ext3. You'd open the file with O_ATOMIC and a write() would either be completely atomic or would return -EFOO without having written anything.

The thing which prevents this is the ranking order between journal_start() and lock_page().

It's not trivial but also not too hard to change things so that journal_start() can rank outside lock_page() - this would also offer some CPU savings.

Can't say that I'm terribly motivated about the feature though.