Kernel Traffic #162 For 14 Apr 2002

It used to be, under DOS, that ".COM" files were loaded and "executed" even if they were text. Then, when the ".EXE" file came out, it would be executed if the first two bytes were 'MZ' so you could make a text file with the first two characters "MZ" and save it as "CRASH.EXE" and that's what it would do. All ".BAT" files were assumed to be interpreted by 'COMMAND.COM', the "shell", as scripts. This means that you can make a ".BAT" file called "COMMAND.BAT", with interesting results.

When FAT-32, NTFS, VFAT, Windozes file-system(s) were developed all bets are off. Long file-names are the result of a 'container-file' concept and anything goes.

So the only way to guess at these file's execution capabilities is to read the name --and it's a bad guess.

If the files are NOT set to 'executable' as read by Linux, then samba will not work. For the files to be visible to WIN/Clients, they must have all bits set. This 'feature' can be used to make DOS/Win files temporarily off-limits to WIN/Clients (like during a backup).

The authors of that code made it GPL. You have no right to change that. Its exactly the same as someone taking all your code and making it binary only.

You are

• subverting a digital rights management system [5 years jail in the USA]
• breaking a license

but worse than that you are ignoring the basic moral rights of the authors of that code.

Every single line of code I ever submitted to Linus is -pure- GPL. It bears a GPL header. That includes my part of the vmalloc_to_page work. It has never been available to non GPL modules. You took code I and many others own and exposed it as a library for non GPL users. If they use it that way they are violating copyright law, and they *will* get cease and desist letters.

Anyone using any code of mine in the kernel with non GPL code does so on the basis of the legal doctrine of what is or is not a derivative work, and they do so on their own legal assessment. Taking code I am one of the authors of and making it convenient for people like veritas to use in non GPL code is quite different. Its theft plain and simple.

Well, you're all wrong, bthththt...

Removing the .._GPL() is in this case correct, but not for any of the reasons mentioned, but simply because even Ingo agreed that it shouldn't be _GPL since it's explicitly meant for drivers that shouldn't have any knowledge whatsoever about the VM internals. GPL or not.

The fact that the code was back-ported from 2.5.x and that the _GPL still is there too is just a mistake, partly because I've not gotten any updates from Ingo..

So Linus is allowed to arbitarily export other peoples contributions ? I think we need to clear this one up and understand what people think the actual rules are around here. As I understand it the original code was

1. extracted from bttv and is code which I and DaveM partly wrote
2. was submitted by Gerd who did the extra work and kept it as _GPL when he first exported it. (in 2.4 its relevant to expose it as we have the V4L1 not V4L2 interface)

Nobody seems to have remembered to ask permission around here

No.

I've only submitted the 2.4.x backport to Marcelo, and the only reason I've export it as _GPL there is that it is _GPL in 2.5.x. Having that symbol without _GPL in 2.4 and with _GPL in 2.5 would be very bad style and would upset people who start using that function in 2.4 and notice later on that it is exported more strict in 2.5 ...

I'll happily submit a patch to Marcelo to remove the _GPL once it is gone in Linus 2.5 tree.

The 2.5.x code (which I grabbed for the backport) was submitted by Ingo and updated by Linus says my BK tree changelog. Ingo obviously did *not* a simple cut&paste of DaveM's code from bttv. The 2.5.x vmalloc_to_page() function handles pte-highmem and preemtable kernels, that code was never in bttv.

Trivial change to modutils, keeping backwards compatibility, so EXPORT_SYMBOL_GPL == EXPORT_SYMBOL_INTERNAL.

When the flamers and lawyers agree on what they really mean by EXPORT_SYMBOL_GPL or its replacement and everybody agrees on what the keyword should be, let me know and I will roll a new modutils. Otherwise, leave me out of this flamewar.

Let's look at the possible design for 2.5 first, then 2.4.

The meaning of EXPORT_SYMBOL_INTERNAL is that it can be used to export symbols by base kernel subsystems (static or modular) to other base kernel subsystems which can be compiled as a module. So, if the symbol is thus exported by what is thought of as "base kernel" then only modules that claim to be "base kernel" should use it. Similarly, if it is exported by a driver, then only modules closely associated with that driver (for drivers split in multiple modules) should be able to use it.

In other words, exporting symbols should not be based on the consumer's license because that is technically inappropriate criterion.

As for implementation, perhaps EXPORT_SYMBOL_INTERNAL could look like:

EXPORT_SYMBOL_INTERNAL(runqueue_lock, "scheduler");

and the corresponding module that implements a "scheduler" can claim its rights by:

MODULE_CLASS("scheduler");

A module should be able to belong to multiple classes, of course, i.e. it could provide both "scheduler" to get runqueue_lock and "filesystem" to get register_filesystem().

So, modutils would check module's classes and resolve or deny the corresponding symbol. And EXPORT_SYMBOL(sym) would mean "disable class check by modutils for this symbol".

I am not suggesting to throw away MODULE_LICENSE from .modinfo, only that it should have nothing to do with exporting symbols (i.e. it should be like author/description etc fields).

Now, all the above does not look like 2.4 matter, it should probably be implemented in 2.5. As for 2.4 perhaps it should be a simple matter of changing the actual name EXPORT_SYMBOL_GPL to EXPORT_SYMBOL_INTERNAL which would place INTERNAL_ instead of GPLONLY_ in .kstrtab. And the thing that would link with it in the module should not be MODULE_LICENSE but be a new macro:

MODULE_SYMBOLS_INTERNAL;

A module that doesn't make any special claims (for compatibility) gets only those exported by EXPORT_SYMBOL, whilst a module that claims access to MODULE_SYMBOLS_INTERNAL gets also those exported by EXPORT_SYMBOL_INTERNAL.

There should be no "licensing implications" but simply the (documented) fact that if a proprietary module writer is stupid enough to make a MODULE_SYMBOLS_INTERNAL claim his module will break far more often both with respect to existence _and_ semantics/implemention of those entries exported only "internally". But that is their own problem -- we should neither help them nor prevent them from doing their work and earning their living. That is my opinion. If the vendor has so many resources to spend on monitoring Linux kernel development (and therefore inevitably getting involved and _helping_ with it!) to stay uptodate with every implementation of internal symbols, then that is fine -- so much the better for Linux. But if they want to write stable and maintainable modules then they will never put MODULE_SYMBOLS_INTERNALS. We should be always convincing proprietary software writers by our technical superiority rather than by making their lives tough based on a whim of whoever chooses which license to allow exporting his symbol to.

Simple, ethical and makes everyone happy, or, at least those who understand that the design of a Unix-like operating system should be driven by technical superiority rather than by using marketroid's weapons against them (he who lifts up the sword shall fall by the sword).

That would be the case _only_ under yet another interpretation of GPL. Strange thing about GPL is that there are so many interpretations to choose from :)

It is your interpretation that matters, not mine, so how can I convince you? But I am entitled to an opinion that your interpretation is, in some sense, wrong. Namely, in the sense that it is inconsistent with the similar situation in the case of libraries or even system calls. I don't see why exporting kernel symbols should be so radically different and extremely sensitive to the nature of the consumer's license for some symbols but not for others...

Tigran, the difference is very very fundamental, please think about it, and please try to ignore the fact that you are working for Veritas. Internal modules of the GPL kernel are just a technical modularization of a complete and unified GPL-ed work. We want it to stay consistent, we want *our* work to be fully and completely debuggable, supportable and we want to be able to understand and develop any part of it. This is our intent.

the moment you start to argue 'but why cannot we just add this set of EXPORTs and put our binary-only modules here and there' you are in essence questioning our freedom to specify the license of our work. You are abusing the internal modularization mechanizm to put in code that we cannot debug, cannot read and cannot develop and cannot support in any reasonable way. It's like exchanging kernel/sched.o with your binary-only implementation and not publishing the source code of it. If you want to play such games then you have to implement the other 4 million lines as well, nobody prevents you to do that.

system calls are a published, honored, maintained interface. User-space applications are indpendent works not derived from the GPL-ed kernel in any way. Hence the exception.

historically we have also chosen to to provide a different type of API towards more or less clearly separate works, like independently developed, non-GPL hardware drivers. Let yourself not be confused by the fact that the same technical mechanizm is also used to demand-link separate smaller parts of the kernel work as well.

The impact of binary-only modules on the kernel's development architecture is not zero but not overly significant either, so most of us are pretty pragmatic about that, as long as binary-only module vendors are not abusing this mechanizm to impact the integrity of the kernel and create clearly derived works without obeying the rules of the GPL. And it's clear that internal symbols are just that: internal, still part of the kernel work. [and directly resulting from that, they obey the GPL.]

I consider 'abuse' for example a kernel derivative with a 'modified' scheduler. The day it will be possible to put a binary-only sched.o into the kernel i'll stop doing Linux. I am not here to develop some 'lite' version of the OS, where all the interesting stuff happens behind closed doors. I'm not here either to see the quality of the OS degrade due to sloppy programming in widely used binary-only modules, without being able to fix it.

The GPL right now protects our work from being abused in such a way - it's illegal to provide a binary-only sched.o and compile a kernel product from that, because the resulting kernel is still one work and the whole work must still be under the GPL. It's equally illegal to recover the location of sched.o in the final kernel image and runtime relink it with a binary-only sched.o. It's equally illegal to accomplish the same over the internal module interface.

Think about it, every separate .o in the Linux kernel can be equivalently expressed in terms of a EXPORT-ed module interface, GPL-ed header files and a closed-source module. There is a good reason why the GPL forbids such freely defined 'module interfaces' to be added. Think of the GPL as the price you pay for being able to use the Linux kernel's source code or being allowed to link to it - you are not forced in any way to do that.

and no, you have no *right* to link a binary-only sched.o to the Linux kernel - even if you develop a sched.c 'separately' - and intuitively feel that it's somehow a separate work, the end result is still a derivative of the kernel. And this violation of the license is illegal in most countries, it's even a crime in some countries.

After thinking about it for a while (and careful reading of your explanation) I must conclude that your view is probably safer, i.e. in the long term it is probably better for the Linux kernel to protect its status along the lines you described, even if it is not as "smooth" or "convenient to everyone" as the scheme I was talking about.

On the issue of what should be considered a derivative and what shouldn't, from your email it seems (and it's not a bad thing, imho) that the Linux kernel is protecting itself to make sure that "interesting" functionality is either already in the kernel or exists as a GPL'd module.

To make this thought clearer, let's say that there is no GPL'd journalling filesystem for Linux (i.e. reiserfs, ext3 and others suddenly disappeared). Then, to make your thoughts consistent you would need to disable the exported interfaces required for development of a journalling filesystem. Because, otherwise, you would be working on a "lite" OS and "interesting stuff will happen behind the closed doors". As I said, it is not necesserily "bad", i.e. it may well be necessary for Linux's survival and therefore I am all for it. I only thought that there is something _technically_ unpleasant or "wrong" about it... Maybe "wrong" is the wrong word, but you know what I mean.

Side note: this only works partially.

For example, of the 297 changesets in 2.5.8-pre1, 147 were from email patches (the 50% ratio seems to have held up pretty well over time: about half of the submissions I get are old-style patches, with half being BK merges. Of course, for me the advantage of BK is that the BK merge half often required _much_ less than 50% of the time).

Anyway: of the 147 patches, 125 were merges from Dave Jones (only 123 are marged as such - two were re-attributed by me by hand by editing the emails directly).

And of those 123 changesets marked as coming from Dave, most were obviously written by others, and Dave acted as integrator (which is not unimportant, of course - 99% of what I do myself is only integration these days).

So the statistics get skewed by details like this - if is only a partial map of who actually _wrote_ the patch.

(The same is sometimes true of the BK patches themselves, not just the email patches I get from integrators liek Dave who still use diffs to merge with me. It depends on who and how the original BK changeset was created, of course).

I think I see. If you get a GNU patch from someone like Dave who got it from someone else, then it shows up as Dave in your changelogs. Hmm. Do you think it is worthwhile to try and do some parsing of the message such that we can set $BK_USER to the right person if the message contains the info? Dave is pretty good about doing stuff like

Patch description...
Originally from Matt Domsch.

If we formalized that a bit we could get the annotations right. I know it may sound like an ego stroking exercise, but it's actually very useful when looking at the annotated history to see who did something, it makes it easier to track them down and feed them new changes/suggestions/etc. Yeah, it makes it easier to bug them with questions too, but that tends to happen anyway.

Time flies. Just ran some stats in Linus' tree.

Linus created his tree on 2002-04-03 17:03:45-08:00.

3177 changesets (logical changes/patches/whatever you want to call them). 55000 deltas in 11832 files.

BitKeeper overhead for maintaining this information, the changeset stuff, is about 7MB uncompressed. It started at about 1.5MB for the initial baseline, so we're growing at about 3MB/month, which is a problem.

Since I'm typing, here's the set of things we're currently concerned with and working on for the Linux kernel tree:

1. Performance of updates. The actual updates themselves are fairly fast but the integrity checks hurt. We've already released a new version which does less integrity checks if you turn on partial_check: yes in the config file.
2. Space overhead. The design of the changeset data structures doesn't scale well enough and we know how to do better.
3. LODs. We've been stewing on the discussions that occurred here on the whole "I want to have people test this so it needs to be in BK but then later I want it out of BK". We have a LOD design that makes that possible and also gives you named containers for each maintainer, cherry picking, etc.
4. Nested repositories. This gives you the ability to chop up the kernel and have only the parts you actually need on your disk, that's the theory at least. We have commercial customers who really want this and I think it may be useful for the kernel as well.

Those are the main ones and that's enough to keep us busy for several months at least.