Kernel Traffic
Latest | Archives | People | Topics
Wine
Latest | Archives | People | Topics
GNUe
Latest | Archives | People | Topics
Czech
Home | News | RSS Feeds | Mailing Lists | Authors Info | Mirrors | Stalled Traffic
 

Kernel Traffic #211 For 30 Mar 2003

By Zack Brown

Table Of Contents

Mailing List Stats For This Week

We looked at 2758 posts in 14502K.

There were 593 different contributors. 316 posted more than once. 217 posted last week too.

The top posters of the week were:

1. BK->CVS Real-Time Mirror

11 Mar 2003 - 21 Mar 2003 (110 posts) Subject: "[ANNOUNCE] BK->CVS (real time mirror)"

Topics: Assembly, Real-Time, Samba, Version Control

People: Larry McVoyBrandon LowWayne ScottBen CollinsAndreas DilgerMartin J. BlighAndrea ArcangeliJens AxboeH. Peter AnvinDana LacosteJohn BradfordTheodore Y. Ts'oPavel Machek

Larry McVoy announced:

We've been working on a gateway between BitKeeper and CVS to provide the revision history in a form which makes the !BK people happy (or happier).

We have the first pass of this completed and have a linux 2.5 tree on kernel.bkbits.net and you can check out the tree as follows (please don't do this unless you are a programmer and will be using this. Penguin Computing provided the hardware and the bandwidth for that machine and if you all melt down the network they could get annoyed. By all means go for it if you actually write code, though, that's why it is there.)

    mkdir ws
    cd ws
    cvs -d:pserver:anonymous@kernel.bkbits.net:/home/cvs co linux-2.5

Each of the releases are tagged, they are of the form v2_5_64 etc.

Linus had said in the past that someone other than us should do this but as it turns out, to do a reasonable job you need BK source. So we did it. What do we mean by a reasonable job? BitKeeper has an automatic branch feature which captures all parallel development. It's cool but a bit pedantic and it makes exporting to a different system almost impossible if you try and match what BK does exactly. So we didn't. What we (actually Wayne Scott) did was to write a graph traversal alg which finds the longest path through the revision history which includes all tags. For the 2.5 tree, that is currently 8298 distinct points. Each of those points has been captured in CVS as a commit. If we did our job correctly, each of these commits has the same timestamp across all files. So you should be able to get any changeset out of the CVS tree with the appropriate CVS command based on dates.

We also created a ChangeSet file in the CVS tree. It has no contents, it serves as a place to capture the BK changeset comments. Each file which is part of a changeset has an extra comment which is of the form

        (Logical change 1.%d)

where the "1.%d" matches the changeset rev. So you can look for all files that have (Logical change 1.300) in their comments to reconstruct the changeset. NOTE! That information is actually redundant, the timestamps are supposed to do the same thing, let us know if that is not working, we'll redo it. I expect we'll find bugs, please be patient, it takes 4 hours of CPU time on a 2.1Ghz Athlon to do the conversion, that's a big part of why this has taken so long. That's after a week's worth of optimizations.

Each ChangeSet delta has a BK rev associated with it in the comments. We'll be giving you a small shell script which you can use to send Linus patches that include the rev and we'll modify BK so that it can take those patches with no patch rejects if you used that script.

We have a first pass of a real time gateway between BK and this CVS tree done. Right now it is done by hand (by me) but as soon as it is debugged you will see this tree being updated about 1-3 minutes after Linus pushes to bkbits.

Once you guys look this over and decide you like it, we'll do the same thing for the 2.4 tree.

We're also talking to an unnamed (in case it doesn't work out) Linux company who may host bkbits.net for us. If they do that, we'll turn the GNU patch exporter feature in BKD. That means that you'll be able to wget any changeset as a GNU patch, complete with checkin comments. I'm working with Alan on the format, I think we're close though I have to run the latest version past him.

If all of this sounds nice, it is. It was a lot of work for us to do this and you might be wondering why we bothered. Well, for a couple of reasons. First of all, it was only recently that I realized that because BK is not free software some people won't run BK to get data out of BK. It may be dense on my part, but I simply did not anticipate that people would be that extreme, it never occurred to me. We did a ton of work to make sure anyone could get their data out of BK but you do have to run BK to get the data. I never thought of people not being willing to run BK to get at the data. Second, we have maintained SCCS compatible file formats so that there would be another way to get the data out of BK. This has held us back in terms of functionality and performance. I had thought there was some value in the SCCS format but recent discussions on this list have convinced me that without the changeset information the file format doesn't have much value.

Our goal is to provide the data in a way that you can get at it without being dependent on us or BK in any way. As soon as we have this debugged, I'd like to move the CVS repositories to kernel.org (if I can get HPA to agree) and then you'll have the revision history and can live without the fear of the "don't piss Larry off license". Quite frankly, we don't like the current situation any better than many of you, so if this addresses your concerns that will take some pressure off of us.

Another goal is to have the freedom to evolve our file formats to be better, better performance and more features. SCCS is holding us back. So you should look hard at what we are providing and figure out if it is enough. If you come back with "well, it's not BitKeeper so it's not enough" we'll just ignore that. CVS isn't BitKeeper. On the other hand, we believe we have gone as far as is possible to provide all of the information, checkin comments, data, timestamps, user names, everything. The graph traversal alg captures information at an extremely fine granularity, absolutely as fine is possible. We have 8298 distinct points over the 2.5.0 .. 2.5.64 set of changes, so it is 130 times finer than the official releases. If you think something is missing, tell us, we'll try and fix it.

The payoff for you is that you have the data in a format that is not locked into some tool which could be taken away. The payoff for us is that we can evolve our tool as we see fit. We have that right today, we can do whatever we want, but it would be anywhere from annoying to unethical to do so if that meant that you couldn't get at the data except through BitKeeper. So the "deal" here is that you get the data in CVS (and/or patches + comments) and we get to hack the heck out of the file format. Our changes are going to move far faster than CSSC or anyone else could keep up without a lot of effort. On the other hand, our changes are going to make cold cache performance be much closer to hot cache performance, use a lot less disk space, a lot less memory, and a lot less CPU.

Brandon Low extolled:

before Linus started using BK, close to 50% of the revision data that is now saved was completely lost in the process of him merging patches by hand into his repository. I mean be realistic, do you think that Linus kept perfect track of EVERY single ' ' ';' ')' that he changed when merging a patch with minor rejects with his repo? Do you think that every single time that he made a 1 line change or merged a 1 line change that was sent to this list it was documented and recorded? I doubt it. So now we are able to get a publicly available CVS repository with close to two times the data that was ever available before, and infinitely more than was ever available to anyone outside of Linus' own head.

I personally think that Larry has done an amazing job supporting this project and it's goals, and I will give him a big "Thanks for all your support and hard work" at this time. I think that those of you complaining about this as bitmover clearing the road to steal our data should take a long hard look at what you are really saying and consider what BK has given us that we never had before because nothing before was ever usable by Linus.

Pavel Machek noted that the number of ChangeSets in the BitKeeper repository was at that time 17000, about twice what was made available in the CVS version. Wayne Scott explained:

The ChangeSet file has many csets and we only capture around 1/2 of them in CVS ChangeSet file. The extra ChangeSets are grouped together with the merge cset where they were added to the path we are recording. That is correct, but it is not the whole story.

What happens is that most csets modifiy a non overlapping set of files. So while we didn't get every delta to the ChangeSet file, we did capture >90% of the actual changes to the source files in the tree.

Elsewhere, Ben Collins took exception to the fact that BitMover wanted to change the file format. He said:

You are giving us approximately 90% of our data in exchange for the one thing that made using bitkeeper not a total sellout; the fact that the revision history of the repo was still accessible without proprietary software.

I honestly appreciate the work that you and BitMover do for the kernel, but not giving us access to 100% of _our_ data is unacceptable to me. Quite honestly, I think your move is to restrict the possible alternatives to the BK client (the CSSC based ones like I and others had done), which were able to extract 100% of the data, even if they couldn't make use of it in the same way as bitkeeper. Atleast it was there.

You've made quite a marketing move. It's obvious to me, maybe not to others. By providing this CVS gateway, you make it almost pointless to work on an alternative client. Also by providing it, you make it easier to get away with locking the revision history into a proprietary format.

Later he clarified, "I am not against the CVS->BK gateway. I'm all for it. But it's kind of sour given that he now wants to change the disk format of the repo to make it harder to get the data from it. If all he announced was "you now have a CVS->BK repo", I wouldn't be complaining, I'd be patting him on the back." Andreas Dilger replied, "He didn't say "now I'm going to make the repo harder to get data from it". What he said was "now I'm free to change the format from SCCS to something that is more efficient for BK to use". Who knows, maybe the new format will be _easier_ to reverse engineer/parse using 3rd party tools? Also, it's not like he can change things overnight, because there are lots of customers/users who have repos in the old SCCS format, and he doesn't want to completely throw away his current code just to piss off some whiny l-k users. At worst, if it bothers you so much, you can take up the now seemingly forgotten Linux trait of "taking things into your own hands and fixing it to your own needs" and write bk_evil_format_2_CVS conversion tool instead of bitching on l-k about it." Martin J. Bligh also replied to Ben:

As long as we continue to get all the data in an open format, I'm not sure this really matters, personally. If there's some data loss, let's focus on that issue ... but it seems there isn't at the moment.

I'd rather we *didn't* go trying to clone BK and make it file-format compatible underneath ... that seems more incendiary than useful. Cloning other products is always a loosing game, the best you can do is catch them. Personally, I'd prefer we spent the effort making a usable simple SCM that 95% of us can use that does merges and stuff, and not bother trying to follow someone else in file format.

Of course, I'm in no position to dictate to others what they should implement, do what you like ... just my personal opinion. But there's always the possiblity we can make something that fits kernel development *better*, rather than playing catchup to BK all the time ;-)

Larry replied:

My personal opinion is that BK maps only so so well onto the kernel development effort. It's not horrible, it's closer than any other SCM, but it could be better. The kernel guys tend to be "more loose" than commercial guys, i.e., stuff is tried, it sits in Alan's tree for a while or DaveJ's tree and then is rejected if it is found to be bad. You really need a sort of "lossy" SCM system, one which is willing to throw data away. BK is absolutely not about losing information, we view everything as valuable, even bad ideas. That matches the commercial world better than the Linux world.

I _think_ that Arch is closer. You will definitely give up some stuff if you move to Arch but you will also gain some stuff. Arch is willing to pick and choose, we aren't, we're sort of an all or nothing answer. Pavel is all hot and bothered about PRCS but PRCS is sort of BK without the distribution, gui tools, and scripting. It's a step backwards as far as I can tell (don't get me wrong, we've acknowledged the coolness of PRCS on our website for years and I tried to team up with Josh, I'm a fan). You should really look at Arch, it may be a better fit. And these days, if you could find a better fit, none of us at BitMover would shed a tear if you moved off BK. This has *not* been a pleasant experience for us.

Andrea Arcangeli also said to Ben:

You shouldn't care less of the disk format. You *can't* run bk in the first place to reach those files, it's by pure luck that somebody is been fine to give away his right to write free software (oh and proprietary software too but we don't care :) in the SCM arena and to provide this info to you via rsync or whatever proxy or open protocol that tytso mentioned is doable.

Before you can remotely care about the disk format you've to reverse engeneer the network protocol first, having more proprietary stuff there won't make differences for us. And of course it makes perfect sense for Larry to hide the stuff better, but even if he encrypts it, the secret key has to be in the bk binary, I mean, it's all in open source assembly anyways, if you figured out the network protcol, you shouldn't have an order of magnitude more of troubles to figure out the new file format too. NOTE: I don't want to discuss the legal details of reading the open source assembly, this was only an example ;).

really, what we care is the data, and what I discussed in the last weeks with Larry about the kernel CVS at first sigh seems enough for kernel developers, what matters is the _mainline_ evolution. All other trees matters much less (and NOTE: all important non mainline trees don't use bitkeeper anyways). If getting the changesets with dates will be too hard I assume Larry could help on it. Some script should do it pretty well thanks to the logical tag in the log. I know it's not the most useful format for export but this is reliable, documented and open and it makes it trivial to checkout and search the file logs. which makes it very usable immediatly w/o the need of new software which is good for us kernel developers in the short term. This is a good short/mid term solution.

IMHO cloning bitkeeper would be an option if Larry would be supporting it, but that is obviously not the case.

There is no point to complain about the change of format of files in the Larry has all the rights to change the file format even after you reverse engeneered it the first second third fourth time, so all your effort will break in seconds. You can spend the rest of your life to keep up with Larry and he'll always be ahead of you. We have to do this with the SMB protocol because there's no open ""exporter"", but here Larry provided the data, and the data belongs to the community in the first place so there's no need to slowdown innovation here trying to catch up with closed proprietary protocols. And note: if you don't like that linux is developed with bk you should speak with Linus not with Larry. That is Linus's choice, Larry couldn't make that change.

If you complain about the file format change, it means you realized right now you did a mistake in depending on bk in the first place.

I think we reached a point of balance here that will solve all the collisions. The CVS is a "stability" point. The lack of data-availability with CVS or similar open protocol would force us to reverse engeneer bk to access the data, and the availability of CVS immediatly make us wasting time reverse engeneering bk. Cloning bitkeeper is a waste of time if the CVS just exports the data correctly.

Please focus on this: the only thing we miss is the visibility of the jfs tree and similar other bits that aren't even guaranteed to be merged in mainline. But that doesn't worry me at all, in one year from now if the jfs tree didn't merge correctly it won't matter what was in such dead tree.

If you want to contribute, stop these threads, and start importing CVS into a more powerful SCM and let us know an URL where we can access the data from there. I will only answer to a working URL, either that or live with CVS. The SCM can be evolved over tiem. If this new underground domain will be better than bitkeeper than jfs and Linus as well could join us in the future. In the meantime CVS will do fine and it guarantees the openess of the linux info. As you probably know I don't have much time in helping with SCM developement by I can t try give my $.02 (or at the very least I want to be still allowed to give my $.02! ;).

I won't answer further emails about this issue to avoid hurting the l-k traffic too much (the last bk threads even made me overlook the BK->CVS announcement after a one day and half of email backlog go figure ;).

And of course many thanks to Larry for the BK->CVS effort! While I think it was due, it certainly takes some relevant effort to do it.

Jens Axboe was also bothered by the move to modify the file format. In response to Ben's initial complaint, he agreed completely, and said he might have to stop using BitKeeper, if the situation continued as it was going. Andreas pointed out that this was a pretty gloomy attitude, considering BitMover had just provided an almost complete CVS gateway. He said, "Some people will just never be happy no matter what you give them." And Jens replied:

I've been very happy with BK, been using it shortly after Linus started doing so. Mostly out of curiosity at first, later because it was actually quite useful. I even see myself as a fairly pragmatic individual, but even so I do find it increasingly difficult to defend my BK usage.

So please stop thinking you can judge that easily by pushing me into your nice little 'some people will never be happy bla bla' category.

Andreas apologized for lumping Jens in with the other anti-BK folks; and said he wished the BitKeepere flame wars would stop.

Close by, H. Peter Anvin said:

From what I can gather, the question is very simple:

"Can we get our data out of BK into some kind of open format?"

It's an important question. If the answer is "yes, but only the stuff that can be mapped onto CVS" then that's a significant data loss, and if BitMover changes the data format without documentation, then there is no longer a way to get all the data out.

Presumably the CVS exporter could get augmented with some kind of metadata export... perhaps an XML schema that describes how the various points are to be linked or whatnot... it won't turn CVS into BK overnight (so Larry can still sleep at night), but it would give BitMover the freedom to change their data format.

Dana Lacoste replied, "if CVS loses all that data, is that BK's fault? BK's so powerful because it has more information than anyone else, but it's not their fault (and it's not proprietary data) that no-one else can deal with the data when it's exported, now is it????" H. Peter replied, "You're missing the point completely. Of course it's not BK's fault that CVS can't represent the data. However, one of the (valid!) selling points of BK was "we won't hold your data hostage." That requires that you can export both the data and the metadata into some kind of open format. Since CVS clearly can't be that open format (CVS being insufficiently powerful), the additional metadata needs to be available in some kind of auxilliary form. It's then, of course, not BK's fault that CVS can't possibly make use of that auxilliary metadata."

John Bradford pointed out, "I thought that BK has been able to export everything to a text file since the first version." And Larry said:

bk export -tpatch -r1.900 > patch.1.900
bk changes -v -r1.900 > comments.1.900

Been there forever. So has ways to get all the metadata from the command line without having to reverse engineer the file format. See

http://www.bitkeeper.com/manpages/bk-prs-1.html

it's all there. Always has been.

Wayne wanted me to point that it is easy to write the BK to CVS exporter completely from the command line, we prototyped it that way, the only reason we rewrote part of it in C was for performance. The point being that you guys could have done this yourself without help from us because all the metadata is right there. Ditto for anyone else worried about getting their data out of BK now or in the future. The whole point of prs is to be able to have a will-always-work way to get at the data or the metadata, it makes the file format a non-issue.

H. Peter was very happy to hear this, and asked if the output from that could be made available automatically, along with the CVS tree. And Theodore Y. Ts'o said:

More importantly, even if someone isn't allowed to use the BK command line tool because once upon time, a long time ago, they submitted a patch to arch or subversion, they can still find someone is allowed to set up a bk daemon under the terms of the FUL, connect to the BK daemon using a http client, and extract the full diff of any changeset that way. This doesn't have to be the bkd on bkbits.net; anyone who is authorized to use BK under the terms of the FUL can set up a bk daemon to be listening on a port of any machine for which they have shell access (it doesn't even require root privs). And every last changeset can always be made available using this path.

So to the people are complaining that they won't be able to get out their data if a future version of BK uses a more powerful representation than SCCS files ---- would you like some more whine with your cheese?

2. Linux 2.5.64-mm8 Released

16 Mar 2003 - 20 Mar 2003 (20 posts) Archive Link: "2.5.64-mm8"

People: Andrew MortonMike Galbraith

Andrew Morton announced:

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.64/2.5.64-mm8/

3. Fix For Ancient Scheduler Bug

16 Mar 2003 - 20 Mar 2003 (5 posts) Archive Link: "[patch] sched-2.5.64-bk10-C4"

Topics: Big O Notation, Scheduler

People: Ingo MolnarAndrew MortonMike Galbraith

Ingo Molnar said:

the attached patch fixes a fundamental (and long-standing) bug in the sleep-average estimator which is the root cause of the "contest process_load" problems reported by Mike Galbraith and Andrew Morton, and which problem is addressed by Mike's patch.

the bug is the following: the sleep_time code in activate_task() over-estimates the true sleep time by 0.5 jiffies on average (0.5 msecs on recent 2.5 kernels). Furthermore, for highly context-switch intensive and CPU-intensive workloads it means a constant 1 jiffy over-estimation. This turns the balance of giving and removing ticks and nils the effect of the CPU busy-tick, catapulting the task(s) to highly interactive status - while in reality they are constantly burning CPU time.

the fix is to round down sleep_time, not to round it up. This slightly under-estimates the sleep time, but this is not a real problem, any task with a sleep time in the 1 jiffy range will see timekeeping granularity artifacts from various parts of the kernel anyway. We could use rdtsc to estimate the sleep time, but i think that's unnecessary overhead.

the fixups in Mike's scheduler patch (which is in -mm8) basically work around this bug. The patch below definitely fixes the contest-load starvation bug, but it remains to be seen what other effects it has on interactivity. In any case, this bug in the estimator is real and if there's any other interactivity problem around then we need to deal with it ontop of this patch.

this bug has been in the O(1) scheduler from day 1 on basically, so i'm quite hopeful that a number of interactivity complaints are fixed by this patch.

4. Local User Security Exploit Against 2.2 And 2.4 Kernels

17 Mar 2003 - 27 Mar 2003 (106 posts) Archive Link: "Ptrace hole / Linux 2.2.25"

Topics: Disks: IDE, FS: ext3, User-Mode Linux

People: Alan CoxMathieu LafonBen PfaffMatthew GrantTomas SzepeJeff GarzikArjan van de VenBen LaHaiseJeff Dike

Alan Cox reported:

The Linux 2.2 and Linux 2.4 kernels have a flaw in ptrace. This hole allows local users to obtain full privileges. Remote exploitation of this hole is not possible. Linux 2.5 is not believed to be vulnerable.

Linux 2.2.25 has been released to correct Linux 2.2. It contains no other changes. The bug fixes that would have been in 2.2.5pre1 will now appear in 2.2.26pre1. The patch will apply directly to most older 2.2 releases.

A patch for Linux 2.4.20/Linux 2.4.21pre is attached. The patch also subtly changes the PR_SET_DUMPABLE prctl. We believe this is neccessary and that it will not affect any software. The functionality change is specific to unusual debugging situations.

We would like to thank Andrzej Szombierski who found the problem, and wrote an initial patch. Seth Arnold cleaned up the 2.2 change. Arjan van de Ven and Ben LaHaise identified additional problems with the original fix.

A number of folks reported breakage with Alan's patch. Mathieu Lafon reported, "The patch breaks /proc/<pid>/cmdline and /proc/<pid>/environ for 'non dumpable' processes, even for root. We need to access theses proc files for processes monitoring. Included is a patch to restore this functionnality for root."

Ben Pfaff also noticed a problem, and said to Alan, "I am concerned about this change because it will break sandboxing software that I have written, which uses prctl() to turn dumpability back on so that it can open a file, setuid(), and then execve() through the open file via /proc/self/fd/#. Without calling prctl(), the ownership of /proc/self/fd/* becomes root, so the process cannot exec it after it drops privileges. It uses prctl() in other places to get the same effect in /proc, but that's one of the most critical." Alan replied:

The dumpability is per mm, which means that you have to consider all the cases of a thread being created in parallel to dumpability being enabled.

So consider a three threaded process. Thread one triggers kernel thread creation, thread two turns dumpability back on, thread three ptraces the new kernel thread.

Proving that is safe is non trivial so the current patch chooses not to attempt it. For 2.4.21 proper someone can sit down and do the needed verification if they wish.

Matthew Grant also said, of Alan's initial post:

This patch really breaks UML using the skas mode of thread tracing skas3 patch on quite a significant amount of machines out there. The skas mode is a lot more secure than the traditional UML tt mode. I guess this is related to the below...

I am running a UML site that a lot of hospitals ad clinics in Bangldesh depend on for there email. It allows them to work around the corruption and agrandidement of the ISPs over there. The skas3 mode patch is needed for the site to run securely. Tracing thread mode does not cut it.

There are also a large number of other telehoused ISP virtual hosting machines that use this stuff, and it is actually proving to be quite reliable.

I have attached the skas3 patch that Jeff Dike is currently using, and the patch that you have produced. Could you please look into the clash between them, and get it fixed.

Thank you - there are lots out there who will appreciate this.

Shortly thereafter, Alan said, "I take patches."

Elsewhere, Arjan van de Ven brought Alan's patch up to kernel 2.4.21pre5, and posted it. Tomas Szepe asked:

So what happens now?

Is this critical enough for 2.4.21 to go out? Or can it wait like some other fairly serious stuff such as the ext3 fixes? What about the current state of IDE?

Would it make sense to repackage 2.4.20 into something like 2.4.20-p1 or 2.4.20.1 with only the critical stuff applied?

Alan replied, "If you build your own kernels apply the patch, if you use vendor kernels then you can expect vendor kernel updates to appear or have already appeared. " But Tomas said Alan had avoided his questions.

Jeff Garzik also said to Tomas' initial query:

There shouldn't be a huge need to rush 2.4.21 as-is, really. If you want an immediate update, get the fix from your vendor.

Plus, it's a local root hole, and there are still tons of those left out there to squash.

5. Deprecating .gz Format On kernel.org

19 Mar 2003 - 25 Mar 2003 (71 posts) Archive Link: "Deprecating .gz format on kernel.org"

Topics: Compression, Modems

People: H. Peter AnvinTigran AivazianJamie LokierEric SandallArjan van de VenMartin J. BlighKurt GarloffArnaldo Carvalho de Melo

H. Peter Anvin said:

At some point it probably would make sense to start deprecating .gz format files from kernel.org.

I am envisioning this as a three-phase changeover:

a) Get all mirrors to carry .bz2 format. This would affect the following sites:

DUTH:format=gz
GARBO:format=gz
HCMC:format=gz
IGLU:format=gz
LINUXAID:format=gz
LLARIAN-NET:format=gz
MINET-FR:format=gz
NC-ORC:format=gz
PCSS:format=gz
PROGRAMVAREVERKSTEDET:format=gz
PUB-FTP-UNIVERSITY-OF-OLDENBURG:format=gz
RN-RNO:format=gz
TASK:format=gz
TELEPAC:format=gz
TENGU-EASYNET-FR:format=gz
UNC-METALAB:format=gz
WEBLAB:format=gz

b) Once that is done, change the robots to no longer require .gz files; .bz2 files uploaded would be signed but no .gz file would be generated.

-> If we get a complete loss of data here, all .gz files would be lost.

c) At some point, deprecate .gz uploads entirely and remove all the old .gz files. After that point .gz files uploaded would be treated just like .Z, .zip or any other "unmanaged" compression format.

Now, the questions that come up are:

i) Does this sound reasonable to everyone? In particular, is there any loss in losing the "original" compressed files?

ii) Assuming a yes on the previous question, what time frame would it make sense for this changeover to happen over?

Martin J. Bligh was thrilled with all of this, and hoped the change would happen as soon as possible. But Tigran Aivazian pointed out to H. Peter:

there is at least one reason for the "original" .gz files. Here are the logical steps:

a) any Linux distribution contains their own "linux" package with the source base being "vanilla" Linux .tar.gz file

b) switching such to .tar.bz2 will make building the package longer because of longer extract times

c) re-running tar to generate a .tar.gz from .tar.bz2 and store the .tar.gz instead will make customers suspicious --- i.e. they will have to ask "is this _really_ a plain Linux tree or do I need to run diff(1) to verify, just in case?"

See the reasoning? However, I agree that this reason is very weak. But you were interested in any reasons, including weak ones

Jamie Lokier said:

Personally I fetch the .bz2 tar files for a few base kernel versions, but I fetch the .gz patch files.

This is so that I can "zgrep" through the patch files looking for which version changed some feature or API.

bzgrep exists, but it is way too slow.

So if there were only .bz2 patch files, I would fetch them and convert them back into .gz files on my local mirror.

Which is ok of course, but then the signatures don't match any more.

Well, that's my really weak reason for liking .gz patches.

Eric Sandall suggested getting the signature from the tar file instead of the compressed file, so the compression method wouldn't matter anymore. Jamie pointed out that it would take a lot of time to uncompress the file, just in order to check the signature. And Eric replied, "True...for large files it'd be nice to know if you have the correct tarball _before_ spending all that CPU time decompressing it. It's a trade off, mostly, CPU time for more generic useage."

Elsewhere, Arjan van de Ven remarked, "I can't speak for the others, but Red Hat Linux uses the .bz2 files in kernel rpms." Arnaldo Carvalho de Melo added that Connectiva also did this. Kurt Garloff added SuSE to that list. Bruno Cornec added MandrakeSoft. Eric also said, "And Source Mage (http://www.sourcemage.org). :)" Dagfinn Ilmari Mannsaker said Debian did as well. Jon Portnoy included Gentoo.

There was more discussion of the main point, and at one point H. Peter said:

OK, there seems to be enough resistance to this to put it off for a year or two. Since that means I don't have to do any work, that plan has inherent appeal to me.

In other words, the current setup will remain for now. HOWEVER, I would like to recommend that mirror sites start carrying .bz2 files if they want to carry only one format.

Martin asked, "Can we at least switch the default upload format to bz2? I would think that's a reasonably simple change to the robot? For those of us uploading stuff over a slow modem link, the extra efficiency would be much appreciated." But H. Peter replied, "No, I don't want to muck with working scripts. I suggest setting up a script to upload to the /staging area and convert automatically. I have put a script called bz2togz in /usr/local/bin to help out."

6. ksymoops 2.4.9 Released

20 Mar 2003 (1 post) Archive Link: "Announce: ksymoops 2.4.9 is available"

People: Keith OwensJames W. LaferriereMaciej W. Rozycki

Keith Owens announced:

ftp://ftp.<country>.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4

ksymoops-2.4.9.tar.gz           Source tarball, includes RPM spec file
ksymoops-2.4.9-1.src.rpm        As above, in SRPM format
ksymoops-2.4.9-1.i386.rpm       Compiled with 2.96 20000731, glibc 2.2.5
patch-ksymoops-2.4.9.gz         Patch from ksymoops 2.4.8 to 2.4.9.

Changelog extract

The change to decode the Code: line in two chunks allows architectures that have variable length instructions to safely dump the code before eip. The code from eip onwards is always reliable, the code before eip may not be reliable, this is reflected in the headings before each chunk of decode output.

Updating ksymoops alone has no effect on the decode. The arch specific kernel dump routine must :-

(a) Add the string " VLI" to the line containing the eip/psr/pc/ip/psw to tell ksymoops that this dump has variable length instructions. For example EIP: 0060:[<c014fedf>] VLI Not tainted

(b) Dump bytes before and after eip, on one code line. ksymoops handles up to 64 bytes of Code: data.

(c) Enclose the eip byte in <> or ().

If you do not want to decode before eip with variable length instructions, do not change your arch tree. ksymoops will continue to decode as before.

Architectures that already dump code before the eip do not need to be changed. AFAIK they all have fixed length instructions.

Some people have reported problems building ksymoops, with unresolved references in libbfd (htab_create, htab_find_slot_with_hash). Try http://www.cs.helsinki.fi/linux/linux-kernel/2002-13/0196.html first, if that does not work, contact the binutils maintainers. This is not a ksymoops problem, ksymoops only uses libbfd. Any unresolved references from libbfd are a binutils problem.

7. I2C Cleanups

20 Mar 2003 (19 posts) Archive Link: "[BK PATCH] i2c driver changes for 2.5.65"

Topics: I2C, Version Control

People: Greg KH

Greg KH posted a bunch of BitKeeper changesets, explaining:

Here are some more i2c driver cleanups. These do the following items I mentioned in my last set of patches:

There is also a i2c spelling patch in here, and I've added the i2c-isa driver, which almost isn't even a driver at all, based on the size of it.

Even with adding a new driver, the sum of these patches is still a smaller number of lines than we started with, which is always nice.

Please pull from: bk://kernel.bkbits.net/gregkh/linux/i2c-2.5

Things left to do after this:

8. lk-changelog.pl 0.83 Released

21 Mar 2003 (1 post) Archive Link: "lk-changelog.pl 0.83"

Topics: Version Control

People: Matthias Andree

Matthias Andree announced:

lk-changelog.pl aka. shortlog version 0.83 has been released.

This script is used by Linus and Marcelo to rearrange and reformat BK ChangeSet logs into a more human-readable format, and the official repository is bk://gkernel.bkbits.net/BK-kernel-tools/

The changes are listed at the end of the script below.

You can always download this script and GPG signatures from http://mandree.home.pages.de/linux/kernel/

My thanks go to Vitezslav Samel who has spent a lot of time on digging out the real names for addresses sending in BK ChangeSets.

9. Minutes From The March 21 LSE Conference Call

21 Mar 2003 (1 post) Archive Link: "Minutes from March 21 LSE Call"

People: Hanna LinderJens AxboeMichael CohenCliff White

Hanna Linder posted:

LSE Tech Call Minutes March 21, 2003

Minutes Compiled by Hanna Linder. Please send corrections or additions to lse-tech@lists.sourceforge.net

I. Matthew Fanto: The Cello Disk Scheduling Framework

If you have seen on the lkml the schedulers being worked on, like: cfq, deadline, and anticipatory io schedulers. Cello is another one. It is a two level disk scheduler.

The two levels are a request is made and is put into a class specific queue. Currently there are three different applications classes:

real time,
interactive best effort,
throughput intensive.

Different algorithms and insertion methods work better for each type so it maintains different queues for each class type.

After assigning to queue it assigns weights. there is a 4th queue which is class unaware and it chooses which tasks to take based on weights and queue.

There are two ways to assign weights:

         proportionate time allocation.
                doesnt take into account amount of data being transfered.
         proportionate byte allocation.
                takes into account the amount of data going to transfer.
there are advantages to each method.

Most of the code is done. Bill Irwin saw the code last night. No one else has yet. Hanna encouraged him to send it out anyway.

Matt said he has a paper he will put on line. Jens Axboe read the paper a while ago and wanted to see it actually work.

Cliff offered to help Matt run it under the STP so they can help test it. Sometimes STP isnt happy so make sure to cc cliff white to get help getting it working.

Gerrit reiterated the importance of sending out the code.

Originally got the project from mjc (Michael Cohen) and found lots of performance increases based on his implementation in the user space. He asked Matt to work on it in the kernel.

Matt is going to send out a patch today to lse-tech.

II. Cliff White - Scheduling benchmarks

Has been redoing the AIM stuff. Should have an early alpha release next week. Should work with both aim7 and aim9. He is switching some of the static stuff to dynamic to make it easier to work with. He hopes to have something runnable next week.

What it does is take a list of microbenchmarks, decides on a num of tasks for each user, forks a certain number of children. each child runs a number of tasks in a random order.

Currently looking at a problem where if 20-30 children forking seeing 3-4 seconds delta between children starting and stopping even though they are all doing the same thing. The lifetime of the parent is about 50 seconds. There is a lot of contention between the children so it may or may not be a problem. At the very least this looks like a good way to stress the scheduler.

10. IDE Todo List

22 Mar 2003 - 23 Mar 2003 (18 posts) Archive Link: "IDE todo list"

Topics: Disks: IDE, Disks: SCSI, Forward Port, Hot-Plugging, PCI, Power Management: ACPI, Serial ATA

People: Alan CoxJens AxboeMark Lord

Alan Cox posted his IDE todo list:

(Minus some stuff which is NDA'd because it involves unreleased chips etc)

Jens Axboe took exception to the "Finish verifying 256 sector I/O or larger on LBA48 [How to handle change dynamically on hotplug ?]" item, saying:

That is basically impossible. How are you going to handle the case where you have a queue full of 256 request writes, and the plugged in disk chokes on them? And insolvable unless you start setting aside requests simply for this purpose. Also breaks the pseudo atomic segments that a single request represents. This is just way beyond ugly...

This is a generic problem of course, and the typical answer is to go by the rules of the lowest common denominator if hot plug can cause you queue limits to be violated (may be other problems than simply max sector count).

Alan said, "I don't think its impossible at all. Remember if you hotplug a drive you *dont* want the pending I/O to hit the new drive!" And Jens replied, "In that case it could be done, the key point is that no resizing needs to be done. The rest is purely driver implementation :)"

11. LUFS Userland FS 0.9.5 Released

24 Mar 2003 (1 post) Archive Link: "[ANNOUNCE] LUFS (Linux Userland File System) 0.9.5 released"

People: Florin Malita

Florin Malita announced:

LUFS 0.9.5 has been released.

LUFS (http://lufs.sourceforge.net) is a hybrid userspace file system framework. It comes with a bunch of useful file system modules: sshfs, ftpfs, gnutellafs, gvfs, locasefs, gvfs, cardfs, cefs, etc.

2.4 & 2.5 kernel patches available at http://sourceforge.net/project/showfiles.php?group_id=57332

12. Repository For Version Control Systems

24 Mar 2003 (1 post) Archive Link: "ftp://ftp.kernel.org/pub/scm/"

Topics: Version Control

People: H. Peter Anvin

H. Peter Anvin announced:

I have set up a tree on kernel.org for SCM repositories. Currently it contains a mirror of the bk->cvs repository tree at:

ftp://ftp.kernel.org/pub/scm/linux/kernel/bkcvs/

The main intent for this is to make it possible to rsync the full repository; we currently don't have CVS pserver access or anything like that (maybe later.)

This tree is mirrored hourly from kernel.bkbits.net.

I'm hoping someone with a BK license and a kernel.org account could add the bk exports file to this tree, as well.

13. Linux 2.5.66 Released

24 Mar 2003 - 27 Mar 2003 (36 posts) Archive Link: "Linux 2.5.66"

People: Linus Torvalds

Linus Torvalds announced 2.5.66, saying, "A lot of changes all over. Most notably probably the fbcon updates, it's really all over the map - mostly a lot of very small fixes."

14. Open POSIX Test Suite 0.9.0

24 Mar 2003 (1 post) Archive Link: "[ANNOUNCE] Open POSIX Test Suite 0.9.0"

Topics: POSIX

People: Rolla N. SelbakJim Houston

Rolla N. Selbak announced:

Release 0.9.0 of the Open POSIX Test Suite is now available at http://posixtest.sourceforge.net. This third release contains POSIX conformance tests for the POSIX functions:

Threads (90% - not including tagged or rwlock-related functions)
Signals (90% complete)
Message queues (100% complete)
Semaphores (100% complete)
Timers (100% complete - tags TMR and CS)

It also contains bug fixes from 0.2.0.

The release notes that appear on download describe how to compile and run these tests.

The README page and the Open POSIX Test Suite website (above) give more information on the project goals and progress as well as information on how to contribute or contact us if you are interested.

Many thanks to Jim Houston, Jerome Marchand and other members of the POSIX testing community for their bug fixes, patches, and suggestions on how to improve the 0.2.0 suite.

The Open POSIX Test Suite is an open source test suite with the goal of creating conformance test suites, as well as potentially functional and stress test suites, to the functions described in the IEEE Std 1003.1-2001 System Interfaces specification. Initial work is focusing on timers, threads, semaphores, signals, and message queues.

Feel free to contact posixtest-discuss@lists.sourceforge.net if you would like further information.

15. SysFS Migration

25 Mar 2003 - 26 Mar 2003 (7 posts) Archive Link: "add eeprom i2c driver"

Topics: FS: sysfs, I2C, Version Control

People: Jan DittmerGreg KH

Jan Dittmer posted a patch and said, "This adds support for reading eeproms. Tested against latest bk changes with i2c-isa." But Greg KH replied:

I'd like to hold off in submitting the i2c chip drivers just yet, due to the changes for sysfs that are going to be needed for these drivers.

As an example of the changes necessary, here's a patch against the i2c cvs version of the eeprom.c driver that converts it over to use sysfs, instead of the /proc and sysctl interface. It's still a bit rough, but you should get the idea of where I'm wanting to go with this. As you can see, it takes about 100 lines of code off of this driver, which is nice.

I'm copying the sensors list too, as they wanted to see how this was going to be done.

Jan replied, "Looks good, I'll try to come up with a converted version of via686a later. Should tidy up a lot." At around that time, Greg also said:

Ok, in finding out a bit more of what the EEPROM driver is trying to do, it looks like it qualifies for the "binary file in sysfs" exception. It's just exporting a binary blob of data that exists in hardware through the kernel to userspace, for userspace to do with it what it wants to.

I'll work on converting the driver to that interface later tonight, unless someone beats me to it :)

16. JFS 1.1.2 Released

25 Mar 2003 (1 post) Archive Link: "[ANNOUNCE] JFS 1.1.2"

Topics: FS: JFS

People: Dave Kleikamp

Dave Kleikamp announced:

Release 1.1.2 of JFS was made available today.

Drop 65 on March 25, 2003 includes fixes to the file system and utilities.

Utilities changes

File System changes

Note: The 2.4.21 and 2.5 kernel.org development kernels are kept up to date with the latest JFS code. The file system updates available on the web site are only needed for maintaining earlier 2.4 kernels.

For more details about JFS, please see our website: http://oss.software.ibm.com/jfs

17. Regular Patches Becoming Second-Class Citizens

26 Mar 2003 (12 posts) Archive Link: "[BK PULL] PCMCIA changes"

Topics: Version Control

People: Linus TorvaldsDominik BrodowskiRussell King

Up until now, Linus Torvalds has maintained that developers who used BitKeeper would not receive better treatment than developers who did not; he would be just as likely to apply a patch that came from BitKeeper as from traditional tools. The following little thread points to a change in that situation:

Russell King asked Linus to pull some PCMCIA patches from Russell's local BitKeeper repository. Linus replied, "Pulled. Russell, since you use BK _and_ you're working on PCMCIA, can you work as the middle man for the patches that Dominik has been sending out? They all look sane, and the "driver services" socket add/remove abstraction in particular looks like something that is needed. I just didn't have time to check them out more deeply and test them." And Dominik Brodowski replied, "Fine with me - the four patches are on their way to Russell."

18. Linux 2.4.21-pre6 Released

26 Mar 2003 - 27 Mar 2003 (24 posts) Archive Link: "Linux 2.4.21-pre6"

Topics: Disks: IDE, Version Control

People: Marcelo TosattiChristoph HellwigLinus Torvalds

Marcelo Tosatti announced 2.4.21-pre6, saying, "We are approaching -rc stage. I plan to release -pre7 shortly which should fixup the remaining IDE problems (thanks Alan!) and -rc1 later on." Christoph Hellwig gave a growl, and said, "once again this is not tagged in BK. Could you _please_ ask Linus for his nice update release, tag and publish script?" And Linus Torvalds replied:

Hey, I'm a retard. I don't actually do the tagging with a script, since I usually want to create the tree privately first, and run a final compile cycle on it. So I tag it by hand, and then I have a script that outputs a list of commands that I just cut-and-paste directly. It's ugly, but it works (and it means that running the script doesn't _do_ anything: I have to actually take one last look at what I'm going to do before I start it all up).

The "release script" is this piece of crap:

        #!/bin/sh
        echo "bk export -w ../linux-$2"
        echo "bk export -h -tpatch -rv$1.. > ../patch-$2"
        echo "export PATH=\$PATH:/home/torvalds/BK/tools"
        echo "changelog v$1 v$2 > ../ChangeLog-$2"
        echo "shortlog --width=72 < ../ChangeLog-$2 > ../ChangeLog"
        echo "cd .."
        echo "tar cf linux-$2.tar linux-$2"
        echo "gzip -9 patch-$2"
        echo "gzip -9 linux-$2.tar"
        echo "touch LATEST-IS-$2"

and that's it. I'd just do "../release-script 2.5.66 2.5.67" from my kernel directory when I want to generate a 2.5.67 release.

Ok, I'm embarrassed to even admit to doing this. Don't rub it in. It's useful to me because sometimes I don't do a full release - I just want to pre-generate a change-log of the diff, for example, to judge whether I forgot about something.

19. NFS-utils 1.0.3 Released

27 Mar 2003 (1 post) Archive Link: "ANNOUNCE : nfs-utils 1.0.3"

Topics: FS: NFS

People: Neil Brown

Neil Brown said:

This is announcement for the release of nfs-utils 1.0.3

It is available from sourceforge:
http://nfs.sourceforge.net
or kernel.org
http://www.{countrycode}.kernel.org/pub/linux/utils/nfs/

This release is primarily for people using or planning to use nfsd on 2.5 series kernels.

Other user's may upgrade if they like. There are a few small fixes which might be of interest. See the Change Log.

The system call interface for exporting filesystems via NFS is defined using types like "__kernel_dev_t" which should only be used internally to the kernel. Using these in a user-space interface was a poor decission.

There are patches available for 2.5 which change __kernel_dev_t, at least for i386, to be 32 bit instead of 16 bit. This is an incompatable change and nfs-utils does not work correctly on kernels with these patches.

This version of nfs-utils uses an alternate interface which is available in 2.5 to export file systems. This interface is not sensitive to changes in kernel internal type definitions, and so will work independantly of these patches.

If the new interface is not available (as it is not in 2.4), this version of nfs-utils will fall back to the old style interface. Thus this version should work on all platforms and all kernel releases.

It may be that when these patches are finally included into the mainline, we will able to keep the old systemcall interface working. However as this is not certain, using 1.0.3 (or later) is probably the best choice.

 

 

 

 

 

 

Sharon And Joy
 

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.