|Home | News | RSS Feeds | Mailing Lists | Authors Info | Mirrors | Stalled Traffic|
Table Of Contents
|1.||26�Feb�2003�-�17�Mar�2003||(200 posts)||BitBucket: A BitKeeper Competitor; Version Control Discussion|
|2.||8�Mar�2003�-�14�Mar�2003||(12 posts)||kconfig Update|
|3.||10�Mar�2003�-�16�Mar�2003||(6 posts)||perfctr 2.5.0 Released|
|4.||11�Mar�2003�-�13�Mar�2003||(3 posts)||VMRegress 0.8a Released|
|5.||13�Mar�2003�-�14�Mar�2003||(7 posts)||NetFlow Data Support|
|6.||13�Mar�2003||(2 posts)||Some Developers Unhappy With Linus Dropping Patches|
|7.||13�Mar�2003||(1 post)||Modutils 2.4.23 Released|
|8.||13�Mar�2003�-�15�Mar�2003||(19 posts)||i2c Updated To The New Driver Model|
|9.||13�Mar�2003�-�14�Mar�2003||(3 posts)||Linux RAID FAQ Version 0.0.12 Released|
|10.||14�Mar�2003||(2 posts)||Status Of Promise Driver Development|
|11.||14�Mar�2003�-�17�Mar�2003||(66 posts)||BitMover Considers Lawsuit Over BitBucket Development|
|12.||15�Mar�2003||(1 post)||Mailing List Stats|
|13.||16�Mar�2003�-�17�Mar�2003||(5 posts)||XFS, ReiserFS, And ext3 Comparisons|
|14.||16�Mar�2003�-�17�Mar�2003||(7 posts)||Support For NUMAQ Machines With More Than 8 IOAPICs|
|15.||16�Mar�2003||(1 post)||KDB Kernel Debugger 4.0 For 2.4.19, 2.4.20, i386 And ia64|
|16.||17�Mar�2003||(1 post)||LKST (Linux Kernel State Tracker) 1.4 Released|
|17.||17�Mar�2003||(1 post)||SquashFS 1.2 Compressed Filesystem Released|
|18.||17�Mar�2003||(7 posts)||BitKeeper Gateway To CVS|
|19.||17�Mar�2003||(4 posts)||New Configuration Organization For IPX Under 2.5|
|20.||17�Mar�2003||(2 posts)||kernel.org Planned Downtime|
|21.||17�Mar�2003�-�18�Mar�2003||(15 posts)||Linux 2.5.65 Released|
|22.||18�Mar�2003�-�19�Mar�2003||(14 posts)||Kernel 2.5.65-mm1 Released|
|23.||18�Mar�2003||(1 post)||VM Subsystem Documentation|
|24.||18�Mar�2003||(1 post)||New relayFS High-Speed Data Relay Filesystem|
|25.||18�Mar�2003�-�19�Mar�2003||(9 posts)||cvsps Project To Support BitKeeper Metadata In CVS|
|26.||18�Mar�2003||(1 post)||Blank-Screen On Demand|
|27.||18�Mar�2003�-�19�Mar�2003||(7 posts)||Linux On Small Systems|
|28.||19�Mar�2003||(2 posts)||I2O Update|
|29.||19�Mar�2003||(2 posts)||Essential Reality P5 Data Glove Not HID Compliant|
|30.||19�Mar�2003||(1 post)||New OpenHPI Implementation For SAForum's HPI|
IntroductionTom Lord, maintainer of the Arch version control system, needs some help. He's flat broke and can't find work. Arch is a really important project, and it would be great to keep him healthy and housed so he can keep working on it. If you like, you can give him some money as "firstname.lastname@example.org" on PayPal, or maybe someone could help him find work. Check out his resume if that's a possibility.
Mailing List Stats For This Week
We looked at 1890 posts in 9305K.
There were 495 different contributors. 261 posted more than once. 200 posted last week too.
The top posters of the week were:
1. BitBucket: A BitKeeper Competitor; Version Control Discussion
26�Feb�2003�-�17�Mar�2003 (200 posts) Archive Link: "BitBucket: GPL-ed BitKeeper clone"
Topics: Microkernels: Hurd, Version Control
People: Pavel Machek,�Larry McVoy,�John Bradford,�Christoph Hellwig,�Adam J. Richter,�David Lang,�Jeff Garzik,�Andrea Arcangeli,�H. Peter Anvin,�Joel Becker,�Olaf Hering,�Olivier Galibert,�Linus Torvalds,�Zack Brown,�Alan Cox,�Daniel Phillips,�Horst von Brand
Pavel Machek announced BitBucket, a tool intended to one day operate on BitKeeper repositories:
I've created little project for read-only (for now ;-) bitkeeper clone. It is available at www.sf.net/projects/bitbucket (no tar balls, just get it fresh from CVS).
Part of readme follows.
Install CSSC from <http://cssc.sf.net/> (it is also available as Debian package). You may need to apply cssc.diff.
Here you get following tools:
extracts files from BK repository. You can get repository by rsync -zav --delete nl.linux.org::kernel/linux-2.5 .
pull new version of repository, compute differences from the last time and apply them to directory with *your* sources.
compare two versions (specify versions from top-level s.ChangeSet)
To get a list of all changesets, do prs linux-2.5/SCCS/s.ChangeSet.
Larry McVoy replied, "BitKeeper is a trademark, please don't use the BitKeeper name when describing BitBucket. Thanks." Later he clarified that Pavel was "not allowed to say "BitBucket is a GPL-ed clone of BitKeeper" because that implies that BitBucket does what BitKeeper does and nothing could be farther from the truth." Pavel apologized, and said he'd fix the wording to be "Goal of this project is to create version managment system compatible with BitKeeper."
As a result of this "don't use the BitKeeper name" exchange, some people started ribbing Larry by referring to BitKeeper as "KitBeeper", "ButtCreeper" and other names.
Pavel Janik felt that it was a waste of time to start yet another version control project; and that in this case, Pavel's work actually supported a proprietary tool. Christoph Hellwig said Pavel was free to spend his time on BitBucket if he chose. John Bradford asked, "What is the goal of the BitBucket project, though? To develop a version control system, or to annoy Larry?" Christoph replied, "Given the annoucement probably access the kernel BK tree in real time." And Pavel clarified, "To be able to access kernel version history without touching bk. Annoying Larry is just a side effect, altrough I agree selection of project name was "interesting" ;-)."
Elsewhere, Adam J. Richter replied to Pavel's initial announcement:
Thank you for taking some initiative and improving this situation by constructive means. You are an example to us all, as is Andrea Arcangeli with his openbkweb project, which you will probably want to examine and perhaps integrate (ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/openbkweb).
bitbucket is about 350 lines of shell scripts, documentation and diffs, the most interesting file of which is FORMAT, which documents some reverse engineering efforts on bitkeeper internal file formats. bitkbucket currently uses rsync to update data from the repository. openbkweb is 500+ lines of python that implements enough of the bitkeeper network protocol to do downloads, although perhaps in inefficiently. That sounds like some functionality that you might be interested in integrating.
I think the suggestion made by Pavel Janik that it would be better to work on adding BitKeeper-like functionality to existing free software packages is a bit misdirected. BitKeeper uses SCCS format, and we have a GPL'ed SCCS clone ("cssc"), so you are adding functionality to existing free software version control code anyhow.
However, I would like to turn Pavel Janik's point in what I think might be a more constructive direction.
Aegis, BitKeeper and probably other configuration management tools that use sccs or rcs basically share a common type of lower layer. This lower layer converts a file-based revision control system such as sccs to an "uber-cvs", as someone called it in a slashdot discussion, that can:
You might want to keep in the back of your mind the possibility of someday splitting off this lower level into a separate software package that programs like your bitkeeper clone, aegis could use in common. If the interface to this lower level took cvs commands, then it could probably replace cvs, although the repository would probably be incompatible since the meaning of things like checking in multiple files together with a single comment would be different, and there would be other kinds of changes to represent beyond what cvs currently does. Using a repository format that is compatible with another system (for example bitkeeper or aegis) would make such a tool more useful, and if such a tool makes it easier for people to migrate from a prorprietary system to a free one, that's even better, so your starting with bitkeeper's format seems like an excellent choice to me.
Thanks again for starting this project. I will at least try to be a user of it.
David Lang corrected, "the openbkweb project didn't reverse engineer the BK network protocol, it used the HTTP access that is provided on bkbits.net to download the individual items and created a repository from that." He added, "bitbucket uses rsync as that is the most efficiant way to get a copy of the repository without trying to talk the bitkeeper protocol. it is FAR more efficiant and accruate then the openbkkweb interface."
Someone suggested supporting something like Subversion, instead of starting yet another version control project, and Alan Cox said that the repositories people needed to access were in BitKeeper format, and so a tool was needed to access them. Jeff Garzik replied:
"BK format"? Not really. Patches have been posted (to lkml, even) to GNU CSSC which allow it to read SCCS files BK reads and writes.
Since that already exists, a full BitKeeper clone is IMO a bit silly, because it draws users and programmers away from projects that could potentially _replace_ BitKeeper.
Pavel replied, "Read-only access to the bk repositories is the first goal. Then, I'll either add write support (unlikely) or feed it into some existing version control system to work with that. I'm still not sure what's the best."
Close by, Andrea Arcangeli replied to Jeff's point about SCCS compatibility, saying, "you never tried what you're talking about. there's no way to make any use of the SCCS tree from Rik's website with only the patched CSSC. The whole point of bitbucket is to find a way to use CSSC on that tree. And the longer Larry takes to export the whole data in an open format (CVS, subversion or whatever), the more progress it will be accomplished in getting the data out of the only service we have right now (Rik's server). Sure, CSSC is a foundamental piece to extract the data out of the single files, but CSSC alone is useless. CSSC only allows you to work on a single file, you lose the whole view of the tree and in turn it is completely unusable for doing anything useful like watching changesets, or checking out a branch or whatever else useful thing. As Pavel found _all_ the info we are interested about is in the SCCS/s.ChangeSet file and that has nothing to do with CSSC or SCCS." He suggested, "Jeff, please uninstall *notrademarkhere* from your harddisk, install the patched CSSC instead (like I just did), rsync Rik's SCCS tree on your harddisk (like I just did), and then send me via email the diff of the last Changeset that Linus applied to his tree with author, date, comments etc... If you can do that, you're completely right and at least personally I will agree 100% with you, again: iff you can." But Jeff replied, "You're missing the point: A BK exporter is useful. A BK clone is not. If Pavel is _not_ attempting to clone BK, then I retract my arguments. :)" Pavel confirmed he was working on an exporter, not a clone.
But H. Peter Anvin said, "I disagree. A BK clone would almost certainly be highly useful. The fact that it would happen to be compatible with one particular proprietary tool released by one particular company doesn't change that fact one iota; in fact, some people might find value in using the proprietary tool for whatever reason (snazzy GUI, keeping the suits happy, who knows...)" Jeff agreed such a tool would be useful, but he said it would still most likely harm other version control projects. He said:
While people would certainly use it, I can't help but think that a BK clone would damage other open source SCM efforts. I call this the "SourceForge Syndrome":
Q. I found a problem/bug/annoyance, how do I solve it?
A. Clearly, a brand new sourceforge project is called for.
My counter-question is, why not improve an _existing_ open source SCM to read and write BitKeeper files? Why do we need yet another brand new project?
AFAICS, a BK clone would just further divide resources and mindshare. I personally _want_ an open source SCM that is as good as, or better, than BitKeeper. The open source world needs that, and BitKeeper needs the competition. A BK clone may work with BitKeeper files, but I don't see it ever being as good as BK, because it will always be playing catch-up.
H. Peter said he didn't disagree with any of this, but had just been talking about whether a BitKeeper clone would be useful. But he agreed with all of Jeff's objections, and added, "Personally, I've spent quite a bit of time with OpenCM after a suggestion from Ted T'so. It's looking quite promising to me, although I haven't yet used it to maintain a large project." Close by, Joel Becker replied to Jeff's assertion that folks should devote themselves to improving existing tools, rather than starting up a new one. Joel said:
Normally, I'd agree with you Jeff. However, none of the current open source SCM systems are architected in a way that can operate like BK.
I've been using subversion for a while now. It pretty much fixes all the problems that CVS had, AS LONG AS you accept the CVS style of version control. That style doesn't work for non-central work like the kernel.
The one thing BK does that makes it worthwhile is the three-way merge. This (and the resulting DAG) make handling code from Alan, from Linus, from Andrew, and from everyone else possible. With CVS, subversion, or any other SCM I've worked with, you have to hand merge anything past the first patch. Ugh.
This requires architecture, and (AFAIK) BitBucket is the first try at it. Compatibility with the proprietary tool that does it already is a good thing.
Olaf Hering said, "Ah, finally we got to the root of the "problem"."
Elsewhere, Olivier Galibert took up the question of what really would be required for any version control system to replace BitKeeper. He responded to Adam's post, specifically Adam's list of desired features:
1. process a transaction against a group of files atomically,
2. associate a comment with such a transaction rather than with just one file,
3. represent symbolic links, file protections
4. represent file renames (and perhaps copies?)
Olivier added his own fifth item, saying:
5. Represent merges. That's what is making cvs branches unusable.
Frankly, if you want all of that you'd better design a repository format that is actually adapted to it. The RCS format is not very good, the SCCS weave is a little better but not by much (it reminds me of Hurd, looks cool but slow by design). Larry did quite a feat turning it into a distributed DAG of versions but I'm not convinced it was that smart, technically. In particular, everthing suddendly looks much nicer when you have one file per DAG node plus a cache zone for full versions.
But anyway, what made Bitkeeper suck less is the real DAG structure. Neither arch nor subversion seem to have understood that and, as a result, don't and won't provide the same level of semantics. Zero hope for Linus to use them, ever. They're needed for any decently distributed development process.
Hell, arch is still at the update-before-commit level. I'd have hoped PRCS would have cured that particular sickness in SCM design ages ago.
Atomicity, symbolic links, file renames, splits (copy) and merges (the different files suddendly ending up being the same one) are somewhat important, but not the interesting part. A good distributed DAG structure and a quality 3-point version "merge" is what you actually need to build bk-level SCMs.
Pavel asked Olivier to elaborate on the importance of the DAG (Directed Acyclic Graph) structure. Pavel was under the impression that "this "real DAG" structure is more or less equivalent to each developer having his owm CVS repository..." Olivier elaborated, "Nope. CVS uses RCS, and RCS only knows about trees, not graphs. Specifically, branch merges are not tagged as such, and as a result CVS is unable to pick up the best grandparent when doing a merge. That's the main reason of why branching under CVS is so painful (forgetting about the performance issues)." Pavel replied, "I see. But I still somehow can not understand how merging is possible. Merge possibly means work-by-hand, right? So it is not as simple as noting that 1.8 and 22.214.171.124 were merged into 1.9, no? [And what if developer did really crap job at merging that, like dropping all changes from 126.96.36.199?]" Olivier went into more detail:
Calling A and B the versions to merge and R the reference, diff3 uses this algorithm (probably the simplest possible):
Better algorithms do the alignments per-character instead of per-line, detect moved and changed functions, detect duplicate inserts, etc. None, of course, is perfect, as Larry could tell you.
Now if the development went that way:
1.7 -> 188.8.131.52 (branching, i.e. copy) v v v 184.108.40.206 1.8 v v -> 220.127.116.11 (merge) 1.9 v v v 1.10 v v -> 18.104.22.168 (merge) v v v 22.214.171.124 v v 1.11 <- (merge)
Pretty much standard, a developper created a new branch, made some changes in it, synced with mainline, synced with mailine again a little later, made some new changes and finally folded the branch back in the mainline. Let's admit the developper changes don't conflict by themselves with the mainline changes.
CVS, for all the merges, is going to pick 1.7 as the reference. The first time, for 126.96.36.199, it's going to work correctly. It will fuse the 1.7->1.8 patch with the 188.8.131.52->184.108.40.206 patch and apply the result to 1.7 to get 220.127.116.11. The two patches have no reason to overlap. 18.104.22.168->22.214.171.124 will essentially be identical to 1.7->1.8, and 1.8->126.96.36.199 will essentially be identical to 188.8.131.52->184.108.40.206.
As soon as the next merge, i.e 220.127.116.11, it breaks. CVS is going to try to fuse the 1.7->1.10 patch with the 1.7->18.104.22.168 patch. But 1.7->1.10 = 1.7->1.8+1.8->1.10 and 1.7->22.214.171.124 ~= 1.7->126.96.36.199+1.7->1.8. So they have components in common, hance they _will_ conflict.
If CVS had taken the latest common ancestor by keeping in the repository the existence of the 1.8->188.8.131.52 link, it would have taken the 1.8 version as the reference. The patches to fuse would have been 1.8->1.10 and 1.8->184.108.40.206, which have no reason to conflict.
Same for the next merge, the optimal merge point is in that case 1.10, and it ends up being a null merge, i.e. 1.11 is a copy of 220.127.116.11.
You can see the final structure is a DAG, with each node having a max of 2 ancestors. And that's what PRCS and bk are working with, fundamentally.
Pavel asked, "So, basically, if branch was killed and recreated after each merge from mainline, problem would be solved, right?" Linus Torvalds replied:
Now think three trees. Each merging back and forth between each other.
Or, in the case of something like the Linux kernel tree, where you don't have two or three trees. You've got at least 20 actively developed concurrent trees with branches at different points.
Trust me. CVS simple CANNOT do this. You need the full information.
Give it up. BitKeeper is simply superior to CVS/SVN, and will stay that way indefinitely since most people don't seem to even understand _why_ it is superior.
Zack Brown (yes, I) said, "You make it sound like no one is even interested ;-). But it's not true! A lot of people currently working on alternative version control systems would like very much to know what it would take to satisfy the needs of kernel development. Maybe, being on the inside of the process and well aware of your own needs, you don't realize how difficult it is to figure these things out from the outside. I think only very few people (perhaps only one) really understand this issue, and they aren't communicating with the horde of people who really want to help." And Larry McVoy said:
There are parts of BitKeeper which required multiple years of thought by people a lot smarter than me. You guys are under the mistaken impression that BitKeeper is my doing; it's not. There are a lot of people who work here and they have some amazing brains. To create something like BK is actually more difficult than creating a kernel.
To understand why, think of BK as a distributed, replicated, version controlled user level file system with no limits on any of the file system events which may happened in parallel. Now put the changes back together, correctly, no matter how much parallelism there has been. Pavel hasn't understood anything but a tiny fraction of the problem space yet, he just doesn't realize it. Even Linus doesn't know how BitKeeper works, we haven't told him and I can tell from his explanations that he gets part of it but not most of it. That's not a slam on Linus or Pavel or anyone else. I'm just trying to tell you guys that this stuff is a lot harder than you think. I've told people that before, like the SVN and OpenCM guys, and the leaders of both those efforts showed up later and said "yup, you're right, it is a hell of a lot harder than it looks". And they are nowhere near being able to do what BK does. Ask them if you have doubts about what I am saying.
Merging is just one of the complex areas. It gets all the attention because it is hard enough but easy enough that people like to work on it. It's actually fun to work on merging. Ditto for the graph structure, that's trivial. The other parts aren't fun and they are more difficult so they don't get talked about. But they are more important because the user has no idea how to deal with them and users do know how to deal with merge problems, lots of you understand patch rejects.
Rename handling in a distributed system is actually much harder than getting the merging done. It doesn't seem like it is, but we've rewritten how we do it 3 times and are working on a 4th all because we've been forced to learn about all the different ways that people move things around. CVS doesn't have any of the rename problems because it doesn't do them, and SVN doesn't have 1/1000th of the problems we do because it is centralized. Centralized means that there is never any confusion about where something should go, you can only create one file in one directory entry because there is only one directory entry available. In BK's case, there can be an infinite number of different files which all want to be src/foo.c.
Symbolic tags are really hard. What?!? What could be easier than adding a symbolic label on a revision? Well, in a centralized system it is trivial but in a distributed system you have to handle the fact that the same symbol can be put on multiple revs. It's the same problem as the file names, just a variation. Add to that the fact that time can march forward or backwards in a distributed system, even if all the events were marching forward, and the fun really starts. I personally have redone the tags support about 6 times and it still isn't right.
Security semantics are hard in a distributed system. Where do you put them, how do you integrate them into the system, what happens when people try and work around them? In CVS or SVN you can simply lock down the server and not worry about it, but in BK, the user has the revision history and they are root, they can do whatever they want.
Time semantics are the hardest of all. You simply can't depend on time being correct. It goes forwards, backwards, and sideways on you and if you think you can use time you don't have the slightest idea of the scope of the problem. Again, not a problem for CVS/SVN/whatever, all the deltas are made against the same clock. Not true in a distributed system.
That's a taste of what it is like. You have to get all of those right and the many other ones that I didn't tell you about or you might as well not bother. Why? Because the problems are very subtle and there isn't any hope of getting an end user to figure out a subtle problem, they don't have the time or the inclination. We've seen users throw away weeks of work just because they didn't understand the merge conflict so they start over on an updated tree. And those people will understand the rename corner cases? Not a chance.
The main point here is that if you think that BK happened quickly, by one guy, you are nuts. It started in May of 1997, that's almost 6 years ago, not the 2 years that Pavel thinks, and I had already written a complete version control system prior to that, so this was round two. Even with that knowledge, I wasn't near enough to get BK to where it is today, there is more than 40 man years of effort in BK so far. A bunch of people, working 60-90 hour weeks, for almost 6 years. Not average people, either, any one of these people would be a staff engineer or better at Sun (salaries for those people are in the $115K - $140K range).
The disbelievers think that I'm out here waving the "it's too hard" flag so you'll go away. And the arrogant people think that they are smarter than us and can do it quicker. I doubt it but by all means go for it and see what you can do. Just file away a copy of this and let me know what you think three or four years from now.
Oh, by the way, you'll need a business model, I found that out 2 or 3 years into it when my savings ran out. Oh, my, you might not be able to GPL it! Why it might even end up being just like BitKeeper with an evil corporate dude named Pavel running the show. Believe me, if that happens, I'll be here to rake him over the coals on a daily basis for being such an evil person who doesn't understand the point of free software. I can't wait.
Zack took this post and tried to boil it down to a feature-set, and posted the result. Pavel said he'd be willing to host that document in the BitBucket CVS tree; and Daniel Phillips gave a pointer to some earlier discussion. Zack incorporated this into his feature-set document, and posted an update. The discussion began to turn into a general consideration of preferred features for a version control system, with some folks complaining that the discussion was off-topic for linux-kernel. Horst von Brand and others attempted to correct various misconceptions in Zack's document, and at one point Larry complained that, since Zack had released the document under the GPL, and "Since a substantial amount of the information in there is what I said, Zack has no right to impose any license on the information. It's a bit unethical if you ask me, it's my copyright, not his. And I didn't impose any silly license on it."
The discussion continued to go further and further into feature descriptions, with some folks asking for the moon, and others saying that even BitKeeper's distributed approach was unnecessary. More folks said the discussion was off-topic, and suggested going to the OpenCM or arch mailing lists; and the thread petered out.
2. kconfig Update
8�Mar�2003�-�14�Mar�2003 (12 posts) Archive Link: "[PATCH] kconfig update"
People: Roman Zippel,�Christoph Hellwig,�Petr Baudis
Roman Zippel said, "It took a bit longer than I wanted, but here is finally another kconfig update. There are two important changes: I included Romain's gtk front end and the support for the menuconfig keyword. Unfortunately the first is lacking a bit support for the latter. Romain, you have to check that menu entries of type P_MENU can now also have a config symbol. I looked a bit at it myself but my gtk knowledge is insufficient. :) Other changes are small parser fixes and the config list in qconf has a parent entry, so it should be more obvious, how to get to a parent menu. I don't expect bigger problems, so I want to send this patch soon to Linus. The patch is at http://www.xs4all.nl/~zippel/lc/patches/kconfig-2.5.64.diff" Christoph Hellwig asked, "Any chance you could take a look at the patch that links lxdialog directly to menuconfig instead of requiring the separate binary? It has been around for a long time and seems like a very worthwhile change, imho." And Petr Baudis said, "It is me responsible for the delays and not being integrated yet, unfortunately I didn't have time for proper debugging one problem in it yet :-( (broken window resizing handler; Roman proposed some solution which I didn't manage to try yet). I hope I will finally give it a final kick really soon."
3. perfctr 2.5.0 Released
10�Mar�2003�-�16�Mar�2003 (6 posts) Archive Link: "perfctr-2.5.0 released"
Topics: Ioctls, Profiling
People: Mikael Pettersson,�J.A. Magallon,�Nils Smeds,�William Cohen
Mikael Pettersson said:
Version 2.5.0 of perfctr, the Linux/x86 performance monitoring counters driver, is now available at the usual place: http://www.csd.uu.se/~mikpe/linux/perfctr/
Quick guide to porting source code from perfctr-2.4 to perfctr-2.5:
Other noteworthy changes from perfctr-2.4:
Version 2.5.0, 2003-03-10
J.A. Magallon asked:
Perhaps this has been asked for a million times, but I'm new to perfctrs...
Is there any tool available to profile a program based on this ? I have seen perfex, but that gives total counts. I would like something like gprof... We are now optimizing some software and I would like to make my colleagues leave Windows (they use Intel's VTune) and go to Linux.
Or at least compare the same kind of things between VTune on win and 'something' in Linux that also uses the counters. They don't seem to trust gprof. And, looking at the results, I'm beginning to untrust VTune...
Nils Smeds said, "Take a look at HPCView from Rice unversity. It might do what you are looking for? http://www.cs.rice.edu/~dsystem/hpcview/index.html" And William Cohen added, "You might also look at OProfile (http://oprofile.sourceforge.net/). There is an option in the oprofpp command to dump data OProfile collects in gprof format."(In response to this issue of Kernel Traffic, Christopher Curtis sent in a link to Intel's native Linux VTune. -- Ed: [24 Mar 2003 09:15:00 -0800]
4. VMRegress 0.8a Released
11�Mar�2003�-�13�Mar�2003 (3 posts) Archive Link: "vmregress test on linux 2.5.62"
Topics: Virtual Memory
People: Dave Olien,�Mel Gorman
Dave Olien said to Mel Gorman:
I've modified your vmregress test form linux 2.4 so it works on linux 2.5.63. I fixed up some of the core routines to understand new modifications to kernel vm structures, so it all compiles and runs.
There are still some issues with the perl scripts that collect data and pipe it to gnuplot. Some of the plots of vmstat output don't work.
You can see what I've modified at the URL
There's a VMR_README file that describes the files there.
Excellent stuff, thanks. This prompted me to dust off my old test box, get 2.5.64 on it, new modutils etc etc and get working back on VM Regress. I'm releasing 0.8a for you to take a look at it, it is available from http://www.csn.ul.ie/~mel/projects/vmregress/ and directly downloadable from http://www.csn.ul.ie/~mel/projects/vmregress/vmregress-0.8a.tar.gz .
I'm not going to be working on this for another few days (hence the rushed release) so I wanted you to see what I did with your stuff so far. Rather than merging your stuff blindly, I took a few extra steps
1. The changes you made broke VM Regress from working with 2.4.anything so I sat down and made the configure script understand both 2.4 and 2.5 build systems. The way it is now, a
will compile the modules and be loadable against 2.4.20 and 2.5.64 . It generates different makefiles depending on the kernel release and I'm fairly sure I got it right. My "extensive" testing involved loading the vmregress_core, sizes.o and zones.o modules and printing the reports. They worked fine for that, but note the "a" part of 0.8a, I didn't test anything else.
2. vmr_mmzone.h is going to be my mapping between the different names of structs and fields between 2.4 and 2.5 . this (you'll love this) involves having #defines which map some VM Regress type to the actual kernel structure. So for example, on 2.4 it's
#define C_ZONE struct zone_t
and 2.5 it's
#define C_ZONE struct zone
This looks (and is) horrible but there is very few name discrepancies so it doesn't confuse things too much and I reckon it is better than having a 2.4 and 2.5 version of VM Regress for such minor differences.
3. I kept my old indenting style, but I think I incorporated all your bug fixes. I need to double check this though, I have a few other bugs on the ToDo list I want to clean up anyway
4. I blindly included the OSDL scripts into an osdl/ directory. I haven't looked at them in detail yet, I presume they are doing magic OSDL stuff for the moment until I get back onto it. If I don't though, I would still like to get the OSDL tests integrated into the tool rather than having them separate.
I think I have 99% if not 100% of your work integrated in nicely so that VM Regress still works on both sets of trees. The rudimentary ChangeLog so far is below
Dave was thrilled, and said he'd look over the new version.
5. NetFlow Data Support
13�Mar�2003�-�14�Mar�2003 (7 posts) Archive Link: "NetFlow export"
People: Florian Weimer,�Jakob Oestergaard,�David Luyer
Florian Weimer asked, "Are there any patches which implement kernel-based NetFlow data export?" Jakob Oestergaard said, "Netramet works very well - it's userspace (and very flexible indeed)." Florian Weimer replied, "According to the documentation, it can receive NetFlow data and process it, but it cannot generate such data." Jakob replied:
I never used it to receive netflow data.
I used netramet to generate flow information on a linux-based backbone router, and to periodically move the flow information to another node for batch processing/analysis.
I used the netramet tools to extract that information from the netramet "meter" (the flow measurement daemon running on the router), for storage on the remote accounting/processing machine. Some custom Perl scripts process the flow information thereafter.
You asked for netflow data export. Netramet can give you something similar to netflow (I never used netflow, but from what I hear, netramet is similar only more flexible).
But Florian said he needed the NetFlow data format, not any substitute. Jakob replied:
I suspect it's easier to patch netramet, than start a new kernel-based project.
If Netramet really supports reading netflow data, I guess it would not be a huge undertaking to implement write support too. But I'm not familiar with the Netramet source - really, you should look at it yourself or ask the developers.
David Luyer added:
NetFlowMet is the NetFlow module for netramet. But it's only a receiver, not a transmitter.
However, 'ntop' can generate NetFlow packets from a Linux system, using libpcap.
I suggest you read http://www.switch.ch/tf-tant/floma/software.html
6. Some Developers Unhappy With Linus Dropping Patches
13�Mar�2003 (2 posts) Archive Link: "Patches, effort, motivation"
Topics: FS: sysfs, PCI, Version Control
People: Russell King,�Alan Cox
Russell King lamented:
I'm getting to the point of completely giving up kernel development outside the ARM tree, which basically means leaving serial, pci, and pcmcia to other people.
I have been trying for the last 5 days to get Linus to accept two patches small patches:
a patch to register tty_devclass using postcore_initcall().
This is necessary before I can push the next set of serial changes. Without it, the serial layer will oops in sysfs when it tries to register drivers.
Both of these patches have been sent to Linus several times over the past 5 days, and each time they have been completely ignored without explaination.
I have a huge pile of patches backing up here. I'm at the point where I'm going to give up updating them to bk-curr, and testing them before sending each to Linus due to the number of patches. Going around this loop just takes too much of my time to make it worth while.
My backlog currently consists of:
8 PCI patches (1 needs to be further cut up)
10 PCMCIA patches (1 needs to be further cut up)
1 tty_io.c patch
16 serial csets
there are some dependencies between these patches, and getting the stuff applied in the right order is absolutely paramount to ensure that things don't break.
So, any suggestions on how to handle this better, and above all get Linus to start applying stuff?
Alan Cox invited Russell to submit patches into his -ac tree.
7. Modutils 2.4.23 Released
13�Mar�2003 (1 post) Archive Link: "Announce: modutils 2.4.23 is available"
People: Keith Owens,�Bill Nottingham,�Amit S. Kale,�Richard Henderson
Keith Owens announced:
modutils-2.4.23.tar.gz Source tarball, includes RPM spec file modutils-2.4.23-1.src.rpm As above, in SRPM format modutils-2.4.23-1.i386.rpm Compiled with gcc 2.96 20000731, glibc 2.2.2. modutils-2.4.23-1.ia64.rpm Compiled with gcc 2.96-ia64-20000731, glibc-2.2.3. patch-modutils-2.4.23.gz Patch from modutils 2.4.22 to 2.4.23.
Modutils 2.4.24 will be out in the next few days. The only change planned for 2.4.24 is to complain about modules using the default behaviour of "export all symbols" on architectures that use function descriptors. Such architectures must explicitly export symbols, otherwise gcc does not generate function descriptors and inter-module references break spectacularly.
8. i2c Updated To The New Driver Model
13�Mar�2003�-�15�Mar�2003 (19 posts) Archive Link: "[BK PATCH] i2c driver changes for 2.5.64"
Topics: I2C, Version Control
People: Greg KH
Greg KH announced:
Here's a set of i2c driver changes that start the conversion of the i2c core and drivers over to the kernel driver model. Eventually this will allow all of the sysctl and proc mess to be removed for this subsystem.
These patches add i2c driver bus and i2c adapter driver support to the driver core. They also export a needed symbol from the driver core (which Pat Mochel agreed with doing.) The patches also add three i2c controllers that have been in the i2c cvs tree for a long time, and are on a lot of people's machines.
The i2c core needs Christoph's previous patches to work properly, and with that patch, and these patches, it all works properly on my machines (tested on 4 different types of i2c controllers.)
Oh, and the i2c development team has given the ok for me to send these patches to you, and for me to do this work.
Please pull from: bk://kernel.bkbits.net/gregkh/linux/i2c-2.5
Things left to do after this:
9. Linux RAID FAQ Version 0.0.12 Released
13�Mar�2003�-�14�Mar�2003 (3 posts) Archive Link: "Posting of the Linux RAID FAQ"
Topics: Disk Arrays: RAID, FS: sysfs
People: Nandu Nair,�John Jasen
Gregory Leblanc posted the latest version 0.0.12 of his Linux RAID FAQ. Nandu Nair replied, "Thanks for taking the time to write this FAQ. It is very informative. One question I would like answered and possibly included in the FAQ is as follows. When an unclean shutdown ( or similar event, but not bad disks ) causes the raid array to be resynced on a subsequent reboot and if the resync process is crunching more resources than desired - possibly due to the presence of multiple raid volumes that need resyncing - , is there a way for a user process to request the kernel to stop the resync process for the time being? In other words can the user choose to wait and do the resync at what he thinks is a more appropriate time - in terms of resouce availability - ( at night, for instance ) if he is willing to take up the risk involved ? If yes, how can the kernel raid-recovery processes be stopped/controlled ?" John Jasen replied:
You have /proc/sys/dev/raid/speed_limit_max and _min, where you can specifiy upper and lower bounds for how fast the raid resyncs.
I imagine you could use cron or at to whack together something to arrange higher speed resyncs during offhours.
10. Status Of Promise Driver Development
14�Mar�2003 (2 posts) Archive Link: "Promise SATA, status"
Topics: Serial ATA
People: Andre Hedrick,�Stephan von Krawczynski
Andre Hedrick reported, "For the folks out there with this hardward and no support: Promise and I are back in communication and may have reached an agreement for developing the needed driver." Stephan von Krawczynski said happily, "Great news! Thank you for your efforts."
Previously, as covered in Issue�#206, Section�#6� (12�Feb�2003:�Promise Spits On Free Software) , Promise had been unwilling to provide the needed information.
11. BitMover Considers Lawsuit Over BitBucket Development
14�Mar�2003�-�17�Mar�2003 (66 posts) Archive Link: "Never ever use word BitKeeper if Larry does not like you"
Topics: Version Control
People: Pavel Machek,�Larry McVoy,�Alan Cox
Pavel Machek said:
Never ever use word KitBeeper, Larry thinks he owns the world.
The page originally said "Goal of this project is to create version managment system compatible with <prohibited word>".
PS: I know forwarding personal message to the mailing list is not polite, but this is more of legal threat than personal message...
PPS: About Toyota: I may not be able to call my company Toyota (if you have registered trademark in czech republic, which I doubt), but I'm sure able to say that my car contains same engine as Toyota 1234cdt. And as you named directory in the repository BitKeeper, there's no chance not to mention it.
He quoted a private email from Larry McVoy:
Several folks criticized Pavel for publishing private email, but Alan Cox said:
When you are dealing with someone apparently being a corporate bully you have no choice quite often but to publicize stuff. Whether Larry is actually unreasonable is kind of hard to tell without all the context.
If Larry feels bitbucket is a lot like bitkeeper then I see his point. If he's moaning about things like "foobar is a tool for reading BitKeeper repositories" then I guess he forgot to take his pills this morning 8)
Unfortunately all the current bad feeling and suspicion keeps magnifying probably minor misunderstandings into large ones
Various folks were critical of Pavel for publishing private email, and various folks were critical of Larry for sending the mail in the first place.
12. Mailing List Stats
15�Mar�2003 (1 post) Subject: "statistics for this mailinglist"
Topics: Klibc, Microsoft, Version Control
People: Folkert van Heusden,�Jens Axboe,�Larry McVoy,�Martin J. Bligh,�David S. Miller,�Linus Torvalds,�Alan Cox,�Russell King,�Pavel Machek,�Andrew Morton,�Greg KH,�William Lee Irwin III
Folkert van Heusden automated his mboxstats script to post the mailing list statistics at regular intervals:
Overall statistics ------------------ Statistics created on: Sat Mar 15 04:02:23 2003 First message was written at: 2003/02/24 11:22:10 Last message was written at: 2003/02/24 23:55:45 Total number of messages: 3040 Total size: 17455KB Total number of writers: 657 Number of people who wrote >1 message: 340 Total number of lines: 297404 Average lines per message: 97 Total header length (lines): 145698 Average header length (lines): 47 The header is on average 48.99% of the message (lines). The header is 43.92% bytes in size of the total. Average number of bits information per byte: 0.7502 Total number of unique user-agents: 0 Total number of unique organisations: 79 Total number of unique top-level domains: 66 Importance ---------- Low : 0.00% Normal: 1.61% High : 0.00% (the rest is unspecified) Top writers | # msgs|av size| total|time| e-mail address ---+-------+-------+------+----+-------------------------------- 1] 96| 4922| 461KB|1332| "Martin J. Bligh" <email@example.com> 2] 81| 4656| 368KB|1142| Andrew Morton <firstname.lastname@example.org> 3] 78| 3626| 276KB|0000| Alan Cox <email@example.com> 4] 60| 6447| 377KB|1420| Greg KH <firstname.lastname@example.org> 5] 55| 4274| 229KB|1312| Linus Torvalds <email@example.com> 6] 48| 6912| 324KB|1535| William Lee Irwin III <firstname.lastname@example.org> 7] 47| 4500| 206KB|1428| Larry McVoy <email@example.com> 8] 39| 5444| 207KB|1516| Pavel Machek <firstname.lastname@example.org> 9] 38| 3932| 145KB|1515| Jens Axboe <email@example.com> 10] 37| 6609| 238KB|1801| Russell King <firstname.lastname@example.org> Top subjects | # msgs|av size| total|time| subject ---+-------+-------+------+----+-------------------------------- 1] 111| 4362| 472KB|1405| Minutes from Feb 21 LSE Call 2] 71| 5372| 372KB|1311| [ANNOUNCE] BK->CVS (real time mirror) 3] 64| 4807| 300KB|1229| BitBucket: GPL-ed KitBeeper clone 4] 43| 3677| 154KB|1252| 2.5.63 accesses below %esp (was: Re: ntfs OOPS (2.5.63)) 5] 38| 4345| 161KB|0833| About /etc/mtab and /proc/mounts 6] 38| 4307| 159KB|1012| Never ever use word BitKeeper if Larry does not like you 7] 35| 3684| 125KB|1207| bio too big device 8] 32| 4385| 137KB|0742| [BK PATCH] klibc for 2.5.64 - try 2 9] 31| 4517| 136KB|1515| [PATCH] s390 (7/13): gcc 3.3 adaptions. 10] 29| 4265| 120KB|1451| [BUG] 2.5.63: ESR killed my box! Top receivers | # msgs|av size| total|time| e-mail address ---+-------+-------+------+----+-------------------------------- 1] 859| 8109|6802KB|1315| email@example.com 2] 211| 9306|1917KB|1336| firstname.lastname@example.org 3] 128| 6791| 848KB|1114| Andrew Morton <email@example.com> 4] 76| 4281| 317KB|1308| "Martin J. Bligh" <firstname.lastname@example.org> 5] 72| 5868| 412KB|1136| Alan Cox <email@example.com> 6] 44| 3815| 163KB|1655| firstname.lastname@example.org 7] 43| 4396| 184KB|1228| Larry McVoy <email@example.com> 8] 35| 4177| 142KB|0941| "'Jens Axboe'" <firstname.lastname@example.org> 9] 34| 6007| 199KB|0654| email@example.com 10] 34| 4542| 150KB|1350| Greg KH <firstname.lastname@example.org> Top CC'ers | # msgs|av size| total|time| e-mail address ---+-------+-------+------+----+-------------------------------- 1] 1316| 5138|6603KB|1202| email@example.com 2] 216| 7992|1685KB|1333| firstname.lastname@example.org 3] 106| 5193| 537KB|1237| Andrew Morton <email@example.com> 4] 101| 7061| 696KB|1303| Alan Cox <firstname.lastname@example.org> 5] 62| 4950| 299KB|1021| "Martin J. Bligh" <email@example.com> 6] 42| 6218| 255KB|1347| Larry McVoy <firstname.lastname@example.org> 7] 40| 5520| 215KB|1051| email@example.com 8] 34| 4574| 151KB|1125| "David S. Miller" <firstname.lastname@example.org> 9] 33| 4260| 137KB|1136| Greg KH <email@example.com> 10] 31| 3782| 114KB|1322| Alan Cox <firstname.lastname@example.org> Top of top-level-domain ---------------------------------------------------------- 1] com 1145 2] org 474 3] de 291 4] net 187 5] uk 160 6] cz 86 7] ru 67 8] edu 61 9] au 60 10] nl 48 Timezones ---------------------------------------------------------- 1] 777 +0100 2] 731 -0800 3] 357 -0500 4] 196 +0000 5] 115 -0600 6] 71 +1100 7] 69 +0300 8] 64 +0200 9] 50 -0700 10] 43 +0900 Top organisations ---------------------------------------------------------- 1] 198 2] 48 The Domain of Holomorphy 3] 29 OSDL 4] 23 none 5] 19 Open Source Devlopment Lab 6] 17 Transmeta Corporation, Santa Clara CA 7] 16 Working Overloaded Linux Kernel 8] 15 HOME 9] 15 IBM Deutschland GmbH 10] 14 daimi.au.dk Top user-agents ---------------------------------------------------------- 1] 56 Mutt/1.4i 2] 40 Mutt/1.5.3i 3] 26 KMail/1.5 4] 24 Ximian Evolution 1.2.2 5] 20 Mutt/1.3.28i 6] 17 Mutt/18.104.22.168i 7] 12 KMail/1.4.3 8] 9 Mutt/1.2.5i 9] 8 Microsoft Outlook Express 6.00.2800.1106 10] 8 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20021003 Messages per day ---------------------------------------------------------- Sunday 213 **************** Monday 387 ***************************** Tuesday 507 ************************************* Wednesday 564 ***************************************** Thursday 442 ********************************* Friday 386 ***************************** Saturday 128 ********** Messages per Month ---------------------------------------------------------- Jan 0 Feb 1060 ********************************* Mar 1566 ************************************************ Apr 0 May 0 Jun 0 Jul 0 Aug 0 Sep 0 Oct 0 Nov 0 Dec 1 * Messages per day-of-the-month ---------------------------------------------------------- 1 2 * 2 0 3 0 4 0 5 0 6 1 * 7 2 * 8 127 *********** 9 211 ****************** 10 217 ******************* 11 221 ******************* 12 314 *************************** 13 230 ******************** 14 241 ********************* 15 0 16 0 17 0 18 0 19 0 20 0 21 0 22 0 23 2 * 24 170 *************** 25 285 ************************* 26 250 ********************** 27 211 ****************** 28 142 ************* 29 0 30 0 31 1 * Messages per hour ---------------------------------------------------------- 1 33 ********* 2 47 ************* 3 24 ******* 4 22 ****** 5 12 **** 6 22 ****** 7 42 ************ 8 87 ************************ 9 137 ************************************* 10 173 ********************************************** 11 185 ************************************************** 12 163 ******************************************** 13 182 ************************************************* 14 183 ************************************************* 15 184 ************************************************* 16 181 ************************************************ 17 155 ****************************************** 18 106 ***************************** 19 146 *************************************** 20 119 ******************************** 21 123 ********************************* 22 103 **************************** 23 104 ****************************
13. XFS, ReiserFS, And ext3 Comparisons
16�Mar�2003�-�17�Mar�2003 (5 posts) Archive Link: "FileSystem XFS vs RiserFS vs ext3"
Topics: Disk Arrays: LVM, Disk Arrays: RAID, FS: NFS, FS: ReiserFS, FS: XFS, FS: ext3
People: Alex Lau,�Bernd Eckenfels,�Hans-Peter Jansen,�Bryan Whitehead
Alex Lau asked, "I get basic understanding of the functions and different between XFS, RiserFS and ext3. But in high volumn read write enviornment (database, NFS email server etc), which will provide better preformance?" Bernd Eckenfels replied:
NFS is a bit tricky. Reiser used to be broken on it, and at least from large XFS NFS Servers I know that they tend to be unstable, still.
For the Database Servers, I am not sure how well they operate with journaling filesystems. I think Linux Journal had an article on performance on that.
Reiser might be your bet, depending on the usage pattern of the filename space, with Ext3 catching up. Personally I love the XFS features for resizing in connection with LVMs, but i guess you can have that with Ext3 and Reiser, too.
Hans-Peter Jansen replied, "The last big problems with NFS on big ReiserFS shares where fixed with 2.4.19 (IIRC). Since then, at least I haven't experienced any logic induced failures in this area with my heavily used diskless setups on up to 80 GB ReiserFS, shared with NFS. I enjoy this configuration a lot." And Bryan Whitehead added, "I've never had problems with XFS+NFS on any of my servers..."
Alex posted the results of his own tests:
Thanks for all the input. Here is some info after I get from my setting. Hardware config: Tyan 2466 Duel MP2200+ 512MB, SX6000 4 ide 120GB 7200RPM RAID5
------------------------------------------------------------------------- StPeter:/mnt/part1# sfdisk -l /dev/sda Disk /dev/sda: 351905 cylinders, 64 heads, 32 sectors/track Warning: extended partition does not start at a cylinder boundary. DOS and Linux will interpret the contents differently. Warning: The first partition looks like it was made for C/H/S=*/255/63 (instead of 351905/64/32). For this listing I'll assume that geometry. Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 0+ 44860 44861- 360345951 5 Extended /dev/sda2 0 - 0 0 0 Empty /dev/sda3 0 - 0 0 0 Empty /dev/sda4 0 - 0 0 0 Empty /dev/sda5 0+ 3646 3647- 29294464+ 83 Linux /dev/sda6 3647+ 7293 3647- 29294496 83 Linux /dev/sda7 7294+ 10940 3647- 29294496 83 Linux /dev/sda8 10941+ 14587 3647- 29294496 83 Linux /dev/sda9 14588+ 18234 3647- 29294496 83 Linux /dev/sda10 18235+ 21881 3647- 29294496 83 Linux /dev/sda11 21882+ 25528 3647- 29294496 83 Linux /dev/sda12 25529+ 29175 3647- 29294496 83 Linux /dev/sda13 29176+ 32822 3647- 29294496 83 Linux /dev/sda14 32823+ 36469 3647- 29294496 83 Linux /dev/sda15 36470+ 44860 8391- 67400676 83 Linux ------------------------------------------------------------------ StPeter:/mnt/part1# hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 64 MB in 1.60 seconds = 40.00 MB/sec StPeter:/mnt/part1# hdparm -T /dev/sda /dev/sda: Timing buffer-cache reads: 128 MB in 0.47 seconds =272.34 MB/sec ------------------------------------------------------------------ StPeter:/mnt/part1# mount /dev/sda5 on /mnt/part1 type xfs (rw,noexec,nosuid,nodev) /dev/sda6 on /mnt/part2 type reiserfs (rw,noexec,nosuid,nodev) /dev/sda7 on /mnt/part3 type ext3 (rw,noexec,nosuid,nodev) StPeter:/mnt/part1# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda5 28G 4.8M 27G 1% /mnt/part1 /dev/sda6 28G 33M 27G 1% /mnt/part2 /dev/sda7 27G 33M 26G 1% /mnt/part3 StPeter:/mnt/part1# time cp -rf /usr/src/kernel-source-2.4.20 ./ real 1m3.501s user 0m0.140s sys 0m2.680s StPeter:/mnt/part2# time cp -rf /usr/src/kernel-source-2.4.20 ./ real 0m3.696s *************** so fast... user 0m0.110s sys 0m3.570s StPeter:/mnt/part3# time cp -rf /usr/src/kernel-source-2.4.20 ./ real 1m29.697s user 0m0.090s sys 0m2.490s *ext3 used the most space StPeter:/mnt/part3# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda5 28G 180M 27G 1% /mnt/part1 /dev/sda6 28G 191M 27G 1% /mnt/part2 /dev/sda7 27G 211M 25G 1% /mnt/part3 -------------------------------------------------------- StPeter:/mnt/part1# time rm -rf kernel-source-2.4.20/ real 0m23.351s user 0m0.050s sys 0m1.250s StPeter:/mnt/part2# time rm -rf kernel-source-2.4.20/ real 0m1.297s user 0m0.010s sys 0m0.890s StPeter:/mnt/part3# time rm -rf kernel-source-2.4.20/ real 0m1.062s user 0m0.000s sys 0m0.690s -------------------------------------------------------
14. Support For NUMAQ Machines With More Than 8 IOAPICs
16�Mar�2003�-�17�Mar�2003 (7 posts) Archive Link: "[PATCH][ANNOUNCE] 32way/8quad NUMAQ booting with 16 IOAPICs, 223 IRQs"
People: Zwane Mwaikambo
Zwane Mwaikambo said:
I've managed to put together a patch which allows a NUMAQ box with more than 8 IOAPICs to boot, the current workaround is to force it to 8 or lower. This was achieved by setting up percpu idts thus allowing us to implement per node vector spaces. The previous hurdle was running out of vectors to assign, the current one is running out of irqs, something which i'll further be looking at.
Later on i'm planning to move to per node IDTs so that i can fix f00f workaround and also save space on nodeless systems. To do this will probably require early_cpu_to_node workalikes (or perhaps just use logical_smp_processor_id and apic_to_node).
Thanks to the folks at OSDL (and Mark for giving up his 4quad for so long) for providing me access to the 8quad.
15. KDB Kernel Debugger 4.0 For 2.4.19, 2.4.20, i386 And ia64
16�Mar�2003 (1 post) Archive Link: "Announce: kdb v4.0 is available for kernels 2.4.19, 2.4.20, i386 and ia64"
People: Keith Owens
Keith Owens announced:
<=== You know what they say about .0 releases. Caveat emptor. ===>
The biggest change is this release in the way that kdb captures data from each cpu. Prior to v4.0, the controlling cpu (first to get into kdb) would try to pull the other cpus into kdb by sending them an IPI. Doing a backtrace on the active tasks required that kdb switch its context into each cpu in turn.
This works in most cases, but when the system is really wedged, it is difficult to get data from the other cpus. Of course this is precisely the case when we want data from the other cpus. This problem is particularly noticeable on ia64, when machine checks (MCA) or INIT interrupts can disable normal IPI or dive down into SAL, taking control away from the OS. Also on IA64 the non-maskable interrupt is actually masked (http://external-lists.valinux.com/archives//linux-ia64/2001-May/subject.html, look for 'Replacements for local_irq_xxx').
In kdb v4.0, each cpu pushes its state via a global array. This allows any cpu to do a backtrace on any other cpu, from a known starting point. It even handles the cases when IA64 requires that cpus rendezvous and spin in SAL. The push model also makes it easier to detect when a cpu is dead, an event which would often hang the old kdb pull model.
On ia64, kdb v4.0 gives decent backtraces from MCA callback, MCA rendezvous and the INIT monarch event. The next step in kdb v4.1 is to detect hung spinloops and break out of them, and to support INIT slave events. The detection and debugging of hung spinloops is waiting on acceptance of my new spinlock code for IA64 (http://external-lists.valinux.com/archives//linux-ia64/2003-March/004976.html).
16. LKST (Linux Kernel State Tracker) 1.4 Released
17�Mar�2003 (1 post) Archive Link: "Release of LKST 1.4"
People: Yumiko Sugita
Yumiko Sugita announced:
I'd like to announce publication of Linux Kernel State Tracer (LKST) 1.4, which is a tracer for Linux kernel.
In release 1.4, we especially improved about event-buffer. Please refer to changesto1.4.txt in documents for details.
LKST binaries, source code and documents are available in the following site,
We prepared a mailing list written below in order to let users know update of LKST.
To subscribe, please refer following URL,
And if you have any comments, please send to the above list, or to another mailing list written below.
17. SquashFS 1.2 Compressed Filesystem Released
17�Mar�2003 (1 post) Archive Link: "[ANNOUNCE]:Squashfs1.2 released (compressed filesystem)"
Topics: Compression, FS: SquashFS, Small Systems
People: Phillip Lougher
Phillip Lougher announced:
Squashfs1.2 has been released.
Squashfs is a highly compressed read-only filesystem for Linux 2.4. It uses zlib to compress files, inodes, and directories. All blocks are packed to minimise the data overhead, and block sizes of between 4K and 32K are supported. It is intended to be used as a filesystem for archival use and in embedded systems where low overhead is needed.
1.2 has added append capability to Mksquashfs. This means that new files and directories can be added to pre-existing filesystems without needing to rewrite the original data. For large filesystems this can be of major benefit. Because mksquashfs peforms duplicate checking against the original filesystem, it means that updated directories can be appended over time, and only the changed files will take extra space. The unchanged files will be detected as duplicates, and will share data. This exhibits a basic versioning capability.
Squashfs1.2 is available from http://squashfs.sourceforge.net.
18. BitKeeper Gateway To CVS
17�Mar�2003 (7 posts) Subject: "BK->CVS is live"
Topics: Modems, Version Control
People: Larry McVoy,�Jan-Benedict Glaw,�Sebastian D.B. Krause,�Tomas Szepe,�David Woodhouse,�Jan-Benedict
Larry McVoy announced:
I think those repositories are stable enough you can start to count on them. Sam and others have looked over their changes and think that the CVS tree has accurate data.
The CVS repository is at
and it has two top level directories, linux-2.4 and linux-2.5.
Jan-Benedict Glaw suggested, "Allowing rsync on the repository could help people on slow links (modem) esp. as CVS isn't exactly known to be fast and evvective. I'd love to have it rsyncable (as we have it for mips-linux:-)" And David Woodhouse suggested that CVSup might also be good in conjunction with this. Sebastian D.B. Krause said, "That would be nice and easy to set up because Larry could just use cvsup with the current CVS-repository. And it would be much faster than cvs." And Tomas Szepe put in, "rsync would be totally splendid. Say, Larry, is that feasible?" But there was no reply in that thread.
19. New Configuration Organization For IPX Under 2.5
17�Mar�2003 (4 posts) Archive Link: "Where did IPX on 2.5 go?"
People: Maciej Soltysiak,�Brian Davids,�Randy Dunlap
Maciej Soltysiak reported, "i tried to find IPX support in 2.5 via make menuconfig, it is not there. (or where it used to be) There is nothing about IPX also in .config but net/ipx files are there. It is 2.5.64-bk12. Was it removed or i am missing something here." Randy Dunlap replied that it should show up under the LLP option, but that LLP had to be enabled for IPX to appear. Brian Davids also said to Maciej, "It's under Networking Support, Networking Options, ANSI/IEEE 802.0 - aka LLC (IPX, Appletalk, Token Ring). Yeah it's a little more buried than before, but it's still there." Maciej found it, and was happy.
20. kernel.org Planned Downtime
17�Mar�2003 (2 posts) Archive Link: "kernel.org: Extended downtime announcement"
Topics: Disk Arrays: RAID
People: H. Peter Anvin
H. Peter Anvin said:
In order to make the best use of available space, we will unfortunately have to reconfigure the RAID arrays on the main kernel.org server. This will result in extended downtime for all kernel.org services, starting at 16:00 PST/24:00 UTC this evening, March 17, 2003. For ftp/www/rsync/filehub.kernel.org, this downtime should be in the ballpark of 2-3 hours; with luck less; for mirrors.kernel.org, this downtime is expected to be 3-4 days. Services will be phased in as they become available, so if you get a "connection refused" from one service other services might still be operational.
Hopefully this will be reasonably painless.
He replied to himself a few hours later:
OK, so I'm an idiot.
The correct downtime estimate is about 6-7 hours. This is a matter of simple arithmetric, but somehow I never bothered to actually calculate it out.
Just to be on the safe side I'm going to estimate kernel.org being back up around 00:00 PST/08:00 UTC.
21. Linux 2.5.65 Released
17�Mar�2003�-�18�Mar�2003 (15 posts) Archive Link: "Linux 2.5.65"
Topics: Version Control
People: Linus Torvalds,�John Cherry
Linus Torvalds announced kernel 2.5.65:
I've delayed this too long, but Ingo found why the scheduler sometimes did bad things, and this should all be good.
A lot of fairly small changes all over the map, see the full changelog for details.
John Cherry posted some compilation statistics:
Compile statistics: 2.5.65
2.5.64 2.5.65 -------------------- ----------------- bzImage (defconfig) 14 warnings 14 warnings 0 errors 0 errors bzImage (allmodconfig) 30 warnings 30 warnings 9 errors 12 errors modules (allmodconfig) 2356 warnings 2421 warnings 99 errors 100 errors
Compile statistics have been for kernel releases from 2.5.46 to 2.5.65 at: www.osdl.org/archive/cherry/stability (will be updated by 6PM PST).
Other stability-related links:
OSDL Stability page:
Nightly linux-2.5 bk build:
2.5 porting items:
2.5 porting items history:
22. Kernel 2.5.65-mm1 Released
18�Mar�2003�-�19�Mar�2003 (14 posts) Archive Link: "2.5.65-mm1"
People: Andrew Morton
Andrew Morton announced:
kernel.org is being slow. Should later appear at
It turns out that calling disk request_fns from timer/tasklet context is not permitted because a few old drivers like to sleep in that function.
keventd cannot be used for this because it can deadlock. So another kernel thread per CPU has been reluctantly added.
23. VM Subsystem Documentation
18�Mar�2003 (1 post) Archive Link: "VM documentation"
Topics: Virtual Memory
People: Mel Gorman
Mel Gorman announced:
Yet another release in the usual places. The main reasons for the release is a correction on the subject of vmalloc more than anything else and the rearrangement of chapters to present the material in more logical order. I am hoping there will only be one, or at most two more releases after this before it's done and dusted (famous last words).
Understanding the Linux Virtual Memory Manager
Few important reasons for this release but still, it brings me closer to just finalising it and releasing it fully.
24. New relayFS High-Speed Data Relay Filesystem
18�Mar�2003 (1 post) Archive Link: "[ANNOUNCE] relayfs, a high-speed data relay filesystem"
People: Tom Zanussi
Tom Zanussi announced:
A kernel patch implementing relayfs, a Linux filesystem designed to simplify the buffering and efficient transfer of large amounts of data from kernel to user space for those facilities that need to do such things, is now available as:
The relayfs home page, which will host the latest versions of the above and related patches and info is:
A preliminary description of the characteristics and API for relayfs was previously posted to this list. Here's a link to that post:
I've included at the end of this mail the relayfs doc file (Documentation/filesystems/relafys.txt), which describes the API in detail, as currently implemented.
Also, the Linux Trace Toolkit has been reworked to make use of relayfs; a preliminary kernel patch and updated user tools are available at the LTT website, and can be used as a concrete working example of using the API:
I've tested the relayfs code (via the updated LTT code) on my UP PII machine and so far haven't seen any problems with it, but I still need to do SMP testing and otherwise pound more heavily on it - I plan on generating some benchmark data, which in addition to putting relayfs through its paces, should give some idea of how much overhead might be introduced by the API/implementation itself, as compared with the benchmark data previously posted for LTT without it:
I wouldn't expect to see much difference, as the underlying logging code is essentially the same.
25. cvsps Project To Support BitKeeper Metadata In CVS
18�Mar�2003�-�19�Mar�2003 (9 posts) Subject: "[ANNOUNCE] cvsps support for parsing BK->CVS kernel tree logs"
Topics: Compression, MAINTAINERS File, Networking, PCI, Version Control
People: David Mansfield,�Andrea Arcangeli
David Mansfield announced:
I've just added (updated) lightly tested support for the BK->CVS kernel trees to cvsps (www.cobite.com/cvsps) in version 2.0b4. The purpose of this effort is to recreate the BK ChangeSet meta-data that is embedded in the 'cvs log' data in these trees. BTW, cvsps is GPL software :-p. I'd like to thank Larry and Andrea for helping me track down some issues with this effort. This is still a BETA version, though, and I haven't given this enough testing, so be nice. It works for me.
This version is tested and works against this morning's linux-2.4 and linux-2.5 trees, and contains a few workarounds for specific issues in those trees. See below for information on these problems.
The output of cvsps looks like:
------------------------------ PatchSet 999 Date: 2002/07/11 19:50:46 Author: alan Branch: HEAD Tag: (none) Log: [PATCH] Fix several pdc202xx problems Misnaming of 20270 as 20268R Failure of LBA48 on 20262 Incorrect speed detection because the old driver used host not drive side cable detect PDC202xx handling for quirks in udma reporting off some drives LBA48 for PIO mode BKrev: 3d2dd386wJMnehoOAhv3wL991IfXVQ Members: ChangeSet:1.999->1.1000 MAINTAINERS:1.74->1.75 drivers/ide/ide-features.c:1.4->1.5 drivers/ide/ide-pci.c:1.18->1.19 drivers/ide/pdc202xx.c:1.11->1.12 include/linux/pci_ids.h:1.44->1.45 -----------------
You can also get a diff of this PatchSet using the '-g' option to cvsps.
There are currently 2,798 PatchSets in the linux-2.4 tree, and 8,382 in the linux-2.5 tree.
Quick start instructions
Download, build and install cvsps from http://www.cobite.com/cvsps
Get the 2.0b4 (or latest) version.
Create a working directory with the tree of your choice:
cvs -d:pserver:email@example.com:/home/cvs co linux-2.4/Makefile
[ IMPORTANT: cvsps doesn't currently support an option for setting the compression level so PLEASE, edit your .cvsrc and put 'cvs -z4' to enable compression ]
cvsps [-x] --bkcvs
This basically runs a 'cvs rlog' against the tree, parses, and caches all of the revision history as PatchSets. It also outputs all of the PatchSet summaries to stdout, so you may want to '>/dev/null' the first time.
Subsequent 'cvsps' commands do not need the '--bkcvs' unless you are updating (-u, not completely tested) or rebuilding (-x, always works) the cache file.
Now you can use cvsps to browse the patchsets at your leisure, without loading the cvs server (except to generate diffs). See cvsps -h for the many ways you can slice and dice the information.
I welcome any feedback.
I have currently encountered two problems with the log format.
1) someone has committed sections of 'cvs log' text into the log. This causes quite a headache for my parser, because false end-of-log-message markers are present in the log. Fortunately, Larry has put a '(Logical change x.yyyy)' marker at the end of each log message, see alse 2)
2) Not all log messages are terminated by a '(Logical change x.yyy)' marker. A single revision of one file is missing this marker, Larry is looking into why this may have happened.
Both of these problems are being worked around by my 'Adaptive Crap Filter (notTM)' code. Don't look at it. It'll kill you.
Andrea Arcangeli asked:
what about the deleted files? should we teach cvsps how to diff the old revision fetched with cvs up -p against /dev/null to make a completely coherent patch?
this is with 2.4
Directing PatchSet 2742 to file ../patches-2.4//2742.patch
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
cvs [update aborted]: no such directory `drivers/usb/hcd'
David Mansfield said, "This has been fixed already in 2.0b5 (on www.cobite.com/cvsps website). It was all working fine if you had a checked out tree. But doesn't work to use 'update -p' if the sub-directory isn't checked out (update -p is what is used in your version). The new code uses 'cvs co -p -r x.y repository/file' which works fine. The patches that are generated by '-g' now need 'patch -p1' to be applied inside the working directory."
Andrea, in his same post, had gone on to say, "I also wonder if cvsps is so accurate also w/o the --bkcvs option (i.e. w/o atomic commits from bk). are the dates guaranteed to be the same for all files w/ a normal cvs tree?" To which David had replied, "No. On a normal CVS repository, and without the --bkcvs, a heuristic is used to recreate a 'commit'. The commit must have the same author, the same message and the time of commit must be within a fuzz-factor number of seconds (default 300, settable via the -z option). The reason the fuzz-factor is needed is that a commit over a slow link, or a very large commit in general can take a lot of time, and the log time will vary for each file committed. It actually works quite well."
Also in his same post, Andrea had asked, "what about the -f option? why can't I use it at the same time with -r? Can I use multiple -f at the same time? That is getting very cool and so useful." And David had said, "Oops. You found a bug. See the attached patch against 2.0b5 (should work against any recent version). By design though, all of the 'filter' options can be combined. Also by design, you cannot specify multiple instances of any option (except -d and -r, where the first and second instance have special meaning - start vs end)." Anrea said he'd try out the patch.
Also in his original response, Andrea asked, "Would also be nice to export the API of the cvsps internals to python (i.e. to allow to efficiently parse the cvsps metadata files in .cvsps from scripts that will give the flexibility of parsing the data as you want or to quickly write a gui fronthand). This is low prio though, having -f working together with -r and all the other options is much more interesting at this point IMHO. Being able to specify a directory as a file would also be very useful." And David said, "The file is actualy a substring match. If the -f argument matches as a substring the filename it will count as a match. So you can specify directory names just as is." Andrea replied:
so it doesn't sound a regex. Being able to specify more than 1 -f would be very useful. Either that or regex would do it fine too with '^net/core', so as far as I can write stuff like -f '^net/core|^net/ipv4' I'm fine.
I also think using match by default in the regex is cleaner. So I can write -f 'net/core|net/ipv4' w/o bothering about the ^ because it become implicit. And I can still use '.*net/core.*' if I want a substring regex. I think substring search will be not common.
But really, any kind of way you implement the 'multiple file' thing is fine as far as I can specify more than 1 file ;).
David posted a patch to allow the -f option to take a regular expression.
At one point, Andrea also remarked, "BTW, I think cvsps should be shipped together with cvs and integrated over time in the unstable branch, this is a major feature for any cvs user. Storing metadata locally is the way to go, over time the whole tree should be cached local (as an option). This is actually the major cvs improvement I seen since years. And I'm glad I can contribute to it unlike many kernel developers out there."
26. Blank-Screen On Demand
18�Mar�2003 (1 post) Archive Link: "Blank-screen-on-demand"
People: Pavel Machek
Pavel Machek announced, "This allows user to blank screen on demand. On some machines (philips velo) backlight takes more energy than rest of system, and this becomes pretty critical."
27. Linux On Small Systems
18�Mar�2003�-�19�Mar�2003 (7 posts) Archive Link: "Linux on 16-bit processors"
People: Joel Jaeggli,�J.A. Magallon,�Xavier Bestel,�Alan Cox
Mick Weiss wanted to run Linux on cheap, 16-bit machines, and wondered if any particular Linux version would be best for that. Joel Jaeggli suggested:
the economics aren't really there as far as I can tell given the cost of embeded 386 and 486 class cpu's to say nothing of tiny powerpc and arm cpu's.
J.A. Magallon also suggested:
It doesn't need an mmu, boots on a Palm. ;) Look in 'uClinux Ports'
Or http://www.linux.org/projects/ports.html, look for m68k ports, don't know if any of them work on cpus below 68020.
Xavier Bestel replied, "It works on an Amiga 500 (plain 68000)."
Alan Cox also said to Mick, "The kernel side is fairly easy if you have a couple of megs of ram. The ucLinux tree supports mmuless systems and a fair variety of processors. User space is more of an issue. The standard Linux userspace is designed for systems with disks and paging, the uclinux stuff is smaller and the ELKS userspace is tinier still."
28. I2O Update
19�Mar�2003 (2 posts) Archive Link: "PATCH: I2O updates"
People: Alan Cox
Alan Cox posted some patches and said, "This brings the I2O layer up to date. It fixes the i2o scsi hang on boot. It fixes the stack abuse by the i2o_proc code and it merges i2o_pci into i2o_core. I2O is now dead so no other transports are likely to appear, so this always rather messy abstraction can go."
29. Essential Reality P5 Data Glove Not HID Compliant
19�Mar�2003 (2 posts) Archive Link: "HID Patch for Essential Reality P5 Glove"
People: Jason McMullan
Jason McMullan said:
This patch adds the Essential Reality P5 data glove to the HID blacklist. It's yet another device that advertises itself as a HID, and doesn't comply with the HID specs.
The P5 is an inexpensive ($99 USD) data glove, with 5 finger flexion sensors and 8 IR location sensors.
A user-space library to access the P5 is available at http://www.evillabs.net/~jmcc/p5/
30. New OpenHPI Implementation For SAForum's HPI
19�Mar�2003 (1 post) Archive Link: "[ANNOUNCE] OpenHPI -- an implementation for SAForum's HPI"
People: Andrea L. Brugger
Andrea L. Brugger announced:
I'd like to introduce the OpenHPI project at http://openhpi.sf.net. The intent of this project is to produce an implementation of the Service Availability Forum's Hardware Platform Interface (HPI). HPI provides a universal interface for creating resource system models, typically for chassis and rack based servers, but extendable for other problem domains such as clustering, virtualization, and simulation.
We are currently in the design phase of the project and are soliciting involvement with others in the community.
Our goal is to have modular hardware support that can be implemented using a plugin architecture. This would allow a top-level OpenHPI implementation to be independent of the underlying hardware platform(s). We are also just getting a build environment up and a stubbed-out library implementation available. Work has also just begun regarding the planning and development of a certification suite for the HPI implementation.
For more information and how you can become involed please see...
Sharon And Joy
Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.