Kernel Traffic #135 For 1 Oct 2001

You can run some of the tests made especially for testing latency, like an audio benchmark. One such test is at http://www.gardena.net/benno/linux/latencytest-0.42.tar.gz

Obviously a heavily tasked I/O benchmark is useful, I have used dbench in the past (ftp://samba.org/pub/tridge/dbench/) (try it with 16 processes or so), but I have been told I should use bonnie.

You can time normal things, too. `time make dep clean bzImage' is always a favorite :)

Under UP, enabling preemption helps all of this (the recent linuxdevices article on preemption shows a 30x decrease in latency.). Both myself and Nigel have run dbench with good results for -16. See http://kpreempt.sourceforge.net/ for some more.

After a few days of developement I think I'm ready to release the VM rewrite I did.

The alternate vm will be included in 2.4.10pre9aa1 (or anwways the very next -aa release) and I'll maintain it in the -aa tree. It is supposed to provide:

1. stable kswapd, avoid the kswapd 100% load of the cpu problem (this is provided by the classzone design, btw I improved the implementation a little bit compared to the 2.3/2.4.0-test patches, now I try to do things as lazily as possible without the bookkeeping in the pagealloc/pagefreeing)
2. optimal performance, avoid slowdowns after multiple runs of workloads and avoid swapout storms (for databases not using O_DIRECT)
3. you will get swap+ram of available virtual memory

At the moment it's of course still a bit experimental and subject to changes but I'm writing this email on top of it and it's perfectly usable.

This isn't an hack/band-aid or a small set of changes, it's a complete rewrite from scratch of the whole memory balancing including garbage collections lru lists, kswapd etc...

That is the wrong direction to go into.

We'll be completely screwed on NuMA with the classzone patch. I've said so before, I'll say so again.

The basic approach of the classzone patch is _wrong_, in making global decisions where no "globality" exists.

I bet that the improvements are from other things, not from classzone itself. And I will bet that if we start doing classzones, we'll regret it a LOT in a few years.

Ok, the big thing here is continued merging, this time with Andrea.

I still don't like some of the VM changes, but integrating Andrea's VM changes results in (a) better performance and (b) much cleaner inactive page handling in particular. Besides, for the 2.4.x tree, the big priority is stability, we can re-address my other concerns during 2.5.x.

This also merges the blkdev in page cache patch, and that will hopefully make it noticeably easier to do the "do bread() with page cache too", at which point a lot of the current ugly synchronization issues will go away.

Oh, and it gets the direct-IO improvements from Andrea too.

[ Other small patches from Andrea merged just to make future merges easier. ]

The console locking merge with Andrew Morton also moves us a bit closer to the -ac tree..

Andrea,

Personally I think it is too late in the 2.4.x series to integrate your code.

I have nothing against the code itself (the "old" code also had bugs), but a major VM rewrite at this point seems to be dangerous if we want a stable VM.

Don't you agree that your code can introduce new stability bugs ?

Linus, please, if you want to integrate Andrea's code do it in 2.5.x, but not 2.4.x. (yes, I'm expecting you to scream at me now :))

Without hesitation? Hardly.

The bug fixes in -ac that aren't merged into -linus are that way BECAUSE NOBODY HAS SENT ME MERGES.

Alan works on it quite intensively, but the fact is, that for the -ac merge, Alan seems to be able to merge it slower than -ac grows. Which is why I actually started asking people to merge their parts from -ac into -linus to help Alan. That's how the other merge in -pre11 happened.

The aa tree has existed for a long time, and is actually mainted in better chunks than the -ac tree, which makes it easier to merge. Admittedly my and Alans tastes are often closer than mine and Andreas, which means that the -aa tree merges are more painful for _me_ ;)

Funny that you mention it, because I actually have a cunning plan, and you're an unwitting part of it.

Or actually, I hope you're a "witting" part of it, because it's going to be your code.

Take your "struct block_device" code, add a "struct address_space" to it, and whenever a block device inode is opened, make the inode->i_mapping point to &bdev->b_data, and voila..

You already get all the reference counting right, and it's the only sensible place to do it anyway, wouldn't you agree?

I thought you'd be thrilled. It seems to match your lazy allocation patch very well..

Yes, we can make it work that way (see downthread). Yes, combined with lazy allocation (and pipefs-like scheme) it can turn into nice, _working_ code.

But right now we have

1. broken patch applied to the tree
2. somewhat tested patch that may (after being modified so that it would _apply_ to -pre11) fix the breakage. Once it's tested enough to be consider as a candidate for inclusion, that is.

IMNSHO it's somewhat less than ideal situation. I've already talked to with Andrea and once he gets back to life (no, 'e's just sleepin') we'll try to do something usable. Life would be much simpler if aforementioned cunning plan included sending mail to participants (me and Andrea) before putting the patch into the tree, though. Oh, well...

That's the part I like about the page-cache bdev patch. It has a lot of fairly ugly warts, but all of them seem to be really fixable with _other_ cleanups, at which point only the good parts remain.

I agree that the timing may leave something to be desired. But we had the discussion about fixing pagecache-bdev consistency wrt the regular buffer cache filesystem accesses a week or so ago, and the fact is that nobody really seems to have started working on it - because everybody felt that you have to get everything done at once.

I don't have that feeling. I'm happy with having partial merge with ugly warts, if it means that you can get to the final stage _without_ having to have all the problems fixed at one time.

So now we have two _smaller_ merges that will fix two other issues, and remove all the horridness from the original merge.

The real question is why can't we just open 2.5 and only fix the VM to start with? Leave the kernel at 2.4.10-pre10, and possibly use the -ac VM code (which has diverged from mainline), and leave people (Alan, Ben, Marcello, et. al.) who want to tinker with it in small increments and do the drastic stuff in 2.5.

Make it clear that 2.5 is still ONLY for VM and other bug fixes at this point, and not all of the long-awaited 2.5 rework YET. If it turns out that the VM fixes are done quickly, then they can be back-ported to 2.4. If it takes longer than expected you can open 2.5 up to general development.

With the right "management" it will be not much different than the current situation, except that people won't have fits about such huge changes going into 2.4. I think this is your subconcious telling you that you really wanted to start 2.5 a month ago.

Look, the problem is that Linus is being an asshole and integrating conflicting ideas into both the VM and the VFS, without giving anybody prior notice and later blame others.

Just look at how he's now trying to force Al Viro into implementing his ideas yesterday because he broke stuff again...

If you want a stable kernel, use Alan's kernel.

Rik, in case you've missed that, I can and do speak for myself. I had spent ten years dodging the draft; when I decide to get enlisted into something it will happen on _my_ decision and _my_ conditions. When I decide that I'm being forced into something I do not accept - you'll know it from posting with URL of forked tree.

FWIW, I'm less than thrilled by the Andrea's patch, but it is salvagable. I'm also less than thrilled by the whole situation with VM - all sides of it. I seriously suspect that we need a limited multi-way fork in that area, so that you guys would stop stepping on each others' toes. I'm taking no part in your merry 5-way clusterfuck - sort that mess out between yourselves.

Again, when I decide that situation is unacceptable for me - I'll simply fork the tree. I do _not_ appreciate being enlisted into anyone's holy wars, so unless you really want to go _way_ up in my personal shitlist (several positions below .ru DoD) - don't play politics in my vicinity.

I am rewriting he wrapfs from the fist project and is now in a debugging stage, it is now quite ready for experimental tests.

The idea is orinigally from FiST, a stackable file system. But the FiST owner Erez seems given up to maintain the project. At the time I receive the code, it is so buggy, even unusable, lots of segmentation fault problems. I have debugging the fs for quite a while. Now it is useful in just use as a file system wrapper. It is useful in chroot environments and hardlinks aren't available. It wraps a directory and mount to another directory on tops of any filesystems. I wish to maintain this file system development since FiST's idea is good. It allow to use this base wrapfs as a template and then you can do encryption and other operations on it with fast development time. If any kernel file system maintainer is interested, please contact me .. I would like to get help to finish up my debugging work. The result will be GPL'ed. I will also package it with the fistgen package as a file system development tool.

I'm currently doing some research on implementiation of a new file system for my diplomathesis. It's about designing and implementing a network file system with security in mind. For those interested, there is a short introduction about why and how on my website (look at the Secure Internet File System section):

http://www.linuxnetworks.de/security/index.html

I have started work on the patch for /proc/sys/kernel/tainted with the corresponding modutils and ksymoops changes. insmod of a non-GPL module ORs /proc/sys/kernel/tainted with 1, insmod -f ORs with 2.

What to do about modules with no license? Complain and taint or silently ignore? A lot of modules in -ac14 have no MODULE_LICENSE, probably because they have no MODULE_AUTHOR. IMHO the default should be complain and taint, even though it will generate lots of newbie questions to l-k.

An ext3 patch against linux 2.4.10 is at

http://www.uow.edu.au/~andrewm/linux/ext3/

This patch is *lightly tested* - ie, it boots and does stuff. The changes to ext3 are small, but the kernel which it patches has recently changed a lot. If you're cautious, please wait a couple of days.

The patch retains the buffer-tracing code. This will soon be broken out into a separate patch to make ext3 suitable for submission for the mainstream kernel.

Changelog:

• Fix an oops which could occur at unmount time due to non-empty orphan list. This could be triggered by an earlier error during a truncate.
• Merge Ted's directory scan speedup heuristic.
• Remove the abort_write() address_space_operation by ensuring that all prepare_write() callers always call commit_write().
• A number of changes to suit the new 2.4.10 VM and buffer-layer design.

If this passes, everyone working in computer security can be arrested and thrown in prison for life. In addition, people such as Kevin Mitnick can be thrown back in prison even though they have already paid for their crime (double jeopardy?).

http://www.securityfocus.com/news/257

The "WEP crack" fallout will be interesting to watch also.

In theory under the new law anyone whos computer was infected by nimda/codered could be imprisoned for life -- the new law says nothing about intent. So basically we would have a few million microsoft windows users serving life sentences...

When people are crashing planes into buildings and killing people by the thousands, hacking laws should be tough. The US has shut off internet access from Cyprus and several other places, and I've noticed a fall-off of hacking on my servers -- GOOD!.

Maureen O'Gara at Client Server News is based in NY, and from what she describes, the entire city is in a terrible state. Let anyone in New York know who is our friend on this list that the Utah Native American Church has sent James Mooney to New York City to conduct ceremonies for the victims and their families. The mayor's office has given us permission to conduct our ceremonies there for these people without fear of police harassment.

I am sending him enough peyote to trip out half the city. Anyone in NY who needs to find healing who is a member of our linux "Family" is welcome at these ceremonies. These people involved in this terrifying ordeal need to sit in a tepee and go somewhere else for a couple of days with the sacred medicine.

New York folks who wish to be involved in these ceremonies can call 212-755-0968 or 212-929-9396 to find out where and when. We have so far hosted thousands of the victims in these ceremonies. All are welcome and their families. The laws in New York allow non-Indians to use peyote for religious purposes of any race, unlike Utah. Tell our brothers we open our doors to those in need of spiritual and emotional healing for the people of New York.

These ceremonies are **FREE**. The Utah NAC is picking up the tab.

In what way has the recent violent acts differed significantly from acts which have been ongoing world-wide for, well, always? Is it that "it doesn't happen here!"? This is an increadibly US centric world view.

When the world seems to be at peace, it is easy to ask for your rights. It is when the war comes to you that you really need them, and that they are hardest to request.

This was a violent crime, commited by men who were willing to die. It was a failure of physical security; and massive databases will not make it harder for someone who is willing to sacrifice themselves.

But they will affect the ability of the population to conduct acts of civil disobediance and rebellion, upon which this and many other contries are founded.

The war in the world is not new, we are simply used to ignoring it. And for this, we are widely hated and scorned.

I will not grant the statement that "This sort of thing must be prevented at all costs". There are some prices I will not pay, and some that I will immediately distrust anyone who asks me to pay them.

I know that I am not making friends with this post, but my conscience demands that I respond to your blind aquescense of rights. I want a world in which my children can choose their life, even if the cost is a reduction in 'security'.

E-fucking-nough people. Stop this thread now, it is off topic.

There are many places out there where constructive conversations on this topic can be had, but vger is not one of them.

Please don't make Matti and I add more keyword filters to vger's list system to prevent this.