Kernel Traffic #49 For 3 Jan 2000

It will never happen on Linux.

I'm suprised an SGI person hasn't learnt from past mistakes. IRIX is unstable, and unmaintainable, and please just face it - it's because SGI had the "cool feature of the day" disease.

Thread-private mappings WILL NOT HAPPEN. You can obviously do them in SGI Linux, but that way lies madness, and it's something I'll keep pointing out in public.

You can have thread-private _pointers_. Just have different mappings of the same hardware context if you have to, and just have different pointers to it in different threads.

thread-private mappings are FUNDAMENTALLY broken. They completely break the whole point of having a thread in the first place, and I can only suggest that if you want them you should look at a great concept that was invented, oh, thirty years ago by people more intelligent than you or I.

Namely "fork()". Which gives you the thread-private mapping you want.

If you want threads with shared address spaces, then make them SHARED. None of this private stuff. It's not on the table, and it won't be. I've explained to people why before, and I bet I'll end up explaining again, but the short and sweet of it is that you CANNOT do thread-private mappings without losing most of the performance advatages of a thread in the first place.

And once you lose the performance-advantage of threads, not all that much else remains. Threads certainly aren't any easier to program than processes...

I agree with Linus. IRIX is full of things that noone but a marketing bunny would like, and Linux shouldn't follow in those misguided footsteps.

My understanding of what IRIX had to do to get their semantics to work, based on having been a kernel engineer at SGI for 3 years, so I've looked at this code, is that when you have private mappings, you typically end up replacing the TLB miss handler. The details get sort of messy, but the problem can be summed up as the address space does not belong to a thread, in fact it is the other way around. As long as your threads either share completely or don't share completely, then you only have one VM struct and one set of page tables to walk. As soon as you start having thread private mappings, then you end up having to have different (usually overlapping) page tables. Hence the new TLB miss handler which sorts out the mess.

You really really don't want to do what SGI did. What you do want to do is clearly state what your needs are - leaving out any details about how you want those needs solved - and keep restating them until someone says "ah - you do that like this" and Linus likes it.

I'm not talking about implementing Thread-private mappings for the entire system. What I have done is when the process mmaps the accel region set a flag. This way if the process creates new threads or forks it will have a private mapping. The impact of private mapping is thus minimized. Once a process unmaps the accel region the flag is turned off. The only way preformance can be killed on the system is if the process that mmapped the accel region creates a bunch of threads and none of the threads use the accel region. I just don't see that happening. Also how many threads will a well designed OpenGL library have. You don't want to go crazy here creating a bunch of threads.

The question we have to ask why do we want private mapping. The reason DRE (Direct Rendering Engine) needs this is to ensure a page fault happens. If you don't want private mapping for thread in this case then tell me a way to ensure every thread or forked process that uses the accel engine page faults. Why do you need a page fault? Because the page fault is what I used to serialize access to the accel engine on SMP machines and to save and restore the graphics context. As Jon Leech pointed out each threads need to refer to *different* graphics contexts. OpenGL is based on the graphics pipeline. Ideally each part of the pipeline would be used by a different process. Say we have SMP machine with one video card. Ideally a different process on each CPU would be in different places in the graphics pipeline. The problem is one thread could have the hardware in a state that could alter the desired results for the other threads since the other threads most likely need the hardware in another state.

Also as Jon pointed out the API requires that these contexts be identified by some thread-specific mechanism available to the graphics library, not by explicit stack pointers in the application. Either the threaded OpenGL API is broken or DRE for linux-SGI is. If this the case then linux will need its own special threaded OpenGL library compared to all the other platforms which don't require this special rewrite.

The thing that makes me admire UNIX is not that it's UNIX. The thing I like about UNIX is that even after 30 years, the fundamentals stand out. The basic notion of files as unstructured streams of data, and the UNIX fork/exec way of doing things (most everybody else does a "spawn()", which does not have the same philosophy at all).

And quite frankly, especially when it comes to threads, Linux has a DESIGN. It's not the hodge-podge of threading issues that we call "pthreads" that came about from different vendors tryign to solve the same problem in different ways (inluding user-mode only solutions). It's a CONCEPT, the same way "fork()" is a concept.

The difference between a "concept" and a "random collection of routines" is that the concept will survive, and the routines won't.

But that also requires that people don't mess up the concept by thinking that it's "just an implementation". Because once lost, you won't be able to undo the changes. You can fix bugs, but you can't easily remove the "feature of the day".

You've broken the concept of shared TLB's. You've then superglued the broken parts together and said "if you look at it from the right angle you cannot SEE the cracks".

But the cracks ARE THERE! They make the system more complex internally, and even when you don't see the cracks you may notice that the thing doesn't quite stand up straight.

You broke the whole notion of interchangeable, anonymous, shared TLB mappings.

You broke the notion that when a thread is switched, no MM work needs to be done.

You broke the notion that you can simply look at "process->mm" and determine whether it can share the TLB state with another thread (even if that TLB state in between was used by an anonymous kernel thread).

In short, you took a notion and dirtied it until it was just a random collection of routines that superficially LOOK like it has a design.

In short, you didn't care about the BEAUTY.

No. You're asking the WRONG question.

There is no way in HELL we want private mappings. End of story.

Use processes and SysV shared memory if that is what you're after.It works today, and it gets you EXACTLY the same semantics that you are apparently after. Sure, you have to think about the problem another way, but it's just a mirror image.

Instead of saying "ok, I want private mappings in a shared address space", you say "ok, I want shared mappings in private address spaces".

What's the difference? Doesn't soundlike much, no?

But look at it from a NOTIONAL standpoint. You don't break anything by taking a private address space and adding a shared object to it - we've had that notion for a long time, and then you HAVE a private TLB and a private page table to play with.

In contrast, if you break the sharedness of a shared address space, YOU DON'T HAVE ANYTHING LEFT! You just broke the bubble, and it popped. You turned it into a private address space, for no gain (you already HAD private address spaces, so you just degenerated the whole system).

Please keep in mind that what the SGI folks are complaining about here and how the 3dfx has to do are radically different issues.

Most commodity 3D graphics hardware these days cannot be interrupted mid-operation, have it's state fully saved, have another renderer's state fully restored, and let the latter continue where he left off. You must complete the full operation you are currently in the middle of before you can let someone else have the card.

Whereas most SGI, Sun, and other vendor's higher end cards allows you to arbitrarily stop a renderer mid-place, save the card state, and restore the card state another thread has. This can happen at nearly arbitrary locations, so the following works:

Thread 1	Thread 2
OP = RECTANGLE
X = 0
Y = 0
	OP = LINE (takes fault, thread1's graphics card state is saved, thread2's is restored and mappings are removed from thread1's space)
	X1 = 5
	Y1 = 10
	X2 = 10
	Y2 = 10 (draws the line)
W = 50 (takes fault, graphics state restored, thread2's state saved and his mappings removed)
H = 50 (draws the rectangle)

Commodity PCI/AGI 3D graphics cards cannot do this, which is why the userland locking solution exists at all. However for cards that can do the above, this is what people want.

Look out when you're condescending.

Commodity hardware is where it is at when it comes to 3D. Forget about CAD work etc - 3D is all about games, and probably always will be. It's just a fact that the game market is about a million times larger than the traditional 3D market ever was, and as a result they have more resources.

It's the i386 all over again. The "non-professional" 3D solutions are already getting level with the "professional" ones.

I don't understand why people are so hung up about page faults.

I think it's ENTIRELY because of historical baggage, and the particular implementation under Irix.

What I'm surprised about is that nobody seems to just come out and say:

Page faults are BAD. Playing with the page tables is EXPENSIVE. Page faults fundamentally are NOT thread-safe, because page tables are fundamentally shared among threads.

Ok. Nobody else said it, so now I have.

YOU SHOULD NOT PLAY MM GAMES! They do not scale in SMP, they do not scale with threads, and the costs of missing are absolutely huge. The whole thing is also extremely hard to debug, and implies a much tighter coupling between the kernel and the X server than there should ever be!

You can do a _regular_ SMP-safe lock with _real_ thread safety and no faulting behaviour in a few instructions. We're talking maybe 50 cycles here - about 40 cycles for the actual two locked instructions, and a very generous 10 cycles to check whether you are the old owner and going to the switch routine if not).

Note that IF you have to switch contexts, the regular lock will be a hell of a lot faster than taking page faults, so let's just ignore that case: page faulting obviously loses, and there's no way anybody can seriously claim anything else.

So let's look at the no contention case, where you got the lock, and everything was fine. You spent 50 cycles on verifying it. Big deal. That's 50 _CPU_ cycles. Not memory cycles, not PCI cycles. In exchange for those 50 cycles you get:

• you can debug things
• it's thread-safe
• you have much lower latency if you DO get graphical context switching, and you can actually test it in user space first!

Oh. And btw. It's already been done. See the 3dfx driver.

So forget this playing with mmap and page faults. Use mmap() as a way to access the physical hardware, but NOT as a way to switch contexts. Ok?

After doing too many last-minute updates of critical code that we really shouldn't have left this late (Both the mm layer and the SCSI layer was changed quite a lot: we'll be better for it, but I'd have been happier if we hadn't needed to), I'm going to calm things down. I've released 2.3.33 which fixes a few smaller problems with 2.3.32, and I'll let it quiet down a bit for a while.

We're obviously not going to have a 2.4 this millenium, but let's get the pre-2.4 series going this year, with the real release Q1 of 2000.

Contrary to popular misconceptions, PEOPLE DON'T CARE!

The fact that our forefathers were Pascal-programmers, and started counting from one does not mean that we have to continue that mistake forever. We've since moved on to C, and the change from 1999->2000 is a lot more interesting in a base-10 system than the change from 2000->2001.

The reference point of our timekeeping is based on an event where the uncertainty about the timing is much more than a year, and was made up several hundred years AFTER the fact. As such, if you want to be a stickler, you might as well say that the next millenium may have started several years ago.

So please stop sending me email. You don't have to celebrate if you don't want to. But let the rest of the world who doesn't care about silly irrelevant details (what's a millenium to you anyway) just go on with our life.

NEXT year I may agree with you. I'll join the ranks of people with no life but the ability to count in another 360 days or so. But that's mainly in order to have an excuse to go out to town.

For those of you interested in this:

• According to the bible, Jesus was born when Herod Agrippa was ruling in Judea. Herod Agrippa is documented has having died in 4 BC.
• According to the bible, Joseph and Mary had gone to the city of Bethlehem because of the first Great Roman Census. History records that these censii were held every 14 years with the second being in AD 7 and the third in AD 21.

These two facts imply that the second millenium ended on 31st December 1993, and we have been in the third millenium for nearly six years now.

With the filesystems/VFS situation looks so (and I'm not going into IWBNI area, only code that needs fixing):

• ADFS, AFFS, HFS, NTFS, QNX4 - blatantly broken.
• UFS - needs cleanup/fixes.
• loopback, ramdisk, raid - more or less broken.
• CODA - will need serious testing after the expected large patch.
• procfs - needs decision on namespace policy/namespace rework. And proper dealing with races on module offloading, but that's old story.

That's just the most pressing stuff. I'm going to fork the -bird (aka VFS-CURRENT) after 2.4.0 and will feed the stable/well-tested stuff back into the main tree (with intention to collapse it after 2.5 will open), but it would be nice if we could fix at least the stuff mentioned above _before_ 2.4.

Taking my working list the key items seem to be

• About 70% of device drivers dont have the setup code fixed. In many cases that makes them unusable non modular
• About 1/3rd of the ISA device drivers using memory mapped IO are still broken AF_UNIX sockets are broken, and there are numerous IP layer bugs only fixed in DaveM's trees that have to be fixed, either by extracting them or going softnet.
• The SPX sockets are totally broken
• The PIII/Athlon/K6/Pentium memcpy etc acceleration code needs merging
• NCR5380 is broken for SMP
• AHA152x is broken for SMP
• Anyuser can crash the code due to module races (eg open)
• RAID 0.90 needs merging
• Vmalloc needs to take flags for DMA optionally (for 32bit PCI scatter gather)
• Getblk/mark buffer races in most file systems
• Protection for inode size fields is wrong right now

Less critical stuff

• Switching to the new i2c code core and the new bttv driver. Affects only some tv cards so is ok
• Per process rt signal queue limits
• Syncppp should use the new generic ppp code
• Back merge 2.2.13/2.2.14 fixes

I have the i2c stuff sorted mostly (thanks to Gerd and co), the 2.2.13/14 stuff I will do next week and is all bug fixes, Im still trying to stomp all the isa memory mapped I/O issues. I'll also sort the vmalloc cases out.

Most of the above is bug fixing or driver stuff so is post freeze work. The raid and PIII stuff are not. I'd also prefer you to look at the core softnet changes and say yes/no, then draw your line the side of it you choose. The softnet work is the other 50% of the scaling work, without it 2.4 wont scale much better than 2.2 for real world situations. That bothers me.

I've got a list of other stuff (Erez stackable fs, telephony API work, performance counters, lm-sensors, ibcs) that are in the nice but so be it category. I have no problem with those landing outside of 2.4.0 or staying as add ons. Skipping softnet though would I think be a mistake.

The algorithms I know about seem to be along the lines of the following:

1. Read a dozen or so pages of source code looking for whatever bug one is after. Drink several large mugs of coffee whilst doing so.
2. Toss a coin. If heads, go to step 3; if tails, go to step 4; if it lands on its edge, go to step 5.
3. Note a bug one wasn't looking for, and fix it. Go to step 1.
4. Overlook the bug one was looking for. Go to step 1.
5. Have a brainwave, and go directly to the bug and fix it, even though it was nowhere near the code one was looking at. Go to step 1.

Put any timescale you care to against that lot.

The problem is that NFS relies on the user sending the RPC authentication each and every time we access data on the server. In order not to get a permissions error suddenly if the user changes euid while s/he is reading/writing to the open file, we therefore want to use the same RPC authentication info throughout the file's lifetime. Ideally that means taking the RPC auth info that was valid when opening the file (since this is more or less in line with a POSIX filesystem's behaviour with permission checking at file open only) and caching it somewhere.

The most practical way of implementing this policy is therefore to hide the RPC auth in the file descriptor structure (I use the private data field), and pass that info via the file pointer to readpage/writepage/whatever else needs it.