Derek Glidden gave an exploit to make his machine entirely unresponsive for
several minutes. He pinned this behavior to the VM in 2.4, and added that he
felt there were still serious problems before the 2.4 VM could be considered
stable. Others confirmed the behavior, and there was a long discussion. And
one point Miles Lane said, Mike and others, I am
getting tired of your comments. Sheesh. The various developers who actually
work on the VM have already acknowledged the issues and are exploring fixes,
including at least one patch that already exists. It seems clear that the
uproar from the people who are having trouble with the new VM's handling
of swap space have been heard and folks are going to fix these problems.
It may not happen today or tomorrow, but soon. What the heck else do you
want?
Derek replied that his original post had been intended to provide
some useful data-points, and Miles replied, Actually,
I think your original message was useful. It has spurred a reevaluation of
some design assumptions implicit in the VM in the 2.4 series and has also
surfaced some bugs. It was not you who I felt was sending enflammatory
remarks, it was the folks who have been bellyaching about the current swap
disk space requirements without offering any new information to help developers
remedy the situation. So, thanks for bringing the topic up. :-)
Linus Torvalds also replied to Derek's original post:
Now, this may well be true, but what you actually demonstrated is that "swapoff()" is extremely (and I mean _EXTREMELY_) inefficient, to the point that it can certainly be called broken. It got worse in 2.4.x not so much due to any generic VM worseness, as due to the fact that the much more persistent swap cache behaviour in 2.4.x just exposes the fundamental inefficiencies of "swapoff()" more clearly. I don't think the swapoff() algorithm itself has changed, it's just that the algorithm was always exponential, I think (and because of the persistent swap cache, the "n" in the algorithm became much bigger). So this is really a separate problem from the general VM balancing issues. Go and look at the "try_to_unuse()" logic, and wince. I'd love to have somebody look a bit more at swap-off. It may well be, for example, that swap-off does not correctly notice dead swap-pages at all - somebody should verify that it doesn't try to read in and "try_to_unuse()" dead swap entries. That would make the inefficiency show up even more clearly.
Felix von Leitner reported that some of the kernel headers would cause
userland libraries such as dietlibc that depended on them, to export the
wrong API. He pointed out a simple fix, but David S. Miller replied, Don't use kernel headers for userspace. Kernel headers
and user headers are distinctly different namespaces, and you have pointed
out only one of many places where we use different names/structures/etc. for
some kernel networking headers vs. what userspace wants.
Felix von
Leitner complained, What choice do I have?
Duplicate everything and then be out of sync when the specs change?
To
which Linus Torvalds replied:
Yes.
Even more preferably - write user-space headers that have _only_ the minimum amount of code in them. The kernel headers have a lot of cruft that is kernel-only, and that means that if you compile user space using them, your compiles will be slower than they should be.
The basic issue is that the kernel will _refuse_ to follow the "namespace of the day" rules of C89, C99, POSIX, BSD, SuS, GNU .. the list goes on. The kernel headers are not meant to be used in user space, and will not have the strict namespace rules that a lot of standards spend so much time playing with.
There aren't that many things that are actually useful in the kernel headers anyway. Most of them (like the IPv6 stuff) are really specified in other places in the first place.
Peter J. Braam put together some ext3 rpms and made them available at ftp://ftp.clusterfilesystem.com/pub/ext3. P.A.M. van Dam asked where he could find the ext3 patch against the 2.4 kernel, and Andrew Morton replied:
All the details are at http://www.uow.edu.au/~andrewm/linux/ext3/
Current status is "pretty solid". Performance is good. It's basically untested against LVM and RAID. It can deadlock under heavy load if you're using quotas.
Avoid those things and you shouldn't have any problems.
Porting it (back) over to -ac and fixing up the quota problems is basically the next activity on the list.
Dawson Engler wrote a utility to check source code for bugs. Specifically,
he said, You can look at this checker as
essentially tracking whether the information from an untrusted source
(e.g., from copy_from_user) can reach a trusting sink (e.g., an array index).
The main limiting factor on its effectiveness is that we have a very incomplete
notion of trusting sinks. If anyone has suggestions for other general places
where it's dangerous to consume bad data, please send me an email.
He listed fifteen bugs he'd discovered with this tool. Alan Cox quickly
patched many of them, and Oliver Xymoron added:
I wrote something similar to this for userspace (via ld preload). Most of my checks followed strings around and made sure they were length checked so as to avoid stack overflows, but I also checked args to open, etc..
In your case, basically all destinations are trusting sinks at some level: userspace gives you data to put it somewhere. You might want to instead check that data is passing through functions that 'detaint', by checking capabilities, etc. I bet that you'll find that something like 90% of code paths are covered by a few common security checks. And that most of the remainder could be rewritten to be.
Eric S. Raymond reported finding several new configuration symbols in 2.4.6pre2 that had no corresponding documentation. He listed them:
CONFIG_AMD7409_OVERRIDE
CONFIG_BLK_DEV_AMD7409
CONFIG_BLK_DEV_OSB4
CONFIG_BLUEZ
CONFIG_BLUEZ_HCIEMU
CONFIG_BLUEZ_HCIUART
CONFIG_BLUEZ_HCIUSB
CONFIG_BLUEZ_L2CAP
Maksim Krasnyanskiy claimed the CONFIG_BLUEZ items, saying they belonged to the Bluetooth subsystem, adding that Linux was the first OS to have official Bluetooth support. He described those options:
CONFIG_BLUEZ
Bluetooth is low-cost, low-power, short-range wireless technology. It was designed as a replacement for cables and other short-range technologies like IrDA. Bluetooth operates in personal area range that typically extends up to 10 meters. More information about Bluetooth can be found at http://www.bluetooth.com
Linux Bluetooth subsystem consist of several layers:
HCI Core (device and connection manager, scheduler)
HCI Device drivers (interface to the hardware)
L2CAP Module (L2CAP protocol)Say Y here to enable Linux Bluetooth support and to build HCI Core layer.
To use Linux Bluetooth subsystem, you will need several user-space utilities like hciconfig and hcid. These utilities and updates to Bluetooth kernel modules are provided in the BlueZ package. For more information, see http://bluez.sf.net.
If you want to compile HCI Core as module (hci.o) say M here.
Not unsure ? say N.
CONFIG_BLUEZ_L2CAP
L2CAP (Logical Link Control and Adaptation Protocol) provides connection oriented and connection-less data transport. L2CAP support is required for most Bluetooth applications.
Say Y here to compile L2CAP support into the kernel or say M to compile it as module (l2cap.o).
Not unsure ? say M.
CONFIG_BLUEZ_HCIUART
Bluetooth HCI UART driver. This driver is required if you want to use Bluetooth devices with serial port interface.
Say Y here to compile support for Bluetooth UART devices into the kernel or say M to compile it as module (hci_uart.o).
Not unsure ? say M.
CONFIG_BLUEZ_HCIUSB
Bluetooth HCI USB driver. This driver is required if you want to use Bluetooth devices with USB interface.
Say Y here to compile support for Bluetooth USB devices into the kernel or say M to compile it as module (hci_usb.o).
Not unsure ? say M.
CONFIG_BLUEZ_HCIEMU
Bluetooth Virtual HCI device driver. This driver is required if you want to use HCI Emulation software.
Say Y here to compile support for Virtual HCI devices into the kernel or say M to compile it as module (hci_usb.o).
Not unsure ? say M.
Randy Dunlap suggested saying "Sure? say M" instead of "Not unsure? Say M", and Maksim agreed and posted a revised list shortly thereafter.
Matt Nelson had a project that would require about 6G of RAM under Intel,
and he wanted to know if there were any caveats he should be aware of. Doug
McNaught suggested getting an Alpha machine, and explained, Pointers on IA32 are still 32 bits, so (as I understand it) each
process can address a maximum of 4GB. You would have to allocate multiple
chunks (in shared memory or mmap()ed files, so they're persistent) and map
them in and out as needed (which could get hairy).
But he added,
if you can split your task up into multiple
processe such that no process needs to address more than 4GB, an IA32 machine
will work fine.
Later Matt said, Thanks
to all that replied. I've never needed to use so much memory before, and
was ignorant to how much magic was implemented in the 64G support on IA32.
Unfortunately, there's not quite enough magic in there for my needs... now
to find an affordable SMP Alpha system....
Craig Lyons of Promise Technology sent a patch to Andre Hedrick, explaining,
In the 2.4 kernel there is support for some of our
products (Ultra 66, Ultra 100, etc.). As you may or may not know, our Ultra
family of controllers (which are just standard IDE controllers and do not have
RAID) use the same ASIC on them as our FastTrak RAID controllers do. The 2.4
kernel will recognize our Ultra family of controllers, but there is a problem
in that a FastTrak will not be recognized as a FastTrak, but as an Ultra.
Consequently, the array on the FastTrak is not recognized as an array,
but instead each disk is seen individually, and the users data cannot be
properly accessed. We have a patch that fixes this and are wondering if it
is possible to get this patch into the kernel.
Andre replied, I do not want or need your company's patches, period.
I will not take or accept or approve of any dirty code that allows the a
poorly written binary driver that can not control its ISR and it interferes
irresponsiblily with the native ATA driver.
Bert Hubert chastised, Craig contacted you to
find out what was wrong and you should explain to him what the problems are,
and how he could solve them. Linus would accept patches written by Bill
Gates if they were licensed right and coded properly, so I don't see why
Promise should be an exception.
He also explained to Craig, The procedure is to publish the patch publically and have
people comment and try it. They will often find that your code is not up
to par or does things in ways that do not please the kernel people. No evil
is intended, it is just that the kernel developers are a choosy bunch. But
given the right prodding they will tell you how you could change your code
so that it is acceptable. Alternatively, people here might see what needs
to be done from your patch, and do it themselves.
But Andre said,
No I would not take their code and apply it.
I might not even want to look at it. All I want is the API rules to the
signatures and we have them now. We do not need their driver.
Rob
Landley felt that, as a maintainer, Andre should be more "open" about their
patches. Andre replied, I have seen one version
and I got physically sick.
The thread ended there, but elsewhere, under the Subject: Eye2Eye a hope for Promise to Join Linux, Andre said to Craig:
I would like to publicly thank you for coming to the table of GNU/GPL with an open perspective. After 90 minutes on the phone, of which 45 minutes were me pointing out issues promblems and complaints w/ 20 minutes on ways to work on solutions in the near and distant future and the listening to your concerns and questions between my moments of interruption.
The next conversion will not have the burst-in moments because it will be in person or my cell battery will be fully charged.
Since you have stated "I will not make promise, I can not keep" this is a good thing and it will go a fair way to clean up messes from the past on both sides.
I look forward to Promise working with Linux in meaningful and productive ways.
Please reply and correct anything that is mistated by me or verify the correctness. This will show an action of good-faith before all those watching here.
Craig replied:
Andre and I did indeed have a nice conversation on the phone. Thank you again for taking the time to talk with me and offering your assistance. As I stated on the phone, we are making a large commitment of resources to supporting Linux by releasing drivers and utilities for our products, including the FastTrak. I know we have plans to release source for our Ultra and SuperTrak series cards, but at this point I'm not sure that the way we are going to be supporting FastTrak is what you would like to see. As I said, while I cannot guarantee anything that I don't have the authority to deliver, I will pass on your requests. I will try to be an advocate for Promise in the Linux community, and an advocate for the Linux community to Promise. If the company has concerns, I will let you know what they are, and then maybe you can tell us if we are off-base with those concerns or not.
I would invite anybody to contact me if you have any suggestions, any requests, whatever. As I told Andre, I won't promise something I can't personally deliver, but I will do whatever I can to help out. I'm also trying to get a technical point of contact so that you don't have to deal with a marketing weenie who doesn't understand half of what you're saying ;).