Wine Traffic #238 For 3�Sep�2004

By Brian Vincent

Table Of Contents

Introduction

This is the 238th issue of the Wine Weekly News publication. Its main goal is to play blackjack. It also serves to inform you of what's going on around Wine. Wine is an open source implementation of the Windows API on top of X and Unix. Think of it as a Windows compatibility layer. Wine does not require Microsoft Windows, as it is a completely alternative implementation consisting of 100% Microsoft-free code, but it can optionally use native system DLLs if they are available. You can find more info at www.winehq.org (http://www.winehq.org)

Mailing List Stats For This Week

We looked at 226 posts in 958K.

There were 57 different contributors. 30 posted more than once. 25 posted last week too.

The top posters of the week were:

1. Creating Type Libraries

28�Aug�2004�-�31�Aug�2004 (16 posts) Archive Link: "working together on stdole.tlb and a end to dcom9x"

Topics: RPC / COM / OLE

People: Steven Edwards,�Mike Hearn,�Alastair Bridgewater,�cvs,�Huw Davies,�Filip Navara,�,�ReactOS

Wine's DCOM has improved over the years, but it's still lacking some important functionality. Type libraries (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/automat/htm/chap9_49pv.asp) are one such problematic area. Steven Edwards asked this week, " Once again cross-posting because I would like to see if this is something we can work together on. What is left to do for us to be able to build and have a working implementation of stdole.tlb or any other type libs? I ask because it seems Wines DCOM implementaton is almost good enough for most apps except for this file and ReactOS is quickly getting to the point that it can run most of the same stuff Wine can without DCOM9x installed."

Mike Hearn described where the problem is:

It basically involves implementing the ICreateTypeLib interfaces in our OLE dlls. Nyef did some great work on this but for stdole32.tlb we need the other version, I think.

Once this is done maybe we can pull the stdole32.tlb generation program used in Crossover upstream. Right now it's pretty useless for Wine as it has to be run on Windows to work.

Filip Navara asked for more details, since it appeared ICreateTypLib2 was already implemented. Alastair Bridgewater replied:

The ICreateTypeLib2 interface is used to create the new-style MSFT typelibs. The older ICreateTypeLib interface is used for the old-style typelibs. While the newer interface also implements the older one, when you ask for the old interface with CreateTypeLib() you get a completely different implementation.

I don't see why it necessarily follows that stdole32 wouldn't work as an MSFT typelib, but the implementation of ICreateTypeInfo2 is badly incomplete in places, and I would trust it to completely mess up any semi-complex type library you attempt to create with it. Specifically, any type library that involves functions or variables.

What he meant by that is, there are two different kinds of typelibs. It's best explained by a comment in dlls/oleaut32/typelib.h, " There are two known file formats, those created with ICreateTypeLib have the signature "SLTG" as their first four bytes, while those created with ICreateTypeLib2 have "MSFT"."

Along these line, Huw Davies agreed using the MSFT format might work, " You're right, an MSFT stdole32 might work fine - I guess we should at least try it. So for fun I've attached (http://www.winehq.com/hypermail/wine-devel/2004/08/att-0639/01-stdole32.c) the program we used to generate CrossOver's stdole32.tlb (Actually you need to take the generated file and wrap it up in a resource only dll, but that's a detail). Since it was designed to be compiled with MSVC, it uses the L"" construct for olestrings - you can hack around this by compiling with -fshort-wchar for now."

Filip Navara tried it out, and did get something out, " I tried to change CreateTypeLib to CreateTypeLib2 and running it with Wine DLLs. It produced bunch of FIXMEs, but generated the type library in the end. The result is attached. Compared to the type library generated using the same executable with Windows DLL only two bytes differ. Is there some type library dump program that can be used to find out what are the differences or am I left alone and should I try to find out the actual differences other way?"

Steven Edwards found something that might help figure out what was generated:

After a quick google search. I dont know if this will give you all the information you need but it does have a sample program that can do simple dumping of typelibs.

There was no reply and the thread ended there.

2. Scheduling (con't)

24�Aug�2004�-�31�Aug�2004 (21 posts) Archive Link: "Re: [ck] Re: Threading issues? [ck-request@vds.kolivas.org: ck Digest, Vol 3, Issue 16] "

Topics: Architecture

People: Mike Hearn,�Ove Kaaven,�Robert Shearman,�Con Kolivas,�Andreas Mohr,�,�Ove K�ven,�Rob Shearman,�TransGaming

If you've been following Linux kernel news, you've probably seen the discussions over the past year concerning schedulers. There's been a lot of work done in this area, but it's a very tricky problem to make every process run exactly the way it was intended without interfering with other ones. A few weeks ago (WWN issue #236 (http://www.winehq.com/?issue=236#Threading%20Issues?) ) Andi Mohr forwarded a message from a kernel development list that described a problem with Wine's interaction with the Linux scheduler. More discussion occured this week after Mike Hearn asked, " One thought that occurred to me, and this is just a random theory, is that maybe the issue is not with the Wine code but the Win32 code run on top of it. Do you know how 2.6 scheduling compares with 2.4 and Windows (NT) scheduling? Could it be that some apps are written to expect Windows-style scheduling and fail to work if they don't get it?"

Now, I'll ruin the ending of this thread right now to tell you that nothing seems to have been resolved. However, it pointed out some fascinating things about Windows, Wine and Linux. Con Kolivas thought about it for a while, but didn't have an answer. Windows treats scheduling differently than Linux and Con didn't have the background on how that worked. Now, Wine relies on the Linux kernel to do the actual scheduling of threads, however some Windows applications expect scheduling to work a specific way.. something different than what the kernel actually does. Ove K�ven explained how scheduling works on Windows and some of the pitfalls associated with it:

A book I have described how it works...

The Windows scheduler is in its most basic form remarkably similar to the scheduling policy known in Linux as SCHED_RR, with a few "enhancements". For example, the thread owning the foreground window gets a priority boost relative to other threads in the system. There's also some stuff in there to help deal with the priority-inversion problem known to real-time programmers, I think. (When a low priority thread can keep blocking the execution of a high priority thread by holding a lock the high priority thread needs, with the result that medium priority threads get to run but the high priority thread does not, it's called priority inversion, if I understand right.)

Also note that Windows allows a Win32 process to boost its own priority all the way to what they call "real time". Only root can do this under Linux. I'm not sure if you need administrator privileges to do this under Windows (probably not), but since every Windows user runs as administrator anyway, it's probably not unlikely that many applications expect this ability anyway.

Internally, Windows assigns each thread a priority, determined by process priority class, thread priority class (all programmatically set), and some other factors such as whether it's running in the foreground (owns the foreground window) or not. Threads with the same scheduling priority are scheduled round-robin. Threads with higher priority preempt anything with lower priority, so that lower priority threads will not get to run as long as a higher priority thread is runnable. It is of course recommended and assumed that applications use high priority only for threads that won't use much CPU, since if they do, execution of that thread will block all threads with lower priority than itself.

The book described no kind of dynamic scheduling based on how much CPU time a thread is spending, only the aforementioned "is in foreground" thing. Does imply that a Win32 app running an infinite loop could run fine in the background, but would hang the system if brought to the foreground, I think (assuming Windows doesn't see that it has hung, which it probably checks by seeing if it responds to messages).

Rob Shearman then provided a link that explains more detail about the priorities a thread may have:

As decribed here:

There are 31 priority levels and only users with Administrator privileges can set processes to the highest priorities of 16-31. I doubt there are programs that require these high priorities, although we could find potential targets by printing a message for programs that try and set their priority to a high level.

Ove took a look and then came back with a real world example of where this problem occurs:

OK, so you need administrator to set a realtime priority, but you are still able to set a high priority on yourself without being an administrator under Windows. This is not possible under Linux, where a non-root process can only reduce, not elevate, its own priority.

For what it's worth, at TransGaming we know for a fact that some games (maybe it was Battlefield 1942) does; they elevate the priority of one of their most critical threads, and seem to use interesting mechanisms to regulate how much time this high-priority thread use relative to the other threads. In any case it does help the game's smoothness to experimentally elevate this thread's priority under Linux, but this is not really a solution because of permissions.

Mike knew TransGaming had problems with 2.6 and wondered if they had solved them. Ove explained:

we still consider 2.6 too unstable to support the use of, but the problem is probably related to what's being discussed here, which is why I'm interested.

One interesting problem we've seen is the one of wineserver using a whole lot more CPU now than before. A scenario where this could happen is if the game has a relatively low-priority thread which gets to do a lot of waits on critical sections, and some high-priority threads that does interesting stuff, also usually holding those critical sections, so there's always contention. The game might not care much exactly how often the low-priority thread gets scheduled, as long as the resource protected by the critical section is not in use whenever it does get scheduled (which it probably usually isn't if the Win32 thread priority system is used).

Well, my theory is that in this situation, with the great new interactivity-boosting Linux scheduler, the wineserver is considered very interactive, and gets scheduled a lot more than the high-priority threads, just so it can satisfy the wait requests of the low-priority thread.

This particular situation is a problem which the shared-memory wineserver we have in Cedega now should help somewhat against, I suppose, but that probably doesn't make this problem less interesting for you.

And of course there are threads we use that gets penalized a lot. For example, the DirectSound mixing thread would by no means be considered interactive by the scheduler. We would love to elevate the priority of this thread, for example, but it's not possible because of permissions.

Con took it all in detailed some of the challenges being faced:

To be honest, knowing how linux manipulates priorities dynamically and that more of the OS is threaded at different layers from windows, it is a miracle than any of the high performance gaming might run smoothly.

Audio runs as a separate thread outside of wine potentially through who knows how many layers as a combination of both process and kernel context so that's already complicating the issue by having a separate thread. X does the same again. So what the win32 game is doing under wine when it asks for audio or graphics to be done will be handled completely differently to windows. How blocking is done between the graphics and audio calls, when you have a highly cpu bound task (the game) as well is definitely complex and not just the kernel that is responsible here. Add to that the fact that wine itself runs a few threads and as I said before, it's a miracle it runs smoothly at all.

He went on to explain that nice seems to be the best solution. Changing the scheduler just to adapt to one program might not be the wisest decision and nice seems to perform what's required to get things to run smoothly. Unfortunately it comes with the caveat Ove had explained, only root can get the priority levels to behave properly. Andi Mohr thought this was a good reason for the idea that's been tossed around of having Linux's scheduler become a "pluggable" type interface, " This shows that dynamic reloading of different schedulers is a very useful thing: if playing some game on Wine doesn't work with the currently used scheduler, you just load a specially adapted scheduler (say a "Wine" scheduler for close Windows scheduling compatibility ;) in order to have some great gaming session, and that's it. (of course it'd be useful to have a universally usable scheduler instead, but that is very difficult if not impossible to achieve) "

3. epoll (con't)

29�Aug�2004�-�31�Aug�2004 (8 posts) Archive Link: "Re: wineserver epoll"

Topics: IO, Architecture

People: Shachar Shemesh,�Alexandre Julliard,�Mike Hearn,�Marcus Meissner,�

Shachar Shemesh submitted a patch to implement epoll() functionality, for background on this see the past two issues, #236 (http://www.winehq.com/?issue=236#poll%20vs.%20epoll) and #237 (http://www.winehq.com/?issue=237#poll%20vs.%20epoll%20(con't)) :

After much talk, the patch seems ready for commit. Since there are many systems with slightly older glibc that have the header but do not have the implementation in glibc, or that have a stub implementation that is guaranteed to fail, we are calling the kernel functions directly. This also allows compiling on older systems than the code actually ships to.

Accordingly, the autoconf checks for epoll_create in glibc were dropped.

ChangeLog:
Shachar Shemesh
configure.ac
configure
include/config.h.in

server/fd.c

Alexandre pointed out a problem with the patch, " That's wrong; if the libc function exists you should use it, direct system calls should only be used as a fallback. And when using system calls you must not depend on the libc header, you have to define the structures yourself."

Shachar thought there might be a problem with that, " But what to do if the library functions exist during compilation, and do not during runtime? Merely linking with them will cause wineserver not to load, claiming that it requires "GLIBC version 3.2.3". Some wine uses (crossover office being one of them) require you to compile once and run almost everywhere, which run counter to this behavior."

Shachar also wondered what was wrong with using the libc header. With regards to building on one version of glibc and running on an older one, Alexandre explained it won't really work anyway. CrossOver gets around it by building "against a very old glibc" . With regard to using the libc header if it's available, Alexandre said, " The libc definitions must be used with the libc function, and the kernel definitions with the system call. There is no guarantee at all that they will remain identical (check the 64-bit file support for an example of how that can happen)."

Mike Hearn thought of some other ways to do it, " So I guess Shachar could use dlopen, or possibly the epoll functions are weak symbols - quite a lot of glibc functions are these days. In that case you can just mark them with the weak attribute and that's no problem. "

Marcus Meissner didn't like that:

Guys, pretty pretty please, just use the glibc function, add a regular configure check and use it if it does not return -ENOSYS.

All this confusing handling of dlopening etc etc is way overblown.

4. Richedit Control

28�Aug�2004 (1 post) Archive Link: "Richedit control"

Topics: Architecture

People: Mike McCormack,�

Ah.. the Rich Edit control. It's everyone's favorite control to ignore. What exactly is it, you ask? Well, basically the Rich Edit control (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/CommCtls/richedit/richeditcontrols.asp) is a word processor. Ever use wordpad? That's the incarnation of an executable built around this library. It's also a lot of work to replicate. Mike McCormack announced this week he's going to take a stab at working on it:

I'm going to play with the richedit control to try get it to draw RTF documents with formatting. My plan is to write a program that can display but not edit, then let somebody else add the editting functionality afterwards :) Once I can get the attached program to work, the plan is to change the code in text-writer.c to add the formatting information to the richedit control via standard richedit messages. If anybody is already working on it, or wants to have a go, let me know.

5. Darwine Display Driver

29�Aug�2004 (1 post) Archive Link: "Wine's X11drv integration in Mac OS X"

Topics: Status Updates

People: Pierre d'Herbemont,�

Last week (issue #237 (http://www.winehq.com/?issue=237#Quartz%20Display%20Driver) ) we discussed a new Quartz display driver, or at least the beginnings of one. Well, it seems perhaps that development might be scrapped in favor of a different plan. Pierre d'Herbemont outlined the direction but went on to describe the challenges being faced:

We are currently porting Wine to Mac OS X. Currently Wine has a display driver for X11, and we would like to continue to use it for the Mac OS X port: to display the GUI interface, we don't need the Aqua UI elements since we prefer to use Windows theming system, and given the complexity and amount of work done in x11drv . But we also would like to be able to build something that behaves like a normal Carbon or Cocoa App, with a Dock Icon, and a Menu Bar, which would be managed by some Carbon code in Wine itself.

I had a look to the X11CallCarbonAndCocoa [1] sample code, but the trouble is that the X11 window still belongs to X11.app, a mouse click in the Dock to get the App's Windows to the front doesn't work for the X11 Window, and the menu bar for this Window is still the X11.app's menu bar. So I'd like to know if there is a way for the Wine Carbon instance to own the X11 Windows Wine created. At least it would be a feature that we'd like to see :)

Pierre sent the email to quite a few lists, but it didn't appear anyone had an answer. There was some dismay expressed at using X since the integration with MacOS X apparently isn't that good. Of course, the point of Pierre's request was to find out if anyone knew how to fix that.

6. OLE For Process Control?

2�Sep�2004 (11 posts) Archive Link: "Wine and industrial communication like OPC"

Topics: RPC / COM / OLE

People: Rickard Svensson,�Mike Hearn,�Rein Klazes,�Juan Lang,�,�Rein Klaze

Rickard Svensson asked if Wine could be used to support some specific COM-based interfaces:

I have some questions about Wine and how/if it can be used with industrial communication like OPC (MicroSoft Com/DCom objects, DDE and more)

My short question is: Can I use Wine to make OPC communication work on a Linux system?

OPC "OLE for Process Control" A "general" standard for communication in industrial systems, that unfortunately is totally depending on Windows(DCOM)

On www.opcfoundation.org (http://www.opcfoundation.org) there are some companies who have made Unix OPC clients, but the license costs are high. There is also OPC true XML that is now a real product, that looks promising.

I'm in need to make/use some kind of OPC api (Wine DCom) for Linux, a api like that would make many companys in the industrial computer interested. Or as a backup I could have a Win program running in Wine on the Linux machine talking with the main app on the same computer and talking OPC to other machines that need that.

I hope Wine and/or you can help me in this case. It would be a tragedy if we had to switch operating system just because of this.

Mike Hearn had an optimistic answer:

Answer: almost certainly yes, with a catch:

  1. Wines current builtin DCOM is not up to scratch for what you want, and it will take a long time to get there. Really, given how unglamourous network DCOM hacking is and how few people have worked on it, it's unlikely to happen until funding is available.
  2. So ... you'd have to use "native" DCOM. This is a redistributable that can be downloaded from microsoft.com and installed into Wine. You'd have to read the EULA very carefully but I suspect you need a Windows license in order to use it.

However, even if you do, simply buying a copy of Windows then throwing away the box may actually still be easier than switching your project to another OS entirely.

Rein Klazes wondered if it would really be that easy, " Mike, I was hesitating to give this answer. Typically OPC will be used distributed, over an ethernet link. Did anyone test wine & native DCOM in such a way? Can you actually communicate between an application running under Wine with an application that runs under real Windows?"

Mike still thought it could work:

I think it may have been used for Exchange5/Outlook connections, but you are right in that I don't know if this works.

Wine's sockets support is good though, far better than its builtin DCOM. Based on what I know of the DCOM network protocol, I don't think we'd have any problems with it.

Only issue would be if you needed NT only features, as only DCOM for Windows 98 works natively on Wine.

Off the top of my head, I can't think of any big issues. Maybe dcomcfg doesn't quite work yet etc, but fixing any problems that arise would be much easier than migrating to a totally different OS, or bringing our own DCOM up to scratch

Juan Lang pointed out there are some potential problems that definitely aren't easily fixed:

I can think of a couple, but they may be able to be worked around:

With regards to using named pipes, that uses something like SMB for communication. Juan Lang explained it begins delving into an area that has traditionally been Samba's domain - their daemons sit on the ports required for communication:

The reason I've backed off from this is that it's hard to do correctly, and anything less than completely correct isn't going to be accepted by A.J. Ideal for us would be a file descriptor that we can do atomic operations on; that'd be duplicable like the rest of our handles. Not sure how we'd get that from smbd (see below for a different idea), and Samba3's public API is too high-level to be of much use to Wine. (Samba4 might be better.)

The ports issue prevents us from receiving connections, i.e. acting as a DCOM server. This is probably less important than being able to act as a client, at least for a while.

I asked Steve French (Linux CIFS module) whether he'd consider exposing named pipes in the Linux filesystem. He said he would, then asked me whether I'd given any thought to naming convention. I hadn't and I haven't had as much time to experiment with this of late, so I haven't responded. This method would allow us to act as a CIFS (TCP port 445) client. Acting as an SMB (TCP port 139) client isn't handled by the CIFS module, it's done by the SMBFS module. Not sure what Steve's plans are here.

Anyway, any of you guys interested in seeing RPC work under Wine (Greg, you still lurking around here?) might want to play around with this if you have time. I might myself once I land somewhere for a while, but no promises.

Sharon And Joy

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.