This is the 237th issue of the Wine Weekly News publication. Its main goal is to eat Vicodin. It also serves to inform you of what's going on around Wine. Wine is an open source implementation of the Windows API on top of X and Unix. Think of it as a Windows compatibility layer. Wine does not require Microsoft Windows, as it is a completely alternative implementation consisting of 100% Microsoft-free code, but it can optionally use native system DLLs if they are available. You can find more info at www.winehq.org
There seems to be an interesting trend going on.. or perhaps it's
just a bunch of weird coincidences. Over the past month I've noticed
a lot of off-hand references to Wine in a bunch of different sources;
everything from software reviews to mailing lists (besides Wine!) that
I'm subscribed to. It seems like Wine is appearing on the radar screen
for more and more people. For example, consider this
linux.com article that describes using Wine to
run a Waste
client. (Waste is a
P2P app for instant messaging and file transfers incorporating a lot of
encryption.) The fact that Wine is involved seems more of an afterthought.
Also check out this Linux review that comments,
Here's what I need to do next. ... Learn how to
use Wine - a clever way of running Windows programs under Linux - for my
accounts program.
Trend, coincidence, or an editor in search of
a news story.. you decide.
The Darwine folks popped up this week to let us know what they're working on. It's been quite a few months since we've given an update on the project (see WWN issue #207 for the last one.) There's definitely a lot of exciting stuff going on in that area and it seems like preliminary support for running i386 Windows binaries on MacOS X is in place. A slew of emails hit wine-devel this week, the first of which was Sanjay Connare's announcement:
I am proud and happy to announce the release of a new binary labeled Darwine 20040820 DP. This binary is based upon the 20040813 wine release. The majority of changes for this release can be viewed:
http://cvs.winehq.org/cvsweb/wine/ChangeLog
It is being released as a DEVELOPER PREVIEW so USE AT YOUR OWN RISK! It is not yet suited for mass distribution or general user use. This release in addition to all of the fixes from the wine team includes many Mac OS X optimizations and fixes. It also includes WineHelper a native Mac OS X application that helps launch .exe's. It allows for .exe's to be double clicked in the finder like any other application and essentially bypasses the need to run wine via the command line.
The Darwine libraries have been moved from their default installation location of /usr/local to a more Mac OS X friendly location /Library/Darwine. Applications and WineHelper are installed in /Applications/Darwine.
This new binary also automatically uninstalls the previous binary if it is found to be installed. For uninstall to occur the user must enter an administrator's password. If the correct password is entered the uninstaller will return some errors however the old version of Darwine will be removed and the install of this new binary can commence by reopening the installer package. A new uninstaller is included with this release that simplifies the uninstall process thanks to the new location of libraries.
The binary has been split into two releases one being this release and a soon to be released Darwine SDK. The Darwine SDK will contain the source for both wine and WineHelper, all of the header files, winegcc, winebuild tools and many examples and scripts that will allow open source apps to be compiled using these tools.
Please report bugs to the darwine SourceForge Bug Tracking System, and to the darwine-devel mailing-list. Also any comments, questions or concerns can also be posted to the darwine-devel mailing-list. Enjoy this new release.
It looks like the port has really begun to integrate like native MacOS X apps and seems to be getting some userland attention. It's also great to see they were able to release a new binary based on the most recent vintage of Wine. Jim White followed up on Sanjay's announcement to point out a few other things; not the least of which was a change in the Darwine website from SourceForge to OpenDarwin.org. Jim also provided a link to a recent interview:
Other Darwine news is that there was an interview with moi on osViews:
Text:
http://www.osviews.com/modules.php?op=modload&name=News&file=article&sid=2046
Broadcast:
http://www.osviews.com/modules.php?op=modload&name=News&file=article&sid=2047
Then Jim mentioned that work with integrating with the QEMU emulator was progressing:
And perhaps the most interesting news is that Darwine's lead developer Pierre d'Herbemont has sucessfully ported QEMU (fast X86 emulator) to Mac OS X:
QEMU on Mac OS X
by Pierre d'Herbemont
2004-07-04 at 11:07:13I am proud to announce that QEMU is now working on Mac OS X.
- Original announcement on the qemu-devel list
- Download the installer for Mac OS X
- Screenshots of QEMU running on Mac OS X
- The FreeOSZoo, to download ready-to-run images of QEMU virtual computers, pre-installed with a Free Operating System and a set of popular free software.
I would like to thanks Matt Reda for its collaboration.
Jim has an excellent explanation in the interview linked above about why QEMU integration is important:
osViews: Why aren't you just porting QEMU and then running Wine through the port of QEMU?
Jim White: QEMU is a processor... um, its a computer emulation... and so it runs an operating system. So, QEMU is a virtualization environment. So, its doing that job. It already runs Linux... under QEMU that just came to pass this last week or two. So Wine would run now under Linux... under QEMU.
But that's not what Darwine's about. Darwine is the idea that by incorporating the x86 emulation with the Wine environment, that we can run applications without running a separate operating system.
[....]
The reason that Darwine has a special case [which] could be interesting is... when you talk about emulation, then the next question people ask after, 'does it work' is 'well, how fast does it go?' So one of the reasons why Darwine... could in the long run be worth while is on that issue alone.
The opportunities for optimization are very very high with the Darwine approach because the problem that a general processor emulator has is its compilation window... Like QUMU compiles instructions... and compiles them into native code and runs that, but since it doesn't know things like 'what are code blocks' and this sort of thing... its very limited in what it can do there.... its window translating scheme.
But with Darwine, down the road... since we know things like, 'what are code blocks' things could could actually be natively translated. {This allows} a block of code to be compiled in native code and not have to use dynamic compilation at all. Performance could potentially get MUCH better than systems that use operating system virtualization.
Guess this must be the week of Darwine news.. another thread popped up
up about using Wine + QEMU on MacOS X. Now, normally a lot of discussion
like this takes place over on the Darwine mailing lists. Mike Hearn made
a comment that prompted Jim White to explain rude comments were exactly
the reason discussion of Darwine was not done on Wine mailing lists.
Alexandre, however, encouraged them to use Wine's mailing lists to bounce
ideas off other developers,
that's not the general attitude here, you guys are
very much welcome on wine-devel. So please don't hesitate to CC
wine-devel on anything that can be of interest, even if it's not
strictly a Wine issue; this makes it possible for us to catch
potential problems earlier on, and it helps us to get some context so
that when you submit a patch we have a better idea of the big picture
behind the patch.
That actually seemed to open the floor to some discussion that's not seen very often on the list. In this case it centered around getting Wine to use QEMU as a CPU-level emulator, but without running a separate operating system on top of it. Now, in the interview above Jim alludes to running Linux on QEMU running on MacOS X. Within that sandbox he could also run Wine. The "holy grail" seems to be bypassing the need for an operating system altogether and plugging Wine directly into QEMU. However, that's a lot easier said than done; I already have a headache just trying to wrap my head around it. I'm probably going to botch this whole summary, partly because I don't fully understand all the issues and partly because it seems like some discussion also drifted off-list.
The first issue that seemed to pop up involved endian issues. An x86 machine is considered a "little-endian" architecture and a PowerPC is considered "big-endian". Now, what that means is if you have something like an integer there is a corresponding binary string consisting of a bunch of 1's and 0's. Let's take a really simple example simplified to the point where the example is wrong, but the idea is right. Let's examine what happens to the number 65,534 when it's stored in memory. First, you can break 65534 down into a hexidecimal number FFFE, or in hex notation 0xFFFE. Now, the geeks among you (and judging from this audience it's probably most of you) will truly recognize that for what it is - two bytes of information. (For the sake of this discussion we won't get into the fact that the bytes 0xFFFE can be represented as binary bits; namely, 11111111 11111110.)
Up to this point the whole world of mathematics and computer science agrees on everything. Things begin to fall apart when we need to store these bytes in computer memory. Now, on both big-endian and little-endian architectures each memory location addresses 1 byte. How these memory locations are assigned differ though. On a big-endian machine (PPC) the most significant byte (in this case, 0xFF) is stored at a memory location with the lowest address, the next most significant byte (0xFE) is stored in the next address, etc. So if you were to look at a chunk of memory in sequential order you would just see FF FE stored there. If you move over to a little-endian machine it's different (not really more complicated.. just differently complicated). There we store the least significant byte in the first memory location. So if you were to look at that same sequence of memory on a little-endian machine you'd see "FE FF" stored there. What we mean by "look" is something like, grab a pointer to an address in memory and then keep indexing it until we've sucked in all the data we want. What got sucked in? Well.. as you can see.. it depends on the endianess. Now, we've used a primitive example of a 16-bit integer. Things get worse when you consider a 32-bit architecture and how something like a string of characters might get misinterpreted.
Now, what the Darwine guys want to do is take a Windows executable compiled for a little-endian machine (x86), run it through Wine, and then into QEMU to make sure the instructions are translated properly into big-endian PPC commands. Ouch.. makes.. head.. hurt.. Mike pointed out just one potential problem:
what I meant by that statement is that if you're running x86 Wine in QEMU you can't access the Carbon APIs directly. I think in the mode where you run foreign binaries QEMU will perform syscall conversion but obviously this is of limited use - the OS X APIs aren't accessible via syscalls.
One thing you could do I guess is expose each OS X API that you wanted as a new "fake" syscall and then use a customized version of QEMU that converted it into a Mac API call, ie marshal the calls in and out of the emulated environment. You'd still pay a performance penalty but that's unavoidable. I'm not sure how you'd deal with callbacks.
Jim White had a different idea to deal with that problem:
The ideal method would be to handle this with a trick loader that can link to PPC code which is escaped from the X86 emulation. We can automatically generate X86 DLLs for the PPC dynamic libraries (and hence the OS X APIs) which have little emulation escape thunks in them. There is zero performance penalty since the emulator can compile such escapes as direct native calls. Callbacks use a similar sort of little thunk which calls the emulator with a given X86 address.
I would really like to get away from having to write all the little wrappers to deal with the namespace problem. I believe we can solve that with a suitable naming convention as we began to do with the winegcc patch.
Alternatively we could go with with dynamic binding as proposed by Alexandre and have a syscall (or emulation escape) to call PPC code. This is probably the easiest to implement but leaves us with extra call overhead (although apparently this is something that is currently done in Wine and is quite small).
By using suitable macros we can make the source independent of which method is used.
The thing we don't get away from is that any code which calls OS X routines from X86 code *will* have to deal with endianess. There is overhead here, but we can be careful to use code that the emulator does a good job translating. Of course if there is substantial code (which would be unusual) it can simply be compiled as a PPC dynamic library.
As for swapping functions, the preceding discussion reiterates my belief that we should confine such fussy business to the relatively small bit of code that calls into OS X, will be relatively stable over the long haul, and over which we have full control. And as for the concern over having Wine be X86 and hence incurring emulation overhead, this would be one of the first bits of code which would be a candidate for having its emulator-compiled code cached.
Now, part of the reason I included that snippet of discussion is to outline some of the complex things that need to be dealt with. The actual thread put forth a roadmap of what the Darwine guys want to do that's much easier to understand:
Lofty goals? Perhaps. However, given the number of porting ideas that have fallen by the wayside and given how far Darwine has come.. maybe these guys can pull it off. Special thanks to Pierre d'Herbemont for reviewing this thread, no thanks to me for making a bunch of changes after that.
And finally in Darwine/MacOS X news, Rib Rdb announced a new Wine display driver to work with Quartz. On MacOS X, Quartz is the low-level 2D graphics rendering library upon which the Aqua desktop is built and apps integrating with it provide a common look and feel. Apps that don't integrate with it, such as native X applications, seem out of place. Currently Wine supports two main display drivers - one is for graphical apps using X (x11drv) and the other is for commandline apps (ttydrv.) There have been efforts in the past to add others, such as the SDL driver (see WWN issue #114), but there hasn't been much of a reason. Rib explained that there's still much work to do in order for the driver to be functional:
Attached is a very preliminary driver for native graphics on OS X. So far all it does is let you make windows. No events or drawing work yet (I was baffled by the GetDC function. It looked like the existing drivers only call windows functions, and I didn't see any way for setting the window's context.) To compile this requires a patch by Pierre d'Herbemont to deal with conflicting function names in windows and Carbon. He said he will send it here. Also, to get the windows to focus, you have to place the executable in an application bundle. I just took one I had sitting around, copied the wine binary over the executable in the bundle and ran it from the commandline like so:
/Users/ribrdb/Desktop/winemine.app/Contents/MacOS/CarbonTest winemine
Things are getting busy with work, so I'm not sure when I'll have more time to work on this.
Mike Hearn wondered if the effort was really worth it,
Given the complexity and amount of work done on the x11drv it
seems silly to duplicate all that effort. The x11drv is worked on by the
entire Wine development team right now, so any quartzdrv would always be
playing catchup. Well, if they want to spend their time doing that, fine
by me
Lionel Ulmer disagreed and thought it could be a valuable exercise:
Well, by 'the entire Wine development team' you mean Alexandre :-) ?
I, for one, am happy that people are using our driver model to write new drivers as it could serve as a proof of concept that our driver model is sound (think about GL / Cairo / 'DBI engine' kind of drivers).... Heck, if we only wanted to support one driver, why do all the job of doing a clean separation and not include X11 in Wine's GDI directly ?
Mike pointed out that there were other people besides Alexandre that had recently contributed to the X driver. He went on to explain that the driver model has already proven to be sound:
Well the driver model is lifted from Windows so I think it is sound. And as for why we use separate drivers, well, when the DLL separation is complete it means we can make a real Windows box display via X11 which is Just Plain Cool :)
It also allows for portability to operating systems that don't have X11. MacOS X does have X, in fact it's a modified XFree but with some proprietary bits added. So I'm thinking the added effort is not worth the gain. I'm not a Mac user though so who knows, maybe the extra native-ness is worth all the pain
Some discussion then moved over to irc and the debate was rehashed.
From time to time we point to the Wine Janitorial page and Fun Projects list as a good way for aspiring developers to get involved. hint, hint One of those janitorial tasks is to prevent Unicode functions from using ASCII functions to do their dirty work.. and in the process perform a lossy conversion. Well, that list is definitely shrinking thanks to a lot of individuals and some of the things left on the list seem harder than others to finish. We generated the list of cross-calls (or "Wide to ASCII" aka "W->A") using a little tool Patrik Stridvall wrote years ago called winapi_check. Vincent Béron decided to point out this week that.. oops.. we're probably wrong about how complete that list is. In fact, it's not like we're a little wrong.. we seem to have missed a bunch:
I played a bit with smatch and Wine (after reading http://people.redhat.com/mstefani/wine/smatch/), to try to get over what I mentionned in my last email (regarding winapi_check and cross-calls in parenthesis).
I attach the smatch script to generate the results I have. It's still not perfect (the listview.c references are wrong, for example), but as it's tied to gcc it can detect some calls which otherwise would not be caught. For example, the calls to GetWindowLongA in header.c are in a macro.
Not all of them are bugs: caution is needed when checking, although reducing the number of false positive would likely be welcome (either by modifying the script or patches).
The list was quite lengthy with 286 possible cases. Compare that to the 54 we list on the janitorial page and it looks as we haven't even started. Rob Shearman pointed out the latest CVS might contain less since a very recent patch corrected calls to GetWindowLongA. Vincent reran the test and found things improved! We dropped to 270! (Sorry, I just got done watching The Daily Show, so pretend Jon Stewart is reading this paragraph out loud.) Rolf Kalbermatter did point out that calls in the shell related DLL's might not be a problem. In a different thread, Vincent described how this has been overlooked:
Also take note (Francois? Any perl hacker?) that the method used to find the cross-calls (winapi_check) is not 100% proof. There are some valid C statements which contain a cross-call which are not found. Look at dlls/lzexpand/lzexpand_main.c GetExpandedNameW:
INT WINAPI GetExpandedNameW( LPWSTR in, LPWSTR out )
{
INT ret;
DWORD len = WideCharToMultiByte( CP_ACP, 0, in, -1, NULL, 0, NULL, NULL );
char *xin = HeapAlloc( GetProcessHeap(), 0, len );
char *xout = HeapAlloc( GetProcessHeap(), 0, len+3 );
WideCharToMultiByte( CP_ACP, 0, in, -1, xin, len, NULL, NULL );
if ((ret = GetExpandedNameA( xin, xout )) > 0)
MultiByteToWideChar( CP_ACP, 0, xout, -1, out, strlenW(in)+4 );
HeapFree( GetProcessHeap(), 0, xin );
HeapFree( GetProcessHeap(), 0, xout );
return ret;
}The A call is between parenthesis and winapi_check can't find it. winapi_check would need to be modified to find those, as I'm sure that's not the only one in the whole tree.
Everyone promptly ignored the thread and forgot they had ever read it.
Last week we covered Shachar Shemesh's inquiry into using epoll() instead of poll() (see WWN #236.) Well, Shachar took a stab at the idea presented and returned this week with a patch:
Attached is a non-perfect patch for review. This is a migration of the wineserver to use epoll instead of poll (if it's available).
current known issue with this patch:
- Will not compile if HAVE_SYS_EPOLL_H is not 1 (i.e. - won't compile if epoll not available at compile time)
- Segfaults on wine exit.
- Lots of debug asserts.
Comments welcome.
Mike McCormack did offer some suggestions:
If you're going to use syscalls instead of epoll_create etc, then you don't need to check whether epoll functions exist in glibc, since you're not using them anyway.
In any case, seeing that getting your patch accepted will take some effort, it's probably a good idea to use the glibc version functions first, and deal with the problem of missing epoll functions in older glibcs in a separate patch.
Shachar ran some benchmarks and posted the results:
The patch is not yet ready for commit, but I do have preliminary benchmarks: The attached program (compiled as winelib) was used. ulimit -n was raised to 10240.
With the current wine code:
real 0m41.076s
user 0m5.070s
sys 0m7.722sthe main wine process was taking 10% CPU time, and wineserver was taking over 60% cpu time. load average was over 2
With my preliminary path:
real 0m20.985s
user 0m5.316s
sys 0m11.421smain wine process was still taking 10%, but so was wineserver. load average was about 1.5.
We can see that there is a significant drop in actual execution time, even though there is an INCREASE in user+system processing time. I believe this is not a fluke (results are pretty consistent), but rather that the poll behavior was taking a huge toll on the system in areas where it wasn't attributed towards wine, even though it was wine related. I would take the real time measurement as the indicative one.
Ideas for better benchmarks would be greatly appreciated.