Wine Traffic #36 For 27�Mar�2000

By Eric Pouech

Table Of Contents

Introduction

This is the 36th release of the Wine's kernel cousin publication. It's main goal is to distribute widely what's going on around Wine (the Un*x windows emulator).

Wine 20000326 has been released. Announce reads:

Mailing List Stats For This Week

We looked at 129 posts in 650K.

There were 39 different contributors. 24 posted more than once. 21 posted last week too.

The top posters of the week were:

1. Wine's resource compiler

�Archive Link: "PrgWin95: Extern and #pragma/#line/#error support"

People: Bertho Stultiens,�Dimitrie Paun,�Francois Gouget,�,�Uwe Bonnes,�Patrik Stridvall,�Marcus Meissner

While continuing his effort to test the Petzold's Windows Programming samples (see this issue (wn20000313_34.html#0) ), Francois Gouget sent a patch to correct a few issues in wrc - the Wine's resource compiler -. Those fixes mainly dealt with some uncorrectly supported preprocessor directives (like #error, #line or #pragma)

Bertho Stultiens (the author of wrc) explained:
The problem is that wrc has a semantical preprocessor instead of a lexical preprocessor. There are quite a few other constructs that also fail in wrc. I have been working on a new preprocessor for the past two months (and it is working), but it is not yet completely done. Main problem is that I need to rewrite the macro-expansion code for nested macros (to prevent functional recursion) and definitely speed it up because it takes 3.5 seconds to compile some files (oh well, it does 20000 lines per second though; almost as fast as gcc:-). My coming changes also correct the other 2 major issues: filenames and usertype resources.

Dimi Paun started then a long thread by asking whether if it was worth creating Wine's own preprocessor embedded inside wrc, instead of using gcc as a preprocessor.

Marcus Meissner and Patrik Stridvall pointed out that gcc's preprocessor wasn't available on all platforms, but that configure could look at any decent preprocessor, like plain cpp. Uwe Bonnes and Patrik Stridvall also noted some discrepancies between gcc's preprocessor and cpp - like some different command line options handling, but also, a different handling of the "\r\n" sequence as a resource file terminator -, but Patrik was confident that those "features" could be circumvented. He demonstrated it by sending a patch which allowed to compile resource files on Solaris (which was working with both cpp and gcc as preprocessors).

Some people (like Uwe Bonnes and Patrik Stridvall) remarked that using plumbing (feeding wrc's input with a preprocessor output) ended up with poor error reporting to the end user.

But Dimi went further:
the point is that rewriting and reinvention of the wheel for the single purpose that it is more convenient to have it included in wine is a bit silly. We require a C compiler (for now we require gcc with is OK IMO), we require a make program (don't know if gmake is required though), perl, yacc, lex, etc. It would be simply silly to reimplement all this programs simply because some systems don't have them. They are free on the net, download them.
, and asked Bertho if he agreed to strip the preprocessor part from wrc.

Bertho had a rather different opinion:
Well, a good question. Stripping the preprocessor could be an advantage if it was solely used with wine. But, this is not the case. There have been other uses of wrc which require the preprocessor built in.

--

To make it short [ahum;-];

I intended wrc to be versatile enough to be used both within wine (because we lacked a real resource-compiler back then) and powerful enough to be used within other projects which have nothing to do with wine, but want to use windows-resources (for whatever reason). The wine-license made this a perfect option and I kinda liked it too.

So, keeping that in mind, the whole discussion is a bit artificial and not changing my work nor my attitude in putting a real preprocessor into wrc's frontend. This will be entirely optional for use and a command-line option can be used to bypass it.

Even if Alexandre would want to reject the upcoming patches (which I do not think he will), then it would still not change my attitude because the changes will be published by other means. Solely because I feel that wrc should be capable of compiling resources in a stand-alone manner and no one should be required more than wrc to get resources compiled.

and he went on explaining with more details the changes underway
Yes, wrc will be faster. That is actually the primary technical reason for me to split wrc into a preprocessor/compiler combination. The current mix (a bad hack) was there to help out for the time being. The extra rules in the scanner and parser make it awfully complex to change anything of substance in any part. This is why user-resources and filenames are wrong. I could not get this put into the current design without making an even worse hack.

Wrc's backend will be cleaned up so it will understand enough C and C++ constructs so that they can be filtered out and for the rest it will be a 'clean' and 'compatible' compiler. All lexical substitutions (i.e. the preprocessor) will be done in the frontend. Whether you choose to use it is not my primary concern.

Then for reinventing the wheel. Yes, many have done it before. But, do you know a full-working preprocessor in the public domain that can be put into wrc without problems? I don't. The rest I know is GPL.

A good bunch of posters approved Bertho's approach, and Francois Gouget shed some more light
I would like to point out that even if you use gcc as the pre-processor, as I did, wrc must still be aware of some directives like '# <line> <file> and '#pragma'. I seemed to me that this was not always clear in the messages.

We need to handle '# <line> <file>

We need to support the '#pragma'

As a conclusion, wrc will keep its integrated preprocessor (keep posted for the day where Bertho releases the new wrc version).

2. Wine and XFree 4.0

�Archive Link: "Wine problem with XFree 4.0"

People: Jeremy White,�Ove Kaaven,�Alexandre Julliard,�Ulrich Weigand,�Andrew Lewycky,�,�Ove K�ven

Following post from last week (wn20000320_35.html#0) regarding the incompatibilities between Wine and XFree 4.0, Jeremy White reopened an old discussion on pthread support in Wine:
I tried to understand exactly what the problem was if we linked Wine with X 4.0, libpthread, and OpenGL. Really, the core issue is why we can't link in libpthread.

To rehash what is already known:

After a scan of the glibc 2.1 source code (and no experimental evidence whatsoever), I found only the following two problems:
  1. malloc and free will fail badly: I see two issues. First, the code for malloc tests to see if __pthread_initialize is available. If so, it invokes it. I believe that __pthread_initialize will actually run correctly, and start the pthread_manager. I have no proof of this. Second, as malloc and free manipulate internal structures, they rely on macros from malloc-t.h. The ones I inspected were mutex_init, mutex_lock, et. all. Basically, if libpthread is linked in, and the weak symbols for __pthread_mutex_xxx are available, then malloc will use them, otherwise they turn into NOPs (and malloc is presumably running in a single thread environment).

    However, in our case, the code for pthread_mutex_lock will be invoked. That code, in turn, grabs the pthread_self descriptor from the GS register. Of course, we're not really a pthread and boom down she goes.
  2. errno will fail: Again, same basic issue.

However, I found no further instances where glibc examines the weak aliasing on the pthread functions. Of course, I did not look at the source code for X 4.0 or any other library to see in what ways it depends on the weak aliasing of pthreads.

As I understand Ove's pthread patch, his patch resolves this problem. I don't have his patch in front of me, but I believe he did this by registering Wine threads with libpthread (setting up the GS register), and by correspondingly trapping the creation of threads from libpthread and registering them with Wine.

Jeremy also asked why Ove's patch never made its way inside the Wine's CVS tree.

Ove K�ven gave some more indication on his work:
My original pthread patch defined only the __pthread_xxx routines that glibc used, so that libc would use them instead of assuming a single-thread environment, since wine IS multithreaded.

pthread was normally never linked in before XFree86 4.0 appeared, so when it did, I had to update the patch to also define standard pthread symbols to override pthread's own (libwine is before pthread-using X11 libs on the link command line...)

The OpenGL and MT parts of XFree86 4.0 uses pthreads directly, without going through any weak aliases. In other words, they explicitly link it in.

I just implemented the pthread primitives on top of Win32 synchronization primitives. For example, pthread_mutex_init() is a wrapper for InitializeCriticalSection(), pthread_mutex_lock() is a wrapper for EnterCriticalSection(), etc...
and regarding the rejection cause:
it is currently quite glibc-specific (Alexandre calls in un-portable, although I wouldn't myself consider it impossible to port to other systems, like FreeBSD), and Alexandre really wants to see a portable fix for libc threadsafety before slipping in system-specific workarounds that ~90% of the developers would consider "good enough" (and the rest wouldn't know how to fix).

Alexandre Julliard jumped in:
the theory is that if it's broken for everybody, it's more likely to get fixed than if it only breaks on "exotic" platforms. Since it still hasn't been fixed it seems I'll have to revise my theory...

I think at this point we have two possibilities: one is to provide our own complete libpthread implementation to override the libc one, i.e. the extended version of Ove's patch.

The other solution is to try to use the existing libpthread to create our threads; if we can work around the %gs register problem, this would have the advantage of better integrating with the native libraries (for instance gdb would then be able to understand what's going on with threads). This is IMO a more elegant solution, but the big question is: can it be made to work?

Ulrich Weigand raised some more problematic points:

Actually, I'm not sure whether any version of libpthread uses both of those constructs at the same time (as both are used to determine the current thread), but I've seen (different) versions that use either one.

The first item fools Wine's manipulation of the LDT (Wine as libpthread thinks he's the only one tweaking the LDT entries), and puts a hard constraint (%gs register shall not be touched, which is something emulated Windows's program don't ensure).

One possible solution to directly use pthreads would be to:
One possible solution to remove the restrictions on Win32 apps would be to treat Win32 apps the same way as Win16 apps, using explicit transition thunks. That way, we could switch from the pthreads stack to the Wine stack, and could save/restore %gs as well ... Furthermore, this approach would also allow to use compilers that don't support stdcall, and it might simplify a processor emulator version of Wine. On the other hand, there's quite a bit of overhead on every transition :-/

Alex Korobka also pointed out that pthread was missing the suspend/resume feature needed to implement the ThreadSuspend(), ThreadResume() API calls.

Andrew Lewycky reported:
I have a patch that runs wine on pthreads in glibc 2.1, that works for 32-bit programs at least. I didn't bother with the LDT or %gs stuff since linuxthreads-0.8 does not use it, due to bugs in the 2.0 kernels. (So it may come back some day.)
, which gave some good hope on being able to use pthread, but also brings lots of issues, as Ulrich already pointed out: the different libpthread version use different mechanisms to identify a thread (%gs, stack...), opening various concerns from all the versions.

Ulrich and Andrew then discussed some other concerns with libpthread while executing the exception handlers (which also have issues with identifying the current thread).

As of today, no global solution has been reached so far.

Sharon And Joy

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.