Wine Traffic #4 For 18 Jul 1999
Table Of Contents
Introduction
This is the fourth release of the experimental Wine's kernel cousin publication.
It's main goal is to distribute widely what's going on around Wine (the Un*x
windows emulator).
Another publisher seems to get interested in Wine.
Francis Beaudet (from Macadamian) announced that they are planning to release
a win95 look for the file-open dialogs.
I'll be on vacations for the next couple of weeks, so there won't be any new issues
until I get back.
1.
Crtdll Implementation
Archive Link: "Crtdll Implementation"
People:
Patrik Stridvall, Marcus Meissner, Patrik Stridval, Albert den Haan, , ReactOS
Patrik Stridvall sent out a RFC about the implementation of the CRTDLL and
the way to use functions (and function prototypes) defined both by Windows
in CRTDLL and the underlying Unix OS libc.
Editor's note: for clarity, the reader must not be confused by
native. Here, it does not refer to Windows's libraries being used by
Wine, but rather the host OS libraries being used by Wine.
What Patrik wants to achieve is something like:
CRTDLL calls libc
Other parts of Wine calls CRTDLL when ever possible.
We needn't do 2 if we doesn't want to.
To provide at least the step 1/, Patrick proposed:
On thing that any solution must do is to add the CRTDLL include files,
ctype.h, stdio.h, stdlib.h, string.h and time.h for example.
One problem is that both #include <stdio.h> and #include "stdio.h"
will include CRTDLL stdio.h. Not good? However GNU C has a option
-I- that fixes this problem. Only a small patch include-separation.diff
is needed to make Wine support that.
Now #include <stdio.h> include libc stdio.h and
#include "stdio.h" include CRTDLL stdio.h. This makes Wine dependent
on GNU C, but at first it already is and secondly it will only be
during the transition process as I will show later.
The second problem is that since CRTDLL will call libc it
must include both libc and CRTDLL include files.
That is quite a problem as it will cause massive namespace collision.
However I think I have found an hopefully acceptable solution.
Using preprocessor "magic" it seems to be possible to rename the name
spaces to avoid collsions. Instead of doing
#include <stdio.h> /* libc */
#include "stdio.h> /* CRTDLL */
you do
#include "native/stdio.h" /* libc */
#include "internal/stdio.h" /* CRTDLL */
Please look at the files include/internal/*.h and include/native/*.h,
and native/*.c, to see how it is done.
The files include/native/*.h and native/*.c is generated using the
script tools/make_native using the files native/*.spec, perhaps
the files internal/*.h should be generated as well.
Patrik noted that with thr preprocess "magic"
... it might be possible to avoid generating the files during the
compile phase and they might be portable between platforms and in the CVS
and only regenerated when the interface changes, that is not to often once
things stabilize.
and that this scheme also
requires that the supported platform has som sort of ELF-like dlopen.
Marcus Meissner quite not agree and liked better the other way round
(not using CRTDLL but rather libc for all the APIs). But, he
acknoledged the use of it for some specific functions, especially
due to lack of threadsafeness of some of the FILE*
and memory allocation functions.
Patrik Stridval then precised some points:
- If _every_ CRTDLL function are to be implemented in a _100%_ Windows
compatible way, primarly for the emulator, but also for the library,
it is essential that we have some way to enable inclution of both libc
and CRTDLL includes in order to access the libc
structures/macros/functions that might not be exactly the same as the
CRTDLL ones (alignment, order of fields, missing/extended fields) etc.
There might also be thread-safe problem like incomptible or misssing
thread safe function. This way we might have a thread-safe CRTDLL on
top of a non-thread-safe libc.
- The Wine code that uses CRTDLL instead, needn't check for missing or
diffrently behaving libc functions since that will be checked/fixed at
CRTDLL level.
- It makes it possible to take this one step further a CRTDLL that only
calls KERNEL32 and/or NTDLL. That will be useful in ther project like
ReactOS that are in themselves standalone OS:es and do not have a native
libc. ReactOS for example does have a KERNEL32 and/or NTDLL (I don't
remember which) and their CRTDLL is quite rudimentary last time I
checked. They might be other NT clones with similar problems. It is IMO
good design and might increase the uses of Wine and indirectly bring
more programmers to help.
- As I hinted in 3 this scheme is extensible beyond libc/CRTDLL. It might
provide as way to include both native and internal include files that has
namespace collisions and a way to link files with namespace collisions.
- All this begs the question. Why use libc at all? Why not in the future do
direct syscalls since that will enable use of specific things that the
native kernel is particullary good at. This will be a lot more work of
course, but why not it will be faster. :-)
- Win64: In Win64, long is 32 bit, and in 64-bit Unix it is 64-bit, this
might cause problems, since long in native libc is 64-bit and if we use a
compiler that generates 32-bit longs, linking with native directly libc
might be a problem, if we use 64-bit long we will get other problems,
especially for WineLib. Out of the ashes into the fire. :-)
Patrik proposed also a stepped approach:
- Making CRTDLL a 100% Windows compatible libc with every function
implemented for use with the emulator. My preprocessor "magic" scheme might
help doing that.
- Making WineLib applications use CRTDLL instead of libc.
To do this we need some sort of elf dlopen mechanism and/or possible
other "preprocessor" magic, because of namespace problems.
- Making Wine call CRTDLL for portabillity reasons.
- Making a CRTDLL that calls KERNEL32/NTDLL instead of libc.
and asked:
- Where should we stop?
- Is my scheme robust/portable enough to do this?
- Have I missed something?
- What do you think?
Patrik also took a look at integrating ReactOS CRTDLL into Wine. His first
conclusions are that many of the commonly used
functions seem to be implemented using KERNEL32 calls which is exactly what
we (or at least I) want.
, but licence is an issue: some files claim to be BSD, other files claim to be
LGPL, and there are files indirectly claiming to be GPL by refering to a
file COPYING in the topmost directory.
Albert den Haan from Corel explained in details the strategies currently
used to help porting their applications:
We have been concerned with what does the compiler link to
when the Wine headers are used on top of glibc headers when compiling
Corel's Win32 applications. And in some cases we would rather have
the option to call glibc functions instead of the CRTDLL ones ---
especially for services like dynamic memory allocation.
Over the past few months we have had to map many of the stdlib and stdio
functions to their CRTDLL implementations via macros in crtdll.h and
#including crtdll.h from other strategic headers (windef.h catches most
of the cases).
This allows most of the source stuff we have tried to find the
declarations they need. It also allows us to experiment in choosing the
combinations of implementations for various services by editing one
header file and recompiling.
Albert wishes to keep the configurability I want
to tune source ported programs
.
Patrik and Albert also agreed that having all the Wine code relying on
a single portable CRTDLL will help porting Wine from OS to OS.
Nevertheless, Patrick points that Albert has different objectives: You need to keep in mind that I am looking for the
best possible _technical_ solution in the long run, you (Corel) are
looking for the best possible solution for your immediate problems.
However there is a large overlap between us where compromises
are possible. Fast and small footprint is _very_ important for me too,
but I prefer good before soon.
2.
CRTDLL implementation
Archive Link: "CRTDLL implementation"
People:
Albert den Haan, Patrick Stridval, Patrik Stridval, Ove Kåven, Alexandre Julliard,
Albert den Haan also discussed about memory allocation under Wine:
-
In short: The current web of Wine memory allocations is an
inconsistent mess that will have to be cleaned up eventually; especially
for those of us interested in source code ports.
-
Why I did this: I noticed that some of the resources returned
by Wine API implementations were allocated using HEAP_* functions, others
using malloc, and yet others using xmalloc. Then our source compiled
libraries would not use the correct freeing mechanism to release the
resource again and there would be a lot of memory leakage with warnings
of this nasty situation.
-
What I did: A few weeks ago I tried the essentially the same
operation on the xmalloc and xfree calls on the Corel Wine tree. This stuff
has been soaking for a while here and I have to say that I am frustrated with
the results.
I hacked up crtdll.h to use Wine's HEAP* allocators as much as possible.
This involved providing maps using preprocessor macros for the ?alloc
functions and free.
This was not enough. A lot of dynamic memory in wine is allocated with
xmalloc and xcalloc because the coder did not want to specifically
handle an error condition. That memory is (usually :) properly released
with free; but I had mapped that to CRTDLL_free above and I saw a _lot_
more error messages when CRTDLL_free and its HEAP_* primatives were
properly upset when asked to free memory it didn't allocate.
So I mapped the x?alloc functions, and reimplemented them in terms of
the HEAP_functions (which was gratifyingly easy), and broke wine's
initialization into a million (O.K. a lot of) pieces. xmalloc is used
in a _lot_ of places. Some of them are executed before the Wine heap
system is up and running with the usual "undefined" results.
Analysis:
Wine's current implementation has been somewhat indiscriminate in its
dynamic memory use. The current mix of host OS allocation and HEAP
allocation may not affect programs running under allocation (that may or
may not exit cleanly anyway) but source ported programs seem to be
really tweaking the problem cases.
This stuff is really endemic. It will take some time to find the
mismatches and resolve them. This includes making sure the free's match
the "mallocs" and that the wine implementation code still uses the most
efficient memory management for each task at hand.
For example the Wine server
must use the host OS's allocation system
because it cannot trust (or access?) the client's heaps. I suspect that
the Wine server will be used to
implement shared memory eventually and
we are trying to plan for the future, right? Right?
Other portions of the Wine implementation code could use either the host
OS heap or the Wine HEAP implementation. I'm no expert on this, but I
suspect the utilities that cross address spaces would be hard either way
but I hope we can avoid the additional overhead of using the wine heap
wherever possible!
Stuff that escapes Wine to be twiddled by the client applications are
problematical. If the client expects Win32 memory management then it
must be given the context it expects. For the apps that are less picky
then Wine implementors can make a choice, but they must be consistent.
Right now I don't know how consistent Wine's memory management strategy
is, but what I have seen is both heartening (no disasterous stuff has
shown up) and scary (there are a lot of xmalloc calls out there).
Proposals
- Move the current xmalloc.h file from include/xmalloc.h to
include/wine/xmalloc.h. Provide a short term stub include/xmalloc.h for
the short term) This will provide finer control over which flavour of
xmalloc you want to use in a given file.
- Rename the Wine xmalloc implementations to WINE_xmalloc
(HEAP_xmalloc?) and such. (see 1.)
- Start churning through the Wine implementations and start
discriminating calls to xmalloc as outlined above. I know which ones I
want to fix! :)
- Experiment with using the HEAP memory allocation stuff selectively.
If profiling shows that some operations are faster using the host O.S's
dynamica allocation, use it! If testing reveals that some apps require
Win32 memory allocation then that should be configurable (runtime vs
compiled? We'll debate that later)
Patrik Stridval went a step further: do we
need the special behavior of xmalloc and friends at all. Obviously, we need
a way to choose between the heap malloc and the native libc malloc, but does
are there any code _really_ needs the special behavior that xmalloc has? Any
code that does should be fixed instead IMHO.
He also added that The native malloc is probably
faster on most platforms since it is one of the most common libc calls
they are heavily optimized on most (all?) platforms. HeapAlloc is AFAIK
not very optimized so I wouldn't be suprised if it is _much_ slower.
Albert noted that changing xmalloc into *malloc could be not so easy because
xmalloc does not return if the
allocation fails, it exit() the thread immediately
and will be a substantial effort in itself. Many of these
allocations happen in situations (e.g. early in wine's initialization) when
any failure is a good reason to take wine down so the user can provide more
resources. Granted, there are cases where fancier error recovery would be a
good idea but let the people who need that stuff code it when the need
it.
Ove Kåven noticed that "host OS heap" (malloc) is
thread-unsafe under Wine.
He also reminded his work in the debugger
(changing malloc with HeapAlloc) which produced very slow results (at least
while loading symbols).
Albert detailed some of the allocation issues he run into:
- X does not like something about Wine allocated memory (This may be my
fault),
- the g++ new and delete operators are fragile with the prototype
implementation I tried,
- The LIBRES resources hacks are executed before Wine's HEAP
implementation is initialized. (obvious once you look at it, this will
go away soon, err someday, "How are those elf-dlls coming Ulrich?
Bertho?" :)
- The registry is read in before the Wine HEAP is initialized. And this
service is not under the control of the server. Yet.
Alexandre gave his point of view on the CRTDLL issue:
-
CRTDLL: Our CRTDLL should only implement functions that cannot be
used straight from libc. There is absolutely no reason to duplicate
functions like strcpy() when the libc one can be called directly.
Trying to make Wine portable to a platform that doesn't even
have a libc is a complete waste of time IMNSHO.
We need a scheme to reroute libc calls in Winelib applications to our
CRTDLL implementation; this can be done at compile-time with macros,
or at link-time with ELF tricks. We probably want to implement both
possibilities and pick the more appropriate one depending on the
current platform capabilities. In either case, it should not be
necessary to make a single change to the Winelib app source code.
The extent to which CRTDLL redefines libc calls can be different for
different platforms; for instance on a platform where the libc wchar_t
is 16-bit (or where we don't care about binary compatibility), we can
call straight to libc for all wide-char functions. The principle is
that we should use as much of the native libc as possible, as this
will be better for performance and interoperability with other
applications.
Ove's pthread hack is a good way to make more libc functions useable
directly; it will probably be included as an option someday but I'd
like to have a more generic solution in place first.
-
Include files: We need to provide our own set of the standard
include files (stdio, stdlib, etc.) at least for definitions that need to
be different from the native libc. It would be nice to have a scheme that
allows redefining only things that need it, so that we don't have to
recreate a complete set of headers. Albert's patch seems a reasonable
approach to solve this problem.
An issue is that we need to have the higher level parts of Wine use
our own standard headers, while the Wine core (CRTDLL, KERNEL, etc.)
needs to use the libc headers. I'm afraid that doing this with
#include <xx> vs. #include "xx" is very error-prone.
-
Memory allocation: Inside Wine, libc malloc() and friends
should never be used directly. All memory allocations should ultimately
go through HeapAlloc(). It may someday be possible to have HeapAlloc
simply wrap a critical section around libc malloc() for better performance
(at least in the case of the per-process heap; the shared system heap
obviously cannot be accessed with malloc at all).
The server is an exception because it is a separate application and
doesn't use threads, so it can make use of the standard libc (and it
doesn't have access to any of the Windows APIs anyway).
Existing malloc() calls in higher-level DLLs can be left alone, as
they will be rerouted to the CRTDLL malloc() once we get this feature
in place. malloc() calls in the core should probably be replaced by
HeapAlloc. xmalloc() should be removed completely IMO, it's not used
heavily anyway, apart maybe in the registry code.
Patrik didn't think that tunneling through CRTDLL the existing libc
functions is completly doable, especially when dealing with 64 bit
Unix platforms providing a 32 bit CRTDLL.
But, he reexplained how his patch could help (see also previous topic):
The patch tries to solve the following problems.
- Making it possible to include both libc and CRTDLL include files
in same file in order to easier implement CRTDLL using libc.
- Making it possible to gradually make all non core Wine files
use CRTDLL instead of libc.
- Making is possible to for WineLib apps to use CRTDLL instead of libc.
- Making it possible for WineLib apps to choose between using
the CRTDLL or the libc variant of a funtion.
The current patch doesn't support 4 yet, but it is easy to add support for
it.
It introduced two new include directories include/internal and
include/native,
if you want the internal (mainly CRTDLL) version of a file you do
#include "internal/stdio.h"
and if you want the native version you do
#include "native/stdio.h"
The files can be made work in two different modes depend on the current
configuration.
- [default] internal/stdio.h renames printf to CRTDLL_printf
to avoid namespace collision with libc printf and native/stdio.h renames
NATIVE_Symbol_printf to printf.
- internal/stdio.h does no renaming at all and native/stdio.h makes
NATIVE_Symbol_printf to a function pointer that dlopen("libc.so", "printf")
is assigned in the initialization code.
(1) works but, since the current implementation does nothing.
(2) doesn't work, since to work 100% it requires that all non core libc
calls are routed through CRTDLL. Currently it doesn't even link.
The idea is use mode (1) while gradually transforming the code to support
(2), and when (2) works we make (2) the default and perhaps even remove the
support for (1) but that can be discussed in the future.
To support this we unforunately has to temporary during the transition from
(1) to (2) used the -I- option (GCC specific) so the correct headers is included.
Now we can fix the files on by one by changing for example
#include <stdio.h>
to
#include "internal/stdio.h"
and fixing the compile errors and in some cases bugs due to
CRTDLL libc differences.
When (2) works and if we decide to dump support for (1)
remove the -I- option and just rename all #include "internal/stdio.h"
to #include <stdio.h> since the internal stdio.h has priority over
the native stdio.h. Of course we have to rename the #include <stdio.h>
include in native/stdio.h to #include "/usr/include/stdio.h" but it doesn't
matter since it is a generated files anyway and thus we can have configure
easily change the path on system that the libc include is not /usr/include.
No final decision has been made yet to integrate either Patrik's or Albert's
patches.
3.
Binary resources and CVS
Archive Link: "Binary resources and CVS"
People:
Francis Beaudet, Ulrich Weigand, Patrik Stridval, Bertho Stultiens, Pavel Roskin, , Ove Kåven
Francis Beaudet asks how I would go about
submitting a BMP (or any binary file) to the CVS tree.
Ulrich Weigand proposed to distribute
bitmaps in XPM format (which is source code), and either use
tools (like netpbm) to convert them to BMP by Makefile rules,
or maybe even teach wrc to load bitmaps from XPM format directly
(and convert them to BMP in the binary resource format)?
Patrik Stridval disliked the use of netpbm for porting
issues. He'd rather see wrc convert XPM to BMP.
Bertho Stultiens agreed that wrc
should handle xpm format automagically.
, and asks for a
small and easy library to do it (out of ImageMagic which is
too large). Ulrich Weigand proposed libXpm, but noted that it would
introduce dependency on libX11 itself as well.
Ove Kåven rather proposed to embed bitmaps as hex codes as
Borland's Resource Workshop does. Bertho Stultiens confirmed that
wrc knows about this syntax.
Pavel Roskin reminds some option from CVS:
4.
Wine clipboard enhancement plan
Archive Link: "Wine clipboard enhancement plan"
People:
Noel Borthwick, Gérard Patel, Ulrich Weigand, Pavel Roskin, Albert den Haan, Alexandre Julliard,
Noel Borthwick he's looking into making some changes
to the windows clipboard code in order to allow the wine clipboard to
integrate better with the X clipboard.
.
He proposed (for comments):
There are two X clipboard selection mechanisms which seem to be relatively
widely used:
- The X selection content is specified through the built in XA_PRIMARY
selection atom in X. For example if you select something in an X terminal
window, the terminal window takes ownership of the selection by calling
XSetSelectionOwner() and specifying the XA_PRIMARY selection atom. When a
client wants to paste this selection it calls XConvertSelection again
specifying XA_PRIMARY and the target data format atom. This then triggers an
XSelectionRequestEvent for the specified format.
- The X clipboard content is specified through the CLIPBOARD selection
atom. The CLIPBOARD atom does not appear to be predefined as part of the
XLib standard, but it is used by many programs for example Netscape as well
as the XClipboard clipboard viewer. (XA_CLIPBOARD is defined in an X utility
header in X11/include/X11/Xmu/Atoms.h) If you select something in a terminal
window the content won't show up in XClipboard. However if you copy
something using Edit|Copy in Netscape you will see it in XClipboard (via the
CLIPBOARD atom) as well as you can paste it to an terminal (via the or
XA_PRIMARY atom).
This is the current scenario with an app running under Wine:
Currently the clipboard in wine only responds to the first mechanism. If you
select and copy text to the Wine clipboard, Wine takes over ownership of
only the XA_PRIMARY selection atom but does not deal with the CLIPBOARD
atom. As a result of this you cannot paste a selection from Wine into
another clipboard aware Linux app or see the selected data in XClipboard for
example. One should be able to paste Wine's selection into a terminal
window. However this seems to be broken. (GlobalSize fails in
GetClipboardData in HeapValidateHeap - block xxx is not inside heap)
The converse works partially. You can select something in an X terminal and
paste it into Wine using XA_PRIMARY. However if you copy something into the
clipboard using the CLIPBOARD atom from a clipboard aware app like Netscape,
Wine won't know about this selection since it doesnt yet know about this
atom.
Proposed changes to Wine's clipboard implementation:
- Whenever something is copied into the clipboard via SetClipboardData, in
addition to taking ownership of the XA_PRIMARY selection atom also take
ownership of the CLIPBOARD atom (XA_CLIPBOARD).
(This would be done in X11DRV_CLIPBOARD_SetData) This will wire the
XSelectionRequest event to the CLIPBOARD atom as well, allowing us to
process it in EVENT_SelectionRequest.
- Whenever GetClipboardData is called, always get the selection from the X
clipboard first, by calling XConvertSelection instead of checking in Wine's
internal clipboard. This is the sequence of events:
- First check the CLIPBOARD atom. If the format is available there use it.
- If nobody owns the CLIPBOARD atom try the XA_PRIMARY atom.
The advantage of checking the X clipboard is that if an external app to wine
has placed data on the clipboard we can retrieve that. If Wine itself is the
owner of the clipboard selection this still works, since our
EVENT_SelectionRequest will handle this request.
The Wine GetClipboardData clipboard code which currently looks into the
internal clipboard(via CLIPBOARD_RenderFormat) should NOW be invoked from
EVENT_SelectionRequest. Effectively this makes Wine's internal clipboard
behave like a cache which is accessed in response to an XSelectionRequest.
Also see point 5 below.
- Make all Windows clipboard formats route through this X-clipboard
mechanism. i.e. we should define X atoms for each all registered windows
clipboard formats as well as application defined formats.
RegisterClipboardFormat should be responsible for passing on requests to
create these atoms to the clipboard driver. The atom names could be the
format names prefixed with something like WCF_ (Wine Clipbord Format)
- Wine's EVENT_SelectionRequest should check the "target" field in
XSelectionRequestEvent and handle the target atom name as follows:
- If the target atom is a predefined target type such as XA_STRING or
XA_PIXMAP or XA_BITMAP(?) we can map these into native Windows clipboard
formats CF_TEXT or CF_BITMAP respectively and convert the clipboard content
if available, into an X compatible format. Bitmaps could be converted to
Pixmaps for example. Maybe would could do something intelligent with some
other native X target types as well.
- If the target atom is a Wine registered clipboard target type (prefixed by
WCF_) we translate the name to the windows format name (stripping WCF_) and
retrieve the appropriate format from the windows clipboard if available.
Typically such a request would come from a Wine application, but in theory
an external client could request a WCF_* format from a Wine application!
- Move Wine's internal clipboard implementation details to the clipboard
driver level (x11drv/clipboard.c). Wine's clipboard code will largely
delegate to the driver which will maintain the clipboard cache.
EVENT_SelectionRequest will directly manipulate the internal clipboard cache
and if necessary send a WM_RENDERFORMAT message to the window if the format
is set for delayed rendering.
This will simplify Wines clipboard code and hide the implementation details
in the clipboard driver level, where they belong.
Gérard Patel reminded Noel about a discussion he had with Ulrich Weigand
regarding this issue:
- Gérard's note:
I have looked a bit to the copy/paste problem, and here is my
idea. The clipboard got broken because of the separation of threads
in true 32 bits threads. To implement this, and to prepare to
separate address spaces, each thread/process has a separate
heap. What happen is that when an application copy data in the
clipboard :
- it copy the data in a data area in its private heap
- it signals X that the application has placed data on the clipboard.
When another application request data from the clipboard, X
signals an event to the Wine application and asks for the data
(text). Wine retrieves then the handle to the data area, get the
data, and sends it to X. The problem is that the context is
different. As I understand it, in an X event, Wine is in
a different context, so uses a different heap. So the handle
to the data, saved in a global variable, is not found in
this heap.
There is an error message about it that is very significant.
err:heap:HeapValidate Heap 40330000: block 40903ba0 is not inside heap
That's why it works (usually) in the other direction. When another
application has placed data in the X clipboard, and a Wine
application request data from clipboard, there is no X event,
and so there is no context change. But you can't copy from a
Wine application since when the other application request data,
Wine can't find the data (it is searching it in the wrong heap)
Now, how to fix ? To do it, it would be necessary to implement
a shared memory scheme and control the access through the wine
server (this is a citation of Ulrich Weygand; I have absolutely no
idea of how implement it in fact).
So a true solution will (probably) to wait for the completion of
the separation of address spaces; at this moment, I presume that
the Wine gurus will implement some mechanisms to share memory
and then the clipboard will work like it should.
- Ulrich's note:
Ah. Looking at the clipboard source, it seems that 32-bit clipboard
access could
never have worked, at least not between different
processes. The 32-bit GlobalAlloc has always returned a handle
valid only in the calling process.
The difference now is just that the X event processing is now done
always in a separate Wine process, while it used to be done in
one of the running processes. So, if you were running just a
single Windows application, it (accidentally) worked ...
Pavel Roskin then provided some more information on Motif's behavior
(quoted from Lesstif documentation):
Introduction
To understand how the Clipboard works, you first need to understand the
ICCCM specification (currently at version 2 in R6; you can find a
PostScript version in docs/hardcopy/ICCCM). Read through it, frequently
referring to O'Reilly Volume Zero (the protocol reference) and Volume Two
(the Xlib reference), about how selections should work, and how the CLIPBOARD
selection in particular should be used. It will take you about a month
to fully understand how all this should work. Come back when you're done.
Basics
Oh, you're back. Is your head swimming with visions of SelectionNotify,
SelectionClear, XGetSelectionOwner(), and other choice nuggets? Have you
seen the beautiful fogginess of EventHandlers, struggled with the concepts
of Atoms, properties on windows, and format atoms?
Good. You're ready for the first nugget of information about the M*tif
clipboard.
Forget everything you've learned, it doesn't apply in 90% of the cases.
There are two coexisting mechanisms in Motif; there's the Motif way, and
there's the X way. Both work, but the differences are a little messy
until
you understand what gets used, and when. (talk about compliant apps like
xclipboard vs Motif applications here). The mechanism uses selections,
but
doesn't do so in the traditional sense.
There are two selections: the XA_MOTIF_CLIP_LOCK, and the XA_CLIPBOARD.
The XA_MOTIF_CLIP_LOCK selection is used to serialize access to the
CLIPBOARD selection between M*tif applications. The XA_CLIPBOARD
selection
is used to indicate which app just added data to the clipboard via a copy
(normal M*tif case), or the application that just cut or copied something
to the clipboard (X case).
Contrary to O'Reilly, the Motif clipboard does NOT store data that you
pass to XmClipboardCopy in the CLIPBOARD property of the window that you
passed in (section 18.5 or section 17.5, depending on the version of
Volume
Six (or Six-A, respectively) that you're reading. Instead, it's stored on
the root window itself. Unless I've got the only version of Motif that
does this try using xprop on the root window after you've copied something
to the clipboard; if you don't see something like _MOTIF_CLIP_ITEM_*,
let me know). However, ICCCM compliant applications do.
Noel proposed to stick to his first XA_CLIPBOARD implementation (to be
compatible with X and KDE) and then to deal with
different X flavors ... at the Clipboard driver level. Since in any case
I was considering moving some of the clipboard code to the driver level,
potentially Motif specific issues could be addressed in a different Motif
clipboard driver.
Ulrich Weigand proposed a quick fix:you might simply
change the clipboard code to allocate data on the shared heap instead of
using GlobalAlloc. Alternatively, you might always use the 16-bit heap,
even for storing 32-bit data, as the 16-bit heap is always shared.
Regarding separate address spaces, Ulrich added we'll
need to think of a way to pass clipboard data to the other process.
Actually, wasn't your suggestion to always pass data via the X clipboard,
even between two Wine processes? This would nicely fix that problem ...
Noel noted that this might not work because the X
clipboard always works in "delayed render" mode. i.e. we need to return the
actual WINE clipboard data in an X atom only in response to an
XSelectionRequest event.
. Ulrich confirmed that since Wine currently
uses a single connection to X server in a service thread.
Albert den Haan pointed out another issue: The solution
described below looks to work for as long as the (single) application is up
(and therefore its service thread) but will not work as Win#+ does when the app
exits. It seems if we want to have this sort of persistence then the current
wine set-up will not work. We have enough trouble with non functional
zombies as it is!
Albert proposed to use the wine server thread (and mentionned some other
possible extension for serialization and/or persistence - like the registry).
Noel agreed and proposed that the server could keep a
copy of the data on (app) exit or alternatively the clipboard data could be
rendered to X atoms on app exit. So the clipboard server would just need to
access and return the appropriate atoms to the requesting client.
Alexandre Julliard pointed out that It can't be part of
the Wine server because the server doesn't have an X connection; but it should
be easy to write a small Winelib app whose only purpose is to hang on the
clipboard data.
5.
Address Space Separation / Stability vs Correctness
Archive Link: "Address Space Separation / Stability vs Correctness"
People:
Gavriel State, Ulrich Weigand, , Gérard Patel, Ian Schmidt
Gavriel State expressed some concerns regarding address spaces:
There has been a fair bit of talk on the list lately about implementing
address space separation for different processes. While I agree that
it is certainly important to emulate the Win32 runtime architecture as
closely as possible, I'm a bit concerned (or perhaps just confused)
about the necessity of this feature.
At present, it is possible to run multiple Win32 apps by starting
seperate Wine processes manually at the command line, which would then
start seperate Wine server processes along with the app. These processes
cannot communicate amongst each other using standard Win32 IPC APIs,
may have problems due to unserialized access to registry files, etc.
Some of this may be solvable by having a shared Wine server process.
Extending the Wine server model in this way is
not what people are
discussing as seperate address spaces though, right?
Now, when a Win32 app tries to create another process using the
CreateProcess API, this process is currently created in a shared
address space. This allows us to use GlobalAllocced memory as an
IPC mechanism, and generally makes it easy to provide the IPC functions
that might be expected in these circumstances.
The problem with the shared address space model is that it does not
provide the memory protection that would be provided with the seperated
model, and that the new process will not have the same memory layout
it would have had in Windows, right?
If that's all it is, why is it a big deal? Unless I'm mistaken,
providing seperated address spaces will be a
big deal, requiring
marshalling of all message data, and various other tweaky-to-get-right
tasks. On the other side of the coin, how common is the use of
CreateProcess amongst the apps people want to run? Is this useful
mostly to support the use of Program Manager/Explorer as a shell to
launch other apps? Where it is used, how important is the separation
of address spaces? Are there apps that aren't working right because
of the lack of this feature?
Finally, what's the current state of development on this? Have people
actually started working on it, or is it still in the discussion stage?
If so, are those of you working on it seeing a big reduction in
stability? Or am I just being a worry-wart.
Gavriel's main question is to make the right choice between stability
(what Corel's looking for to release apps) and correctness (add as
many features as possible), and main some parallel with Linux mode of
operation between stable and experimental versions. He asked wether
this was something to consider in the future, and since Corel's has
been maintaining its own stable CVS tree, I
could envisage our tree becoming a basis for that at some point.
Ian Schmidt listed some programs needing this separate address space
feature (mainly because one programs has its relocation info stripped, so
it can only be mapped at a precise address, and it fails it this address
is in use): Microsoft Visual Studio (VC++ 6.0), many games.
Gérard Patel mentionned InstallShield based installation not working
because of this.
Ulrich Weigand explained in details of the separate address space stuff:
Well, in a way it is: the goal would be to map the CreateProcess() API
to fork(), so that processes created by CreateProcess()
are more or
less the same as processes started separately on the command line,
but using a shared Wine server.
This would actually be doable right now, it would just need a small
amount of reshuffling of the process creation sequence, and a call
to fork() at the right place. The reason why it isn't done yet is
exactly that communication between processes would then stop working,
because currently the IPC mechanisms do indeed rely on the fact the
shared heap and all 16-bit global memory
are shared. So, before
this is done, the IPC mechanisms would need to be adapted.
The
big problem are the applications that are not relocatable, because
the relocation info was stripped during link. These *will not run*,
unless loaded exactly at the intended base address.
Well, as far as I see it, we are still in a discussion stage, although
a lot of preparations are already being done. In a sense, the whole
concept of using a Wine server process
is in fact a preparation for
inter-address space IPC.
Also, the work done lately in the area of rehashing the Win16 scheduler,
removing the one global TASK_Reschedule loop and running each process
in its own thread, is of course the first step towards full separation:
currently, every process runs at least in its own thread, while not yet
in its own address space.
As to stability: Unfortunately, there are indeed stability problems,
caused primarily by the switch towards using multiple threads. This
has exposed several thread-safety related problems, which are still
largely unresolved (primarily the problem that libc is not thread-safe
due to missing pthreads synchronization, but there are also Wine-internal
thread-safety problems still lurking, e.g. in the area of asynchronous
X event handling / window repainting/activation).
Apart from these problems (that are already there), I don't think the
switch to separate address spaces would cause any further stability
problems, certainly not for single-process applications (i.e. those which
don't call CreateProcess() anyway).
As far as I can see, the main changes still to be done are these:
- Use shared mappings to provide real shared memory between address
spaces for the shared heap and 16-bit global memory.
[ While this would probably affect even single-process applications,
a shared memory area to which only one process attaches should not
be any less stable than normal memory ... ]
- Marshal the parameters for inter-process messages.
[ If there are no inter-process messages, this will never happen. ]
- Clean up the use of global variables (e.g. instead of a global list
of window classes, use per-process lists etc. On the other hand,
the list of windows needs to be global, so this will either be
moved to shared memory or to the Wine server.)
[ If anything, this should improve stabililty, because this should
also improve thread-safety, even for single-process apps ... ]
To summarize: Apart from the current thread-safety related stability
problems (which need to be resolved in any case before Wine can be
declared 'stable'), I don't think the necessary preparations towards
the switch to separate address spaces will introduce fundamental
stability problems. (That's not to say there is no possibility of
bugs being introduced here or there, of course, but as far as I can
tell, no new
fundamental problems, like thread-safety, should arise.)
6.
DOS memory mapping (and some bits of paranoia)
Archive Link: "DOS memory mapping (and some bits of paranoia)"
People:
Ian Schmidt, Ulrich Weigand, Norman Stevens, Alexandre Julliard, , Ove Kåven, Eric Pouech
Ian Schmidt reported some crash while running Excel 2000:
Debugging the Excel 2000 crash, it turns out to be caused by some very strange
code. Specifically, inside it's main message handler (the app starts up and
runs fine waiting for a message to this point) it does the following (in rough
pseudocode - I may be misinterpreting some things a little, but I don't want to
post disassembled bits of Office either).
- Call GetVersionEx()
- If it's Win9X, call a second function with hardcoded parameters 0x800 and
0xf0000.
- The second function checks if the first parameter is non-zero - if it is, it
returns.
- Otherwise it proceeds to add up all the bytes starting at where the second
parameter points to, for as many bytes as the first parameter indicates and
returns that as the result. (the caller appears to ignore this result however)
On WINE 0xf0000 is invalid so this causes a nice crash. On Win98 a test
program I wrote that does the same thing runs fine (no faults, BSODs, etc) and
picks up what appears to be pieces of BIOS or something (what IS at linear
address 0xf0000 in the PC architecture? I never did any DOS programming). I
assume NT would exhibit the WINE behavior - anyone wanna try it?
Ian also put out some paranoid questions whether this could be "anti-wine" code?
Ulrich Weigand answered that In Win9[58], the first 4MB
page is shared between all processes, and contains the memory image of the boot
DOS session (1MB + 64K), followed by the low part of the 16-bit global heap.
So it would indeed appear that at linear address 0xf0000 the BIOS ROM should be
visible.
He also turned down the anti-wine code, but something linked
to some features of BIOSes (noting It is beyond me why a
mere application should depend on BIOS implementation details; if the OS would
do it, it still wouldn't be nice, but a little more understandable :-/)
Norman Stevens also doubted it's anti wine code: More
likely it is anti-piracy. The linear address 0xf0000 is the start of the bios rom,
so if you write the checksum into the executable as it is installed you can
prevent people running the program on a different computer.
Ian Schmidt didn't agree because the app run fine with just
zeros mapped there. So if it's anti-piracy it's broken (buggy? Microsoft? :). It
does execute the check every 5 seconds or so after the initial test (on a WM_TIMER).
Maybe it's really just an attempt to make people buy faster computers so they won't
notice how large Win2k is :)
Ulrich proposed a patch for it, and added If there are more
direct accesses to the boot DOS image, we might even think about mapping our
version of that image (DOSMEM_dosmem) actually at address 0 ...
Eric Pouech didn't like mapping all the 1MB at 0 because it would remove NULL
pointer deref faults (at least the first page should not be mapped, thus
disabling access to BIOS data segment at 0x00000400).
Alexandre Julliard proposed we could map it
there, but protect the first page, and un-protect it on the first access >=
0x400. This would still allow catching most NULL pointers accesses, and it
could make it possible to avoid forking a separate process for running DOS
programs (at least with ELF binaries).
Ian reported that patch from Ulrich didn't solve the issue, but with using
an expanded version of another one provided by Ove Kåven, he got Excel 2000 to
run minus some graphical glitches (loading / saving files
worked, typing stuff into cells worked, graphs/charts seem to work, etc).
Sharon And Joy