due to the number of regressions I found few days ago, it occurred to me that we really need an API test suite now. Thus I went ahead and implemented a test functionality for LoadModule16(). The result: 7 of the 21 tests that I implemented were FAILING !! That's ONE THIRD. And that for such a highly basic function as LoadModule16(). That's quite incredible.

I guess we really should change our development model from trying tons of programs to *systematically* testing functions and Windows mechanisms now. If we can show everyone where stuff is failing, it might be a lot easier to attract new people.

I attached a preview of the posting I intend to post on *tons* of Windows devel newsgroups ("Call For Volunteers"). That way we might actually get hold of hundreds of Windows developers helping us implement a complete test suite (complete tests of up to 12000 Windows functions). Not to mention the additional PR we might get out of this...

If this works, then we'd need someone in charge of managing the assignment of specific windows functions to specific programmers. I won't do that (very limited time). Anyone from Codeweavers up to the task ? Or maybe someone non-developer and rather foreign to the Wine devel could do that task ?

About the program: My goal was to get a program with very easy parsing interface: fields separated by colons, nothing fancy and difficult to implement, ... A .exe can be found at: http://home.nexgo.de/andi.mohr/download/wt_ldr16.exe

Please comment on both my intended posting and the way I programmed the first version of the test suite (I'm not extremely happy with the current program; if you have any improvements, then get them here ASAP !).

As 12000 Windows functions is an enormous number, I think we should really start doing this NOW instead of years later once Wine really needs this. Wine does not extremely need this yet, but we're getting pretty damn close to that already.

Just looked at the tool. It only provides gateway from Perl to wine API functions, right?

Advantages I see in using script-based tests:

1. tests can be used without recompilation
2. Perl is familiar to many developers
3. programming in scripting language is usually easier than in C
4. Perl is widely used for testing purposes and I expect to find many testing tools in Perl.

Cons:

1. Many current Wine contributors don't know Perl
2. one more level of abstraction does not give significant advantages in this application. On the contrary, it is more difficult to locate cause of problems because developer has to go trough one more, often not familiar layer. Absense of strict typing in this layer will hurt a lot.

Advantages of using C-based tests:

1. compilation has to be used to run the tests. In some cases this is an advantage. Before running the tests you'd better be sure it at least compiles in both environments.
2. C is the most familiar to developers language and the language itself is simpler than Perl
3. Documentation for Win32 API is C-oriented. Examples, given in documentation are in C or easy translated to C
4. Developers already have some testing code snippets in C

For Perl all you need is the winetest app that is in the tree, plus a bit of Makefile glue which is pretty trivial to do.

In fact here's a 10-minute hack to add a make test target. With that all you have to do is create a test script in dlls/xxx/tests/foo.test, put the expected output in tests/foo.test.ref (presumably generated by running the test under Windows), add your script to the makefile and run make test.

1. Modify the makefiles so that 'make test' (or make regress, or pick your poison) does something interesting, and does so without much hassle.
2. Define a standard for how tests should work. For example, if I wanted to write a test for FunctionFoo(), I should be able to create wine/programs/winetest/tests/foo.pl, add 'foo.pl' to winetest/tests/Makefile.in, and my test should automatically become part of the regression suite.
   
   (Note that adding tests/foo.c is just as trivial).
   
   I believe that the defacto standard chosen at this point is that a test prints 'OK' if all went well, and it doesn't on error, and prints some clue as to what went wrong.
3. Add a section to the documentation on how to do the same.

A test unit is an executable or script. You can name it anyway you like (please no space in the names, it's always annoying). All test units should be non-interactive. A test unit called xxx generates two outputs:

• its exit code
• text output on either or both of stdout and stderr, both of which are normally redirected to a file called 'xxx.out'.

A test succeeds if:

• its exit code is 0
• and its output, 'xxx.out' matches the reference output according to the rules described later.

Reciprocally it fails if:

• its exit code is non zero Either because one aspect of the test failed and thus the test unit decided to return a non-zero code to indicate failure, or because it crashed and thus the parent got a >= 128 error code.
• or because its output differs from the reference output established on Windows

Under this model each test unit may actually be comprised of more than one process (for instance to test CreateProcess, inter-process messaging, inter-process DDE, etc.). All that counts is that the original process does not finish until the testing is complete so that the testing framework knows when to check the test output and move on.

(There is no provision for hung tests. A time out based mechanism, with a large time out, like 5 minutes, could do the trick.)

A test unit can also exercise more than one aspect of one or more APIs. But, as a rule of thumb, a specific test should not exercise more than a couple to a handful related APIs (or up to a dozen in extreme cases). Also, more than one test could exercise different aspects a given API.

So when running the Wine regression tests, if we find that 3 tests out of 50 failed, it means that three processes had an incorrect exit code or output out of fifty. One should then analyze in more details what went wrong during the execution of these processes to determine which specific API or aspect of an API misbehaved. This can be done either by looking at their output, by running them again with Wine traces, or even by running them in a debugger.

When you run the test, every test case either passes or fails, which results in a classification into four subsets:

PASS Test case was expected to pass, and it did
FAIL Test case was expected to pass, but failed
XPASS Test case was expected to fail, but passed
XFAIL Test case was expected to fail, and it did

Only a test case in the FAIL category causes the whole test run to fail, and is reported even in the test run summary. The other categories are only reported as total numbers.

If you are getting nonzero FAIL numbers, you have introduced a regression. Nonzero PASS and XFAIL numbers are expected; if you get nonzero XPASS numbers you might look at the cases in question and decide whether you want to remove the 'expected to fail' flag.

I ported a sample of my tests from C to the Perl framework. Want to share my experiences.

Good:

1. 1) AFAICS technically you could do all the things you could do in C - working with structures, using callback functions, using threads. Existing problems probably can be ironed out.
2. additional processing of the test results can be done easier and quicker than in C

Bad:

1. constants must be explicitely declared or plain numbers used instead
2. no documentation on the framework yet.
3. in many cases data sizes must be specified manually - for structures processing, function parameters.
4. more difficult to track issues. than in C

Note, issues (1) and (2) probably can be fixed.

As some of you might remember I'm the lead developer of the Odin project. (project for OS/2 with the same goals as Wine). Odin is partly based on Wine code. The core dlls (gdi32, user32, kernel32) + some special dlls (winmm, directx, winsock) use some Wine code, but is basically incompatible with Wine for obvious reasons. Most of the remaining dlls are based entirely on Wine code.

The last few years there has been little cooperation between both projects for several reasons. One of them is that we have far fewer active developers and cooperation sometimes proved to be difficult. Some source files of the two project had grown apart since changes were never merged back into Wine.

Now, I'm trying to change this situation, but am wondering how to proceed. I can post bug fixes to this list, but it will be difficult for me to actually test these changes in Wine itself. (too time consuming) Posting diff files is not always possible due to the major differences between source files.

Merging changes from Wine into our source tree is no problem. We can do that ourselves. However, it would be nice if compiler incompatibilites (we use IBM VisualAge C++, not GCC) could be resolved. There aren't many, but the IBM compiler is not always as flexible as GCC and of course doesn't understand any GCC extensions. It would be a big help if incompatible code could be changed. Switching compilers is not an option for us. I can post a (small) list of changes we had to make.

The license of our project is not compatible with Wine, but the dlls that are largely based on Wine (such as comctl32, shell32, ole) are an exception to the project license. The sources of those dlls are released under the Wine license. In case there is ever interest in the other sources, then you can just ask me. I'd be happy to 'donate' them to Wine.

For more information about Odin you can visit http://odin.netlabs.org (some information is outdated though). There you can find out where to download the sources (cvs).

The next post from me will contains some patches for bugs in Wine that I've recently found.

I just got this - Lindows released 2 screenshots of their Lindows: www.lindows.com/screenshots

Now, IF they did use wine (and thats a big if - from the screenshots they're running either Office 2K or office XP, Internet Explorer, the fonts looks like MS TTF fonts, and I think those screenshots are modified a bit - look at the 2nd screenshots, behind the start menu - look at the "file/edit/view" menu - it should be alligned to the left) and they'll contribute some parts of their wine to the standard wine - then wine will be in much better shape then what is it today.

While doing research for my site on the Microsoft antitrust trial ( http://www.kegel.com/remedy/ ), I came across a nasty little EULA for a product called MSNBC News Alert. The EULA is at http://www.msnbc.com/tools/newsalert/naeula.asp and says

MSNBC Interactive grants you the right to install and use copies of the SOFTWARE PRODUCT on your computers running validly licensed copies of the operating system for which the SOFTWARE PRODUCT was designed [e.g., Microsoft Windows(r) 95; Microsoft Windows NT(r), Microsoft Windows 3.x, Macintosh, etc.].

I've just experimentally, in my own copy of the CVS, changed the code in misc/cdrom.c so that it handles mixed mode CDs the same as data CDs for retrieving CD labels. (In the process of trying to get Age of Empires 1 to run in single-player mode).

And surprisingly, it worked! (mind you AOE is not fast under wine, but that's another problem).

Does anyone have any comments on this approach? if not I'll submit a patch...

I guess it might be music CDs with builtin biography data/programs often that are problematic.

If you get hold of such CDs, then you need to experiment with CDROMREADMODE2, CDROMREADMODE1 and/or CDROMREADRAW ioctl()s (in /usr/include/linux/cdrom.h) to get back sector data. I already verified once that you can get the CD label of "problematic" mixed-mode CDs this way. I couldn't figure out the mechanism that e.g. Windows uses to calculate the serial number of these CDs, though.

You might probably want to figure out a way to have misc/cdrom.c also support label and serial reading of problematic mixed-mode CDs, in a modular way (i.e. one that doesn't look too awful code-wise ;)

Ok - I've been doing some serious experimenting here, including reading the iso 9660 spec :)

I've found some interesting things too.

First, it seems that some CD's don't just store the 'best' data in higher-numbered volume descriptors. Some seem to store 'everything' in the type 1 (primary) descriptor, and only store the modified info in the type 2 (secondary) descriptor. So the routine CDROM_Data_FindBestVoldesc (it returns the highest-numbered descriptor) might need some rethinking...

Secondly, I found a real curiosity in one particular CD: Microsoft's Multimedia Beethoven (about 1992) isn't ISO9660 at all. It's a mixed mode CD, and is readable in Windows, as well as Wine, but the label is screwy (Windows XP gets it wrong too). Instead of using the specification identifier 'CD001' (ISO9660) it uses 'CDROM' and has two dwords stored at the start of the volume descriptor, as well as having a few other discrepancies. I have no conclusive proof as to what specification it's written, but I'm guessing its possibly the original High Sierra format, or perhaps an even earlier, semi-proprietary, spec. Either way, it makes me wonder whether some of the 'strange' mixed-mode CDs are of a similar nature.

Anyway, what I'm thinking is to rewrite the code that deals with CD labels to take into account what I've found so far - I'll also include support to get the label from the 'CDROM' type I've found, so we'll have the right label for that CD too. Quite possibly, that'll fix the 'strange' mixed mode CD's too.

If anyone has an odd CD lying around, they can email me the first 65536 bytes of the CD (hopefully no vendor will care about copyright on their volume descriptors!) so I can compare the structure of the volume descriptors - I've got about 8 of them so far, just use 'dd if=/dev/scd0 of=cd.dmp bs=65536 count=1' and compress the result before emailing it to me :)