ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Tue Jun 25, 2019 5:51 pm

All times are UTC




Post new topic  Reply to topic  [ 11 posts ] 
Author Message
PostPosted: Wed Jun 05, 2019 10:57 pm 
User avatar

Joined: Thu Apr 18, 2019 6:45 pm
Posts: 11
Intro
I finally got this to work under Windows, but because of the amount of time I spent trying to make this work, I am compelled to share my experiences and comments on doing so. I don't want others to waste nearly as much time as I did trying to make this work.

I was initially unsure whether to make this thread under Tutorials or here. I settled for here because I wanted to also make some comments on the build scripts and compatibility. Prior to simply asking in the QBMS Errors thread if there was a Windows specific build script, I could find no mention of there being one on aluigi.altervista.org nor in this forum. Nor could I find any mention of a user having attempted to compile under Windows.

I'll spare most the experience of wandering in the dark trying to build using the makefile prior, but much of the changes I had to make with the makefile were present in that batch script already. For future reference to curious individuals: Don't try to build under Windows with the Makefile. It's hell and you'll waste more time chasing dubious compilation errors than you may want to admit.

For convenience, I'll quote aluigi's build script here:
aluigi wrote:
I use a .bat file for doing the whole compiling job on Windows using gcc 4:
http://aluigi.org/bms/compa.bat

I also made some minor edits to the script for convenience, readability, and a couple comments about compilation:
Attachment:
File comment: Slightly modified batch script to build quickbms in windows.
compile_quickbms.zip [2.51 KiB]
Downloaded 8 times

Some of the following might be common sense solutions for some readers, but I'd rather be thorough and complete than not.

Potential Pitfalls
1. Missing includes to library headers shipped with MinGW.
Code:
src\quickbms.c:172:14: fatal error: openssl/crypto.h: No such file or directory
     #include <openssl/crypto.h>
For some reason this occurred with all my MinGW installations. Headers and libs located in the opt directory aren't in the search paths by default (ie OpenSSL). As such, you may want to manually add opt\include and opt\lib paths to the compilation script.

2. Missing includes for mcrypt and tomcrypt.
Add these include paths to the compilation script. (My modified script already has this addition.)
Code:
-Isrc\libs\libmcrypt -Isrc\libs\libtomcrypt

3. Several "skipping incompatible" library errors while linking.
You're trying to use an x86_64 toolchain + libraries. Install the i686 toolchain and try with that.

4. Failure to link xcompress.lib
Code:
ld.exe: xcompress.lib(obj/i386/lci.obj): warning: line number count (0x2) exceeds section size (0x1)
xcompress.lib(obj/i386/lci.obj): could not read symbols: Invalid operation
collect2.exe: error: ld returned 1 exit status
Don't use GCC 7 or newer. The library xcompress.lib will be incompatible with the binutils that are shipped with GNU GCC 7 and 8. If you're hellbent on using latest possible GCC toolchain, then use GCC 6.4.0. It will also work with GCC 5 and 4.

5. Undefined reference to `BCrypt_init`
It's possible that bcrypt.h has been included by another file (possibly wincrypt.h). I've observed this in both GCC 5 and GCC 6. So, unless you're Mr. Aluigi himself or you actually are attempting to build and test on a Win98 machine... It should be satisfactory enough to make these two changes and add -lbcrypt to the compilation script:
Code:
// In file /src/myenc.c : Comment out the mybcrypt include and use MinGW's
//#include "extra/mybcrypt.h"
#include <bcrypt.h>
// [ ... About 110 lines below ... ]
//if(!BCrypt_init()) return -1;  // The comment on this line in-file suggests only needed for Win98


Final Words
Could you put the batch script in an easy-to-find location... Like the page for quickbms on your site.


Last edited by Brolijah on Mon Jun 10, 2019 9:56 pm, edited 2 times in total.

Top
   
PostPosted: Wed Jun 05, 2019 11:44 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 10553
Yeah compiling quickbms on Windows is complicated and it's a sort of "card castle" where everything is balanced and works on the building environment that I use for it.

Compiling on Windows is ever painful with all programs (except helloworld.c) so this is not really a surprise, various versions of VS, mingw/gcc/cygwin and so on... really painful.

1) I guess that #include is mandatory anyway. Did you cross-compile or compiled natively?

2) weird and interesting, I will review that

3) yes definitely, DO NOT TRY x86_64

4) this is interesting and also partially easy to bypass since a portion of the code from xcompress is available with source code (used on Linux)

5) this is really weird but the mybcrypt.h wrapper works well, do you get any specific error?
Here my BCrypt_Init is linked and perfectly called at runtime (just checked) and all the API referenced.
I don't have a sample ready to test and share unfortunately.

Ah, in case someone is curious... yes there are some users of quickbms who run it on Win98 :D


Top
   
PostPosted: Thu Jun 06, 2019 12:58 am 
User avatar

Joined: Thu Apr 18, 2019 6:45 pm
Posts: 11
(The list was just to answer any potential questions for people who come across these things as I did. Didn't mean them to sound like I was asking those questions here.)
Note 1 is just... weird. You'd think headers and libraries installed with MinGW would all be in GCC's default search path. Yet, /opt/include and /opt/lib never was for any of my installations. MinGW puts openssl in there and not the main include path. I compiled natively using MinGW-w64 GCC toolchains.
Note 4... I couldn't find much on xcompress. Didn't realize some of it was open-source.
Note 5. GCC complains about two things: (1) warning implicit declaration of BCrypt_init in myenc.c and (2) undefined reference to BCrypt_init at link time. I did think it very strange that GCC thought the call to BCrypt_init was an implicit declaration although it had already been defined with a body in the header... If you feel willing to look at the disassembly, I can attach the object file for quickbms.o. The actual function for BCrypt_init is just gone. (The easy check should've been find where "bcrypt.dll" is referenced before LoadLibrary, but the string is literally nowhere.)


Top
   
PostPosted: Thu Jun 06, 2019 1:45 am 
User avatar

Joined: Thu Apr 18, 2019 6:45 pm
Posts: 11
Attached below is the object file I compiled for quickbms.c using the normal code in myenc.c.
BCrypt_init should have been compiled into this object by inclusion. "quickbms.c" includes "myenc.c", and "myenc.c" includes "mybcrypt.h". Except by my search, it's nowhere. In fact, the function call is undefined according to IDA.


Attachments:
quickbms_temp.zip [1.8 MiB]
Downloaded 8 times
Top
   
PostPosted: Thu Jun 06, 2019 7:11 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 10553
Maybe your gcc already has __BCRYPT_H__ defined since it has been already loaded and mybcrypt.h is ignored.

Anyway with a vanilla Mingw using gcc 4 x86 on Windows (native) there are no issues and it compiles on the fly without spending hours with errors.
I remember the old t-dragon toolchain (but probably older than 2015) on which it worked perfectly too, ever on the fly using compa.bat.


Top
   
PostPosted: Mon Jun 10, 2019 9:47 pm 
User avatar

Joined: Thu Apr 18, 2019 6:45 pm
Posts: 11
aluigi wrote:
Maybe your gcc already has __BCRYPT_H__ defined since it has been already loaded and mybcrypt.h is ignored.

Oh god I can't believe I didn't try to test that scenario. I tried a stupid little test and I see now bcrypt is already included by another header.
Code:
In file included from C:/mingw-w64/i686-6.4.0-win32-dwarf-rt_v5-rev0/mingw32/i686-w64-mingw32/include/wincrypt.h:764:0,
                 from C:/mingw-w64/i686-6.4.0-win32-dwarf-rt_v5-rev0/mingw32/i686-w64-mingw32/include/windows.h:95,
                 from C:/mingw-w64/i686-6.4.0-win32-dwarf-rt_v5-rev0/mingw32/i686-w64-mingw32/include/winsock.h:10,
                 from C:\mingw-w64\i686-6.4.0-win32-dwarf-rt_v5-rev0\mingw32\opt\include/openssl/dtls1.h:71,
                 from C:\mingw-w64\i686-6.4.0-win32-dwarf-rt_v5-rev0\mingw32\opt\include/openssl/ssl.h:1699,
                 from src\quickbms.c:173:

That answers that, I guess. I'll update the OP to reflect this. I believe someone, someday, will stumble across this thread, and I hope it'll be of some use to them.
As a side memo, the whole reason why I set out to compile quickbms myself was because I and a couple others on Discord wanted the fixed ALLZ code built in with quickbms. Thank you again for your help and this multi-useful tool.


Top
   
PostPosted: Tue Jun 11, 2019 8:51 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 10553
Wow, compiling the whole quickbms just for ALLZ?
Wasn't enough your stand-alone tool for doing that job?


Top
   
PostPosted: Tue Jun 11, 2019 7:09 pm 
User avatar

Joined: Thu Apr 18, 2019 6:45 pm
Posts: 11
Two reasons: It's just more convenient. Do you remember the Jump Stars ALAR archive type? That's also one of those Aqualead formats. The guys who were researching that game got stumped when they got to ALLZ from what I read on some 15 year old forum posts. Since games using ALAR are also likely using ALLZ or vice-versa, we could just use one tool for both.
The second reason is the Zanki Zero people's files are not packed in ALAR files... rather they're just ALLZ files stacked back to back. No header to them or anything, which is weird. So they need to split up those files then decompress. A time consuming process that could probably be scripted with just one tool.


Top
   
PostPosted: Wed Jun 12, 2019 8:42 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 10553
Very interesting.

I hope to make a simple step-by-step for an error-free compiling once I have some free time.

After all one day I had to open this Pandora's box :D
Being the only author and maintainer of QuickBMS everything will stop once I will stop working on it, so it's necessary to have a plan B ready in place.

My only request is to avoid forks, the situation is already a mess with the older versions of the tool, therefore forks are definitely my nightmare.
I'm ready and available to implement ANY feature and crazy idea that has a sense and works (and doesn't take time to be implemented), I'm very open on that side.


Top
   
PostPosted: Thu Jun 13, 2019 12:20 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 10553
As promised, this is the simple step-by-step for compiling quickbms on Windows.
There are NO requirements and it's meant for everybody.

1)
Download gcc 4.9.4
https://sourceforge.net/projects/mingw- ... z/download

2)
unzip gcc on your disk, maybe in c: (it will be c:\mingw32)

3)
Download the current quickbms dependencies
http://aluigi.org/bms/quickbms_dependencies.zip

4)
unzip the include folder in c:\mingw32\i686-w64-mingw32

5)
unzip the lib folder in c:\mingw32

6)
download the quickbms source code and extract the src folder in c:\quickbms (it will become c:\quickbms\src)
http://aluigi.org/papers/quickbms_src.zip

7)
download compa.bat and put it in c:\quickbms
http://aluigi.org/bms/compa.bat

8)
open compa.bat with a text editor and:
- remove line 16: set TEMP=z:\temp
- remove line 17: set TMP=z:\temp
- replace all the occurrencies of c:\mingw with c:\mingw32

9)
run the console (cmd.exe)

9)
go in the quickbms directory and set the path of the Mingw compiler:
Code:
set PATH=c:\mingw32\bin


10)
Code:
compa.bat


11)
for compiling quickbms_4gb_files.exe
Code:
compa.bat -DQUICKBMS64



Notes:

* quickbms 0.10.0 has a compiling bug in src\utils.c at lines 440 and 442, you can just delete them because they are unused.

* there is also the BCrypt problem reported by Brolijah, it's enough to remove line 1431 in src\myenc.c and add -lbcrypt after compa.bat:
Code:
compa.bat -DQUICKBMS64 -lbcrypt


* if you need to compile it fast for your tests remove "-s -O2" from compa.bat

* gcc 8.1.0 works perfectly too:
https://sourceforge.net/projects/mingw- ... z/download

* this step-by-step will be fixed after I will release quickbms 0.10.1


Top
   
PostPosted: Thu Jun 13, 2019 1:19 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 10553
I have manually patched xcompress.lib for setting the line number field of lci.obj and ldi.obj to zero :)
The patched version is already included in quickbms_dependencies.zip
For the tech guys:
Code:
           original_xcompress.lib (MD5 8f4165d4d9a173c06bc00dc5a3ab3aab)
  Offset   |        patched_xcompress.lib
-----------------------------
000A40D0   02       00
0010B4AC   02       00


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 11 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited