Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Mon Jun 21, 2021 5:13 pm

All times are UTC

Post new topic  Reply to topic  [ 2 posts ] 
Author Message
PostPosted: Sat Sep 26, 2020 7:54 pm 

Joined: Wed Sep 09, 2020 10:26 am
Posts: 2
I came here earlier and posted my progress with this tool in the totally wrong board, and in there I said I was thinking of writing a compression algorithm... well, here it is.

To recap: I reversed the decompression function for Okage: Shadow King, and made a tool in Python for extracting the assets in the game's .XPF archives.
Now there's an options to pack the assets back into .XPF archives, because I wrote a compression function based on the decompressor.
It recompresses decompressed files just as well (tends to actually produce identical compression streams), and even better (even smaller files), than the original tool the developers used.

Attached is the tool written in Python. When fed an .XPF file, it will extract and decompress all contained files. When fed a folder, it will compress all files and pack them into an .XPF file.
Also included are C++ implementations of the same decompression and compression functions standalone, because I simply don't know to make an equivalent tool in this language - but I made these out of necessity because speed is an issue with Python; it may very well take hours to compress 1 file, while CPP takes mere seconds to do the same. See the recompression table under EDIT 2.

EDIT: Fixed a blunder that would confound anyone reading the compression algorithm. Also attempted to speed up the Python version a bit, and added a readme to help you use it.

EDIT 2: Added a compression levels option, which includes medium compression in a reasonable timeframe, and an option to completely match the compression in the original game files.
I selected this big file below to demonstrate the different levels. It's probably in the extreme end of time differences, but that also serves to clearly show what the levels are capable of.
Recompression table for DATA\BTLDATA\ALPHA\FIELD\BEILY41.XPF using OSK Tool.py
Level   Elapsed   Compressed size
0       0.02s     346,764 (No compression.)
1       9.6s      111,640 (Low compression; short writes, short seeks.)
2       9.6s      111,585 (Shallow compression; long writes, short seeks.)
3       128.14s   94,211  (Identical game compression; short writes, long seeks.)
4       118.21s   93,716  (Maximum compression; long writes and long seeks.)

EDIT 3: I just found out that you can completely bypass Python's slowness by running the script using PyPy instead of official Python - it matches the C implementations in speed, taking 1 second where it would otherwise take 1 minute... Wish I knew that earlier.

File comment: Includes: OSK Tool.py, OSK Decompress.cpp, OSK Compress.cpp, readme.txt
OSK Tool and code.zip [7.54 KiB]
Downloaded 95 times

Last edited by Inuk Syooperstar on Mon Nov 02, 2020 7:09 pm, edited 1 time in total.
PostPosted: Sat Oct 10, 2020 11:59 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12297
Well done and thanks for the code.
I will support this compression in quickbms 0.11, I just added it to the beta today and I will update the public beta next week.
In quickbms I will not handle the 32bit field at the beginning because I prefer to make the algorithm usable in the comtype_scan2 scanning.

Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 2 posts ] 

All times are UTC

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited