ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Tue Sep 19, 2017 11:38 am

All times are UTC




Post new topic  Reply to topic  [ 3 posts ] 
Author Message
PostPosted: Thu Aug 07, 2014 3:23 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6661
During the reverse engineering of an archive or an unknown file it may happen to see that it uses compression due to some parameters found in the index table and/or due to its "scrambled" content:
Image


Usually there are some tricks to know if it's a known compression algorithm, for example zlib starts with 0x78, lzma with 0x5d followed by some zeroes, lzss and lzo show parts of the uncompressed content and so on.

But if we don't know the algorithm or we want to be sure of its name or we want to know what's the result which is closer to the original uncompressed file, we need to use the following script and bat file:
http://aluigi.org/papers/bms/comtype_scan2.bat
http://aluigi.org/papers/bms/comtype_scan2.bms

The following is the situation in our folder, with dump.dat that is our compressed file:
Image


And this is the runtime help of comtype_scan2.bat:
Image


Let's insert this command-line to start the scan:
Code:
comtype_scan2.bat comtype_scan2.bms dump.dat output

Please note that if we already know what is the uncompressed size, it's HIGHLY recommended to add it to the command-line like in this example:
Code:
comtype_scan2.bat comtype_scan2.bms dump.dat output 0x7cf


During the scanning QuickBMS will show lot of messages and errors.
That's perfectly normal.
Usually you will notice that it freezes like in this case:
Image


No problem, press CTRL-C and type 'n':
Image


Finally we reach the end of the scanning:
Image


The next step is the manual checking of the results dumped in the output folder.
There are some ways to automize this process, anyway the simplest way is ordering the files by size in decrescent order:
Image


And then open them one-by-one with a hex editor:
Image


That 8.dmp seems to contain valid PNG data, let's try to open it with an image viewer:
Image


Bingo, that's the correct algorithm.

Now open defs.h text file inside the QuickBMS source code (src folder in quickbms.zip) and check what algorithm is that number 8:
Image


Yeah, the algorithm is lzo1x.

Don't think that it's ever so easy to find the correct algorithm, sometimes you don't know the name of the file and its content is a custom format or a raw audio/image.


Attachments:
img9.png [8.86 KiB]
Not downloaded yet
img8.png [19.11 KiB]
Not downloaded yet
img7.png [37.13 KiB]
Not downloaded yet
img6.png [29.28 KiB]
Not downloaded yet
img5.png [10 KiB]
Not downloaded yet
img4.png [10.39 KiB]
Not downloaded yet
img3.png [10.02 KiB]
Not downloaded yet
img2.png [10.32 KiB]
Not downloaded yet
img1.png [18.6 KiB]
Not downloaded yet
img0.png [18.82 KiB]
Not downloaded yet
Top
   
PostPosted: Thu Aug 07, 2014 5:10 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6661
Ah, I have attached the original dump.dat in case someone wants to make his own tests.

You can even create it by yourself with quickbms:
Code:
comtype lzo1x_compress
get SIZE asize
clog "dump.dat" 0 SIZE SIZE


Attachments:
dump.zip [959 Bytes]
Downloaded 167 times
Top
   
PostPosted: Tue Jan 17, 2017 8:15 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6661
I want to stress the fact that the comtype scanner should be used only if you know really what you are doing.

Very quickly:

- do you have a file that may contain chunks of compressed data?
DO NOT USE the comtype scanner

- do you have a raw file that may contain anything?
DO NOT USE the comtype scanner

- do you have a raw file that you are sure contain compressed data from offset 0 till its end?
YES, USE the comtype scanner

- is the comtype scanner a way to find compressed chunks of data in a file?
NO

- is the comtype scanner a way to find what algorithm is used on a specific piece of data?
YES, the compressed data must cover the whole file, so if the file is 0x123 bytes big and the compressed data is from offset 0 to 0x10 or from offset 0x10 to 0x123 it will fail!

- example, if you use comtype scanner on a ZIP archive you will find absolutely NOTHING

- example, if you use comtype scanner on the compressed part of a ZIP archive you will have success (deflate algorithm)

In general the rule is not using the scanner except if you want to waste your time and your resources, that's up to you but then don't complain with quickbms for your faults.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 3 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited