ZenHAX
https://zenhax.com/

QuickBMS - Scan all the supported compressions
https://zenhax.com/viewtopic.php?f=4&t=23
Page 1 of 1

Author:  aluigi [ Thu Aug 07, 2014 3:23 pm ]
Post subject:  QuickBMS - Scan all the supported compressions

During the reverse engineering of an archive or an unknown file it may happen to see that it uses compression due to some parameters found in the index table and/or due to its "scrambled" content:
Image


Usually there are some tricks to know if it's a known compression algorithm, for example zlib starts with 0x78, lzma with 0x5d followed by some zeroes, lzss and lzo show parts of the uncompressed content and so on.

But if we don't know the algorithm or we want to be sure of its name or we want to know what's the result which is closer to the original uncompressed file, we need to use the following script and bat file:
http://aluigi.org/papers/bms/comtype_scan2.bat
http://aluigi.org/papers/bms/comtype_scan2.bms

The following is the situation in our folder, with dump.dat that is our compressed file:
Image


And this is the runtime help of comtype_scan2.bat:
Image


Let's insert this command-line to start the scan:
Code:
comtype_scan2.bat comtype_scan2.bms dump.dat output

Please note that if we already know what is the uncompressed size, it's HIGHLY recommended to add it to the command-line like in this example:
Code:
comtype_scan2.bat comtype_scan2.bms dump.dat output 0x7cf


During the scanning QuickBMS will show lot of messages and errors.
That's perfectly normal.
Usually you will notice that it freezes like in this case:
Image


No problem, press CTRL-C and type 'n':
Image


Finally we reach the end of the scanning:
Image


The next step is the manual checking of the results dumped in the output folder.
There are some ways to automize this process, anyway the simplest way is ordering the files by size in decrescent order:
Image


And then open them one-by-one with a hex editor:
Image


That 8.dmp seems to contain valid PNG data, let's try to open it with an image viewer:
Image


Bingo, that's the correct algorithm.

Now open defs.h text file inside the QuickBMS source code (src folder in quickbms.zip) and check what algorithm is that number 8:
Image


Yeah, the algorithm is lzo1x.

Don't think that it's ever so easy to find the correct algorithm, sometimes you don't know the name of the file and its content is a custom format or a raw audio/image.

Attachments:
img9.png [8.86 KiB]
Not downloaded yet
img8.png [19.11 KiB]
Not downloaded yet
img7.png [37.13 KiB]
Not downloaded yet
img6.png [29.28 KiB]
Not downloaded yet
img5.png [10 KiB]
Not downloaded yet
img4.png [10.39 KiB]
Not downloaded yet
img3.png [10.02 KiB]
Not downloaded yet
img2.png [10.32 KiB]
Not downloaded yet
img1.png [18.6 KiB]
Not downloaded yet
img0.png [18.82 KiB]
Not downloaded yet

Author:  aluigi [ Thu Aug 07, 2014 5:10 pm ]
Post subject:  Re: QuickBMS - Scan all the supported compressions

Ah, I have attached the original dump.dat in case someone wants to make his own tests.

You can even create it by yourself with quickbms:
Code:
comtype lzo1x_compress
get SIZE asize
clog "dump.dat" 0 SIZE SIZE


Attachments:
dump.zip [959 Bytes]
Downloaded 257 times

Author:  aluigi [ Tue Jan 17, 2017 8:15 am ]
Post subject:  Re: QuickBMS - Scan all the supported compressions

I want to stress the fact that the comtype scanner should be used only if you know really what you are doing.

Very quickly:

- do you have a file that may contain chunks of compressed data?
DO NOT USE the comtype scanner

- do you have a raw file that may contain anything?
DO NOT USE the comtype scanner

- do you have a raw file that you are sure contain compressed data from offset 0 till its end?
YES, USE the comtype scanner

- is the comtype scanner a way to find compressed chunks of data in a file?
NO

- is the comtype scanner a way to find what algorithm is used on a specific piece of data?
YES, the compressed data must cover the whole file, so if the file is 0x123 bytes big and the compressed data is from offset 0 to 0x10 or from offset 0x10 to 0x123 it will fail!

- example, if you use comtype scanner on a ZIP archive you will find absolutely NOTHING

- example, if you use comtype scanner on the compressed part of a ZIP archive you will have success (deflate algorithm)

In general the rule is not using the scanner except if you want to waste your time and your resources, that's up to you but then don't complain with quickbms for your faults.

Author:  usabdt [ Wed Nov 14, 2018 6:36 am ]
Post subject:  Re: QuickBMS - Scan all the supported compressions

comtype_scan2.bat not used on win 10 ?

Author:  aluigi [ Wed Nov 14, 2018 12:41 pm ]
Post subject:  Re: QuickBMS - Scan all the supported compressions

@usabdt
It works with win10 too, do you get any error and what error?

Author:  usabdt [ Wed Nov 14, 2018 12:51 pm ]
Post subject:  Re: QuickBMS - Scan all the supported compressions

aluigi wrote:
@usabdt
It works with win10 too, do you get any error and what error?

can not run the file comtype_scan2.bat . Can you capture your operation when doing on Win 10 ??

Author:  aluigi [ Wed Nov 14, 2018 6:26 pm ]
Post subject:  Re: QuickBMS - Scan all the supported compressions

usabdt wrote:
can not run the file comtype_scan2.bat .

Details?
Anyway that's something meant only for advanced users. If you want support for a format or a compression ask on the forum and do NOT try it, just as written in my FAQ post above.

Page 1 of 1 All times are UTC
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/