Free Game Research Forum | Official QuickBMS support | twitter @zenhax
It is currently Tue Jan 31, 2023 8:23 pm

All times are UTC

Post new topic  Reply to topic  [ 24 posts ]  Go to page Previous 1 2
Author Message
PostPosted: Sun Aug 11, 2019 5:54 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12984
Yeah, unfortunately it doesn't support files larger than 2Gb.
I guess I can easily update it if necessary. but really people need to scan files that big?

PostPosted: Mon Aug 12, 2019 1:46 pm 

Joined: Wed Aug 24, 2016 1:50 am
Posts: 3
aluigi wrote:
Yeah, unfortunately it doesn't support files larger than 2Gb.
I guess I can easily update it if necessary. but really people need to scan files that big?

Nah... I just thought it wasn't a known 'bug' since it wasn't stated anywhere. Could be easily bypassed by just extracting the needed chunk out into a separate file and using offzip on it though, for anyone who encounter the same issue.

PostPosted: Fri Dec 09, 2022 9:27 am 

Joined: Sun Jan 06, 2019 8:23 am
Posts: 5
aluigi wrote:
Offzip - http://aluigi.org/mytoolz.htm#offzip

dump zlib/deflate compressed data, help in reversing file formats, why scanning compressions isn't possible with any algorithm

Imagine a situation in which you have a file and you have no idea of its content or you don't know its format to create a parser.

The first thing to try is usually checking if there is compressed data in it and extracting it on the fly or you can use some information about the compressed and uncompressed size to write a file format parser.

The most used compression algorithm in the world is deflate, also known as zlib.
Deflate and zlib are basically the same algorithm but with a small important difference:
  • deflate [url=http://www.ietf.org/rfc/rfc1951.txt‎]RFC1951[/url] is the pure compressed stream: no headers, just data
  • zlib [url=http://www.ietf.org/rfc/rfc1950.txt‎]RFC1950[/url] is a deflate stream with a small header at the beginning and a CRC at the end
    You can recognize it due to the presence of the 0x78 byte at the beginning of the compressed data
Just for the record, there is also the Gzip [url=http://www.ietf.org/rfc/rfc1952.txt‎]RFC1952[/url] compression but it's mainly a container rather than a stream compression and it supports other algorithms, not only deflate.

The problem of compression algorithms is that without a CRC it's impossible to know if the decompressed data is valid or just a sort of "false positive".

In this case zlib gives excellent results and we can really scan a whole file extracting all the compressed streams almost without false positives.

Instead with deflate we will get many false positives and so it's up to us to understand if it's used for real or not.

There is a command-line tool of mine which is very helpful to scan the zlib and deflate streams, it's called offzip.
Like all the tools of mine there is a runtime help that is displayed when you launch the tool without arguments, so refer to it for any additional option.

The quick command-line example is the following:
offzip -S file.dat 0 0

It will scan the file "file.dat" searching for zlib compressed streams and will return the offset, the compressed and decompressed size and the total number of data found.

It's also possible to dump the results directly in a folder:
offzip -a file.dat output_folder 0

If you want to scan the file for raw deflate compressed data use the following:
offzip -z -15 -S file.dat 0 0

The "-z" option specifies the windowBits value used by the [url=http://zlib.net]zlib[/lib] library where a positive number is used for zlib and a negative one for deflate.

The following is an example of successful zlib scan:
Offset file unzipper 0.3.5
by Luigi Auriemma
e-mail: aluigi@autistici.org
web:    aluigi.org

- open input file:    steamservice.idb
- zip data to check:  32 bytes
- zip windowBits:     15
- seek offset:        0x00000000  (0)

| hex_offset | blocks_dots | zip_size --> unzip_size |
  0x00000105  ...................... 4342469 --> 18112512
  0x004243d3  ...................... 1687404 --> 11886592
  0x005c0348  ........ 14626 --> 49152
  0x005c3c73  . 601 --> 1380

- 4 valid zip blocks found

And how you know if there are false positives?
Simple, you will see lot of dots without the compressed/uncompressed sizes on the right and many error messages.
With deflate (-z -15) don't trust the results, often they are false positives except if you see many subsequent sequences.

Personally I find offzip very useful when I work on not-so-simple archives and I want a quick way to retrieve the compressed and uncompressed size values that I can search in the file with a hex editor to locate the header.
In this case I use the -x option that dumps the size values in hexadecimal format.
I take one of the two value, search it in 32bit little endian with the hex editor and then I check if there is also the other value close to it.
After you locate the header containing information about the archived files, it's more easy to have an idea of the structure used for each entry.

Just to recap, if you want to:
  • know if there are zlib compressed streams: offzip -S file.dat 0 0
  • know if there are deflate compressed streams (false positives!): offzip -S -z -15 -q file.dat 0 0
  • dump the zlib results: offzip -a file.dat c:\output_folder 0
  • dump the results in one unique file, useful when there are chunked files: offzip -a -1 file.dat c:\output_folder 0
  • analyze an archive to retrieve the index table containing offset/zsize/size: offzip -S -x file.dat 0 0
  • dump the results in a file (maybe to use later with packzip [work-in-progress!]): offzip -a -L c:\output_folder\dump.txt file.dat c:\output_folder 0

Hello, I wanted to ask a question, do you have a complete guide on how to learn how to decipher 3d models of headers in hexadecimal format, etc?

PostPosted: Sat Dec 10, 2022 4:31 pm 

Joined: Sat Dec 10, 2022 4:20 pm
Posts: 1
Is there a way to solve the problem "no valid full zip data found", I can't make another save because other saves had lost, I've only got this one.

Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 24 posts ]  Go to page Previous 1 2

All times are UTC

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited