ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Wed Feb 24, 2021 7:54 pm

All times are UTC




Post new topic  Reply to topic  [ 5 posts ] 
Author Message
PostPosted: Mon Feb 08, 2021 8:25 am 

Joined: Sun Jul 26, 2020 11:28 am
Posts: 4
Hey, aluigi! I've been using your fat2_fat3 bms script for a while now and it's been working mostly fine! However it seems when decompressing certain files in .dat file, random numbers of bytes at EOF could go "missing", resulting incomplete extraction.

Here is an example for one file (generated\databases\generic\shopitemsubtype.lib):

Image
(xbox360 vs pc and .lib file is cross platform in little endian)

Here is the archive I extracted it from: https://mega.nz/file/IXpygbzZ#XCKswR0YI ... FR6Gbamtfw

Another example when more bytes at EOF go missing (graphics\_materials\grassam-m-20120326110838.material.bin in multicommon.dat)
Attachment:
File comment: original file
original_grassam-m-20120326110838.material.bin [1.09 KiB]
Downloaded 15 times

I manually "fixed" it by comparing to PC file:
Attachment:
File comment: fixed file
fixed_grassam-m-20120326110838.material.bin [1.1 KiB]
Downloaded 13 times

Raw data stream cut from .dat file:
Filehash - 0x2b69ca0dbd7a1f5e;
Size - 1127;
ZSize - 490;
Offset - 99530512;
Attachment:
File comment: raw uncompressed data
raw_grassam-m-20120326110838.material.bin [490 Bytes]
Downloaded 14 times

From what I understand, lzma_dynamic has to deal with truncated data but it isn't "perfect" as we see here. Is there a solution to this problem?


Top
   
PostPosted: Wed Feb 17, 2021 4:49 am 

Joined: Sun Jul 26, 2020 11:28 am
Posts: 4
Bump


Top
   
PostPosted: Wed Feb 17, 2021 3:28 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 11982
The error reported by the lzma decompressor is LZMA_STATUS_NEEDS_MORE_INPUT.
I remember that the farcry stuff was a mess and I can't say if the compressed stream was invalid (as I remember) or there was an additional byte that we need to read (I doubt, maybe try uploading a sample of 490+1 bytes).

lzma_dynamic is correct in its work based on the provided input data.


Top
   
PostPosted: Sun Feb 21, 2021 2:36 am 

Joined: Sun Jul 26, 2020 11:28 am
Posts: 4
aluigi wrote:
The error reported by the lzma decompressor is LZMA_STATUS_NEEDS_MORE_INPUT.
I remember that the farcry stuff was a mess and I can't say if the compressed stream was invalid (as I remember) or there was an additional byte that we need to read (I doubt, maybe try uploading a sample of 490+1 bytes).

lzma_dynamic is correct in its work based on the provided input data.

Here is the raw data plus the next 5 bytes (basically the LZMA header of another compressed file). I found that when decompressing with lzma_dynamic, the last byte (of compressed data) isn't needed to get the same result, and the first byte of the raw data is the size (when it is not 0) of "uncompressed data" at EOF that usually needs to append to decompressed data.

However, there are many files starting with 0 and can't decompress correctly :(


Attachments:
File comment: +additional 5 bytes
raw+5byte_grassam-m-20120326110838.material.bin [495 Bytes]
Downloaded 3 times
Top
   
PostPosted: Sun Feb 21, 2021 2:44 am 

Joined: Sun Jul 26, 2020 11:28 am
Posts: 4
Here is an example of file with "uncompressed data"

Filehash - 0x46aa7a5dbd72be90;
Size - 1126;
ZSize - 484;
Offset - 99558983;


Attachments:
File comment: with 9 bytes of uncompressed data
raw_grassam-m-20120326165902.material.bin [484 Bytes]
Downloaded 3 times
Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 5 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited