ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Fri Nov 16, 2018 3:42 pm

All times are UTC




Post new topic  Reply to topic  [ 313 posts ]  Go to page Previous 112 13 14 15 16 Next
Author Message
PostPosted: Mon Aug 20, 2018 12:55 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
@haiwood
Please no off-topic.
I check most of the topics on the forum and if something is in the wrong section or lacks some details I fix it automatically.
And if I don't reply to a topic it means I don't have a positive solution at the moment.


Top
   
PostPosted: Mon Sep 10, 2018 7:19 am 

Joined: Sat Feb 06, 2016 12:00 am
Posts: 7
I decided to go back and extract/decrypt my Sega Genesis ROMS from my gold edition CD and Steam for archival purposes.

When I ran the 2 scripts(Sega_classics.bms and Sega_classics_gold.bms) on the QuickBMS page in 0.9.0 every file decrypted was incorrect. I ran the same exact scripts with old versions 0.7.5 and 0.7.3 and both correctly decrypted the .Pak files so I know the problem lies somewhere in the newer version. But I don't know what version the bug started in.

Additionally, I decided to compare the extracted files with a hex editor. What I found was that 0.9.0 added 3 bytes to the beginning of every ROM file decrypted. Every ROM started with these 3 bytes "EF BB BF" added. If you deleted these 3 bytes and saved, the edited files were identical to the versions created by the older QuickBMS versions.


Top
   
PostPosted: Thu Sep 13, 2018 6:54 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
Thanks for the feedback. Can you upload one of these roms so I can check it?

Some weeks ago I noticed the following post on reddit that may be related to the same bug (I have to check it but don't know if I will have all the files for the test):
https://www.reddit.com/r/FFXVPCmods/com ... _not_work/


Top
   
PostPosted: Tue Sep 18, 2018 7:22 am 

Joined: Sat Feb 06, 2016 12:00 am
Posts: 7
aluigi wrote:
Thanks for the feedback. Can you upload one of these roms so I can check it?


There is still an encrypted ROM from my gold edition in this previous thread: viewtopic.php?f=9&t=2101

It uses the script you added in that thread(sega_classics_gold.bms). The produced file was titled correctly regardless of which version of QuickBMS I used. If an older version of QuickBMS is used(I used .75) it should produce a use-able ROM. If the script is run in .90 it should create a ROM that is 3 bytes more(and they should be the bytes I posted above, at least that's what every ROM had added to it for me.) If I deleted the first 3 bytes of the .90 ROM with a hex editor it ended up the same as the ROM created with the older version.

From my testing, a ROM extracted from a Steam version .pak with the "sega_classics.bms" script was the same as a ROM extracted from a Gold edition .pak with the gold script(provided both paks were the same underlying game). This was the case as long as the same version of QuickBMS was used(I.E. the game extracted from .90 would be the same whether Steam or Gold edition(including the 3 extra bytes), and if .75 was used the steam and Gold would be the same as well).


Top
   
PostPosted: Wed Sep 19, 2018 9:43 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
Uhmmm I tried sega_classics_gold.bms with the g0009.pak sample you provided in that topic and the result is the same in quickbms 0.9.0 or 0.7.7 or 0.7.1.
I need an exact way to replicate the problem.


Top
   
PostPosted: Wed Sep 19, 2018 6:41 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
Ah, then do also another test.
Open quickbms.exe (0.9.0) with a hex editor, go at offset 0x95c686 and replace the byte 0x2c with 0x00.
Retry the script and let me know if now the result is good or wrong.


Top
   
PostPosted: Wed Sep 19, 2018 10:38 pm 

Joined: Sat Feb 06, 2016 12:00 am
Posts: 7
aluigi wrote:
Ah, then do also another test.
Open quickbms.exe (0.9.0) with a hex editor, go at offset 0x95c686 and replace the byte 0x2c with 0x00.
Retry the script and let me know if now the result is good or wrong.


The Hex change seems to have worked.

As far as how I was running it, I was using WINE command line "quickbms sega_classics_gold.bms g0009.pak"
The hex change you said though seems to have fixed it.


Top
   
PostPosted: Wed Sep 19, 2018 11:10 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
Perfect, now I know what was the problem :D


Top
   
PostPosted: Wed Sep 26, 2018 5:15 pm 

Joined: Fri May 27, 2016 2:28 pm
Posts: 53
NameCRC command always return a empty string, if using Temporary or Memory file as listfile.


Top
   
PostPosted: Wed Oct 03, 2018 1:26 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
Could you please provide an example?

Here it seems to work well:
Code:
set MEMORY_FILE10 binary "string1\nstring2\nmy_message\nstring3\n"
math NAME_CRC = 0x3c449c53

#test1
namecrc DUMMY 0 MEMORY_FILE10 32 0x04c11db7 "32 -1 -1 0 1 0"
namecrc NAME NAME_CRC

#test2
namecrc NAME NAME_CRC MEMORY_FILE10 32 0x04c11db7 "32 -1 -1 0 1 0"

print "%NAME%"


Top
   
PostPosted: Fri Oct 05, 2018 7:05 pm 

Joined: Fri Oct 05, 2018 6:52 pm
Posts: 5
Hi aluigi! First of all, your projects rock! Your work on integrating so much compression algos in quickbms is beyond appreciation!
I've been fiddling around with data from R6 Siege (they changed their packaging again, old scripts don't work anymore). I know that compressed chunks with compression type 0x1 are in zstd. I successfully uncompressed them with python binding of this lib. However, quickbms fails to decode those pieces. In my case it fails with calculated uncompressed length -20. I suppose, your implementation may have fell a bit behind from official releases.
Also, I managed to find lots of Export signatures from ZSTD in the exe of the game with x64dbg, but signsrch doesn't find any traces of ZSTD. Maybe new signatures?
If needed, I can provide you a sample chunk that quickbms fails to decompress. Or any other data you need.


Top
   
PostPosted: Fri Oct 05, 2018 7:40 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
zstd requires that both the compressed and uncompressed size are correct otherwise it returns the error -72 (ZSTD_error_srcSize_wrong), there is a long and detailed discussion here: viewtopic.php?p=37343#p37343

If you get the error/length -20 then it's the error ZSTD_error_corruption_detected that can be invoked in tons of places inside the zstd library, so it's impossible to know what caused it except that it's some malformed byte in the compressed stream.
quickbms simply calls the standard function ZSTD_decompress on the provided data so there is not much room for errors on its side :)

It's highly possible that the python wrapper adopts ZSTD_decompressStream instead of ZSTD_decompress, just like the .NET one described in the linked discussion.

The compressed chunk, the expected decompressed size and the decompressed chunk may be useful, yes.


Top
   
PostPosted: Fri Oct 05, 2018 8:30 pm 

Joined: Fri Oct 05, 2018 6:52 pm
Posts: 5
Dammit... I started gathering the data for you and noticed that comp_size from .forge and actual file's size differ. It turned out I was writing out files in text mode, not binary. So those extra bytes broke the data...
Now, after reexporting that chunk quickbms unpacked it with no problems at all. So false alarm here. Guess I should sleep more before coding and check my data before posting. xD
The only thing that remains is lack of zstd table's signatures in signsrch, but that's kinda out of topic for this thread.
Anyways, thank you for your time and effort! Complementi a maestro!


Top
   
PostPosted: Sat Oct 06, 2018 8:31 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
Eh signsrch would need some new signatures in the database but the problem is collecting all those that have been introduced (or where missing) since the last database update made some years ago.

In the meantime if you need support for zstd feel free to add the following to signsrch.sig:
Code:
################################################################################
TITLE
    zstd v0.1
TYPE
    32
DATA
    0x1eb52ffd

################################################################################
TITLE
    zstd v0.2
TYPE
    32
DATA
    0x22b52ffd

################################################################################
TITLE
    zstd v0.3
TYPE
    32
DATA
    0x23b52ffd

################################################################################
TITLE
    zstd v0.4
TYPE
    32
DATA
    0x24b52ffd

################################################################################
TITLE
    zstd v0.5
TYPE
    32
DATA
    0x25b52ffd

################################################################################
TITLE
    zstd v0.6
TYPE
    32
DATA
    0x26b52ffd

################################################################################
TITLE
    zstd v0.7
TYPE
    32
DATA
    0x27b52ffd

################################################################################
TITLE
    zstd v0.8
TYPE
    32
DATA
    0x28b52ffd


Top
   
PostPosted: Sat Oct 06, 2018 4:27 pm 

Joined: Wed May 23, 2018 1:00 pm
Posts: 39
Alugi, I've tried to extract latest fifa online 3. And i get this message:
the uncompressed data (-1) is bigger than the allocated buffer (7455533) Last script line before the error or that produced the error:
55 clog "" OFFSET ZSIZE SIZE
Please help


Top
   
PostPosted: Sat Oct 06, 2018 5:32 pm 

Joined: Fri Oct 05, 2018 6:52 pm
Posts: 5
Thanks for the signaturess! Idk, maybe if you're interested, I can provide you a dict table for Titanfall2 compression algo. I found it's offset. From what I know it is lzma based, but not the exact copy. Alltho I didn't bother to check exactly due to lack of experience at the time. I also have an offset of an actual decompress function. Unfortunately, I haven't finished it's disassembly. I made a python port that works but only partially. There are still some errors in a branch or two that cause it to fail on certain archives.


Top
   
PostPosted: Sat Oct 06, 2018 8:12 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
@MITA996
This topic is not for games.
The topics for that game are viewtopic.php?t=1000 and viewtopic.php?t=3036
Just remember that raw_ea_10fb_dumper.bms is prone to false positives and there are no solutions for that.

@Ulibos
Have you already tried with the comtype_scan.bat/bms scanner?
If you are sure that it's a customized lzma (or lzma2) then it wouldn't be a simple job, currently I prefer to stay away from additional work :)


Top
   
PostPosted: Wed Oct 10, 2018 1:38 pm 

Joined: Fri Oct 05, 2018 6:52 pm
Posts: 5
I did check Titanfall2 yesterday. I ran signsrch and didn't find any meaningful clues about compression algos in that binary. It is quite possible tho that inited vars are packed because I later added the signatures I have from the game and it didn't find them either. Or I did smth wrong...
I also revisited my data. The archives are packed at bit level. Default value mask is 0xffffff, i.e. at least integers are encoded in 24 bits instead of 32. Further more, archive header is also encoded in such way. It goes like this:

$arg4 is a function's arg, by default it equals 0xffffff, i.e. a bit trimming mask

$b1: 6 bits - encodes length of next block in bits
$b2: $b1 bits - packed size (it's encoded in a fancy way. Irc basically you read $b1 bits long block and slap extra bit on top. Example: $b1=4, $b2=0b0010, result=0b10010)
$b3: bitlen($arg4) bits - length of trimming mask in bits (i.e. how long (in bits) ints are in this package)
$b4: bitlen($arg4) bits - another mask, generally $b4==$b3
some skipped space. there is a padding of 0xD bits, but the resulting shift is then rounded to length in bytes. Irc in some edge cases you can get extra byte padded depending on previous blocks length. Will need to lookup the math later.
$b5: bitlen($b3 or $b4) - unpacked size
As a result of all this stuff actual data starts from like byte 14, bit 5, so you have all your data bit shifted and need to account for that. TF2 reads blocks and than stitches them with bit masks. I don't know for sure, but it seems like lzma doesn't do such hardcore bit fuckery, so I quess it's a proprietary compression algo.
Please keep in mind, I'm writing this data from my memory so some blocks's sizes or bit masks might be slightly wrong. Actual decoded data is at home.


Top
   
PostPosted: Sun Oct 21, 2018 5:18 pm 

Joined: Fri May 27, 2016 2:28 pm
Posts: 53
Strange thing. In the last two versions: 8.4.0 - 9.0.0, ScanDir command return a memory allocation error (line 678) when any filter is specified. Even for one .txt in folder.


Top
   
PostPosted: Sun Oct 21, 2018 6:42 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9238
I guess that feature has been never tested since there were at least 2 bugs in it :D
Will be fixed in 0.9.1


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 313 posts ]  Go to page Previous 112 13 14 15 16 Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited