ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax
It is currently Tue Jul 05, 2022 3:03 pm

All times are UTC




Post new topic  Reply to topic  [ 19 posts ] 
Author Message
PostPosted: Tue Aug 05, 2014 1:44 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12982
Ubisoft use a specific file format for their games, you can recognize it by the 2TAF and 3TAF magic signature at the beginning of the archive.

The format is known to be used by:
FarCry 3 - FAT2
The Crew - FAT2
Watch Dogs - FAT3

The format is divided in two files:
- FAT containing the index table
- DAT containing only the data referenced by FAT

One of the bad things of formats used by multiple games and multiple platforms is the existence of multiple customizations.
For example the same archive on Xbox360, PS3 and PC is handled in different ways.

This difference is related mainly to the handling of compressed files because FAT3 on Xbox uses LZX (aka xmem), PC too (it's rare, maybe they use the same DX sdk) and PS uses zlib.
To make the situation even more difficult is the fact that the compression is chunked so there is no compressed and uncompressed size but a sequence of compressed parts of the file.

FAT2 is easier because doesn't use chunks but also in that case there are at least 2 different compression algorithms: lzo and lzma.

A bad, very bad, thing of this format is that its files are nameless, there is only a name CRC to identify them.

There are various tools to handle the format for specific games, moreover due to the name CRC limitation.

I worked on FAT2/3 just recently and I release a script for quickbms that should be compatible with all the games and all or almost all the platforms:
http://aluigi.org/papers/bms/fat2_fat3.bms

For reference, the following is a FAT2 file from The Crew:
Code:
32 54 41 46 05 00 00 00 01 00 00 00 2a 10 00 00   2TAF........*...
61 0d 00 00 00 00 00 00 10 d1 b9 82 73 fa 06 00   a...........s...
16 02 00 c0 3c 70 62 00 b1 e3 01 00 00 00 00 00   ....<pb.........
30 6c d8 3c 14 af 15 00 73 4e 00 40 a7 a5 90 00   0l.<....sN.@....
61 3a 00 00 00 00 00 00 b5 dc b4 62 73 27 1d 00   a:.........bs'..
1d 08 00 c0 d7 a3 24 00 e1 b4 00 00 00 00 00 00   ......$.........
cb b5 da 79 88 ec 2d 00 cc 16 00 c0 f4 a7 57 00   ...y..-.......W.
91 f1 00 00 00 00 00 00 c2 a6 dc 01 56 e1 37 00   ............V.7.
8b 21 00 40 24 f6 79 00 81 ad 0a 00 00 00 00 00   .!.@$.y.........
76 a5 e1 df a6 dd 48 00 f9 64 02 40 c4 be 0b 00   v.....H..d.@....

While the following is a FAT3 from Watch Dogs:
Code:
33 54 41 46 08 00 00 00 03 04 37 00 ab 12 00 00   3TAF......7.....
3b 06 23 00 82 05 02 00 00 11 00 80 1a 9d 97 00   ;.#.............
5c 02 30 00 82 05 02 00 f0 11 00 80 c2 54 97 00   \.0..........T..
3f 99 3e 00 82 85 00 00 30 03 00 e0 4b 24 17 00   ?.>.....0...K$..
9b 7d 52 00 00 00 00 00 a0 01 00 00 7d 03 20 00   .}R.........}. .
28 0e 63 00 82 05 02 00 30 11 00 80 20 0c 97 00   (.c.....0... ...
fa f3 67 00 82 05 08 00 c0 46 00 80 2d 23 83 00   ..g......F..-#..
95 04 6a 00 82 05 10 00 a0 89 00 80 33 5d 82 00   ..j.........3]..
cf 5f 6a 00 82 05 10 00 a0 cf 00 80 23 fa 8b 00   ._j.........#...
7b a3 6e 00 0a 1a 04 00 11 34 00 60 4c 0a 1f 00   {.n......4.`L...


Top
   
PostPosted: Fri Aug 22, 2014 10:34 am 

Joined: Fri Aug 22, 2014 10:30 am
Posts: 1
Hi How to open .loc file

Image


Attachments:
main_english.rar [561.19 KiB]
Downloaded 361 times
Top
   
PostPosted: Sun Dec 09, 2018 7:58 pm 

Joined: Sun Dec 09, 2018 7:53 pm
Posts: 2
Could you take a look at the format on The Crew 2 archives? This BMS script doesn't work with them.
I'd be really thankful if you could release an updated script for this!


Attachments:
sample.zip [4.44 MiB]
Downloaded 220 times
Top
   
PostPosted: Mon Dec 10, 2018 11:12 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12982
oodle compression, script 0.1.5:
http://aluigi.org/bms/fat2_fat3.bms


Top
   
PostPosted: Mon Dec 10, 2018 12:16 pm 

Joined: Sun Dec 09, 2018 7:53 pm
Posts: 2
Thank you!


Top
   
PostPosted: Sat Nov 13, 2021 7:37 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
aluigi wrote:
... A bad, very bad, thing of this format is that its files are nameless, there is only a name CRC to identify them.
...

Would it be a very bad idea if the script could extract them for their names as they are now but with their CRC64Bit hashes?
Would help a lot to identify them :) would be very helpful if you could update the script to support that.


Top
   
PostPosted: Sat Nov 13, 2021 8:22 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12982
Sure, at line 244 replace 'set NAME string ""' with:
Code:
xmath TMP_CRC "NAME_CRC >> 32"
string NAME p "%08x%08x" TMP_CRC NAME_CRC


Additionally I'm quite sure there is a bug at line 75 where "get DUMMY longlong" should be "get NAME_CRC longlong" other version 9 wouldn't have the correct crc associated.
Please let me know how it works.


Top
   
PostPosted: Sun Nov 14, 2021 3:46 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
aluigi wrote:
Sure, at line 244 replace 'set NAME string ""' with:....
...a bug at line 75 where "get DUMMY longlong" should be "get NAME_CRC longlong" ...
Please let me know how it works.

I would like to confirm both recommendations above work as expected, and the results are accurate. Thank you so much. The script can definitely be updated for future use that way no one will get confused what the files are as i did :?
Before:
Code:
  Offset           Filesize   Name
  00000000170f8ea7 263333     0000000000000000.dat
  000000003e7acfa3 22         0000000000000001.dat
  00000000247ba630 485613     0000000000000002.dat
  0000000083451d28 134246     0000000000000003.dat
  00000000836031e7 263333     0000000000000004.dat

After:
Code:
  Offest           FileSize   CRC64
  00000000170f8ea7 263333     0002CE399C7095C2
  000000003e7acfa3 22         00035ACDB463C178
  00000000247ba630 485613     0005161C58F1FAE8
  0000000083451d28 134246     000A94756572D236
  00000000836031e7 263333     000E9033917CE2FF

All these Ubisoft games built on Dunia engine, have actual file names but need to be manually edited/added, so how can i feed the list with the names so the assets can be extracted with file/folder structure based on their reversed hashes?

Example hashes above would be extracted like so in their file/folder structure:
Code:
project\road66\base\area\area_01\area_01_06\secteur_01_06_06_00\detailid_01_06_06_00.xbt
project\road66\generated\area\area_02\area_02_10\secteur_02_10_05_01\ai_02_10_05_01.van
project\road66\generated\area\area_01\area_01_09\secteur_01_09_07_01\ai_01_09_07_01.van
project\road66\generated\area\area_05\area_05_11\secteur_05_11_02_03\displacement_05_11_02_03.hms
project\road66\base\area\area_05\area_05_11\secteur_05_11_02_05\detailid_05_11_02_05.xbt


I tried something like:
Code:
quickbms.exe script.bms filenames.txt input output

And few more variations, but to no avail, is there a way to do it? I cant figure it out :(


Top
   
PostPosted: Sun Nov 14, 2021 7:17 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12982
Perfect.

For that feature you mention exists the NameCRC command.
I don't know what crc/hash algorithm is used on these filenames and if it's supported by the crc engine of quickbms.
If you know the algorithm filenames.txt must contain only the filenames, otherwise it must contain a hexadecimal crc followed by the name (like "1122334455667788 path\file.txt").

In that case would be enough to replace the whole content of the GET_NAME_FROM_CRC function with:
namecrc NAME NAME_CRC "filenames.txt" 64


Top
   
PostPosted: Sun Nov 14, 2021 10:56 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
aluigi wrote:
Perfect....I don't know what crc/hash algorithm is used on these filenames and if it's supported by the crc engine of quickbms....

Based on: http://aluigi.altervista.org/bms/quickb ... engine.txt looks like its supported, its the Jones variant
aluigi wrote:
In that case would be enough to replace the whole content of the GET_NAME_FROM_CRC function with:
namecrc NAME NAME_CRC "filenames.txt" 64

I did try, yet every single assets gets overwritten by a name like: (2).bin
And not sure i fully understand how to modify the function :(
Te names inside the filenames.txt are with out any other characters besides the actual path as described above for those 5 CRC64 hashes, no ""


Top
   
PostPosted: Mon Nov 15, 2021 5:16 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 12982
So I guess this is the correct command:
namecrc NAME NAME_CRC "filenames.txt" 64 0xad93d23594c935a9 "64 0 0 0 1 0"

Regarding the function I meant, from:
Code:
startfunction GET_NAME_FROM_CRC
    set NAME string ""
    if NFO_EXISTS != 0
        for z = 0 < NAMES
            getarray TMP_CRC 10 z
            if TMP_CRC == NAME_CRC
                getarray NAME 11 z
                math z = NAMES  # break is bugged
            endif
        next z
    endif
endfunction

To:
Code:
startfunction GET_NAME_FROM_CRC
    namecrc NAME NAME_CRC "filenames.txt" 64 0xad93d23594c935a9 "64 0 0 0 1 0"
    if NAME == ""
        xmath TMP_CRC "NAME_CRC >> 32"
        string NAME p "%08x%08x" TMP_CRC NAME_CRC
    endif
endfunction


Top
   
PostPosted: Tue Nov 16, 2021 9:45 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
Thank you, i tested both TC games and it works flawlessly.


Top
   
PostPosted: Fri Mar 18, 2022 10:59 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
The script will NOT export strings that happen to have a space in the entire full path, it gets skipped, how can i avoid that?
Ideal would be to take in to consideration entire string line, spaces or not.
Can that be achieved?


Top
   
PostPosted: Mon Apr 04, 2022 7:23 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
Would anyone else happen to know how to make the script extract the strings that have spaces in name at all?
Right now the whole string is skipped.


Top
   
PostPosted: Mon Apr 04, 2022 9:54 pm 

Joined: Sat Sep 28, 2019 7:00 pm
Posts: 630
I suppose more explanation about the issue is required. I think cyberspeed is talking about situation, when list with filenames contains strings with spaces, but CRC is not calculated properly for them. In the same time, explicit declaration works fine. For example, for CRC32 with default parameters:
[extracted\test file.bin] with NAME_CRC = 0x402C6A32 -> doesn't work (output will be 402C6A32.dat)
[0x402C6A32 extracted\test file.bin] with NAME_CRC = 0x402C6A32 -> works fine (output will be "extracted\test file.bin")

@cyberspeed: If you can calculate those crc manually, you can use list with pre-defined crc values, this way it will work as expected.

_________________
You can request AES keys on rin forums (the list with keys is also there)
AES keys finder and latest UE4 bms scripts: in this post


Top
   
PostPosted: Tue Apr 05, 2022 8:16 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 104
spiritovod wrote:
...I think cyberspeed is talking about situation, when list with filenames contains strings with spaces, but CRC is not calculated properly for them...

That is correct.
spiritovod wrote:
@cyberspeed: If you can calculate those crc manually, you can use list with pre-defined crc values, this way it will work as expected.

I always assumed the script is written to not support spaces, or that it would require more code to support them, using your suggestion works as it should, i just had no idea it would require the HASH to be included even though quickbms natively supports that CRC64 algo. Thanks non the less.
I was confused when Luigi said above, if the CRC is unknown the CRC64 hash must be provided but if its supported by quickbms it is not necessary, so now i know.


Top
   
PostPosted: Tue Apr 05, 2022 9:14 pm 

Joined: Sat Sep 28, 2019 7:00 pm
Posts: 630
@cyberspeed: Um, no, it's more like a quickbms bug. Like explained in the documentation, you may use list with paths only and list with pre-defined values + paths, both should work in the same way for supported crc schemes. For example:
[extracted\test2.bin] with NAME_CRC= 0x2AB02897 -> works fine (output will be "extracted\test2.bin")

I didn't dig into quickbms code, but it may happen if you'll use space as delimiter for detecting pre-defined value in a string, or something. For example, you have [0x402C6A32 extracted\test file.bin] and you need to know, if it contains pre-defined value or not - in that case you search for the first space and split the string into pre-defined value and path. But what if you have only path with a space - such string will be splited in the same way (though it shouldn't). It's just a guess though.

_________________
You can request AES keys on rin forums (the list with keys is also there)
AES keys finder and latest UE4 bms scripts: in this post


Top
   
PostPosted: Thu Jun 23, 2022 9:22 am 

Joined: Thu Jun 23, 2022 9:19 am
Posts: 1
Could you take a look at the format on X360 Far Cry 4 archives? This BMS script doesn't seem to work with them.
I'd be really thankful if you could release an updated script for this!

Sample file, 6MB: https://ufile.io/t5w8iy65


Top
   
PostPosted: Thu Jun 23, 2022 10:00 am 

Joined: Wed Jun 01, 2016 5:53 pm
Posts: 132
@Davida where did you get that *.NFO file? Beacause it contains filenames + hashed version. Post all of them if there are any others.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 19 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited