ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Wed Dec 13, 2017 2:40 pm

All times are UTC




Post new topic  Reply to topic  [ 659 posts ]  Go to page 1 2 3 4 533 Next
Author Message
 Post subject: Mass Effect Andromeda
PostPosted: Thu Mar 16, 2017 6:42 pm 
User avatar

Joined: Sat Nov 05, 2016 2:19 pm
Posts: 20
I got the MEA EA and want to extract the files from it. This is the first time messing around with Frostbite 3 for me and I'm having trouble.

During my research I found the following helpful resources:

Questions:
  • .toc files, wtf? What are they used for? The .sb files look more like a table of content. Are the .sb files enough to unpack?
  • .toc encryption? In other games if the file is encrypted it starts with x00D1CE. In MAE it doesn't but always starts with x??D1 and ends with
    Code:
    B9 73 E9 D4 88 43 59 08 A5 9F 30 6A E4 D2 79 94
    B2 BB D1 7F 75 F4 D0 D1 01 00 00 40 65 21 61 64
    6E 58 64 24 5E 21 72 66 4F 73 72 44 79 49 72 49
    21 78 56 67 48 65 41 21 36 56 63
    are they encrypted? The D1 makes me think so.
  • How does variable4 work? It says in the manual it's a Frostbite specific type.

I've uploaded a sample of .toc/.sb pairs here and a sample of .cas/.cat files here.

Any help is much appreciated :)


Top
   
PostPosted: Fri Mar 17, 2017 3:43 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7239
These toc files look obfuscated yes.
Two weird things: the second byte is ever 0xd1 even if the others are ever different and that 0xd0 0xd1 0x01 0x00 0x00 "@e!adnXd$^!rfOsrDyIrI!xVgHeA!6Vc" at the end.

To know how the data is read in variable4 take a look at src\file.c searching "variable4":
Code:
            i = 0;
            do {
                c = fgetxx(fdnum, 1, error);
                if(*error) break;
                retn |= ((c & 0x7f) << i);
                i += 7;
            } while(c & 0x80);


Top
   
PostPosted: Fri Mar 17, 2017 9:44 pm 

Joined: Fri Mar 17, 2017 9:41 pm
Posts: 14
Hey i found this but don't work for me with extraction files =), if help you =)

http://files.1337upload.net/BF4_Python_ ... 18017e.rar


Last edited by apollon08170 on Fri Mar 17, 2017 10:17 pm, edited 1 time in total.

Top
   
PostPosted: Fri Mar 17, 2017 10:16 pm 

Joined: Fri Mar 17, 2017 9:41 pm
Posts: 14
When i execute bf4dumper.py with files in path

Code:
#This script runs through all toc files it can find and uses that information to extract the files to a target directory.
#Often the assets are actually stored in cascat archives (the sbtoc knows where to search in the cascat), which is taken care of too.
#The script does not overwrite existing files (mainly because 10 sbtocs pointing at the same asset in the cascat would make the extraction time unbearable).

#Adjust paths here.

bf4Directory    = r"C:\Users\Apollon\Desktop\EXTRACTION\DATA"
targetDirectory = r"C:\Users\Apollon\Desktop\EXTRACTION\extract" #do yourself a favor and don't dump into the Users folder (or it might complain about permission)


###The following paths do not require adjustments (unless the devs decided to rename their folders). Note that they are relative to bf4Directory.

#As files are not overwritten, the patched files need to be extracted first.
#The script will dump all tocs it can find in these two folders+subfolders:
tocRoot  = r"Update" #patched and xpack files FIRST
tocRoot2 = r"Data"   #unpatched vanilla files SECOND

#Note: The "Update" tocRoot contains both patch (for vanilla AND xpack) and unpatched xpack files. The reason it still
#      works correctly is because it goes through the folders alphabetically, so the patch comes first.


#Feel free to comment out one or both cats if they don't exist (some Frostbite 2 games shipped without cats).
#Although in that case you could just as well use an invalid path, i.e. not change anything.
catPath       = r"\cas.cat"
updateCatPath = r"\cas.cat"

.........
.....
...


The cas.cat file is "not found" ?

Code:
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
========= RESTART: C:\Users\Apollon\Desktop\EXTRACTION\bf4dumper.py =========
Unpatched cat not found.
Patched cat not found.
>>>


Do you have an idea ?


Top
   
PostPosted: Sun Mar 19, 2017 9:08 am 

Joined: Sun Mar 19, 2017 9:07 am
Posts: 6
I tried to use the script for FIFA17, no good. Hope someone can find why!


Top
   
PostPosted: Sun Mar 19, 2017 3:43 pm 
User avatar

Joined: Tue Feb 02, 2016 2:35 am
Posts: 981
I have never expected Bioware to get their hands on Frostbite 3 AGAIN.

If you want to dump just the .cas/.cat files then use my modified script. I want to know if it works on this "game" after you`ve done using it.


Attachments:
frostbite.rar [3.22 KiB]
Downloaded 173 times

_________________
Expect my username to be everywhere, whether you're using the internet or not.
Top
   
PostPosted: Sun Mar 19, 2017 6:59 pm 

Joined: Thu Aug 13, 2015 5:08 pm
Posts: 21
AnonBaiter wrote:
I have never expected Bioware to get their hands on Frostbite 3 AGAIN.

If you want to dump just the .cas/.cat files then use my modified script. I want to know if it works on this "game" after you`ve done using it.


Doesn't work for me, tries to open "cas_-446005142.cas" after pointing the script at a cas.cat file when the files are named cas_01.cas, cas_02.cas etc.

Code:
- enter in folder C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage
  coverage file 0     0%   0          8637248    . offset 00000000
- open input file C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage\cas.cat
- enter in folder C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage
- open input file C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage\cas_-446005142.cas

- error in src\file.c line 247: fdnum_open()
Error: No such file or directory

Last script line before the error or that produced the error:
  429 open FDSE TMP 1

Press ENTER or close the window to quit


Top
   
PostPosted: Sun Mar 19, 2017 7:24 pm 
User avatar

Joined: Tue Feb 02, 2016 2:35 am
Posts: 981
Oh god.

Try to run quickbms through command-line and use -0 -V with the script, then the file in which you want to extract and post the results here after you`re done. After all I need to fix this even if I refuse to get the game by myself.

_________________
Expect my username to be everywhere, whether you're using the internet or not.


Top
   
PostPosted: Sun Mar 19, 2017 8:00 pm 

Joined: Thu Aug 13, 2015 5:08 pm
Posts: 21
AnonBaiter wrote:
Oh god.

Try to run quickbms through command-line and use -0 -V with the script, then the file in which you want to extract and post the results here after you`re done. After all I need to fix this even if I refuse to get the game by myself.


Code:
C:\Users\Snowpiercer\Documents\QuickBMS>quickbms -0 -V "C:\Users\Snowpiercer\Documents\QuickBMS\Scripts\frostbite_mea.bms" "C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage\cas.cat" "C:\Users\Snowpiercer\Documents\MEA_Extract"

QuickBMS generic files extractor and reimporter 0.8.0
by Luigi Auriemma
e-mail: me@aluigi.org
web:    aluigi.org
        (Mar 12 2017 - 15:03:46)

                   quickbms.aluigi.org  Homepage
                            zenhax.com  ZenHAX Forum
                               @zenhax  Twitter & Scripts

- current_folder: C:\Users\Snowpiercer\Documents\QuickBMS
- bms_folder:     C:\Users\Snowpiercer\Documents\QuickBMS\Scripts
- exe_folder:     C:\Users\Snowpiercer\Documents\QuickBMS
- file_folder:    C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage
- output_folder:  C:\Users\Snowpiercer\Documents\MEA_Extract
- temp_folder:    C:\Users\Snowpiercer\AppData\Local\Temp\
- open input file C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage\cas.cat
- open script C:\Users\Snowpiercer\Documents\QuickBMS\Scripts\frostbite_mea.bms
- set output folder C:\Users\Snowpiercer\Documents\MEA_Extract

  offset   filesize   filename
--------------------------------------
. 00000000 get     EXT        "cat" -1002
.
- enter in folder C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage
  coverage file 0     0%   0          8637248    . offset 00000000
- open input file C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage\cas.cat
. 0000022c idstr              "NyanNyanNyanNyan" 16
    4e 79 61 6e 4e 79 61 6e 4e 79 61 6e 4e 79 61 6e   NyanNyanNyanNyan
. 0000023c get     FILES      0x0003a923 4
. 00000240 getdstr ZERO       "" 20
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
    00 00 00 00                                       ....
. 00000244 getdstr HASH       "" 20
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
    00 64 1f 50                                       .d.P
. 00000258 get     FOFFSET    0xd288a5cd 4
. 0000025c get     FSIZE      0xf45b90c6 4
. 00000260 get     ZERO       0x2753e5e2 4
. 00000264 get     PACK_NUM   0xe56a806a 4
. 00000268 get     PACK_NAME  "cas" -1001
- enter in folder C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage
- open input file C:\Program Files (x86)\Origin Games\Mass Effect Andromeda\Data\Win32\streaminginstall\spinitialinstallpackage\cas_-446005142.cas

- error in src\file.c line 247: fdnum_open()
Error: No such file or directory

Last script line before the error or that produced the error:
  429 open FDSE TMP 1


Top
   
PostPosted: Sun Mar 19, 2017 8:08 pm 

Joined: Sun Mar 19, 2017 9:07 am
Posts: 6
AnonBaiter wrote:
Oh god.

Try to run quickbms through command-line and use -0 -V with the script, then the file in which you want to extract and post the results here after you`re done. After all I need to fix this even if I refuse to get the game by myself.


Code:
- open input file I:\Origin\Mass Effect Andromeda\Data\Win32\loc\en.toc
- open script C:\Users\Robin-PC\Desktop\quickbms\frostbite.bms
- set output folder O:\New folder

  offset   filesize   filename
--------------------------------------
- enter in folder I:\Origin\Mass Effect Andromeda\Data\Win32\loc
  coverage file 0     0%   0          115857     . offset 00000000
- open input file I:\Origin\Mass Effect Andromeda\Data\Win32\loc\en.TOC

- signature of 3 bytes at offset 0x00000000 doesn't match the one
  expected by the script:

  this one: "���"
  fd d1 e5                                          ...

  expected: ""
  00 d1 ce                                          ...

- 0 files found in 0 seconds
  coverage file 0     0%   3          115857     . offset 00000003

Press ENTER or close the window to quit


Here is what I got.


Top
   
PostPosted: Sun Mar 19, 2017 9:11 pm 
User avatar

Joined: Tue Feb 02, 2016 2:35 am
Posts: 981
I`m working on it. Don`t count on me for .sb/.toc files though.

EDIT: Okay, on the frostbite.bms script, replace this line:
Code:
   if ZERO != "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      goto -16 0 SEEK_CUR
   endif
With this line:
Code:
   if ZERO != "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      goto -16 0 SEEK_CUR
   else
      goto 0 0 SEEK_CUR
   endif
See if it works.

_________________
Expect my username to be everywhere, whether you're using the internet or not.


Last edited by AnonBaiter on Sun Mar 19, 2017 10:06 pm, edited 1 time in total.

Top
   
PostPosted: Sun Mar 19, 2017 9:36 pm 

Joined: Thu Aug 13, 2015 5:08 pm
Posts: 21
AnonBaiter wrote:
I`m working on it. Don`t count on me for .sb/.toc files though.

EDIT: Okay, on the frostbite.bms script, replace this line:
Code:
   if ZERO != "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      goto -16 0 SEEK_CUR
   endif
With this line:
Code:
   if ZERO != "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      goto -16 0 SEEK_CUR
   else
      # do nothing
   endif
See if it works.


Getting the exact same output as before.


Top
   
PostPosted: Sun Mar 19, 2017 9:54 pm 
User avatar

Joined: Sat Nov 05, 2016 2:19 pm
Posts: 20
aluigi wrote:
These toc files look obfuscated yes.
Two weird things: the second byte is ever 0xd1 even if the others are ever different and that 0xd0 0xd1 0x01 0x00 0x00 "@e!adnXd$^!rfOsrDyIrI!xVgHeA!6Vc" at the end.

To know how the data is read in variable4 take a look at src\file.c searching "variable4":
Code:
            i = 0;
            do {
                c = fgetxx(fdnum, 1, error);
                if(*error) break;
                retn |= ((c & 0x7f) << i);
                i += 7;
            } while(c & 0x80);

Thanks, how can I deobfuscate the .toc files? And can you tell me what is in them that isn't in the .sb files?

apollon08170 wrote:
Hey i found this but don't work for me with extraction files =), if help you =)

http://files.1337upload.net/BF4_Python_ ... 18017e.rar

I checked that out thank you.

AnonBaiter wrote:
I have never expected Bioware to get their hands on Frostbite 3 AGAIN.

If you want to dump just the .cas/.cat files then use my modified script. I want to know if it works on this "game" after you`ve done using it.

Can you tell me how to save the bundles? What's up with ebx, dbx, res, chunk, bmm?
And can you tell me what's usually in the .toc files? And why is there so many duplicates?

I can read cas.cat files and .sb files with the attached project, it's a VS C# program. Currently working on writing the bundles to disk.


Attachments:
MEA Unpack.zip [3.38 MiB]
Downloaded 330 times
Top
   
PostPosted: Sun Mar 19, 2017 10:12 pm 
User avatar

Joined: Tue Feb 02, 2016 2:35 am
Posts: 981
swinei wrote:
Can you tell me how to save the bundles? What's up with ebx, dbx, res, chunk, bmm?
And can you tell me what's usually in the .toc files? And why is there so many duplicates?
What? I don't have the game, and I won't intend to have it anytime soon. You don't want to know why.

_________________
Expect my username to be everywhere, whether you're using the internet or not.


Top
   
PostPosted: Sun Mar 19, 2017 10:22 pm 
User avatar

Joined: Sat Nov 05, 2016 2:19 pm
Posts: 20
AnonBaiter wrote:
What? I don't have the game, and I won't intend to have it anytime soon. You don't want to know why.

Now I'm really curios :P why not?

You seem to know the Frostbite engine that's why I ask. I don't have any other Frostbite games to look at. I can also upload more sample files if you need anything specific.


Top
   
PostPosted: Sun Mar 19, 2017 11:06 pm 
User avatar

Joined: Tue Feb 02, 2016 2:35 am
Posts: 981
swinei wrote:
You seem to know the Frostbite engine that's why I ask. I don't have any other Frostbite games to look at.
Well, to be honest though I'm just a beginner on this stuff.
swinei wrote:
I can also upload more sample files if you need anything specific.
Thanks for the offer, but I already have a Frostbite 3 game on my HDD. Which is enough since my Frostbite "research" is limited to games made by EA DICE with that engine. That habit of mine might change sometime though, just not today.

_________________
Expect my username to be everywhere, whether you're using the internet or not.


Top
   
PostPosted: Mon Mar 20, 2017 3:41 am 

Joined: Sun Mar 19, 2017 9:07 am
Posts: 6
AnonBaiter wrote:
swinei wrote:
You seem to know the Frostbite engine that's why I ask. I don't have any other Frostbite games to look at.
Well, to be honest though I'm just a beginner on this stuff.
swinei wrote:
I can also upload more sample files if you need anything specific.
Thanks for the offer, but I already have a Frostbite 3 game on my HDD. Which is enough since my Frostbite "research" is limited to games made by EA DICE with that engine. That habit of mine might change sometime though, just not today.


So is there any possible to extract files from toc and sb files?


Top
   
PostPosted: Mon Mar 20, 2017 3:54 am 
User avatar

Joined: Tue Feb 02, 2016 2:35 am
Posts: 981
No ideas at the moment.

_________________
Expect my username to be everywhere, whether you're using the internet or not.


Top
   
PostPosted: Mon Mar 20, 2017 7:45 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7239
AnonBaiter wrote:
if ZERO != "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"[/code]See if it works.

That can't work because you can't use C escapes in conditions.
Strings containing escapes are allowed only in those few commands where it's written "cstring" in quickbms.txt (unfortunately this is an "heritage" of the original bms language).

If TOC is encrypted there is no way to extract data from it using the current script.


Top
   
PostPosted: Tue Mar 21, 2017 8:28 am 
User avatar

Joined: Sat Nov 05, 2016 2:19 pm
Posts: 20
The CAS entries are compressed with Zstandard. Yay comptype scanner :D
compression type = 0F 70

The resId for textures is 1809719482

So now I have the res entry with the DDS header info and the chunk entry with the texture data.

The DDS header info has the 16 byte chunk id at position 32.
The last 16 bytes is maybe the name, but multiple entries can have the same name hm.
Width and height shorts start at 22 (I think).

I've attached an example pair, any help figuring out the header would be much appreciated!

edit: I started on a 010 template with info from here and here
Code:
//------------------------------------------------
//--- 010 Editor v7.0.2 Binary Template
//
//      File:
//   Authors:
//   Version:
//   Purpose:
//  Category:
// File Mask:
//  ID Bytes:
//   History:
//------------------------------------------------

int mipOneEndOffset;
int mipTwoEndOffset;
int textureType;
int pixelFormat;
short unknown;

short depth;
short sliceCount;

short width;
short height;

byte numSizes;
byte firstMip;

int dummy;

char chunkID[16];

local int i;
for(i=0;i<15;i++){
    int mipSize;
}

int chunkSize;

byte dummyy[16];

int nameHash;
char name[16];


AnonBaiter wrote:
Well, to be honest though I'm just a beginner on this stuff.

Ha me too ;)
AnonBaiter wrote:
Thanks for the offer, but I already have a Frostbite 3 game on my HDD. Which is enough since my Frostbite "research" is limited to games made by EA DICE with that engine. That habit of mine might change sometime though, just not today.

Could you upload some decrypted .toc files please? I wanna know whats in them


Attachments:
dump.zip [30.76 KiB]
Downloaded 120 times


Last edited by swinei on Tue Mar 21, 2017 9:34 am, edited 1 time in total.
Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 659 posts ]  Go to page 1 2 3 4 533 Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited