ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Tue Sep 25, 2018 6:53 pm

All times are UTC




Post new topic  Reply to topic  [ 5 posts ] 
Author Message
PostPosted: Tue Dec 12, 2017 7:24 am 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 37
Hello,

Was wondering if the following script, http://aluigi.altervista.org/papers/bms/frostbite.bms and nfstr_blueprintbundle.bms
can be adjusted to add the following known resource types and once files get extracted they get the correct extensions (instead of .dat), thus preventing same file overwritten and making a mess.
Code:
    0xC6DBEE07: ".AnimatedPointCloud",
    0xD070EED1: ".AnimTrackData",
    0x51A3C853: ".AssetBank",   
    0x957C32B1: ".AtlasTexture",
    0x428EC9D4: ".BundleRefTableResource",
    0xAFECB022: ".CompiledLuaResource",
    0xF04F0C81: ".Dx11ShaderProgramDatabase",
    0xBCC7FB86: ".Dx11Texture",
    0x6BDE20BA: ".Dx12Texture",
    0xE565EB15: ".DxShaderDatabase",
    0x10F0E5A1: ".DxShaderProgramDatabase",
    0x5C4954A6: ".DxTexture",
    0x85AC783D: ".EAClothAssetData",
    0x387CA0AD: ".EAClothData",
    0x85EA8656: ".EAClothEntityData",
    0x70C5CB3E: ".EnlightenDatabase",
    0xE156AF73: ".EnlightenProbeSet",
    0x59CEEB57: ".EnlightenShaderDatabase",
    0xC6CD3286: ".EnlightenStaticDatabase",
    0x5BDFDEFE: ".EnlightenSystem",
    0xEF23407C: ".FifaPhysicsResourceData",
    0xE36F0D59: ".HavokClothPhysicsData",
    0x4864737B: ".HavokDestructionPhysicsData",
    0x91043F65: ".HavokPhysicsData",
    0xEB228507: ".HeadMorphResource",
    0x9C4FAA17: ".HeightfieldDecal",
    0x0DEAFE10: ".IesResource",
    0xC78B9D9D: ".ImpulseResponse",
    0x36F3F2C0: ".IShaderDatabase",
    0xC417BBD3: ".ITexture",
    0x86521D6C: ".LinearMediaAsset",
    0xC611F34A: ".MeshEmitterResource",
    0x49B156D4: ".MeshSet",
    0x1091C8C5: ".MorphTargetsResource",
    0x31E779A2: ".MovieTexture",
    0xB2C465F6: ".NewWaveResource",
    0x30B4A553: ".OccluderMesh",
    0xC664A660: ".PamReplayResource",
    0x3B9D1688: ".PSDResource",
    0x319D8CD0: ".RagdollResource",
    0x3568E2B7: ".RawFileData",
    0x41D57E10: ".RenderTexture",
    0x7AEFC446: ".StaticEnlightenDatabase",
    0x2D47A5FF: ".SwfMovie",
    0x6BB6D7D2: ".Terrain",
    0x15E1F32E: ".TerrainDecals",
    0xA23E75DB: ".TerrainLayerCombinations",
    0x22FE8AC8: ".TerrainStreamingTree",
    0x9D00966A: ".UITtfFontFile",
    0x1CA38E06: ".VisualTerrain",
    0xEFC70728: ".ZoneStreamerGrid",
##    0x24A019CC:".DAIresType24A019CC",
##    0x5E862E05:".DAIresType5E862E05",
##    0x59C79990:".DAIresType59C79990",
##    0x76742DC8:".DAIresType76742DC8",
##    0x41759364:".NFSPresType41759364",
##    0x2EBF5E85:".NFSRresType2EBF5E85",

Basically to work same way a python dump script does and adds adequate extension to the files it finds in sb/toc,cas/cat files.
Is it doable?

cheers


Top
   
PostPosted: Mon Dec 18, 2017 12:33 am 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 37
.
your the only one that can answer I guess, is it doable? practical? useless? etc... hehe


Top
   
PostPosted: Mon Dec 18, 2017 8:28 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8837
Where is located that 32bit value? is it a little endian magic at offset 0? Like 0xEFC70728 -> 28 07 c7 ef


Top
   
PostPosted: Mon Dec 18, 2017 9:33 pm 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 37
aluigi wrote:
Where is located that 32bit value? is it a little endian magic at offset 0? Like 0xEFC70728 -> 28 07 c7 ef

id-daemon knows what's going on saying those are the actual names of the resTypes, so I don't know how they are "translated".
so basically "49B156D4=MeshSet", "6BDE20BA=Texture" so on and so forth.

As far as I can tell, or mostly speculating, its the value inside each bundle where is located in every sb/toc where it tells what files to extract, if a bundle has lets say 10ebx 10meshset 10resType those entries should hold that info, unfortunately I am not that knowledgeable, every python script made for any frostbite game has that info, not 100% sure though, located somewhere in these lines it should explain how it gets that offset/entry.
Code:
def hex2(num): return hexlify(pack(">I",num)) #e.g. 10 => '0000000a'
class Stub(): pass #generic struct for the cat entry

def readCat(catDict, catPath, rootPath):
    """Take a dict and fill it using a cat file: sha1 vs (offset, size, cas path)"""
    cat=cas.unXor(catPath)
    cat.seek(0,2) #get eof
    catSize=cat.tell()
    cat.seek(40) #skip nyan
    casDirectory=os.path.dirname(catPath)+"\\" #get the full path so every entry knows whether it's from the patched or unpatched cat.
    while cat.tell()<catSize:
        entry=Stub()
        sha1=cat.read(20)
        entry.offset, entry.size, dummy, casNum = unpack("<IIII",cat.read(16))
        entry.path=casDirectory+"cas_"+("0"+str(casNum) if casNum<10 else str(casNum))+".cas"
        #if (dummy==0 and entry.size>4000000):
        if dummy==0: catDict[sha1]=entry


def dump(tocPath, targetFolder):
    """Take the filename of a toc and dump all files to the targetFolder."""
    print "Dumping '%s'..." % tocPath
    #Depending on how you look at it, there can be up to 2*(3*3+1)=20 different cases:
    #    The toc has a cas flag which means all assets are stored in the cas archives. => 2 options
    #        Each bundle has either a delta or base flag, or no flag at all. => 3 options
    #            Each file in the bundle is one of three types: ebx/res/chunks => 3 options
    #        The toc itself contains chunks. => 1 option
    #
    #Simplify things by ignoring base bundles (they just state that the unpatched bundle is used),
    #which is alright, as the user needs to dump the unpatched files anyway.
    #
    #Additionally, add some common fields to the ebx/res/chunks entries so they can be treated the same.
    #=> 6 cases.

    toc=cas.readToc(tocPath)
    if not (toc.get("bundles") or toc.get("chunks")): return #there's nothing to extract (the sb might not even exist)
    sbPath=tocPath[:-3]+"sb"
    sb=open(sbPath,"rb")

    for tocEntry in toc.bundles:
        if tocEntry.get("base"): continue
        sb.seek(tocEntry.offset)

        ###read the bundle depending on the four types (+cas+delta, +cas-delta, -cas+delta, -cas-delta) and choose the right function to write the payload
        if toc.get("cas"):
            bundle=cas.Entry(sb)
            #make empty lists for every type to make it behave the same way as noncas
            for listType in ("ebx","res","chunks"):
                if listType not in vars(bundle):
                    vars(bundle)[listType]=[]
                   
            #The noncas chunks already have originalSize calculated in Bundle.py (it was necessary to seek through the entries).
            #Calculate it for the cas chunks too. From here on, both cas and noncas ebx/res/chunks (within bundles) have size and originalSize.
            for chunk in bundle.chunks:
                chunk.originalSize=chunk.logicalOffset+chunk.logicalSize
                   
            #pick the right function
            if tocEntry.get("delta"):
                writePayload=casPatchedPayload
                sourcePath=None #the noncas writing function requires a third argument, while the cas one does not. Hence make a dummy variable.
            else:
                writePayload=casPayload
                sourcePath=None
        else:
            if tocEntry.get("delta"):
                #The sb currently points at the delta file.
                #Read the unpatched toc of the same name to get the base bundle.
                #First of all though, get the correct path.

                #Does it work like this?
                #   Update\Patch\Data\Win32\XP1\Levels\XP1_003\XP1_003.toc
                #=> Update\Xpack1\Data\Win32\XP1\Levels\XP1_003\XP1_003.toc
                xpNum=os.path.basename(tocPath)[2] #"XP1_003.toc" => "1"
                split=tocPath.lower().rfind("patch")
                baseTocPath=tocPath[:split]+"xpack"+xpNum+tocPath[split+5:]
                if not os.path.exists(baseTocPath): #Nope? Then it must work like this:
                    #   Update\Patch\Data\Win32\XP1Weapons.toc
                    #=> Data\Win32\XP1Weapons.toc
                    baseTocPath=tocPath[:split-7]+tocPath[split+6:] #just cut out Update\Patch
                #now open the file and get the correct bundle (with the same name as the delta bundle)   
                baseToc=cas.readToc(baseTocPath)
                for baseTocEntry in baseToc.bundles:
                    if baseTocEntry.id.lower() == tocEntry.id.lower():
                        break
                else: #if no base bundle has with this name has been found:
                    pass #use the last base bundle. This is okay because it is actually not used at all (the delta has uses instructionType 3 only).
                   
                basePath=baseTocPath[:-3]+"sb"
                base=open(basePath,"rb")
                base.seek(baseTocEntry.offset)
                bundle = noncas.patchedBundle(base, sb) #create a patched bundle using base and delta
                base.close()
                writePayload=noncasPatchedPayload
                sourcePath=[basePath,sbPath] #base, delta
            else:
                bundle=noncas.unpatchedBundle(sb)
                writePayload=noncasPayload
                sourcePath=sbPath

        ###pick a good filename, make sure the file does not exist yet, create folders, call the right function to write the payload 
        for entry in bundle.ebx:
            targetPath=targetFolder+"/bundles/ebx/"+entry.name+".ebx"
            #if "sound/" in entry.name: continue
            if prepareDir(targetPath): continue
            writePayload(entry, targetPath, sourcePath)

        for entry in bundle.res: #always add resRid to the filename. Add resMeta if it's not just nulls. resType becomes file extension.
            targetPath=targetFolder+"/bundles/res/"+entry.name+" "+hexlify(pack(">Q",entry.resRid))
            #if not "d_assault_newera_skb_01" in entry.name: continue
            if entry.resMeta!="\0"*16: targetPath+=" "+hexlify(entry.resMeta)
            if entry.resType not in resTypes: targetPath+=".unknownres "+hex2(entry.resType)
            else: targetPath+=resTypes[entry.resType]
            if prepareDir(targetPath): continue
            writePayload(entry, targetPath, sourcePath)

        for i in xrange(len(bundle.chunks)): #id becomes the filename. If meta is not empty, add it to filename.
            entry=bundle.chunks[i]
            targetPath=targetFolder+"/chunks/"+hexlify(entry.id) +".chunk" #keep the .chunk extension for legacy reasons
            #if bundle.chunkMeta[i].meta!="\x00": targetPath+=" firstMip"+str(unpack("B",bundle.chunkMeta[i].meta[10])[0])
            #chunkMeta is useless. The same payload may have several values for firstMips so chunkMeta contains info specific to bundles, not the file itself.
            if prepareDir(targetPath): continue
            #if hexlify(entry.id)[:8]=="882CD8F1":
            writePayload(entry, targetPath, sourcePath)

    #Deal with the chunks which are defined directly in the toc.
    #These chunks do NOT know their originalSize.
    #Available fields: id, offset, size
    for entry in toc.chunks:
        targetPath=targetFolder+"/chunks/"+hexlify(entry.id)+".chunk"
        if prepareDir(targetPath): continue
        if toc.get("cas"):
            try:
                catEntry=cat[entry.sha1]
                #if not checkchunk(catEntry.path,catEntry.offset):
                #if hexlify(entry.id)[:8]=="1f175f8e":
                try:
                    process = subprocess.Popen(["fb_zstd.exe",catEntry.path,str(catEntry.offset),str(catEntry.size),targetPath],stderr=subprocess.PIPE,startupinfo=startupinfo)
                    process.communicate() #this should set the returncode
                    if process.returncode:
                        print process.stderr.readlines()
                except:
                    print "Error executing fb_zstd."
                    print catEntry.path,str(catEntry.offset),str(catEntry.size),targetPath
            except:
                continue
        else:
            if not checkchunk(sbPath,entry.offset):
                LZ77.decompressUnknownOriginalSize(sbPath,entry.offset,entry.size,targetPath)

    sb.close()

Sorry If I cant be of much help.


Top
   
PostPosted: Tue Dec 19, 2017 3:15 am 

Joined: Wed Mar 23, 2016 5:11 am
Posts: 37
what I forgot to mention is that, when we have an error like this from the python script,
Code:
KeyError: 947691693

it means that:
Code:
decimal 947691693->hex 387CA0AD->resType EAClothData

that's just an example, but I have no clue how all works, sorry


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 5 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited