ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Fri Apr 20, 2018 8:25 pm

All times are UTC




Post new topic  Reply to topic  [ 11 posts ] 
Author Message
PostPosted: Sun Mar 18, 2018 1:06 am 

Joined: Sun May 01, 2016 10:06 pm
Posts: 61
Due to my recent frustrations with extracting textures from PhyreEngine-based games that use stock PhyreEngine asset format I've decided to make a Noesis script that parses the structure (at least important parts of it) and allows for viewing/extracting 2D textures.
Frustrations may get worse, as PhyreEngine got to the Switch.

Doesn't work with cubemaps and 3D textures but it may be possible to.
Based on ToCS II Acewell's script and it's pretty much done.

So far I've checked it work with:
[PC] SAO: Hollow Realization
[PC] Secret Ponchos
[PC] Trails of Cold Steel I/II (double checked with this)
[PC] Tokyo Xanadu eX+
[PC] Final Fantasy X HD - has PTextureCubeMap
[PC] Final Fantasy XII: The Zodiac Age - has PTextureCubeMap (in RGBA16F, no less!)
[PC] SAO: Re: Hollow Fragment - has PTexture3D
[PS3] OreImo Happy enD - one sample; these samples might be from the same game
[PS3] Super Heroine Chronicle - one sample
[PS3] Unnamed title from this thread
[PS4] Final Fantasy XII: The Zodiac Age
[PS4] Tokyo Xanadu eX+ (and this)

Feedback and samples to work out kinks are appreciated.


Attachments:
File comment: 0.2.2 - Additional safeguards against models and shaders and some code cleanup.
tex_PhyreEngine_phyre.zip [2.64 KiB]
Downloaded 13 times


Last edited by TheUkrainianBard on Thu Apr 19, 2018 1:28 pm, edited 30 times in total.
Top
   
PostPosted: Sun Mar 18, 2018 12:16 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 61
Changelog:
Code:
0.2.2 - Additional safeguards against models and shaders and some code cleanup.
0.2.1 - Fixed *everything* (hopefully) PS4 by actually using GNF image header data inside Phyre structure.
0.2 - Dirty support for PS4 DXT1/3/5/ARGB8 textures via faking a GNF header.
0.1.5 - Dropped ".dds."/".png."/".bmp." file name requirement. Now it's just ".phyre" extension (SAO: Re: Hollow Fragment PC textures have no extension at all for ex.)
0.1.4 - Added BC7 support.
0.1.3 - Added Phyre structure read support for PS4 textures (cannot show them properly yet).
0.1.2 - Added BC5 (used in FFXII Zodiac Age normals for ex.) and A8 support and a safeguard against cubemaps.
0.1.1 - Slight code cleanup.
0.1 - Initial release.

Bonus: Phyre structure of a PS3 OreImo Happy enD sample (copy and paste into your notepad):
Code:
1-0|PAssetReference
1-1|0x00000018|0x00000004|m_id = 0x00000001|1| @ 0x00000018
1-2|0x0000001c|0x00000004|m_asset = 0x0000000a|10| @ 0x0000001c
1-3|0x00000020|0x00000004|m_assetType = 0x00000003|3| @ 0x00000020
2-0|PBase
3-0|PClassDescriptor
4-0|PClusterHeader
5-0|PClusterHeaderGCM
5-4|0x00000048|0x00000004|m_vramBufferSize = 0x000bf400|783360| @ 0x00000048
5-5|0x0000004c|0x00000004|m_vramBufferAlignment = 0x00000080|128| @ 0x0000004c
5-6|0x00000050|0x00000004|m_hostBufferSize = 0x00000000|0| @ 0x00000050
5-7|0x00000054|0x00000004|m_hostBufferAlignment = 0x00000080|128| @ 0x00000054
6-0|PClusterHeaderBase
6-8|0x00000000|0x00000004|m_phyreMarker = 0x50485952|1346918738|PHYR @ 0x00000000
6-9|0x00000004|0x00000004|m_size = 0x00000058|88| @ 0x00000004
6-10|0x00000010|0x00000004|m_instanceListCount = 0x00000002|2| @ 0x00000010
6-11|0x00000008|0x00000004|m_packedNamespaceSize = 0x00000dd5|3541| @ 0x00000008
6-12|0x00000014|0x00000004|m_arrayFixupSize = 0x00000003|3| @ 0x00000014
6-13|0x00000018|0x00000004|m_arrayFixupCount = 0x00000001|1| @ 0x00000018
6-14|0x0000001c|0x00000004|m_pointerFixupSize = 0x0000000a|10| @ 0x0000001c
6-15|0x00000020|0x00000004|m_pointerFixupCount = 0x00000003|3| @ 0x00000020
6-16|0x00000024|0x00000004|m_pointerArrayFixupSize = 0x00000000|0| @ 0x00000024
6-17|0x00000028|0x00000004|m_pointerArrayFixupCount = 0x00000000|0| @ 0x00000028
6-18|0x0000002c|0x00000004|m_pointersInArraysCount = 0x00000000|0| @ 0x0000002c
6-19|0x00000030|0x00000004|m_userFixupCount = 0x00000002|2| @ 0x00000030
6-20|0x00000034|0x00000004|m_userFixupDataSize = 0x00000011|17| @ 0x00000034
6-21|0x00000038|0x00000004|m_totalDataSize = 0x000000f4|244| @ 0x00000038
6-22|0x0000003c|0x00000004|m_headerClassInstanceCount = 0x00000000|0| @ 0x0000003c
6-23|0x00000040|0x00000004|m_headerClassChildCount = 0x00000000|0| @ 0x00000040
6-24|0x0000000c|0x00000004|m_platformID = 0x47434d00|1195592960|GCM @ 0x0000000c
6-25|0x00000044|0x00000004|m_physicsEngineID = 0x00000000|0| @ 0x00000044
7-0|PInstanceListHeader
7-26|0x00000000|0x00000004|m_classID[1] = 0x00000001|1| @ 0x00000e2d
7-26|0x00000000|0x00000004|m_classID[2] = 0x0000000a|10| @ 0x00000e51
7-27|0x00000004|0x00000004|m_count[1] = 0x00000001|1| @ 0x00000e31
7-27|0x00000004|0x00000004|m_count[2] = 0x00000001|1| @ 0x00000e55
7-28|0x00000008|0x00000004|m_size[1] = 0x00000054|84| @ 0x00000e35
7-28|0x00000008|0x00000004|m_size[2] = 0x000000a0|160| @ 0x00000e59
7-29|0x0000000c|0x00000004|m_objectsSize[1] = 0x00000028|40| @ 0x00000e39
7-29|0x0000000c|0x00000004|m_objectsSize[2] = 0x000000a0|160| @ 0x00000e5d
7-30|0x00000010|0x00000004|m_arraysSize[1] = 0x0000002c|44| @ 0x00000e3d
7-30|0x00000010|0x00000004|m_arraysSize[2] = 0x00000000|0| @ 0x00000e61
7-31|0x00000014|0x00000004|m_pointersInArraysCount[1] = 0x00000000|0| @ 0x00000e41
7-31|0x00000014|0x00000004|m_pointersInArraysCount[2] = 0x00000000|0| @ 0x00000e65
7-32|0x00000018|0x00000004|m_arrayFixupCount[1] = 0x00000001|1| @ 0x00000e45
7-32|0x00000018|0x00000004|m_arrayFixupCount[2] = 0x00000000|0| @ 0x00000e69
7-33|0x0000001c|0x00000004|m_pointerFixupCount[1] = 0x00000002|2| @ 0x00000e49
7-33|0x0000001c|0x00000004|m_pointerFixupCount[2] = 0x00000001|1| @ 0x00000e6d
7-34|0x00000020|0x00000004|m_pointerArrayFixupCount[1] = 0x00000000|0| @ 0x00000e4d
7-34|0x00000020|0x00000004|m_pointerArrayFixupCount[2] = 0x00000000|0| @ 0x00000e71
8-0|PString
8-35|0x00000000|0x00000004|m_buffer = 0x7265736f|1919251311|resource/image_convert/src/tweeticon_00.dds @ 0x00000e9d
9-0|PTexture2DGCM
9-36|0x00000024|0x00000050|m_samplerState
10-0|PTexture2D
11-0|PTexture2DBase
11-37|0x0000001c|0x00000004|m_width = 0x00000240|576| @ 0x00000ee5
11-38|0x00000020|0x00000004|m_height = 0x00000154|340| @ 0x00000ee9
12-0|PTextureCommonBase
12-39|0x00000000|0x00000004|m_format = 0x50546578|1347708280|PTexture2D @ 0x00000f69
12-40|0x00000005|0x00000001|m_memoryType = 0x41524742|1095911234|ARGB8 @ 0x00000f74
12-41|0x0000000c|0x00000004|m_mipmapCount = 0x00000009|9| @ 0x00000f7a
12-42|0x00000010|0x00000004|m_maxMipLevel = 0x0000000b|11| @ 0x00000f7e
12-43|0x00000014|0x00000004|m_textureFlags = 0x00000000|0| @ 0x00000f82
13-0|PTextureSamplerStateGCM
13-44|0x00000004|0x0000004c|m_buffers
14-0|PSharray<PTextureBufferGCM>
14-45|0x00000004|0x00000048|m_u
14-46|0x00000000|0x00000004|m_count
15-0|PTextureBufferGCM
15-47|0x00000008|0x00000018|m_gcmTextureBlock
15-48|0x00000020|0x0000001c|m_textureRegs
15-49|0x0000003c|0x00000004|m_offsetInAllocatedBuffer
16-0|CellGcmTexture
16-50|0x00000000|0x00000001|format
16-51|0x00000001|0x00000001|mipmap
16-52|0x00000002|0x00000001|dimension
16-53|0x00000003|0x00000001|cubemap
16-54|0x00000004|0x00000004|remap
16-55|0x00000008|0x00000002|width
16-56|0x0000000a|0x00000002|height
16-57|0x0000000c|0x00000002|depth
16-58|0x0000000e|0x00000001|location
16-59|0x0000000f|0x00000001|_padding
16-60|0x00000010|0x00000004|pitch
16-61|0x00000014|0x00000004|offset
17-0|PGpuSyncableGCM
18-0|PTextureRegistersGCM
18-62|0x00000000|0x00000004|m_offset
18-63|0x00000004|0x00000004|m_format
18-64|0x00000008|0x00000004|m_formatDisableCompare
18-65|0x0000000c|0x00000004|m_imageRect
18-66|0x00000010|0x00000004|m_control3
18-67|0x00000014|0x00000004|m_control1
18-68|0x00000018|0x00000004|m_control1DisableCompare


Last edited by TheUkrainianBard on Wed Apr 18, 2018 7:26 pm, edited 15 times in total.

Top
   
PostPosted: Sun Mar 25, 2018 5:31 pm 

Joined: Wed Nov 15, 2017 1:54 pm
Posts: 15
SAO has lots of textures in BC7 format, so you might want to add support for that. I don't know if noesis can decode bc7 though.

Also if your script works with SAO, it should also work with the Secret of Mana remake on pc. You can add that to the list.


Top
   
PostPosted: Sun Mar 25, 2018 6:24 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 61
Thank you.
Noesis can decode BC7:
Code:
output = rapi.imageDecodeDXT(input, Width, Height, noesis.FOURCC_BC7)

Could you confirm it working with those SAO:HR BC7 textures and Secret of Mana remake?

The list is non-exhaustive but what I've personally confirmed working. Ideally this script should work with all PhyreEngine texture assets (at least on PC and PS3 for now).


Top
   
PostPosted: Sun Mar 25, 2018 6:45 pm 

Joined: Wed Nov 15, 2017 1:54 pm
Posts: 15
Too lazy to test it on files now, moved some of them to external hdd. I worked on both sao and secret of mana files and their structures are exactly the same. So it will work :D


Top
   
PostPosted: Tue Mar 27, 2018 4:29 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 61
I wonder if this is some sorta tiling taking place on PS4...
UPD: Well, I have extracted something somehow and those are GNF images


Attachments:
File comment: 3. Correct dimensions and vertical flip
(dxt5)imbg021.png [382.86 KiB]
Not downloaded yet
File comment: 2. Creating a GNF header inside Noesis script and importing it as .gnf
(dxt5)imbg021.dds.gnm.png [406.62 KiB]
Not downloaded yet
File comment: 1. Just using Morton order unswizzling
(dxt5)imbg021.dds.gnm.phyre.png [385.33 KiB]
Not downloaded yet


Last edited by TheUkrainianBard on Wed Apr 18, 2018 7:28 pm, edited 1 time in total.
Top
   
PostPosted: Thu Apr 05, 2018 4:11 pm 

Joined: Thu Apr 05, 2018 4:06 pm
Posts: 3
Thank you! Already solved my problem~


Attachments:
note_help000.dds.phyre.zip [314.06 KiB]
Downloaded 14 times
Top
   
PostPosted: Thu Apr 05, 2018 4:52 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 61
@klose_rinz
Hm, is that from PS4 re-release of Sen no Kiseki aka Sen no Kiseki Kai ?


Top
   
PostPosted: Fri Apr 06, 2018 3:57 pm 

Joined: Thu Apr 05, 2018 4:06 pm
Posts: 3
TheUkrainianBard wrote:
@klose_rinz
Hm, is that from PS4 re-release of Sen no Kiseki aka Sen no Kiseki Kai ?


It's from Tokyo Xanadu ex+. Sen no kiseki 3 and Sen no kiseki Kai are on 4.70+ version, and we cannot get the files.

Your code works well with the .dds.phyre file from this game.
But there are some problems when opening .png.phyre(ARGB8) files.

I change the code with the following, then I can get the picture, but the colors are not right.
You can view the vis_127 in the attachment. The left above of the picture is dark sky with nearly pure black color. But in the extracted picture, it's two different bandings with green and blue.
So I think the order of the color is not the problem, maybe there are some tricks in the pixels which are different from PS3 swizzling.

Code:
    if Platform == "GNM\x02":
        FakeGNF = FakeGNFHeader + data
        FakeGNF = rapi.loadTexByHandler(FakeGNF, ".gnf")
        FakeGNF.name = "test.gnf"
        data = rapi.imageGetTexRGBA(FakeGNF)
        try:
            if imgFmt == "ARGB8":
               if IsSwizzled == 1:
                  # data = rapi.imageFromMortonOrder(data, imgWidth, imgHeight, 4)
               data = rapi.imageDecodeRaw(data, imgWidth, imgHeight, "b8 g8 r8 a8")
            data = rapi.imageFlipRGBA32(data, imgWidth, imgHeight, 0, 1)
            texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, noesis.NOESISTEX_RGBA32))
        except:
            texList.append(FakeGNF)
    else:
        data = rapi.imageFlipRGBA32(data, imgWidth, imgHeight, 0, 1)
        texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, noesis.NOESISTEX_RGBA32))

    return 1


Also, if I manually change IsSwizzled to 1, the code "data = rapi.imageFromMortonOrder(data, imgWidth, imgHeight, 4)" will get exception.


Attachments:
title.png.pc.png [3.68 MiB]
Not downloaded yet
vis127.pngout.png [3.69 MiB]
Not downloaded yet
title.png.zip [4 MiB]
Downloaded 15 times
Top
   
PostPosted: Fri Apr 06, 2018 6:41 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 61
Yeah, that's the problem with having a very limited amount of samples. I didn't realise that phyre structure had all necessary GNF image data (I kinda did realise for GCM, but there was no real need to use them)...
Attached file is what I got after I've actually used that GNF image data.

I've updated the script in the first post.


Attachments:
title.png [3.65 MiB]
Not downloaded yet
Top
   
PostPosted: Sat Apr 07, 2018 6:54 pm 

Joined: Thu Apr 05, 2018 4:06 pm
Posts: 3
TheUkrainianBard wrote:
Yeah, that's the problem with having a very limited amount of samples. I didn't realise that phyre structure had all necessary GNF image data (I kinda did realise for GCM, but there was no real need to use them)...
Attached file is what I got after I've actually used that GNF image data.

I've updated the script in the first post.


Many thanks~


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 11 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited