ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Tue Oct 23, 2018 4:56 pm

All times are UTC




Post new topic  Reply to topic  [ 31 posts ]  Go to page 1 2 Next
Author Message
PostPosted: Sun Mar 18, 2018 1:06 am 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Due to my recent frustrations with extracting textures from PhyreEngine-based games that use stock PhyreEngine asset format I've decided to make a Noesis script that parses the structure (at least important parts of it) and allows for viewing/extracting 2D textures.
Frustrations may get worse, as PhyreEngine got to the Switch.
Based on ToCS II Acewell's script and it's pretty much done.

What the script allows to do: preview and export supported 2D textures from PC/PS3/PSVita/PS4 in stock PhyreEngine asset format ("PHYR" or "RYHP" magic).
What the script doesn't allow to do: preview and export cubemaps (PTextureCubeMap), 3D textures (PTexture3D), atlas maps, models, shaders, fonts etc.

Titles confirmed working:
PC titles:PS3 titles:PSVita titles:PS4 titles:
Feedback and samples to work out kinks, add new features or add new formats supported are appreciated.


Attachments:
File comment: For QuickBMS: 0.4.2 - very slow when the texture is Morton order swizzled (about 2 pixels per MHz per second for Core i5)
PhyreEngine_PTexture2D_phyre.zip [4.8 KiB]
Downloaded 50 times
File comment: For Noesis: 0.4.2
tex_PhyreEngine_phyre.zip [4.64 KiB]
Downloaded 85 times
File comment: For QuickBMS: add ".phyre" extension to file based on it's "RYHP" magic.
So far was required for only one game: SAO: Re: Hollow Fragment.

add_phyre_extension.bms [96 Bytes]
Downloaded 84 times


Last edited by TheUkrainianBard on Thu Sep 20, 2018 5:29 pm, edited 44 times in total.
Top
   
PostPosted: Sun Mar 18, 2018 12:16 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Changelog:
Code:
0.4.2 - GCM: Fixed ARGB1555 and ARGB4444 decoding
           | Fixed RGBA8/ARGB8 decoding (0.4 regression fix)
           | Added L8 and LA88 support
        GCM, DX11: L8 and LA88 are upconverted to RGB and RGBA respectively so now they are viewed/exported as proper grayscale
                 | Workaround for no grayscale decoding in Noesis
        ALL: Eliminated almost all hardcoding
           | Added "keep import name" option - may or may not keep the right import filename
           | Changed "keep Block Compression" option to "keep DXT" - Noesis limitation, decodes BC5/7 to RGBA in any case
0.4.1 - GCM: Re-added DXT3 support (0.4 regression fix)
        GCM, DX11: Added experimental "keep Block Compression" option
                 | May not work for more advanced BCn, like BC5
        ALL: Made parsing a bit less hardcoded; code cleanup
0.4 - GCM: Added multiple textures in single file support
         | Added ARGB1555 and ARGB4444 support
      GCM, GNM: Made parsing a bit less hardcoded
0.3 - GXM: Added support via importing a complete GXT image present in asset structure
[Deprecated] 0.2.3+cubemaps - DX11: Added DXTn/BCn experimental cubemap support
                                  | Unlikely to keep it - too much effort to support
0.2.3 - ALL: Added some comments; code cleanup
0.2.2 - ALL: Added additional safeguards against models and shaders; code cleanup
0.2.1 - GNM: Fixed *everything* (hopefully) by using GNF image header data inside Phyre structure
0.2 - GNM: Dirty support for DXT1/3/5/ARGB8 textures via faking a GNF image header and importing as a GNF image
0.1.5 - ALL: Dropped ".dds."/".png."/".bmp." file name requirement
           | Now it's just ".phyre" extension
0.1.4 - DX11: Added BC7 support
0.1.3 - GNM: Added Phyre structure read support
           | Cannot show them properly yet
0.1.2 - DX11: Added BC5 and A8 support
        ALL: Added a safeguard against cubemaps
0.1.1 - Code cleanup
0.1 - Initial release


Last edited by TheUkrainianBard on Thu Sep 20, 2018 5:22 pm, edited 24 times in total.

Top
   
PostPosted: Sun Mar 25, 2018 5:31 pm 

Joined: Wed Nov 15, 2017 1:54 pm
Posts: 53
SAO has lots of textures in BC7 format, so you might want to add support for that. I don't know if noesis can decode bc7 though.

Also if your script works with SAO, it should also work with the Secret of Mana remake on pc. You can add that to the list.


Top
   
PostPosted: Sun Mar 25, 2018 6:24 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Thank you.
Noesis can decode BC7:
Code:
output = rapi.imageDecodeDXT(input, Width, Height, noesis.FOURCC_BC7)

Could you confirm it working with those SAO:HR BC7 textures and Secret of Mana remake?

The list is non-exhaustive but what I've personally confirmed working. Ideally this script should work with all PhyreEngine texture assets (at least on PC and PS3 for now).


Top
   
PostPosted: Sun Mar 25, 2018 6:45 pm 

Joined: Wed Nov 15, 2017 1:54 pm
Posts: 53
Too lazy to test it on files now, moved some of them to external hdd. I worked on both sao and secret of mana files and their structures are exactly the same. So it will work :D


Top
   
PostPosted: Tue Mar 27, 2018 4:29 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
I wonder if this is some sorta tiling taking place on PS4...
UPD: Well, I have extracted something somehow and those are GNF images


Attachments:
File comment: 3. Correct dimensions and vertical flip
(dxt5)imbg021.png [382.86 KiB]
Not downloaded yet
File comment: 2. Creating a GNF header inside Noesis script and importing it as .gnf
(dxt5)imbg021.dds.gnm.png [406.62 KiB]
Not downloaded yet
File comment: 1. Just using Morton order unswizzling
(dxt5)imbg021.dds.gnm.phyre.png [385.33 KiB]
Not downloaded yet


Last edited by TheUkrainianBard on Wed Apr 18, 2018 7:28 pm, edited 1 time in total.
Top
   
PostPosted: Thu Apr 05, 2018 4:11 pm 

Joined: Thu Apr 05, 2018 4:06 pm
Posts: 3
Thank you! Already solved my problem~


Attachments:
note_help000.dds.phyre.zip [314.06 KiB]
Downloaded 58 times
Top
   
PostPosted: Thu Apr 05, 2018 4:52 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
@klose_rinz
Hm, is that from PS4 re-release of Sen no Kiseki aka Sen no Kiseki Kai ?


Top
   
PostPosted: Fri Apr 06, 2018 3:57 pm 

Joined: Thu Apr 05, 2018 4:06 pm
Posts: 3
TheUkrainianBard wrote:
@klose_rinz
Hm, is that from PS4 re-release of Sen no Kiseki aka Sen no Kiseki Kai ?


It's from Tokyo Xanadu ex+. Sen no kiseki 3 and Sen no kiseki Kai are on 4.70+ version, and we cannot get the files.

Your code works well with the .dds.phyre file from this game.
But there are some problems when opening .png.phyre(ARGB8) files.

I change the code with the following, then I can get the picture, but the colors are not right.
You can view the vis_127 in the attachment. The left above of the picture is dark sky with nearly pure black color. But in the extracted picture, it's two different bandings with green and blue.
So I think the order of the color is not the problem, maybe there are some tricks in the pixels which are different from PS3 swizzling.

Code:
    if Platform == "GNM\x02":
        FakeGNF = FakeGNFHeader + data
        FakeGNF = rapi.loadTexByHandler(FakeGNF, ".gnf")
        FakeGNF.name = "test.gnf"
        data = rapi.imageGetTexRGBA(FakeGNF)
        try:
            if imgFmt == "ARGB8":
               if IsSwizzled == 1:
                  # data = rapi.imageFromMortonOrder(data, imgWidth, imgHeight, 4)
               data = rapi.imageDecodeRaw(data, imgWidth, imgHeight, "b8 g8 r8 a8")
            data = rapi.imageFlipRGBA32(data, imgWidth, imgHeight, 0, 1)
            texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, noesis.NOESISTEX_RGBA32))
        except:
            texList.append(FakeGNF)
    else:
        data = rapi.imageFlipRGBA32(data, imgWidth, imgHeight, 0, 1)
        texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, noesis.NOESISTEX_RGBA32))

    return 1


Also, if I manually change IsSwizzled to 1, the code "data = rapi.imageFromMortonOrder(data, imgWidth, imgHeight, 4)" will get exception.


Attachments:
title.png.pc.png [3.68 MiB]
Not downloaded yet
vis127.pngout.png [3.69 MiB]
Not downloaded yet
title.png.zip [4 MiB]
Downloaded 56 times
Top
   
PostPosted: Fri Apr 06, 2018 6:41 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Yeah, that's the problem with having a very limited amount of samples. I didn't realise that phyre structure had all necessary GNF image data (I kinda did realise for GCM, but there was no real need to use them)...
Attached file is what I got after I've actually used that GNF image data.

I've updated the script in the first post.


Attachments:
title.png [3.65 MiB]
Not downloaded yet
Top
   
PostPosted: Sat Apr 07, 2018 6:54 pm 

Joined: Thu Apr 05, 2018 4:06 pm
Posts: 3
TheUkrainianBard wrote:
Yeah, that's the problem with having a very limited amount of samples. I didn't realise that phyre structure had all necessary GNF image data (I kinda did realise for GCM, but there was no real need to use them)...
Attached file is what I got after I've actually used that GNF image data.

I've updated the script in the first post.


Many thanks~


Top
   
PostPosted: Fri Jul 27, 2018 11:11 am 

Joined: Fri Jul 27, 2018 10:44 am
Posts: 4
i would like to say thanks for this plugin, i used it for Secret of Mana and Tokyo Xanadu, very pleased with the outcome,
i noticed you added support for GXM for the PSVita, so i tried to use it on Dragon Quest Builders as the file name says it is a dds.GXM.phyre, but i get this error

Output extension has set output file type to:
.dds - DDS Image
Detected file type: PhyreEngine texture
[i] Platform: 'GXM\x01'.
Traceback (most recent call last):
File "C:\noesisv4176\plugins\python\tex_PhyreEngine_phyre.py", line 141, in noepyLoadRGBA
print("[i] Phyre asset import name: "+m_buffer)
TypeError: must be str without null bytes, not str

i get this kind of error on some files i try, i also tried the ps3 version which is dds.GCM.phyre, but that also puts out this error

Output extension has set output file type to:
.dds - DDS Image
Detected file type: PhyreEngine texture
[i] Platform: 'GCM'.
Traceback (most recent call last):
File "C:\noesisv4176\plugins\python\tex_PhyreEngine_phyre.py", line 141, in noepyLoadRGBA
print("[i] Phyre asset import name: "+m_buffer)
TypeError: must be str without null bytes, not str


but some images view and can be exported, just not the BG folder stuff which i am guessing that it is some kind of archives? if so how do i go about getting the images inside it out if it is possible, thank you advance.


Top
   
PostPosted: Sat Jul 28, 2018 10:15 am 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Please post files that don't work in any kind of archive (zip for ex). Or try commenting the line 141 by putting # at the very start of it.


Top
   
PostPosted: Sun Jul 29, 2018 3:13 pm 

Joined: Fri Jul 27, 2018 10:44 am
Posts: 4
thank you, it allowed me to view some textures, misaligned though, easy fix, though it won't veiw the other textures in these files, only some, i did find out that there is more than one format in there, using texture-finder, and was able to get more that way, allot more work but still could find more, here is a sample of a small file that i cannot view, and couldn't view with texture-finder either, maybe not a texture not sure, i know models are stored in there too

3mb file for blocks
1mb for environments

i am hoping to get more textures out, just would like an easier way, i have seen a few programs for other phyre engine games that gets just about everything out, just nothing for Dragon Quest Builders, only to unpack the psarc file, which is where i am now, and i know it uses phyre, the ps3 version is GCM and the vita version is GXM, but i am only able to get the ps3 version so far, some of the vita but not much, i don't mind which ever version works as long as i am happy to get get the files out, anyways thank you for the help it has gotten me more files, ill leave the samples here in hopes more people maybe can get them out...


Top
   
PostPosted: Mon Jul 30, 2018 1:09 am 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Here's a temporary solution specifically for those "one phyre file - multiple 2D textures inside" like your "blocks" file. But there is an enormous regression - it can't open anything else from PS3, so you have to shuffle files from this and the first post.
After I'll fix the regression - I'll delete the file from this post.

Even more samples are appreciated - maybe like a complete folder archived.


Last edited by TheUkrainianBard on Wed Aug 01, 2018 10:04 pm, edited 1 time in total.

Top
   
PostPosted: Tue Jul 31, 2018 5:37 pm 

Joined: Fri Jul 27, 2018 10:44 am
Posts: 4
omg thank you so much, it even got the bump map image too, i wasn't expecting to get that, but now very happy to have,

here is the whole data file, everything is in the buildgra folder for images that i am aware of

DQB 226mb archive

its large but it is everything, anyways thank you so much for helping, even though it is a 2 year old game.


Top
   
PostPosted: Tue Jul 31, 2018 7:06 pm 

Joined: Sat Feb 24, 2018 7:24 am
Posts: 4
TheUkrainianBard, can you add support for this sample:


Attachments:
battle.rar [164.86 KiB]
Downloaded 17 times
Top
   
PostPosted: Wed Aug 01, 2018 10:06 pm 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
@Rainx


Top
   
PostPosted: Thu Aug 02, 2018 6:40 am 

Joined: Sat Feb 24, 2018 7:24 am
Posts: 4
TheUkrainianBard wrote:
@Rainx

Error:
Code:
WARNING: Could not load external texture
Traceback (most recent call last):
  File "D:\noesis\plugins\python\tex_PhyreEngine_phyre.py", line 276, in noepyLoadRGBA
    FakeGNF.name = "test.gnf"
AttributeError: 'NoneType' object has no attribute 'name'


Top
   
PostPosted: Thu Aug 02, 2018 9:31 am 

Joined: Sun May 01, 2016 10:06 pm
Posts: 79
Strange, it worked for me and there was no regression for my PS4 samples.

Update: I've fixed regressions, first post updated.


Attachments:
File comment: DDS
battle.zip [164.7 KiB]
Downloaded 16 times
File comment: PNG
battle.png [334.19 KiB]
Not downloaded yet
Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 31 posts ]  Go to page 1 2 Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited