ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Fri Jul 20, 2018 4:30 am

Unanswered topics | Active topics


»

All times are UTC



Post new topic  Reply to topic  Page 1 of 1
  [ 16 posts ] 
  Print view Previous topic | Next topic 
Author Message
puggsoy
 Post subject: XMemDecompress code
PostPosted: Tue Jan 06, 2015 1:11 am 
User avatar

Joined: Sat Dec 13, 2014 1:01 am
Posts: 161
I recently discovered that QuickBMS supports XMemDecompress, the compression method used in XNB files for XNA games. I was curious to look at the code itself but couldn't find a find that seemed to be related to it in the source.

Would you be able to direct me towards where it is in the source?
Top
   
Reply with quote  
Ekey
 Post subject: Re: XMemDecompress code
PostPosted: Tue Jan 06, 2015 6:33 am 

Joined: Sat Aug 09, 2014 2:34 pm
Posts: 759
source\libs\mspack\
Top
   
Reply with quote  
puggsoy
 Post subject: Re: XMemDecompress code
PostPosted: Tue Jan 06, 2015 7:56 am 
User avatar

Joined: Sat Dec 13, 2014 1:01 am
Posts: 161
Thanks Ekey :) I've heard that the method is also known as LZX so I assume that the relevant code is in lzx.h and lzxd.c.
Top
   
Reply with quote  
aluigi
 Post subject: Re: XMemDecompress code
PostPosted: Tue Jan 06, 2015 10:33 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
As far as I know the code in mspack is not 100% compatible with xmemdecompress, I remember to have tried it some months/years ago.
Top
   
Reply with quote  
Ekey
 Post subject: Re: XMemDecompress code
PostPosted: Tue Jan 06, 2015 9:39 pm 

Joined: Sat Aug 09, 2014 2:34 pm
Posts: 759
Well, if you want i can share module with defines the Xbox Compression APIs :)
Top
   
Reply with quote  
aluigi
 Post subject: Re: XMemDecompress code
PostPosted: Wed Jan 07, 2015 12:54 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
The Xbox SDK is the only place where is located the xmemdecompress code.
I'm not aware of open source versions, I heard that someone probably tried to do something for a tool related to a game (Unreal engine?) but I don't remember and if I tried it, it didn't work.
So check xcompress.h and xcompress.lib if you want to use xmemdecompress.
Top
   
Reply with quote  
puggsoy
 Post subject: Re: XMemDecompress code
PostPosted: Thu Jan 08, 2015 12:41 am 
User avatar

Joined: Sat Dec 13, 2014 1:01 am
Posts: 161
How does QuickBMS use XMemCompress? The XNB script works perfectly on some files I tried.
Top
   
Reply with quote  
aluigi
 Post subject: Re: XMemDecompress code
PostPosted: Thu Jan 08, 2015 8:07 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
xcompress.lib
Top
   
Reply with quote  
beedy
 Post subject: Re: XMemDecompress code
PostPosted: Sat Aug 26, 2017 9:24 am 

Joined: Sat Aug 26, 2017 8:09 am
Posts: 8
I ask my question in this topic.

I need help to re-compressing Xbox 360 NHL Legacy chunklzx compressed texture packs using quickbms. I copied attached part of fightnight.bms to get compression algorithm and I can now decompress files correctly but when I re-importing modified textures with quickbms it appears problems with a changed sizes.

Decompressed texlibs are same sized and every textures inside texlibs are also same sized, but compressed file sizes should be different sizes.
I have no idea what the lzx-compression do for files during the compression process, but the result of re-importing with quickbms is not perfect.

Is there way to compress the textures to chunklzx with xmemdecompress algorithm to a new file using quickbms or other way instead of re-injecting it inside original file?

Here you can see the result of re-importing/injecting using quickbms
http://imgur.com/a/kJnhJ

Attachment contains the sample file.

Code:
# Fight Night Champion / SSX Xbox 360 (*.big) (script 0.3.4)
# script for QuickBMS http://quickbms.aluigi.org

get NAME basename
get EXT extension
string NAME += "_unpacked."
string NAME += EXT

endian big
    comtype xmemdecompress
    savepos TMP
    goto OFFSET
    idstring "chunklzx"
    get DUMMY long
    get FULLSIZE long
    get SIZE long
    get CHUNKS long
    get DUMMY long
    get DUMMY long
    get DUMMY long
    get DUMMY long
    log MEMORY_FILE3 0 0
    append
    savepos OFFSET
    for j = 0 < CHUNKS
        math OFFSET x= 0x8
        for # very lame, made on the fly
            math T = OFFSET
            math T %= 0x10
            if T == 8
                break
            endif
            math OFFSET += 8
        next
        goto OFFSET

        get ZSIZE long
        get ZIP long
        savepos OFFSET
        if ZIP == 3
            clog MEMORY_FILE3 OFFSET ZSIZE SIZE
        elif ZIP == 4
            log MEMORY_FILE3 OFFSET ZSIZE
        else
            print "unknown ZIP type %ZIP%, contact me!"
            cleanexit
        endif
        math OFFSET += ZSIZE
    next j
    append
    get MYSIZE asize MEMORY_FILE3
    if MYSIZE != FULLSIZE
        print "Alert: MEMORY_FILE3 %MYSIZE% != %FULLSIZE%"
    endif
    log NAME 0 FULLSIZE MEMORY_FILE3


Attachments:
texlib_0_29_0.rx2.zip [1.85 MiB]
Downloaded 56 times
Top
   
Reply with quote  
aluigi
 Post subject: Re: XMemDecompress code
PostPosted: Sun Aug 27, 2017 10:25 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
If you just want to compress one file:
Code:
comtype xmemlzx_compress
get SIZE asize
clog "new.dat" 0 SIZE SIZE
Top
   
Reply with quote  
beedy
 Post subject: Re: XMemDecompress code
PostPosted: Mon Aug 28, 2017 5:20 pm 

Joined: Sat Aug 26, 2017 8:09 am
Posts: 8
Thank you aluigi for your answer. Yes I want to compress one file and then inject the file onto .big archive and then injecting the .big archive to original game .iso file using Xbox image browser. a File compression is the only problem to get perfect texturemods in game. More about that

When I compress uncompressed file with this:
Code:
comtype xmemlzx_compress
get SIZE asize
clog "new.dat" 0 SIZE SIZE

I get:
http://imgur.com/a/Hszrt
Should be:
http://imgur.com/a/zxF5a

How to fit that with chunklzx script? It should need parse a header for compressed file. 00 3B 10 00 is uncompressed file size, 00 00 00 0F is a number of chunks, what is 40 00 00 (get SIZE long)? I will keep learning but I'm very thankfully if you can help me whit that.
Top
   
Reply with quote  
aluigi
 Post subject: Re: XMemDecompress code
PostPosted: Mon Aug 28, 2017 6:20 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
Basically what you want is a rebuilder because chunklzx is not a compression, it's a file format with its fields and rules.
Additionally it's even not very "standard" in the sense that some fields are unknown and I used some work-arounds.
I don't write rebuilders so I can't help.

The script doesn't support reimporting for various reasons:
- the file is split in chunks, in theory quickbms can do the job but in practice users report ever problems
- usage of MEMORY_FILE to reassemble the file (but this is something that can be fixed)
- compression algorithms that will crash the game like lz4 viewtopic.php?p=5500#p5500 (I don't remember if xmem may give the same problems)

The only thing which is far from being a solution and still doesn't solve the possible third problem is modifying the script to extract the single chunks, reimporting will work well in that way. Let me know if you want to try and if that's ok for your needs.

The alternative is writing a rebuilder by yourself.
Top
   
Reply with quote  
beedy
 Post subject: Re: XMemDecompress code
PostPosted: Mon Aug 28, 2017 8:09 pm 

Joined: Sat Aug 26, 2017 8:09 am
Posts: 8
Thank you, I have never written any programs so it's quite challenging at this situaition. However I can understand hex code a little so it's a good start. I copied first 0x40000 (got this from chunklzx offset 0x10) bytes from uncompressed .rx2 to a new file. Then I do that xmemlzx_compress and first bytes of new.dat are identical with original chunklzx file. \o/

original chunklzx file starts at 0x30
http://imgur.com/3Gbsiwk
New compressed .dat file:
http://imgur.com/a/jkqyW
Top
   
Reply with quote  
beedy
 Post subject: Re: XMemDecompress code
PostPosted: Fri Sep 01, 2017 11:43 am 

Joined: Sat Aug 26, 2017 8:09 am
Posts: 8
Hi Aluigi!
I got 15 chunks of compressed data to a "new.dat" file using this:
Code:
endian big
CHUNKS = 15
OFFSET = 0
append
for i = 0 < CHUNKS
   if OFFSET > 0x340000
      SIZE = 0x31000
   else
      SIZE = 0x40000
   endif
   comtype xmemlzx_compress
   clog "new.dat" OFFSET SIZE SIZE
   math OFFSET += 0x40000
next i

I got chunks identical with original chunklzx file.

Original chunklzx has two long fields before every chunks: compressed size of chunk for example 00 01 00 2E and unknown 00 00 00 03. Could you help me how to add these fields before every chunks to "new.dat"?
Top
   
Reply with quote  
aluigi
 Post subject: Re: XMemDecompress code
PostPosted: Wed Sep 06, 2017 4:25 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
Code:
comtype xmemlzx_compress
endian big
math CHUNK_SIZE = 0x40000
get SIZE asize

set NAME string "new.dat"
log NAME 0 0
for OFFSET = 0 != SIZE
    math TMP = SIZE
    math TMP - OFFSET
    if TMP < CHUNK_SIZE
        math CHUNK_SIZE = TMP
    endif

    clog MEMORY_FILE OFFSET CHUNK_SIZE CHUNK_SIZE
    get CHUNK_ZSIZE asize MEMORY_FILE
    putvarchr MEMORY_FILE2 0 CHUNK_ZSIZE long
    putvarchr MEMORY_FILE2 4 3           long
    append
    log NAME 0 8           MEMORY_FILE2
    log NAME 0 CHUNK_ZSIZE MEMORY_FILE
    append
next OFFSET + CHUNK_SIZE
Top
   
Reply with quote  
beedy
 Post subject: Re: XMemDecompress code
PostPosted: Fri Sep 08, 2017 7:20 am 

Joined: Sat Aug 26, 2017 8:09 am
Posts: 8
Thank you! :D
Top
   
Reply with quote  
Display posts from previous:  Sort by  
Post new topic  Reply to topic  Page 1 of 1
  [ 16 posts ] 

»

All times are UTC

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited