ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Fri Jul 20, 2018 2:51 pm

All times are UTC




Post new topic  Reply to topic  [ 39 posts ]  Go to page Previous 1 2
Author Message
PostPosted: Sat Feb 24, 2018 3:08 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
doesn't exist "force" in reimport2.
NEVER use "force" in reimport mode.


Top
   
PostPosted: Wed Feb 28, 2018 7:24 pm 

Joined: Sun Jul 10, 2016 11:07 am
Posts: 60
wow, this is really great news!

cant wait to try it out.


Top
   
PostPosted: Sat Mar 03, 2018 10:01 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
quickbms 0.8.4 will return a warning if you use reimport2 on sequential content, for example chunked files or files saved with "savepos OFFSET ; log output.dat OFFSET 0x100" and so on. That will avoid to corrupt the archive without giving any explanation to the user since currently (0.8.3) the new data is appended at the end of the file and the game will still read the old content due to the lack of an offset reference (it's sequential).

Additionally it will also zero the original space if the file is bigger, that will save space when you will distribute the compressed edited archive.
I wasn't sure if enabling or not this feature but it offers also the advantage of knowing if something is wrong with the reimport operation, let's say something doesn't work as expected and the game tries to read the old content, it will be just zeroes (invalid) and it's easy to notice it (crash, errors, corrupted images/text).

Hopefully there will be no bugs with these two changes :D


Top
   
PostPosted: Tue May 08, 2018 5:24 pm 
User avatar

Joined: Mon Jun 27, 2016 5:45 pm
Posts: 6
This is an amazing step forward, now if only we could add new files then it would be perfect!


Top
   
PostPosted: Tue May 08, 2018 9:39 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
That's impossible because you can only edit the existent information.
Adding new files require the full 100% knowledge of the specific target format (and version) and a whole rebuilder.


Top
   
PostPosted: Mon Jun 25, 2018 4:25 pm 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
I used the reimport with a sequential file .... after the file was translated even though the new file = the original file size (uncompress size) .... but I was told "compress size bigger". If I make the smaller volume, my reimport file gets inserted "null" characters at the end and the game can not read ... do i have any solution for this?


Top
   
PostPosted: Wed Jun 27, 2018 3:23 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
reimport2 can't be used for files stored in sequential way.
You need to use reimport.bat and being sure that your compressed/uncompressed files is smaller/equal than the original file.
Remember to use the backup copy if you already tried reimport/reimport2 and it failed, just to be sure it didn't corrupt the file.


Top
   
PostPosted: Sat Jun 30, 2018 1:00 pm 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
aluigi wrote:
reimport2 can't be used for files stored in sequential way.
You need to use reimport.bat and being sure that your compressed/uncompressed files is smaller/equal than the original file.
Remember to use the backup copy if you already tried reimport/reimport2 and it failed, just to be sure it didn't corrupt the file.

i use reimport.bat The size of the text file is smaller than the original file, file gets inserted "null" characters at the end, and the game can not read this file... only text file. model, font, video...do not have this problem


Top
   
PostPosted: Sun Jul 01, 2018 3:35 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
That's exactly how it must work so it's perfect from my side.
Your file is short and therefore it fills the difference (old size - new size) with zeroes.
If you want to avoid that make a file with the same size of the original putting maybe spaces at the end if it's a text file.


Top
   
PostPosted: Sun Jul 01, 2018 12:02 pm 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
Thanks aluigi, I'm also using this way


Top
   
PostPosted: Mon Jul 02, 2018 2:30 pm 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
aluigi wrote:
That's exactly how it must work so it's perfect from my side.
Your file is short and therefore it fills the difference (old size - new size) with zeroes.
If you want to avoid that make a file with the same size of the original putting maybe spaces at the end if it's a text file.


sorry for bothering me many times, i really have a problem. After "add spaces" to new uncompress size = old uncompress file, but only a number of files active ... some files are still quickbms automatically added "null" ... my game can not load the file containing the signature self "null" ....
I add spaces with hex editor, and make sure the new file = the original file
You can modify my quickbms.exe file. Instead of adding "null", it will automatically add "spaces" if the size is smaller than the original file. I wait for help from you
This is the image depicting the problem

Image


Top
   
PostPosted: Mon Jul 02, 2018 10:04 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
If quickbms adds null bytes it means that the size of the file is not the same of the original one. Simple.

Do you have script and archive for doing a test? Upload them.
I can't promise nothing because if quickbms does that it means it's perfectly correct, I just want to do a check in case there is a rare bug.


Top
   
PostPosted: Tue Jul 03, 2018 2:45 am 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
aluigi wrote:
If quickbms adds null bytes it means that the size of the file is not the same of the original one. Simple.

Do you have script and archive for doing a test? Upload them.
I can't promise nothing because if quickbms does that it means it's perfectly correct, I just want to do a check in case there is a rare bug.


i use this script http://aluigi.zenhax.com/bms/qq_sfc.bms

I have made the uncompress size = uncompress size of the original file. But, "compress size" is not equal, compress size is smaller than original file can reimport .... I have no way to both "uncompress size" and "compress size" = original file ... I use hex editor only Can edit uncompress size = original file

Or you can check this script so that it can get names, folders (not sequentially). Then I can freely reimport the file using reimport2.bat


Top
   
PostPosted: Tue Jul 03, 2018 9:56 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
You can't use the reimport feature with that script for various reasons:
- sequential files require reimport.bat (no compressed/uncompressed size field modification, so no reimport2.bat)
- lz4 requires that the specified compressed/uncompressed size perfectly match the available data

Obviously it's rare that the compressed size matches the original one since it depends by the entropy of the file and the various settings and implementations of the compression algorithm.
Long story short: there is no solution except for manual hex editing or just writing a rebuilder from scratch.


Top
   
PostPosted: Thu Jul 05, 2018 2:42 pm 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
aluigi wrote:
You can't use the reimport feature with that script for various reasons:
- sequential files require reimport.bat (no compressed/uncompressed size field modification, so no reimport2.bat)
- lz4 requires that the specified compressed/uncompressed size perfectly match the available data

Obviously it's rare that the compressed size matches the original one since it depends by the entropy of the file and the various settings and implementations of the compression algorithm.
Long story short: there is no solution except for manual hex editing or just writing a rebuilder from scratch.


I still have to do you again ...
After using the hex editor, edit the size = the original file by "add spaces"
this is the original file:
Image

file editing:
Image

file after reimport:
Image

  I think there was a bug in the script or quickbms that caused quickbms to auto-fix, adding "null" to make it bigger than the original file. from 16,777 to 16,813 bytes

Here are 3 files: original, my edit, after reimport:
https://drive.google.com/open?id=1HuEBcsYq-OBQC8vswJhNN5GPDgrOo3Gn

You can check it again
thanks


Top
   
PostPosted: Thu Jul 05, 2018 5:17 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
Uhmmm no, there is no bug.

You can use only reimport.bat, NOT reimport2.bat as I said.

You didn't provide the original archive but I made a simple test here because the format is very simple so I created an archive from scratch and everything (extract->reimport->reextract) worked perfectly as expected.

If you have that wrong result with reimport.bat and quickbms 0.9.0 then upload the original archive.


Top
   
PostPosted: Thu Jul 05, 2018 6:22 pm 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
aluigi wrote:
Uhmmm no, there is no bug.

You can use only reimport.bat, NOT reimport2.bat as I said.

You didn't provide the original archive but I made a simple test here because the format is very simple so I created an archive from scratch and everything (extract->reimport->reextract) worked perfectly as expected.

If you have that wrong result with reimport.bat and quickbms 0.9.0 then upload the original archive.

i use reimport.bat and quickbms 0.90, not reimport 2.
Why reextract files (after reimport.dev) are resized?
original archive https://drive.google.com/open?id=1JAwJ7wKo7O_JA8W85YOUJYKviwBzVXtW


Top
   
PostPosted: Fri Jul 06, 2018 1:01 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 8468
Finally I can replicate the issue with the archive and I confirm what I wrote few posts above:
Quote:
- lz4 requires that the specified compressed/uncompressed size perfectly match the available data


Now the technical explanation:

quickbms uses the following code for performing the decompression:
Code:
size = LZ4_decompress_safe_partial(in, out, zsize, size, *outsize);
where size is the expected decompressed size (for example 0x4189 in your case) while *outsize is the size of the output buffer that can be any number bigger than size (for example 0x41ad).

lz4 doesn't have a termination bit so it continues to extract data till the result is:
Code:
targetOutputSize >= result < dstCapacity

Code:
/*!
LZ4_decompress_safe_partial() :
    This function decompress a compressed block of size 'srcSize' at position 'src'
    into destination buffer 'dst' of size 'dstCapacity'.
    The function will decompress a minimum of 'targetOutputSize' bytes, and stop after that.
    However, it's not accurate, and may write more than 'targetOutputSize' (but always <= dstCapacity).
   @return : the number of bytes decoded in the destination buffer (necessarily <= dstCapacity)
        Note : this number can also be < targetOutputSize, if compressed block contains less data.
            Therefore, always control how many bytes were decoded.
            If source stream is detected malformed, function returns a negative result.
            This function is protected against malicious data packets.
*/
LZ4LIB_API int LZ4_decompress_safe_partial (const char* src, char* dst, int srcSize, int targetOutputSize, int dstCapacity);

Your reimported file DOES NOT have the matching compressed size 0xa97, it has something smaller like 0xa8d but still lz4 will decompress bytes which means also all the padded data between 0xa97 and 0xa8d till the dstCapacity is full.
That's why you see the zeroes in your reextracted file.

quickbms is perfectly correct in what it does, indeed your game isn't able to load the edited file due to the unmatching compressed size.

End of the technical explanation.


Top
   
PostPosted: Fri Jul 06, 2018 3:03 am 

Joined: Tue Apr 24, 2018 9:56 am
Posts: 28
aluigi wrote:
Finally I can replicate the issue with the archive and I confirm what I wrote few posts above:
Quote:
- lz4 requires that the specified compressed/uncompressed size perfectly match the available data


Now the technical explanation:

quickbms uses the following code for performing the decompression:
Code:
size = LZ4_decompress_safe_partial(in, out, zsize, size, *outsize);
where size is the expected decompressed size (for example 0x4189 in your case) while *outsize is the size of the output buffer that can be any number bigger than size (for example 0x41ad).

lz4 doesn't have a termination bit so it continues to extract data till the result is:
Code:
targetOutputSize >= result < dstCapacity

Code:
/*!
LZ4_decompress_safe_partial() :
    This function decompress a compressed block of size 'srcSize' at position 'src'
    into destination buffer 'dst' of size 'dstCapacity'.
    The function will decompress a minimum of 'targetOutputSize' bytes, and stop after that.
    However, it's not accurate, and may write more than 'targetOutputSize' (but always <= dstCapacity).
   @return : the number of bytes decoded in the destination buffer (necessarily <= dstCapacity)
        Note : this number can also be < targetOutputSize, if compressed block contains less data.
            Therefore, always control how many bytes were decoded.
            If source stream is detected malformed, function returns a negative result.
            This function is protected against malicious data packets.
*/
LZ4LIB_API int LZ4_decompress_safe_partial (const char* src, char* dst, int srcSize, int targetOutputSize, int dstCapacity);

Your reimported file DOES NOT have the matching compressed size 0xa97, it has something smaller like 0xa8d but still lz4 will decompress bytes which means also all the padded data between 0xa97 and 0xa8d till the dstCapacity is full.
That's why you see the zeroes in your reextracted file.

quickbms is perfectly correct in what it does, indeed your game isn't able to load the edited file due to the unmatching compressed size.

End of the technical explanation.


I understand, sad that there is no solution. thank you


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 39 posts ]  Go to page Previous 1 2

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited