ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Wed Oct 17, 2018 10:43 am

All times are UTC




Post new topic  Reply to topic  [ 17 posts ] 
Author Message
PostPosted: Tue May 26, 2015 4:17 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Hello,
First off Very Great job! on this toolset. The tool extracts everything perfectly, however the reimport fails do to file size. I tried it by just extracting down to the file I wanted to edit and tried to reimport WITHOUT any edits just export then import and still get a file size difference error. This is what I get on importing the unchanged file. Any help would be greatly appreciated.

E:\Quickbms and Script-14-1-0>quickbms -G -w -r
QuickBMS generic files extractor and reimporter 0.6.3a
by Luigi Auriemma
e-mail: me@aluigi.org
web: aluigi.org
(May 6 2015 - 10:55:55)

http://quickbms.aluigi.org
http://twitter.com/luigi_auriemma
http://zenhax.com

- GUI mode activated, remember that the tool works also from command-line
where are available various options like folder scanning, filters and so on
- select the BMS script or plugin to use
- select the input archives/files to extract, type * or "" for whole folder and
subfolders
- select the output folder where extracting the files
- REIMPORT mode enabled!
- open input file E:\Quickbms and Script-14-1-0\patch.bundle
- open script E:\Quickbms and Script-14-1-0\witcher3.bms
- set output folder E:\Quickbms and Script-14-1-0\00_mod

offset filesize filename
--------------------------------------
Error: file "gameplay\items\def_loot_containers.xml"
the reimport option acts as a reimporter and so you cannot reinsert a
file if it's bigger than the original otherwise it will overwrite the
rest of the archive or cannot be loaded correctly:
new size: 31442 (1179468 uncompressed)
old size: 27897 (1179468 uncompressed)
- do you want to skip this file? (y/N/force)
y will continue with the next file
N (default) will terminate QuickBMS
force will force the reimporting of the file

Thanks,
Tom


Top
   
PostPosted: Tue May 26, 2015 8:20 pm 

Joined: Sat Aug 09, 2014 2:34 pm
Posts: 764
You can't reimport if modified file size is larger than the original.

tsanford01 wrote:
new size: 31442 (1179468 uncompressed)
old size: 27897 (1179468 uncompressed)


Top
   
PostPosted: Tue May 26, 2015 8:30 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
I know that the problem is the file was not modified or even opened just extracted then reimported and the filesize came up different.


Top
   
PostPosted: Tue May 26, 2015 10:06 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
There is no solution for that problem.
QuickBMS already uses the best compression algorithms available, in this case lz4 and doboz are just the original algorithms and zlib is just a mix of 3 algorithms of which the best one is used.

I don't know what algorithm is used for that specific file, let me know.


Top
   
PostPosted: Wed May 27, 2015 1:26 am 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Thank you for your response. Quickbms seems to work fine on all the smaller files in the same .bundle they are all xmls and work on both export and import. it is the larger ones that seem to cause the problems. Very strange, the .bundle wouldn't have different compressions in the same archive I assume.


Top
   
PostPosted: Wed May 27, 2015 8:21 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
Can you provide a link to that patch.bundle or any other archive that gives the same problem in reimport mode?
I really doubt to be able to do something but, who knows, maybe there is an option to use in doboz or lz4 to compress better and I didn't use it... just guessing.


Top
   
PostPosted: Wed May 27, 2015 4:13 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Thanks again for taking your time to help with this if possible. I have included a file xml.bundle that contains the file def_loot_containers.xml which is the exact file as before but the bundle itself is smaller. Just FYI I have successfully imported several of the smaller files in that bundle it is the larger ones that pose a challenge.

Included xml.bundle,witcher3.bms used, and extracted file in question.

https://bitbucket.org/tsanford01/witcher-projects/downloads/Witcher%20Import%20problem.rar

Best regards,
Tom


Top
   
PostPosted: Wed May 27, 2015 6:45 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
Thanks a lot.
Unfortunately I don't have good news.

The file has a size of 1178586 bytes and we must shrink it to 27910 bytes.

The following are the results obtained by calling the functions of lz4:
Code:
LZ4HC  31493
LZ4HC2 30619
LZ4    52918

The first thing to notice is that lz4hc2 compresses more than the lz4hc used in quickbms.
I don't know if I missed this function or wasn't available when I implemented it in quickbms, but it will be used for sure in the next version.

As you can see we are still very far from the 27910 we need to obtain.


Top
   
PostPosted: Wed May 27, 2015 6:49 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
I have an update.
I have just used the latest lz4 (the one in quickbms is just few weeks old) and I have better results:
Code:
LZ4HC  27910
LZ4HC2 27813
LZ4    52918
This is an excellent news, the problem is solved :D

I guess I will release the new quickbms quite soon so you can start the modding.


Top
   
PostPosted: Wed May 27, 2015 7:13 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
You are awesome !! Thanks for all your help. I look forward to using your excellent tool more soon :)

Regards,
Tom


Top
   
PostPosted: Sat May 30, 2015 8:45 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
I have a bad new about Witcher 3.
The following is what I wrote to another user to explain why the reimported files don't work:
Code:
the problem is just the lz4 algorithm so it's not something caused by
quickbms.

If you compress a file with lz4, add some random data at its end and
then try to decompress it, you willl get the same error.

I guess that lz4 doesn't have a end marker like zlib and so it
decompresses everything it gets till it reaches an error like the end
of the output buffer.

As already said, I can do nothing about it because Witcher uses the
same library and in fact the reimported file doesn't work as you tested.
The reason that it doesn't work even if you didn't modify it is caused
by the compression used in quickbms that is better than the one of the
game (same algorithm with better compression option) and so the
compressed file is smaller than the original but the compressed size
is the same of the original file resulting in "garbage" after the new
file.

The only solution is using a rebuilder, I have already seen some solutions on github.


Top
   
PostPosted: Tue Jun 02, 2015 11:52 am 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Thanks again for your hard work !.. I have been working with this and noticed the problem as well. I have been working with a couple of bundles that contain .xml documents. It works fine on smaller files but not the larger ones get the extra garbage data. Looking at the hex lz4 seems to add a header or key of some sort to the beginning and several bytes to the end. Using the lz4 library right off of github I can compress the file then hex edit it to remove extra bits and it works fine. Very strange issue however.

Best regards,
Tom

P.s. - I have to hex edit out the garbage created after the file and replace all bits with 00 to make up the correct file size.


Top
   
PostPosted: Tue Jun 02, 2015 2:26 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Hello again sorry to drag this on. But, I had a thought. We are trying to use the latest LZ4 for compression to get the correct size and the release just before the latest wouldn't work. The Witcher 3 has been in development for years, so my question is how are we certain LZ4 was used and the very latest release at that? As you can tell I am a hobby programmer no pro by any standard, so I am not questioning your skill or anything just trying to learn something. As for end marker and such, the bundle holds a file path into the bundle, an offset, a compressed size and a decompressed size, so I figure it just reads the amount of data between offset and end of compressed size.


Regards,

Tom


Top
   
PostPosted: Wed Jun 03, 2015 7:13 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
Witcher 3 uses LZ4_compress_HC probably with compressionLevel 9, if you use it you should notice that the size of the compressed file is the same of the original.

I guess you are writing a rebuilder right?
You should have no problems in that case, the problem exists only with reinjectors.


Top
   
PostPosted: Fri Jun 05, 2015 8:18 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Hello again,

Yes I a attempting to rebuild some files by relocating them in the bundle changing offset, size, etc. What I am finding is that minor changes to a file say change a single variable from 1 to 9 etc. Then recompile with LZ4 r129 using HC at 9 I still get a file with many extra bytes of data even after removing the header key etc. I just can't get them to work in game with a single change. Even removing comments and such to make them the exact same size and leaving where them are suppose to be using both with your nice Quickbms import or direct hex edit it just won't work, very frustrating. These are simple xml files too.. Using hex workshop and comparing files that single change causes several differences in the file. And ideas?

I of course am not asking you to help mod the game. I am just not understanding the LZ4 compression I guess. Llooking at the posted frame format:
Quote:
A compliant decompressor must be able to decompress at least one working set of parameters that conforms to the specifications presented here. It may also ignore checksums. Whenever it does not support a specific parameter within the compressed stream, it must produce a non-ambiguous error code and associated error message explaining which parameter is unsupported

https://github.com/Cyan4973/lz4/blob/master/lz4_Frame_format.md
So, I am wondering how your tool is able to decompress the files without parameters besides that it is an LZ4 compression. Is the magic number, frame descriptor, etc. not needed? That would give me a clue as to how to recompress it properly. Like possible a header located in another file.. just a guess.

P.s. - Any file I have found compressed with zlib can be edited very easily. I can move them, expand them, whatever.

regards,
Tom


Top
   
PostPosted: Sat Jun 06, 2015 2:16 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 9049
quickbms simply uses this code to decompress:
Code:
size = LZ4_decompress_safe(in, out, zsize, size);

Remember that you don't have to use the same compression algorithm and you don't even have to use compression at all, after you have edited offset/zsize/size edit also the flag.


Top
   
PostPosted: Sat Jun 06, 2015 9:43 pm 

Joined: Tue May 26, 2015 3:58 pm
Posts: 10
Simply a brilliant idea.. lose the troublesome LZ4 altogether change the flags go zlib like the easy to manipulate files and roll on.

You have been great help!
Thanks,
Tom


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 17 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited