ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Sat Jan 20, 2018 9:29 am

All times are UTC




Post new topic  Reply to topic  [ 252 posts ]  Go to page Previous 17 8 9 10 1113 Next
Author Message
PostPosted: Sun Mar 05, 2017 1:45 pm 

Joined: Thu Aug 07, 2014 10:28 pm
Posts: 137
having trouble with oodle in quickbms.
It works fine for some chunks but fails on others.
I am thinking mabee they are using newer version?
https://github.com/powzix/kraken

00 00 EC 80 00 00 00 00 00 00 04 00 2C 7F 93 3E 6E 1C 20 23 00 00 00 00 77 C8 03 00 93 DC D7 6B - works - sample1
00 00 F0 80 00 00 00 00 00 00 04 00 3C 49 E6 C0 E5 E4 23 23 00 00 00 00 33 CF 03 00 EC 9F 78 FB - fails - sample2
00 00 F4 80 00 00 00 00 00 00 04 00 FB B5 3B 80 18 B4 27 23 00 00 00 00 57 CE 03 00 21 D0 4B 7C - crash - sample3

decompressed size is always 0x40000
compressed size is the size of the sample


Attachments:
samples.zip [729.97 KiB]
Downloaded 54 times
oodle_decompressors.zip [2.46 MiB]
Downloaded 51 times
Top
   
PostPosted: Sun Mar 05, 2017 2:09 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
Oodle has long history of compatibility problems of its own algorithms, it's enough to check the changelog:
http://www.radgametools.com/oodlehist.htm
What I mean is that data created with a version of oodle may not be compatible with other versions (even newer versions).
Quickbms 0.7.7 uses oodle 2.3.0.

Anyway quickbms simply calls OodleLZ_Decompress on the data "as is", it performs absolutely no operation on the bytes of the input data (that happens only if you specify a raw algorithm, not the default behaviour).


Top
   
PostPosted: Sun Mar 05, 2017 2:15 pm 

Joined: Thu Aug 07, 2014 10:28 pm
Posts: 137
the chunks are from the same archive.
quickbms decompressed 19gb from the file fine then failed on everything for the remaining 12gb starting at this sample here.
this is ps4 sample.
looks like it might be the 2.40 version they used.
Oodle 2.4.0 is up--New Hydra automatically selects Kraken/Mermaid/Selkie and new Mermaid+ compressor with slightly higher compression!


Top
   
PostPosted: Sun Mar 05, 2017 4:12 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
Unfortunately 2.3.0 was the only oodle dll available on Warframe when I released quickbms 0.7.7
Currently the 2.4.1 is available but I don't know when I'm going to work on the next quickbms.
In your case it's probably more easy to write a simple tool from scratch, after all it's just one API to call... ok it's not that simple to be honest because in the past the oodle developers changed even the prototype of the main API! :O


Top
   
PostPosted: Wed Mar 08, 2017 4:23 am 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
Just thought about something... Few times I had problems with break operator, those problems often was solvable, but sometimes they give real headaches, so I have idea about solution for this problem. Why not add labels for the cycles? And when we need to break some cycle then we could point break to the label of that cycle.

Also I think that labels for the script in general are also needed, this could help a lot in writing custom decompress functions, because disassembled code often can have lots of branches and it is not that easy to unwrap such code, there also can be some platform specific optimizations inside that code which gives even more confusion. So labels are must have in such situations.


Top
   
PostPosted: Wed Mar 08, 2017 12:14 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
Compression and custom encryption algorithm should not implemented in bms language.
The correct way is dumping the function/dll (maybe in a MEMORY_FILE to avoid external files) and using it with CallDLL.

Labels for the cycles?
The language is meant to be simple and as close as possible to the original one.


Top
   
PostPosted: Wed Mar 08, 2017 12:36 pm 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
Then how to fix that very annoying bug in the other way? Sometimes break is pretty much useless in its current state, either it doesn't work as it should or gives unpredictable result, but many decompression algorithms rely on break in the cycles, there is no other solution for such cases.


Top
   
PostPosted: Wed Mar 08, 2017 12:41 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
The work-around I use when I need it (rarely to be honest) is an OK variable set to 1 by default, and set to 0 when I have to break or I set the initial value of the cycle to a value that causes its termination (for example i = 9999 if for i = 0 < 100).

Just curious, what compression algorithms are you implementing in bms language?
I bet it's graphic stuff, right?


Top
   
PostPosted: Wed Mar 08, 2017 12:44 pm 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
aluigi wrote:
Compression and custom encryption algorithm should not implemented in bms language.
The correct way is dumping the function/dll (maybe in a MEMORY_FILE to avoid external files) and using it with CallDLL.


Dumping function from where to where? From MIPS code to where? Or from PowerPC? Or from ARM? Games exist not only for PC, don't you understand it? And the only way is to disassemble game's native code, find decompression routine and translate it manually into human readable code. If quickbms can do so many things then it should handle the most basic functions flawlessly. Otherwise why you implemented all those functions at all?


Top
   
PostPosted: Wed Mar 08, 2017 12:47 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
"break" wasn't available in the original language.
So can't you just translate the reverse engineerd function into C code and use it (yeah "dumping" since there is no need of dll if it's simple) with calldll?


Top
   
PostPosted: Wed Mar 08, 2017 12:48 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
And in any case I prefer to have a "break" that works in most of cases rather than not having it at all.


Top
   
PostPosted: Wed Mar 08, 2017 12:59 pm 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
aluigi wrote:
"break" wasn't available in the original language.
So can't you just translate the reverse engineerd function into C code and use it (yeah "dumping" since there is no need of dll if it's simple) with calldll?

I can, but otherwise what for we have quickbms? For me personally quickbms is very handy because of its simplicity, it works out of the box and the most important - it doesn't take too much time to write script which will do the job.


Top
   
PostPosted: Wed Mar 08, 2017 1:12 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
I know.
Let's say the "break" problem gets fixed, it will remain the huge slowness issue affecting the bms code used for compression and encryption stuff (input->operations->output). Even 10 megabytes of input data can take minutes to be elaborated.
Regarding the readability, it's impossible to compare the length and readability of a function written in bms (long and chaotic) with the same written in C.

So the easiest and most elegant solution is just a precompiled function or dll used with CallDLL.
It's fast, you can embed it directly in the script and if you want to keep the source code you can just put it in a comment.
There is even the possibility to use it in the Log/CLog command through the Comtype and Encryption instructions, the input/input_size/output/output_size will be automatically handled.
The only negative point is that the user will be prompted to acknowledge the usage of the code, just that.


Top
   
PostPosted: Wed Mar 08, 2017 1:15 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
There are various examples on my website about calldll, the following covers both calldll and the commented original function:
http://aluigi.org/bms/tales_of_berseria.bms

While this is the most known example related to calldll used directly in log/clog:
http://aluigi.org/bms/rfactor2.bms


Top
   
PostPosted: Thu Mar 09, 2017 8:28 am 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
I still think that quickbms must have ability to do everything on its own, in other words it must become self-contained tool which doesn't require external bells and whistles.

And I really whish that in one day labels in conjunction with jump operator will be added as a new feature. Is it really that hard to add such feature?


Top
   
PostPosted: Thu Mar 09, 2017 12:34 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
Quote:
I still think that quickbms must have ability to do everything on its own, in other words it must become self-contained tool which doesn't require external bells and whistles.

In my opinion here there is a misunderstanding because quickbms is not an interpreter of a programming language, we have just a limited set of instructions with the final goal of extracting data from an input (file, network, memory and so on).
The problems raise when people start to use quickbms instead of a programming language.
I'm glad of that but making (for example) a 3d model converter in bms instead of python is not a good option.

For example this is a script I made years ago to implement the lzss algorithm in bms language: http://aluigi.altervista.org/bms/pure_l ... t_only.bms
It's huge and slow, much better to implement it as one-line pre-compiled binary to use with calldll.
Or better to use the native one, if people have custom lossless compression algorithms they want to see natively in quickbms then it's enough to contact me and I will add them immediately in the next release.

Quote:
And I really whish that in one day labels in conjunction with jump operator will be added as a new feature. Is it really that hard to add such feature?

My goal is to fix the "break" command, but I don't want to have something that will start to no longer work due to this change, and being "break" a sort of experimental instruction it's priority is really very low.

Currently the development of quickbms is focused on new algorithms to implement and bugfixes, so everything that doesn't require time and that will not affect the current features (fixing "break" is a bugfix with possible effects on the stability).
A feature like the one you proposed (which is basically the label+goto of C) is of no use to users, as you said that "may" be useful only if you reverse engineer a compression/encryption algorithm and want to implement it as bms language. But in that case the solution already exists: calldll.

Could you please tell me what's your problem with calldll?
It's easy and elegant, I really don't get what's the problem.


Top
   
PostPosted: Thu Mar 09, 2017 1:53 pm 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
Quote:
Could you please tell me what's your problem with calldll?
It's easy and elegant, I really don't get what's the problem.

The problem is that not always you can have access to all needed toolchain for that. And when you want to do something only once, is speed really important? Absolutely no. When job is done, the used script becomes pretty much useless and the only thing you can borrow further is your experience. But if I really need to use programming language, then I can use it from start to finish, right? Why do I need quickbms then? But no, I need it because it's much easier to deal with, no need to bother about dependencies, compiling and debugging. Convenience is the most important factor here, the less dependencies the better.


Top
   
PostPosted: Thu Mar 09, 2017 2:06 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
Can you provide some context?
I mean what you are doing, what programming language do you know, how big is the function you are reverse engineering, why quickbms would be better than python or C, and so on.

I'm thinking to what I can do, maybe there are easy solutions that I can implement in the next version.
If a 100% working "break" solves all the problems then I will put it at highest priority and finally closing this bug.
"Label" + "goto" is probably possible (I mean that it's probably not a huge work) if "break" works, obviously "goto" would not be used because already exists that instruction, maybe it can be an additional feature of "break" or "continue" (yeah there is even an experimental continue instruction), like "break label1" or "continue label2".
Just thinking...


Top
   
PostPosted: Thu Mar 09, 2017 2:43 pm 

Joined: Tue Dec 20, 2016 8:18 pm
Posts: 15
aluigi wrote:
Can you provide some context?
I mean what you are doing, what programming language do you know, how big is the function you are reverse engineering, why quickbms would be better than python or C, and so on.


My main hobby is modding, I never concentrate my doings on one language because of that. Different games have different gameplay core, some use their own script language, some use common ones, some use C, so when I deal with a new game, I need to use its language and because most of programming languages have lots of similarities it's not that hard to learn rules of the new one.

And why I really asked for labels is because it can help to make quickbms more versatile. Of all decompression algorithms supported to this day I had only two very lucky cases when quickbms detected used compression of the files and those cases were pretty laughable, one was packbits-RLE and the second was deflate, lol, I could figure it out by my own if I wasn't so lazy in that day... So I thought that It would be great to have some sort of backdoor in quickbms where it will be possible to create custom decompression functions by converting disassembled code into script. Actually RISC processors, for example, have very simplistic instruction set, so the only thing left for such possibility is labels and jump.


Top
   
PostPosted: Thu Mar 09, 2017 2:55 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 7450
Ok, I'm doing some tests with "break" after a quick fix that includes also the dormant "continue" instruction.
Everything seems ok but obviously I need more tests cases because I'm sure there is something missing or wrong, it would be too good to have fixed this issue in less than 5 minutes :D

If you can produce some scripts that I can test, it would be great.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 252 posts ]  Go to page Previous 17 8 9 10 1113 Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited