After I did lots of research in sound formats, archives, models and animations I finally decided to get to most complicated task I've heard of: Megatextures. I was able to reverse the modified version of HDP they used and decompress megatextures. The devs improved the coding algorithm in compare to standard HDP.
Doom Tool released http://aviacreations.com/wraith
. And here's The brief history of megatexture research.March 22nd
The first look into mega2 files. I found they contain tons of small pieces - tiles. From xentax thread I got info that's probably HDP or JXR compression. After downloading HDP specs and sources it becomes clear why nobody was able to decode it before. Walls of code. Very complicated algorithm, even having the specs.March 25th
I found decoding routine in Doom and got the first tile decoded by it from memory.
Making standard headers for the raw data (with all available options tried) surely fails.March 26-30th
I needed to finish cloth extraction for Hitman 5, and D-Walker animations for MGSV, so nothing was done here.April 3rd
After some days of studying the format specs and raw data I came to conclusion that it was modified, non standard HDP format. So the only way to decode it was to carefully study their routine step-by-step and find what exactly was changed. But it looked so tangled up even after some debugging and multiple reading through the doc, that I said: "no. its TOO complex. i think i wont be able to do this"April 7th
After some more debugging and studying, and changing the code, the very first monochrome tile was decoded. I couldn't believe it first, but it couldn't be just by accident. This was correctly decoded tile.April 8th
More in-depth research showed that most changes were made deep inside of the adaptive variable-length bitstream reading which mostly concerned colored images. Some changes found, and I could decode about half of the image, and then it fails:April 9th
Lots of hard work to dig into the very core of this crazy algorithm, many more changes were found and at some point I got this:
This was the very first 3-channel tile (not very colorful, but its colored) decoded from megatexture. After that, It was all history. Just some little tweaks and checks, was a little bit different in all games, but the problem was finally solved. You can count, it took 13 days.