ABClassic or other game? You have to make the game's APK debuggable, clone the APK decompiled on your computer, fake Android.mk, connect your device to your computer, call NDK-GDB (from Android SDK), use breakpoints on application functions that call for loading LUA files, load a level, you should hit the breakpoint, do x/4x and x/s on the given addresses and one should give you the encryption key. The current key works fine for me, I'll try that file (and script entirely) and see if there's a problem... UPDATE: That file works fine for decryption (just delete the 9 header bytes) and open as an archive, try updating OpenSSL or use the BMS script. You can decompile with "java -jar unluac.jar level766.lua > level766_decomp.lua".
BMS script is not work, I use OpenSSL.
And, how to recompile levels to apk?