ZenHAX

Now that aluigi wrote a pickle decoder I tried to modify the rpa extraction script to decode the new Ren'py persistent file format to something more user readable but after reading the documentation and experimenting with the code it seems my understanding of quickbms is not good enough to do it myself so I tough I'd ask for help someone more experienced than me.

What I know so far is that there are 2 types of persistent file formats (new cPickle and old pickle) and they are both zlib compressed.

I've attached an example of both file types if anyone is wiling to take a look.

Where are used these formats? In savegames?
You should use python pickle to fully load the objects.
The pickle.bms script was created only for having an easy and quick (and python-less) access to the data.

They are used to store the game configuration and other types of data that is shared between player games that can't be stored in the save files.

For example for "Gallery" or "New Game +".

P.S. Is there a easy way to save unpickled version of the file with python?

The best i managed to do so far based on RPA archives is:

# RenPy persistent unpickler 0.1

init python:
 try:
  import cPickle as pickle
 except:
  import pickle
 import sys
 import zlib

 f = open("game/saves/persistent", "rb")
 data = f.read().decode("zlib")
 f.close()

 index = pickle.loads(data)
 
 f = open("game/saves/persistent.txt", "wb")
 f.write(str(len(index.keys()))+"\\n")
 for key in index.keys():
  for offset, dlen, start in index[key]:
   f.write(key+"\\n")
   f.write(str(offset)+"\\n")
   f.write(str(dlen)+"\\n")
   f.write(str(start)+"\\n")
   f.write("\\n")
 
 f.close()

but it returns an error:

Full traceback:
  File "renpy/bootstrap.py", line 295, in bootstrap
    renpy.main.main()
  File "renpy/main.py", line 419, in main
    game.context().run(node)
  File "game/unpickle_persistent.rpy", line 3, in script
    init python:
  File "renpy/ast.py", line 814, in execute
    renpy.python.py_exec_bytecode(self.code.bytecode, self.hide, store=self.store)
  File "renpy/python.py", line 1719, in py_exec_bytecode
    exec bytecode in globals, locals
  File "game/unpickle_persistent.rpy", line 22, in <module>
    f.write(str(len(index.keys()))+"\\n")
TypeError: 'NoneType' object is not callable

And using python produces this error (the same if I manually unpack the file and use the unpacked version with the original code):

python -O 1.py 1.dat 1.txt
Traceback (most recent call last):
  File "1.py", line 18, in <module>
    index = pickle.loads(data)
  File "renpy/persistent.py", line 30, in <module>
    from renpy.loadsave import dump, dumps, loads
  File "renpy/loadsave.py", line 653, in <module>
    unknown = renpy.object.Sentinel("unknown")
AttributeError: 'module' object has no attribute 'object'

1.py code:

try:
 import cPickle as pickle
except:
 import pickle
#import io
#import os
import sys
import zlib
#import renpy
#import types
#import pickle
#from cStringIO import StringIO

f = open(sys.argv[1], "rb")
data = f.read().decode("zlib")
f.close()

index = pickle.loads(data)

f = open(sys.argv[2], "wb")
f.write(str(len(index.keys()))+"\\n")
for key in index.keys():
 for offset, dlen, start in index[key]:
  f.write(key+"\\n")
  f.write(str(offset)+"\\n")
  f.write(str(dlen)+"\\n")
  f.write(str(start)+"\\n")
  f.write("\\n")
 
 f.close()