Recently I have returned to the possibility of trying to add GameSpy3D support to my GameSpy Master Server Emulator. So far, I have been able to utilize gsmsalg properly to service the \\basic\\secure\\<challenge key> and the \\validate\\<validate key> properly from server to client. After research I have found out what happens next is the server simply sends back an encrypted version of the standard 4 byte + 2 byte server list. It does not send a \\final\\ for this as the total size of the decrypted packet is part of the encoded header.
Aluigi was kind enough to go back into his archives and find a wireshark ACP dump of a server query with GameSpy3D from years ago. I wrote a tool to successfully use the entire dump and decode it into the proper list.
At this point, I am totally stuck on how to encrypt it. Aluigi never wrote an encoder for this protocol since it was only used by GS3D. I suspect that it's probably very much the same as enctype2 and later where it func6/func7 have a func6e/func7e where at the end of the func7e function two things are swapped at the end of that particular round. With that said, I don't know how to initialize things so it works in reverse.
Attached is the code I have written to test decoding aluigi's packet dump successfully. If for some reason the attachment does not work it is also available at: http://maraakate.org/EncType1_Decode_Test.7z
Is there anyone out there who can take a look to see what I have to do in order to get the encoding part initialized properly? Or at least guide me in the right direction? I am very ignorant when it comes to crypto, not really a strong suite of mine.
Thanks in advance!