ZenHAX

Free Game Research Forum | Official QuickBMS support | twitter @zenhax | SSL HTTPS://zenhax.com
It is currently Tue Oct 17, 2017 1:51 am

All times are UTC




Post new topic  Reply to topic  [ 10 posts ] 
Author Message
PostPosted: Sun Oct 02, 2016 9:05 pm 

Joined: Sun Oct 02, 2016 8:46 pm
Posts: 5
Using gcc on OS X to compile quickbms_src

it is spitting out a lot of warnings and errors at the moment. I have suppressed some by modifying the Makefile:

Code:
EXE      = quickbms
# OS X aliases gcc/g++ to clang. force make to gnu compilers here (from Homebrew installation)
CC      = /usr/local/Cellar/gcc/6.2.0/bin/gcc-6
CXX      = /usr/local/Cellar/gcc/6.2.0/bin/g++-6
# -m32 because QuickBMS has been tested only on 32bit systems and gives problems using 64bit native code
# add -I flag for <malloc.h> .. on OS X its in malloc/ folder.
CFLAGS   += -m32 -s -O0 -fstack-protector-all -fno-unit-at-a-time -fno-omit-frame-pointer -I/usr/include/malloc/
CDEFS   += -DDISABLE_UCL -DDISABLE_MCRYPT -DDISABLE_TOMCRYPT
CLIBS   += -lstdc++ -ldl -lcrypto -lssl -lz -lbz2 -lstdc++ -lm -lpthread -llzo2
PREFIX   = /usr/local
BINDIR   = $(PREFIX)/bin
SRC      = $(EXE).c

all:
   $(CC) $(SRC) $(CFLAGS) -o $(EXE) \
   \
   -D_7ZIP_ST \
   \
   compression/*.c* encryption/*.c* extra/mem2mem.c libs/lzma/LzmaDec.c libs/lzma/Lzma2Dec.c libs/lzma/Bra86.c libs/lzma/LzFind.c libs/lzma/LzmaEnc.c libs/lzma/Lzma2Enc.c libs/mspack/*.c libs/ppmd/*.cpp libs/aplib/depacks.c libs/brieflz/depacks.c libs/brieflz/brieflz.c compression/jcalg1_static.lib libs/zziplib/*.c libs/bcl/*.c libs/szip/*.c libs/lzhl/*.cpp libs/tdcb/*.c libs/libkirk/*.c libs/7z_advancecomp/*.cc libs/iris/*.cpp libs/old_cabextract/lzx.c libs/mrci/*.cpp libs/lz4/*.c libs/snappy/snappy.cc libs/snappy/snappy-c.cc libs/snappy/snappy-stubs-internal.cc libs/snappy/snappy-sinksource.cc libs/mmini/mmini_huffman.c libs/mmini/mmini_lzl.c libs/clzw/lzw-dec.c libs/clzw/lzw-enc.c libs/lzlib/lzlib.c libs/blosc/blosclz.c libs/gipfeli/*.cc libs/liblzg/src/lib/decode.c libs/liblzg/src/lib/encode.c libs/liblzg/src/lib/checksum.c libs/doboz/*.cpp libs/amiga/* libs/sphlib/c/*.c libs/shadowforce/*.cpp libs/zstd/common/*.c libs/zstd/compress/*.c libs/zstd/decompress/*.c libs/zstd/dictBuilder/*.c libs/zstd/legacy/*.c -Ilibs/zstd -Ilibs/zstd/common -Ilibs/zstd/legacy libs/azo/unAZO.cpp libs/azo/Decoder/MainCodeD.cpp libs/azo/Common/x86Filter.cpp libs/nintendo_ds/*.c libs/ctw/*.c libs/grzip/libgrzip.c libs/heatshrink/heatshrink_decoder.c libs/heatshrink/heatshrink_encoder.c libs/libzling/*.cpp  -Ilibs/ecrypt/include -Ilibs/libcsc -D_7Z_TYPES_ libs/libcsc/csc_dec.cpp libs/libcsc/csc_filters.cpp libs/libcsc/csc_memio.cpp -DDENSITY_FORCE_INLINE=inline -Drestrict=__restrict__ libs/density/src/*.c libs/density/src/spookyhash/src/*.c libs/brotli/dec/*.c libs/brotli/common/*.c libs/libbsc/adler32/adler32.cpp libs/libbsc/bwt/bwt.cpp libs/libbsc/coder/coder.cpp libs/libbsc/coder/qlfc/qlfc.cpp libs/libbsc/coder/qlfc/qlfc_model.cpp libs/libbsc/filters/detectors.cpp libs/libbsc/filters/preprocessing.cpp libs/libbsc/libbsc/libbsc.cpp libs/libbsc/lzp/lzp.cpp libs/libbsc/platform/platform.cpp libs/libbsc/st/st.cpp libs/shoco/shoco.c libs/ms-compress/src/*.cpp libs/lzjody/lzjody.c libs/lzjody/byteplane_xfrm.c disasm/disasm.c disasm/cmdlist.c disasm/assembl/assembl.c libs/mydownlib/mydownlib.c libs/TurboRLE/trlec.c libs/TurboRLE/trled.c libs/TurboRLE/ext/mrle.c libs/lhasa/lib/*_decoder.c libs/lhasa/lib/crc16.c libs/dipperstein/*.c libs/liblzf/lzf_d.c libs/liblzf/lzf_c_best.c libs/zopfli/*.c libs/lzham_codec/lzhamcomp/*.cpp libs/lzham_codec/lzhamdecomp/*.cpp libs/lzham_codec/lzhamlib/*.cpp -Ilibs/lzham_codec/include -Ilibs/lzham_codec/lzhamcomp -Ilibs/lzham_codec/lzhamdecomp -DLZHAM_ANSI_CPLUSPLUS libs/dmsdos/*.c libs/tornado/Tornado.cpp libs/tornado/Common.cpp libs/PKLib/*.c extra/mybits.c libs/lz5/lz5*.c libs/ppmz2/*.c* libs/libdivsufsort/*.c libs/xxhash/*.c extra/xalloc.c libs/lzfse/src/*.c  libs/hsel/myhsel.cpp libs/hsel/HSEL.cpp \
   \
   $(CLIBS) $(CDEFS)

install:
   install -m 755 -d $(BINDIR)
   install -m 755 $(EXE) $(BINDIR)/$(EXE)

.PHONY:
   install


using OS X El Capitan Version 10.11.6

any help would be appreciated :D


Attachments:
File comment: stdout and stderr of make using the Makefile in OP on OS X El Capitan Version 10.11.6
make.log.zip [90.55 KiB]
Downloaded 52 times
Top
   
PostPosted: Mon Oct 03, 2016 10:53 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6850
The first error is the missing LZO library: lzo/lzo1.h
You need it, or you can add -DDISABLE_LZO to skip it.

The second is a bug in libzpaq that apparently requires -Dunix or it thinks you are windows.

The 99,9% of the other errors are related to assembly and in-line assembly.

The last errors are related to lzham and the usage of "_is_pod".

What I can do on my side is:
  • for lzo nothing because it's a requirement and it's used in many games, so install it or use -DDISABLE_LZO
  • I can report the libzpaq problem to the author or add a commented variable in the Makefile containing both -Dunix and -I/usr/include/malloc/
  • skip yamb256/yamb.c if it's __APPLE__
  • skip all the (useless) amiga compressions if it's __APPLE__
  • lzham already checks __APPLE__ so I don't know why you get __is_pod instead of "std::__is_pod<T>::__value", it's a third-party library so the solution is probably just skipping it


Top
   
PostPosted: Mon Oct 03, 2016 9:51 pm 

Joined: Sun Oct 02, 2016 8:46 pm
Posts: 5
aluigi wrote:
The first error is the missing LZO library: lzo/lzo1.h
You need it, or you can add -DDISABLE_LZO to skip it.

The second is a bug in libzpaq that apparently requires -Dunix or it thinks you are windows.

The 99,9% of the other errors are related to assembly and in-line assembly.

The last errors are related to lzham and the usage of "_is_pod".

What I can do on my side is:
  • for lzo nothing because it's a requirement and it's used in many games, so install it or use -DDISABLE_LZO
  • I can report the libzpaq problem to the author or add a commented variable in the Makefile containing both -Dunix and -I/usr/include/malloc/
  • skip yamb256/yamb.c if it's __APPLE__
  • skip all the (useless) amiga compressions if it's __APPLE__
  • lzham already checks __APPLE__ so I don't know why you get __is_pod instead of "std::__is_pod<T>::__value", it's a third-party library so the solution is probably just skipping it


Thanks for the fast response. I installed LZO, defined unix like you said. To skip yamb algorithm and amiga compressions - what's the easiest way to do that? As for the lzham issue, it looks like it is using "std::__is_pod<T>::__value" but there is an issue with it still:

Code:
libs/lzham_codec/lzhamdecomp/lzham_traits.h:71:33: error: expected ')' before '__is_pod'
    #define LZHAM_IS_POD(T) std::__is_pod<T>::__value


EDIT: It actually wants "__is_pod", and not "std::__is_pod<T>::__value", so scratch the lzham issue.


Top
   
PostPosted: Tue Oct 04, 2016 4:50 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6850
Very good, I guess that __is_pod thing can be solved with a -D_is_pod=__is_pod without source code modification :)

Regarding amiga it's enough to remove "libs/amiga/*" from the Makefile and remove all the "QUICK_COMP_CASE(AMIGA_" lines in perform.c (they are included in a #if defined).

For yamb.c it's probably more easy to just delete all the code from line 440 till 611.

To recap, the additional line in the Makefile necessary to compile on OSX is:
Code:
-I/usr/include/malloc/ -Dunix -D_is_pod=__is_pod


Top
   
PostPosted: Tue Oct 04, 2016 2:51 pm 

Joined: Sun Oct 02, 2016 8:46 pm
Posts: 5
Ok, I removed amiga and yamb using the methods you gave. That cleared up the log to look at the sneaky errors :_)

#1
on __APPLE__, sys/ptrace.h uses different signal names for ptrace(...
I added this to io/process.c on line 13 to fix it
Code:
    #ifdef __APPLE__
        #define PTRACE_TRACEME      PT_TRACE_ME
        #define PTRACE_PEEKTEXT     PT_READ_I
        #define PTRACE_PEEKDATA     PT_READ_D
        #define PTRACE_PEEKUSER     PT_READ_U
        #define PTRACE_POKETEXT     PT_WRITE_I
        #define PTRACE_POKEDATA     PT_WRITE_D
        #define PTRACE_POKEUSER     PT_WRITE_U
        #define PTRACE_CONT         PT_CONTINUE
        #define PTRACE_KILL         PT_KILL
        #define PTRACE_SINGLESTEP   PT_STEP
        #define PTRACE_GETREGS      PT_GETREGS
        #define PTRACE_SETREGS      PT_SETREGS
        #define PTRACE_GETFPREGS    PT_GETFPREGS
        #define PTRACE_SETFPREGS    PT_SETFPREGS
        #define PTRACE_ATTACH       PT_ATTACH
        #define PTRACE_DETACH       PT_DETACH
        #define PTRACE_GETFPXREGS   PT_GETFPXREGS
        #define PTRACE_SETFPXREGS   PT_SETFPXREGS
        #define PTRACE_SYSCALL      PT_SYSCALL
        #define PTRACE_SETOPTIONS   PT_SETOPTIONS
        #define PTRACE_GETEVENTMSG  PT_GETEVENTMS
        #define PTRACE_GETSIGINFO   PT_GETSIGINFO
        #define PTRACE_SETSIGINFO   PT_SETSIGINFO
    #endif


#2
In libs/lzham_codec/lzhamdecomp/lzham_platform.cpp line 65:
"__asm {int 3};" was giving syntax errors
the correct function call to invoke debug/trap signal on OS X is "__builtin_trap();"
so it becomes:
Code:
void lzham_debug_break(void)
{
#if LZHAM_USE_WIN32_API
   DebugBreak();
#elif (TARGET_OS_MAC == 1) && (TARGET_IPHONE_SIMULATOR == 0) && (TARGET_OS_IPHONE == 0)
   __builtin_trap();
#else
   assert(0);
#endif
}



The last error I'm getting is coming from libiconv headers themselves and not quickbms. So that's confusing. see:
Code:
/usr/local/opt/libiconv/include/iconv.h:139:55: error: expected ';', ',' or ')' before 'uc'
 typedef void (*iconv_unicode_char_hook) (unsigned int uc, void* data);
                                                       ^~
/usr/local/opt/libiconv/include/iconv.h:144:3: error: unknown type name 'iconv_unicode_char_hook'
   iconv_unicode_char_hook uc_hook;
   ^~~~~~~~~~~~~~~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:155:61: error: expected ';', ',' or ')' before '*' token
               void (*write_replacement) (const unsigned int *buf, size_t buflen,
                                                             ^
/usr/local/opt/libiconv/include/iconv.h:157:15: error: expected ';', ',' or ')' before 'void'
               void* callback_arg,
               ^~~~
/usr/local/opt/libiconv/include/iconv.h:164:28: error: expected ';', ',' or ')' before 'code'
              (unsigned int code,
                            ^~~~
/usr/local/opt/libiconv/include/iconv.h:198:3: error: unknown type name 'iconv_unicode_mb_to_uc_fallback'
   iconv_unicode_mb_to_uc_fallback mb_to_uc_fallback;
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:199:3: error: unknown type name 'iconv_unicode_uc_to_mb_fallback'
   iconv_unicode_uc_to_mb_fallback uc_to_mb_fallback;
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:216:52: error: expected ';', ',' or ')' before 'namescount'
 extern void iconvlist (int (*do_one) (unsigned int namescount,
                                                    ^~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:219:24: error: expected ';', ',' or ')' before 'void'
                        void* data);


EDIT: The iconv syntax error goes away when I use clang which makes sense. But then quickbms source is full of syntax errors because it prefers gcc (clang doesn't support nested functions so a lot breaks)

Can I also remove calls to iconv without quickbms breaking/losing major functionality ?


Top
   
PostPosted: Tue Oct 04, 2016 7:24 pm 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6850
#1
Well done :)
I will use your fix for sure.

#2
Try to add -D__builtin_trap() to the Makefile

#libiconv
Can you try to comment the "#define USE_LIBICONV" in quickbms.c?


Top
   
PostPosted: Tue Oct 04, 2016 9:52 pm 

Joined: Sun Oct 02, 2016 8:46 pm
Posts: 5
libiconv solution worked :)

Now come linker errors..

First, because we removed amiga compression, I had to comment out these lines (unz.c:5590)
Code:
extern int UnSquash(unsigned char *, int);// __asm__("UnSquash");
int amiga_unsquash(unsigned char *in, int insz, unsigned char *out) {
    int     i;

    // copy to out setting the correct endianess
    for(i = 0; i <= (insz - 4); i += 4) {
        *(u32 *)(out + i) = ntohl(*(u32 *)(in + i));
    }

    return UnSquash(out, insz);
}


Next, because you're forcing 32bit compilation, and my local lzo and openssl libs are compiled as 64 bit, they get ignored by ld.
I downloaded the latest source of openssl and lzo, and built the libs as 32 bit, then changed quickbms Makefile to use them.

And now I have quickbms compiled and running on OS X :). Thanks again for your help


Top
   
PostPosted: Wed Oct 05, 2016 3:14 am 

Joined: Sun Oct 02, 2016 8:46 pm
Posts: 5
My goal was to compile quickbms to run the compression type scanner. It worked - data is lzss compressed

Here's comtype_scan2.bat re-written as a shell script:

Code:
help() {
   echo ""
    echo "QuickBMS comtype scanner 2 (version 0.1.1b)"
    echo "by Luigi Auriemma"
    echo "e-mail: aluigi@autistici.org"
    echo "web:    aluigi.org"
    echo ""
    echo "you must specify:"
    echo "- the path of comtype2_scan.bms"
    echo "- the input file compressed with the unknown algorithm"
    echo "- the output folder where placing the unpacked files"
    echo ""
    echo "example:"
    echo "  comtype_scan2 c:\comtype_scan2.bms c:\dump.dat c:\output_folder [max_size]"
    echo ""
    echo "if an algorithm doesn't return immediately press CTRL-C"
    echo "and answer 'n' [no] when will be asked to \"terminate batch job\""
    echo ""
}

if [ $# -ne 3 ]; then
    help
    exit 1
fi

for i in {1..700}; do
   quickbms -a $i $4 $1 $2 $3
done


Top
   
PostPosted: Wed Oct 05, 2016 8:57 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6850
Excellent


Top
   
PostPosted: Wed Oct 05, 2016 10:41 am 
Site Admin
User avatar

Joined: Wed Jul 30, 2014 9:32 pm
Posts: 6850
Ah, I successfully compiled on OSX too :)
There will be a commented step-by-step in the next Makefile for OSX to make the life easier.


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 10 posts ] 

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Powered by phpBB® Forum Software © phpBB Limited