1) reverse engineer the program to code the "hash" function
binary is compiled with debug, it's annoying but feasible
used IDA for static analysis + immdbg for debugging
enventually, see that it's not a hash but an encoding (so invertible)
2) invert the encoding and decode the "hash" to get the flag

$ python encode.py abcdefgh
input: abcdefgh (8)
f1: "aC&\noU<$'*-0369<" 6143260a6f553c24272a2d303336393c
f2: 'GBEDCGAKGPFFDMCECHCKCNDADDDGDJDM' 474245444347414b47504646444d43454348434b434e444144444447444a444d
f3: 'MDJDGDDDADNCKCHCECMDFFPGKAGCDEBG' 4d444a444744444441444e434b43484345434d44464650474b41474344454247
MDJDGDDDADNCKCHCECMDFFPGKAGCDEBG

$ python decode.py MDJDGDDDADNCKCHCECMDFFPGKAGCDEBG
input: MDJDGDDDADNCKCHCECMDFFPGKAGCDEBG (32)
f3r: 'GBEDCGAKGPFFDMCECHCKCNDADDDGDJDM' 474245444347414b47504646444d43454348434b434e444144444447444a444d
f2r: "aC&\noU<$'*-0369<" 6143260a6f553c24272a2d303336393c
f1r: 'abcdefgh' 6162636465666768
abcdefgh

$ python decode.py HEEEBEODOAMBICKDFFCHGAGEDFDCGDGH
input: HEEEBEODOAMBICKDFFCHGAGEDFDCGDGH (32)
f3r: 'HGDGCDFDEGAGHCFFDKCIBMAODOEBEEEH' 48474447434446444547414748434646444b4349424d414f444f454245454548
f2r: 'v6#SF\x06rU:(\x1c\x0e>ADG' 76362353460672553a281c0e3e414447
f1r: 'v@m0s@lcentr0' 76406d3073406c63656e747230
v@m0s@lcentr0