CSAW CTF - RE300 - FTP

The challenge binary is a 64 bit executable. The objective was to login as user blankwall by finding the password. The supplied password is processed using a hash function at 0x00401540. Below is the decompiled code of the function:

uint32_t calc_hash(char *input)
{
  int counter; 
  uint32_t state; 

  state = 5381;
  for (counter = 0; input[counter]; ++counter)
    state = 33 * state + input[counter];
  return state;
}

The output of this function must be equal to 0x0D386D209. The length of the password is not known and state variable is 32-bit, hence modulo 0xffffffff. I used Z3 to solve this and multiple solutions are possible. Below is the solver:

#!/usr/bin/env python

from z3 import * 

# >>> (0x1505 * 0x21 * 0x21 * 0x21) < 0xD386D209
# True
# minimum length of password
passlen = 4

def init_solver(passlen):
    s = Solver()
    vectors = {}
    for c in range(passlen):
        vec = BitVec(str(c), 8)
        vectors[c] = vec
        s.add(And(0x20 < vec, vec < 0x7F))
    return s, vectors

while True:
    s, vectors = init_solver(passlen)
    state = 0x1505 
    for i in range(passlen):
        vec = ZeroExt(24, vectors[i])
        state = state * 0x21 + vec
    s.add(state == 0xD386D209)

    if s.check() == sat: break
    passlen += 1

m = s.model()
c = lambda m, i: chr(m[vectors[i]].as_long())
print ''.join([c(m,i) for i in range(passlen)])

This gives the password UIcy1y. Login using the password as

con.write('USER blankwall' + chr(0xa))
password = 'PASS UIcy1y\x00' + chr(0xa)
con.write(password)

The file re_solution had the flag, flag{n0_c0ok1e_ju$t_a_f1ag_f0r_you}