Rolling My Own picoCTF 2021 Solution

Description

1. Step 1Understand the hash-as-code trick

   Observation

   The binary never compares your input against a stored string. It calls mmap and mprotect to mark a buffer executable and jumps into it, which means the MD5 output bytes are being run as machine code rather than compared as a hash.

   The binary appends an 8-byte salt to your input, splits the combined string into 12-byte chunks, and MD5-hashes each chunk. It then pulls four consecutive bytes starting at a specific offset out of each hash (the starting offsets used are 8, 2, 7, and 1 for the four chunks respectively) and concatenates those four 4-byte slices into a 16-byte buffer that it marks executable and calls. A wrong password yields random bytes that fault; the intended password yields bytes that form a valid little routine.

   bash

   # In Ghidra, find the function that mmaps/mprotects a buffer RWX and calls it.

   Copied!

   bash

   # Trace back: input + salt -> 12-byte chunks -> MD5 -> selected bytes -> executed.

   Copied!

   What didn't work first

   Tried: Try to crack the binary by patching the conditional jump so it always takes the success branch.

   There is no conditional jump to patch. The binary never compares your password against a stored value; it executes the bytes that hashing your input produced. A wrong password does not fail a check, it generates garbage machine code and segfaults. Patching jumps in the wrapper loop achieves nothing.

   Tried: Use 'strings' on the binary hoping to recover the password or the salt values in plaintext.

   strings does show the salt values, which are 8-byte literals in the data section, but not which bytes of which MD5 output get executed, or in what order. Reverse the offset-selection logic in Ghidra to learn which 4-byte slice of each hash reaches the shellcode buffer.

   Learn more

   Why this is "rolling your own". The author avoided storing the password by making the password itself the only input that hashes into runnable code. This is a real obfuscation technique: a hash sub-sequence is interpreted as machine code, and the salt is tuned so the desired instruction bytes only appear for one specific key. It is clever, but it is fully reversible because MD5 is fast to brute-force over a tiny 4-character search space per constraint.

   The target shellcode the routine must produce is:

   48 89 FE             mov rsi, rdi
   48 BF F1 26 DC B3 07 00 00 00   mov rdi, 0x7b3dc26f1
   FF D6                call rsi
   C3                   ret

2. Step 2Turn the shellcode into MD5 byte constraints

   Observation

   Each 4-byte slice that gets executed comes from a known offset inside a specific MD5 digest, and each chunk's salt is a fixed literal in the data section. So the short 4-character prefix of every chunk can be brute-forced independently to match the required bytes.

   Each 12-byte chunk is 4 unknown password characters followed by an 8-byte salt that Ghidra reveals. The four salts (one per chunk) plus the required hash bytes give four independent constraints. With the leading characters hinted as 'D1v1', you brute-force the remaining 4-character groups so that MD5(group + salt) carries the needed byte at the needed offset.

   python

   python3 - <<'PY'
   import hashlib, itertools, string
   
   # salts and (offset, required 4 bytes) recovered from the binary
   chunks = [
       (b"GpLaMjEW", 8, bytes([0x48, 0x89, 0xFE, 0x48])),
       (b"pVOjnnmk", 2, bytes([0xBF, 0xF1, 0x26, 0xDC])),
       (b"RGiledp6", 7, bytes([0xB3, 0x07, 0x00, 0x00])),
       (b"Mvcezxls", 1, bytes([0x00, 0xFF, 0xD6, 0xC3])),
   ]
   
   alphabet = (string.ascii_letters + string.digits).encode()
   password = b""
   for salt, off, want in chunks:
       for combo in itertools.product(alphabet, repeat=4):
           guess = bytes(combo)
           h = hashlib.md5(guess + salt).digest()
           if h[off:off + len(want)] == want:
               password += guess
               break
   print("password:", password.decode())   # -> D1v1d3AndC0nqu3r
   PY

   Copied!

   Expected output

   password: D1v1d3AndC0nqu3r

   Each group is only 4 characters over a ~62-symbol alphabet, so each constraint solves in well under a second. Concatenating the four recovered groups gives the full 16-character password.

   What didn't work first

   Tried: Attempt to reverse the MD5 hash directly (e.g., using an online hash lookup or hashcat) to recover each 4-character group.

   Online MD5 rainbow tables cover common words and short dictionary strings, not arbitrary 4-character combinations salted with an 8-byte binary suffix. Hashcat fails too, because you need to match 4 bytes at a specific offset rather than the full digest, and its standard modes compare whole hashes with no partial-offset option. The brute-force loop checks exactly the relevant slice.

   Tried: Brute force all 16 characters at once as a single combined search.

   A 16-character alphanumeric brute force means 62 to the 16th power, which is hopeless. But each 12-byte chunk is independent: the four groups never interact in the hash computation. That makes each 4-character sub-problem at most about 14 million candidates, solvable in under a second. Treating it as one monolithic search throws away the structure the challenge is built on.

   Learn more

   Why brute force is the right tool. MD5 is not invertible, but you are not inverting it. You only need a 4-byte agreement at a fixed offset, and the preimage you control is 4 printable characters. That is at most a few million hashes per chunk, trivial on a laptop. The salts ensure each chunk has a unique answer, so the four groups concatenate into one password.

   For more on recovering keys from hash constraints and custom checkers, see the CTF Encodings guide.

3. Step 3Submit the password

   Observation

   Concatenating the four recovered 4-character groups gives one 16-character password. Submit it to the netcat service and the right shellcode executes.

   Connect and enter the recovered password. The service rebuilds the shellcode from your input, executes it, and prints the flag.

   bash

   nc mercury.picoctf.net <PORT_FROM_INSTANCE>

   Copied!

   bash

   # Enter: D1v1d3AndC0nqu3r

   Copied!

   Learn more

   The 16-character password splits into the four 4-character groups the brute force recovered, in order: D1v1, d3An, dC0n, qu3r.

Flag

The checker hashes your password and runs the hash bytes as machine code, so only the password whose MD5 outputs assemble into valid shellcode works. Reverse the byte-selection scheme, then brute-force each 4-character group so MD5(group+salt) carries the required byte. The password is D1v1d3AndC0nqu3r.