Rolling My Own picoCTF 2021 Solution

Description

1. Step 1

   Understand the hash-as-code trick

   Observation

   I noticed the binary never compared the input to a stored string but instead called mmap/mprotect to mark a buffer executable before jumping into it, which suggested the MD5 output bytes were being used directly as machine code rather than as a stored hash for comparison.

   The binary appends an 8-byte salt to your input, splits the combined string into 12-byte chunks, and MD5-hashes each chunk. It then pulls four consecutive bytes starting at a specific offset out of each hash (the starting offsets used are 8, 2, 7, and 1 for the four chunks respectively) and concatenates those four 4-byte slices into a 16-byte buffer that it marks executable and calls. A wrong password yields random bytes that fault; the intended password yields bytes that form a valid little routine.

   bash

   # In Ghidra, find the function that mmaps/mprotects a buffer RWX and calls it.

   Copied!

   bash

   # Trace back: input + salt -> 12-byte chunks -> MD5 -> selected bytes -> executed.

   Copied!

   What didn't work first

   Tried: Try to crack the binary by patching the conditional jump so it always takes the success branch.

   There is no conditional jump to patch. The binary does not compare your password to a stored value at all - it literally executes the bytes produced by hashing your input. A wrong password does not fail a check; it generates garbage machine code that causes a segfault before any comparison can happen. Patching jumps in the wrapper loop does nothing useful.

   Tried: Use 'strings' on the binary hoping to recover the password or the salt values in plaintext.

   Strings shows the salt values embedded in the binary (they are 8-byte literals in the data section), but it cannot tell you which bytes of which MD5 output are executed or in what order. You still need to reverse the offset-selection logic in Ghidra to know which 4-byte slice of each hash ends up in the shellcode buffer.

   Learn more

   Why this is "rolling your own". The author avoided storing the password by making the password itself the only input that hashes into runnable code. This is a real obfuscation technique: a hash sub-sequence is interpreted as machine code, and the salt is tuned so the desired instruction bytes only appear for one specific key. It is clever, but it is fully reversible because MD5 is fast to brute-force over a tiny 4-character search space per constraint.

   The target shellcode the routine must produce is:

   48 89 FE             mov rsi, rdi
   48 BF F1 26 DC B3 07 00 00 00   mov rdi, 0x7b3dc26f1
   FF D6                call rsi
   C3                   ret

2. Step 2

   Turn the shellcode into MD5 byte constraints

   Observation

   I noticed that each 4-byte slice executed as shellcode was drawn from a known offset within a specific MD5 digest, and the salt for each chunk was a fixed literal in the binary's data section, which suggested I could brute-force the short 4-character prefix of each chunk independently to match the required bytes at those offsets.

   Each 12-byte chunk is 4 unknown password characters followed by an 8-byte salt that Ghidra reveals. The four salts (one per chunk) plus the required hash bytes give four independent constraints. With the leading characters hinted as 'D1v1', you brute-force the remaining 4-character groups so that MD5(group + salt) carries the needed byte at the needed offset.

   python

   python3 - <<'PY'
   import hashlib, itertools, string
   
   # salts and (offset, required 4 bytes) recovered from the binary
   chunks = [
       (b"GpLaMjEW", 8, bytes([0x48, 0x89, 0xFE, 0x48])),
       (b"pVOjnnmk", 2, bytes([0xBF, 0xF1, 0x26, 0xDC])),
       (b"RGiledp6", 7, bytes([0xB3, 0x07, 0x00, 0x00])),
       (b"Mvcezxls", 1, bytes([0x00, 0xFF, 0xD6, 0xC3])),
   ]
   
   alphabet = (string.ascii_letters + string.digits).encode()
   password = b""
   for salt, off, want in chunks:
       for combo in itertools.product(alphabet, repeat=4):
           guess = bytes(combo)
           h = hashlib.md5(guess + salt).digest()
           if h[off:off + len(want)] == want:
               password += guess
               break
   print("password:", password.decode())   # -> D1v1d3AndC0nqu3r
   PY

   Copied!

   Expected output

   password: D1v1d3AndC0nqu3r

   Each group is only 4 characters over a ~62-symbol alphabet, so each constraint solves in well under a second. Concatenating the four recovered groups gives the full 16-character password.

   What didn't work first

   Tried: Attempt to reverse the MD5 hash directly (e.g., using an online hash lookup or hashcat) to recover each 4-character group.

   Online MD5 rainbow tables cover common English words and short dictionary strings, not arbitrary 4-character alphanumeric combinations salted with an 8-byte binary suffix. Hashcat would also fail because you need to match only 4 bytes at a specific offset of the digest, not the full hash - standard hashcat modes compare the entire hash, so there is no built-in mode for 'partial-offset match'. The brute-force loop in the script is the correct approach because it checks exactly the relevant byte slice.

   Tried: Brute force all 16 characters at once as a single combined search.

   A 16-character alphanumeric brute force is astronomically large (62^16 possibilities). The key insight is that each 12-byte chunk is independent - the four groups never interact in the hash computation - so each 4-character sub-problem (62^4 at most ~14 million) can be solved separately in under a second. Treating it as one monolithic search breaks the divide-and-conquer structure the challenge is built on.

   Learn more

   Why brute force is the right tool. MD5 is not invertible, but you are not inverting it. You only need a 4-byte agreement at a fixed offset, and the preimage you control is 4 printable characters. That is at most a few million hashes per chunk, trivial on a laptop. The salts ensure each chunk has a unique answer, so the four groups concatenate into one password.

   For more on recovering keys from hash constraints and custom checkers, see the CTF Encodings guide.

3. Step 3

   Submit the password

   Observation

   I noticed the brute-force script produced a single 16-character password by concatenating the four recovered 4-character groups, which suggested submitting it directly to the netcat service to trigger the correct shellcode execution and reveal the flag.

   Connect and enter the recovered password. The service rebuilds the shellcode from your input, executes it, and prints the flag.

   bash

   nc mercury.picoctf.net <PORT_FROM_INSTANCE>

   Copied!

   bash

   # Enter: D1v1d3AndC0nqu3r

   Copied!

   Learn more

   The 16-character password splits into the four 4-character groups the brute force recovered, in order: D1v1, d3An, dC0n, qu3r.

Flag

The checker hashes your password and runs the hash bytes as machine code, so only the password whose MD5 outputs assemble into valid shellcode works. Reverse the byte-selection scheme, then brute-force each 4-character group so MD5(group+salt) carries the required byte. The password is D1v1d3AndC0nqu3r.