Description
A binary accepts a 100-character hex key (lowercase a-f, 0-9) and validates it against a hardcoded target using a custom jumble function. Recover the key and XOR-decrypt the flag.
Setup
Download the otp binary and flag.txt from the challenge page.
Make the binary executable: chmod +x otp
chmod +x otpSolution
Want to try it yourself first?
The guided walkthrough reveals hints one step at a time.
Step 1Analyze the binary in Ghidra
ObservationThe challenge provides a compiled binary and no source. Static reverse engineering in Ghidra is what will show the validation logic, the jumble transformation, and the hardcoded target the input is checked against.Load otp in Ghidra and look at main. The program takes a command-line argument, validates that each character is a lowercase hex digit (a-f or 0-9), applies a custom jumble transformation, and compares the result to a hardcoded 100-character string. The transformation and the target string are visible in Ghidra's decompiler view.Learn more
Ghidra is a free, open-source reverse engineering framework developed by the NSA. Its decompiler converts raw machine instructions back into readable C-like pseudocode. Here it reveals the validation pattern: apply a jumble() function to each character, then call strncmp() against a fixed expected string.
The key insight is that you do not need to algebraically invert the jumble math. The comparison is done with strncmp(), which you can observe at runtime. There are only 16 possible input characters (hex digits), so you can test each one at each position empirically.
Step 2Set a GDB breakpoint at strncmp to observe both arguments
ObservationThe Ghidra decompiler shows the validation ending in a strncmp that compares the jumbled input against a fixed target. A GDB breakpoint there reads both string pointers from RDI and RSI, so you can see directly whether a candidate character matches.Run the binary under GDB with a 100-character test input and set a breakpoint at the strncmp call. When the breakpoint hits, inspect the RSI and RDI registers: one holds your jumbled input, the other holds the expected target. On a 64-bit machine, strncmp arguments are passed in registers. This lets you see whether any character position matches the target.bashgdb ./otpbash(gdb) break strncmpbash(gdb) run $(python3 -c "print('6' + '0'*99)")bash(gdb) x/s $rsibash(gdb) x/s $rdiWhat didn't work first
Tried: Reading the jumbled output from RAX instead of RDI/RSI after strncmp returns
After strncmp returns, RAX holds the integer result, 0 for a match, not a pointer to either string. The string pointers are only live in RDI and RSI at the moment the function is entered, so set the breakpoint at the call site or function entry, not at the return.
Tried: Using 'info registers' to find the expected target, then reading all 100 characters from memory at once
The full expected string is in memory, but which of RDI and RSI holds the target rather than your jumbled input depends on argument order, which varies by compiler. Read the wrong pointer and you get your own input back, which looks like a match at every position. Check which register holds bytes that stay constant across runs before trusting the read.
Learn more
Manually testing one character reveals the mapping: putting in 'a' produces 'c', going up by two each step, so 'b' produces 'e', 'c' produces 'g', and so on. Since the first expected byte is 'M' and the mapping increments by 2, '6' maps to 'M', which GDB confirms. Once the first character matches, the rest can be automated.
On x86-64, the first two arguments to any function are passed in
rdiandrsi. Inspecting them at the strncmp breakpoint shows both the jumbled version of your input and the hardcoded expected string side by side, without modifying the binary.Step 3Automate character-by-character recovery with a Python GDB script
ObservationThe 100-position key allows only 16 hex characters per position, and the strncmp oracle confirms matches one byte at a time. That is 1,600 GDB invocations, so a Python script beats repeating the manual process 100 times.Write a Python script that calls GDB in a loop. For each of the 100 positions, try all 16 hex characters. At the strncmp breakpoint, compare RSI and RDI. When they match at the current position, that character is correct. Record it and move to the next position. After all 100 positions the full key is recovered.pythonpython3 solve_proc.pyWhat didn't work first
Tried: Trying to invert the jumble math algebraically from the Ghidra pseudocode instead of using the GDB oracle
Ghidra sometimes misrepresents integer widths, sign extension, and modular arithmetic, producing a subtly wrong inverse formula. That yields a 100-character key which still fails strncmp when the binary runs it. The GDB oracle sidesteps the problem entirely, because it never needs the math: it only watches which input matches the first N bytes.
Tried: Running the GDB script with the candidate character appended at the end rather than at position N with '0' padding after it
The jumble function runs per character, but strncmp checks the whole string from byte 0. Put the known-good prefix plus a candidate at positions 0 to N with the rest wrong and strncmp still returns nonzero, even when position N is right. Pad positions N+1 through 99 with a fixed neutral character so only 0 to N are being tested.
Learn more
The script runs the binary under GDB with the prefix of known-good characters, then the candidate character, then 99 minus the current position '0' characters to pad to 100. At the strncmp breakpoint it reads RSI and RDI. If the first N+1 bytes match, the candidate character is correct.
This reduces the brute force from 16^100 (completely infeasible) to 16 * 100 = 1,600 GDB invocations. Each check is independent per position because the jumble function is applied per character. Running the script fills in the key character by character: it starts with the '6' found manually, then recovers all 100 characters automatically.
Step 4XOR the recovered key with flag.txt
ObservationThe challenge name references a One-Time Pad, and the binary produces a 100-character hex key. Decoding that hex to 50 bytes and XORing it with flag.txt reverses the encryption.The recovered key is a hex string. Decode it to bytes and XOR it byte-by-byte with the contents of flag.txt. The result is the plaintext flag. An online XOR tool also works for this step: paste the hex key and flag.txt bytes, then convert the hex output to ASCII.pythonpython3 -c " key = bytes.fromhex('<recovered_100_char_key>') flag_enc = open('flag.txt','rb').read() print(''.join(chr(a^b) for a,b in zip(key,flag_enc))) "Expected output
picoCTF{cust0m_jumbl3s_4r3nt_4_g0Od_1d3A_...}What didn't work first
Tried: Treating the recovered key as ASCII text rather than a hex string and XORing it directly with flag.txt
The 100-character key is hex-encoded, so two characters make one byte. Pass it to XOR as a 100-byte ASCII string and the key length doubles and every byte misaligns, giving garbage. Call bytes.fromhex() first to turn it into its 50-byte binary form.
Tried: XORing the full flag.txt file length against the key without checking that the lengths match
If flag.txt is longer than the 50-byte decoded key, zip() silently truncates and the tail of the flag disappears. If it is shorter, some key bytes go unused and the output still looks fine. zip() without a length check hides the mismatch; itertools.zip_longest or an explicit assertion surfaces it.
Learn more
The One-Time Pad (OTP) XORs each plaintext byte with the corresponding key byte to produce ciphertext. XORing the ciphertext with the same key recovers the plaintext because XOR is its own inverse:
(plaintext XOR key) XOR key = plaintext.The challenge's title is ironic: this is a broken OTP. A real OTP requires a key generated from a cryptographically secure source. Here the key is recoverable because the validation logic leaks one character at a time through GDB observation. The flag message confirms this: custom jumbles are not a good idea.
Interactive tools
- XOR CipherXOR-decrypt hex or text ciphertext with a known key, or brute-force the single-byte key automatically.
Flag
Reveal flag
picoCTF{cust0m_jumbl3s_4r3nt_4_g0Od_1d3A_...}
The trailing hex suffix (e.g. 15e89ca4 or 33ead16f) is generated per instance. GDB with a breakpoint at strncmp lets you observe both the jumbled input and the expected target in RSI/RDI. Testing all 16 hex characters per position (1,600 total checks) recovers the full 100-character key, which is then XORed with flag.txt.