Patching Binaries, Cracking Crackmes, and Writing Keygens for CTF

Introduction

A crackme is the smallest complete reverse engineering problem: a program that reads a password, transforms it somehow, compares the result against something it already knows, and prints a flag only on a match. Nothing else happens. There is no network protocol to model, no heap to groom, no cryptography to attack. Every byte of the puzzle is sitting in one file on your disk.

That makes crackmes the best possible place to learn a habit that pays off everywhere else in reverse engineering: deciding which door to walk through before you start working. Beginners open the binary in a decompiler and start reading, because reading is the obvious move. It is frequently the slowest one. A binary that spends two hundred lines deriving a password from MD5 digests can be beaten in ninety seconds by breaking on strcmp and printing a register, and no part of that ninety seconds requires understanding MD5.

This guide lays out the four doors into a password check, explains how to pick one from symptoms you can observe in under a minute, and then covers the two things that get in the way: packers that hide the code from static analysis, and anti-debug checks that hide it from dynamic analysis. It assumes you can already navigate a decompiler and set a breakpoint. If you cannot yet, start with Ghidra for reverse engineering and the GDB guide, then come back.

The four doors

Every password check has the same shape. Somewhere there is a value the program expects, somewhere there is a transformation applied to your input, and somewhere there is a branch that decides between success and failure. You can attack any of those three places, or sidestep all of them.

The choice is not a matter of taste. It is decided by one question: where does the flag actually come from? If the binary prints the flag itself, patching is legitimate and fast. If the binary only says "correct" and the real flag comes from a server that wants the password you typed, patching the local copy tells you nothing, and you need door 1, 2, or 4.

Layer 0: is it packed?

Before any of the four doors, answer one question: does the file on disk contain the code that runs? A packer compresses the real program into a data blob and prepends a small stub that decompresses it into memory at startup and jumps to it. The program behaves identically, but a decompiler pointed at the file sees only the stub: a tight loop shuffling bytes, and nothing resembling the logic you came for.

UPX is the packer you will meet in CTF, and it is polite enough to sign its work. The markers are the literal string UPX! and the section names UPX0 and UPX1.

file out
strings out | grep -i upx
readelf -S out | head -20
 
# If the markers are there, decompress in place:
upx -d out

When the markers are absent but the symptoms persist, you are looking at a custom or modified packer, and the generic answer is to stop fighting it statically. Let the stub do its job, then take the memory image after it finishes: break at the entry point, single step until control transfers somewhere far from the stub, and dump the mapped region. The unpacking routine is a decompressor you do not have to understand as long as you are willing to read its output instead of its code.

Packer is exactly this in its simplest form: the name is the hint, strings shows the UPX markers, and one upx -d turns an opaque file into an ordinary ELF whose flag is sitting in a format string as ASCII hex.

Door 1: read the check

The honest door. Load the binary, find main, and follow the data. What makes this fast or slow is not the decompiler, it is whether you rename things as you go. A decompiled loop full of local_c and local_a8 is unreadable; the same loop with i and input is often obvious. Renaming is not cosmetic, it is the analysis.

Two patterns cover most of what you will meet:

The obfuscated constant. The expected password never appears as a string in the file, because it is stored transformed and decoded at startup. In Bypass Me, main calls a decode routine before it ever prompts, and that routine XORs twelve hardcoded bytes with 0xAA. Running strings on that binary is a waste of time by construction, because the plaintext never exists on disk. Once you can see the loop, the recovery is one line:

# the twelve bytes read straight out of the .rodata blob
python3 -c "print(bytes(b ^ 0xAA for b in bytes.fromhex('f9dfdacfd8f9cfc9d8cfde9b')).decode())"

The mangler. The program does not compare your input to a secret, it compares a transformation of your input to a stored value. This is the pattern in Classic Crackme 0x100, where three rounds of index-dependent modular arithmetic run over every byte before a memcmp. The trap here is subtle and catches almost everyone: the constants you can read out of the binary are the post-transformation target. Typing them in as the password makes the program mangle them a second time, and the comparison fails.

Gatekeeper is worth reading as an example of how little code a check can be while still being non-obvious. It enforces two constraints at once: the input string must be exactly three characters, and its numeric value must exceed 999. No decimal number satisfies both. The resolution is that the conversion is strtol with base 16 hardcoded, so 3e8 is three characters and 1000 at the same time. That fact is visible in one line of disassembly and invisible from the outside.

Door 2: steal the answer

The single highest-value trick in crackme solving, and the one most under-used by beginners: if the program has to compare your input against something, that something is in memory at the moment of comparison. You do not need to know how it got there.

Every derivation, however elaborate, ends with a plaintext value sitting in a register. Reading it is cheaper than reproducing it.

On x86-64 System V, the first two arguments to a function are in rdi and rsi. For strcmp(user_input, expected) that puts the expected value in rsi. Break there and print it:

gdb ./crackme
(gdb) break strcmp
(gdb) run
# type anything at the prompt
(gdb) x/s $rsi
(gdb) x/s $rdi

If the binary is statically linked or the comparison is inlined, break on the call site address instead of the symbol. If it uses memcmp, the length is in rdx, which tells you exactly how many bytes to dump:

(gdb) break memcmp
(gdb) run
(gdb) x/s $rsi
(gdb) x/32bx $rsi

keygenme is the canonical demonstration of why this door exists. The binary assembles its license key at runtime from a chain of MD5 digests and sprintf calls. Inverting that chain is not possible, because MD5 is not invertible. Reading the finished key out of the stack frame after the assembly completes takes one breakpoint:

# Address taken from Ghidra: the call to strlen fires once the key is complete
(gdb) break *0x<ADDRESS_FROM_GHIDRA>
(gdb) run
(gdb) x/s $rbp-0x30

The same idea generalises past strings. In jitfp, the password checker only works on its own host, because the values it needs are patched into the live process from outside rather than stored in the file. Static analysis of that binary cannot succeed by design. Resolving each function pointer in the running process to the character it accepts is the entire solution, and it is door 2 applied one byte at a time.

Door 3: patch the branch

Patching means editing the instruction stream so the program takes the path you want. In a password check that is nearly always one conditional jump. You have three edits available, and choosing between them matters less than people think as long as the replacement is the same length as the original.

Locate the branch, then edit the file offset that corresponds to it:

objdump -d --start-address=0x401000 crackme | grep -n 'jne\|je\|cmp'
 
# Say the jne sits at virtual address 0x401180. Get its file offset:
#   file_offset = vaddr - section_vaddr + section_file_offset
readelf -S -W crackme | grep -w .text   # gives both section values
 
# With .text at vaddr 0x401000, offset 0x1000, that jne is at file offset 0x1180:
printf '\x74' | dd of=crackme bs=1 seek=$((0x1180)) conv=notrunc
 
# Confirm the edit landed on the instruction you meant, by virtual address:
objdump -d crackme | grep -B2 -A2 '401180:'

Ghidra can do the same edit with more safety rails: select the instruction, use Patch Instruction to rewrite it, then export with Save As and the Original File format so the rest of the file is preserved byte for byte. That path is worth learning because it also works when the patch is several instructions long, where hand-computing offsets gets error-prone.

No Way Out and Need for Speed are both patch problems rather than password problems. In the second, the binary destroys its own key after a short timer, so the goal is not to guess anything but to change the timer or the branch that depends on it. Once you accept that the program is a text file you are allowed to edit, that stops feeling like cheating and starts feeling like the intended solution, because it is.

The Python-flavoured version of the same door is patchme.py, where the "binary" is source and the patch is a one-line edit. It is the same skill with the disassembly step deleted, which makes it a good first exposure to the idea.

Door 4: write the keygen

A keygen is a program that produces valid passwords. You need one when the check is a transformation you must run backwards, and especially when a remote service will demand the real key that your locally patched binary never made you produce.

The question that decides whether a keygen is even possible is: is the transformation invertible? Three answers, three strategies.

The Classic Crackme transformation is the invertible kind, and its structure is the reason why. Each round adds an offset that depends only on the character index, never on the character value. That makes every round a shift, and shifts compose and undo cleanly:

target = 'lxpyrvmgduiprervmoqkvfqrblqpvqueeuzmpqgycirxthsjaw'
 
def unround(s):
    out = []
    for j, ch in enumerate(s):
        offset = ((j * 0x55) ^ 0x33) & 0xF   # recompute the round's index term
        out.append(chr((ord(ch) - ord('a') - offset) % 26 + ord('a')))
    return ''.join(out)
 
password = target
for _ in range(3):          # the binary applies three rounds, so undo three
    password = unround(password)
print(password)

When the transform is not invertible but the input space is small, brute force is a perfectly respectable keygen. A four-character lowercase key is 456,976 candidates, which a Python loop chews through faster than you can read this sentence. Reach for that before reaching for cleverness.

keygenme-py sits between the two: the validation is readable Python, so the inversion is a matter of reading carefully rather than reasoning about registers. It is the friendliest entry point in the cluster and worth doing before the compiled ones.

When the binary fights back

Doors 2 and 3 both assume you can run the program under your control. Anti-debugging is the set of tricks that make that assumption false. None of them are unbeatable. All of them are designed to cost you time, and knowing the catalogue turns each one back into a thirty-second obstacle.

The Windows anti-debug series walks this ladder deliberately. WinAntiDbg0x100 has a single IsDebuggerPresent call, and the fix is to break on the TEST instruction and set EAX to zero so the conditional jump reads a clean result. WinAntiDbg0x200 stacks three checks, which teaches the real lesson: map every guard statically before you attach, so you can place all the breakpoints in one pass instead of discovering them one crash at a time.

WinAntiDbg0x300 is the instructive one, because it makes debugging genuinely impractical: it is UPX packed, and it runs a loop that continuously re-checks for a debugger. There is no register to patch at the right moment when the check never stops happening. The answer is to give up on dynamic analysis entirely, unpack, patch the jump statically in Ghidra, and run the modified file with no debugger attached at all. Anti-debugging defends against debuggers; it does not defend against editing.

Gatekeeper and Bypass Me both reward this framing too: neither is really hard, but both punish committing to one tool and grinding.

Local crack, remote flag

The most common wasted hour in this category comes from patching a binary whose flag was never in it. Read the challenge brief for a nc line. If there is one, the architecture is almost always: a local binary you can analyse freely, and a remote service holding the flag that will accept exactly one string.

Under that architecture, patching proves nothing. Your patched copy prints "correct" for any input, which is precisely as informative as printing nothing. The remote service is unpatched, so you need the actual password, which means door 1, 2, or 4.

That third row is the setup in Bypass Me: the binary is setUID on the remote host, so the flag is readable only through the program itself. Copy it down with scp to analyse at leisure, recover the password locally, then spend it on the remote instance.

picoCTF challenges

Worked walkthroughs for every challenge in this cluster, ordered so that each one adds exactly one idea to the previous.

Quick reference

# Triage
file target && strings target | head -40
strings target | grep -i upx        # packed?
objdump -d target | grep -c ''      # tiny output means packed or stripped
 
# Unpack
upx -d target
 
# Steal the expected value
gdb ./target
(gdb) break strcmp     # or memcmp / strncmp
(gdb) run
(gdb) x/s $rsi         # second argument: the expected string
(gdb) x/32bx $rsi      # when it is bytes rather than text
 
# Find the branch
objdump -d target | grep -n 'cmp\|test\|jne\|je'
 
# Patch one byte (jne 0x75 -> je 0x74)
printf '\x74' | dd of=target bs=1 seek=$((0xOFFSET)) conv=notrunc
 
# Bypass a Linux ptrace check without patching the file
gdb -ex 'catch syscall ptrace' -ex run ./target

Where to go next: Ghidra for the static half, GDB for the dynamic half, x86-64 assembly when the decompiler output stops being enough, and angr when the transform is invertible in principle but tedious in practice and you would rather let a solver do it.

Sources and further reading

The primary documentation behind every technique above.

• UPX for the packer format, its markers, and the exact semantics of -d versus -t.
• Ghidra for the decompiler and its Patch Instruction workflow, and the GDB manual for breakpoint and memory examination syntax.
• ptrace(2) for why a process can only be traced once, which is the mechanism behind the Linux anti-debug check, and IsDebuggerPresent for the Windows equivalent and the PEB flag it reads.
• The ELF specification for the section and segment layout you need when converting a virtual address to the file offset a patch must be written at.