Buffer Overflow and Binary Exploitation for CTF

Introduction

You paste a long run of As into a prompt, the program prints Segmentation fault, and for a second nothing seems to have happened. But you just overwrote the address the CPU was about to jump to, and that single crash is the entire game. Binary exploitation (pwn) is one of the most hands-on CTF categories. Instead of attacking web application logic or cracking ciphers, you analyze compiled programs, find memory safety bugs, and craft inputs that redirect execution - typically to pop a shell or print a flag.

The technique is old and undefeated. Aleph One published "Smashing the Stack for Fun and Profit" in Phrack issue 49 in 1996, and the payload layout in that article (padding, saved frame pointer, return address) is the same one you will type today. Three decades later, MITRE still ranked CWE-787, the out-of-bounds write class that contains stack overflows, second on its 2024 Top 25 Most Dangerous Software Weaknesses.

Some of the plumbing did get fixed. The C standard committee removed gets outright in C11 after deprecating it in C99 TC3, because the function has no way to know how large its destination is; the Linux manual page still carries the line "Never use this function." CTF binaries put it back on purpose, which is why gets in a disassembly is the fastest confirmation you have a stack overflow.

picoCTF has an unusually well-structured binary exploitation track. The challenges progress from crash-only stack overflows, through format string leaks, all the way to multi-step heap exploits. This guide covers each technique in order of the picoCTF challenge series they appear in.

Stack basics

When a function is called, the CPU pushes the return address onto the stack - the address of the next instruction to execute when the function returns. The function then allocates space for its local variables (the stack frame) immediately above the return address.

A stack buffer overflow occurs when a local array (buffer) receives more data than it can hold, overwriting adjacent memory including - crucially - the saved return address. By controlling what goes there, you control where the program jumps on function return.

# Memory layout (grows downward)
| higher addresses |
|------------------|
| saved return addr| <-- overwrite this
| saved RBP        |
| buffer[64]       | <-- fills from here
| lower addresses  |
 
# A 64-byte buffer needs 64 + 8 (RBP) = 72 bytes padding
# before you reach the return address (x86-64)

The 8 bytes for the saved RBP are not a convention someone picked, they are the System V AMD64 ABI frame layout: the return address sits at %rbp+8 and the saved base pointer at %rbp, both 8 bytes wide because the ABI defines a 64-bit pointer. On 32-bit i386 the same two slots are 4 bytes each, which is the single most common reason a payload that worked on one binary is off by 8 on another.

Use pwntools' cyclic to find the exact offset without counting manually. It generates a de Bruijn sequence in which every substring of length n (default 4) appears exactly once, so the four bytes sitting in the crashed instruction pointer identify their own offset. With the default lowercase-alphabet of 26 characters that gives a unique run 456,976 bytes long, far more than any stack frame you will meet.

python3 -c "from pwn import *; print(cyclic(200))" | ./binary
# binary crashes with a cyclic pattern in rsp
 
# Find the offset from the crashing address
python3 -c "from pwn import *; print(cyclic_find(0x6161616c))"  # -> 44

The offset is different for every binary. The 72 above (64-byte buffer plus 8 for the saved RBP) is one worked example; the 44 that cyclic_find returns here belongs to a different example binary with a smaller buffer and layout. Never assume a number, always measure the offset for the binary in front of you with cyclic.

ret2win

A ret2win challenge has a function somewhere in the binary (commonly called win, flag, or secret) that prints the flag or spawns a shell. The goal is to overwrite the return address with the address of that function.

# Find the win function address
objdump -d ./binary | grep win
# or
nm ./binary | grep win
# or in pwntools
elf = ELF('./binary')
win_addr = elf.symbols['win']

Build the payload: padding to reach the return address, then the win function address:

from pwn import *
 
elf    = ELF('./binary')
p      = process('./binary')  # or remote('host', port)
offset = 44                   # bytes until return address
 
payload = flat(
    b'A' * offset,
    elf.symbols['win'],
)
p.sendlineafter(b'Input: ', payload)
p.interactive()

Format string bugs

A format string vulnerability occurs when user-controlled input is passed as the first argument to printf (or sprintf, fprintf) without a format string:

// Vulnerable
printf(user_input);
 
// Safe
printf("%s", user_input);

Format strings can be exploited for both reads and writes:

Reading memory (leak addresses)

# Dump stack values as hex
python3 -c "print('%p.%p.%p.%p.%p.%p.%p.%p')" | ./binary
 
# Read a specific stack offset
python3 -c "print('%7$p')" | ./binary   # 7th argument

Writing memory (%n)

The %n specifier writes the number of bytes printed so far to a pointer on the stack. By placing a target address on the stack and using positional arguments, you can write arbitrary values to arbitrary addresses - typically to overwrite a return address or a GOT entry. Both halves of that trick are standardised: POSIX defines %n and the %N$ positional form, so %7$p reads the seventh argument on every conforming libc rather than being a glibc quirk.

Note what stops it in production. Building with -D_FORTIFY_SOURCE=2 makes glibc abort at runtime when a format string containing %n lives in writable memory, and GCC's -Wformat-security warns at compile time about printf(user_input). A challenge binary that reaches you with a working %n was compiled with those turned off on purpose.

# pwntools fmtstr_payload automates the write
from pwn import *
 
p        = process('./binary')
offset   = 6      # position of your input on the stack
target   = 0x...  # address to overwrite
value    = 0x...  # value to write
 
payload  = fmtstr_payload(offset, {target: value})
p.sendline(payload)

Heap exploitation

Heap exploitation targets bugs in dynamic memory allocation: use-after-free, double-free, and heap overflow. Modern glibc uses the tcache (per-thread cache) for small allocations, which is the primary target in introductory CTF heap challenges.

The tcache arrived in glibc 2.26 (2017) as a lock-free fast path, and its constants matter when you are exploiting it. There are 64 tcache bins covering chunk sizes up to 1,032 bytes on 64-bit, and each bin holds at most 7 chunks before allocations spill into the fastbins and unsorted bin. That 7 is why heap challenges so often ask you to allocate and free exactly eight objects: the eighth is the one that behaves differently.

Which glibc you are facing decides which attack still works. glibc 2.29 added a key field to freed tcache chunks so a naive double-free is detected, and glibc 2.32 introduced safe-linking, which stores the forward pointer XORed with the chunk address shifted right by 12 bits. Under safe-linking you must leak a heap address before you can forge a tcache pointer at all. Check the target's libc version first with ./libc.so.6 or strings libc.so.6 | grep GLIBC.

tcache poisoning

After a free(), glibc writes the address of the next free chunk into the freed chunk's metadata. If you can write to a freed chunk (use-after-free), you can overwrite this pointer and make the next malloc() return a chunk at an arbitrary address - for example, a function pointer table or a stack-resident variable.

1. Allocate chunk A
2. Free chunk A      -> tcache: [A]
3. Write to A        -> corrupt the next pointer in A's metadata
4. malloc()          -> returns A (pops from tcache)
5. malloc()          -> returns the corrupted address you wrote
                        (now you control what malloc gives out)

PIE and ASLR bypass

ASLR (Address Space Layout Randomization) randomizes the base address of the stack, heap, and shared libraries at every run. PIE (Position-Independent Executable) extends this to the binary itself. Without a leak, you cannot predict where your target function lives.

The bypass is always the same: find a way to leak a runtime address, compute the binary base from it, then calculate the address of your target function.

Linux exposes the policy in one file. The kernel's kernel.randomize_va_space sysctl takes three values: 0 disables randomization, 1 randomizes stack, mmap, and VDSO, and 2 (the default on every mainstream distribution) adds the heap. On x86-64 the mmap region is randomized with 28 bits of entropy by default, so guessing a libc base blind is 268 million attempts and not a strategy.

It was a strategy on 32-bit. Shacham and co-authors showed in "On the Effectiveness of Address-Space Randomization" (CCS 2004) that PaX on 32-bit x86 left only 16 bits of useful entropy, and they brute-forced a real Apache target in an average of 216 seconds. That result is the reason 64-bit address space, not randomization alone, is what makes ASLR worth having, and the reason every modern challenge hands you a leak instead.

Leaking a PIE address

# Common sources of leaks:
# 1. Format string: %p chain until you see an address in binary range
# 2. printf with %s on a pointer that points inside the binary
# 3. The binary prints an address itself (helpful challenge setup)
 
# Once you have a leak:
leaked_addr  = int(p.recvline(), 16)
binary_base  = leaked_addr - elf.symbols['known_function']
win_addr     = binary_base + elf.symbols['win']

pwntools primer

pwntools is the standard Python library for binary exploitation in CTF. It handles process interaction, struct packing, ROP chain construction, and shellcode generation. If you are new to Python scripting for CTF in general, the Python for CTF guide covers binary I/O, encoding, and socket scripting before getting to pwntools.

Install

pip install pwntools

Boilerplate

from pwn import *
 
context.binary = elf = ELF('./binary')
context.arch   = 'amd64'   # or 'i386'
 
# Local
p = process('./binary')
 
# Remote
p = remote('challenge.host', 1337)
 
# Useful helpers
p.sendline(payload)          # send + newline
p.sendlineafter(b'> ', payload)  # wait for prompt then send
p.recvuntil(b'0x')           # receive until marker
leak = int(p.recvline(), 16) # parse hex address
p.interactive()              # hand control to your terminal

Struct packing

p64(0xdeadbeef)   # little-endian 8-byte pack (64-bit)
p32(0xdeadbeef)   # little-endian 4-byte pack (32-bit)
flat(b'A'*44, p64(win))  # build a payload inline

Quick reference

Sources and further reading

Every offset, entropy figure, and glibc version above is checkable against one of these. When an exploit misbehaves, the ABI and the glibc release notes usually explain it faster than a debugger does.

• Aleph One, "Smashing the Stack for Fun and Profit" (Phrack 49-14, 1996). The original, and still the clearest.
• System V AMD64 ABI (frame layout, 16-byte stack alignment, argument registers) and the ELF specification (ET_DYN versus ET_EXEC).
• Linux kernel sysctl documentation for randomize_va_space, and Shacham et al., "On the Effectiveness of Address-Space Randomization" (CCS 2004) for the 16-bit brute-force result.
• glibc release notes for 2.26 (tcache), 2.29 (double-free key), and 2.32 (safe-linking).
• POSIX printf specification (%n and positional %N$ arguments), gets(3), and glibc source fortification.
• pwntools documentation, in particular cyclic and de Bruijn sequences.
• CWE-787: Out-of-bounds Write for the canonical description of the defect class, and the glibc memory allocation manual for what lies past the end of the buffer once the overflow leaves the stack.