picoCTF Solutions
picoCTF 2026Reverse EngineeringMedium

Secure Password Database picoCTF 2026 Solution

Published: March 20, 2026

Description

A password program proudly shows what it stores in its database output. Download system.out, inspect the binary, and recover what it is really checking.

Setup

Download system.out

Download system.out and inspect it.

Use strings, file, and reverse engineering tools to analyse the output.

bash
file system.out
bash
strings system.out

Solution

Walk me through it
  1. Step 1Identify the file type
    Download system.out and check what kind of file it is.
    bash
    file system.out
    bash
    strings system.out | grep -i 'picoCTF|flag|password'
    Learn more

    The file command examines the first few bytes of a file (the magic bytes) to identify its actual format, regardless of the file extension. Many formats begin with a unique byte signature: ELF executables start with 0x7f454c46 ("ELF"), ZIP files with 0x504b0304 ("PK"), and Java class files with 0xcafebabe. The file extension is just a naming convention and can be misleading.

    strings extracts printable ASCII/UTF-8 sequences of a minimum length (default 4 characters) from any binary file. While simple, it is often the fastest way to find hardcoded passwords, error messages, URLs, or flag strings embedded directly in a binary. Many CTF challenges at the beginner level are solved simply by running strings | grep picoCTF, making it always worth trying first before reaching for heavier tools.

  2. Step 2Try to find the flag in binary strings
    Run strings to search for a direct or hex/base64-encoded picoCTF flag.
    bash
    strings system.out | grep picoCTF
    bash
    strings system.out | grep -E '[0-9a-fA-F]{40,}'
    python
    python3 -c "import re, base64; data=open('system.out','rb').read(); print(re.findall(rb'picoCTF\{[^}]+\}', data))"
    Learn more

    Binary programs often store sensitive strings in their .rodata (read-only data) section, which holds string literals and compile-time constants. If a flag or password is hardcoded directly, strings will find it. The regex pattern [0-9a-fA-F]{40,} targets long hexadecimal strings that might be encoded representations of the flag - 40+ hex characters corresponds to a 20+ byte value, which is typical for encoded flags or hashes.

    The Python one-liner directly scans the raw bytes for the picoCTF{...} pattern, which catches cases where the string is present in the binary but strings might miss it due to null bytes or non-printable characters interspersed in the data. This brute-force binary scan is effective because CTF flags have a known prefix and suffix structure.

  3. Step 3Trace the binary with ltrace
    Run the binary under ltrace and watch the arguments passed into strcmp/memcmp. The second argument is whatever the binary is comparing your input against - often the flag or a password that unlocks it.
    bash
    chmod +x system.out
    bash
    ltrace -s 256 ./system.out <<< 'AAAA'
    bash
    # Look for strcmp/memcmp calls; the second arg is the expected value
    Learn more

    A typical ltrace line looks like:

    strcmp(0x7ffffffde000 "AAAA", 0x40c0d8 "correctflag{...}")  = -1

    That second pointer plus the string after it is the expected value - the secret. -s 256raises the printable string limit (default 32) so long flags don't get truncated.

    ltrace works at the PLT/GOT level: it patches the GOT entries for library functions to log the call before forwarding to the real implementation. strace does the same for syscalls (read, write, open). Limitation: ltrace only sees dynamically linked library calls. If the binary is statically linked or rolls its own comparison loop, drop into GDB.

    See the GDB CTF guide for the breakpoint-driven path and Ghidra for reverse engineering for the static-analysis path.

  4. Step 4Analyse the transformation with GDB or objdump
    If ltrace shows the comparison happens but the second arg looks like garbage, the flag is packed (gzip, zlib, XOR). Dump .rodata and check the head bytes for known magic numbers, then break on strcmp in GDB to inspect memory live.
    bash
    # Dump the head of .rodata - look for 1f 8b (gzip), 78 9c (zlib), or other magic
    bash
    objdump -s -j .rodata system.out | head -40
    bash
    objdump -d system.out | grep -A5 'cmp\|strcmp\|memcmp'
    bash
    # GDB: strcmp may fire many times before the real check, keep continuing
    bash
    gdb -q ./system.out
    bash
    (gdb) break strcmp
    bash
    (gdb) run <<< "AAAA"
    bash
    (gdb) x/s $rsi
    bash
    (gdb) c
    bash
    (gdb) x/s $rsi
    bash
    (gdb) c       # repeat until you see picoCTF{ in $rsi
    Learn more

    Magic-byte recognition for packed .rodata:

    • 1f 8b = gzip header. Pipe through gunzip.
    • 78 9c / 78 da = zlib stream. Use python -c 'import zlib,sys;sys.stdout.buffer.write(zlib.decompress(open(...,"rb").read()))'.
    • 50 4b 03 04 = ZIP. unzip.
    • 7f 45 4c 46 = ELF. The binary embeds another binary.

    GDB workflow on strcmp. break strcmp + run drops you at the first strcmp call. In x86-64, $rdi is arg1 (your input), $rsi is arg2 (the expected value). x/s $rsi prints the expected string. strcmp may fire many times before the real check - libc setup, locale code, and helper routines all call it. Keep typing c (continue) and x/s $rsi until you see something that looks like the flag.

  5. Step 5Try XOR brute-force decode
    If the flag is XOR-encoded in the binary, try every single-byte key against the raw binary data.
    python
    python3 - <<'EOF'
import re
data = open('system.out', 'rb').read()
for key in range(1, 256):
    dec = bytes(b ^ key for b in data)
    for m in re.finditer(rb'picoCTF\{[^}]+\}', dec):
        if m.group().startswith(b'picoCTF{'):
            print(f'key=0x{key:02x}: {m.group().decode()}')
EOF
    Learn more

    Single-byte XOR encoding is one of the simplest obfuscation techniques: XOR every byte of the plaintext with the same key byte. It provides no real cryptographic security because there are only 255 possible keys (1-255; key=0 is a no-op), all of which can be tried in milliseconds. The attack script exploits the known plaintext of picoCTF{ - because the flag must start with this prefix, any successful decryption will produce exactly this string, making it trivial to verify which key is correct.

    XOR encoding is used in malware to obfuscate strings (command-and-control URLs, registry keys, error messages) from basic static analysis tools. Malware analysts routinely apply this same brute-force technique when reverse engineering malicious binaries. More sophisticated malware uses multi-byte XOR keys or rolling keys, but the same principle applies - the known structure of the plaintext (e.g., Windows API function names, HTTP headers) enables a crib dragging attack to recover the key.

    This brute-force approach works efficiently because Python can process binary data extremely fast in a tight loop, and the comparison is a simple regex match. For a 1MB binary, all 255 keys can be tried in well under a second. The pattern [{][^}]+[}] is written with character classes to avoid escaping issues inside Python byte strings.

Flag

picoCTF{s3cur3_p4ssw0rd_db_...}

Reverse engineering challenge. Try strings first, then ltrace to capture strcmp arguments, then objdump .rodata, then XOR brute-force decode. The program transforms the password before storing and comparing it.

Want more picoCTF 2026 writeups?

Browse picoCTF 2026Full library

Useful tools for Reverse Engineering

Related reading

What to try next