picoCTF Solutions
picoCTF 2026Reverse EngineeringMedium

Secure Password Database

Published: March 20, 2026

Description

A password program proudly shows what it stores in its database output. Download `system.out`, inspect the binary, and recover what it is really checking.

Setup

Download system.out

Download system.out and inspect it.

Use strings, file, and reverse engineering tools to analyse the output.

file system.out
strings system.out

Solution

  1. Step 1Identify the file type
    Download system.out and check what kind of file it is.
    file system.out
    strings system.out | grep -i 'picoCTF|flag|password'
  2. Step 2Try to find the flag in binary strings
    Run strings to search for a direct or hex/base64-encoded picoCTF flag.
    strings system.out | grep picoCTF
    strings system.out | grep -E '[0-9a-fA-F]{40,}'
    python3 -c "import re, base64; data=open('system.out','rb').read(); print(re.findall(rb'picoCTF\{[^}]+\}', data))"
  3. Step 3Trace the binary with ltrace
    Run the binary under ltrace to capture the arguments to strcmp/memcmp at the moment of the password check. The comparison value is the stored (possibly transformed) password.
    chmod +x system.out
    ltrace -s 256 ./system.out <<< 'test'
    # Look for strcmp/memcmp calls — the second argument is the expected value
  4. Step 4Analyse the transformation with GDB or objdump
    If ltrace doesn't reveal the flag directly, set a breakpoint at the comparison function to read memory, or use objdump to find hardcoded comparison values in the binary.
    objdump -d system.out | grep -A5 'cmp\|strcmp\|memcmp'
    objdump -s -j .rodata system.out | strings
    gdb -q ./system.out -ex 'break strcmp' -ex 'run <<< "AAAA"' -ex 'x/s $rdi' -ex 'x/s $rsi' -ex 'quit'
  5. Step 5Try XOR brute-force decode
    If the flag is XOR-encoded in the binary, try every single-byte key against the raw binary data.
    python3 - <<'EOF' import re data = open('system.out', 'rb').read() for key in range(1, 256): dec = bytes(b ^ key for b in data) m = re.search(rb'picoCTF[{][^}]+[}]', dec) if m: print(f'key=0x{key:02x}: {m.group().decode()}') EOF

Flag

picoCTF{s3cur3_p4ssw0rd_db_...}

Reverse engineering challenge. Try strings first, then ltrace to capture strcmp arguments, then objdump .rodata, then XOR brute-force decode. The program transforms the password before storing and comparing it.