Easy as GDB picoCTF 2021 Solution

Description

1. Step 1Locate the check_answer function
   Load the binary in Ghidra or use GDB to identify the function that validates input. The binary prompts for a flag, mangles it, and then calls a check_answer function that iterates character by character, incrementing a counter for each correct character. Find the address where the counter is stored (e.g., at ebp - 0x18).
   bash

   gdb -q ./brute

   Copied!

   bash

   (gdb) info functions

   Copied!

   bash

   (gdb) disas main

   Copied!
   Learn more

   The check_answer function counts how many input characters match the correct flag, storing the count at a known stack offset. By reading this offset after each guess attempt, you know exactly how many characters you have right without needing to reverse engineer the full mangling algorithm.

2. Step 2Use GDB to brute-force one character at a time
   Set a breakpoint inside check_answer where the match count is available. Try each character of the alphabet and keep the one that increments the count. Use pwntools to script GDB as a subprocess so you don't need to type this manually for every character.
   python

   python3 << 'EOF'
   import subprocess
   import string
   
   # Run GDB as a subprocess and interact with it
   alphabet = string.ascii_uppercase + string.ascii_lowercase + string.digits + "{}_"
   flag = "picoCTF{"  # known prefix
   
   while not flag.endswith("}"):
       best_count = len(flag)
       best_char = None
       for c in alphabet:
           guess = flag + c + "A" * (40 - len(flag) - 1)
           # Run GDB, set breakpoint at check_answer counter address, read the count
           gdb_script = f"""
   file brute
   break *0x56555a97
   run <<< '{guess}'
   printf "%d\n", *(int*)($ebp - 0x18)
   quit
   """
           result = subprocess.run(
               ["gdb", "-batch", "-ex", gdb_script.strip()],
               capture_output=True, text=True
           )
           # Parse the count from GDB output
           for line in result.stdout.splitlines():
               try:
                   count = int(line.strip())
                   if count > best_count:
                       best_count = count
                       best_char = c
                   break
               except ValueError:
                   pass
       if best_char:
           flag += best_char
           print(f"Flag so far: {flag}")
   
   print(f"Final flag: {flag}")
   EOF

   Copied!
   Learn more

   Timing/oracle attack via instruction count. The check_answer function advances a counter for each correct character before stopping. By comparing the counter value for different input guesses, you can tell which character is correct (it produces a higher counter). This is equivalent to a side-channel attack: instead of breaking the cipher, you exploit observable behavior differences.

   An alternative approach is using valgrind --tool=callgrind to count instructions: a correct character causes more instructions to execute (the loop goes one iteration further). The character that produces the most instruction executions is the correct one for that position.

   In GDB, printf "%d\\n", *(int*)($ebp - 0x18) reads a 4-byte integer from the stack frame at the location of the match counter. This address was found by reading the check_answer disassembly.

Flag

Set a breakpoint in check_answer and read the per-character match counter - the character that increments it is correct. Automate with GDB scripting to brute-force the flag one character at a time.