Easy as GDB picoCTF 2021 Solution

Description

1. Step 1

   Locate the comparison loop

   Observation

   I noticed the binary takes input and compares it to the flag character by character, which suggested there must be a specific compare instruction inside a loop where both the encoded input byte and the expected byte are simultaneously live in registers and readable without reversing the encoding.

   Load the binary in Ghidra/GDB. It reads your input, runs it through an encoding pipeline (an XOR pass plus a permutation/reorder of the bytes), then compares the encoded input against an encoded copy of the real flag, one byte at a time in a loop. Find the exact compare instruction (the cmp/loop body) and set a breakpoint there; that is where the two bytes being compared are live in registers (commonly AL = your encoded byte, DL = the expected byte).

   bash

   gdb -q ./brute

   Copied!

   bash

   (gdb) info functions

   Copied!

   bash

   (gdb) disas main

   Copied!

   bash

   # find the per-byte cmp instruction inside the check loop

   Copied!

   Expected output

   FLAG: picoCTF{I_5D3_A11DA7_...}

   What didn't work first

   Tried: Set a breakpoint on the strcmp or memcmp function expecting the whole flag to be compared at once.

   The binary does not call strcmp or memcmp. It uses a hand-rolled loop that compares one byte at a time with a cmp instruction inside the loop body. Breaking on strcmp fires nothing and the program exits cleanly without hitting any breakpoint, so you see no comparison data at all. You need to find and break on the actual cmp inside the loop via disassembly.

   Tried: Use strings on the binary hoping the encoded or plain-text flag appears as a printable string.

   The binary stores an XOR-permuted encoding of the flag, not the raw flag bytes. strings will surface other literals (error messages, format strings) but the flag bytes are non-ASCII after encoding and will not appear as a recognizable string. Only dynamic inspection at the moment of comparison reveals the expected value.

   Learn more

   Rather than invert the XOR-plus-permutation math, you read the expected byte straight out of the comparison. At the compare instruction the program has already computed the expected encoded byte for the current position and placed it in a register next to yours, so a single register read leaks it. Recover one correct character per position and the flag falls out.

2. Step 2

   Brute-force one character at a time by reading the compared bytes

   Observation

   I noticed that at the comparison breakpoint the expected encoded byte sits in DL alongside my encoded byte in AL, which suggested scripting GDB to try each candidate character and accept whichever one makes AL equal DL at the position corresponding to the current flag length.

   Script GDB: for the current known prefix, try each candidate next character, let the breakpoint at the compare fire for that position (the (len(prefix))th hit), and read the two bytes being compared. When your encoded byte (AL) equals the expected byte (DL) at that position, the candidate is correct. Append it and continue until the closing brace.

   python

   python3 - <<'EOF'
   import subprocess, string
   
   alphabet = string.ascii_letters + string.digits + "{}_"
   flag = "picoCTF{"
   CMP = 0x5655598e          # address of the per-byte compare; read from your disasm
   
   while not flag.endswith("}"):
       pos = len(flag)
       found = None
       for c in alphabet:
           guess = flag + c
           gdb = f"""
   file brute
   break *{hex(CMP)}
   commands
     printf "AL=%d DL=%d\n", $al, $dl
     continue
   end
   run <<< '{guess}'
   quit
   """
           out = subprocess.run(["gdb","-batch","-ex",gdb.strip()],
                                capture_output=True, text=True).stdout.splitlines()
           # inspect the (pos)th compare: accept c when AL == DL there
           cmps = [l for l in out if l.startswith("AL=")]
           if len(cmps) > pos:
               al, dl = (int(x.split("=")[1]) for x in cmps[pos].split())
               if al == dl:
                   found = c; break
       if not found: break
       flag += found
       print("flag so far:", flag)
   print("FLAG:", flag)
   EOF

   Copied!

   What didn't work first

   Tried: Hardcode the breakpoint address 0x5655598e as a fixed value without verifying it in your own build or instance of the binary.

   ASLR is disabled for this 32-bit binary but the compare address is specific to the provided binary on the challenge server. If you downloaded a different build or the binary was re-released, the address will differ. The breakpoint silently fires at the wrong instruction or never fires, so the AL/DL reads produce garbage. Always confirm the cmp address from your own disassembly with disas main or objdump before scripting.

   Tried: Try to count only the (pos)th compare hit by reading the total output line count instead of filtering lines that start with AL=.

   GDB emits extra lines (breakpoint notices, program output) mixed in with the printf output, so a raw line-count index lands on the wrong output line. The filter for lines starting with AL= is necessary to isolate the printf output from the noise. Without it the extracted byte values are wrong and the character match never fires.

   Learn more

   The breakpoint address 0x5655598e and the exact registers used are binary-specific; confirm them from your disassembly (look at the cmp in the check loop and which registers feed it). For the published instance this recovers picoCTF{I_5D3_A11DA7_0db137a9}.

   A coarser alternative is valgrind --tool=callgrind instruction counting: a correct character makes the compare loop run one iteration further, so the candidate with the highest instruction count at each position is correct. The direct AL/DL read is faster and exact when you can pin the compare instruction.

Flag

The binary XORs and permutes your input, then compares it byte-by-byte against the encoded real flag. Break at the per-byte compare and read the two bytes (AL = yours, DL = expected); when they match, the guessed character is correct. Script GDB to recover the flag one position at a time. Addresses are binary-specific.