Easy as GDB picoCTF 2021 Solution

Description

1. Step 1Locate the comparison loop

   Observation

   The binary compares your input to the flag one character at a time. That means there is a single cmp instruction inside a loop where both the encoded input byte and the expected byte are live in registers at once, readable without reversing the encoding at all.

   Load the binary in Ghidra/GDB. It reads your input, runs it through an encoding pipeline (an XOR pass plus a permutation/reorder of the bytes), then compares the encoded input against an encoded copy of the real flag, one byte at a time in a loop. Find the exact compare instruction (the cmp/loop body) and set a breakpoint there; that is where the two bytes being compared are live in registers (commonly AL = your encoded byte, DL = the expected byte).

   bash

   gdb -q ./brute

   Copied!

   bash

   (gdb) info functions

   Copied!

   bash

   (gdb) disas main

   Copied!

   bash

   # find the per-byte cmp instruction inside the check loop

   Copied!

   Expected output

   FLAG: picoCTF{I_5D3_A11DA7_...}

   What didn't work first

   Tried: Set a breakpoint on the strcmp or memcmp function expecting the whole flag to be compared at once.

   The binary never calls strcmp or memcmp; it uses a hand-rolled loop comparing one byte at a time with a cmp instruction. Break on strcmp and nothing fires: the program exits cleanly having hit no breakpoint, so you see no comparison data. Find the actual cmp in the disassembly and break there.

   Tried: Use strings on the binary hoping the encoded or plain-text flag appears as a printable string.

   The binary stores an XOR-permuted encoding of the flag, not the raw bytes. strings surfaces other literals, error messages and format strings, but the encoded flag bytes are non-ASCII and never form a recognizable string. Only inspecting at the moment of comparison reveals the expected value.

   Learn more

   Rather than invert the XOR-plus-permutation math, you read the expected byte straight out of the comparison. At the compare instruction the program has already computed the expected encoded byte for the current position and placed it in a register next to yours, so a single register read leaks it. Recover one correct character per position and the flag falls out.

2. Step 2Brute-force one character at a time by reading the compared bytes

   Observation

   At the comparison breakpoint the expected encoded byte sits in DL and yours sits in AL. So script GDB to try each candidate character and keep whichever makes AL equal DL at the position matching the current flag length.

   Script GDB: for the current known prefix, try each candidate next character, let the breakpoint at the compare fire for that position (the (len(prefix))th hit), and read the two bytes being compared. When your encoded byte (AL) equals the expected byte (DL) at that position, the candidate is correct. Append it and continue until the closing brace.

   python

   python3 - <<'EOF'
   import subprocess, string
   
   alphabet = string.ascii_letters + string.digits + "{}_"
   flag = "picoCTF{"
   CMP = 0x5655598e          # address of the per-byte compare; read from your disasm
   
   while not flag.endswith("}"):
       pos = len(flag)
       found = None
       for c in alphabet:
           guess = flag + c
           # GDB's -ex takes one command per flag, so put the whole script in a
           # file and load it with -x. Feed the guess through a file too, because
           # "run < file" is the redirection form GDB itself supports.
           open("input.txt", "w").write(guess + "\n")
           with open("cmd.gdb", "w") as fh:
               fh.write(f"""break *{hex(CMP)}
   commands
     silent
     printf "AL=%d DL=%d\\n", $al, $dl
     continue
   end
   run < input.txt
   quit
   """)
           out = subprocess.run(["gdb","-batch","-x","cmd.gdb","./brute"],
                                capture_output=True, text=True).stdout.splitlines()
           # inspect the (pos)th compare: accept c when AL == DL there
           cmps = [l for l in out if l.startswith("AL=")]
           if len(cmps) > pos:
               al, dl = (int(x.split("=")[1]) for x in cmps[pos].split())
               if al == dl:
                   found = c; break
       if not found: break
       flag += found
       print("flag so far:", flag)
   print("FLAG:", flag)
   EOF

   Copied!

   What didn't work first

   Tried: Hardcode the breakpoint address 0x5655598e as a fixed value without verifying it in your own build or instance of the binary.

   ASLR is off for this 32-bit binary, but the compare address belongs to the exact build on the challenge server. Download a different build, or catch a re-release, and the address shifts, so the breakpoint fires on the wrong instruction or never fires and the register reads are garbage. Confirm the cmp address from your own disassembly, with disas main or objdump, before scripting anything.

   Tried: Try to count only the (pos)th compare hit by reading the total output line count instead of filtering lines that start with AL=.

   GDB mixes breakpoint notices and program output in with the printf output, so indexing by raw line count lands on the wrong line. Filtering for lines that start with AL= isolates the printf output from the noise. Without that filter the extracted byte values are wrong and the character match never fires.

   Learn more

   The breakpoint address 0x5655598e and the exact registers used are binary-specific; confirm them from your disassembly (look at the cmp in the check loop and which registers feed it). For the published instance this recovers picoCTF{I_5D3_A11DA7_0db137a9}.

   A coarser alternative is valgrind --tool=callgrind instruction counting: a correct character makes the compare loop run one iteration further, so the candidate with the highest instruction count at each position is correct. The direct AL/DL read is faster and exact when you can pin the compare instruction.

Flag

The binary XORs and permutes your input, then compares it byte-by-byte against the encoded real flag. Break at the per-byte compare and read the two bytes (AL = yours, DL = expected); when they match, the guessed character is correct. Script GDB to recover the flag one position at a time. Addresses are binary-specific.