WinAntiDbg0x300 picoCTF 2024 Solution

Description

1. Step 1

   Unpack with UPX

   Observation

   I noticed that running strings on the raw executable showed UPX! markers and section names like UPX0 and UPX1, which indicated the binary was compressed with UPX and needed to be unpacked before Ghidra could analyze the real code.

   The binary is packed with UPX. Running strings on the packed file shows UPX! markers and section names like UPX0 and UPX1. Run upx -d WinAntiDbg0x300.exe to unpack it in place, giving you the real code for Ghidra to analyze. Load the PDB file supplied with the challenge during Ghidra import to get function names and variable names for free.

   What didn't work first

   Tried: Loading the packed binary directly into Ghidra and trying to analyze the decompiled output.

   Ghidra sees the UPX decompression stub, not the real code. The decompiled output looks like a tiny program that copies memory around and jumps to a computed address - none of the real function names or logic are visible. You must unpack with upx -d first so Ghidra sees the actual executable code.

   Tried: Skipping the PDB import and trying to rename functions manually from context clues.

   Without the PDB, Ghidra labels everything FUN_00401234 and DAT_00403000. The anti-debug logic and flag decryption routine become very hard to find among dozens of unlabeled functions. The challenge supplies the PDB file specifically so you can import it and get the original symbol names, which makes the correct code path obvious.

   Learn more

   UPX (Ultimate Packer for eXecutables) compresses a binary and prepends a decompression stub. At runtime the stub decompresses the original code into memory and jumps to it. Malware authors use UPX (or custom packers) to shrink payloads and hinder static analysis. The telltale UPX! magic bytes visible via strings make identification trivial. upx -d reverses the packing in place.

   Importing the PDB (Program Database) file during Ghidra analysis provides all the original symbol names: function names, variable names, and type information. This makes reverse engineering dramatically easier. To load a PDB in Ghidra: import the executable, then before analyzing, go to File > Add to Program and browse to the PDB file. Ghidra will warn about an inexact match; accept it.

2. Step 2

   Understand why you cannot use a debugger

   Observation

   I noticed the challenge description explicitly warned that the binary would fight the debugger and that an infinite loop constantly checks for one, which suggested I needed to understand the anti-debug mechanism before attempting to bypass it, and that DebugView was the safe observation method.

   In Ghidra, examine WinMain and the challenge_create_thread function. The manage_child_process function creates a mutex; if the mutex already exists (meaning a child has been spawned), the child attempts to attach a debugger to the parent and terminate it. The main loop runs forever and the decrypt_flag code is unreachable because an unconditional JMP bypasses it. DebugView captures OutputDebugString calls, letting you see any debug output the program emits without attaching a debugger.

   What didn't work first

   Tried: Attaching x64dbg or OllyDbg to the unpacked binary and trying to step through the anti-debug checks.

   The binary spawns a child process that detects the debugger and terminates the parent within seconds. Every attempt to pause or step causes the process to die before you reach anything useful. The solution is to not use a debugger at all - DebugView captures the output passively without triggering the child process termination logic.

   Tried: Running the binary in a terminal and looking for flag output in stdout or a console window.

   The flag is printed via OutputDebugString, which writes to the Windows debug message stream and does not appear in any console window or file. Standard stdout redirection with > output.txt captures nothing. Only DebugView (or a proper kernel-level debugger, which the binary fights) can intercept these messages.

   Learn more

   DebugView (Dbgview.exe from Microsoft Sysinternals) is a passive monitor for OutputDebugString output. It captures debug messages printed by any process on the system without attaching a debugger, so the anti-debug child-process logic never fires. Run DebugView (32-bit mode for 32-bit binaries) and then run the program normally as administrator.

   The every-5-second output in DebugView ("no debugger was present, exiting successfully") confirms the program is running and its debug output is reachable, but the flag is never printed because the loop never exits to the decrypt_flag call. The solution is to patch the binary so the unconditional JMP is replaced with NOPs, letting execution fall through to the decrypt_flag path.

3. Step 3

   Patch the binary in Ghidra

   Observation

   I noticed in Ghidra's assembly view that the challenge_create_thread function contained an unconditional JMP that looped forever, preventing the decrypt_flag call from ever being reached, which suggested replacing that JMP with NOP instructions to let execution fall through to the flag decryption path.

   In Ghidra's assembly view, locate the unconditional JMP at the bottom of the loop in challenge_create_thread. Select those bytes, go to Patch Instruction (or use the byte-level editor), and replace them with NOP instructions. Ghidra's keyboard shortcut for patching to NOPs is Ctrl+Shift+G. Then export the patched binary: File > Export Program > Windows PE. Run the patched binary as administrator with DebugView open and wait up to 5 seconds for the flag to appear.

   What didn't work first

   Tried: Patching the IsDebuggerPresent check or the child-process mutex logic instead of the unconditional JMP in the main loop.

   Neutralizing the anti-debug checks does not help because the loop never exits regardless of whether a debugger is detected - the JMP is unconditional, not conditional. You must patch the JMP that keeps the loop running, not the IsDebuggerPresent comparisons above it.

   Tried: Exporting the patched binary with File > Export Program using the default 'Original File' format instead of 'Windows PE'.

   The 'Original File' export writes back only the bytes that were already in the original file structure and may not include Ghidra's byte-level patches. Choosing 'Windows PE' as the format tells Ghidra to reconstruct a valid PE with all patched bytes applied, producing an executable that actually reflects your NOPs.

   Learn more

   A NOP (No Operation, opcode 90) is an instruction that does nothing. Replacing a JMP with five NOPs (a JMP instruction is typically 2 to 5 bytes; fill with enough NOPs to cover the original instruction size) causes execution to continue to the next instruction rather than jumping back to the loop head.

   Ghidra's Export Program function writes the patched binary with your in-place edits preserved. The output is a valid PE executable that you can run on Windows. This workflow of: unpack, analyze, patch in Ghidra, export, run with DebugView, is the canonical approach to binaries that actively fight debuggers.

   The challenge_create_thread function also calls compute_hashes functions that modify a global variable. All of those must complete before the flag is decrypted, which is why the flag appears after a delay (about 5 seconds) rather than immediately after the patched JMP is bypassed.

4. Step 4

   Read the flag from DebugView

   Observation

   I noticed that the binary uses OutputDebugString to print the flag rather than stdout, and that DebugView was already set up to capture that output stream, so running the patched binary with DebugView open was the final step to retrieve the flag.

   After running the patched binary as administrator with DebugView open, wait up to 5 seconds. The program prints 'you got the flag' and then the flag string to the DebugView output pane. Copy picoCTF{...} from there.

   Learn more

   The flag is printed via OutputDebugString, which writes to the Windows debug message stream rather than stdout. Only DebugView (or a proper debugger, which the binary fights) can capture this output. That is why the challenge hints specifically mention DebugView.

   Running as administrator is required because the binary checks for admin privileges as one of its conditions. The patched binary still runs through those checks but now reaches the decrypt_flag path regardless of the debugger presence, so the flag is printed after all the hash computations complete.

Flag

Patching the infinite-loop JMP to NOPs and running the patched binary with DebugView reveals the flag after a short delay.