Binary Instrumentation 3 picoCTF 2026 Solution

Description

1. Step 1

   Run the binary and observe the failure

   Observation

   I noticed the challenge description said the binary was 'designed to write the flag' but something went wrong, which suggested running it first to observe the actual failure mode before opening a disassembler.

   Run bin-ins3.exe. It exits without producing output or printing an error related to a missing path. Even if you create the expected output directory, the file written there contains garbage or an error message rather than the flag. This tells you the problem is not a simple WriteFile argument bug. Open the binary in a disassembler (Ghidra or IDA Free) and look at the entry point region to understand the structure.

   bash

   mkdir C:\random

   Copied!

   bash

   .\bin-ins3.exe

   Copied!

   bash

   # File is created but contains an error message, not the flag.

   Copied!

   bash

   # Open in Ghidra or IDA to inspect the structure.

   Copied!

   Expected output

   picoCTF{t11_4r3_4p15_n07h1n9_3l53_...}

   What didn't work first

   Tried: Running strings on bin-ins3.exe and grepping for picoCTF to find the flag statically.

   strings only extracts printable sequences from the on-disk bytes, which are the packer stub, not the payload. The payload is compressed or encrypted inside the stub, so the flag string does not appear in plaintext in the file. You will get no hits because the interesting content is only decompressed into memory at runtime.

   Tried: Creating C:\random\output_flag.txt manually before running the binary, expecting the flag to appear in it.

   The binary does attempt to write to that path, but the value it writes is derived from the in-memory payload after the header-erasure function has already run. Without hooking that function, the payload executes in a corrupted state and writes an error message or garbage instead of the flag. Pre-creating the directory does not fix the root cause.

   Learn more

   PE packers are tools that compress or encrypt a Windows executable (PE file) and wrap it in a stub that decompresses the payload into memory at runtime, then transfers execution to the original entry point. The on-disk binary is the stub; the real program lives in an encrypted or compressed blob inside it. Packers are used to reduce file size, obfuscate code, and make static analysis harder.

   When you open a packed binary in a disassembler, you see only the stub code. The interesting logic - the flag-writing routine in this case - is hidden in the payload that gets decompressed at runtime. This is why static strings searches on the original file turn up nothing useful.

2. Step 2

   Identify the header-erasure function with frida-trace

   Observation

   I noticed that running the binary produced garbage output rather than a flag even with the expected output path present, which suggested the payload was being corrupted in memory before execution and that a header-erasure routine in the packer stub was responsible.

   Use frida-trace to get a dynamic call trace as the binary runs. The function sub_1400015A0 (as named by IDA; Ghidra may label it differently) is called immediately after the payload is expanded into memory. Its job is to overwrite the first bytes of the payload with zeros, destroying the MZ and PE headers so that the unpacked code is harder to dump after the fact. This is a common anti-analysis trick in packers. You need to hook this function and dump the payload before it runs. See the Frida for binary instrumentation post for the tooling basics.

   bash

   pip install frida-tools

   Copied!

   bash

   # Trace all function calls to find the header-erasure point:

   Copied!

   bash

   frida-trace .\bin-ins3.exe -I 'bin-ins3.exe'

   Copied!

   bash

   # Watch for a call that fires just before execution transfers to the payload.

   Copied!

   bash

   # Cross-reference with the disassembly to confirm sub_1400015A0 or equivalent.

   Copied!

   What didn't work first

   Tried: Running frida-trace with -i (lowercase) instead of -I to trace the header-erasure function by name.

   The -i flag filters by function name pattern, not module name. Since the header-erasure function is an unlabeled internal sub-routine (sub_1400015A0 in IDA, not an export), it has no matching symbol name to filter on. Only -I (uppercase, include-module) traces all instrumented functions inside the module, including unnamed internal calls. Using -i with a pattern like 'sub_*' silently matches zero functions.

   Tried: Using x64dbg or OllyDbg to step through and find the erase function instead of frida-trace.

   A traditional debugger can work but requires manually setting a breakpoint at the function and is tedious to correlate with Ghidra offsets because the module base shifts with ASLR on every run. frida-trace gives a timestamped call log automatically without manually computing rebased addresses, making it much faster to identify the exact call that fires just before execution transfers to the payload.

   Learn more

   After a packer decompresses its payload, the payload exists as a valid PE image in memory - it has an MZ header at offset 0, a PE signature, section headers, and runnable code. At this moment it could be dumped to disk and executed directly. To prevent this, some packers call a header-erasure routine that zeroes out the first 0x1000 bytes (or just the MZ/PE signatures) before jumping to the original entry point. Once erased, the in-memory image no longer looks like a valid PE, foiling naive memory-dumping tools.

   frida-trace auto-generates JavaScript handler stubs for every matched function and logs entry/exit with argument values. Using -I <module> (include module) traces all exports and internal calls within the module, giving you a live call graph without writing any instrumentation code yourself.

3. Step 3

   Hook the header-erasure function with Frida to dump the payload

   Observation

   I noticed frida-trace identified sub_1400015A0 as the function called immediately after payload decompression, and its first argument pointed to an MZ header in memory, which suggested hooking it with Frida's Interceptor to read and save the full payload PE before the function could erase the headers.

   Write a Frida Python script that spawns the binary, hooks the header-erasure function, reads the entire payload from memory using the SizeOfImage value from the PE optional header, and writes the bytes to a local file before the header is destroyed. The script receives the dump via Frida's message-passing channel.

   python

   cat > dump_payload.py << 'EOF'
   import frida, sys
   
   ERASE_FUNC_OFFSET = 0x15A0  # offset of sub_1400015A0 from module base
   OUTPUT_FILE = "payload_dumped.exe"
   
   js_hook = """
   const modBase = Module.getBaseAddress("bin-ins3.exe");
   const eraseFunc = modBase.add(${ERASE_FUNC_OFFSET});
   
   Interceptor.attach(eraseFunc, {
       onEnter(args) {
           // args[0] is a pointer to the start of the unpacked PE in memory.
           const peBase = ptr(args[0]);
   
           // Validate MZ signature.
           const mz = peBase.readU16();
           if (mz !== 0x5A4D) { console.log("Not an MZ header, skipping."); return; }
   
           // Walk the PE header to get SizeOfImage.
           const peOffset = peBase.add(0x3C).readU32();
           const peSignature = peBase.add(peOffset).readU32();
           if (peSignature !== 0x00004550) { console.log("Bad PE signature."); return; }
   
           // SizeOfImage is at PE header + 0x18 (OptionalHeader) + 0x38 (SizeOfImage).
           const sizeOfImage = peBase.add(peOffset + 0x18 + 0x38).readU32();
           console.log("PE found. SizeOfImage:", sizeOfImage);
   
           const bytes = peBase.readByteArray(sizeOfImage);
           send("dump", bytes);
           console.log("Payload sent. Detaching.");
           Interceptor.detachAll();
       }
   });
   console.log("Hook installed at offset 0x15A0.");
   """;
   
   def on_message(message, data):
       if message.get("payload") == "dump" and data:
           with open(OUTPUT_FILE, "wb") as f:
               f.write(data)
           print(f"Payload written to {OUTPUT_FILE} ({len(data)} bytes).")
   
   pid = frida.spawn([r".\bin-ins3.exe"])
   session = frida.attach(pid)
   script = session.create_script(js_hook)
   script.on("message", on_message)
   script.load()
   frida.resume(pid)
   sys.stdin.read()
   EOF

   Copied!

   python

   python dump_payload.py

   Copied!

   bash

   # payload_dumped.exe is now a valid PE on disk.

   Copied!

   What didn't work first

   Tried: Using a hardcoded size (e.g. 0x10000 or 0x100000) instead of reading SizeOfImage from the PE optional header.

   A hardcoded size either cuts the dump short (missing code sections, causing crashes if you try to run it) or reads past the allocated region (producing a dump padded with zeros or garbage from adjacent allocations). SizeOfImage is the authoritative value the packer itself used when calling VirtualAlloc, so it is the exact byte count of the valid payload image.

   Tried: Attaching to the already-running process with frida.attach() instead of using frida.spawn() + frida.resume().

   The header-erasure function fires very early in the binary's execution, often within milliseconds of the process starting. Attaching to a running process is too slow - the function will have already fired and the headers already erased before the Interceptor hook is installed. spawn() pauses the process at creation so the hook can be loaded before any code runs.

   Learn more

   The PE optional header stores SizeOfImage at a fixed offset: 0x3C from the start of the file gives the PE header offset; the optional header follows the 20-byte COFF header (PE signature 4 bytes + COFF 20 bytes = offset 0x18 from PE offset); SizeOfImage is at offset 0x38 within the optional header. Reading this value before the headers are erased tells you exactly how many bytes the packer allocated for the payload image.

   Frida's send() function passes arbitrary data from the instrumented process back to the Python controller through a secure pipe. The second argument is a raw byte buffer (an ArrayBuffer from readByteArray()), so the full binary image crosses the process boundary without any encoding overhead.

   The offset 0x15A0 is relative to the module load address. Because Windows uses ASLR, the absolute address changes on every run, but Module.getBaseAddress() always gives the current base, so adding the constant offset lands on the right function regardless of ASLR randomization.

4. Step 4

   Extract the flag from the dumped payload

   Observation

   I noticed that payload_dumped.exe was a valid PE image with intact headers, which suggested running strings on it to surface any embedded text including the base64-encoded flag rather than attempting to re-execute the dump.

   Run strings on the dumped payload or grep for the picoCTF base64 prefix. The payload contains a base64-encoded flag string. Decode it to get the plaintext flag.

   bash

   strings payload_dumped.exe | grep -i 'picoctf\|cGljb0'

   Copied!

   bash

   # You will see: cGljb0NURnt0MTFfNHIzXzRwMTVfbjA3aDFuOV8zbDUzXzc5MjcyZjVifQo=

   Copied!

   bash

   echo 'cGljb0NURnt0MTFfNHIzXzRwMTVfbjA3aDFuOV8zbDUzXzc5MjcyZjVifQo=' | base64 -d

   Copied!

   What didn't work first

   Tried: Grepping the dump for 'picoCTF{' directly without the base64 prefix alternative.

   The flag is stored base64-encoded, so the literal string 'picoCTF{' does not appear in the payload binary. The grep will return no results. You need to also search for 'cGljb0' (the base64 encoding of 'picoC') to catch the encoded form, as shown in the command above.

   Tried: Running the dumped payload_dumped.exe directly and reading the output file it produces.

   The dumped PE image has its MZ/PE headers present (that is why we dumped it before erasure), but the section alignment and image base assumptions in the dump may not match where Windows loads it, causing an import resolution failure or crash on startup. Reading the embedded base64 string with strings is faster and more reliable than trying to re-execute the dump.

   Learn more

   The packed binary tries to write the flag to C:\random\output_flag.txt at runtime, but the flag it would write is itself base64-encoded rather than plaintext. The real flag is baked into the payload binary as a string literal. Dumping the payload and running strings on it is a faster path than trying to reconstruct the file-write flow.

   Why base64? Storing a flag or any sensitive string as a base64 literal is a mild obfuscation technique: it avoids the literal picoCTF{ prefix showing up in a raw binary search, but it is trivially reversible once you know to look for it. The strings utility extracts any sequence of printable ASCII characters above a minimum length (default 4), making it ideal for quickly surveying what text is embedded in a binary.

Flag

The binary is a custom PE packer. It decompresses a payload PE into memory, then calls a header-erasure function (sub_1400015A0) before executing the payload. Hooking that function with Frida captures the payload before its headers are wiped. The payload contains a base64-encoded flag string that decodes to the flag.