Binary Instrumentation 3 picoCTF 2026 Solution

Description

1. Step 1Run the binary and observe the failure

   Observation

   The description says the binary was meant to write the flag and something went wrong. Run it first and see how it fails before opening a disassembler.

   Run bin-ins3.exe. It exits without producing output or printing an error related to a missing path. Even if you create the expected output directory, the file written there contains garbage or an error message rather than the flag. This tells you the problem is not a simple WriteFile argument bug. Open the binary in a disassembler (Ghidra or IDA Free) and look at the entry point region to understand the structure.

   bash

   mkdir C:\random

   Copied!

   bash

   .\bin-ins3.exe

   Copied!

   bash

   # File is created but contains an error message, not the flag.

   Copied!

   bash

   # Open in Ghidra or IDA to inspect the structure.

   Copied!

   What didn't work first

   Tried: Running strings on bin-ins3.exe and grepping for picoCTF to find the flag statically.

   strings reads the bytes on disk, and those are the packer stub. The payload is compressed inside it, so no flag string exists in the file at all. The interesting content only appears in memory at runtime.

   Tried: Creating C:\random\output_flag.txt manually before running the binary, expecting the flag to appear in it.

   The binary does try to write there, but what it writes comes from the payload after the header-erasure function has already run. Uncorrected, the payload executes in a broken state and emits an error or garbage. Creating the directory changes nothing.

   Learn more

   PE packers are tools that compress or encrypt a Windows executable (PE file) and wrap it in a stub that decompresses the payload into memory at runtime, then transfers execution to the original entry point. The on-disk binary is the stub; the real program lives in an encrypted or compressed blob inside it. Packers are used to reduce file size, obfuscate code, and make static analysis harder.

   When you open a packed binary in a disassembler, you see only the stub code. The interesting logic - the flag-writing routine in this case - is hidden in the payload that gets decompressed at runtime. This is why static strings searches on the original file turn up nothing useful.

2. Step 2Identify the header-erasure function with frida-trace

   Observation

   It writes garbage rather than a flag even with the output path in place, so the payload is corrupted in memory before it runs. A header-erasure routine in the packer stub is doing it.

   Use frida-trace to get a dynamic call trace as the binary runs. The function sub_1400015A0 (as named by IDA; Ghidra may label it differently) is called immediately after the payload is expanded into memory. Its job is to overwrite the first bytes of the payload with zeros, destroying the MZ and PE headers so that the unpacked code is harder to dump after the fact. This is a common anti-analysis trick in packers. You need to hook this function and dump the payload before it runs. See the Frida for binary instrumentation post for the tooling basics.

   bash

   pip install frida-tools

   Copied!

   bash

   # Trace all function calls to find the header-erasure point:

   Copied!

   bash

   frida-trace .\bin-ins3.exe -I 'bin-ins3.exe'

   Copied!

   bash

   # Watch for a call that fires just before execution transfers to the payload.

   Copied!

   bash

   # Cross-reference with the disassembly to confirm sub_1400015A0 or equivalent.

   Copied!

   What didn't work first

   Tried: Running frida-trace with -i (lowercase) instead of -I to trace the header-erasure function by name.

   Lowercase -i filters by function name, not module. The header-erasure routine is an unlabeled internal function with no export symbol, so there is no name to match and a pattern like sub_* silently matches nothing. Uppercase -I traces every instrumented function in the module, unnamed calls included.

   Tried: Using x64dbg or OllyDbg to step through and find the erase function instead of frida-trace.

   A debugger works, but you set the breakpoint by hand and rebase every Ghidra offset yourself, because ASLR moves the module base each run. frida-trace gives a timestamped call log without any of that, which makes finding the call just before the handoff much quicker.

   Learn more

   After a packer decompresses its payload, the payload exists as a valid PE image in memory - it has an MZ header at offset 0, a PE signature, section headers, and runnable code. At this moment it could be dumped to disk and executed directly. To prevent this, some packers call a header-erasure routine that zeroes out the first 0x1000 bytes (or just the MZ/PE signatures) before jumping to the original entry point. Once erased, the in-memory image no longer looks like a valid PE, foiling naive memory-dumping tools.

   frida-trace auto-generates JavaScript handler stubs for every matched function and logs entry/exit with argument values. Using -I <module> (include module) traces all exports and internal calls within the module, giving you a live call graph without writing any instrumentation code yourself.

3. Step 3Hook the header-erasure function with Frida to dump the payload

   Observation

   frida-trace shows sub_1400015A0 called right after decompression, with its first argument pointing at an MZ header in memory. Hook it and save the whole payload PE before the function erases those headers.

   Write a Frida Python script that spawns the binary, hooks the header-erasure function, reads the entire payload from memory using the SizeOfImage value from the PE optional header, and writes the bytes to a local file before the header is destroyed. The script receives the dump via Frida's message-passing channel.

   python

   cat > dump_payload.py << 'EOF'
   import frida, sys
   
   ERASE_FUNC_OFFSET = 0x15A0  # offset of sub_1400015A0 from module base
   OUTPUT_FILE = "payload_dumped.exe"
   
   js_template = """
   const modBase = Module.getBaseAddress("bin-ins3.exe");
   const eraseFunc = modBase.add(__ERASE_OFFSET__);
   
   Interceptor.attach(eraseFunc, {
       onEnter(args) {
           // args[0] is a pointer to the start of the unpacked PE in memory.
           const peBase = ptr(args[0]);
   
           // Validate MZ signature.
           const mz = peBase.readU16();
           if (mz !== 0x5A4D) { console.log("Not an MZ header, skipping."); return; }
   
           // Walk the PE header to get SizeOfImage.
           const peOffset = peBase.add(0x3C).readU32();
           const peSignature = peBase.add(peOffset).readU32();
           if (peSignature !== 0x00004550) { console.log("Bad PE signature."); return; }
   
           // SizeOfImage is at PE header + 0x18 (OptionalHeader) + 0x38 (SizeOfImage).
           const sizeOfImage = peBase.add(peOffset + 0x18 + 0x38).readU32();
           console.log("PE found. SizeOfImage:", sizeOfImage);
   
           const bytes = peBase.readByteArray(sizeOfImage);
           send("dump", bytes);
           console.log("Payload sent. Detaching.");
           Interceptor.detachAll();
       }
   });
   console.log("Hook installed at offset 0x15A0.");
   """
   
   js_hook = js_template.replace("__ERASE_OFFSET__", hex(ERASE_FUNC_OFFSET))
   
   def on_message(message, data):
       if message.get("payload") == "dump" and data:
           with open(OUTPUT_FILE, "wb") as f:
               f.write(data)
           print(f"Payload written to {OUTPUT_FILE} ({len(data)} bytes).")
   
   pid = frida.spawn([r".\bin-ins3.exe"])
   session = frida.attach(pid)
   script = session.create_script(js_hook)
   script.on("message", on_message)
   script.load()
   frida.resume(pid)
   sys.stdin.read()
   EOF

   Copied!

   python

   python dump_payload.py

   Copied!

   bash

   # payload_dumped.exe is now a valid PE on disk.

   Copied!

   What didn't work first

   Tried: Using a hardcoded size (e.g. 0x10000 or 0x100000) instead of reading SizeOfImage from the PE optional header.

   A hardcoded size either truncates the dump, losing code sections, or reads past the allocation and pads the file with zeros and neighboring garbage. SizeOfImage is the number the packer itself passed to VirtualAlloc, so it is exactly the valid image.

   Tried: Attaching to the already-running process with frida.attach() instead of using frida.spawn() + frida.resume().

   The erasure fires within milliseconds of the process starting, so attaching to a running process is too late: the headers are gone before the hook installs. Spawning under Frida pauses at creation, and the hook loads before any code runs.

   Learn more

   The PE optional header stores SizeOfImage at a fixed offset: 0x3C from the start of the file gives the PE header offset; the optional header follows the 20-byte COFF header (PE signature 4 bytes + COFF 20 bytes = offset 0x18 from PE offset); SizeOfImage is at offset 0x38 within the optional header. Reading this value before the headers are erased tells you exactly how many bytes the packer allocated for the payload image.

   Frida's send() function passes arbitrary data from the instrumented process back to the Python controller through a secure pipe. The second argument is a raw byte buffer (an ArrayBuffer from readByteArray()), so the full binary image crosses the process boundary without any encoding overhead.

   The offset 0x15A0 is relative to the module load address. Because Windows uses ASLR, the absolute address changes on every run, but Module.getBaseAddress() always gives the current base, so adding the constant offset lands on the right function regardless of ASLR randomization.

4. Step 4Extract the flag from the dumped payload

   Observation

   The dump is a valid PE with intact headers. Run strings over it for the embedded base64 rather than trying to execute it.

   Run strings on the dumped payload or grep for the picoCTF base64 prefix. The payload contains a base64-encoded flag string. Decode it to get the plaintext flag.

   bash

   strings payload_dumped.exe | grep -i 'picoctf\|cGljb0'

   Copied!

   bash

   # You will see: cGljb0NURnt0MTFfNHIzXzRwMTVfbjA3aDFuOV8zbDUzXy4uLn0K

   Copied!

   bash

   echo 'cGljb0NURnt0MTFfNHIzXzRwMTVfbjA3aDFuOV8zbDUzXy4uLn0K' | base64 -d

   Copied!

   Expected output

   picoCTF{411_4r3_4p15_n07h1n9_3l53_...}

   What didn't work first

   Tried: Grepping the dump for 'picoCTF{' directly without the base64 prefix alternative.

   The flag is base64-encoded, so the literal prefix never appears in the payload and the grep returns nothing. Search for cGljb0 as well, which is that prefix encoded.

   Tried: Running the dumped payload_dumped.exe directly and reading the output file it produces.

   The dump does have its headers, which is why we took it before erasure, but the section alignment and image base may not match where Windows loads it, so imports fail to resolve or it crashes on startup. Reading the embedded base64 with strings is faster and surer.

   Learn more

   The packed binary tries to write the flag to C:\random\output_flag.txt at runtime, but the flag it would write is itself base64-encoded rather than plaintext. The real flag is baked into the payload binary as a string literal. Dumping the payload and running strings on it is a faster path than trying to reconstruct the file-write flow.

   Why base64? Storing a flag or any sensitive string as a base64 literal is a mild obfuscation technique: it avoids the literal picoCTF{ prefix showing up in a raw binary search, but it is trivially reversible once you know to look for it. The strings utility extracts any sequence of printable ASCII characters above a minimum length (default 4), making it ideal for quickly surveying what text is embedded in a binary.

Flag

The binary is a custom PE packer. It decompresses a payload PE into memory, then calls a header-erasure function (sub_1400015A0) before executing the payload. Hooking that function with Frida captures the payload before its headers are wiped. The payload contains a base64-encoded flag string that decodes to the flag. The flag is shown abbreviated on this page; work the steps above to recover the full value.