Binary Instrumentation 3 picoCTF 2026 Solution

Description

1. Step 1Run the binary and trace WriteFile to find the bug
   Run the binary, see no output, then trace WriteFile with frida-trace. The auto-generated handler logs each call's args; you'll see arg 3 (nNumberOfBytesToWrite) is 0, which is why nothing prints. See Frida for binary instrumentation for the broader workflow.
   bash

   unzip -P picoctf bin-ins3.zip

   Copied!

   bash

   ./bin-ins3

   Copied!

   bash

   # No output. Trace WriteFile to see why:

   Copied!

   bash

   frida-trace ./bin-ins3 -i 'WriteFile'

   Copied!

   bash

   # In the generated handler, log args[2]; you will see it is 0.

   Copied!
   Learn more

   WriteFile is a Windows API function from kernel32.dll that writes data to a file handle. Its third parameter, nNumberOfBytesToWrite, controls how many bytes from the buffer are actually written. Setting it to zero means the call succeeds but writes nothing - the flag buffer exists in memory but never reaches output.

   This kind of intentional sabotage (or accidental bug) is a realistic scenario: a developer might zero-initialize a length variable, forget to set it, or a compiler optimization might constant-fold it to zero. The x64 Windows calling convention passes the first four integer/pointer arguments in rcx, rdx, r8, r9, with additional args on the stack. So WriteFile(hFile, lpBuffer, nNumberOfBytesToWrite, lpNumberOfBytesWritten, lpOverlapped) maps to rcx, rdx, r8, r9, [rsp+0x20]. Frida's args array indexes this consistently across architectures: args[2] is always the third argument regardless of how the ABI passes it.

   Binary instrumentation is the practice of modifying or observing a program's behavior at runtime without altering the binary on disk. Dynamic instrumentation tools like Frida, Pin (Intel), and DynamoRIO intercept function calls, inspect and modify arguments, read/write memory, and replace implementations - all while the process runs normally.

2. Step 2Hook WriteFile() with Frida to fix the byte count
   Hook WriteFile and override args[2] (nNumberOfBytesToWrite) with a real length. Read the buffer as a fixed byte window since Windows strings may not be null-terminated and the flag may contain non-UTF-8 bytes.
   bash

   pip install frida-tools

   Copied!

   js

   cat > fix.js << 'EOF'
   // Frida script: hook WriteFile and fix nNumberOfBytesToWrite
   const WriteFile = Module.getExportByName("kernel32.dll", "WriteFile");
   
   Interceptor.attach(WriteFile, {
       onEnter(args) {
           // args[1] = lpBuffer, args[2] = nNumberOfBytesToWrite (currently 0)
           const bufPtr = args[1];
           // Read a fixed window. Windows strings are not always null-terminated,
           // and the flag may contain non-UTF-8 bytes, so readByteArray() is safer
           // than readUtf8String().
           const N = 256;
           const bytes = new Uint8Array(bufPtr.readByteArray(N));
           // Trim at the first null (best heuristic for C strings).
           let end = bytes.indexOf(0);
           if (end < 0) end = N;
           if (end > 0) {
               args[2] = ptr(end);
               console.log("Fixed WriteFile byte count to:", end);
               // Print as latin-1 so all 256 byte values render losslessly.
               console.log("Content:", String.fromCharCode.apply(null, bytes.slice(0, end)));
           }
       }
   });
   EOF

   Copied!

   bash

   frida -l fix.js ./bin-ins3

   Copied!
   Learn more

   Frida is a dynamic instrumentation toolkit that injects a JavaScript engine (V8) into a target process. Once injected, your JavaScript code runs inside the target process with full access to its memory and can intercept any function call using Interceptor.attach(). The onEnter callback fires before the hooked function executes, letting you read and modify the arguments via the args array.

   Module.getExportByName() resolves a named export from a loaded DLL or shared library to its memory address. This works because Windows DLLs (and Linux .so files) maintain an export table - a mapping from function names to addresses - that Frida can read from the in-process module list.

   The pattern of modifying function arguments in onEnter is called argument patching and is one of the most common Frida use cases. Real-world applications include bypassing license checks (patching a return value from a validation function), SSL pinning bypass (replacing certificate comparison results), and API fuzzing (injecting malformed arguments to find crashes).

3. Step 3Read the flag from the output
   With the byte count fixed, WriteFile outputs the flag to stdout or the target file. Read it from the Frida console output.
   Learn more

   After Frida patches the nNumberOfBytesToWrite argument, the WriteFile call proceeds normally and the OS kernel writes the full flag buffer to the file handle. If the handle is stdout (handle value 1 on Windows, or the handle returned by GetStdHandle(STD_OUTPUT_HANDLE)), the flag appears in the console.

   An alternative approach is to use onLeave instead of onEnter: the onLeave callback fires after the function returns, giving access to the return value. For debugging, you can log the buffer content in onEnter without modifying the arguments, which is a purely passive instrumentation useful for understanding what data a function receives.

   This challenge illustrates a broader security concept: runtime integrity. Production software often uses anti-tamper measures (code signing, integrity checks, anti-debug tricks) to prevent exactly this kind of instrumentation. Learning Frida teaches you both how to bypass such measures and how to implement them defensively.

Flag

The binary calls WriteFile() with nNumberOfBytesToWrite = 0, preventing any output. A Frida script intercepting WriteFile() and overriding the byte count argument with the real flag length causes the flag to be written.