Frida and Binary Instrumentation for CTFs: A Beginner's Path

Strings already has the flag, and that's the lesson

Open picoCTF 2025 Binary Instrumentation 1. The challenge title says you need a runtime hooking framework. The README hints at something called Frida (a tool that injects JavaScript into a running process and lets you intercept function calls in flight). You download the Windows executable, you remember the first thing every reverse engineer learns, and you type:

strings bininst1.exe | grep -i pico

stringsis the Unix utility that scans a file for runs of printable ASCII characters and prints them. It's pure static analysis, fast, dumb, and lossless on anything a developer left as a literal in the binary.

The flag scrolls past, base64-encoded. You decode it. You move on. Same trick on Binary Instrumentation 2. The challenge author's own note says "no instrumentation is required." You scratch your head, mark both solved, and never install Frida.

That used to be me. I followed an Android Frida tutorial for two hours, found out none of it applied to a Windows PE, gave up, and came back six months later to a Linux binary that took five minutes to hook. Most of the friction in this tool sits in the wrong tutorials, not the tool.

The strings shortcut on bininst1 and bininst2 is the wrong way to use the challenge. The picoCTF curriculum ladders four binary-instrumentation problems in a deliberate order, and the unintended strings shortcut on the first two is the cheap onramp, not the lesson. The lesson is the next two. bin-ins3 asks the binary to write the flag and pins nNumberOfBytesToWrite to zero, so nothing reaches disk. bin-ins4 ships the flag through WS2_32!send to a server that no longer exists, so nothing reaches the wire either. strings stops working. Frida is the only reasonable path.

The numbers in the titles are misleading. A reader skimming the catalog sees 1, 2, 3, 4 and assumes the same technique gets harder. The picoCTF dashboard even labels the first two as medium and the second two as hard, which is true on points and false on technique. The first two are unintended-strings warmups; the second two are the actual exercises. Skipping the install on the warmups is what makes the second two feel impossible.

The challenge title is a curriculum hint, not a difficulty rating. Skipping the intended path on the easy onramp means meeting Frida next time under harder conditions, without the basics.

This guide is the path the curriculum points at. Install Frida, hook a Win32 export with five lines of JavaScript, fix a broken argument at runtime, capture a payload before it leaves the process, and then cross the bridge that almost no tutorial covers: hooking a stripped Linux ELF where address space layout randomization (ASLR) moves the function's address every run.

Frida by doing: install, attach, hook a Win32 call

Fridais a dynamic binary instrumentation (DBI) toolkit by Ole André V. Ravnås. The short version: it injects a small JavaScript engine into your target process, and from inside that engine you can read the target's memory, replace its functions, and intercept any call by writing a callback. Static analysis (what strings and Ghidra do) reads the bytes at rest. Dynamic instrumentation reads what the bytes do when the program runs. The two answer different questions, and a CTF challenge usually tells you which.

Install on Linux or macOS:

pip install frida-tools
# Linux only, allow attaching to non-child processes
sudo sysctl kernel.yama.ptrace_scope=0

You get two commands worth knowing: frida for spawning a target and loading a script, and frida-tracefor auto-generating handler scaffolds when you don't want to write the JavaScript yourself. Skip frida-serverfor now; that's the daemon for hooking Android phones.

Now solve bininst1 the intended way. The binary calls kernel32!Sleep with a value large enough that you'd outlive the program. Replace Sleep with a function that returns immediately:

// hook.js
const sleep = Module.findExportByName("kernel32.dll", "Sleep");
Interceptor.replace(
  sleep,
  new NativeCallback(function (ms) { return; }, "void", ["uint32"])
);

frida -f bininst1.exe -l hook.js

Three lines of script and the flag pops in seconds. Module.findExportByNameresolves the function by its export name from the DLL's export table, the same table that lets Windows wire up imports at load time. Interceptor.replaceswaps the function's entry point for a native callback you provide; that callback returns immediately so Sleep becomes a no-op. NativeCallbacktells Frida the function's signature so the calling convention lines up.

For bininst2, the binary tries to write the flag through CreateFileA and WriteFile, but its filename argument is the literal placeholder string "<Insert path here>". Easier than writing JavaScript: let frida-trace generate the handlers for you and edit them in place.

frida-trace -i CreateFileA -i WriteFile -f bininst2.exe -X KERNEL32
# Edit __handlers__/kernel32.dll/CreateFileA.js to substitute a real filename
# Edit __handlers__/kernel32.dll/WriteFile.js to hexdump args[1]

The -X KERNEL32 flag scopes the search to kernel32.dll so frida-trace doesn't generate handlers for every loaded DLL that exposes a function whose name happens to match. Inside CreateFileA.js, replace args[0] with Memory.allocUtf8String('flag.txt')so the call succeeds. The new file path doesn't need to exist beforehand; CreateFileA with the default flags creates one if it's missing. Inside WriteFile.js, hexdump args[1] for the buffer and read args[2].toInt32() for the length, and the encoded flag spills into the trace log.

Both intended solutions are five lines of JavaScript. Skipping the install because strings works is the false economy: it leaves you unprepared for the next two challenges, which are not solvable that way.

Where strings stops working: bin-ins3 and bin-ins4

Now to the part of the curriculum that earns its title. Both 2026 challenges embed the flag in memory, but neither lets it land anywhere strings can scrape.

bin-ins3: fix a broken argument at runtime

The binary calls WriteFile with a valid file handle, a valid buffer, and nNumberOfBytesToWritehardwired to zero. The call succeeds, the OS writes nothing, the program exits silently. The flag exists in process memory for the duration of one function call and then it's gone.

The Frida move is to intercept WriteFile on entry, read the buffer to figure out its real length, and patch args[2] before the kernel sees the call:

// fix.js
const writeFile = Module.getExportByName("kernel32.dll", "WriteFile");
Interceptor.attach(writeFile, {
  onEnter(args) {
    const content = args[1].readUtf8String();
    if (content && content.length > 0) {
      args[2] = ptr(content.length);
      console.log("Patched WriteFile len =", content.length);
      console.log("Buffer:", content);
    }
  }
});

frida -f bin-ins3 -l fix.js

args[1].readUtf8String() dereferences the buffer pointer and reads a null-terminated string. args[2] = ptr(content.length) writes the new length back into the register Frida abstracts as args[2] (r8 on Windows x64). The kernel reads the patched value, writes the real number of bytes, and the flag prints. strings would have worked here too, since the buffer is constructed in .rdata, but that misses the technique. bin-ins4doesn't leave you that escape hatch.

bin-ins4: read the payload before it leaves the process

The binary opens a Winsock connection and calls WS2_32!send with the flag in args[1]. Two things make this hostile to anything but instrumentation. The flag is constructed at runtime, so it never lives in .rdata for strings to find. The destination server is offline, so tcpdump sees a SYN, a RST, and nothing else. The bytes exist for one trip through the kernel and then they evaporate.

Hook send at the application layer:

// intercept.js
const send = Module.getExportByName("WS2_32.dll", "send");
Interceptor.attach(send, {
  onEnter(args) {
    const len = args[2].toInt32();
    if (len > 0) {
      const data = args[1].readByteArray(len);
      const str = new TextDecoder().decode(new Uint8Array(data));
      console.log("send():", str);
    }
  }
});

frida -f bin-ins4 -l intercept.js

The hook fires before the buffer hits the socket layer, so even if the program later wraps the connection in TLS, the data you read here is plaintext. This is the same in-process trick Objectionuses to defeat SSL pinning on Android and iOS, applied to a stripped-down picoCTF challenge. The remote host being unreachable is irrelevant. Frida sees the buffer in memory, and that's where the flag is.

Stack all four challenges side by side. Each row names the bug, says whether stringsstill works on it, and gives the Frida move that actually solves it. Read top to bottom and the curriculum's shape is obvious:

The bridge nobody teaches: hooking a stripped Linux ELF

The picoCTF binary-instrumentation track is Windows-only and uses named DLL exports, which is the easy mode of Frida hooking. Real CTF reverse-engineering challenges (and most malware analysis work) drop you onto a stripped Linux ELF where the function you want has no export name, no debug symbol, and a base address that moves every run. This is the section every Frida tutorial I've read either skips or fakes.

Three concepts run together here, so a quick scaffold first. ELF (Executable and Linkable Format) is Linux's native binary format. PIE (Position-Independent Executable) means the binary can be loaded at any address. ASLR (Address Space Layout Randomization) is the kernel mitigation that picks a different base each run. A stripped binary is one with its symbol table removed; nm and objdump --syms return nothing, and Module.findExportByName(null, "secret_check") returns null. For the full primer, see the ASLR/PIE Bypass guide.

The pattern is the same one Frida documents for shared libraries, applied to the main executable. Find the function's offset in your disassembler, look up the binary's runtime base, and add them:

// hook.js  --  for stripped PIE ./challenge with secret_check at offset 0x12a0
const m = Process.getModuleByName("challenge");
const target = m.base.add(0x12a0);
console.log(`base=${m.base}  target=${target}`);
 
Interceptor.attach(target, {
  onEnter(args) {
    console.log("secret_check called with", args[0], args[1].readCString());
  },
  onLeave(retval) {
    retval.replace(1);
    console.log("forced retval to 1");
  }
});

frida -l hook.js -f ./challenge

Three things tend to trip people up here, and they're all in line two.

First, Process.getModuleByName("challenge") takes the basename of the executable, not null and not the absolute path. Module.findBaseAddress(null) returns null on Linux; this is a documented quirk. Process.enumerateModules()[0].nameis the source of truth if you're unsure.

Second, the offset has to come from the right reference frame. Ghidra by default loads PIE binaries at image base 0x100000, so when you see a function at 0x001012a0, the file offset is 0x12a0. objdump -d ./challenge | grep secret_check confirms it. Hardcoding 0x555555554000 (which a few CTF writeups copy from the Volga CTF f-hash writeup) is a trap: that address is GDB's default load base, not Frida's, and the kernel chooses a different one for every Frida-spawned run.

Third, retval.replace(1) rewrites the return register (raxon x86-64) before control returns to the caller. This is the one-line bypass for any "is this a valid license / password / debugger / flag" check that returns a boolean. You don't need to know what the function does. You only need to know what it returns.

The same pattern handles the trickiest case in CTF reversing: a stripped challenge binary that runtime-decrypts strings, calls strcmp against a derived key, and only then prints the flag. Static analysis sees a wall of xoroperations. Frida lets you hook the constructed function and read both arguments after the decryption is done. You don't reverse the algorithm. You watch the answer fall out.

What Frida doesn't fix

Frida solves a class of problems. Three categories sit outside it, and a CTF beginner who reaches for Frida by reflex on every binary will lose time.

Anti-Frida detection.CTF binaries almost never ship this, but production mobile apps sometimes do, and you'll meet it the moment you graduate from picoCTF to a real bug-bounty target. The checks look for artifacts a Frida-injected process leaves behind: strings like frida-agent in /proc/self/maps, gum runtime threads named gum-js-loop and gmain, listening port 27042, and a stale TracerPidfrom the ptrace handshake. None are hard to bypass and most have a one-line fix, but they add up. The good news: it's rarer than the marketing makes it sound. Promon's 2024 hooking-framework survey tested 144 of the most-popular Android apps and found only three responded appropriately to Frida's presence. Banking, payments, and DRM apps are the loud exceptions; everything else is wide open.

Hot-loop performance.Frida's JavaScript bridge is excellent for tens or hundreds of hooks. It is not excellent for the inside of a tight loop firing tens of thousands of times per second. The docs note that "sending a single message is not optimized for high frequencies" and recommend dropping into CModule (native callbacks compiled at load time) or Interceptor.replaceFastfor the hot path. If you're benchmarking and Frida is the bottleneck, the answer is C, not more JavaScript.

Wrong-tool problems. Frida is for runtime observation and surgical patching. When you actually need:

• Breakpoints, heap inspection, return-oriented programming (ROP) help on a local binary you have on disk, reach for Pwndbg (see the GDB guide).
• Instruction-level coverage tracing for fuzzing, reach for DynamoRIO or Pin.
• To run one function in isolation without the rest of the program, reach for Unicorn.
• To find an input that takes a specific branch, reach for angr or Triton (symbolic execution).

Frida is the cheapest tool when the question is "what does this function do at runtime." For every other question, something else is cheaper.

Quick reference

The five recipes from this guide, in one place:

# Install
pip install frida-tools
sudo sysctl kernel.yama.ptrace_scope=0   # Linux
 
# Replace a function with a no-op
Interceptor.replace(Module.findExportByName("kernel32.dll", "Sleep"),
  new NativeCallback(function (ms) { return; }, "void", ["uint32"]));
 
# Patch an argument before the call
Interceptor.attach(target, { onEnter(args) { args[2] = ptr(64); } });
 
# Read a buffer before it leaves the process
Interceptor.attach(target, {
  onEnter(args) {
    const len = args[2].toInt32();
    console.log(args[1].readByteArray(len));
  }
});
 
# Hook by offset in a stripped PIE Linux ELF
const m = Process.getModuleByName("challenge");
Interceptor.attach(m.base.add(0x12a0), {
  onLeave(retval) { retval.replace(1); }
});
 
# Generate handlers automatically
frida-trace -i 'CreateFile*' -i 'WriteFile' -f program.exe -X KERNEL32

Next time a challenge title says Binary Instrumentation, don't reach for stringsfirst. Reach for Frida, even if you don't need to.

Adjacent reading on this site: GDB guide, Ghidra guide, ASLR/PIE bypass, Buffer overflow guide.