asm4 picoCTF 2019 Solution

Description

1. Step 1Understand string processing in assembly

   Observation

   The function takes a string ('picoCTF_d023b4') rather than a number. So the assembly will dereference a pointer and walk it byte by byte, and that loop pattern is what to understand before simulating anything.

   Open test.S. The function asm4 takes a pointer to the string 'picoCTF_d023b4'. It likely computes a numeric value based on the string contents - perhaps a checksum, hash, or character sum.

   bash

   cat test.S

   Copied!

   What didn't work first

   Tried: Open test.S in a text editor and assume nasm/Intel syntax when reading the instructions.

   test.S is AT&T/GAS syntax, so the operand order is reversed from Intel: source first, destination last. Read it as Intel and every operation looks backwards, and the computed value is wrong. Check for %-prefixed registers and $-prefixed immediates to confirm which syntax you are in.

   Tried: Grep for a hardcoded return value or numeric constant in the file, assuming the answer is embedded literally.

   The function computes its result from the input string at runtime, not from a constant. Grepping for hex literals turns up intermediate magic numbers the algorithm uses, not the answer. You have to run or simulate the whole loop with the real argument.

   Learn more

   When a string pointer is passed to a function, the argument at [ebp+8] is the address of the first character. To access character at index i, the assembly uses movzx eax, byte ptr [reg + i] or loads the pointer and uses an index register.

   Common string-processing loops: iterate while the current character is not null (0x00), computing something with each character (sum of ASCII values, XOR of all chars, polynomial hash, etc.).

2. Step 2Simulate the function

   Observation

   Hand-tracing 32-bit AT&T assembly is error-prone, mostly because the operand order is reversed. Translating the logic to Python is a safer way to compute the return value before committing to a compiled approach.

   Translate the assembly to Python. Set the input string to 'picoCTF_d023b4' and simulate the operations. The final value in eax is the return value.

   Learn more

   Compiling and running the assembly directly is the most reliable approach. Create a C wrapper: extern int asm4(char *s); int main() { printf("0x%x\n", asm4("picoCTF_d023b4")); }. Compile with gcc -m32 wrapper.c test.S -o test -no-pie and run.

3. Step 3Compile and run with a C wrapper (most reliable)

   Observation

   The .S file uses AT&T/GAS syntax with %-prefixed registers, which gcc assembles directly. A small C driver calling asm4 with the exact input string gives the authoritative answer, with no chance of a simulation slip.

   test.S is GAS/AT&T syntax, so let gcc assemble it directly together with a small C driver. Do NOT use nasm here (nasm is Intel-syntax and cannot assemble an AT&T .S file). gcc -m32 produces the 32-bit binary matching the challenge calling convention.

   c

   cat > wrapper.c <<'EOF'
   #include <stdio.h>
   extern int asm4(const char *);
   int main(){ printf("0x%x\n", asm4("picoCTF_d023b4")); return 0; }
   EOF

   Copied!

   bash

   gcc -m32 -no-pie wrapper.c test.S -o asm4_run && ./asm4_run

   Copied!

   Expected output

   0x23e

   What didn't work first

   Tried: Assemble test.S with nasm instead of gcc, since nasm is a well-known assembler for CTF work.

   nasm only understands Intel syntax, and test.S is AT&T/GAS with %-prefixed registers and $-prefixed immediates, so nasm produces a wall of parse errors. gcc with -m32 assembles AT&T .S files natively, no conversion step needed.

   Tried: Compile without -m32 on a 64-bit machine, since 64-bit gcc should handle any assembly.

   The function uses the 32-bit convention: the argument is pushed on the stack and read at [ebp+8]. Without -m32, gcc emits 64-bit code that passes arguments in rdi, so the function reads the wrong location and returns garbage or segfaults. -m32 matches the convention the assembly expects.

   Learn more

   Letting the CPU run the real assembly is faster and less error-prone than hand-tracing. Because the file is AT&T syntax (.S), gcc assembles it natively; the -m32 flag is required on 64-bit Linux to match the 32-bit calling convention the function expects. If your instance uses a different argument string, change it in the wrapper and recompute.

Flag

asm4('picoCTF_d023b4') returns 0x23e. Assemble test.S (AT&T syntax) with gcc -m32 alongside a C driver that calls asm4 - do not use nasm, which cannot assemble an AT&T .S file. The argument string is instance-specific.