asm4 picoCTF 2019 Solution

Description

1. Step 1Understand string processing in assembly

   Observation

   The function takes a string ('picoCTF_d023b4') rather than a number. So the assembly will dereference a pointer and walk it byte by byte, and that loop pattern is what to understand before simulating anything.

   Open test.S. The function asm4 takes a pointer to the string 'picoCTF_d023b4'. It likely computes a numeric value based on the string contents - perhaps a checksum, hash, or character sum.

   bash

   cat test.S

   Copied!

   What didn't work first

   Tried: Assume the operand order is reversed because the file has a .S extension and is built with gcc.

   test.S is a GAS source file, but it opens with the .intel_syntax noprefix directive, so the operands are in Intel order: destination first, source second, and no % or $ prefixes. Reading it as AT&T reverses every instruction and the computed value comes out wrong. Check the first line of the file to see which syntax mode is in effect before tracing anything.

   Tried: Grep for a hardcoded return value or numeric constant in the file, assuming the answer is embedded literally.

   The function computes its result from the input string at runtime, not from a constant. Grepping for hex literals turns up intermediate magic numbers the algorithm uses, not the answer. You have to run or simulate the whole loop with the real argument.

   Learn more

   When a string pointer is passed to a function, the argument at [ebp+8] is the address of the first character. To access character at index i, the assembly uses movzx eax, byte ptr [reg + i] or loads the pointer and uses an index register.

   Common string-processing loops: iterate while the current character is not null (0x00), computing something with each character (sum of ASCII values, XOR of all chars, polynomial hash, etc.).

2. Step 2Simulate the function

   Observation

   Hand-tracing a 32-bit loop over a string is error-prone, mostly because the byte-by-byte accumulation is easy to get off by one. Translating the logic to Python is a safer way to compute the return value before committing to a compiled approach.

   Translate the assembly to Python. Set the input string to 'picoCTF_d023b4' and simulate the operations. The final value in eax is the return value.

   Learn more

   Compiling and running the assembly directly is the most reliable approach. Create a C wrapper: extern int asm4(char *s); int main() { printf("0x%x\n", asm4("picoCTF_d023b4")); }. Compile with gcc -m32 wrapper.c test.S -o test -no-pie and run.

3. Step 3Compile and run with a C wrapper (most reliable)

   Observation

   The .S file is GAS source in Intel-syntax mode, which gcc assembles directly. A small C driver calling asm4 with the exact input string gives the authoritative answer, with no chance of a simulation slip.

   test.S is GAS source (Intel-syntax mode), so let gcc assemble it directly together with a small C driver. Do NOT use nasm here: even though the mnemonics look Intel, the GAS directives (.intel_syntax noprefix, .global, DWORD PTR operands as GAS spells them) are not nasm input. gcc -m32 produces the 32-bit binary matching the challenge calling convention.

   c

   cat > wrapper.c <<'EOF'
   #include <stdio.h>
   extern int asm4(const char *);
   int main(){ printf("0x%x\n", asm4("picoCTF_d023b4")); return 0; }
   EOF

   Copied!

   bash

   gcc -m32 -no-pie wrapper.c test.S -o asm4_run && ./asm4_run

   Copied!

   Expected output

   0x23e

   What didn't work first

   Tried: Assemble test.S with nasm instead of gcc, since the mnemonics are in Intel order and nasm is a well-known assembler for CTF work.

   Intel-order mnemonics are not the same thing as nasm input. test.S is a GAS file: it opens with .intel_syntax noprefix and uses GAS directives such as .global, which nasm does not accept, so nasm produces a wall of parse errors. gcc with -m32 hands the file to gas, which assembles it as is, no conversion step needed.

   Tried: Compile without -m32 on a 64-bit machine, since 64-bit gcc should handle any assembly.

   The function uses the 32-bit convention: the argument is pushed on the stack and read at [ebp+8]. Without -m32, gcc emits 64-bit code that passes arguments in rdi, so the function reads the wrong location and returns garbage or segfaults. -m32 matches the convention the assembly expects.

   Learn more

   Letting the CPU run the real assembly is faster and less error-prone than hand-tracing. Because the file is GAS source (.S), gcc hands it straight to the assembler, Intel-syntax directive and all; the -m32 flag is required on 64-bit Linux to match the 32-bit calling convention the function expects. If your instance uses a different argument string, change it in the wrapper and recompute.

Flag

asm4's argument string is generated per instance, so the returned value is instance-specific; the walkthrough uses picoCTF_d023b4, which returns 0x23e for that instance. Assemble test.S (GAS source in Intel-syntax mode) with gcc -m32 alongside a C driver that calls asm4 - do not use nasm, which does not accept the GAS directives the file uses. If your instance shows a different argument string, rerun the driver with that string. The flag is shown abbreviated on this page; work the steps above to recover the full value.