asm4 picoCTF 2019 Solution

Description

1. Step 1

   Understand string processing in assembly

   Observation

   I noticed the function receives a string argument ('picoCTF_d023b4') rather than a scalar, which suggested the assembly would dereference a pointer and walk the string byte by byte, so understanding that loop pattern was the necessary first step before attempting any simulation.

   Open test.S. The function asm4 takes a pointer to the string 'picoCTF_d023b4'. It likely computes a numeric value based on the string contents - perhaps a checksum, hash, or character sum.

   bash

   cat test.S

   Copied!

   What didn't work first

   Tried: Open test.S in a text editor and assume nasm/Intel syntax when reading the instructions.

   test.S uses AT&T/GAS syntax, so operand order is reversed from Intel syntax - source comes first, destination comes last. Reading it as Intel syntax causes every operation to appear backwards, making the computed value completely wrong. Always confirm the syntax by checking whether registers are prefixed with % and immediates with $.

   Tried: Grep for a hardcoded return value or numeric constant in the file, assuming the answer is embedded literally.

   The function computes its result dynamically from the input string contents at runtime, not from a constant. Grepping for hex literals may find intermediate magic numbers used in the algorithm, not the final return value. The only way to get the correct answer is to simulate or execute the full loop with the actual argument string.

   Learn more

   When a string pointer is passed to a function, the argument at [ebp+8] is the address of the first character. To access character at index i, the assembly uses movzx eax, byte ptr [reg + i] or loads the pointer and uses an index register.

   Common string-processing loops: iterate while the current character is not null (0x00), computing something with each character (sum of ASCII values, XOR of all chars, polynomial hash, etc.).

2. Step 2

   Simulate the function

   Observation

   I noticed that hand-tracing AT&T-syntax 32-bit assembly is error-prone due to reversed operand order, which suggested translating the logic to Python as a safer way to compute the return value before committing to a compiled approach.

   Translate the assembly to Python. Set the input string to 'picoCTF_d023b4' and simulate the operations. The final value in eax is the return value.

   Learn more

   Compiling and running the assembly directly is the most reliable approach. Create a C wrapper: extern int asm4(char *s); int main() { printf("0x%x\n", asm4("picoCTF_d023b4")); }. Compile with gcc -m32 wrapper.c test.S -o test -no-pie and run.

3. Step 3

   Compile and run with a C wrapper (most reliable)

   Observation

   I noticed the file is a .S extension using AT&T/GAS syntax with %-prefixed registers, which indicated gcc could assemble it directly and that a small C driver calling asm4 with the exact input string would yield the authoritative return value without manual simulation errors.

   test.S is GAS/AT&T syntax, so let gcc assemble it directly together with a small C driver. Do NOT use nasm here (nasm is Intel-syntax and cannot assemble an AT&T .S file). gcc -m32 produces the 32-bit binary matching the challenge calling convention.

   c

   cat > wrapper.c <<'EOF'
   #include <stdio.h>
   extern int asm4(const char *);
   int main(){ printf("0x%x\n", asm4("picoCTF_d023b4")); return 0; }
   EOF

   Copied!

   bash

   gcc -m32 -no-pie wrapper.c test.S -o asm4_run && ./asm4_run

   Copied!

   Expected output

   0x23e

   What didn't work first

   Tried: Assemble test.S with nasm instead of gcc, since nasm is a well-known assembler for CTF work.

   nasm only understands Intel syntax, and test.S is written in AT&T/GAS syntax with %-prefixed registers and $-prefixed immediates. Running nasm on it produces a stream of parse errors immediately. gcc (with the -m32 flag) assembles AT&T .S files natively without any conversion step.

   Tried: Compile without -m32 on a 64-bit machine, since 64-bit gcc should handle any assembly.

   The function uses 32-bit calling convention - the argument is pushed on the stack and accessed via [ebp+8]. Without -m32, gcc generates 64-bit code where arguments are passed in registers (rdi), so the function reads from the wrong location and returns a garbage value or segfaults. The -m32 flag is required to match the convention the challenge assembly expects.

   Learn more

   Letting the CPU run the real assembly is faster and less error-prone than hand-tracing. Because the file is AT&T syntax (.S), gcc assembles it natively; the -m32 flag is required on 64-bit Linux to match the 32-bit calling convention the function expects. If your instance uses a different argument string, change it in the wrapper and recompute.

Flag

asm4('picoCTF_d023b4') returns 0x23e. Assemble test.S (AT&T syntax) with gcc -m32 alongside a C driver that calls asm4 - do not use nasm, which cannot assemble an AT&T .S file. The argument string is instance-specific.