asm3 picoCTF 2019 Solution

Description

1. Step 1

   Read the assembly and identify operations

   Observation

   I noticed the challenge description referenced bitwise operations and complex assembly, which suggested I needed to first read the raw source file and catalog each instruction before attempting any manual trace.

   Open test.S. The function asm3 takes three 32-bit arguments. It uses bitwise operations (and, or, xor, shl, shr) and possibly memory accesses. Trace carefully, noting that accessing parts of registers (al, ax vs eax) changes only parts of the value.

   bash

   cat test.S

   Copied!

   What didn't work first

   Tried: Treating all register accesses as full 32-bit eax reads and computing the final value from the whole 32-bit arguments

   When the assembly reads al, ah, or ax, it is only reading 1 or 2 bytes of eax, not the full 32-bit value. Adding the byte-width of the operand to your trace matters: mov al, [ebp+9] loads a single byte from offset 9 off ebp, so applying 0xc264bd5c as a 32-bit value to that offset gives the wrong byte. The correct approach reads the specific byte at the exact stack offset in little-endian order.

   Tried: Running objdump or ndisasm on test.S instead of reading the source directly with cat

   test.S is already an AT&T or Intel syntax assembly source file, not a compiled binary, so objdump and ndisasm expect an ELF or raw binary and will either error or produce garbage output. Reading the .S file directly with cat gives you the human-readable mnemonics you need to trace the operations.

   Learn more

   x86 registers have multiple access widths: eax = full 32-bit, ax = lower 16-bit, ah = upper byte of ax, al = lower byte of ax. Writing to al only changes the low byte of eax; the upper 24 bits are unchanged.

   Key bitwise instructions: movzx eax, al = zero-extend al into eax (clears upper bits). movsx = sign-extend. shl reg, n = shift left by n. shr reg, n = shift right (unsigned). sar reg, n = arithmetic shift right (signed, fills with sign bit).

2. Step 2

   Track partial register operations

   Observation

   I noticed the assembly accessed registers like al, ah, and ax rather than the full eax, which indicated the function was reading individual bytes from the stacked arguments and required careful byte-width tracking to avoid producing the wrong value.

   The arguments are on the stack. The function likely loads specific bytes from the arguments using byte/word pointer accesses. Track each byte manipulation carefully.

   Learn more

   If the assembly accesses [ebp+8] as a dword (32-bit) but then [ebp+10] as a word, it is reading a 2-byte slice from the middle of the first argument. This is a way of extracting specific bytes from a 32-bit value by treating the stack memory as a byte array.

3. Step 3

   Assemble test.S and call asm3 directly (most reliable)

   Observation

   I noticed that the byte-offset arithmetic across three 32-bit arguments was error-prone to trace by hand and that test.S is valid assemblable source, which suggested compiling it as a 32-bit cdecl function and calling it from a C driver to get the exact answer without any manual arithmetic risk.

   asm3 returns a 16-bit value in ax built by pulling specific bytes out of the three stacked arguments (by byte offset, e.g. [ebp+0x9], [ebp+0xd], [ebp+0xe], [ebp+0x12]), doing byte add/sub, and a final 16-bit xor. Rather than hand-trace, assemble test.S as 32-bit and call asm3 with the challenge's three arguments. The printed return value (mask to 16 bits) is the answer; wrap it as picoCTF{0x....}. Note the arguments are instance-specific, so use the exact triple your test.S/prompt shows.

   c

   cat > driver.c <<'EOF'
   #include <stdio.h>
   unsigned int asm3(unsigned int, unsigned int, unsigned int);
   int main(){
       printf("0x%x\n", asm3(0xc264bd5c, 0xb5a06caa, 0xa9820482) & 0xffff);
       return 0;
   }
   EOF
   gcc -m32 -no-pie driver.c test.S -o asm3 && ./asm3

   Copied!

   Expected output

   picoCTF{0x...}

   What didn't work first

   Tried: Compiling driver.c and test.S as 64-bit (omitting -m32) and running the result

   In 64-bit mode the calling convention passes arguments in registers (rdi, rsi, rdx) rather than on the stack, so asm3's [ebp+8], [ebp+0xd], etc. stack-offset reads dereference garbage memory. The compiled binary may segfault or return an entirely wrong value. Adding -m32 forces the cdecl stack-based convention that the assembly was written for.

   Tried: Printing the return value as a full 32-bit int with %x instead of masking to 16 bits with & 0xffff

   asm3 builds its result in the ax (16-bit) register and leaves the upper 16 bits of eax from prior operations in place. Printing the unmasked int can include those high bits, giving a value like 0xdeadXXXX where only XXXX is the actual answer. The challenge expects the answer wrapped as picoCTF{0x....} using only the 16-bit return value.

   Learn more

   Because the function reads its operands by byte offset off the stack, the result depends on all three arguments, and asm3's arguments are randomized per instance. Compiling and calling the real assembly removes any chance of a hand-tracing mistake and always yields the correct value for your exact triple. If you must trace by hand, follow the byte offsets into each 4-byte argument (little-endian) and apply the add/sub/xor in order.

Flag

asm3 extracts specific bytes from the three arguments by stack offset, does byte arithmetic, and returns a 16-bit value in ax. Assemble test.S 32-bit and call asm3 with your instance's exact argument triple to get the value; submit it as picoCTF{0x....}. The arguments (and thus the answer) are instance-specific.