asm3 picoCTF 2019 Solution

Description

1. Step 1Read the assembly and identify operations

   Observation

   The description mentions bitwise operations and complicated assembly. So the first job is reading the raw source and cataloguing each instruction before tracing anything by hand.

   Open test.S. The function asm3 takes three 32-bit arguments. It uses bitwise operations (and, or, xor, shl, shr) and possibly memory accesses. Trace carefully, noting that accessing parts of registers (al, ax vs eax) changes only parts of the value.

   bash

   cat test.S

   Copied!

   What didn't work first

   Tried: Treating all register accesses as full 32-bit eax reads and computing the final value from the whole 32-bit arguments

   Reading al, ah, or ax takes only 1 or 2 bytes of eax, not the full 32-bit value, so the operand width matters in the trace. mov al, [ebp+9] loads a single byte from offset 9, so treating 0xc264bd5c as a 32-bit value at that offset gives the wrong byte. Read the specific byte at the exact stack offset, in little-endian order.

   Tried: Running objdump or ndisasm on test.S instead of reading the source directly with cat

   test.S is assembly source, not a compiled binary, so objdump and ndisasm expect an ELF or raw machine code and either error out or print garbage. Read the .S file with cat and you get the mnemonics you actually need.

   Learn more

   x86 registers have multiple access widths: eax = full 32-bit, ax = lower 16-bit, ah = upper byte of ax, al = lower byte of ax. Writing to al only changes the low byte of eax; the upper 24 bits are unchanged.

   Key bitwise instructions: movzx eax, al = zero-extend al into eax (clears upper bits). movsx = sign-extend. shl reg, n = shift left by n. shr reg, n = shift right (unsigned). sar reg, n = arithmetic shift right (signed, fills with sign bit).

2. Step 2Track partial register operations

   Observation

   The assembly touches al, ah, and ax rather than the full eax. It is pulling individual bytes out of the stacked arguments, so the byte widths have to be tracked carefully or the result comes out wrong.

   The arguments are on the stack. The function likely loads specific bytes from the arguments using byte/word pointer accesses. Track each byte manipulation carefully.

   Learn more

   If the assembly accesses [ebp+8] as a dword (32-bit) but then [ebp+10] as a word, it is reading a 2-byte slice from the middle of the first argument. This is a way of extracting specific bytes from a 32-bit value by treating the stack memory as a byte array.

3. Step 3Assemble test.S and call asm3 directly (most reliable)

   Observation

   Tracing byte offsets across three 32-bit arguments by hand is easy to get wrong, and test.S is valid assemblable source. Compiling it as a 32-bit cdecl function and calling it from a small C driver gives the exact answer with no manual arithmetic.

   asm3 returns a 16-bit value in ax built by pulling specific bytes out of the three stacked arguments (by byte offset, e.g. [ebp+0x9], [ebp+0xd], [ebp+0xe], [ebp+0x12]), doing byte add/sub, and a final 16-bit xor. Rather than hand-trace, assemble test.S as 32-bit and call asm3 with the challenge's three arguments. The printed return value (mask to 16 bits) is the answer; wrap it as picoCTF{0x....}. Note the arguments are instance-specific, so use the exact triple your test.S/prompt shows.

   c

   cat > driver.c <<'EOF'
   #include <stdio.h>
   unsigned int asm3(unsigned int, unsigned int, unsigned int);
   int main(){
       printf("0x%x\n", asm3(0xc264bd5c, 0xb5a06caa, 0xa9820482) & 0xffff);
       return 0;
   }
   EOF
   gcc -m32 -no-pie driver.c test.S -o asm3 && ./asm3

   Copied!

   Expected output

   picoCTF{0x...}

   What didn't work first

   Tried: Compiling driver.c and test.S as 64-bit (omitting -m32) and running the result

   In 64-bit mode the calling convention passes arguments in registers (rdi, rsi, rdx) rather than on the stack, so asm3's [ebp+8] and [ebp+0xd] reads dereference garbage. The binary may segfault or return a wrong value. Adding -m32 forces the cdecl stack convention the assembly was written for.

   Tried: Printing the return value as a full 32-bit int with %x instead of masking to 16 bits with & 0xffff

   asm3 builds its result in the 16-bit ax register and leaves whatever was in the upper half of eax untouched. Printing the unmasked int drags those high bits along, giving something like 0xdeadXXXX where only XXXX is the answer. The challenge wants just the 16-bit return value.

   Learn more

   Because the function reads its operands by byte offset off the stack, the result depends on all three arguments, and asm3's arguments are randomized per instance. Compiling and calling the real assembly removes any chance of a hand-tracing mistake and always yields the correct value for your exact triple. If you must trace by hand, follow the byte offsets into each 4-byte argument (little-endian) and apply the add/sub/xor in order.

Flag

asm3 extracts specific bytes from the three arguments by stack offset, does byte arithmetic, and returns a 16-bit value in ax. Assemble test.S 32-bit and call asm3 with your instance's exact argument triple to get the value; submit it as picoCTF{0x....}. The arguments (and thus the answer) are instance-specific.