asm3 picoCTF 2019 Solution

Description

1. Step 1Read the assembly and identify operations
   Open test.S. The function asm3 takes three 32-bit arguments. It uses bitwise operations (and, or, xor, shl, shr) and possibly memory accesses. Trace carefully, noting that accessing parts of registers (al, ax vs eax) changes only parts of the value.
   bash

   cat test.S

   Copied!
   Learn more

   x86 registers have multiple access widths: eax = full 32-bit, ax = lower 16-bit, ah = upper byte of ax, al = lower byte of ax. Writing to al only changes the low byte of eax; the upper 24 bits are unchanged.

   Key bitwise instructions: movzx eax, al = zero-extend al into eax (clears upper bits). movsx = sign-extend. shl reg, n = shift left by n. shr reg, n = shift right (unsigned). sar reg, n = arithmetic shift right (signed, fills with sign bit).

2. Step 2Track partial register operations
   The arguments are on the stack. The function likely loads specific bytes from the arguments using byte/word pointer accesses. Track each byte manipulation carefully.
   Learn more

   If the assembly accesses [ebp+8] as a dword (32-bit) but then [ebp+10] as a word, it is reading a 2-byte slice from the middle of the first argument. This is a way of extracting specific bytes from a 32-bit value by treating the stack memory as a byte array.

3. Step 3Compute and submit the result
   After tracing all operations, the final value in eax is the return value. Express it as a hex number for the flag.
   Learn more

   For complex assembly, writing an equivalent C program and compiling it is faster than manual tracing: write the assembly as C operations, compile with gcc -O0, and run it with the given arguments. GDB can also evaluate assembly expressions at the ret instruction.

Flag

Trace the bitwise operations and partial register accesses to find the return value of asm3 with the given three arguments.