unpackme picoCTF 2022 Solution

Description

1. Step 1

   Unpack the UPX binary

   Observation

   I noticed the challenge name 'unpackme-upx' and that running strings on the binary returned mostly garbage alongside the magic four-byte tag 'UPX!', which confirmed the binary was compressed with UPX and suggested using upx -d to restore the original ELF before any further analysis.

   Run upx -d unpackme-upx to decompress the binary in-place (or use -o to write a new file). The decompressed ELF can then be analyzed normally.

   bash

   upx -d unpackme-upx -o unpackme-unpacked

   Copied!

   bash

   ls -lh unpackme-upx unpackme-unpacked

   Copied!

   What didn't work first

   Tried: Run strings on the packed binary hoping to find the flag or a readable password.

   The packed binary's code segment is compressed, so strings returns mostly noise and a handful of UPX stub artifacts. The interesting strings (scanf format specifier, the comparison constant printed as a string) only appear after upx -d decompresses the ELF. Strings on the packed file is a dead end.

   Tried: Try upx -d without the -o flag, overwriting the original binary in-place, then discover the original packed file is gone.

   upx -d without -o replaces the input file with the decompressed version, which is fine if you still have the original download. If you then want to compare file sizes or re-run the experiment you need to re-download. Using -o unpackme-unpacked preserves the original alongside the decompressed copy.

   Learn more

   UPX (Ultimate Packer for eXecutables) is a free, open-source packer that compresses executable files. At runtime, the UPX stub decompresses the original code into memory and jumps to the original entry point. Packed binaries are smaller on disk but unpack to their full size in memory.

   Malware commonly uses UPX (or custom packers) to evade antivirus signature scanning - the raw bytes of the packed file don't match the signatures for the malicious code inside. Analysts unpack first, then scan or reverse engineer the inner binary.

   UPX stores a magic header (UPX!) at the end of compressed segments. The -d flag decompresses; -l lists packing info. Running strings on a packed binary shows mostly garbage; after unpacking, meaningful strings like function names and the password comparison become visible.

2. Step 2

   Find the integer check in Ghidra

   Observation

   I noticed the unpacked binary prompted for 'my favorite number' rather than a string password, which suggested the secret was a numeric constant hardcoded in a cmp instruction and that loading the binary into Ghidra's decompiler would expose it directly in the pseudocode of main.

   Open the unpacked binary in Ghidra. Navigate to main() and look for the scanf call reading an integer, followed by a comparison of the local variable against the constant 0xb83cb. The binary asks 'What's my favorite number?' and expects a decimal integer, not a string.

   What didn't work first

   Tried: Search for strcmp or strncmp calls in Ghidra expecting a hardcoded string password.

   The binary uses scanf("%d", ...) and a cmp instruction against an integer constant, not a string comparison. The Functions window and cross-references show no strcmp call at all. Searching for string xrefs yields only the prompt and flag format string, not a password. The secret is a numeric constant visible only as a hex literal in the cmp operand.

   Tried: Run ltrace ./unpackme-unpacked and enter a guess, expecting ltrace to reveal what the binary compares against.

   ltrace intercepts dynamic library calls, but the integer comparison here is a bare cmp instruction inside main - not a library function. ltrace output shows the scanf call but never a strcmp or memcmp, so no comparison value is leaked. GDB with a breakpoint at the cmp instruction or static Ghidra analysis is needed instead.

   Learn more

   Ghidra is NSA's free reverse engineering framework. After importing the binary and running auto-analysis, use the Symbol Tree panel to navigate directly to main. Ghidra's decompiler (open via Window > Decompiler, or the docked Decompile panel) converts the disassembly into readable C pseudocode.

   Finding main in a stripped binary. UPX-unpacked binaries are usually stripped (no main symbol). The standard trick is to find the call to __libc_start_main: the very first argument to it is a function pointer to main. Two ways:

   # objdump way: print the disassembly around every __libc_start_main call.
   objdump -d unpackme-unpacked | grep -B5 '__libc_start_main'
   
   # Look for the immediately-preceding instruction that loads RDI (System V x86-64),
   # e.g. "lea rdi, [rip+0xNNN]" -> RIP-relative address of main.
   
   # In Ghidra: open the entry function (usually _start), follow the first argument
   # loaded into RDI before the call __libc_start_main, double-click the address.

   In Ghidra's decompiler view of main, you will see a pattern like scanf("%d", &local_44) followed by if (local_44 == 0xb83cb). This is a direct integer comparison, not a string comparison. Convert the hex constant to decimal: 0xb83cb = 754635. That is the number you enter.

   Integer vs. string comparison. The disassembly uses a cmp instruction directly on the register holding the scanned integer, rather than calling strcmp. There is no hardcoded string password in this binary. The distinction matters for dynamic analysis too: ltrace will not show a strcmp call here because none exists.

3. Step 3

   Run the binary with the correct number

   Observation

   I noticed the Ghidra decompiler showed the comparison as if (local_44 == 0xb83cb) with scanf reading a decimal integer via %d, which meant converting the hex constant 0xb83cb to its decimal equivalent 754635 and typing it at the prompt would satisfy the check and print the flag.

   When prompted 'What's my favorite number?', enter 754635 (the decimal form of 0xb83cb). The binary compares your input to that constant and, on a match, prints the flag.

   bash

   ./unpackme-unpacked

   Copied!

   bash

   # When prompted, enter: 754635

   Copied!

   Expected output

   What's my favorite number? 754635
   picoCTF{up><_m3_f7w_...}

   What didn't work first

   Tried: Enter 0xb83cb (the hex literal directly) at the prompt instead of the decimal value 754635.

   scanf with the %d format reads a decimal integer from stdin. Typing 0xb83cb is not recognized as hex by %d - scanf reads the leading '0' as zero and stops at 'x', producing the value 0, which fails the comparison. The constant must be converted to decimal (754635) before entering it.

   Tried: Run the still-packed unpackme-upx binary directly instead of the unpacked copy, then enter 754635.

   The packed binary decompresses itself into memory at runtime and would also accept 754635, so the answer is the same. However, running the packed binary without first unpacking it means you cannot inspect the memory image statically - Ghidra would analyze the UPX stub, not the real code. The challenge workflow requires unpacking first so static analysis is possible.

   Learn more

   Once you have the constant from static analysis, simply run the binary normally and type 754635 when prompted. The cmp instruction succeeds and the flag is printed to stdout.

   An alternative to static analysis is dynamic analysis with GDB: set a breakpoint at the comparison instruction, run the binary, and inspect the register or stack value being compared. Because the comparison is a direct integer cmp rather than a library call, ltrace will not capture it. Use strace or GDB instead for dynamic approaches.

   Hex-to-decimal conversion. The constant 0xb83cb can be converted quickly with Python: python3 -c "print(0xb83cb)" prints 754635. Always convert hex constants to decimal before entering them at a decimal scanf prompt, or the comparison will fail.

   strace (system call trace) and ltrace (library call trace) are essential dynamic analysis tools for understanding what a binary does without fully reversing it. They are commonly combined with GDB for a complete picture, especially when dealing with direct integer comparisons that bypass library functions.

Flag

Unpack with `upx -d`, find the integer constant 0xb83cb (754635) in Ghidra's decompiled main(), then enter it when the binary asks 'What\'s my favorite number?'.