Investigative Reversing 0 picoCTF 2019 Solution

Description

1. Step 1Run the binary and observe the output

   Observation

   The challenge gives an executable and a PNG and asks you to run one on the other. So the binary is the encoder that hides the flag inside mystery.png; the flag is not in the image yet.

   Execute mystery (no arguments needed). It reads flag.txt, opens mystery.png in append mode, and writes 26 encoded flag bytes to the end of the file using fputc(). The PNG pixel data is never touched. After running it, exiftool will report 'Warning: There is trailing data after the PNG IEND chunk', which is the tell that the flag bytes were appended past the end of the valid image structure.

   bash

   ./mystery

   Copied!

   bash

   exiftool mystery.png

   Copied!

   What didn't work first

   Tried: Use steghide or zsteg to find hidden data in mystery.png before running the binary.

   steghide and zsteg look for data inside the pixel plane, via LSB or a passphrase, so they find nothing here. The flag is appended after the IEND chunk, outside the valid PNG structure entirely. Only tools that read past IEND, like exiftool or binwalk, will see it.

   Tried: Run strings on mystery.png to recover the flag directly after executing the binary.

   strings filters for runs of four or more printable characters, so it may show fragments, but the bytes at indices 6-15 are shifted by +5 or -3 and can land outside the printable range, which makes strings split or drop them. Read all 26 raw bytes and apply the inverse arithmetic instead.

   Learn more

   A PNG file ends with an IEND chunk (hex bytes 49 45 4e 44 ae 42 60 82). Any bytes that follow the IEND are ignored by image viewers, making this a simple but effective hiding spot. The binary exploits this by opening the file with the C append flag ("a") and writing the flag data one byte at a time with fputc().

2. Step 2Reverse-engineer the encoding in Ghidra

   Observation

   exiftool reports trailing data after the PNG IEND chunk, and the binary is not stripped. Decompiling it in Ghidra should show the exact arithmetic applied to each flag byte before it was appended.

   Open mystery in Ghidra and find the main encoding loop. The binary reads 26 bytes from flag.txt and writes each one to mystery.png using fputc(), but applies arithmetic transforms to disguise some bytes: bytes at indices 6-14 are increased by 5 before writing, and byte at index 15 is decreased by 3. Bytes 0-5 and 16-25 are written without modification.

   bash

   file mystery

   Copied!

   bash

   ghidra mystery &

   Copied!

   Expected output

   mystery: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped

   What didn't work first

   Tried: Assume bytes are XOR-encoded and try XORing them with common keys like 0x20 or the index value.

   XOR is the most common CTF obfuscation, so it is a natural first guess, but the Ghidra decompilation plainly shows addition and subtraction in the fputc loop, not XOR. Applying XOR gives output that does not start with picoCTF{, which is the cue to go back and read the actual operators.

   Tried: Apply the +5/-3 transforms to all 26 bytes uniformly instead of checking the index ranges.

   The encoding is positional: only indices 6-14 were increased by 5, and only index 15 was decreased by 3. Subtract 5 from bytes 0-5 or 16-25 and you break the prefix and suffix. Ghidra shows two separate conditionals keyed on the loop counter, and both need reading precisely before you write the decoder.

   Learn more

   In the Ghidra decompiler output you will see a loop that calls fputc() once per flag character. Two conditionals inside the loop select which arithmetic to apply based on the current index. This kind of light obfuscation is common in beginner reversing challenges: the encoding is not cryptographic, just a positional byte-shift that must be reversed.

3. Step 3Extract and decode the flag bytes

   Observation

   Ghidra shows positional transforms before each fputc() call: +5 for indices 6 through 14, and -3 for index 15. So slice the last 26 bytes off mystery.png and apply the inverse per index.

   Read the last 26 bytes of the modified mystery.png, then reverse the transforms: subtract 5 from bytes at indices 6-14, add 3 to byte at index 15, and leave all other bytes unchanged. The result is the plaintext flag.

   python

   python3 << 'EOF'
   import os, mmap
   
   def memory_map(filename, access=mmap.ACCESS_READ):
       size = os.path.getsize(filename)
       fd = os.open(filename, os.O_RDONLY)
       return mmap.mmap(fd, size, access=access)
   
   with memory_map("mystery.png") as b:
       raw = b[-26:]          # last 26 bytes are the appended flag
       flag = []
       for i in range(6):
           flag.append(raw[i])            # bytes 0-5: no change
       for i in range(6, 15):
           flag.append(raw[i] - 5)       # bytes 6-14: encoded with +5, reverse with -5
       flag.append(raw[15] + 3)          # byte 15: encoded with -3, reverse with +3
       for i in range(16, 26):
           flag.append(raw[i])            # bytes 16-25: no change
       print(bytearray(flag).decode())
   EOF

   Copied!

   What didn't work first

   Tried: Use xxd to hex-dump the last 26 bytes and manually convert them, skipping the Python mmap script.

   xxd shows the encoded bytes with the +5 and -3 shifts still applied, so reading them by eye gives the wrong characters. The Python script automates the per-index arithmetic; doing it by hand across 26 hex values is error-prone and goes wrong unless every index boundary is exactly right.

   Tried: Slice the last 26 bytes by running the binary a second time before extracting, hoping the file resets.

   The binary opens mystery.png in append mode, so each run tacks on another 26 encoded bytes without truncating. Run it twice and the file grows by 52 bytes, and b[-26:] grabs the second copy. Extract right after the first run, or find the IEND offset with binwalk and slice from there rather than using a fixed -26 tail.

   Learn more

   The Python mmap module maps the file into memory so you can index its bytes directly with negative indices. b[-26:] grabs the trailing 26 bytes, which are exactly the flag bytes appended by the binary. Reversing the transforms is straightforward arithmetic: wherever the encoder added 5, the decoder subtracts 5, and wherever the encoder subtracted 3, the decoder adds 3.

Flag

The flag hash suffix is unique per challenge instance. Run the mystery binary, then run the extraction script above to recover your specific flag.