Investigative Reversing 0 picoCTF 2019 Solution

Description

1. Step 1

   Run the binary and observe the output

   Observation

   I noticed the challenge provides both an executable binary and a PNG file and asks you to run the binary on the image, which suggested the binary itself is the encoder that hides the flag inside mystery.png rather than the flag already being present.

   Execute mystery (no arguments needed). It reads flag.txt, opens mystery.png in append mode, and writes 26 encoded flag bytes to the end of the file using fputc(). The PNG pixel data is never touched. After running it, exiftool will report 'Warning: There is trailing data after the PNG IEND chunk', which is the tell that the flag bytes were appended past the end of the valid image structure.

   bash

   ./mystery

   Copied!

   bash

   exiftool mystery.png

   Copied!

   What didn't work first

   Tried: Use steghide or zsteg to find hidden data in mystery.png before running the binary.

   steghide and zsteg look for data embedded inside the image pixel plane using LSB or passphrase-based hiding - they will find nothing because the flag is appended after the IEND chunk, not embedded in the pixel data. The binary writes raw bytes past the end of the valid PNG structure, which only tools that read beyond the IEND marker (like exiftool or binwalk) will detect.

   Tried: Run strings on mystery.png to recover the flag directly after executing the binary.

   strings filters for sequences of printable ASCII 4+ characters, so it may show fragments, but the encoded bytes at indices 6-15 are shifted by +5 or -3 and may fall outside the printable range, causing strings to split or omit them. The correct approach reads all 26 raw bytes and applies the inverse arithmetic transforms, not a printability filter.

   Learn more

   A PNG file ends with an IEND chunk (hex bytes 49 45 4e 44 ae 42 60 82). Any bytes that follow the IEND are ignored by image viewers, making this a simple but effective hiding spot. The binary exploits this by opening the file with the C append flag ("a") and writing the flag data one byte at a time with fputc().

2. Step 2

   Reverse-engineer the encoding in Ghidra

   Observation

   I noticed that exiftool reported trailing data after the PNG IEND chunk and the binary is not stripped, which suggested decompiling it in Ghidra to find the exact arithmetic transforms applied to each flag byte before they were appended.

   Open mystery in Ghidra and find the main encoding loop. The binary reads 26 bytes from flag.txt and writes each one to mystery.png using fputc(), but applies arithmetic transforms to disguise some bytes: bytes at indices 6-14 are increased by 5 before writing, and byte at index 15 is decreased by 3. Bytes 0-5 and 16-25 are written without modification.

   bash

   file mystery

   Copied!

   bash

   ghidra mystery &

   Copied!

   Expected output

   mystery: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped

   What didn't work first

   Tried: Assume bytes are XOR-encoded and try XORing them with common keys like 0x20 or the index value.

   XOR is the most common CTF obfuscation so it is a natural first guess, but Ghidra decompilation clearly shows addition and subtraction operations (not bitwise XOR) inside the fputc loop. Applying XOR will produce garbled output that does not start with picoCTF{, which is the signal to go back to Ghidra and read the actual arithmetic operators.

   Tried: Apply the +5/-3 transforms to all 26 bytes uniformly instead of checking the index ranges.

   The encoding is positional: only indices 6-14 were increased by 5 and only index 15 was decreased by 3. Subtracting 5 from bytes 0-5 or 16-25 will produce wrong character values that break the flag prefix and suffix. Ghidra shows two separate conditional branches keyed on the loop counter - reading both branch conditions precisely is required before writing the decoder.

   Learn more

   In the Ghidra decompiler output you will see a loop that calls fputc() once per flag character. Two conditionals inside the loop select which arithmetic to apply based on the current index. This kind of light obfuscation is common in beginner reversing challenges: the encoding is not cryptographic, just a positional byte-shift that must be reversed.

3. Step 3

   Extract and decode the flag bytes

   Observation

   I noticed Ghidra revealed positional arithmetic transforms (+5 for indices 6-14, -3 for index 15) applied before each fputc() call, which suggested slicing the last 26 bytes from mystery.png and applying the inverse operations per index to recover the plaintext flag.

   Read the last 26 bytes of the modified mystery.png, then reverse the transforms: subtract 5 from bytes at indices 6-14, add 3 to byte at index 15, and leave all other bytes unchanged. The result is the plaintext flag.

   python

   python3 << 'EOF'
   import os, mmap
   
   def memory_map(filename, access=mmap.ACCESS_READ):
       size = os.path.getsize(filename)
       fd = os.open(filename, os.O_RDONLY)
       return mmap.mmap(fd, size, access=access)
   
   with memory_map("mystery.png") as b:
       raw = b[-26:]          # last 26 bytes are the appended flag
       flag = []
       for i in range(6):
           flag.append(raw[i])            # bytes 0-5: no change
       for i in range(6, 15):
           flag.append(raw[i] - 5)       # bytes 6-14: encoded with +5, reverse with -5
       flag.append(raw[15] + 3)          # byte 15: encoded with -3, reverse with +3
       for i in range(16, 26):
           flag.append(raw[i])            # bytes 16-25: no change
       print(bytearray(flag).decode())
   EOF

   Copied!

   What didn't work first

   Tried: Use xxd to hex-dump the last 26 bytes and manually convert them, skipping the Python mmap script.

   xxd shows the raw encoded bytes (with the +5/-3 shifts still applied), so reading them by eye gives the wrong characters without applying the inverse transforms. The Python script is needed to automate the per-index arithmetic - doing it manually on 26 hex values is error-prone and will produce an incorrect flag unless every index boundary is handled exactly right.

   Tried: Slice the last 26 bytes by running the binary a second time before extracting, hoping the file resets.

   The binary opens mystery.png in append mode (C flag 'a'), so every execution appends another 26 encoded bytes without truncating. Running it twice makes the file grow by 52 bytes, and b[-26:] will extract the second copy. The correct approach is to extract immediately after the first run, or to identify the IEND offset with binwalk and slice from there rather than using a fixed -26 tail index.

   Learn more

   The Python mmap module maps the file into memory so you can index its bytes directly with negative indices. b[-26:] grabs the trailing 26 bytes, which are exactly the flag bytes appended by the binary. Reversing the transforms is straightforward arithmetic: wherever the encoder added 5, the decoder subtracts 5, and wherever the encoder subtracted 3, the decoder adds 3.

Flag

The flag hash suffix is unique per challenge instance. Run the mystery binary, then run the extraction script above to recover your specific flag.