investigation_encoded_2 picoCTF 2019 Solution

Description

1. Step 1Analyze the differences from part 1
   This challenge uses a more complex encoding. Run the binary and compare its behavior to part 1. Look for multi-byte operations, multiple passes, or a more complex key schedule.
   bash

   ./mystery encodedData

   Copied!

   bash

   strings mystery

   Copied!

   bash

   ghidra mystery &

   Copied!
   Learn more

   More complex encodings may apply multiple transformations in sequence: first XOR, then shift, then substitute - or the encoding may be position-dependent (the key changes based on the current position in the file). Identifying the sequence of operations is the key challenge.

2. Step 2Decompile the binary
   Use Ghidra to fully decompile the encoding function. Trace through each operation on the data bytes. Pay attention to any loop variables or counters that modify the transformation.
   bash

   ghidra mystery &

   Copied!
   Learn more

   In Ghidra, the Data Type Manager lets you define structures that match how the program interprets its data. This can make the decompilation output much more readable when the code processes structured binary data.

3. Step 3Write the inverse decoder
   Implement the inverse of each encoding operation in reverse order. Apply to encodedData to recover the flag.
   python

   python3 << 'EOF'
   with open('encodedData', 'rb') as f:
       data = bytearray(f.read())
   
   # Apply inverse operations in reverse order
   # (fill in from Ghidra analysis)
   for i in range(len(data)):
       data[i] ^= (i & 0xFF)  # example: position-dependent XOR
   
   print(bytes(data).decode('ascii', errors='replace'))
   EOF

   Copied!
   Learn more

   When reversing a multi-step encoding, always reverse the steps in opposite order: if the encoding does A then B, the decoding must do inverse(B) then inverse(A). Forgetting to reverse the order is a common mistake.

Flag

Decompile the binary to find the encoding sequence, then apply each inverse operation in reverse order to decode the flag.