vault-door-8 picoCTF 2019 Solution

Description

1. Step 1

   Analyze the scramble method

   Observation

   I noticed the description explicitly called out a scramble method that transposes bit pairs, which suggested the first step was to read VaultDoor8.java and understand exactly which switchBits operations are chained together before attempting any reversal.

   Open VaultDoor8.java. Find the scramble() method which applies a series of bit swaps to each character. The checkPassword method calls scramble() on each input character and compares to a hardcoded array.

   bash

   cat VaultDoor8.java

   Copied!

   What didn't work first

   Tried: Search the file for a string literal that looks like the flag and copy it directly.

   The hardcoded array in checkPassword stores scrambled character codes, not readable ASCII. Printing those raw values yields gibberish. The flag only becomes readable after reversing the switchBits operations on each element.

   Tried: Run javac VaultDoor8.java and then java VaultDoor8 with guessed passwords to brute-force the check.

   The class expects interactive input and the password space (printable ASCII to the required length) is too large to brute-force interactively. Static reversal of the scramble is orders of magnitude faster and does not require compiling or running the Java code at all.

   Learn more

   The scramble method typically performs a sequence of bit-pair swaps on an 8-bit character. For example, it may swap bit 0 with bit 1, bit 2 with bit 3, etc. These are self-contained permutations.

   Because each individual switchBits() swap is its own inverse, you can undo the full sequence by executing the same swaps in reverse order. This means you do NOT re-apply scramble() - you apply the individual operations in reverse sequence.

2. Step 2

   Reverse the scramble by running the operations in reverse order

   Observation

   I noticed that each switchBits() call is its own inverse (swapping the same two bit positions twice restores the original), which suggested that applying the same operations in reverse order to the hardcoded byte array would recover the original password characters without needing to run the Java program at all.

   One verified approach: copy the Java source, reverse the order of the bit-swap operations in the scramble method, then apply that reversed scramble to the expected array. This unscrambles each byte back to the original password character. Alternatively, since each individual swap is its own inverse, apply the operations in reverse sequence.

   python

   python3 -c "
   def unscramble(c):
       # Replicate the bit-swap operations from Java scramble()
       # Example: swap bits 0,1 then bits 2,3 then bits 4,5 then bits 6,7
       c = ((c & 0x55) << 1) | ((c & 0xAA) >> 1)
       # Add more swaps if the Java code has them
       return c
   
   enc = [/* paste hardcoded byte array */]
   print(''.join(chr(unscramble(b & 0xFF)) for b in enc))
   "

   Copied!

   Expected output

   picoCTF{...}

   What didn't work first

   Tried: Apply the scramble function directly to each byte in the array instead of reversing its operation order.

   Calling scramble() again does not invert it - it applies the same permutation a second time, producing a doubly-scrambled result. Because the individual swaps are self-inverse but the sequence is not symmetric, you must apply the same swap operations in reversed order, not re-run scramble() from start to finish.

   Tried: Use only the 0x55 / 0xAA adjacent-pair swap and ignore whether the Java code has additional switchBits calls.

   The Java scramble() typically chains multiple different bit-pair swaps (e.g. positions 0-1, then 2-3, then 4-5, then non-adjacent pairs). Implementing only one swap pattern leaves the other transpositions unaccounted for, producing characters that are partially unscrambled and mostly wrong. Read every switchBits() call in the source and replicate each one in reverse order.

   Learn more

   The bitmask 0x55 is 01010101 in binary - it selects all even-positioned bits. The mask 0xAA is 10101010 - it selects all odd-positioned bits. Together they can swap adjacent bit pairs across an entire byte in two operations.

   Read the Java source carefully to replicate the exact sequence of bit swaps. Each swap operation in the Java code must appear in the same order in your Python unscramble function.

3. Step 3

   Submit the flag

   Observation

   I noticed the Python script output a sequence of printable ASCII characters after unscrambling, which confirmed the reversal was correct and that wrapping the result in picoCTF{...} would produce the valid flag.

   The unscrambled characters form the password. Wrap in picoCTF{...} to get the flag.

   Learn more

   This is a classic example of a bijective (one-to-one) encoding function used as obfuscation. Since every possible input maps to exactly one output, the function is fully reversible - the only question is figuring out the reverse mapping.

Flag

Reverse the switchBits scramble operations in VaultDoor8.java