vault-door-8 picoCTF 2019 Solution

Description

1. Step 1Analyze the scramble method

   Observation

   The description calls out a scramble method that transposes bit pairs. So read VaultDoor8.java first and work out exactly which switchBits operations are chained together, before reversing anything.

   Open VaultDoor8.java. Find the scramble() method which applies a series of bit swaps to each character. The checkPassword method calls scramble() on each input character and compares to a hardcoded array.

   bash

   cat VaultDoor8.java

   Copied!

   What didn't work first

   Tried: Search the file for a string literal that looks like the flag and copy it directly.

   The hardcoded array in checkPassword stores scrambled character codes, not readable ASCII. Printing those raw values yields gibberish. The flag only becomes readable after reversing the switchBits operations on each element.

   Tried: Run javac VaultDoor8.java and then java VaultDoor8 with guessed passwords to brute-force the check.

   The class wants interactive input, and the password space at the required length is far too large to brute-force that way. Reversing the scramble statically is orders of magnitude faster and needs no compiling or running of the Java.

   Learn more

   The scramble method typically performs a sequence of bit-pair swaps on an 8-bit character. For example, it may swap bit 0 with bit 1, bit 2 with bit 3, etc. These are self-contained permutations.

   Because each individual switchBits() swap is its own inverse, you can undo the full sequence by executing the same swaps in reverse order. This means you do NOT re-apply scramble() - you apply the individual operations in reverse sequence.

2. Step 2Reverse the scramble by running the operations in reverse order

   Observation

   Each switchBits() call is its own inverse, since swapping the same two positions twice restores the original. So applying the same operations in reverse order to the hardcoded byte array recovers the password, with no need to run the Java at all.

   One verified approach: copy the Java source, reverse the order of the bit-swap operations in the scramble method, then apply that reversed scramble to the expected array. This unscrambles each byte back to the original password character. Alternatively, since each individual swap is its own inverse, apply the operations in reverse sequence.

   python

   python3 -c "
   def unscramble(c):
       # Replicate the bit-swap operations from Java scramble()
       # Example: swap bits 0,1 then bits 2,3 then bits 4,5 then bits 6,7
       c = ((c & 0x55) << 1) | ((c & 0xAA) >> 1)
       # Add more swaps if the Java code has them
       return c
   
   enc = [/* paste hardcoded byte array */]
   print(''.join(chr(unscramble(b & 0xFF)) for b in enc))
   "

   Copied!

   Expected output

   picoCTF{...}

   What didn't work first

   Tried: Apply the scramble function directly to each byte in the array instead of reversing its operation order.

   Calling scramble() again does not invert it; it applies the same permutation a second time and leaves you doubly scrambled. The individual swaps are self-inverse but the sequence is not symmetric, so apply the same swaps in reversed order rather than re-running scramble().

   Tried: Use only the 0x55 / 0xAA adjacent-pair swap and ignore whether the Java code has additional switchBits calls.

   Java's scramble() usually chains several different bit-pair swaps: positions 0-1, then 2-3, then 4-5, then some non-adjacent pairs. Implement only one pattern and the rest go unaccounted for, leaving characters partly unscrambled and mostly wrong. Replicate every switchBits() call from the source, in reverse order.

   Learn more

   The bitmask 0x55 is 01010101 in binary - it selects all even-positioned bits. The mask 0xAA is 10101010 - it selects all odd-positioned bits. Together they can swap adjacent bit pairs across an entire byte in two operations.

   Read the Java source carefully to replicate the exact sequence of bit swaps. Each swap operation in the Java code must appear in the same order in your Python unscramble function.

3. Step 3Submit the flag

   Observation

   Unscrambling gives a run of printable ASCII characters, which confirms the reversal was right. Wrapping the result in picoCTF{...} produces the flag.

   The unscrambled characters form the password. Wrap in picoCTF{...} to get the flag.

   Learn more

   This is a classic example of a bijective (one-to-one) encoding function used as obfuscation. Since every possible input maps to exactly one output, the function is fully reversible - the only question is figuring out the reverse mapping.

Flag

Reverse the switchBits scramble operations in VaultDoor8.java