vault-door-6 picoCTF 2019 Solution

Description

1. Step 1

   Find the hardcoded byte array

   Observation

   I noticed the challenge description stated the vault uses XOR with 0x55, which meant the secret must be stored as a pre-XORed byte array inside the Java source; reading checkPassword in VaultDoor6.java would expose both the key and the encrypted values needed to reverse the cipher.

   Open VaultDoor6.java. The checkPassword method XORs each byte of the input with 0x55 and compares it to a hardcoded byte array. Extract that array.

   bash

   cat VaultDoor6.java

   Copied!

   What didn't work first

   Tried: Run the Java file directly with javac and java to see what the program outputs when you enter guesses

   The program only tells you whether a password is correct or not - it does not print the plaintext. You would need to brute-force 256^n combinations, which is infeasible. The right approach is to extract the hardcoded byte array from the source and reverse the XOR offline.

   Tried: Search for a plaintext string in the source with grep or strings instead of reading the full checkPassword method

   The password is stored as a byte array of XOR-encrypted integers like {0x3b, 0x65, ...}, not as a readable ASCII string, so grep for printable characters finds nothing useful. You need to read the full method to identify the byte array and the XOR key (0x55) before you can decrypt.

   Learn more

   0x55 in binary is 01010101. XORing with this value flips every other bit. Because XOR is its own inverse, applying 0x55 twice returns the original value.

   Single-byte XOR ciphers like this are trivially broken: there are only 256 possible keys, so even brute-force works instantly. With the key given in the source code, a single pass suffices.

2. Step 2

   Decrypt using Python

   Observation

   I noticed that XOR is its own inverse, so applying 0x55 again to each extracted byte would undo the encryption; a quick Python one-liner iterating over the array and printing chr(b ^ 0x55) is the fastest way to recover the plaintext password.

   XOR each byte in the hardcoded array with 0x55 to recover the original password characters.

   python

   python3 -c "
   enc = [/* paste byte array values */]
   print(''.join(chr(b ^ 0x55) for b in enc))
   "

   Copied!

   Expected output

   picoCTF{n0t_mUcH_h4rD3r_tH4n_x0r_...}

   What didn't work first

   Tried: XOR each byte value with 0x55 without masking with 0xFF first, then see garbled or negative-index characters

   Java bytes are signed (-128 to 127), so a value like 0xd3 is stored as -45 in the array. In Python, chr(-45 ^ 0x55) raises a ValueError because chr() requires a non-negative integer. Applying & 0xFF first (e.g. chr((b & 0xFF) ^ 0x55)) converts the signed byte to an unsigned 0-255 value before XORing.

   Tried: XOR with 0xAA (the bitwise complement of 0x55) assuming a double-key scheme

   The source code explicitly uses 0x55 as the XOR key in checkPassword - there is no second key. XORing with 0xAA would invert all bits of each byte (equivalent to NOT), producing garbage characters. The correct key is 0x55 as written in the source.

   Learn more

   In Python, the ^ operator performs bitwise XOR on integers. Since each Java byte value may be signed (range -128 to 127), you may need to apply & 0xFF first to convert to an unsigned 0-255 range before XORing.

3. Step 3

   Submit the flag

   Observation

   I noticed the Python output was a readable ASCII string, which is the raw password; wrapping it in the picoCTF{} format completes the flag for submission.

   The decoded string is the password. Wrap it in picoCTF{...} to form the final flag.

   Learn more

   This challenge illustrates why XOR with a fixed single-byte key provides essentially zero security - it is equivalent to a Caesar cipher operating on individual bits rather than characters.

Flag

XOR every byte in the hardcoded array with 0x55 to get the password characters.