vault-door-7 picoCTF 2019 Solution

Description

1. Step 1

   Understand the bit manipulation

   Observation

   I noticed the checkPassword method in VaultDoor7.java uses bit shift operators and OR-ing to combine characters into 32-bit integers, which suggested the password is packed rather than transformed and can be fully recovered by reversing those shifts.

   Open VaultDoor7.java. The checkPassword method packs four characters at a time into a 32-bit integer using bit shifts, then compares to hardcoded integers. You need to reverse this packing.

   bash

   cat VaultDoor7.java

   Copied!

   What didn't work first

   Tried: Look for a string literal or byte array in the source to read the password directly.

   There is no readable string constant. The hardcoded values are 32-bit integer literals that encode four characters each using bit shifts. You have to extract each byte from each integer mathematically - no string concatenation or array index trick will expose them.

   Tried: Assume the integers are ASCII codes and cast each one directly to a character.

   Each integer holds four packed characters, not one. Casting the full 32-bit value to a char (or chr() in Python) gives a Unicode code point far outside printable ASCII range. You must right-shift by 24, 16, 8, and 0 bits and mask with 0xFF to extract each individual byte.

   Learn more

   The encoding works by shifting each character's ASCII value to a specific bit position within a 32-bit integer. For example, four chars packed as: (c0 << 24) | (c1 << 16) | (c2 << 8) | c3.

   Bit shifting left by N is equivalent to multiplying by 2^N. OR-ing the shifted values combines them into a single integer without overlap, since each character occupies exactly 8 bits (one byte) within the 32-bit word.

2. Step 2

   Reverse the packing in Python

   Observation

   I noticed each hardcoded integer encodes exactly four characters via left shifts of 24, 16, 8, and 0 bits, which suggested writing a short Python script to apply the inverse right shifts with a 0xFF mask to extract each byte as a character.

   For each hardcoded 32-bit integer, extract each 8-bit group by right-shifting and masking with 0xFF. This recovers the four original characters per integer.

   python

   python3 -c "
   ints = [/* paste hardcoded integers */]
   password = ''
   for val in ints:
       password += chr((val >> 24) & 0xFF)
       password += chr((val >> 16) & 0xFF)
       password += chr((val >> 8) & 0xFF)
       password += chr(val & 0xFF)
   print(password)
   "

   Copied!

   What didn't work first

   Tried: Right-shift without masking, e.g. chr(val >> 24) for the first character.

   Without the 0xFF mask, any sign bit or high-order bits left over from the shift can bleed into the extracted value when the integer is negative or large. Masking with 0xFF after shifting ensures you only keep the lowest 8 bits, giving a clean byte in the range 0-255 suitable for chr().

   Tried: Extract bytes using little-endian order (shift 0, 8, 16, 24) instead of big-endian.

   Java packs the characters big-endian: the leftmost character sits in the most significant byte at shift 24. Reversing in little-endian order gives a password with the four characters inside each group reversed, which fails the checkPassword comparison. Match the order from the Java source: 24, 16, 8, 0.

   Learn more

   Masking with 0xFF (binary 11111111) after right-shifting isolates the lowest 8 bits of the result, discarding any higher bits. This is how you extract individual bytes from a multi-byte integer.

   This packing technique is common in low-level code for performance: processing 4 characters at once as a 32-bit word is faster than processing them individually on many architectures.

3. Step 3

   Submit the flag

   Observation

   I noticed the Python script produced a readable ASCII string by joining all extracted characters in order, which confirmed the unpacking was correct and that the string needed only to be wrapped in picoCTF{} to form the final flag.

   Concatenate all extracted characters in order to form the password. Wrap in picoCTF{...} to submit.

   Learn more

   When reversing bit manipulation, always check the shift amounts carefully in the original code. The order of bytes within the integer (big-endian vs little-endian) determines which byte to extract with which shift amount.

Flag

Reverse the 4-bytes-per-int packing scheme from VaultDoor7.java