Java Reverse Engineering for CTF: Decompiling JARs and the Vault Door Series

You have a .jar and no source. Here is the fast path

A challenge handed you a .jar or a single .class file and asked for a flag. Java is not a black box. It compiles to bytecode that keeps method names, field names, strings, and structure, which means a decompiler turns that file back into readable Java in seconds. The whole task is usually: decompile, read the check that validates the password, and run it backwards.

The one-liner answer, if you only read this paragraph:

# a .jar is a zip; pull the class out, then decompile it
unzip -o VaultDoor.jar -d extracted/
java -jar cfr.jar extracted/VaultDoor8.class > VaultDoor8.java
# now read VaultDoor8.java and reverse whatever checkPassword() does

That is the entire pipeline for the picoCTF Vault Door series and almost every other Java reversing challenge you will meet at this level. The rest of this post explains what each piece is, which decompiler to reach for when one produces garbage, and how the Vault Door obfuscation works so you can reverse it by hand. If you have never opened a decompiler, start at the top. If you already have readable Java and just need to undo a scrambled char array, jump to the char-array section.

What is a .class file and a .jar, really?

When you compile VaultDoor8.java with javac, you do not get machine code. You get VaultDoor8.class, a file of bytecode: a compact, stack-based instruction set that the Java Virtual Machine (JVM) interprets at runtime. The format is fully specified in the JVM specification, chapter 4. Every .class file starts with the magic bytes 0xCAFEBABE, which is the easiest way to confirm a mystery file is JVM bytecode.

The reason Java is so easy to reverse is the constant pool. A .class file stores method names, field names, the types of every variable as descriptors, and every string literal in plain text inside that pool. Bytecode is not stripped the way an optimized C binary is. The instructions map almost one-to-one back to the source structure, so a decompiler can rebuild loops, branches, and expressions with high fidelity.

A .class file is not a wall. It is your source code with the variable names mostly intact and the curly braces removed. The decompiler just puts the braces back.

A .jar (Java ARchive) is simpler than it looks: it is a ZIP file with a known layout. Inside you will find the compiled .class files in package directories, a META-INF/MANIFEST.MF that names the entry-point class, and sometimes bundled resources. Because it is just a ZIP, every tool you already have works on it:

$ file VaultDoor.jar
VaultDoor.jar: Java archive data (JAR)
$ unzip -l VaultDoor.jar
  Length      Name
---------  ----------------------
      823  META-INF/MANIFEST.MF
     1442  VaultDoor8.class
$ unzip -p VaultDoor.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Main-Class: VaultDoor8        # this is the class that runs first

Which decompiler do I use: CFR, procyon, jd-gui, or javap?

There is no single best tool. Decompilers disagree, especially on obfuscated input, so the working move is to keep two or three on hand and switch when one produces nonsense. Here is what each is for.

CFR is a single JAR you run from the command line, and it is the one to start with. When CFR emits something that does not type-check or invents a label-goto soup, run the same class through procyon and compare. They use different control-flow reconstruction, so an obfuscation that defeats one often reads cleanly in the other. jd-gui is the graphical option when you would rather click than type.

How do I actually run these on a file?

Three of the four are one-liners. Start by getting the class out of the JAR, or point the tool straight at the JAR if it accepts one.

CFR decompiles a single class or an entire JAR. Send the output to a file so you can search it:

# one class to stdout, redirected to a file
java -jar cfr.jar VaultDoor8.class > VaultDoor8.java
# a whole jar at once, written into ./out/ keeping package dirs
java -jar cfr.jar VaultDoor.jar --outputdir out/
# then just read the file your editor likes
less out/VaultDoor8.java

procyon has the same shape. If CFR choked on a method, this is your second opinion:

java -jar procyon-decompiler.jar VaultDoor8.class > VaultDoor8.procyon.java
# diff the two decompilers to spot where they disagree
diff VaultDoor8.java VaultDoor8.procyon.java

javap is already on your machine if you have a JDK. The -c flag disassembles method bodies; -p includes private members; -v dumps the constant pool too, which is where every string literal lives:

# disassemble every method, including private ones
javap -c -p VaultDoor8.class
# verbose: constant pool, flags, full descriptors
javap -v VaultDoor8.class | less
# grep the constant pool straight for the flag prefix
javap -v VaultDoor8.class | grep -i 'picoCTF'

That last line is worth remembering. Sometimes the flag, or a decisive substring of it, is sitting in the constant pool as a plain string and you never need to understand the logic at all. Always grep for the flag format before you start reading code. A worked example of exactly that shortcut is picoCTF 2022 Safe Opener, where the comparison string is a literal you can read directly out of the source.

Why does decompiled Java look weird, and what do I trust?

Decompiled Java is faithful to the bytecode, not to whatever the author originally typed. The logic is identical, but the surface is rougher. Knowing the common artifacts keeps you from chasing ghosts.

• Synthetic variable names. Locals often come back as n, n2, string, array, or var3. The bytecode kept the types but not always the original local names, so the decompiler invents readable ones. The names mean nothing; the operations mean everything.
• Expanded loops and conditionals. A clean for loop may reappear as a while with a manual counter, or a ternary may become an if/else. Same behavior, different shape.
• Char and int blur together. In the JVM a char is just a 16-bit unsigned integer. Decompilers freely show character constants as their numeric code (97) or compare a char against an int. When you reverse a check you will be doing exactly this arithmetic, so get comfortable reading 'a' and 97 as the same value.
• Occasional goto labels. When control flow is obfuscated, a decompiler that cannot rebuild structured code falls back to labeled break and continue, or even raw labels. That is a signal to try the other decompiler, or to drop to javap -c for the truth.

The discipline that saves you: trust the bytecode over the decompiled source. A decompiler is software with bugs, and an author can craft input that makes it lie. When your reconstructed flag is wrong but the logic looked right, disassemble the method with javap -c and walk the instructions. The stack-machine model is mechanical: iload pushes a local, iconst pushes a constant, ixor pops two and pushes their XOR. It is slower to read but it cannot deceive you.

What is the Vault Door obfuscation trick?

The picoCTF Vault Door series is the canonical Java reversing cluster: eight parts that ramp from trivial to genuinely annoying, all built on the same skeleton. Every Vault Door program reads a password, passes it to a method, and prints success only if that method returns true. The flag is your input wrapped as picoCTF{...}, so the entire game is making the check return true.

The series teaches one idea by escalating it: the password is compared indirectly. Each part adds a transformation between your input and the comparison, and your job is to invert that transformation. Read bottom to top, this is the progression:

The recurring shape, the one you will see in almost every part, is a character array of expected values compared one slot at a time against your transformed input:

public boolean checkPassword(String password) {
    if (password.length() != 32) {
        return false;
    }
    char[] expected = { 'r', '4', 'd', '1', 'x', ... };
    char[] in = password.toCharArray();
    for (int i = 0; i < expected.length; i++) {
        if (in[i] != expected[i]) {     // sometimes after a transform
            return false;
        }
    }
    return true;
}

Vault Door Training and Vault Door 1 exist to teach you the skeleton: open the class, find checkPassword, read the literal. By Vault Door 3 the characters are scattered into the array by a computed index pattern, so you rebuild the string by following the loop's arithmetic. Vault Door 4 through 7 layer encodings and bitwise math on top, and Vault Door 8 combines ugly formatting with a per-character bit scramble, which is the worked example below.

How do I reverse a scrambled char-array check by hand?

This is the heart of the series. Vault Door 8 does not compare your password to an array directly. It first runs each character through a scramble function that swaps bits around, then compares the scrambled result to an expected array. So the relationship is scramble(yourChar) == expected[i], and to recover the password you have to compute unscramble(expected[i]) for every slot.

The decompiled scramble looks roughly like this. A switchBits helper exchanges two bit positions in a byte, and the scramble calls it eight times per character in a fixed order:

public byte switchBits(byte b, int p1, int p2) {
    byte mask1 = (byte)((b >> p1) & 1);
    byte mask2 = (byte)((b >> p2) & 1);
    byte mask3 = (byte)((mask1 ^ mask2) << p1 | (mask1 ^ mask2) << p2);
    return (byte)(b ^ mask3);   // swaps bit p1 and bit p2 when they differ
}
public char[] scramble(char[] arr) {
    for (int i = 0; i < arr.length; i++) {
        byte c = (byte) arr[i];
        c = switchBits(c, 1, 2);
        c = switchBits(c, 0, 3);
        c = switchBits(c, 5, 6);
        c = switchBits(c, 4, 7);
        c = switchBits(c, 0, 1);
        c = switchBits(c, 3, 4);
        c = switchBits(c, 2, 5);
        c = switchBits(c, 6, 7);
        arr[i] = (char) c;
    }
    return arr;
}

Here is the lever that makes this easy. switchBits is an involution: swapping two bits and then swapping the same two bits again returns the original byte. The scramble is a sequence of swaps, so its inverse is the exact same swaps applied in reverse order. You do not need to write an unscramble function at all. You feed the expected array through a scramble whose switchBits calls are listed bottom to top.

Lift the expected array straight out of the decompiled source, paste the original switchBits verbatim, and run the swaps in reverse:

public class Solve {
    // paste switchBits() unchanged from the challenge
    static byte switchBits(byte b, int p1, int p2) {
        byte m1 = (byte)((b >> p1) & 1);
        byte m2 = (byte)((b >> p2) & 1);
        byte m3 = (byte)((m1 ^ m2) << p1 | (m1 ^ m2) << p2);
        return (byte)(b ^ m3);
    }
    public static void main(String[] a) {
        // the 'expected' array copied out of VaultDoor8.class
        char[] exp = { /* the scrambled target chars go here */ };
        for (int i = 0; i < exp.length; i++) {
            byte c = (byte) exp[i];
            // SAME eight swaps, REVERSED order
            c = switchBits(c, 6, 7);
            c = switchBits(c, 2, 5);
            c = switchBits(c, 3, 4);
            c = switchBits(c, 0, 1);
            c = switchBits(c, 4, 7);
            c = switchBits(c, 5, 6);
            c = switchBits(c, 0, 3);
            c = switchBits(c, 1, 2);
            System.out.print((char) c);
        }
        System.out.println();
    }
}

$ javac Solve.java && java Solve
...the recovered password prints here, which is the flag body

The general recipe behind that one example: identify the comparison, identify every transform applied to your input before the comparison, and invert the transforms in reverse order applied to the expected values. For a string-operations variant, the transform might be a StringBuilder.reverse(), a substring shuffle, or a Base64 decode. Same plan, different inverse. The picoCTF 2023 Safe Opener 2 challenge is a clean drill on the string-and-encoding flavor of this exact idea.

Should I read the check statically or patch and run it?

Two strategies recover a flag, and choosing the right one saves time. Reading statically means understanding the check and computing the answer, like the reconstruction above. Patch-and-run means editing the program so it leaks the answer or accepts anything, then executing it.

Read statically when the transform is cleanly invertible and short. The Vault Door reconstructions are this case: a few swaps or one XOR, computed in seconds, no risk of running untrusted code. This is the default for picoCTF-level Java.

Patch and run when the logic is a tangled mess you would rather not fully understand, or when the program itself prints or constructs the flag once a check passes. The trick: you have the decompiled source, so just edit it and recompile, or add a print statement that dumps the expected value:

# 1) decompile to source you can edit
java -jar cfr.jar VaultDoor.jar --outputdir src/
# 2) edit the check: print the expected array, or force success
#    e.g. add  System.out.println(new String(expected));
#    or change  if (in[i] != expected[i])  to never trigger
# 3) recompile just that class and run it
javac src/VaultDoor8.java -d patched/
java -cp patched VaultDoor8

A lighter-weight version that needs no recompile: when the password is built by string operations, paste only the construction code into a tiny scratch main and print the result, exactly as the char-array solver above does. You are not running the challenge, you are running the one method that matters.

picoCTF challenges to practice on

Work the Vault Door series in order. Each part isolates one transformation, and the difficulty curve is deliberate. The skeleton never changes, so by part 8 the only new skill is patience with the bit math.

• Vault Door Training and Vault Door 1: learn the skeleton. Decompile, find checkPassword, read the literal. This is the muscle memory everything else builds on.
• Vault Door 3: the characters are placed into the expected array by a computed index pattern. Follow the loop arithmetic to put them back in order.
• Vault Door 8: the bit-swap scramble worked through above, wrapped in intentionally horrible formatting. Reformat it first, then reverse the swaps.
• Safe Opener and Safe Opener 2: the same Java reversing idea in a different wrapper. The first hides the answer as a literal you grep out; the second mixes string operations and encoding you invert.

Vault Door 4 through 7 are not linked here but follow the same approach: identify the encoding or bitwise transform, invert it, and reassemble. Once you have done Training through 3 and 8, the middle parts are practice reps rather than new concepts.

For the adjacent skills these challenges lean on, the Python reversing post covers the same reverse-the-transform mindset in a scripting language, the Android APK guide extends JAR decompilation to mobile (an APK is largely the same Java pipeline plus Dalvik), the Ghidra guide is where you go when the target is a native binary instead of bytecode, and the Linux CLI post drills the unzip, file, and grep reflexes the whole pipeline depends on.

Quick reference

unzip -o app.jar -d extracted/          # a jar is a zip
unzip -p app.jar META-INF/MANIFEST.MF   # find the Main-Class
xxd app.class | head -1                 # confirm cafe babe magic
java -jar cfr.jar app.jar --outputdir out/   # decompile whole jar (CFR)
java -jar procyon-decompiler.jar X.class      # second opinion
javap -c -p X.class                           # bytecode disassembly
javap -v X.class | grep -i picoCTF            # grep the constant pool
javac Solve.java && java Solve          # run your reconstruction

A .class file carries its own answer key, because a program that checks a password has to hold what it is checking against. Decompile it, read the check, invert the shuffle, and the vault opens.