Description
A new password authentication program that even shows you the password you entered in the database. Isn't that cool? Download system.out and recover what hash value it is actually checking.
Setup
Download system.out and make it executable.
Run it to understand what it prompts for.
chmod +x system.out./system.outSolution
Want to try it yourself first?
The guided walkthrough reveals hints one step at a time.
Step 1
Understand the program in GhidraObservationI noticed the challenge provides a native ELF binary (system.out) and asks to recover the hash value it checks against, which suggested decompiling it in Ghidra to reveal the internal hash algorithm, the obfuscation bytes, and the comparison logic in main().Load system.out into Ghidra and analyze it. Rename variables to make it readable. The program: (1) allocates a stack buffer of 90 bytes, (2) places obfuscation bytes XOR'd with 0xAA at offset 60 into the buffer, (3) asks for your username and reads up to 50 chars, (4) asks how many bytes in length your password is, (5) asks for your hash value as a number. It calls make_secret() which computes a custom hash and compares your number against it. The hash algorithm iterates over each byte: for each byte, new_total = (previous_total * 0x21) + byte_value. Hex 0x21 is decimal 33, making this a polynomial rolling hash.bashfile system.outbashghidra system.out &bash# In Ghidra: rename variables, look at main() and make_secret()/hash()Expected output
system.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
What didn't work first
Tried: Run 'strings system.out' to find the expected hash value or password directly.
strings only shows printable ASCII sequences embedded in the binary. The hash value is computed at runtime by the hash() function, not stored as a literal string. The obfuscation bytes XOR'd with 0xAA are also binary data, not printable text, so they will not appear in strings output. You must actually run or disassemble the binary to recover the hash.
Tried: Provide a password length larger than the actual input to try to leak the obfuscation bytes from the stack buffer.
The program does print extra bytes when the reported length exceeds actual input, but those raw bytes are the intermediate XOR'd data, not the final hash number the server compares against. The server checks the integer output of hash(), not individual buffer bytes. You still need to run make_secret()/hash() to get the value the comparison uses.
Learn more
In Ghidra, right-click on variable names like
local_padto rename them to something meaningful. You can also retype variables (e.g. changeundefined*tochar*) to make the decompiler output more readable.The key insight is that the program asks for "how many bytes in length is your password" - if you provide a length larger than your actual input, the program will try to print the extra bytes, potentially leaking the obfuscation bytes. But this is a distraction - what really matters is the hash comparison at the end.
Step 2
Recover the hash value by running system.out under GDBObservationI noticed that Ghidra revealed a deterministic hash() function driven entirely by bytes embedded in the binary, which suggested using GDB to break on hash() and read the return value from EAX directly rather than reimplementing the polynomial rolling hash in a separate script.The simplest approach: run system.out under GDB, break on the hash() function, type anything for the prompts, then use 'finish' to return from hash(). The return value in register EAX is the expected hash value. Submit that number to the remote server.bashgdb ./system.outbash(gdb) break hashbash(gdb) runbash# When prompted, type anything for usernamebash# For password length, type any numberbash# The breakpoint fires inside hash()bash(gdb) finishbash# After finish, EAX contains the return value (the expected hash)bash(gdb) print $eaxbash# Note down the hash valuebash(gdb) quitbash# Now connect to the remote instance and submit that hash valuebashnc <HOST> <PORT_FROM_INSTANCE>What didn't work first
Tried: Break on main() instead of hash() and step through the entire program to find the hash.
Breaking on main() and stepping instruction by instruction is extremely tedious because the program executes hundreds of setup instructions before reaching hash(). Breaking directly on 'hash' (or 'make_secret' if Ghidra revealed that name) jumps straight to the relevant function. Use 'break hash' then 'run', answer the prompts, and let GDB stop inside the hash function automatically.
Tried: Read the hash from $rax (64-bit) instead of $eax (32-bit) after 'finish'.
The hash() function returns a 32-bit integer, so only the lower 32 bits of $rax are meaningful. Reading $rax may show a larger number if the upper 32 bits contain leftover register garbage. Using 'print $eax' or 'print (int)$rax' ensures you read only the correct 32-bit return value that the comparison in main() actually uses.
Learn more
The
finishcommand in GDB runs until the current function returns, then pauses. The hash fits in 32 bits, so the return value is in theEAXregister. Useprint $eaxto read it. This technique lets you use the program itself as a hash oracle - rather than reimplementing the hash algorithm, you let the binary compute it and just read the result.This works because the hash is deterministic: the obfuscation bytes embedded in the binary produce the same hash value every time. The exact value does not change between runs, so you can compute it locally with GDB and submit it to the remote server.
Step 3
Submit the hash to the remote serverObservationI noticed the hash depends only on obfuscation bytes baked into the binary and not on user input, which meant the value extracted locally from GDB would match what the remote server expects, so submitting it via netcat should yield the flag.Connect to the challenge server with netcat. Enter any username, enter the password length (e.g. 1), then submit the hash value you obtained from GDB. The server will verify it and print the flag.bashnc <HOST> <PORT_FROM_INSTANCE>bash# Enter username: anythingbash# Enter password length: 1bash# Enter hash: <value from GDB EAX>Learn more
The remote instance uses the same binary with the same embedded obfuscation bytes. Because the hash depends only on those embedded bytes (not on the username or anything you provide), the hash value you computed locally is the same one the server expects.
Interactive tools
- Strings ExtractorPull printable text from any binary, library, or image. ASCII and UTF-16 detection, configurable minimum length, flag-like highlight, no command line needed.
- Hex ViewerView text or raw hex bytes as a xxd-style hex dump with byte offset, hex columns, and ASCII sidebar. Highlights printable characters and null bytes.
- File Magic IdentifierIdentify file types from magic numbers. Paste hex bytes or drop a file to detect PNG, JPEG, ZIP, PDF, ELF, PCAP, SQLite, and dozens of other formats.
Flag
Reveal flag
picoCTF{s3cur3_p4ssw0rd_db_...}
Run system.out under GDB, break on hash(), type anything for the prompts, then 'finish' to see EAX with the expected hash value (a 32-bit integer). Submit that number to the remote server to get the flag.