YaraRules0x100 picoCTF 2025 Solution

Description

1. Step 1Collect indicators
   Run strings before and after unpacking. Before: you see "UPX0", "UPX1", and the UPX stub's loader text. After: the original strings appear, including "YaraRules0x100", "NtQuery..." Windows API names, and the phrase "debugger process". These plus the MZ magic become your rule's strings.
   bash

   # Packed (before upx -d): only UPX section names and stub text are visible.
   $ strings suspicious.exe.bak | grep -E 'UPX|Yara|NtQuery|debugger'
   UPX0
   UPX1
   UPX!
   $Info: This file is packed with the UPX executable packer http://upx.sf.net $
   
   # Unpacked (after upx -d): the original program's strings appear.
   $ strings suspicious.exe | grep -E 'UPX|Yara|NtQuery|debugger'
   YaraRules0x100
   NtQueryInformationProcess
   IsDebuggerPresent
   debugger process detected
   
   # Diff the two for everything UPX hid:
   diff <(strings ./suspicious.exe.bak) <(strings ./suspicious.exe) | head

   Copied!
   Learn more

   YARA is an open-source pattern-matching tool originally developed at VirusTotal and now maintained by the security community. It allows malware analysts to write rules that describe file characteristics (byte sequences, string patterns, and logical conditions) and apply those rules to scan files, memory dumps, or network streams. YARA rules are the lingua franca of malware detection and are used in antivirus engines, SIEM platforms, EDR products, and threat hunting tools.

   The strings utility extracts printable character sequences (default minimum length: 4) from any binary file. It is one of the fastest first-pass analysis tools because it reveals hardcoded URLs, registry keys, API calls, error messages, and embedded data without requiring disassembly. Running it before and after unpacking a UPX-compressed executable reveals two different views: the compressed binary shows minimal strings (mostly the UPX stub), while the unpacked binary exposes the original executable's full string table.

   UPX (Ultimate Packer for eXecutables) is a legitimate compression tool for reducing binary size, but it is also extremely popular among malware authors for evading signature-based detection. Unpacking with upx -d recovers the original binary. Some malware modifies the UPX headers to prevent automatic unpacking; in those cases, dynamic unpacking (running the sample in a sandbox until it decompresses itself into memory) or manual analysis with a debugger is required. The hex dumps for CTF guide covers reading PE headers in xxd.

2. Step 2Write the YARA rule
   Combine the indicators into a rule that looks for the MZ header plus YaraRules0x100 and either UPX or NtQuery, or alternatively the entire "debugger process" string. The two-clause or is deliberate; explained below.
   bash

   rule Rule {
       strings:
           $mz = {4D 5A}
           $name = "YaraRules0x100"
           $packer = "UPX"
           $ntquery = "NtQuery"
           $phrase = "debugger process" wide ascii
       condition:
           ($mz and $name and ($packer or $ntquery)) or $phrase
   }

   Copied!
   Learn more

   A YARA rule consists of three sections: meta (optional metadata like author and date), strings (pattern definitions), and condition (boolean logic combining the patterns). String patterns can be plaintext ("text"), regular expressions (/regex/), or hex byte sequences ({4D 5A}). The wide modifier matches UTF-16LE (two bytes per character, common in Windows strings), and ascii matches standard single-byte encoding; using both together covers both formats.

   The MZ magic bytes (0x4D 0x5A) mark the start of every Windows PE (Portable Executable) file: DLLs, EXEs, and SYS files all begin with MZ. Using it as an anchor in the condition ($mz at 0 for strict offset matching, or just $mz for presence anywhere) helps limit false positives to PE files. Combining it with application-specific strings like YaraRules0x100 makes the rule highly specific to this particular sample family.

   Good YARA rule design balances specificity (low false positive rate) with generality (catching all variants of a malware family). Production threat intelligence teams write rules around behavioral indicators (specific API call sequences, anti-analysis techniques like NtQuery for debugger detection) that persist across recompilations, rather than exact byte sequences that change with each build. Resources like YARA documentation, yarGen (automated rule generator), and CAPE sandbox help accelerate rule development from live samples.

   Why the compound condition ($mz and $name and ($packer or $ntquery)) or $phrase? It gives the rule two independent paths to a hit, which is what makes it robust against the harness's sample variations. The first clause matches a PE file that contains both the unique application string YaraRules0x100 and at least one of the anti-analysis indicators (UPX section names if still packed, NtQuery if unpacked). The second clause, $phrase alone, catches any file containing the literal "debugger process" regardless of structure, which fires on samples where the harness stripped the MZ header or removed the application name. Two paths means a sample only has to satisfy one of them to match, dramatically reducing false negatives compared with a single conjunctive condition.

3. Step 3Submit via socat
   Save the rule to sample.txt (or any filename) and pipe it to the grading service with socat. If it matches all test cases, the server returns the flag.
   bash

   socat -t60 - TCP:standard-pizzas.picoctf.net:59919 < sample.txt

   Copied!
   Learn more

   socat (SOcket CAT) is a multipurpose relay tool that creates bidirectional data channels between various types of endpoints: TCP sockets, UDP, files, stdin/stdout, Unix domain sockets, SSL connections, and more. It is the swiss-army knife of network connectivity, frequently used in CTF challenges for connecting to remote services, setting up listeners, and relaying data between protocols. See networking tools for CTF for the full kit.

   The -t60 flag sets a 60-second inactivity timeout, useful here because the grader runs your rule against a sample corpus and complex YARA conditions (especially ones with regex strings or many wide ascii modifiers) can take several seconds per file. Without the timeout, socat's default closes the connection long before the server finishes scoring and you never see the flag. If 60 seconds isn't enough (your rule is heavy, the server is busy, or both), bump it to -t120 and resubmit; the grader doesn't penalize a longer wait. The - endpoint means stdin/stdout, so redirecting a file with < sample.txt sends the file contents as input to the TCP connection.

   In real threat hunting workflows, YARA rules are deployed to scanning infrastructure using tools like YARA-X (the Rust rewrite of YARA), integrated into SIEMs via Velociraptor or THOR for endpoint scanning, or uploaded to sandboxes like VirusTotal and MalwareBazaar to monitor for new samples matching the rule. Writing effective YARA rules is a core competency for malware analysts and threat intelligence teams.

Flag

Any rule that nails at least one unique string plus the PE header works; the combination above passed all server tests.