Integer Overflow and Signedness Bugs for CTF

Introduction

Ask a beginner how to get past a check like if (cost > balance) reject(); and they will look for a way to change the balance. The interesting answer is to leave the balance alone and make the cost negative. Not by typing a minus sign, which the input validation catches, but by asking for a quantity so large that multiplying it by the price overflows the integer type and lands on a negative number. The check then passes honestly: a negative cost really is less than your balance. Subtracting it makes you richer.

That is the entire subject. Integers in C, in Java, in Solidity, and in the CPU itself are not the integers of mathematics. They are fixed-width bit patterns with hard edges, and arithmetic that crosses an edge does not error out or grow: it wraps, silently, and the result is a perfectly valid number that happens to be wrong. Every bug in this guide comes from code that was written as if the mathematical integers were what it had.

An integer bug is not a crash. It is a program computing the wrong answer with complete confidence and continuing to trust it.

This matters beyond the trick, because integer bugs are usually the first link in a chain rather than the payoff. An overflowed length becomes an undersized buffer, which becomes a heap overflow. A truncated size becomes a short allocation with a long copy. The memory corruption guides on this site (buffer overflow and heap exploitation) mostly begin at the point where a length is already wrong. This one is about how it got that way.

How the wrap works

A 32-bit signed integer holds 2³²distinct bit patterns. Two's complement assigns them to the range from -2147483648 to 2147483647 by treating the top bit as a sign. Add one to the largest value and the bit pattern rolls over into the one that means the most negative value. Nothing is detected; the carry out of the top bit is simply discarded.

  2147483647   =  0111 1111 1111 1111 1111 1111 1111 1111
+          1
  ----------
 -2147483648   =  1000 0000 0000 0000 0000 0000 0000 0000

Four boundaries are worth committing to memory, because they are the values you type:

The four bug shapes

Every integer bug you will meet in a CTF fits one of these four. Learning to name the shape from the source is most of the solve, because each shape has exactly one input that triggers it.

Shape 1: wrap-around arithmetic

The canonical form. The program computes something from your input, then validates the result. The validation is correct; the computation is not.

int total = price * quantity;      // 32-bit multiply, wraps silently
if (total > balance) {
    puts("Not enough coins");
    return;
}
balance -= total;                  // subtracting a negative ADDS

Pick a quantity whose product with the price exceeds INT_MAX. The product wraps negative, the check passes, and the subtraction credits your account. Solving for the quantity is one line:

PRICE = 900
for qty in range(1, 10_000_000):
    total = (PRICE * qty) & 0xFFFFFFFF
    if total >= 2**31:                    # interpret as signed
        total -= 2**32
    if total < 0:
        print(qty, total)
        break

Flag Shop is this exactly: you start with 1100 coins, the real flag costs far more, and a multiplication on the purchase quantity wraps to a negative cost that the shop then subtracts. Two Sum strips the scenario away and asks for the mechanism directly: find two positive integers where n1 > n1 + n2. In mathematics no such pair exists. In 32-bit C, any pair whose sum crosses INT_MAX works.

Shape 2: signedness confusion

The most dangerous shape, because it is the one that turns into memory corruption. It happens when the same bit pattern is read as signed in one place and unsigned in another.

char buf[64];
int len;
 
scanf("%d", &len);
if (len > 64) {                 // signed comparison: -1 > 64 is FALSE, so this passes
    puts("Too long");
    return;
}
read(0, buf, len);              // read() takes size_t: -1 becomes 18446744073709551615

Send -1. The bounds check sees a small number and waves it through. The copy sees an enormous one and reads until the stack is destroyed. The check and the use disagree about what the bits mean, and nothing in C forces them to agree.

Gatekeeper is a gentler version of the same type confusion, worth doing because it makes the idea concrete without memory corruption. It demands a three-character input whose numeric value exceeds 999. No decimal number satisfies both constraints. The resolution is that the conversion is strtol with base 16 hardcoded, so 3e8 is three characters wide and 1000 in value. The string length and the numeric value are two different readings of the same input, and the program never reconciles them.

Shape 3: truncation

Assigning a wide integer into a narrow one keeps the low bits and discards the rest. If a length is validated as a 32-bit value and then stored in a 16-bit field, everything above 65535 comes back around to a small number.

unsigned int  requested = get_user_length();   // validated: must be < 100000
unsigned short size     = requested;           // truncated to 16 bits
char *buf = malloc(size);                      // 65536 + 10 allocates 10 bytes
memcpy(buf, src, requested);                   // copies 65546. Heap overflow.

The signature of a truncation bug is two variables holding the same quantity at different widths: one used for the check, the other for the allocation or the copy. Whenever you see a size handled by more than one type, compute what happens at each type boundary.

Related and easy to confuse with truncation: byte order. If the value survives intact but arrives scrambled, you are looking at endianness rather than a size bug. Endianness and Endianness-v2 drill the distinction, which matters as soon as you start packing exploit payloads. See hex dumps for reading multi-byte values out of a dump correctly.

Shape 4: no lower bound

Not strictly an overflow, but it lives in the same family and it shows up more often than any of the others in web and business-logic challenges: the code checks that a value is not too large and never considers that it might be negative.

if (index < item_count) {
    return items[index];      // index = -1 reads BEFORE the array
}
 
// Or, in application logic:
// total = price * quantity, with quantity = -5, credits the account

Shop is the purest example: you hold 40 coins, the flag costs 100, and the shop lets you sell a negative quantity. Nothing overflows at all. The application validated the type (is this a number?) and never the range (is it positive?), and the arithmetic did the rest.

Finding them

With source, grep the arithmetic that touches user input. Without source, read the disassembly for the comparison and the width of the registers involved.

# Source review: arithmetic on input, and narrowing assignments
grep -nE '\*|\+|<<' src/*.c | grep -iE 'len|size|count|qty|price|index'
grep -nE '(short|char|int)[ \t]+[a-z_]+ *= *[a-z_]+;' src/*.c
 
# Binary: signed vs unsigned jumps guarding a size
objdump -d target | grep -E 'jg|jl|jge|jle' | head -40   # signed
objdump -d target | grep -E 'ja|jb|jae|jbe' | head -40    # unsigned
 
# Compile the source with a sanitizer and let it tell you
gcc   -fsanitize=undefined -g chal.c -o chal && ./chal   # signed overflow (it is UB)
clang -fsanitize=integer   -g chal.c -o chal && ./chal   # also unsigned wrap and truncation

Two register-level tells are worth internalising. Movement between register widths is truncation or extension made visible: mov eax, ecx zeroes the upper 32 bits of rax, movsx sign-extends (so 0xFF becomes -1), and movzx zero-extends (so 0xFF becomes 255). Seeing movsx on a length is a strong hint. If reading this level of detail is not yet comfortable, start with x86-64 assembly for CTF.

When the constraint is a system of arithmetic conditions rather than a single boundary, stop solving by hand and hand it to a solver. Z3 models fixed-width machine integers natively with its bitvector theory, which means it wraps exactly the way the target does:

from z3 import BitVec, Solver, sat
 
n1, n2 = BitVec('n1', 32), BitVec('n2', 32)
s = Solver()
s.add(n1 > 0, n2 > 0)          # both positive, as the challenge demands
s.add(n1 + n2 < n1)            # yet their sum is smaller. Only true under wrapping
print(s.model() if s.check() == sat else 'no solution')

Beyond C: web, Python, Solidity

The rules change by language, and the changes decide whether the bug exists at all.

The Solidity row deserves emphasis because it is where this bug class did the most real-world damage. Before version 0.8.0 all arithmetic wrapped silently with no revert, and a long series of token contracts were drained by exactly the code in shape 1. Smart Overflow reproduces it: a deposit function does unchecked uint256 arithmetic, and a deposit of 2²⁵⁶ - 1 wraps the tracked balance into something the contract treats as a fortune. Smart contract bugs covers the surrounding context.

The Python row is why so many CTF services written in Python are not vulnerable to this at all: 2**64 is just a number there. When a Python challenge does have an integer bug, look for the seam where it packs into fixed width, which is almost always struct.pack or an explicit & 0xFFFFFFFF. Shop is a Python target and its bug is shape 4, the missing lower bound, precisely because shape 1 was unavailable.

And the payoff case: when an integer bug produces a wrong size rather than a wrong price, it becomes a memory corruption primitive. Pizza Router chains an out-of-bounds access into heap exploitation, which is the usual endgame for shapes 2 and 3. Pick that thread up in heap exploitation.

picoCTF challenges

Quick reference

# The values to try first, in order
-1              # unsigned max, and the classic signedness probe
0               # division, modulo, and empty-allocation paths
2147483647      # INT_MAX
2147483648      # INT_MAX + 1: wraps to INT_MIN
4294967295      # UINT_MAX
65536           # 16-bit truncation boundary
256             # 8-bit truncation boundary
 
# Simulate the target's arithmetic in Python
python3 -c "v=(900*2500000)&0xFFFFFFFF;print(v-2**32 if v>=2**31 else v)"
 
# Signed vs unsigned jumps in a disassembly
#   jg jl jge jle  -> signed
#   ja jb jae jbe  -> unsigned
 
# Let the compiler find it
gcc   -fsanitize=undefined -g chal.c -o chal   # signed overflow only
clang -fsanitize=integer   -g chal.c -o chal   # unsigned wrap and truncation too

Where this leads: buffer overflow and heap exploitation for what a wrong length becomes, Z3 for solving the constraints automatically, smart contract bugs for the Solidity side, and the pwn roadmap for where this sits in the larger path.

Sources and further reading

The specifications that decide whether a given wrap is defined, undefined, or impossible.

• CWE-190 and CWE-195 for the canonical descriptions of wraparound and signed-to-unsigned conversion errors.
• The C11 draft standard (N1570) for the asymmetry that matters most here: unsigned arithmetic is defined to wrap modulo 2 to the N, while signed overflow is undefined behaviour a compiler may optimise around.
• UndefinedBehaviorSanitizer for finding these automatically when you have the source.
• Solidity 0.8.0 breaking changes for the switch to checked arithmetic and the unchecked escape hatch, and strtol(3) for the base handling behind the type-confusion example.