by: Hacking

Reverse Engineering Binaries is a critical set of techniques that enable attackers to extract sensitive information from, or inject code into, both local and remote executables.

Binary Exploitation Techniques are popular in CTF (Capture The Flag) competitions and less common in bug bounty programs, but they can be applied to enable more complex attack chains.

Mastering Reverse Engineering is crucial for cyber defense, uncovering vulnerabilities, and developing more secure software systems.

MEMORY LAYOUT OVERVIEW

Memory

  • Stack
    • Contains local variables, function parameters, and control flow information.
    • Grows from higher to lower memory addresses.
  • Heap
    • Dynamically allocated memory (e.g. malloc).
    • Grows from lower to higher memory addresses.
  • Data
    • Static and global variables.
  • Code
    • Executable instructions of the program.

Concepts

  • Endianness
    • Big-Endian
      • The most significant byte (MSB) is stored first.
    • Little-Endian
      • The least significant byte (LSB) is stored first.

Functions

  • Prologues
    • Prepare the function for execution, including setting up the stack frame.
      • Shortcut: enter
  • Epilogues
    • Clean up the stack and return from the function.
      • Shortcut: leave
  • Canary (aka Stack Cookie/Guard)
    • Detects stack buffer overflows by checking whether the canary value has been altered, helping prevent the return address from being overwritten and control flow from being hijacked.
  • ROP (Return Oriented Programming)
    • Exploits vulnerabilities by chaining together small snippets of existing code called Gadgets.
  • Gadgets
    • Small sequences of instructions ending in a ret, used to perform arbitrary operations.
  • GOT (Global Offset Table)
    • A runtime address resolution table for global variables and functions in a shared library.
  • Lazy Binding
    • Resolves function addresses on their first call using the PLT, rather than at program startup.
  • PLT (Procedure Linkage Table)
    • Facilitates dynamic linking by redirecting function calls to the GOT with the help of the Dynamic Linker/Loader.
  • ASLR (Address Space Layout Randomization)
    • Protects against buffer overflow attacks by randomizing the memory addresses used by system and program components.
  • Movaps
    • This x86 instruction requires operands to be 16-byte aligned. During exploitation, misalignment will raise an exception and potentially crash the program, making it a common pitfall.
    • The workaround is to add an extra ret right before the function call.

32-bit vs. 64-bit Architecture (x86)

  • 32-bit
    • Parameter Passing: Via stack.
    • Calling Convention: Arguments are pushed onto the stack in reverse order (right to left).
    • System Calls:
      • Instruction: interrupt 128 (int 0x80).
      • Arguments: register eax for the system call number (0x0b = execve), followed by ebx, ecx, edx, esi, edi, and ebp.
    • Address Space: Uses the entire memory address space.
    • Words:
      • word (16 bits)
      • dword (32 bits)
  • 64-bit
    • Parameter Passing: Via registers (RDI, RSI, RDX, RCX, R8, R9), then the stack.
    • Calling Convention: Register order as listed above, followed by stack.
    • System Calls:
      • Instruction: syscall
      • Arguments: register rax for the system call number (0x3b = execve), followed by rdi, rsi, rdx, r10, r8, r9.
    • Address Space:
      • Low canonical addresses: 0x0000000000000000 to 0x00007FFFFFFFFFFF (User space)
      • High canonical addresses: 0xFFFF800000000000 to 0xFFFFFFFFFFFFFFFF (Kernel space)
      • Only 48 bits have meaningful virtual addresses.
    • Words:
      • word (16 bits)
      • dword (32 bits)
      • qword (64 bits)

BASIC ASSEMBLY

  • Registers and their Operands
    • eip (32-bit) / rip (64-bit)
      • Points to the next instruction to be executed.
    • ebp (32-bit) / rbp (64-bit)
      • Points to the base of the current stack frame, used for referencing local variables. While RBP does not hold the return address itself, it often points to the stack position where it is stored.
    • esp (32-bit) / rsp (64-bit)
      • Points to the top of the stack (lowest memory address). Works in conjunction with push and pop.
  • Constructs
    • mov
      • Copies data from source to destination. The pointer is dereferenced (data value).
    • lea
      • Loads the address of the source into the destination. Stores the pointer value itself, not the dereferenced data.
    • ret
      • Pops the return address from the top of the stack into the instruction pointer, resuming execution there.
    • push
      • Pushes a value onto the stack, decrementing the stack pointer.
    • pop
      • Pops a value from the stack, incrementing the stack pointer.
    • call
      • Calls a function by pushing the return address onto the stack and jumping to the function’s address.
    • cmp
      • Works like an IF statement. Compares two values by subtracting the right operand from the left.
  • Operators
    • or, and, and xor
      • Perform bitwise OR, AND, and XOR operations respectively.
    • mul
      • Multiplies two operands.
    • div
      • Divides one operand by another.
    • shl and shr
      • Shift bits left and right respectively.
    • rol and ror
      • Rotate bits left and right respectively.
    • nop
      • No Operation. Does nothing; often used for timing or alignment purposes.

INFORMATION GATHERING

file fileName
  • ELF
    • Linux executable.
  • 80386/i386 or x86-64
    • 32-bit or 64-bit architecture respectively.
  • dynamically linked
    • Uses system libraries; the libraries are not bundled into the binary.
  • not stripped
    • Easier to analyze because debugging information, including function names, is preserved in the binary.
  • LSB
    • Least Significant Bit = Little-endian (bytes appear in reverse order).

See also a list of file signatures that the file command uses to identify file types at [Link].

checksec --file=fileName
  • RELRO
    • “Relocation Read-Only” has the following levels:
      • “Full RELRO” – GOT and PLT entries are resolved before startup and marked read-only, preventing overwrites.
      • “Partial RELRO” – Some entries remain read+write so they can be resolved at runtime, allowing partial overwrites.
      • “No RELRO” – No protection. Uncommon on modern systems.
  • Stack Canary
    • When disabled, shows “No canary found”, meaning there is no protection against buffer overflows overwriting the return address.
  • NX
    • Also known as the No-eXecute bit. When enabled, prevents shellcode from executing on the Stack or Heap.
  • PIC
    • Also known as Position-Independent Code. When enabled, allows shared libraries to be loaded at relative memory addresses.
  • PIE
    • Also known as Position-Independent Executable. When disabled, shows “No PIE”, meaning the program loads at the same memory address every time it runs.
  • RUNPATH
    • At runtime, the dynamic linker searches for shared libraries in this path before falling back to standard locations such as /lib and /usr/lib.
strings fileName
strings fileName -t x
  • Outputs any printable characters to the terminal.
    • Useful for quick information disclosure.
ldd fileName
  • Lists all shared libraries required by a dynamically linked binary.
readelf -l fileName
readelf -a fileName
  • Extracts information about the headers, sections, symbols, and other aspects of an ELF file.
nm fileName
  • Lists symbols from object files (executables).
objdump -d fileName > decompiled.asm
  • Disassembles an executable.
    • Alternatively, use the GUI tool Ghidra [Link].
      • On Kali, install it with sudo apt install -y ghidra.
ropper --file=fileName --search "pop rdi"
  • Finds all gadgets in the binary for use in ROP exploits [Link].
sudo -H python3 -m pip install ROPgadget
ROPgadget --binary fileName
  • Locates gadgets using ROPgadget. Use Radare2 [Link] to find register memory addresses.
r2 fileName

DEBUGGING TOOLS

ltrace ./fileName
  • Intercepts and displays library calls as the program runs.
    • E.g. gets, puts, or printf from libc.
strace ./fileName
  • Intercepts and displays system calls as the program runs.
    • E.g. open, read, write, close, fork, execve, exit, or mmap from the kernel.
gdb ./fileName
  • A powerful debugger that deserves a cheat sheet of its own.
  • A much better experience is available through the Python module gdb-pwndbg [Link].
gdb-pwndbg fileName
  • file fileName
    • Loads the binary to be debugged.
  • info functions
    • Lists the functions in the binary.
  • info registers
    • Shows the contents of all registers.
  • info proc mappings
    • Displays memory mappings including start/end addresses and permissions (rwx).
  • disassemble main
    • Disassembles a function.
  • info stack
    • Shows the stack of a running or crashed execution.
  • backtrace
    • Similar to info stack.
  • break main
    • Adds a breakpoint to a function.
  • break *0xffffffff
    • Adds a breakpoint at a memory address.
  • delete break
    • Removes all breakpoints.
  • x/s 0xffffffff
    • Shows the contents of memory at the given address.
  • cyclic 100
    • Generates a 100-character cyclic pattern for overflow testing.
    • Similar online tool [Link].
  • cyclic -l XXXX
    • Calculates the offset to the given pattern segment.
  • run
    • Executes the binary.
      • n steps to the next instruction.
      • c continues to the next breakpoint or runs freely.
  • run < payload
    • Feeds a payload file into standard input.

See this tutorial for installing Pwndbg + GEF + Peda together [Link].

CRAFTING PAYLOADS MANUALLY

python2 -c 'print "A"*4 + "BBBB" + "\xef\xbe\xad\xde"' > payload
  • For 32-bit: creates AAAABBBB followed by the binary representation of address 0xdeadbeef.
python2 -c 'print "A"*10 + "\xef\xbe\xad\xde\x00\x00\x00\x00"' > payload
  • For 64-bit: creates AAAAAAAAAA followed by the binary representation of address 0x00000000deadbeef.
shellcraft -l
  • Lists available payloads (assembly instructions) for obtaining code execution or a shell within the application runtime.
shellcraft i386.linux.sh
  • Outputs the desired payload in hex.
shellcraft i386.linux.sh -f a
  • Outputs the payload in assembly.

See also msfvenom for additional payloads at [Link].

BONUS

upx -h
upx -d fileName
  • Tool for compressing and decompressing executable files.
hexedit fileName
  • View and edit files in hexadecimal or ASCII.

Pwndbg + GEF + Peda

There is a tutorial for setting up all three tools at once [Link]. Highly recommended to have ready in a Kali base image for pentesting or CTF work.

cd ~ && git clone https://github.com/apogiatzis/gdb-peda-pwndbg-gef.git
cd ~/gdb-peda-pwndbg-gef
./install.sh
./update.sh

IDA Free

Not open-source, but this free tool, originally built for Windows and now available on Linux and macOS as well, is arguably the best visual decompiler [Link].

wget https://out7.hex-rays.com/files/idafree84_linux.run
chmod +x idafree84_linux.run 
./idafree84_linux.run

Radare2 Cutter

A graphical interface for the well-known Radare2 reverse engineering framework [Link].

sudo apt install radare2-cutter -y

Complete summary of the binary on the landing Dashboard:

Disassemble with graphics:

Two popular decompilers available out of the box:

It has an impressively clean interface.

Decompiler Explorer

An online multi-decompiler [Link] [Link].

Windows Reverse Engineering Tools

Binary Ninja is not open-source but offers a free version for Windows, macOS, and Linux [Link].

PEStudio is also not open-source, but its free version is well suited for .NET applications [Link].

ROP Emporium

A free collection of binary exploitation challenges [Link], with a walk-through that builds the knowledge needed to complete them.

Pwnables

Started as a private CTF and became an OpenToAll CTF [Link]. It now contains more than 50 challenges that progressively increase in difficulty.

Libc Database Search

A linked memory address can reveal the version of a library in use, which in turn exposes available functions. This database includes a search feature and provides the libraries themselves for local exploitation testing before targeting a remote system.

  • libc database search [Link]
  • libc-database [Link]
  • Search engine source code [Link]

More Vulnerabilities to Consider

  • Integer Overflow/Underflow
    • Occurs when an arithmetic operation exceeds the maximum binary value a type can hold, causing the value to wrap around.
  • Format String Vulnerability
    • Specific to languages like C and the printf family of functions (the f stands for Format). Without proper input sanitization, this can lead to arbitrary memory reads or writes.

Fast-Paced Reverse Engineering for CTFs

  • PatchELF [Link]
    • Modifies ELF binaries and libraries to add, remove, or alter paths and dependencies.
  • PwnInit [Link]
    • Automates binary exploit challenge setup: marks the binary as executable, downloads the linker and debug symbols, unstrips libc, patches the binary’s RPATH, and fills in a pwntools solve script template.

BONUS OF THE BONUS

Running and debugging x86-32, ARM, and MIPS applications on x86-64:

sudo apt update
sudo apt-get install qemu-user -y
sudo apt install libc6-i386 gdb-multiarch -y
sudo apt install libc6-armel-cross -y
sudo mkdir /etc/qemu-binfmt -p
sudo ln -s /usr/arm-linux-gnueabi /etc/qemu-binfmt/arm
qemu-arm-static ./armv5.bin
qemu-arm ./armv5.bin
sudo apt install libc6-mipsel-cross -y
sudo mkdir /etc/qemu-binfmt -p
sudo ln -s /usr/mipsel-linux-gnu /etc/qemu-binfmt/mipsel
qemu-mipsel-static ./armv5.bin
qemu-mipsel ./armv5.bim