Practical Reverse Engineering (eBook)

Bruce Dang is a senior security development engineering lead at Microsoft focusing on Windows kernel and reverse engineering. Alexandre Gazet is a senior security researcher at QuarksLab focusing on reverse engineering and software protection. Elias Bachaalany is a software security engineer at Microsoft.

Introduction xxiii

Chapter 1 x86 and x64 1

Register Set and Data Types 2

Instruction Set 3

Syntax 4

Data Movement 5

Exercise 11

Arithmetic Operations 11

Stack Operations and Function Invocation 13

Exercises 17

Control Flow 17

System Mechanism 25

Address Translation 26

Interrupts and Exceptions 27

Walk-Through 28

Exercises 35

x64 36

Register Set and Data Types 36

Data Movement 36

Canonical Address 37

Function Invocation 37

Exercises 38

Chapter 2 ARM 39

Basic Features 40

Data Types and Registers 43

System-Level Controls and Settings 45

Introduction to the Instruction Set 46

Loading and Storing Data 47

LDR and STR 47

Other Usage for LDR 51

LDM and STM 52

PUSH and POP 56

Functions and Function Invocation 57

Arithmetic Operations 60

Branching and Conditional Execution 61

Thumb State 64

Switch-Case 65

Miscellaneous 67

Just-in-Time and Self-Modifying Code 67

Synchronization Primitives 67

System Services and Mechanisms 68

Instructions 70

Walk-Through 71

Next Steps 77

Exercises 78

Chapter 3 The Windows Kernel 87

Windows Fundamentals 88

Memory Layout 88

Processor Initialization 89

System Calls 92

Interrupt Request Level 104

Pool Memory 106

Memory Descriptor Lists 106

Processes and Threads 107

Execution Context 109

Kernel Synchronization Primitives 110

Lists 111

Implementation Details 112

Walk-Through 119

Exercises 123

Asynchronous and Ad-Hoc Execution 128

System Threads 128

Work Items 129

Asynchronous Procedure Calls 131

Deferred Procedure Calls 135

Timers 140

Process and Thread Callbacks 142

Completion Routines 143

I/O Request Packets 144

Structure of a Driver 146

Entry Points 147

Driver and Device Objects 149

IRP Handling 150

A Common Mechanism for User-Kernel Communication 150

Miscellaneous System Mechanisms 153

Walk-Throughs 155

An x86 Rootkit 156

An x64 Rootkit 172

Next Steps 178

Exercises 180

Building Confidence and Solidifying Your Knowledge 180

Investigating and Extending Your Knowledge 182

Analysis of Real-Life Drivers 184

Chapter 4 Debugging and Automation 187

The Debugging Tools and Basic Commands 188

Setting the Symbol Path 189

Debugger Windows 189

Evaluating Expressions 190

Process Control and Debut Events 194

Registers, Memory, and Symbols 198

Breakpoints 208

Inspecting Processes and Modules 211

Miscellaneous Commands 214

Scripting with the Debugging Tools 216

Pseudo-Registers 216

Aliases 219

Language 226

Script Files 240

Using Scripts Like Functions 244

Example Debug Scripts 249

Using the SDK 257

Concepts 258

Writing Debugging Tools Extensions 262

Useful Extensions, Tools, and Resources 264

Chapter 5 Obfuscation 267

A Survey of Obfuscation Techniques 269

The Nature of Obfuscation: A Motivating Example 269

Data-Based Obfuscations 273

Control-Based Obfuscation 278

Simultaneous Control-Flow and Data-Flow Obfuscation 284

Achieving Security by Obscurity 288

A Survey of Deobfuscation Techniques 289

The Nature of Deobfuscation: Transformation Inversion 289

Deobfuscation Tools 295

Practical Deobfuscation 312

Case Study 328

First Impressions 328

Analyzing Handlers Semantics 330

Symbolic Execution 333

Solving the Challenge 334

Final Thoughts 336

Exercises 336

Appendix Sample Names and Corresponding SHA1 Hashes 341

Index 343