Security Intelligence (eBook)

Qing Li is Chief Scientist and Vice President of Advanced Technologies for Blue Coat Systems, a worldwide provider of security and network systems. He has 17 issued patents, has received multiple industry awards and has been an active speaker at industry conferences and an active voice in the technology media around the world. Gregory Clark is currently the CEO of Blue Coat Systems, a worldwide provider of security and network systems.

Foreword xv

Preface xvii

Chapter 1 Fundamentals of Secure Proxies 1

Security Must Protect and Empower Users 2

The Birth of Shadow IT 2

Internet of Things and Connected Consumer Appliances 3

Conventional Security Solutions 5

Traditional Firewalls: What Are Their Main Deficiencies? 5

Firewall with DPI: A Better Solution? 9

IDS/IPS and Firewall 11

Unified Threat Management and Next?]Generation Firewall 14

Security Proxy--A Necessary Extension of the End Point 15

Transaction?]Based Processing 18

The Proxy Architecture 19

SSL Proxy and Interception 22

Interception Strategies 24

Certificates and Keys 28

Certificate Pinning and OCSP Stapling 32

SSL Interception and Privacy 33

Summary 35

Chapter 2 Proxy Deployment Strategies and Challenges 37

Definitions of Proxy Types: Transparent Proxy and Explicit Proxy 38

Inline Deployment of Transparent Proxy: Physical Inline and Virtual Inline 41

Physical Inline Deployment 41

Virtual Inline Deployment 43

Traffic Redirection Methods: WCCP and PBR 44

LAN Port and WAN Port 46

Forward Proxy and Reverse Proxy 47

Challenges of Transparent Interception 48

Directionality of Connections 53

Maintaining Traffic Paths 53

Avoiding Interception 56

Asymmetric Traffic Flow Detection and Clustering 58

Proxy Chaining 62

Summary 64

Chapter 3 Proxy Policy Engine and Policy Enforcements 67

Policy System Overview 69

Conditions and Properties 70

Policy Transaction 71

Policy Ticket 73

Policy Updates and Versioning System 77

Security Implications 77

Policy System in the Cloud Security Operation 80

Policy Evaluation 82

Policy Checkpoint 82

Policy Execution Timing 84

Revisiting the Proxy Interception Steps 86

Enforcing External Policy Decisions 90

Summary 91

Chapter 4 Malware and Malware Delivery Networks 93

Cyber Warfare and Targeted Attacks 94

Espionage and Sabotage in Cyberspace 94

Industrial Espionage 96

Operation Aurora 96

Watering Hole Attack 98

Breaching the Trusted Third Party 100

Casting the Lures 101

Spear Phishing 102

Pharming 102

Cross?]Site Scripting 103

Search Engine Poisoning 106

Drive?]by Downloads and the Invisible iframe 109

Tangled Malvertising Networks 113

Malware Delivery Networks 114

Fast?]Flux Networks 117

Explosion of Domain Names 119

Abandoned Sites and Domain Names 120

Antivirus Software and End?]Point Solutions - The Losing Battle 121

Summary 122

Chapter 5 Malnet Detection Techniques 123

Automated URL Reputation System 124

Creating URL Training Sets 125

Extracting URL Feature Sets 126

Classifier Training 128

Dynamic Webpage Content Rating 131

Keyword Extraction for Category Construction 132

Keyword Categorization 135

Detecting Malicious Web Infrastructure 138

Detecting Exploit Servers through Content Analysis 138

Topology?]Based Detection of Dedicated Malicious Hosts 142

Detecting C2 Servers 144

Detection Based on Download Similarities 147

Crawlers 148

Detecting Malicious Servers with a Honeyclient 150

High Interaction versus Low Interaction 151

Capture?]HPC: A High?]Interaction Honeyclient 152

Thug: A Low?]Interaction Honeyclient 154

Evading Honeyclients 154

Summary 158

Chapter 6 Writing Policies 161

Overview of the ProxySG Policy Language 162

Scenarios and Policy Implementation 164

Web Access 164

Access Logging 167

User Authentication 170

Safe Content Retrieval 177

SSL Proxy 181

Reverse Proxy Deployment 183

DNS Proxy 187

Data Loss Prevention 188

E?]mail Filtering 190

A Primer on SMTP 191

E?]mail Filtering Techniques 200

Summary 202

Chapter 7 The Art of Application Classification 203

A Brief History of Classification Technology 204

Signature Based Pattern Matching Classification 206

Extracting Matching Terms - Aho?]Corasick Algorithm 208

Prefix?]Tree Signature Representation 211

Manual Creation of Application Signatures 214

Automatic Signature Generation 216

Flow Set Construction 218

Extraction of Common Terms 220

Signature Distiller 222

Considerations 225

Machine Learning?]Based Classification Technique 226

Feature Selection 228

Supervised Machine Learning Algorithms 232

Naive Bayes Method 233

Unsupervised Machine Learning Algorithms 236

Expectation?]Maximization 237

K?]Means Clustering 240

Classifier Performance Evaluation 243

Proxy versus Classifier 247

Summary 250

Chapter 8 Retrospective Analysis 251

Data Acquisition 252

Logs and Retrospective Analysis 253

Log Formats 254

Log Management and Analysis 255

Packet Captures 259

Capture Points 259

Capture Formats 261

Capture a Large Volume of Data 263

Data Indexing and Query 264

B?]tree Index 265

B?]tree Search 267

B?]tree Insertion 268

Range Search and B+?]tree 270

Bitmap Index 272

Bitmap Index Search 273

Bitmap Index Compression 276

Inverted File Index 279

Inverted File 279

Inverted File Index Query 281

Inverted File Compression 282

Performance of a Retrospective Analysis System 283

Index Sizes 283

Index Building Overhead 285

Query Response Delay 286

Scalability 288

Notes on Building a Retrospective Analysis System 289

MapReduce and Hadoop 289

MapReduce for Parallel Processing 292

Hadoop 293

Open Source Data Storage and Management Solution 295

Why a Traditional RDBMS Falls Short 295

NoSQL and Search Engines 296

NoSQL and Hadoop 297

Summary 298

Chapter 9 Mobile Security 299

Mobile Device Management, or Lack Thereof 300

Mobile Applications and Their Impact on Security 303

Security Threats and Hazards in Mobile Computing 304

Cross?]Origin Vulnerability 305

Near Field Communication 306

Application Signing Transparency 307

Library Integrity and SSL Verification Challenges 307

Ad Fraud 308

Research Results and Proposed Solutions 308

Infrastructure?]Centric Mobile Security Solution 311

Towards the Seamless Integration of WiFi and Cellular Networks 312

Security in the Network 313

Summary 315

Bibliography 317

Index 327

"This book should help any developer, researcher, designer, architect, and even strategist to develop not just solutions, but good solutions, in this dense and evolving area...It could be used as a foundation to certify newcomers to the (security) field and will challenge (professionals) on a horizon of skills beyond security, networking, software design, and system architecture."
--Arnaud Taddei, Director of Security Solutions Architecture, Symantec

"I was looking forward to reading Security Intelligence: A Practitioner's Guide to Solving Enterprise Security Challenges as a way to learn about the Symantec Blue Coat Security Proxy in support of my work at Symantec. However, I was soon pleased to discover that the book is much more than that. While the coverage of security proxies is indeed comprehensive, the book more broadly takes on the task of guiding the reader to an understanding of network security as a whole...
The book is clearly written in an approachable conversational style that is careful not to slip into undefined jargon or assume specialized background of the reader, making it an excellent entrée to what is sometimes an inaccessible field."
--Michael Spertus, Symantec Fellow at Symantec and Adjunct Professor, University of Chicago