Open Source Intelligence Methods and Tools (eBook)

Nihad A. Hassan is an independent information security consultant, digital forensics and cybersecurity expert, online blogger, and book author. He has been actively conducting research in different areas of information security for more than a decade and has developed numerous cybersecurity education courses and technical guides. He has completed many technical security consulting engagements involving security architectures, penetration testing, computer crime investigation, and cyber Open Source Intelligence (OSINT). Nihad has authored four books and scores of information security articles in various global publications. He also enjoys being involved in security training, education, and motivation. His current work focuses on digital forensics, anti-forensics techniques, digital privacy, and cyber OSINT. He covers different information security topics and related matters on his security blog (DarknessGate) and recently launched a dedicated site for Open Source Intelligence resources. Nihad has a BSc honors degree in computer science from the University of Greenwich in the United Kingdom. He can be followed on Twitter at @DarknessGate, and you can connect to him via LinkedIn (DarknessGate).Rami Hijazi holds a master's degree in information technology (information security) from the University of Liverpool. He works for Mericler, Inc., an education and corporate training firm in Toronto, Canada. Rami is an experienced IT professional who lectures on a wide array of topics, including object-oriented programming, Java, e-commerce, agile development, database design, and data handling analysis. He also works as an information security consultant, where he is involved in the design of encryption systems and wireless networks, intrusion detection and data breach tracking, and planning and development advice for IT departments concerned with contingency planning.

Table of Contents 5
About the Authors 13
About the Technical Reviewer 14
Acknowledgments 15
Introduction 16
Chapter 1: The Evolution of Open Source Intelligence 21
Open Source Information Categories 23
OSINT Types 25
Digital Data Volume 25
OSINT Organizations 26
Government Organizations 27
Open Source Center 27
BBC Monitoring 27
Private Sector 27
Jane’s Information Group 28
Economist Intelligence Unit 28
Oxford Analytica 28
Gray Literature Vendors 28
Factiva 29
LexisNexis 30
Parties Interested in OSINT Information 30
Government 30
International Organizations 31
Law Enforcement Agencies 31
Business Corporations 32
Penetration Testers and Black Hat Hackers/Criminal Organizations 32
Privacy-Conscious People 33
Terrorist Organizations 33
Information Gathering Types 34
Passive Collection 34
Semipassive 34
Active Collection 35
Benefits of OSINT 35
Challenges of Open Source Intelligence 36
Legal and Ethical Constraints 37
Summary 38
Notes 39
Chapter 2: Introduction To Online Threats and Countermeasures 41
Online Threats 42
Malware 42
Black Hat Hackers 43
Pharming 43
Phishing 44
Ransomware 47
Adware and Spyware 48
Trojan 49
Virus 49
Worms 49
Scareware 49
Distributed Denial of Service 50
Rootkits 50
Juice Jacking 50
Wi-Fi Eavesdropping 50
Security Software 51
Antivirus 51
Firewall 52
Anti-malware 53
Securing the Operating System 53
Hardening the Windows OS 54
Updating Windows 54
Updating All Installed Programs 55
Locking Your PC Using a USB Drive 55
Using a Less-Privileged User Account 55
Using a Strong Password for Windows 55
Keeping Your User Account Control Turned On 56
Disabling Remote Assistance 56
Making Hidden Files Visible 57
Freezing the Hard Disk 57
Setting a Password for BIOS/UEFI 58
Disabling Unnecessary Ports/Protocols and Services 58
Staying Private in Windows 10 59
Destroying Digital Traces 61
General Privacy Settings 65
Covering Your Laptop Camera 65
Avoiding Pirated Software 65
Handling Digital Files Metadata 66
Physically Securing Computing Devices 70
Online Tracking Techniques 72
Tracking Through IP Address 72
What Is an IP Address? 72
How Is an IP Address Used to Track You Online? 74
Cookies 75
Digital Fingerprinting 77
Script-Based Fingerprinting 77
Canvas Fingerprinting 77
HTML5 78
Checking Your Digital Footprint 78
Browserleaks 78
Panopticlick 78
Secure Online Browsing 79
Configuring Firefox to Become More Private 79
Turning On Private Browsing 79
Changing the Firefox Settings to Become More Private 80
Firefox Privacy Extensions 83
Fighting Against Digital Fingerprinting and Browser Leak 84
Secure Online Communication 84
VPN 85
Proxies 86
DNS Leak Test 87
Online Anonymity 89
Using the TOR Network 89
Tor Browser 90
Hiding Tor Usage 91
Using a VPN 92
Using Tor Bridges 92
Using Pluggable Transports 94
Using the Tails OS and Other Security OSs 96
Sharing Files Securely 97
OnionShare 97
Making Anonymous Payments 99
Prepaid Gift Card 99
Cryptocurrency 100
Encryption Techniques 101
Securing Your Passwords 101
Encrypting Your Hard Drive/USB Sticks 102
Cloud Storage Security 102
Secure E-mail Communications 103
Secure E-mail Providers 104
Secure IM and Online Calling Services 105
Virtualization Technology 106
Android and iOS Emulator 108
Essential Prerequisites 108
Drawing Software and Data Visualization 109
Mind Mapping and Idea Generation Tools 109
FreeMind 109
Storytelling Tools 109
Diagramming Software 109
Apache OpenOffice Draw 109
Google Drawings 110
Note Management 110
TagSpaces 110
KeepNote 110
Data Visualization 110
Microsoft Excel 110
Business Intelligence and Reporting Tools 110
Dradis CE 110
Bookmarking 111
Free Translation Services 112
Final Tips 112
Use a False Identity to Register on Some Websites 112
Be Anonymous 113
Destroy Your Digital Traces Upon Finishing 113
Use Linux 113
Summary 114
Chapter 3: The Underground Internet 115
Layers of the Internet 116
Darknet Users 123
Accessing the Darknet 124
Security Checks When Accessing the Darknet 124
Accessing the Darknet from Within the Surface Web 126
Using Tor 127
Using the Tails OS 129
Warning When Using the Tails OS 134
Searching the Tor Network 135
Other Anonymity Networks 136
I2P 137
Using I2P 137
I2P vs. Tor 142
Freenet 143
Going Forward 143
Summary 144
Notes 145
Chapter 4: Search Engine Techniques 146
Keywords Discovery and Research 148
Using Search Engines to Locate Information 149
Google 149
Google Advanced Operators 152
Google Hacking Database 155
Search Engines Powered by Google 157
Bing 157
Privacy-Oriented Search Engines 159
Other Search Engines 160
Business Search Sites 161
Find Business Annual Records 162
Business Information (Profiles) 164
Metadata Search Engines 166
Code Search 169
FTP Search Engines 170
Automated Search Tools 171
SearchDiggity 171
SearchDome 171
Jeviz 172
Internet Of Things (IoT) Device Search Engines 172
Web Directories 173
Translation Services 175
Website History and Website Capture 177
Website Monitoring Services 179
RSS Feed 181
News Search 182
Customize Google News 183
News Websites 185
Fake News Detection 185
Searching for Digital Files 189
Document Search 189
DOC and DOCX 189
HTML and HTM 189
ODT 189
XLS and XLSX 190
ODS 190
PPT and PPTX 190
ODP 190
TXT 190
PDF 191
File Search Engines 191
Fagan Finder 191
General-Search 192
ShareDir 192
Custom Search Engine 193
Gray Literature 198
Data Leak Information 201
Document Metadata 202
Image 202
Basic Image Search 202
Reverse Image Search 206
Image Manipulation Check 207
OCR Tools 208
Video 210
Basic Video Search 211
Video Analysis 213
File Extension and File Signature List 215
Productivity Tools 215
Screen Capture 215
Download Online Video 216
Easy YouTube Video Downloader Express 216
YooDownload 216
Dredown 217
Video/Audio Converter 217
File Search Tools 218
Summary 220
Notes 220
Chapter 5: Social Media Intelligence 221
What Is Social Media Intelligence? 223
Social Media Content Types 224
Classifications of Social Media Platforms 226
Popular Social Networking Sites 228
Investigating Social Media Sites 229
Facebook 229
Facebook Graph Search 231
Other Useful Facebook Graph Search Commands 240
Tracking Photos Downloaded from Facebook to Its Source Profile 240
Using Google to Search Facebook Content 242
Search for Hashtags on Facebook 242
Using Automated Services to Facilitate Facebook Graph Search 242
Facebook Scanner 243
Graph 243
peoplefindThor 244
Socmint 244
Online Facebook Search Tools/Services 245
Collecting Local Copy of Target Facebook Data 246
Twitter 249
Twitter Search 250
Twitter Advanced Search Operators 251
Twitter Advanced Search Page 255
Online Twitter Search Tools/Services 256
Google+ 259
Searching Google+ 261
Google+ Advanced Search Operator 261
Using Google to Search Within Google+ 264
Searching Google+ Using a Google Custom Search Engine 265
Other Useful Services for Google+ 265
LinkedIn 265
LinkedIn Search 267
Advanced LinkedIn Search Operators 270
Searching LinkedIn Using a Google Custom Search 270
General Resources for Locating Information on Social Media Sites 271
Other Social Media Platforms 272
Pastebin Sites 273
Social Media Psychological Analysis 274
Tone Analyzer 275
Watson Tone Analyzer 275
Facebook and Twitter Prediction 276
Fake Sport 276
Review Meta 276
TweetGenie 276
Summary 276
Notes 277
Chapter 6: People Search Engines and Public Records 279
What Is a People Search Engine? 279
What Are Public Records? 280
Example of Public Records 281
Searching for Personal Details 282
General People Search 282
TruthFinder 282
411 283
Pipl 283
Other 283
Online Registries 286
Vital Records 287
Criminal and Court Search 290
Property Records 291
Tax and Financial Records 292
Social Security Number Search 293
Username Check 293
E-mail Search and Investigation 293
Data Compromised Repository Websites 295
Phone Number Search 297
Employee Profiles and Job Websites 298
Dating Website Search 299
Other Public Records 301
Summary 302
Notes 302
Chapter 7: Online Maps 303
The Basics of Geolocation Tracking 303
How to Find the GPS Coordinates of Any Location on a Map 304
How to Find the Geocode Coordinates from a Mailing Address 306
General Geospatial Research Tools 306
Commercial Satellites 312
Date/Time Around the World 312
Location-Based Social Media 313
YouTube 313
Facebook 314
Using Facebook Graph in the Location Search 314
Facebook Live 315
Twitter 316
Search for Tweets in a Specific Geographical Location 316
Tweet Mapper 318
One Million Tweet Map 319
Qtr Tweets 319
Tweet Map 319
Periscope Map 319
Other Social Media Platforms 320
Strava Heat Map 320
Conducting Location Searches on Social Media Using Automated Tools 321
Country Profile Information 322
Transport Tracking 322
Air Movements 323
Maritime Movements 325
Vehicles and Railway 327
Package Tracking 328
Webcams 329
Digital File Metadata 330
Summary 330
Chapter 8: Technical Footprinting 331
Investigate the Target Website 332
Investigate the Robots.txt File 334
Mirror the Target Website 335
Extract the Links 335
Check the Target Website’s Backlinks 336
Monitor Website Updates 336
Check the Website’s Archived Contents 336
Identify the Technologies Used 337
Web Scraping Tools 340
theHarvester 340
Web Data Extractor 342
Email Extractor 342
Investigate the Target Website’s File Metadata 342
Website Certification Search 343
Website Statistics and Analytics Tools 343
Website Reputation Checker Tools 344
Passive Technical Reconnaissance Activities 345
WHOIS Lookup 345
Subdomain Discovery 347
Using Google Search Operator 348
Using VirusTotal.com 348
DNSdumpster 349
DNS Reconnaissance 350
Route Mapping 350
Common DNS Record Types 351
nslookup Command 352
Netcraft 354
IP Address Tracking 355
Summary 357
Chapter 9: What’s Next? 358
Where Will OSINT Go Next? 358
OSINT Process 360
Final Words 361
Index 362