Data Warehousing and Data Mining Techniques for Cyber Security (eBook)

PREFACE 6
TABLE OF CONTENTS 9
Chapter 1 AN OVERVIEW OF DATA WAREHOUSE, OLAP AND DATA MINING TECHNOLOGY 12
1. MOTIVATION FOR A DATA WAREHOUSE 12
2. A MULTIDIMENSIONAL DATA MODEL 14
3. DATA WAREHOUSE ARCHITECTURE 17
4. DATA WAREHOUSE IMPLEMENTATION 17
4.1 Indexing of OLAP Data 18
4.2 Metadata Repository 19
4.3 Data Warehouse Back- end Tools 19
4.4 Views and Data Warehouse 21
5. COMMERCIAL DATA WAREHOUSE TOOLS 22
6. FROM DATA WAREHOUSING TO DATA MINING 22
6.1 Data Mining Techniques 23
6.2 Research Issues in Data Mining 25
6.3 Applications of Data Mining 25
6.4 Commercial Tools for Data Mining 26
7. DATA ANALYSIS APPLICATIONS FOR NETWORKAVEB SERVICES 27
7.1 Open Research Problems in Data Warehouse Maintenance 30
7.2 Current Research in the area of Data Warehouse Maintenance 32
8. CONCLUSIONS 33
References 33
Chapter 2 NETWORK AND SYSTEM SECURITY 36
1. VIRUSES AND RELATED THREATS 37
1.1 Types of Viruses 38
1.2 Macro Viruses 38
1.3 E-mail Viruses 38
1.4 Worms 39
1.5 The Morris Worm 39
1.6 Recent Worm Attacks 39
1.7 Virus Counter Measures 40
2. PRINCIPLES OF NETWORK SECURITY 41
2.1 Types of Networks and Topologies 41
2.2 Network Topologies 42
3. THREATS IN NETWORKS 42
4. DENIAL OF SERVICE ATTACKS 44
4.1 Distributed Denial of Service Attacks 45
4.2 Denial of Service Defense Mechanisms 45
5. NETWORK SECURITY CONTROLS 47
6. FIREWALLS 49
6.1 What they are 49
6.2 How do they work 50
6.3 Limitations of Firewalls 51
7. BASICS OF INTRUSION DETECTION SYSTEMS 51
8. CONCLUSIONS 52
References 52
Chapter 3 INTRUSION DETECTION SYSTEMS 54
1. CLASSIFICATION OF INTRUSION DETECTION SYSTEMS 55
2. INTRUSION DETECTION ARCHITECTURE 59
3. IDS PRODUCTS 60
3.1 Research Products 60
3.2 Commercial Products 61
3.3 Public Domain Tools 62
3.4 Government Off-the Shelf (GOTS) Products 64
4. TYPES OF COMPUTER ATTACKS COMMONLY DETECTED BY IDS 64
4.1 Scanning Attacks 64
4.2 Denial of Service Attacks 65
4.3 Penetration Attacks 66
5. SIGNIFICANT GAPS AND FUTURE DIRECTIONS FOR IDS 66
6. CONCLUSIONS 68
References 68
Chapter 4 DATA MINING FOR INTRUSION DETECTION 70
1. INTRODUCTION 70
2. DATA MINING FOR INTRUSION DETECTION 71
2.1 Adam 71
2.2 Madam ID 74
2.3 Minds 75
2.4 Clustering of Unlabeled ID 76
2.5 Alert Correlation 76
3. CONCLUSIONS AND FUTURE RESEARCH DIRECTIONS 77
References 78
Chapter 5 DATA MODELING AND DATA WAREHOUSING TECHNIQUES TO IMPROVE INTRUSION DETECTION 80
1. INTRODUCTION 80
2. BACKGROUND 81
3. RESEARCH GAPS 83
4. A DATA ARCHITECTURE FOR IDS 84
4.1 A Software Architecture and Data Model for Intrusion Detection 84
4.2 Data Modeling for Historical Data Analysis Using STAR Schema: 86
4.3 Support for High Speed Drill Down Queries and Detection of AttacksA^irusAVorms 88
4.4 Feature Extraction From Network Traffic Data 89
4.5 Help the Security Officer for Forensic Analysis 90
5. CONCLUSIONS 91
References 91
Chapter 6 MINDS: ARCHITECTURE & DESIGN
1. MINDS - Minnesota INtrusion Detection System 95
2. Anomaly Detection 97
3. Summarization 101
4. Profiling Network Traffic Using Clustering 104
5. Scan Detection 108
6. Conclusion 116
7. Acknowledgements 116
Notes 117
References 117
Chapter 7 DISCOVERING NOVEL ATTACK STRATEGIES FROM INFOSEC ALERTS 120
1. Introduction 121
2. Alert Aggregation and Prioritization 123
2.1 Alert Aggregation and Clustering 123
2.2 Alert Verification and Prioritization 124
3. Probabilistic-Based Alert Correlation 127
3.1 Motivation 127
3.2 Model Description 128
3.3 Parameters in Bayesian Model 131
3.4 Summary 133
4. Statistical-Based Alert Correlation 133
4.1 Motivation 133
4.2 Time Series Analysis 134
4.3 Granger Causality and Granger Causality Test 135
4.4 Procedure of Data Processing in GCT 136
4.5 Applying GCT to Alert Correlation 137
5. Causal Discovery-Based Alert Correlation 140
5.1 Motivation 140
5.2 Introduction to Causal Discovery 141
5.3 Applying Causal Discovery Analysis to Alert Correlation 144
6. Integration of Three Correlation Engines 147
6.1 The Integration Process of Three Correlation Engines 147
6.2 Probability/Confidence Integration 148
6.3 Attack Transition Table Updates 150
6.4 Attack Strategy Analysis 150
7. Experiments and Performance Evaluation 151
7.1 The Grand Challenge Problem (GCP) 151
7.2 GCP Scenario II 157
7.3 Discussion on Statistical and Temporal Correlation Engines 159
8. Related Work 161
9. Conclusion and Future Work 164
References 165
INDEX 169

1. VIRUSES AND RELATED THREATS (p. 36)
This section briefly discusses a variety of software threats. We first present information about computer viruses and worms followed by techniques to handle them.

A virus is a program that can "infect" other programs by modifying them and inserting a copy of itself into the program. This copy can then go to infect other programs. Just like its biological counterpart, a computer virus carries in its instructional code the recipe for making perfect copies of itself. A virus attaches itself to another program and then executes secretly when the host program is run.

During it lifetime a typical virus goes through the following stages:

Dormant Phase: In this state the virus is idle waiting for some event to happen before it gets activated. Some examples of these events are date/timestamp, presence of another file or disk usage reaching some capacity.

Propagation Phase: In this stage the virus makes an identical copy of itself and attaches itself to another program. This infected program contains the virus and will in turn enter into a propagation phase to transmit the virus to other programs.

Triggering Phase: In this phase the virus starts performing the function it was intended for. The triggering phase can also be caused by a set of events.

Execution Phase: In this phase the virus performs its fiinction such as damaging programs and data files.

1.1 Types of Viruses

The following categories give the most significant types of viruses.

Parasitic Virus: This is the most common kind of virus. It attaches itself to executable files and replicates when that program is executed.

Memory Resident Virus: This kind of virus resides in main memory. When ever a program is loaded into memory for execution, it attaches itself to that program.

Boot Sector Virus: This kind of virus infects the boot sector and it spreads when the system is booted from the disk.

Stealth Virus: This is a special kind of virus that is designed to evade itself from detection by antivirus software.

Polymorphic virus: This kind of virus that mutates itself as it spreads from one program to the next, making it difficult to detect using the "signature" methods.

1.2 Macro Viruses

In recent years macro viruses have become quite popular. These viruses exploit certain features found in Microsoft Office Applications such as MS Word or MS Excel. These applications have a feature called macro that people use to automate repetitive tasks.

The macro is written in a programming language such as Basic. The macro can be set up so that it is invoked when a certain function key is pressed. Certain kinds of macros are auto execute, they are automatically executed upon some events such as starting the execution of a program or opening of a file. These auto execution macros are often used to spread the virus.

New version of MS Word provides mechanisms to protect itself from macro virus. One example of this tool is a Macro Virus Protection tool that can detect suspicious Word files and alert the customer about a potential risk of opening a file with macros.