Why Python Forensics?

Chapter Contents

Introduction

The Python programming language and environment has proven to be easy to learn and use and is adaptable to virtually any domain or challenge problem. Companies like Google, Dropbox, Disney, Industrial Light and Magic, and YouTube just to mention a handful are using Python within their operations. Additionally, organizations like NASA’s Jet Propulsion Lab; the National Weather Service; The Swedish Meteorological and Hydrological Institute (SMHI); and Lawrence Livermore National Laboratories rely on Python to build models, make predictions, run experiments, and control critical operational systems.

Before diving straight in, I am sure you would like a little more information about what I will be covering and how a programming environment like Python matches up with digital investigations. Also, you might be interested to know what you will be learning about, generally what the scope of this book is, and how you can apply the concepts and practical examples presented.

The primary purpose and scope of the book is to show you how Python can be used to address problems and challenges within the cybercrime and digital investigation domain. I will be doing this by using real examples and providing the full source code along with detailed explanations. Thus the book will become a set of reference implementations, a cookbook of sorts, and at the end of the day, will hopefully get you involved in developing your own Python forensic applications.

I will be presenting the material without any preconceived notion about your programming expertise (or lack thereof). I only expect that you have an interest in using the examples in the book, expanding on them, or developing derivatives that will fit your situation and challenge problems. On the other hand, this is not a how to programming book, many of those exist for Python along with a plethora of online resources.

So, let us get started by defining just some of the challenges we face in cybercrime and digital investigation. These challenges after all were the catalyst behind the book and have come from the past two decades of working on solutions to assist law enforcement; defense and corporate entities collect and analyze digital evidence.

Cybercrime investigation challenges

Some of the challenge problems that we face in cybercrime investigation include:

The changing nature of investigations: Much of the work over the past two decades has focused on the postmortem acquisition, search, format, and display of information contained on various types of media. I can clearly remember the phone call I received almost two decades ago from Ron Stevens and Tom Hurbanek at the New York State Police. They were investigating a case that involved a Linux computer and were quite concerned about files and other data that might have been deleted that could be impeding the investigation. At that point no technology existed to extract deleted files or fragments that were buried away inside deleted Linux inodes, although several solutions existed for the Windows platform at the time. We worked together to develop algorithms that eventually became a tool named “extractor” that we provided free to law enforcement.

The move from simply extracting data, recovering deleted files, and scouring unallocated or slack space from computers has rapidly shifted just in the last couple of years. Today we focus most of our attention on smart mobile devices, dynamically changing memory, cloud applications, real-time network forensics, automotive data analysis, and weather-based forensics, just to mention a few. In addition, new work is addressing the association of direct digital forensic evidence with a broad range of instantly available electronic information. Whether this information comes from text messages, Facebook posts, tweets, Linkedin associations, metadata embedded in digital photographs or movies, GPS data that tracks our movements or the digital fingerprints left from every Web site we surf, all may be relevant and used in civil or criminal cases. The question is how do we connect these dots while maintaining forensic efficacy?

The widening gap between technology developers and investigators: Investigators, examiners, incident response personnel, auditors, compliance experts tend to come into this field with a background in social science, whereas technology developers tend to have backgrounds in computer science and engineering. Clearly, there are some excellent examples of crossovers in both directions, but the vocabulary, thought process, and approach to problem solving can be quite different. Our goal, as depicted in Figure 1.1, is to leverage Python forensic solutions to close that gap and create a collaborative nonthreatening environment whereby computer science and social science can come together.

The challenge is to develop a platform, vernacular environment where both social scientists and computer scientists can comfortably communicate and equally participate in the process of developing new forensic solutions. As you will see, the Python environment provides a level playing field, or common ground at least, where new innovations and thought can emerge. This has already shown to be true in other scientific fields like Space Flight, Meteorology, Hydrology, Simulation, Internet Technology advancement, and Experimentation. Python is already providing valuable contributions in these domains.

Cost and availability of new tools: With a couple of exceptions (for example, EnCase® App Central), most new innovations and capabilities that come through vendor channels take time to develop and can add significant cost to the investigator’s toolkit. In the past, investigators carried with them just a handful of hardware and software tools that they used to extract and preserve digital evidence. Today, to address the wide range of situations they may encounter, 30-40 software products may be necessary just to perform acquisition and rudimentary analysis of the digital crime scene. Of course this is just the start of the investigative process and the number and variety of analytic tools continues to grow.

The true cost and cost of ownership of these technologies can be staggering, especially when you factor in education and training. The barrier to entry into the field can easily reach high five or even six figures. This is in a field where backlogs continue to grow at law enforcement agencies around the world. Backlogs are also growing within the corporate sector, which is dealing with human resource actions, corporate espionage, insider leaks, and massive amounts of regulatory requirements.

It is obvious that we need a better onramp and new ways for individuals that have both interest and aptitude, to participate and make entry into the digital investigative field easier and more streamlined. As we move forward in time, the digital crime scene will look more and more like the depiction in Figure 1.2.

Data vs. semantics: Due to constrained resources, new technologies and innovation we must move from simple data analysis and search to rapid semantic understanding and even situational awareness. Investigators need tools that can help them narrow in on targets or hotspots within an investigation, in order to better apply resources and more quickly identify important leads. This is especially true for investigations that are ongoing such as active fraud cases, denial of service attacks, sophisticated malicious code breaches, violent crimes such as murder, child abduction, rape, and aggravated assaults that have a digital component.

In addition, we must capture the knowledge and process of the most seasoned...