A new information age

Twenty years ago, your best bet to find information was to go to a library, search through a pile of index cards, find your way through rows of shelves, and borrow a book. Ten years ago, your best bet to let your friends know of your latest endeavors was to call them. These days are long gone.

One of the most remarkable advances in the last decade was the advent of an age of real-time broadcast. An author of this book remembers a recent ride in a hot air balloon that he happened to share with a newly-wed couple. They were taking a lot of pictures. When the ride was over, they were met with friends. The first question they asked their friends was: “Have you seen our air pictures on Facebook?.” Indeed, information about the experience had already preceded them to the ground.

The age of modern broadcast is enabled by a confluence of three technological advances. First, we now have an unprecedented connectivity on the move, apparently, even while in a hot air balloon. Second, we have an increasingly rich set of options for broadcasting information. Twitter, Facebook, YouTube, Instagram, Flickr, and Foursquare are just a few examples. Finally, we live in a world of information acquisition (i.e., sensing) devices that we use on daily basis. Cameras, GPS devices, fitbits, smart watches, and Internet-connected cars are generating significant amounts of data that we may or may not choose to share.

A direct consequence of the age of broadcast is information overload. In point-to-point communication (such as a phone call), a minute consumed by the initiator corresponds to a minute consumed by the responder. Hence, a balance exists between the collective capacity consumed at sources and the collective capacity consumed at sinks. In contrast, on broadcast channels, for every minute consumed at the broadcast source, hours may be collectively consumed by the community of receivers. For example, if a 1-minute broadcast message is read by 1000 recipients, then 1000 minutes are collectively consumed at the receivers for the one minute spent at the source. This is fine when the number of sources is a lot smaller than the number of receivers (e.g., think of the number of radio stations compared to the number of listeners). However, in the current age of democratized real-time broadcast, everyone can be a source. A survey in 2014, suggested that more than 1 Billion users were on Twitter, more than 1.3 Billion on Facebook, and more than 1.5 Billion on Google+. On Twitter, your message is visible to all. The underlying paradigm is one of global broadcast. Given that everyone can be a broadcast source, the balance between production and consumption is disrupted. The proliferation of sensing devices further exacerbates the imbalance. Consequently, a gap widens between the capacity of sources to generate information and the capacity of sinks to consume it. This widening gap heralds a new era of services whose main function is to summarize large amounts of broadcast data into a smaller amount of actionable information for receivers.

A key class of summarization services in the new age of real-time broadcast are services that attain situation awareness. Much of the information uploaded on social media constitutes acts of sensing. In other words, sources report observations they made regarding their physical environment and found worthwhile to comment on. We call this phenomenon, social sensing. In their raw form, however, these observations are not very useful and generally lack reliability. There are conflicting sentiments, conflicting claims, missing data, purposeful pieces of misinformation, and other noise.

Table 1.1 shows examples of tweets collected from Syria on Twitter in August 2013, in the week following the sudden deaths of thousands of citizens in Ghouta; a suburb near the capital, Damascus. It can be seen that many different claims are posted, some of which are true but others are pure conjecture, rumors, or even intentionally planted misinformation. Hence, the use of social sensing data, such as tweets on Twitter, for enhancing situation awareness must be done with care. A key problem is to extract reliable information from large amounts of generally less reliable social sensing data. Recently, significant advances were made on this topic.

We formally define social sensing as the act of collection of observations about the physical environment from humans or devices acting on their behalf. Assessing data reliability is a fundamental research challenge in this field. This challenge, if successfully addressed, may engender a paradigm shift in situation awareness by allowing development of dependable applications with guaranteed correctness properties that rely on the collective observations of untrained, average, and largely unreliable sources. In this chapter, we introduce this problem, go over the underlying technical enablers and motivations of social sensing applications, and discuss several key challenges and state-of-the-art solutions. We then review the organization of the book, chapter by chapter, to offer a reading guide for the remainder of the book.

1.1 Overview

The idea of leveraging the collective wisdom of the crowd has been around for some time [1, 2]. Today, massive amounts of data are continually being collected and shared (e.g., on social networks) by average individuals that may be used for a myriad of societal applications, from a global neighborhood watch to reducing transportation delays and improving the efficacy of disaster response. Little is analytically known about data validity in this new sensing paradigm, where sources are noisy, unreliable, erroneous, and largely unknown. This motivates a closer look into recent advances in social sensing with an emphasis on the key problem faced by application designers; namely, how to extract reliable information from data collected from largely unknown and possibly unreliable sources? Novel solutions that leverage techniques from machine learning, information fusion, and data mining recently offer significant progress on this problem and are described in this book.

In situations, where the reliability of sources is known, it is easy to compute the probability of correctness of different observations. Among other alternatives, one can use, say, Bayesian analysis to fuse data from...