Big Data: Applications in Change Point Detection

I still have the fondest memories of family trips in the summer. As a kid, one of my jobs on the trip was to manage the 35mm rolls of films. I would keep the used films in my backpack, and inform my parents when to purchase fresh films when we were running low. We would come back with rolls and rolls of films, and dad will develop stacks and stacks of photos.

In 1962, Kodak employed 75,000 people and earned more than $1 billion in U.S. sales. It had achieved an incredible success over the course of 70 years¹. The company had gotten to that position by focusing on selling inexpensive cameras to keep consumers buying its film. To Kodak, there was nothing more important to the company’s success than protecting its dominance in film photography.

That fact may very well explain why executives at Kodak ignored the technology that would drive the next generation of cameras. The digital camera was actually first invented by Steve Sasson, an electrical engineer at Kodak, but it was consistently ignored within Kodak, and the invention was never championed — addicted to the profits generated by its 35mm film, Kodak would do nothing that it saw as endangering the success of this business. A lack of early investment in digital photography in the 70’s eventually led to Kodak’s completely missing the digital waves in the 2000’s. In January 2012, Kodak filed for Chapter 11 bankruptcy protection in the United States.

The story of Kodak should be a cautionary tale to any company: ignore the technology potential, and you do so at your own peril.

More and more frequently, B2C companies are embracing the Big Data wave. They are aware of the fact that they are sitting on a mountain of data, and they want to take full advantage of it. They are attempting to extract all the values of the data they have to build intelligence and improve their relationships with customers.

This trend should set off alarm bells for you. The data revolution has been both a blessing and a curse for many industries. B2B is no exception. When more and more of your downstream customers are B2C companies working hard to capitalize end-user data analytics, you can be sure that they will one day turn the analytics onto their supply chain. Indeed, many of them have come to expect the similar personalized offering, product delivery and after-delivery service from you.

Today in B2B, we are at the cusp of an ever-increasing and overwhelming wave of data. And having cleaned your data house, you are more than ready to catch the waves.

Big Data: Goals and Outcomes

At the core, data analytics can broadly be categorized as follows:

1. Descriptive analytics

Examples include traditional query and reporting environments with scorecards and dashboards. In the Big Data era, your data platform should have the capability to perform descriptive analytics — this will allow you to query the data directly on the platform to figure out what happened and why.

2. Predictive analytics

Predictive analytics can help to estimate the probability of a given event in the feature, such as early alert systems, fraud detection, preventive maintenance applications, and forecasting. In the Big Data era, you should have the ability to perform predictive analytics to conduct forecasting, scheduling, and estimate the probability of a back-order, early alert, or fraud likelihood — without having to bring the data out of the platform to do so.

Combining the power of descriptive and predictive analytics gives us pattern mining in big data. This is particularly true in change point detection: outliers and change points are commonly encountered in time series data analysis, but their identification in a data stream hasn’t been easy. But with big data, this is one of the most exciting topics in data mining and statistics.

The proper identification of change points in time series is so crucial because it can be applied in fraud detection, rare event discovery, and detection in trend changes. And this is helpful in pinpointing narrow bounds in operational trends and industry network patterns. This was easier said than done before — in the past, data was too sparse and too coarse, which invariantly meant that the ability to detect signal from noise was weak. But with the massive data, better algorithms designed to handle the volume and high-dimension with a higher accuracy rate, coupled with cheaper computing powers and powerful cloud computing platforms, big data pattern mining is within grasp.

3. Prescriptive analytics

It provides specific and prescriptive recommendations to the user. They address the question — what is the best course of actions given event “x” had happened. Capabilities to perform prescriptive analytics within the data infrastructure further enable you to, say, advising your team the best course of action depending on a set of known facts.

Artificial intelligence cares about building smart agents that live in in world of people and other smart agents. Route-planning, preventive maintenance scheduling, drone flights are all examples of smart agents. The application of artificial intelligence is being introduced aggressively in finance, from portfolio selection, automatic investment advisor, to algorithms for lending target selection and interest rate determination.

Detecting Change Points in Behavioural Patterns

Many customer-facing businesses are already using machine learning to mine data. That fact is not new. Their data typically come in unstructured form as customer reviews of restaurants, retailers and service providers. A vast majority of these data carry with it even less structure than the reviews found on Yelp or Amazon, in terms of sentiment conveyed. For instance, Amazon provides a fine-grained score on a numeric scale for product reviews. However, some sources such as Twitter, Facebook, Google Groups, and Quora contain text data that is much more voluminous, much less unstructured and unlabeled. Many of these businesses are already assessing their brand sentiment using these highly unstructured data.

Typically, text data is injected into a system with a built-in prediction model, with the objective of generating real-time graphs on opinion and sentiment trends. In this context, the machine learning problem was set up as a classification problem.

Sentiment classification is not new — many businesses are already assessing their brand sentiment text analysis in this way. Document labeling and significant feature discovery are already widely used techniques in natural language processing, such as in solving classification problems of predicting customer satisfaction, spam detection, and plagiarism checking. However, most of the current approaches rely heavily on lexicon-based methods, computing the weights of a certain word towards either positive or negative sentiment. Other approaches use document level similarity scores like Term Frequency — Inverse Document Frequency (TF-IDF) or -gram count vectorization to assess the class of a previously unseen document.

However, models like these are difficult to train for the following reasons:

These processes are time-consuming and tedious. In addition, -gram approaches can, at best, only partially model the language context probabilities; namely, the probability of a certain word appearing in the document depends only on the words that precede it, and TF-IDF completely ignores the context in which a word appears altogether.

So, sentiment classification is not new. What is new, however, is that you can now implement a data and software framework to effectively eliminate the above steps from the process of training a language model. Word2Vec computes the probability of occurrence of the current word, and can be considered as a continuous Bag-of-words model. Word2Vec also provides a skip-gram model², which conversely, uses the context probability of the current word, to predict the most likely words to precede and follow it. These word embeddings can be used to derive both syntactic as well as semantic relations in the vector space of the language model.

In fact, you can go one step further. Paragraph vectors³ are a similar concept to Word2Vec, but rely on computing a fixed-length vector representation of a variable length group of words, which could be a sentence, a paragraph, or even an entire document. Herein lies the ability to rapidly detect change points in consumer behaviour: instead of using machine learning to approximate a discrete function, you goal now is to approximate a continuous function — in doing so, you have flipped the old classification problem on its head, and gained the ability to rapidly detect change points in customer spending pattern, industry trends, or say, millennial behaviours.

[1] Wikipedia contributors. “Kodak.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 24 Aug. 2017. Web.

[2] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013

[3] Le, Q., & Mikolov, T. (2014). Distributed Representations of Sentences and Documents.