What is Continuous Intelligence?

A quick guide to the next big thing

Introduction

For the past 10 years, the most successful business applications of machine learning have been quite ethereal: online advertising (e.g. retargeting, website optimization), high-frequency trading, yield management...

Machine learning has often disapointed ventures in the gritty world of operations:

  • Algorithmic performance was not that high compared to traditional linear models.
  • Very few prototypes made it to production.

We have come to understand that for hardcore business challenges, machine learning works best when integrated into a larger continuous intelligence solution.

Here is Gartner's definition of continuous intelligence:

Continuous intelligence combines data and analytics with transactional business processes and other real-time interactions. It leverages augmented analytics, event stream processing, optimization, business rule management and machine learning.

Could that be the next big thing?

***

This page provides an introduction to the four main features of continuous intelligence, and links to related third-party resources.


1.  Streaming data

Streaming data (or event streams, or time series) is the raw material of continuous intelligence: data that accumulates over time, one data point at a time.

High frequency is a possibility, but not an obligation. From an algorithmic and decision management perspective, daily sales data or weekly inventory data are the same as instant sensor data.

What matters is:

  • The sequential nature of the data, as certified by reliable time stamps.
  • The time dependence of the business challenge - meaning that the sequence of events conveys as much information as the events themselves.

Collecting, storing, preprocessing and modeling streaming data is very specific. The following resources clarify these specificities.


2.  Contextual data

When building continuous intelligence solutions, the single most important step is the definition of the target: what do we want to understand? which outcome do we want to affect? are there one or many data streams matching that definition?

But a key feature of continuous intelligence is that understanding the target will require processing other data streams for context. For example:

  • Optimizing natural gaz procurement will require analyzing LNG price movements, but also understanding how oil prices, electricity prices, weather conditions, macroeconomic factors... affect LNG prices.
  • The root-cause analysis of a manufacturing anomaly will require modeling that anomaly's behavior, but also measuring the impact of hundreds of potentially relevant parameters, on and around the production line.

This combination of target and contextual data has three notable consequences:

  • It makes it hard to build continuous intelligence solutions without machine learning, since the more data sources, the less linear models will perform (the famed "curse of dimensionality").
  • It means that continuous intelligence solutions will receive inputs from multiple internal and external sources, which creates specific IT challenges.
  • Data gathering and processing are not free, so how do we identify the data that truly contributes to our continuous intelligence solutions's efficiency?

Resources on machine learning and data selection:


3.  Advanced analytics

Continuous intelligence places a unique burden on data analysis, for two reasons:

  • The goal is to improve (and potentially automate) real-time decision making. This means that our analyses must update every time new data appears ("streaming analytics" or "streaming BI"), which in turn has significant repercussions on IT and algorithms.
  • Because of the nature of time series, the true relevance test for time series analytics is prediction accuracy. Faithfully representing and analyzing streaming data necessarily implies the ability to predict upcoming values. In other words, for continuous intelligence, you can't be prescribe if you don't predict.
The following resources explore the multi-faceted relationship between data analytics and continuous intelligence.

4.  Decision management

Continuous intelligence is decision-oriented: it is about helping managers select the right course of action when multiple factors and potentially conflicting goals are involved.

An important consequence is that continuous intelligence solutions require the combination of many types of models:

  • Linear and machine learning models to make sense of the streaming data - internal and external.
  • Business rules reflecting the company's policies (and often embodying its operational experience) in the corresponding area.
  • Physical models describing the organization of the relevant assets.
  • Human inputs! Capturing the expertise of operators.
  • Optimization models formalizing the business trade-offs.
Resources on decision management, business rules and optimization:

 

When to use

Features

  • Data sequentialization
  • Stationarization and filtering
  • Built-in models
  • Compatibility with ML libraries
  • Aggregation and stacking
  • Custom cost functions
  • Parallelization and distribution
  • Connectors
  • Graph structure
  • Performance monitoring
  • Checkpoints and backups
  • Continual improvement