Esper: Event Stream Processing and Correlationby Thomas Bernhardt and Alexandre Vasseur
Esper is an Event Stream Processing (ESP) and event correlation engine (CEP, Complex Event Processing). Targeted to real-time Event Driven Architectures (EDA), Esper is capable of triggering custom actions written as Plain Old Java Objects (POJO) when event conditions occur among event streams. It is designed for high-volume event correlation where millions of events coming in would make it impossible to store them all to later query them using classical database architecture. A tailored Event Query Language (EQL) allows expressing rich event conditions, correlation, possibly spanning time windows, thus minimizing the development effort required to set up a system that can react to complex situations.
Esper is a lightweight kernel written in Java which is fully embeddable into any Java process, JEE application server or Java-based Enterprise Service Bus. It enables rapid development of applications that process large volumes of incoming messages or events.
Esper is the leading open source Event Stream processing solution, currently available under a GPL license. This article introduces you to the main concepts of event stream processing and correlation and walks you through a sample application (source code and Ant script are available for download).
Introduction to event streams and complex events
Information is critical to make wise decisions. This is true in real life but also in computing, and especially critical in several areas, such as finance, fraud detection, business intelligence or battlefield operation. Information flows in from different sources in the form of messages or events, giving a hint on the state at a given time such as stock price. That said, looking at those discrete events is most of the time meaningless. A trader needs to look at the stock trend over a period, possibly combined with other information to make the best deal at the right time.
While discrete events when looked one by one might be meaningless, event streams--that is an infinite set of events--considered over a sliding window and further correlated, are highly meaningful, and reacting to them with the minimal latency is critical for effective action and competitive advantage.
Introduction to Esper
Relational databases or message-based systems such as JMS make it really hard to deal with temporal data and real-time queries. Indeed, databases require explicit querying to return meaningful data and are not suited to push data as it changes. JMS systems are stateless and require the developer to implement the temporal and aggregation logic himself. By contrast, the Esper engine provides a higher abstraction and intelligence and can be thought of as a database turned upside-down: instead of storing the data and running queries against stored data, Esper allows applications to store queries and run the data through. Response from the Esper engine is real-time when conditions occur that match user defined queries. The execution model is thus continuous rather than only when a query is submitted.
Such concepts are a key foundation of EDA, and have been under active research in more than the last 10 years. Awareness of the importance of such systems in real-world architectures has started to emerge only recently.
In Esper, a tailored EQL allows registering queries in the engine. A listener class--which is basically a POJO--will then be called by the engine when the EQL condition is matched as events flow in. The EQL enables to express complex matching conditions that include temporal windows, joining of different event streams, as well as filtering, aggregation, and sorting. Esper statements can also be combined together with "followed by" conditions thus deriving complex events from more simple events. Events can be represented as JavaBean classes, legacy Java classes, XML document or
java.util.Map, which promotes reuse of existing systems acting as messages publishers.
A trivial yet meaningful example is as follow: assume a trader wants to buy Google stock as soon as the price goes below some floor value-- not when looking at each tick but when the computation is done over a sliding time window--say of 30 seconds. Given a StockTick event bean with a price and symbol property and the EQL
"select avg(price) from StockTick.win:time(30 sec) where symbol='GOOG'", a listener POJO would get notified as ticks come in to trigger the buy order.
ESP and CEP can be used in algorithmic trading, RFID, service-level agreement, adaptive computing, fraud detection, real-time business intelligence and customer relationship management. There are many more use cases you will discover once you get more familiar with Esper through the next section of this article, which builds up on failure detection and customer management use cases. Several use cases are available online at the Esper web site.
Case Study: A Self-Service Terminal Managing System
In this example we consider a self-service terminal system as it exists in airports to allow customers to proceed to self-check in and print boarding passes. The self-service terminal managing system gets a lot of events from all the connected terminals. The event rate is around 500 events per second. Some events indicate abnormal situations such as "paper low" or "terminal out of order." Other events observe activity as a customer uses a terminal to check in and print her boarding pass (see Figure 1).
Figure 1. Event cloud in a terminal managing system
Our primary goal is to resolve self-service terminal or network problems before our customers report them by looking for help, which means higher overall availability and greater customer satisfaction. To accomplish this, we would like to get alerted when certain conditions occur that warrant human intervention: for example, a customer may be in the middle of a check-in process when the terminal detects a hardware problem or when the network goes down. Under these conditions we would like to dispatch a staff member to help that customer, and another staff member to diagnose the hardware or network problem.
We also want to provide a dashboard and summarize activity on an ongoing basis and feed this to a real-time interface. This enables a manager to watch the system in action and spot abnormalities. The system can further compare the summarized activity to stored normal usage patterns.