the words of the auspicious Forrest Gump some things go together like peas and carrots.
Truer words were never spoken. Some things just do go together well,
sometimes by design, often by accident. I don't think anyone actually planned milk and
cookies or popcorn at the movies but nonetheless these things are made
for each other. When it comes to technology the same
harmonious relationships exist.
In the recent Aite report on High Performance Databases (HPDB), the
market for specialized databases
is surveyed along with a handful of vendors in this space.
is a cottage industry where the big database vendors don't
play. It's hard to imagine in this day and age where
database technology is so standardized and mature and a multitude of choice abounds
from commercial products to open source that any other database
technology and a gang of vendors would have a chance. Yet it is happening and it's thriving.
I believe it has to do with a synergistic relationship to event processing. If CEP is the
HPDB's are the "carrots".
These two technologies share two fundamental precepts:
- A focus on Extreme Performance
- Temporal Awareness
I. Extreme Performance,
Speeds and Feeds
These HPDB's which are often referred to as Tick databases,
are found in the same playground as event processing
technologies. In the Capital Markets industry they connect to the same
market sources, consume the same data feeds. Both technologies are
designed to leverage modern multi-core hardware to consume the
ever-increasing firehose of data. By the same token, once that data is
stored on disk, database query performance is equally important.
The massive amount of data collected and is only as good as the
database's ability to query it efficiently thus creating another
(historical) firehose of data which an event processing engine would be
the consummate consumer.
II. Temporal Awareness, when is the data
Time is a basic principle in event processing technology, applications
typically have as a premise to analyze data-in-motion
within a window of time. HPDB's design center is to store and query time series
data. Some of the database vendors even bring time to a higher level
business function. They understand the notion of a business Calendar,
knowing business hours,
business week, holidays, trading hours, etc. Imagine the
simplicity of a query where you want 'Business hours Mon-Fri for the month of February' and
the database itself would know the third Monday was Presidents Day,
skipping over that, thus preventing analytic calculations from
Leveraging the Synergy
These two fundamental shared principles provide the basis for a unique
set of business cases that are only realized by leveraging Event
Processing platforms and High Performance
- Back testing algorithms across massive volumes of historical data compressing time
you could test new trading algorithms against the last 6 months or 1 –
2 years of historical market data but run that test in a matter of
minutes? What if you could be assured that the temporal conditions of
the strategies (i.e. timed limit orders) behaved correctly and
deterministically matching the movement of time in complete
synchronicity with the historical data? These are just a few
of the characteristics that define the harmony between event processing
and high performance (Tick) databases.
- Blending live and historical data in real-time
to obtain volume curves, moving averages, the latest VWAP and other
analytics calculations are possible with these high performance
databases. Leading edge trading algorithms are blending a historical
context with the live market and even News. The winners will be those
that can build these complex algo's and maintain ultra low-latency.
- Pre-Trade Risk Management
exposure is necessary, doing it in real-time to manage market risk is a
mandate. In addition to market data, these high performance
databases can store pre and post trade activity to complement
event-based trading systems and become the basis for trade reporting
In the Trading LifeCycle,
Event Processing and High Performance databases are partner
technologies harmoniously bound together to form a union where the
whole is greater than the sum of the parts. They are the peas and carrots that together create a host of real-world use-cases that would not be possible as individual technologies.
Myself along with my colleague Dan Hubsher we are doing a 3-part Webinar series entitled "Concept to Profit".
The focus is on event processing in the trade lifecycle, but we
include cases that touch upon high performance databases. You can still register for part 2: Building Trading Strategies in the Apama WorkBench where I will focus on the tools for strategy development aimed at the IT developer.
Once again thanks for reading, you can follow me at twitter, here.