Windows

Transcription

Windows

Turning Data into Value
SAS® Event Stream
Processing Engine
David Michiels
Copyright © 2014, SAS Institute Inc. All rights reserved.
Today’s menu
 What is event stream processing?
 Key concepts
 Publish & subscribe
 Live Demo
SAS® EVENT STREAM PROCESSING ENGINE
WHAT IS EVENT STREAM PROCESSING ?
What is a stream?
stream noun \ˈstrēm\
 a natural flow of water that is
smaller than a river
 any flow of liquid or gas
 a continuous flow of
people or things
Process data on the move
ENGINE
DATA IN
(Events)
Process Data
On the Move
Very High speed
Low latency
DATA OUT
(Events)
Detect events of interest
SAS EVENT STREAM PROCESSING
TYPICALENGINE
ESP QUESTIONS
“ give meDetect
the topEvents
3 values
every 5 minutes”
of Interest
Filtering
“Tell me when an
event A was followed by an
Aggregation
event B andPattern
not event
C within 3 minutes”
detection
DATA IN
(called Events)
Calculations
DATA OUT
“Tell me when you
detect
a
3rd
bank
transfer
in
(Events)
Correlations
the last 24 hours
from
the
same
account,
Procedural
coming from 3 different
countries, and pause the
Thresholding
transfer
until
manual
validation”
and
much
more…
“Filter out sensor readings when the device was
in maintenance period”
KEY CONCEPTS
Dataflow centric
SAS EVENT STREAM PROCESSING ENGINE
WINDOW
WINDOW
Event
Stream
DATA IN
FILTER
SOURCE
(Events)
1
Event
Stream
WINDOW
WINDOW
DATA OUT
WINDOW
JOIN
Event
Stream
DATA IN
SOURCE
(Events)
2
SOURCE
3
WINDOW
WINDOW
JOIN
THRESHOLD
WINDOW
DATA OUT
Event
Stream
(Events)
CALCULATIONS
Event
Stream
Event
Stream
Design of the rule model (called “Continuous Query”)
using components (called “Windows”)
(Events)
Event
Stream
WINDOW
DATA IN
Event CALCULATIONS
Stream
(Events)
Continuous update
Each “Window” (Source, Aggregation, Filter, etc.)
Is automatically updated
Each time an Event (data record) comes in
Continuous update
SOURCE WINDOW
ID
SYMB Text
QTY
PRICE
FILTER : QTY >=1000
ID
SYMB
QTY
PRICE
AGGREGATION
SYMB
AVE_PRICE
DATA IN
DATA
OUT
(Events)
(Events)
ID
SYMB
QTY
PRICE
10
IBM
2000
105
Let’s say…
We have market transactions…
We want the current average price of transactions with Quantity >= 1000
We need :
1. An Event Stream Processing Engine…
2. Design a ESP model (Continuous Query) …
3. Start the ESP model…
4. Send Data…
Continuous update
SOURCE WINDOW
ID
DATA IN
SYMB Text
QTY
FILTER : QTY >=1000
PRICE
QTY
QTY
PRICE
SYMB
AVE_PRICE
IBM
2000
105
IBM
105
IBM
110
105
20
MS
1500
26
MS
26
MS
26
IBM
110
40
SYMB
SYMB
10
(Events)
ID
ID
AGGREGATION
IBM
PRICE
3000
115
Window’s states are continuously updated
Events are stored in Memory Depot
Data retention policies
DATA
OUT
(Events)
What is an ESP event?
ID*: int32, SYMB: string, QTY: int32, PRICE: double
N_P FLAG
OPCODE
ID*
SYMB
QTY
PRICE
N
I
10
IBM
2000
105
A data record
With a Schema (datatypes desc.)
With an Operation Code (opcode)
A normal/partial update flag (n,p)
How does it work ?
OPCODES
I
Insert
U
Update
D
Delete
P
UpSert
Continuous update
SOURCE WINDOW
ID
SYMB Text
QTY
FILTER : QTY >=1000
PRICE
ID
SYMB
QTY
PRICE
AGGREGATION
SYMB
AVE_PRICE
10
IBM
2000
125
105
10
IBMIBM
2000
120105
IBM
120
115
125
DATA IN
20
MS
1500
26
20
MS
1500
26
MS
26
(Events)
30
IBM
500
110
40
IBM
40
3000
IBM
115
115
40
IBM
3000
900
115
ID
SYMB
QTY
PRICE
Updates & Deletes are passed along the flow
Window’s states are continuously updated
DATA
OUT
(Events)
PUBLISH & SUBSCRIBE
Publishers & subscribers
WINDOW
WINDOW
Event
Stream
DATA IN
Publishers
send events
to ESP
source
windows
(Events)
Event
Stream
WINDOW
WINDOW
DATA OUT
(Events)
WINDOW
Event
Stream
Event
Stream
DATA IN
(Events)
Event
Stream
WINDOW
WINDOW
WINDOW
WINDOW
Event
Stream
DATA IN
(Events)
Event
Stream
Event
Stream
DATA OUT
(Events)
Subscribers
receive
events from
ESP
windows
Connectors & adapters
WINDOW
WINDOW
Event
Stream
DATA IN
Event
Stream
(Events)
WINDOW
WINDOW
DATA OUT
(Events)
WINDOW
Event
Stream
Event
Stream
DATA IN
(Events)
Event
Stream
WINDOW
WINDOW
WINDOW
WINDOW
Event
Stream
DATA IN
(Events)
Event
Stream
Event
Stream
Connectors are in-process
publishers or subscribers
DATA OUT
(Events)
WINDOW
WINDOW
Event
Stream
DATA IN
(Events)
Event
Stream
WINDOW
WINDOW
DATA OUT
(Events)
WINDOW
Event
Stream
DATA IN
Event
Stream
(Events)
Event
Stream
WINDOW
WINDOW
WINDOW
WINDOW
Event
Stream
DATA IN
(Events)
Event
Stream
DATA OUT
(Events)
Event
Stream
Adapters are stand-alone publishers or subscribers.
Adapters can be networked.
OUT OF THE BOX
PUB/SUB API
File/Socket
XML / JSON
Database (odbc)
SAS® LASR™
Hadoop
Teradata fast adapter
SMTP Subscriber
SAS Dataset
JMS
ESP to ESP
Syslog
IBM WebSphere MQ
Tibco RendezVous
OSIsoft PI
Solace
Tervela
Google Protocol Buffers
C/C++
Java
User Written
All Connectors & Adapters
are build using
the Pub/Sub API
LIVE DEMO
Twitter to SAS LASR®
 Load continuously SAS LASR® from Twitter
streams in real-time
• The tweets are received and processed by ESP from the
Twitter stream as soon as they are tweeted
• and are continuously inserted into a LASR table.
• This LASR table can then be used with SAS Visual
Analytics for analysis
• Tracking keywords:
sasbelux,SFBL14,SFBL,spicyanalytics
Twitter Contest – Tweet to win prizes!
SAS Forum
2. What is specific about an ESP adapter?
A. Adapters can be networked
B.
Adapters are running inside the ESP process
C.
Adapters cannot subscribe to an ESP window
Prizes to win:
Tweet your answer:
Example:
@spicyanalytics 2C
Start of your tweet
Question #
Your answer
1st prize: a ticket for Analytics 2015
2nd prize: a book of Prof Bart Baesens: “Analytics in a big
data world”
3rd to 30th prize: chocolates with pepper
Winners will be contacted post-Forum !
Turning Data into Value

Windows

Transcription

Similar documents