Not Your Father`s Transac on Processing

theNewSQLdatabaseyou’llneveroutgrow
NotYourFather’s
Transac0onProcessing
MichaelStonebraker,CTO
VoltDB,Inc.
HowDoesThisFitinto“BigData”?
! Bigvolume
+  Ihavetoomuchdata
! Bigvelocity
+  Dataiscomingatmetoofast
! Bigvariety
+  Ihavetoomanydatasources
VoltDB
2
HighVelocityApplicaJons
! TradiJonaltransacJonprocessing
! “New”transacJonprocessing
! Highvelocityingest
VoltDB
3
TradiJonalTransacJonProcessing
! RememberhowweusedtobuyairplaneJcketsinthe
1980s
+  Bytelephone
+  Throughanintermediary(professionalterminaloperator)
! Commerceatthespeedoftheintermediary
! In1985,1,000transacJonspersecondwasconsidered
anincrediblestretchgoal!!!!
+  HPTS(1985)
VoltDB
4
TradiJonalTransacJonProcessing
! Workloadwasamixofupdatesandqueries
! ToanACIDdatabasesystem
+  Makesureyouneverlosemydata
+  Makesuremydataiscorrect
! Athumanspeed
! BreadandbuXerofRDBMSs(OldSQL)
VoltDB
5
HowhasTPChangedin25Years?
Theinternet
+  Clientisnolongeraprofessionalterminaloperator
+  InsteadAuntMarthaisusingthewebherself
+  SendsTPvolumethroughtheroof
+  Seriousneedforscalabilityandperformance
VoltDB
6
HowhasTPChangedin25Years?
PDAs
+  YourcellphoneisatransacJonoriginator
+  SendsTPvolumethroughtheroof
+  Seriousneedforscalabilityandperformance
Need in some traditional markets
for much higher performance!
VoltDB
7
AndTPisNowaMuchBroaderProblem
(NewTP)
TheinternetenablesagreenfieldofnewTP
applicaJons
+  MassivelymulJplayergames(stateofthegame,leaderboards,
sellingvirtualgoodsareallTPproblems)
+  Socialnetworking(socialgraphisaTPproblem)
+  RealJmeadplacement
+  RealJmecouponing
+  AndTPvolumesareginormous!!
+  Seriousneedforspeedandscalability!
VoltDB
8
AndTPisNowaMuchBroaderProblem
SensorTagginggeneratesnewTPapplicaJons
+  Marathonrunners(frauddetecJon,leaderboards)
+  Taxicab(scheduling,farecollecJon)
+  DynamictrafficrouJng
+  Carinsurance“bythedrink”
+  Mobilesocialnetworking
+  AndTPvolumesareginormous!!
+  Seriousneedforspeedandscalability!
VoltDB
9
AndTPisNowaMuchBroaderProblem
Electroniccommerceishere
+  WallStreetelectronictrading
+  Real-JmefrauddetecJon
+  MicrotransacJons(throughyourPDA)
+  AndTPvolumesareginormous!!
+  Seriousneedforspeedandscalability!
VoltDB
10
AddinHighVelocityIngest
+  RealJmeclickstreamanalysis
+  MostanythingupstreamfromHadoop
+  Oryourdatawarehouse
+  RealJmeriskassessmentonWallStreet
+  AndTPvolumesareginormous!!
+  Seriousneedforspeedandscalability!
VoltDB
11
Inallcases…..
! Workloadisamixofupdatesandqueries
! Comingatyoulikeafirehose
! SJllanACIDproblem
+  Don’tlosemydata
+  Makesureitiscorrect
! TendstobreaktradiJonalsoluJons
+  Scalabilityproblems(volume)
+  ResponseJmeproblems(latency)
VoltDB
12
PutDifferently
Highvelocityandyou
Youneedtoingestafirehoseinreal
Jme
Youneedtoprocess,validate,enrich
andrespondinreal-Jme(i.e.update)
Youoaenneedreal-JmeanalyJcs
(i.e.query)
VoltDBVoltDB
13
13
1
3
RealityCheck--Size
! TPdatabasesizegrowsattheratetransacJons
increase
! 1TbyteisareallybigTPdatabase
! 1Tbyteofmainmemorybuyableforaround$50K
+  (say)64Gbytesperserverin16servers
! I.e.Moore’slawhaseclipsedTPdatabasesize
! Ifyourdatadoesn’tfitinmainmemorynow,then
waitacoupleofyearsanditwill…..
VoltDB
14
RealityCheck--Performance
! TPC-CCPUcycles
! OntheShoreDBMSprototype
! Elephantsshouldbesimilar
VoltDB
15
ToGoaLotFasterYouHaveto……
! Focusonoverhead
+  BeXerB-treesaffectsonly4%ofthepathlength
! GetridofALLmajorsourcesofoverhead
+  Mainmemorydeployment–getsridofbufferpool
— Leavingother75%ofoverheadintact
— i.e.winis25%
VoltDB
16
SoluJonChoices
! OldSQL
+  LegacyRDBMSvendors
! NoSQL
+  GiveupSQLandACIDforperformance
! NewSQL
+  PreserveSQLandACID
+  Getperformancefromanewarchitecture
VoltDB
17
OldSQL
TradiJonalSQLvendors(the“elephants”)
+  CodelinesdaJngfromthe1980’s
+  “bloatware”
+  MediocreperformanceonNewTP
VoltDB
18
TheElephants
! AreslowbecausetheyspendalloftheirJmeon
overhead!!!
+  Notonusefulwork
! Wouldhavetore-architecttheirlegacycodetodo
beXer
VoltDB
19
LongTermElephantOutlook
!  Upagainst“TheInnovatorsDilemma”
+  Steamshovelexample
+  Diskdriveexample
+  SeethebookbyClaytonChristensonformoredetails
!  Longtermdriaintothesunset
+  Themostlikelyscenario
+  Unlesstheycansolvethedilemma
VoltDB
20
NoSQL
! GiveupSQL
! GiveupACID
VoltDB
21
GiveUpSQL?
! CompilertranslatesSQLatcompileJmeintoa
sequenceoflowleveloperaJons
! SimilartowhattheNoSQLproductsmakeyou
programinyourapplicaJon
! 30yearsofRDBMSexperience
+  Hardtobeatthecompiler
+  Highlevellanguagesaregood(dataindependence,lesscode,…)
+  Storedproceduresaregood!
— OneroundtripfromapptoDBMSratherthanoneoneroundtrip
perrecord
— Movethecodetothedata,nottheotherwayaround
VoltDB
22
GiveUpACID
! Ifyouneeddataconsistency,giving
upACIDisadecisiontotearyour
hairoutbydoingdatabase“heavy
liaing”inusercode
! Canyouguaranteeyouwon’tneed
ACIDtomorrow?
ACID = goodness, in spite of what these guys say
VoltDB
23
WhoNeedsACID?
!  Fundstransfer
+ OranybodymovingsomethingfromXtoY
!  Anybodywithintegrityconstraints
+ Backoutiffails
+ Anybodyforwhom“usuallyshipsin24hours”isnotan
acceptableoutcome
!  AnybodywithamulJ-recordstate
+  E.g.moveandshoot
VoltDB
24
WhoneedsACIDinreplicaJon
!  Anybodywithnon-commutaJveupdates
+  Forexample,+and*don’tcommute
!  Anybodywithintegrityconstraints
+  Can’tsellthelastitemtwice….
!  Eventualconsistencymeans“createsgarbage”
VoltDB
25
NoSQLSummary
! Appropriatefornon-transacJonalsystems
! AppropriateforsinglerecordtransacJonsthatare
commutaJve
! NotagoodfitforNewTP
! Usetherighttoolforthejob
Interes0ng…
Tworecently-proposedNoSQL
languagestandards–CQLand
UnQL–areamazinglysimilarto
(youguessedit!)SQL
VoltDB
26
NewSQL
! SQL
! ACID
! Performanceandscalabilitythroughmodern
innovaJvesoawarearchitecture
VoltDB
27
NewSQL
! NeedssomethingotherthantradiJonalrecordlevel
locking(1stbigsourceofoverhead)
+  Jmestamporder
+  MVCC
+  Yourgoodideagoeshere
VoltDB
28
NewSQL
! NeedsasoluJontobufferpooloverhead(2ndbig
sourceofoverhead)
+  Mainmemory(atleastfordatathatisnotcold)
+  Someotherwaytoreducebufferpoolcost
VoltDB
29
NewSQL
! NeedsasoluJontolatchingforshareddata
structures(3rdbigsourceofoverhead)
+  SomeinnovaJveuseofB-trees
+  Single-threading
+  Yourgoodideagoeshere
VoltDB
30
NewSQL
! NeedsasoluJontowrite-aheadlogging(4thbig
sourceofoverhead)
+  Obviousanswerisbuilt-inreplicaJonandfailover
+  NewTPviewsthisasarequirementanyway
! Somedetails
+  On-linefailover?
+  On-linefailback?
+  LANnetworkparJJoning?
+  WANnetworkparJJoning?
VoltDB
31
ANewSQLExample–VoltDB
! Main-memorystorage
! Singlethreaded,runXactstocompleJon
+ Nolocking
+ Nolatching
! Built-inHAanddurability
+ Nolog(inthetradiJonalsense)
VoltDB
32
Yabut:WhatAboutMulJcore?
! ForAK-coreCPU,dividememoryintoK(non
overlapping)buckets
! i.e.convertmulJ-coretoKsinglecores
VoltDB
33
WherealltheJmegoes…revisited
Before
VoltDB
VoltDB
34
CurrentVoltDBStatus
! RunsasubsetofSQL(whichisgesnglarger)
! OnVoltDBclusters(inmemoryoncommoditygear)
! WithLANandWANreplicaJon
! 70XapopularOldSQLDBMSonTPC-C
! 5-7XCassandraonVoltDBK-Vlayer
! Scalesto384cores(biggestironwecouldgetour
handson)
! Clearlynotethisisanopensourcesystem!
VoltDB
35
Summary
OldTP
NewTP
OldSQLforNewOLTP
!  Tooslow
!  Doesnotscale
NoSQLforNewOLTP
!  Lacksconsistencyguarantees
!  Low-levelinterface
NewSQLforNewOLTP
!  Fast,scalableandconsistent
!  SupportsSQL
VoltDB
36
theNewSQLdatabaseyou’llneveroutgrow
ThankYou