UBER RUSH - QCon New York 2016

UBER RUSH
AND REBUILDING UBER’S DISPATCHING PLATFORM
motivation
CHAPTER 1 OF 8
MOTIVATION TOWARDS MICROSERVICES
STUNTS AND EXPERIMENTS
UBER RUSH, DELIVERY SERVICE
MOTIVATION TOWARDS MICROSERVICES
UBER RUSH REQUIREMENTS
MOTIVATION TOWARDS MICROSERVICES
MULTI-PICKUP
BULK DELIVERIES
MULTI-DISPATCH
SOPHISTICATED MATCHING
CAPACITY MANAGEMENT
UBER EATS REQUIREMENTS
MOTIVATION TOWARDS MICROSERVICES
NO PICKUP LOCATION
TEMPERATURE REGULATION
INVENTORY MANAGEMENT
RE-SUPPLY STATIONS
CHECKOUT FLOW
evolution
CHAPTER 2 OF 8
MONOSERVICE TO MICROSERVICES
MONOLITHIC ARCHITECTURE
CONFIG
USER
CACHE
CARS
POSTPROCESSOR
DISPATCH
MONOSERVICE
MONOSERVICE
ETA
SURGE
GEOCODE
1ST GENERATION MICROSERVICES
MONOSERVICE TO MICROSERVICES
LOGISTICS
SERVICES
DISPATCH
FOUNDATIONAL
SERVICES
PLATFORM
SERVICES
DEPENDENCIES
SUPPLY
ONEDIRECTION
FLIPR
GOLDETA
DEMAND
INV. MGMT
ARBITER
GEOCOD
GEOSPATIAL
OPTIC
RAMEN
CEREBRO
SYNC
LOCKET
MONOSERVICE
2ND GENERATION MICROSERVICES
MONOSERVICE TO MICROSERVICES
LOGISTICS
SERVICES
DISPATCH
FOUNDATIONAL
SERVICES
PLATFORM
SERVICES
DEPENDENCIES
SUPPLY
ONEDIRECTION
FLIPR
GOLDETA
DEMAND
INV. MGMT
ARBITER
GEOCOD
GEOSPATIAL
OPTIC
RAMEN
CEREBRO
DISCO
GEOFENCE
SYNC
LOCKET
RTTR
SCAVENGER
UDESTROY
LUMBERGH
A MICROSERVICE GATEWAY
MONOSERVICE TO MICROSERVICES
LOGISTICS
SERVICES
DISPATCH
GATEWAY
FOUNDATIONAL
SERVICES
PLATFORM
SERVICES
DEPENDENCIES
SUPPLY
ONEDIRECTION
FLIPR
GOLDETA
DEMAND
INV. MGMT
ARBITER
GEOCOD
GEOSPATIAL
OPTIC
RAMEN
CEREBRO
DISCO
GEOFENCE
SYNC
LOCKET
RTTR
SCAVENGER
UDESTROY
LUMBERGH
MOTIVATION TOWARDS MICROSERVICES
THE TRADE-OFFS
UPGRADES ARE PAINFUL
MONOSERVICE
MICROSERVICE
vs.
TEST SUITE IS SLOW
FAILURE IS CATASTROPHIC
CODE IS BRITTLE
DEPLOYS ARE SLOW
topologies
CHAPTER 3 OF 8
MICROSERVICE LAYOUT
INDEPENDENT, INDIVIDUALLY ADDRESSABLE SERVERS
HOST
SERVICE
MICROSERVICE LOAD AVERAGE
DEMAND SERVICE
WORKERS
ARRANGEMENT OF MICROSERVICES
MULTI-TENANT OR DEDICATED HOSTS?
DEMAND
SUPPLY
DISCO
OPTIC
MULTI-TENANT HOSTS
OR
DEDICATED DEMAND
HOST
communications
and fault tolerance
CHAPTER 4 OF 8
MANAGING MICROSERVICE DEPENDENCIES
AUTO-GENERATED CLIENTS
JSON OVER
HTTP
DISCO
MICROSERVICE
DEMAND
MICROSERVICE
THRIFT
OVER HTTP
LUMBERGH
LOAD-BALANCING MICROSERVICES
WITH CLIENT-SIDE LOAD-BALANCING
B
HOT HOST FILE RELOADING
HEALTH CHECKING
A
B
PERSISTENT CONNECTIONS
CONNECTION POOLING
B
A TALKS DIRECTLY TO B
RETRIES
COOPERATIVE MICROSERVICE INSTANCES
FROM INDEPENDENT WORKERS TO COOPERATIVE
232
INDEPENDENT
DEMAND HOSTS
AND WORKERS
COOPERATIVE
DEMAND WORKERS
ACROSS MANY
HOSTS
GOSSIP WITH ONE
ANOTHER AND MAINTAIN A
HASH RING OF EACH
WORKER
COOPERATIVE MICROSERVICE INSTANCES
WITH RINGPOP @ GITHUB.COM/UBER/RINGPOP
232
HASH WORKER ADDRESSES
> hash('10.31.1.2:9000')
53554892
> hash('10.31.8.9:9000')
1325776234
HASH APPLICATION IDS
> hash('33e2dc8c-16fd-4a19-9fad-4ebfc76c66c9')
2312992577
EACH DEMAND WORKER
OWNS A PORTION OF THE
KEYSPACE
> hash('8828169c-69c5-4b79-ae5e-6204c5f615ff')
2640491360
RELIABLE BACKGROUND OPERATIONS
WITH HASH RING TECHNOLOGY
210
232
VNODE
KEYSPACE
(OUTER RING)
FIXED
AND SMALLER
ENTITY
KEYSPACE
(INNER RING)
DYNAMIC
AND LARGER
RELIABLE BACKGROUND OPERATIONS
WITH HASH RING TECHNOLOGY
3
2
1
Riak
DEMAND A WORKER
RECEIVES DELIVERY &
INITIATES DISPATCH
DEMAND A WORKER
WRITES UUID TO VNODE
SET IN THE DB AND
STARTS TIMER
POST /jobs
hash(uuid) % 1024
Demand
A
DEMAND A WORKER
CRASHES BEFORE IT
EXPIRES DISPATCH
RELIABLE BACKGROUND OPERATIONS
WITH HASH RING TECHNOLOGY
5
4
6
Riak
0
DEMAND B DETECTS
MEMBERSHIP CHANGE IN
RING
1023
DEMAND B SCANS
ENTIRE VNODE
KEYSPACE
for vnode in range(0, 1023)
if lookup(vnode) == whoami()
restore(load_uuids(vnode))
DEMAND B LOADS VNODE SET
FROM DB AND RESTORES
BACKGROUND TIMERS
failure, monitoring
and alerting
CHAPTER 5 OF 8
FAILURE TESTING MICROSERVICES
WITH REPEATABLE FAILURE SCENARIOS
FAULT ISOLATION IN MICROSERVICES
WITH DEPLOYMENT PODS
POD 1
GEOFENCE
MICROSERVICE
FLIPR
MICROSERVICE
40.645244,
-73.9449975
POD 2
DEMAND v1
DISPATCH
GATEWAY
SUPPLY v1
POD 2
DEMAND v2
SUPPLY v2
MICROSERVICE ALERTING
WITH GRAPHITE/NAGIOS INTEGRATED THRESHOLD CHECKS
PER REPO THRESHOLDS
IMPORTED PYTHON
BUILT AGAINST GRAPHITE
ALERTS THROUGH NAGIOS
scalability and
sharding
CHAPTER 6 OF 8
PARTITIONING A MICROSERVICE
A SCALABLE GEOSPATIAL INDEX
1
2
864c244
3
> convert(40.645, -73.944)
“864c244”
> hash(“864c244”)
3747631425
READ
OR
WRITE
> lookup(3747631425)
“10.31.1.2:9000”
EARTH IS BROKEN UP INTO
CELLS. EACH CELL HAS AN ID.
GEOSPATIAL READS/WRITES
CONVERTS LAT/LNG TO CELL
ID. CELL ID IS THEN HASHED
ALONG RING.
REQUEST IS EITHER HANDLED
OR FORWARDED BY ONE OF
THE 1300 GEOSPATIAL INDEX
WORKERS.
LOCATION-AWARE MICROSERVICES
WITH CONTEXT-SPECIFIC METADATA
POST /pickup
POST /pickup
?lat=40.70&lng=-73.97
1
X-Uber-City-ID: 1
X-Uber-City-Name: New York
2
DISPATCH
GATEWAY
DEVICE SENDS PICKUP
REQUEST TO DISPATCH
GATEWAY
3
DISPATCH
GATEWAY
DISPATCH GATEWAY RESOLVES
LAT/LNG AGAINST GEOFENCE
SERVICE
SERVICE PROCESSES REQUEST
WITHIN CONTEXT OF CITY
performance and
diagnostics
CHAPTER 7 OF 8
HIGH-PERFORMANCE MICROSERVICES
WITH TCHANNEL @ GITHUB.COM/UBER/TCHANNEL
PERFORMANT
MULTIPLEXING
STREAMING
RETRIES + CIRCUIT BREAKING
POWERS RINGPOP
HIGH-PERFORMANCE MICROSERVICES
WITH NODESTAP @ GITHUB.COM/UBER/NODE-STAP
1
TORCH LIVE PROCESS
2
OPEN FLAMEGRAPH IN BROWSER
DEBUGGING MICROSERVICES
INSPECT INTERNALS WITH NODE REPL
1
CURL REPL ENDPOINT FOR REPL PORT
2
TELNET INTO REPL
3
INSPECT THE STATE OF YOUR WORKER
the next
generation
CHAPTER 8 OF 8
NEXT GENERATION MICROSERVICES
A OVERLAY NETWORK FOR MICROSERVICE ROUTING
REGISTER
REGISTER
DISCO
MICROSERVICE
OPTIC
MICROSERVICE
FORWARD
SEND
SEND
SUPPLY
MICROSERVICE
DEMAND
MICROSERVICE
ROUTING SERVICE
THANKS!
Presented by Jeff Wolski <[email protected]>
Uber is hiring!