2-7 Triple Draw Poker

Playing Draw Poker with Convolu4onal Neural Nets Nikolai Yakovenko 4/22/15 for EE6894 What is Draw Poker? •  Five-­‐card poker with one exchange •  10,000 machines in Las Vegas alone •  Pays out $0.95 to $1.007 per dollar*… *with perfect play •  Add photo of payout table 100% payout with perfect play. The Machine’s Edge What do you do here? Rule #7 Draw 3 to a Royal Flush! Worth $0.80 on average Worth $1.80 on average Easy to get 99.5% payout Just follow these 25 easy rules... No wonder people make mistakes Typical human plays 5% below expectaaon • 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
Four of a kind, straight flush, royal flush 4 to a royal flush Three of a kind, straight, flush, full house 4 to a straight flush Two pair High pair 3 to a royal flush 4 to a flush Low pair 4 to an outside straight 3 to a straight flush (type 1) AKQJ unsuited 2 suited high cards 4 to an inside straight with 3 high cards 3 to a straight flush (type 2) KQJ unsuited QJ unsuited JT suited KQ, KJ unsuited QT suited AK, AQ, AJ unsuited KT suited One high card 3 to a straight flush (type 3) Discard everything Can learning do becer? Data Representaaon All 32 sim results for hand [Kh,Ad,Kc,8h,Qc]:
[]:2000 sample: 0.27 ave 6.00 max
[4d,4h,Jh,5d,4c]
x23456789TJQKA
c..1..........
d..11.........
h..1......1...
s............. 4x13 binary matrix, each card 32-­‐length vector for all draws [Kh]:20000 sample:
0.33 ave 9.00 max
[Ad]:20000 sample:
0.45 ave 25.00 max
[Kc]:20000 sample:
0.33 ave 25.00 max
[8h]:2000 sample:
0.35 ave 25.00 max
[Qc]:20000 sample:
0.44 ave 50.00 max
[Kh,Ad]:2000 sample: 0.41 ave 9.00 max
[Kh,Kc]:2000 sample: 1.54 ave 25.00 max
…
[Kc,Qc]:20000 sample: 0.66 ave 976.00 max
[8h,Qc]:2000 sample: 0.34 ave 9.00 max
[Kh,Ad,Kc]:2000 sample:
1.38 ave 25.00 max
…
[Kh,Kc,8h]:2000 sample:
1.42 ave 25.00 max
[Kh,Kc,Qc]:2000 sample:
1.45 ave 25.00 max
…
[Kh,Ad,Kc,8h]:2000 sample: 1.23 ave 3.00 max
[Kh,Ad,Kc,Qc]:2000 sample: 1.21 ave 3.00 max
[Kh,Ad,8h,Qc]:2000 sample: 0.18 ave 1.00 max
[Kh,Kc,8h,Qc]:2000 sample: 1.21 ave 3.00 max
[Ad,Kc,8h,Qc]:2000 sample: 0.16 ave 1.00 max
[Kh,Ad,Kc,8h,Qc]:2000 sample:
1.00 ave 1.00
max
best result:
[Kh,Kc]:2000 sample:
1.54 ave 25.00 max Why Convoluaonal Network? •  Card games are visual •  Learn properaes like pairs, flushes, straights •  Proximity in the inputs macer [Kh,Qh,4h,3c,Jh]
[2d,2h,7c,6c,5c]
x23456789TJQKA
c.1...........
d.............
h..1......111.
s............. x23456789TJQKA
c...111.......
d1............
h1............
s............. Copy image ConvNet best pracaces Karen Simonyan & Andrew Zisserman explain… hcp://www.robots.ox.ac.uk/~karen/
pdf/ILSVRC_2014.pdf 5-­‐Card Draw Poker: Network Shape input layer shape 100 x 5 x 17 x 17 convoluaon layer l_conv1. Shape (100, 16, 15, 15) convoluaon layer l_conv1_1. Shape (100, 16, 13, 13) maxPool layer l_pool1. Shape (100, 16, 7, 7) convoluaon layer l_conv2. Shape (100, 32, 5, 5) convoluaon layer l_conv2_2. Shape (100, 32, 3, 3) maxPool layer l_pool2. Shape (100, 32, 2, 2) hidden layer l_hidden1. Shape (100, 1024) dropout layer l_hidden1_dropout. Shape (100, 1024) final layer l_out, into 32 dimension. Shape (100, 32) Training Simple Nuanced •  Learn all 32 outputs •  Loss = mean squared error •  Update with Nesterov Momentum •  Bias toward rare cases •  Round off large values –  Easily gets to 67% accurate moves –  99% [0.0 – 4.0] –  <1% [10.0 – 900.0] •  Switch to adapave learning –  AdaDelta works well –  Iniaalize working model Results! Real Return (100k hands) Valida4on% 15x15 same-­‐shape 50k training size $0.930 73% 17x17 valid-­‐shape 90k training size $0.944 70% 17x17 valid-­‐shape 150k training size $0.955 78% 17x17 valid-­‐shape 90k training (longer training) $0.983 77% Don’t trust the averages, since hugely asymmetric payoff. Study your mistakes Biggest Errors 500 hands took 2397.96s
406 no error 52 tiny error
25 small error
17 big error
biggest errors:
(1.14,
(1.12,
(0.85,
(0.83,
(0.82,
(0.81,
(0.75,
(0.71,
…
(0.41,
'[4s,7h,Ah,3h,Jh]',
'[Ts,Kc,Jc,3h,Qc]',
'[Tc,2s,9c,8d,8c]',
'[3d,Td,9s,Ad,8d]',
'[6d,2c,3c,Tc,9c]',
'[Jc,9h,Kh,Ah,2h]',
'[2h,Qs,Jh,Kh,8h]',
'[9s,Th,As,4s,6s]',
'[7h,Ah,3h,Jh]', 1.26, '[4s,7h,Ah,Jh]', 0.12)
'[Kc,Jc,Qc]', 2.015, '[Ts,Kc,Jc,Qc]', 0.897)
'[8d,8c]', 0.852, '[Tc,2s,9c,8c]', 0.0)
'[3d,Td,Ad,8d]', 1.272, '[Td,Ad,8d]', 0.442)
'[2c,3c,Tc,9c]', 1.167, '[]', 0.340)
'[9h,Kh,Ah,2h]', 1.278, '[Jc,Kh,Ah]', 0.465)
'[2h,Jh,Kh,8h]', 1.261, '[Jh,Kh]', 0.513)
'[9s,As,4s,6s]', 1.229, '[As]', 0.511)
'[2s,Jh,Kd,Tc,Qs]', '[Jh,Kd,Tc,Qs]', 0.872, '[Jh,Kd,Qs]', 0.463) Struggling with straights, flushes, straight flushes. Will it learn with more ame? With becer examples? Lessons Learned Do Don’t do •  Keep network lean •  Endlessly fiddle with network shape •  Fiddle with learning rate •  Permute input data –  Deep but simple •  Use adapave learning –  Iniaalize with working model •  Train for a long ame •  Bias toward difficult data –  Becer to get fresh samples Digital or Analog? DeepMind’s Atari AI, acempang to approximate an exact value. Neural Nets do bad “Exact Math” for games Obvious Improvements •  Train on errors –  Directly, or look for similar cases –  Generate more data, permute known cases •  Run much longer, on more data –  Total training: 400k cases, down-­‐sampled to 150k •  Train mulaple models, and vote on result Beyond Draw Video Poker •  Different video payout –  Start training on current model •  Triple Draw –  3 rounds, so train 3 models –  Same network shape, same output –  One big model? •  Incorporate be|ng, opponent hand informaaon Quesaons •  Different network shape? •  How to handle input padding? •  Retrain on rare cases? –  Or a specialty network? –  (Backgammon AIs include 2-­‐5 different networks) Thank you! Bibliography •  GitHub: hcps://github.com/moscow25/deep_draw –  Ping me if you want to run it. Needs bit of cleanup. •  Lasagne: hcp://lasagne.readthedocs.org/en/latest/ •  Network shape (for images) –  hcp://www.robots.ox.ac.uk/~karen/pdf/ILSVRC_2014.pdf –  hcp://vision.stanford.edu/teaching/cs231n/slides/lecture8.pdf •  AdaDelta: hcp://www.machewzeiler.com/pubs/googleTR2012/googleTR2012.pdf •  DeepMind Atari: hcp://www.nature.com/nature/journal/v518/n7540/full/
nature14236.html •  PokerSnowie: hcps://www.pokersnowie.com/about/weaknesses.html •  Wizard of Odds: hcp://wizardofodds.com/games/video-­‐poker/tables/jacks-­‐or-­‐becer/