Presentation - 32nd International Conference on Massive Storage

Presentation - 32nd International Conference on Massive Storage

Incremental Redundancy to Reduce
Data Retention Errors in Flash-based SSDs
31st Interna+onal Conference on Massive Storage Systems and Technology Heejin Park, Jaeho Kim, Jongmoo Choi, Donghee Lee, Sam H. Noh
University of Seoul, Dankook University, Hongik University
1 Outline
§ 
§ 
§ 
§ 
Introduction & Approach
Estimate Error and Period
Effective OPS size and Hot/Cold Separation
Experiments & Conclusion
Massive Storage Systems and Technology (MSST’15) 2 Flash-based SSD
•  Flash based SSD is widely used in many areas
–  Computer system
–  Mobile devices
•  Why flash memory?
– 
– 
– 
– 
– 
Superior performance
Low power consumption
Shock resistance
Small size
Light weight
•  Multi bit-cell
–  Low price
–  Reliability issues are being magnified
Massive Storage Systems and Technology (MSST’15) 3 Reliability of SSD
retenTon errors From: Hagop Nazarian and Sylvain Dubois, Crossbar “The drive for SSDs: What’s holding back NAND flash?” From: Cai, et al, “Error PaRerns in MLC NAND Flash Memory: Measurement, CharacterizaTon, and Analysis”, DATE’ 12 •  Manufacturing process is being developed
–  Endurance decreases
–  High bit error rate (short retention period)
–  More serious in MLC & TLC
•  Types of error in NAND flash based SSDs
–  Program error
–  Read error
–  Retention error (dominant error)
ß Our work is focused on here Massive Storage Systems and Technology (MSST’15) 4 Cause of Retention Error
Floating gate
Control Gate
Source
Channel
• 
Drain
Retention error
–  Errors due to long data retention period
–  Leaking of injected electrons
–  Shifting of threshold voltages
Traps charges in floating gate
11 10 01 00 Vth REF 0 REF 1 REF 2 Massive Storage Systems and Technology (MSST’15) 5 Problem of Retention Error
ECC
01011001
01010001
00101111 00101101 Earlier stage of SSD ECC
01011001
01110001
00101101 01000011 00001111 Can not correct!
Later stage of SSD Massive Storage Systems and Technology (MSST’15) 6 Existing Solutions
ECC
01011001
01110101
00101101 01000011 OOB
•  Strong ECC
Page
–  Reinforce the error correction capability
•  Require considerable OOB space
•  The errors in cold data can’t be corrected
•  Hurts performance and energy efficiency
Massive Storage Systems and Technology (MSST’15) Page
ECC
ECC
OOB
7 Existing Solutions (Cont.)
page with errors 10110001111010
ECC
new page 10111001011010
ECC
read and correct periodically monitoring
10111001011010 ECC
10110001111010
write other page •  Data scrubbing
–  Rewrite data after reading and correcting by using ECC
•  Extra reads, writes, and erase operations
•  Hurts performance and lifetime
Massive Storage Systems and Technology (MSST’15) 8 (Log scale)
Raw Bit Error Rate Observation
1.0E-­‐03 1.0E-­‐04 Correctable by ECC Errors are low when..
- lower P/E cycles
- lower retention period
1.0E-­‐05 1.0E-­‐06 P/E cycles 1500 6000 1.0E-­‐07 1.0E-­‐08 0 100 200 3000 12000 300 400 Time (day)
From : Cai, et al. “Error PaRerns in MLC NAND Flash Memory: Measurement, CharacterizaTon, and Analysis,” in Proc. DATE ’12 •  Retention errors increases as
–  P/E cycles increases
–  Time goes on
Massive Storage Systems and Technology (MSST’15) 9 Our Approach
ECC
01011001
01010001
00101111 00101101 Earlier stage of SSD ECC
Parity
01110001
01011001
01010001
00101101 00001101 01000011 Later stage of SSD •  Write parity considering P/E cycles & reten>on period Massive Storage Systems and Technology (MSST’15) 10 Our Approach (Cont.)
1.0E-­‐08 UPER 1.0E-­‐12 Safe level (1.0E-­‐15)
UPER
1.0E-­‐16 1.0E-­‐20 1.0E-­‐24 1.0E-­‐28 Scrubbing
1.0E-­‐32 1.0E-­‐36 0 Safe period
Data
200 400 600 Time (day)
800 1000 Data
Rewrite by scrubbing •  Safe level : error rate level which is considered unsafe (1.0E-­‐15) •  Safe period : duraCon of the error rate under safe level Massive Storage Systems and Technology (MSST’15) 11 Our Approach (Cont.)
1.0E-­‐08 UPER 1.0E-­‐12 Safe level (1.0E-­‐15)
1.0E-­‐16 UPER
UPER Parity wriBen (IR)
1.0E-­‐20 1.0E-­‐24 1.0E-­‐28 Scrubbing
Scrubbing
1.0E-­‐32 1.0E-­‐36 0 Safe period
Extended safe period
200 400 Delayed scrubbing Data
Parity
Data
600 Time (day)
800 1000 Parity
Data
Rewrite by scrubbing •  Incremental Redundancy (IR)
–  When error rate reached at safe level, write parities dynamically
–  Incrementally reinforce the data recovery capability
Massive Storage Systems and Technology (MSST’15) 12 How to apply parity?
Massive Storage Systems and Technology (MSST’15) 13 IR with Dynamic Vertical Stripe
Chip 0
•  Construct stripe dynamically
–  Reserve some pages
SSD controller
Host
interface
FTL
Chip 1
IR scheme
Chip 2
Chip 3
…
…
…
…
D0
D4
D8
D12
P0
D1
D5
D9
D13
P1
D2
D6
D10
D14
P2
D3
D7
D11
D15
P3
…
…
…
…
D16
D20
D24
D28
D32
D17
D21
D25
D29
D33
D18
D22
D26
D30
D34
D19
D23
D27
D31
D35
…
…
…
…
: Valid page
Px
Dx
: Invalid page
: Parity page
•  Reserved Pages
–  Hot Block
•  Handle write requests
–  Cold Block (stripe)
•  Write parities when safe period expires •  Reinforce error correct capability
•  Low overhead
–  Parity update cost
–  GC cost
: Reserved page
Massive Storage Systems and Technology (MSST’15) 14 Operation of Vertical Stripe
PBN 0 PBN 0 New block D0
D4
D8
D12
P0
D0
D4
D8
D12
P0
D0`
D8`
•  Write operation
– 
– 
– 
– 
Invalidate old data
New data is written to new page
Old data still exist until GC operation
Parity does not need to be updated
Update D0, D8 PBN 0 PBN 0 New block D4
D12
D0
D4
D8
D12
P0
Erase PBN 0 •  GC operation
– 
– 
– 
– 
– 
Unit of victim is block
Valid data are migrated
Parity & invalid data disappear
Can be done as general GC
Does not need to reconstruct stripe
Massive Storage Systems and Technology (MSST’15) 15 Outline
§ 
§ 
§ 
§ 
Introduction & Approach
Estimate Error and Period
Effective OPS size and Hot/Cold Separation
Experiments & Conclusion
Massive Storage Systems and Technology (MSST’15) 16 Analysis of Error Rate
Raw Bit Error Rate
1.0E+00 1.0E-­‐01 100 6000 1.0E-­‐02 P/E cycles 1500 12000 3000 24000 •  Estimate raw bit error rate
1.0E-­‐03 1.0E-­‐04 1.0E-­‐05 –  Key factors of aggravating retention error
P/E cycles Time (day)
–  Make formula form via curve fitting
1.0E-­‐06 1.0E-­‐07 dr : deterioraCon rate funcCon c : P/E cycles d : day 1.0E-­‐08 0 100 200 300 400 Time (day)
•  Compare error rate before and after striping
• 
Uncorrectable Page Error Rate
• 
Uncorrectable Page Error Rate with stripe
-­‐ CPER : Correctable Page Error Rate -­‐ CSER : Correctable Stripe Error Rate Massive Storage Systems and Technology (MSST’15) 17 Safe Period & CG Period
•  Safe period
–  Safe period (Tsafe) : duration of the error rate under safe level
–  Extended safe period (Tesafe) : safe period with parity
Data
Data
Parity
Safe period
Time Extended safe period
-­‐ RBER threshold makes UPER = safe level(1.0E-­‐15) •  GC period
–  Tgc : duration until data will be selected as victim of GC
Data
Time GC period
Selected as vicTm of GC Pu : update probability u : uClizaCon Massive Storage Systems and Technology (MSST’15) 18 When the parity should be written?
•  Write parity when
–  Error rate exceeds safe level before garbage collection
–  Data retention period reaches at safe period
Massive Storage Systems and Technology (MSST’15) 19 Safe Period vs. GC Period •  GC period < safe period
Safe period
GC period
Time Garbage collection
Data
…
u : uClizaCon of block •  GC period > safe period
Data scrubbing
Data
Dwrite : writes by host Dscrub : writes by scrubbing …
Safe period
GC period
Time IR can be used to extend safe period Massive Storage Systems and Technology (MSST’15) 20 Extended Safe Period vs. GC Period •  GC period < extended safe period
Extended safe period
GC period
Time Garbage collection
Data
…
u : uClizaCon of block considering space of pariCes •  GC period > extended safe period
Data scrubbing
Data
…
Extended safe period
GC period
Dwrite : writes by host Dscrub : writes by scrubbing Dparity : parity writes Massive Storage Systems and Technology (MSST’15) Time 21 Outline
§ 
§ 
§ 
§ 
Introduction & Approach
Estimate Error and Period
Effective OPS size and Hot/Cold Separation
Experiments & Conclusion
Massive Storage Systems and Technology (MSST’15) 22 Wasted OPS
Free Block Pool Data 0.2 Scrubbing before GC Used Block Pool ….. Safe period
GC period
Wasted a
Write Wday= 1% 0.15 0.1 0.05 0 Time 4k 5k 6k 7k 8k 9k 10k P/E cycles
•  OPS (Over Provisioning Space)
–  The rest of total space except user space
•  GC period > safe period (or extended safe period)
–  Data scrubbing generates free space before GC operation
–  OPS can not be fully used
•  In case of Hot/Cold Separation, wasted OPS can be reassigned
Massive Storage Systems and Technology (MSST’15) 23 Configuration of Hot/Cold
(1-­‐r) ·∙ Total Writes r ·∙ Total Writes Cold area
Hot area
(1-­‐s) * User Space s ·∙ User Space OPS needs to be separated into 2 parts for hot and cold
We initially set optimal o which is found by Desnoyers
From: Desnoyers, “Analytic Models of SSD write Performance,” ACM Transactions on Storage
Cold OPSOPS Area Hot OPS
(1-­‐o) ·∙ OPS o ·∙ OPS Tune Then tune the o as effective ops size which is suited to safe period
Massive Storage Systems and Technology (MSST’15) 24 OPS Tuning
Scrubbing Safe period
GC period
Safe period < GC period WAFcold_scrub Garbage collec>on GC period < Safe period WAFhot_GC Cold area Safe period
GC period
Hot area wasted OPS Cold OPS
Hot OPS
(1-­‐o) ·∙ OPS o ·∙ OPS Tune Cold OPS
Hot OPS
effecTve OPS o ·∙ OPS + wasted OPS •  If scrubbing occurred one area
–  Wasted OPS can reassigned the other area
Massive Storage Systems and Technology (MSST’15) 25 Definition of WAF
(1-­‐r) ·∙ Total Writes r ·∙ Total Writes Cold area
Hot area
(1-­‐s) * User Space Hot Cold GC GC GC Scrub Scrub GC Scrub Scrub s ·∙ User Space WAF Massive Storage Systems and Technology (MSST’15) 26 Outline
§ 
§ 
§ 
§ 
Introduction & Approach
Estimate Error and Period
Effective OPS size and Hot/Cold Separation
Experiments & Conclusion
Massive Storage Systems and Technology (MSST’15) 27 Experiment Setup
•  Page-mapping FTL simulator
–  With data scrubbing
•  Workloads
–  Synthetic workload (Random & Hot/Cold)
–  Real workload from MSR cambridge
–  Wday means the ratio of updated data per day in user data space.
•  Setting
– 
– 
– 
– 
– 
– 
SSD capacity = 128GB, user space = 80% (initialized)
Page size = 4k
Stripe size = PPB (Page Per Block) = 128
ECC (correct 8 bits / detect 16 bits per codeword)
Page is composed of 8 codewords
GC policy = LRW (Least Recently Written)
Massive Storage Systems and Technology (MSST’15) 28 Scrub 16 Scrub 16 Scrub 12 IR 12 IR 12 IR 8 4 WAF 16 WAF WAF Random workload
8 4 0 8 4 0 0 2K 3K 4K 5K 6K 7K 8K 9K 10K P/E cycles 2K 3K 4K 5K 6K 7K 8K 9K 10K P/E cycles 2K 3K 4K 5K 6K 7K 8K 9K 10K P/E cycles ( Wday = 1% ) ( Wday = 0.5% ) ( Wday = 0.25% ) •  As data are less updated
Scrub : convenConal scrubbing IR : our proposal –  GC period is more longer
–  The safe period expires on lower P/E cycles (GC period > Safe period)
–  WAF increases on earlier stage
•  As P/E cycles increases
–  WAF continuously increases (safe period becomes shorter)
Massive Storage Systems and Technology (MSST’15) 29 Scrub 16 Scrub 16 Scrub 12 IR 12 IR 12 IR 8 WAF 16 WAF WAF Hot/Cold Workload
8 8 4 4 4 0 0 0 2k 3k 4k 5k 6k 7k 8k 9k 10k P/E cycles 2k 3k 4k 5k 6k 7k 8k 9k 10k P/E cycles 2k 3k 4k 5k 6k 7k 8k 9k 10k P/E cycles ( Wday = 1% ) ( Wday = 0.5% ) ( Wday = 0.25% ) •  Trends are similar to random workload
–  As Wday is small, GC period become longer
•  WAF starts to soar for Scrubbing, but for IR
•  IR can release additional writes by scrubbing
Massive Storage Systems and Technology (MSST’15) 30 Real Workload
WAF 10 model(Scrub) 8 sim(Scrub) 6 model(IR) sim(IR) 4 2 0 hm0 mds0 mds1 rsrch0 stg0 stg1 wdev0 web0 Workload •  Configuration
•  6K P/E cycles
•  Derive parameters (s, r, Wday)
•  Results of simulation and model are similar
•  WAF decreases when IR is applied
•  Extended safe period
•  Decrease additional writes by scrubbing
•  More opportunities to be refreshed by writes
Name hm0 mds0 mds1 rsrch0 stg0 stg1 wdev0 web0 s 0.014 0.002 0.002 0.002 0.004 0.002 0.003 0.006 r 0.990 0.986 0.824 0.991 0.991 0.979 0.991 0.992 Avg. Wday 3.19% 1.09% 0.22% 1.54% 2.20% 0.84% 1.03% 1.70% Effect of OPS tuning
10
8
0.6
6
0.4
4
2
0.2
2
0
0
8
0.6
6
0.4
4
0.2
0
OPS ratio
0.8
0.8
2K 3K 4K 5K 6K 7K 8K 9K 10K
1
2K 3K 4K 5K 6K 7K 8K 9K 10K
P/E cycles
( Wday = 1% ) WAF
10
WAF
OPS ratio
1
0
P/E cycles
( Wday = 0.5% ) •  Scrubbing occurs in cold area optimally assigned OPS wasted
•  WAF can be lowered by reassigning wasted OPS
–  Decreases WAF of hot area
•  When the scrubbing also occurs in hot area
–  Total OPS cannot fully used
Massive Storage Systems and Technology (MSST’15) 32 Conclusion
•  Retention error is a serious issue and must be handled
•  IR writes parity dynamically to
–  Reinforce data correct capability
–  Extend the safe period of data
•  IR can be a promising solution
–  With little overhead for striping
–  Data scrubbing operation can be delayed
–  Additional writes by scrubbing can be reduced
Massive Storage Systems and Technology (MSST’15) 33 Thank you
Massive Storage Systems and Technology (MSST’15) 34