Kingston Server Memory

Kingston Technology
Server Architecture and
Kingston Memory Solutions
Ingram Micro
May 2015
Mike Mohney
Senior Technology Manager, TRG
©2015 Kingston Technology Corporation. All rights reserved.
All trademarks and registered trademarks are the property of their respective owners.
Kingston Server Upgrades
• Introduction to Kingston
• Kingston Solutions and Memory Roadmap
• Memory Configurations for Intel server platforms
2
Kingston Snapshot
• Founded in 1987, 27+ years
• Privately Held Company
• Financially Secure
• For 8 years , revenue exceeding $4B
• 4,000 Employees Worldwide
• Worldwide Manufacturing & Testing
• Focused Business:
Memory Modules, Flash Products & Tech Solutions
• Delivering over 1M units per day
• Products distributed in 125 Countries and available in over 30,000 locations
worldwide; widely accepted brand
3
Source: Kingston Technology
Kingston Global Infrastructure
• Manufacturing Centers are all ISO 9001:2008 certified
• 2014 worldwide capacity: Over 30 million units per month
• Over 50 SMT manufacturing lines in 4 global facilities, producing both DRAM & Flash products
*
* 4 Kingston Technology Manufacturing Facilities Worldwide
** Panram and OSE are subcontractors to Kingston
4
Source: Kingston Technology, June 2014
*
*
*
**
**
Kingston Investments & Expertise
Kingston manages all six processes required to build modules, cards, and drives.
Wafer Production
Wafer Testing
Wafer Processing/Chip Production
Component Testing
Module, Card & Drive Production
Module, Card & Drive Testing
5
Strategic Alliances: Server Compatibility Check
Kingston works with many facets of technology to ensure server memory compatibility.
Industry Leading OS,
Software, Hardware
6
PC OEM’s
Motherboard Manufacturers
Intel and Kingston
DDR3 DRAM chip validation testing on
new Advantest tester (DDR4 capable)
Checking early FB-DIMM prototype
against specifications
Since 1998, Intel and Kingston have jointly developed memory technology validation
testing processes for PC133, RDRAM, DDR, DDR2, FB-DIMM, DDR3, LRDIMM, and DDR4.
7
Kingston Performance Upgrades
• Introduction to Kingston
• Kingston Solutions and Memory Roadmap
• Memory Configurations for Intel server platforms
8
Reliability: The Challenges of Memory Testing
Memory Capacity
Number of Cells
1GB
Over 8.5 billion cells
2GB
Over 17 billion cells
4GB
Over 34 billion cells
8GB
Over 68 billion cells
16GB
Over 136 billion cells
Kingston Quality Standard: Zero bad cells
1 bad cell = defective module
9
Semiconductor Reliability
Number Of Field Failures (in units)
This curve of semiconductor failures
over time is called a ‘bathtub’ curve
Early-Life
Failures
Useful Life Of Semiconductor
A typical useful life would be 20 years or more
End-of-Life
Failures
Failure curve for lower-quality
semiconductor products
Time
Ship out from
Factory to user
10
Typically the first few
months of usage
Kingston Production Testing and Dynamic Server Burn-In
100% Testing
• No spot testing! Kingston memory is 100% tested prior to leaving
our factories, as we have always done
Dynamic Server Memory Burn-In
• Shrinking DRAM lithography increases sensitivity to temperature,
forcing higher error rates and exposing weak memory cells
• Kingston’s advanced burn-in process screens these by simulating 3
months of heavy server use at higher temperatures under full
system environment and frequency stress
Proprietary Production Testers
• Kingston’s test engineers designed the first automated production
testers in the memory industry
• New patented automated testers build upon our test engineering
legacy to meet the more stringent demands of evolving memory
technology
35 Quality Control Checkpoints
• Kingston uses AQL sampling methods to re-inspect and re-test
every lot to ensure our quality standards are being met
11
Kingston holds 29 patents
related to memory testing
Memory Roadmap (1987 – 2017)
19871990
FAST
PAGE
MODE
EDO
19971999
PC66
PC100
PC133
2000
DDR200
DDR266
RDRAM DDR333
PC600 DDR400
PC700 RDRAM
PC800 PC1066
Technology
DDR3 (1.5V)
DDR3L (1.35V)
DDR3U (1.25V)
DDR4 (1.2V)
1
2
20012003
Data Rate
20042005
2006
DDR4-2400
20072009
20102013
2014
2015
DDR3L 1.35V
DDR4-2133
DDR2-800
DDR3-1600
FB-DIMMs
DDR3L-1600
DDR2-400
DDR3-800 DDR3-1866
DDR2-533
DDR3-1066 LRDIMM
DDR2-667
DDR3-1333
Module Classification
Peak Bandwidth
800
DDR3-800 / PC3-6400
6400 MB/s or 6.4 GB/s
1066
DDR3-1066 / PC3-8500
DDR3L-1066 / PC3L-8500
8500 MB/s or 8.5 GB/s
1333
DDR3-1333 / PC3-10600
DDR3L-1333 / PC3L-10600
10600 MB/s or 10.6 GB/s
1600
DDR3-1600 / PC3-12800
DDR3L-1600 / PC3L-12800
12800 MB/s or 12.8 GB/s
1866
DDR3-1866 / PC3-14900
14900 MB/s or 14.9 GB/s
2133
DDR4-2133 / PC4-2133
17000 MB/s or 17 GB/s
2400
DDR4-2400 / PC4-2400
19200 MB/s or 19.2 GB/s
2666
DDR4-2666 / PC4-2666
21300 MB/s or 21.3 GB/s
2933
DDR4-2933 / PC4-2933
23400 MB/s or 23.4 GB/s
2016
DDR4-2666
2017
DDR4-2933
New Module Key Location
Form Factors
288-pin DIMM
260-pin SODIMM
Performance
1600MT/s, 1866MT/s, 2133MT/s, 2400MT/s, 2666MT/s, 2933MT/s
4000MT/s (possible extension)
4266MT/s (possible extension)
Power
1.2V with improved refreshes, ½K page for x4
12V for NVDIMM (4-pins)
DRAM Densities/DIMM Capacities
4Gb, 8Gb, 16Gb (x4, x8, x16)
Future DIMM capacities of 64GB, 128GB, 256GB
Slightly thicker for
improved performance
New Features
3D chip stacking support (up to 8 stacked dies)
Point-to-Point Memory Channel addressing
More stable than DDR3 with improved error correction code
Curved edge for better
socket connection
13
“Low Voltage” (LV)
DDR3L = Low Voltage 1.35V JEDEC (industry spec)
• Modules are classified as PC3L-xxx
Reducing memory power consumption 10% by dropping module voltage
requirement, lowering energy costs to data centers
Kingston DDR3L modules are dual-voltage supporting 1.35V / 1.5V
DDR3U (1.25V)
Ultra Low Voltage
Supported on current and legacy server platforms
• Intel SNB/IVB/HSW (Xeon E3+ Series)
• Intel WSM-EP/SNB-EP/IVB-EP/HSW-EP (Xeon 5600/E5 v1/v2 Series)
• Intel WSM-EX/IVB-EX (Xeon E7/v2 Series)
• AMD Magny-Cours+ (Opteron 6100/6200/6300 Series)
• AMD Lisbon+ (Opteron 4100/4200/4300 Series)
Supported on mobile/desktop platforms from Intel and AMD
• Skylake (DDR3L Only), Broadwell, Haswell, & Ivybridge (Core i5 / Core i7)
• AMD A-Series
14
JEDEC Spec
for Mobile, Server
Low Voltage spec
for DDR4 deemed
not possible by
DRAM
semiconductors
Load-Reduced DIMMs for Servers
JEDEC Industry Standard Specification
LRDIMMs overcome the impact of Ranks (Loads) upon memory speed
• Parallel memory bus impacted by memory module Ranks
• Enable greater memory capacity and speeds while reducing “load”
• Require special buffer chips
Eight-Rank modules now possible with less loading on memory bus
• Will enable faster 32GB, 64GB, 128GB LRDIMMs
Supported on 2012+ Intel and AMD servers
Included in DDR4 spec with new, more efficient design – Q1 2015
Kingston launched Q1 2013
DDR3 LRDIMMs
KVR13LL9Q4/32
KVR16LL11Q4/32
KVR18L13Q4/32
DDR4 LRDIMMs
KVR21L15Q4/32
KVR21L15Q4/64
15
Source: Intel IDF, September 2011
Coming soon
Load-Reduced DIMMs for DDR4 – New Design, Reduced Latency
16
Source IDT: www.idt.com/document/whp/ddr4-lrdimms-both-memory-capacity-and-speed
Non-Volatile DIMM (NVDIMM)
What Is It?
A non-volatile memory module directly accessed by the memory controller (approximately
10x faster than PCIe interface). Silicon based memory technology is reaching its
perfomance and efficiently limit.
Types of NVDIMM standards
NVDIMM-F – An all FLASH memory module accessible by the memory controller.
NVDIMM-N – A DRAM memory module made persistent by the use of NAND flash silicon.
NVDIMM-N2 – Similar to NVDIMM-N but with system accessible Flash.
NVDIMM-M – MRAM based memory module accessible by the memory controller.
NVDIMM-x – New technology. PRAM? RRAM? HMC?
Viking ArxCis-NV DDR3 NVDIMM
17
Source: Google Images
Diablo/SanDisk ULLtra DIMM
SK Hynix 16GB DDR4 NVDIMM
Kingston Memory Product Lines
ValueRAM
o Generic / JEDEC Specification
o Ideal for system builders or small
datacenters
OEM / Private Label
System Specific
o Also known as “Branded”
o Guaranteed compatibility
o Ideal for users who want to upgrade
OEM machines
HyperX
o Customized in box solutions
o Enthusiast gaming market
o Contracted supply agreements
o Overclocking
o Brand focused
18
System Specific Memory
Guaranteed Compatible
• Testing conducted in the system in which the memory will be installed
Importance of Branded Memory
• Different chips provide varying performance levels
• Chip Compatibility can vary
• Possible constraints in module height or width
• Presence Detect Configuration
19
ValueRAM Product Decoder
• Kingston ValueRAM is built to industry standard specifications outlined by J.E.D.E.C., the Joint Electron
Device Engineering Council, on which Kingston holds a seat. Intel, AMD, and other computing chipset
architects adhere to these standards for industry agreement and cross-platform compatibility.
• J.E.D.E.C. provides the specifications for Pin Count, Form Factor, Speed, Power, and Memory
Technology Types
20
ValueRAM Server Focused Programs
Intel Certified intended specifically for customers who are using Intel branded server/workstation
motherboards or systems
• Qualified and supported by Intel
• Controlled builds (BOMs)
• PCNs with 45-90 Day Notice
Server Premier intended for server builders with fully controlled BOM requirements (PCB, chip
brand/die revision, etc) regardless of brand of motherboard used
• Fully BOM controlled
• PCNs with 45-90 Day Notice
• Easily manage what is ordered
• No price premium
21
ValueRAM Qualifications
In addition to the strategic partnerships Kingston has with ODMs for submitting memory
for qualification, we also work with independent memory test labs AVL and CMTL Labs
for specific platform, motherboard, or system qualification. Test certificates are
available on our website.
http://www.kingston.com/us/memory/valueram/server
Example:
22
Module Types For Server Platforms
Registered DIMM
Load Reduced DIMM
ECC Unbuffered DIMM
Registered DIMM
Load Reduced DIMM
ECC Unbuffered DIMM
288-pin DIMM
240-pin DIMM
• Optimizing Memory:
• DDR4 is NOT compatible with DDR3 based systems (module will not fit)
• Install memory in kits of identical modules according to memory architecture
• Configure all processors the same way for most bandwidth
• Do Not Mix Module Types within a server.
• Mixing DDR3 (1.5V) and DDR3L (1.35V) will default the memory to 1.5V.
23
DRAM Chip Types Used in Servers
Multi Bit Error
Detection and
Correction
Registered DIMMs
x4
LRDIMMs
1
2
3
4
Registered DIMMs
x8
Single Bit Error
Detection and
Correction
24
LRDIMMs
Unbuffered ECC DIMMs
Unbuffered DIMMs
1 234 5 678
SODIMMs
64-bit Memory Loads (aka Ranks)
• Ranks are virtual modules on a single DIMM,
designed to save space on the motherboard
Single Rank (1R)
• Only one Rank is active at a time, the other ranks
on the DIMM go into standby saving power
• 2R and 4R DIMMs consume less power than
1R DIMMs
64bits
• Memory lanes on the
motherboard become
congested with more memory
loads, so server systems are
limited by the number of ranks
that can be used, as the memory
slows down with more ranks
populated, just line cars in
heavy traffic
Dual Rank (2R)
64bits
64bits
Quad Rank (4R)
64bits
25
64bits
64bits
64bits
Why Add Channels and Increase The Memory Speed?
DDR3 & DDR4 Aggregate Server Memory Bandwidth
120
100
80
GB/s
DDR3-800
60
DDR3-1066
40
DDR3-1333
DDR3-1600
20
DDR3-1866
DDR4-2133
0
Single Channel
(1 DIMM per CPU)
2 x 16GB = 32GB
26
Dual Channel
(2 DIMMs per CPU)
4 x 16GB = 64GB
Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel DDR3 and DDR4 platforms
Quad Channel
(4 DIMMs per CPU)
8 x 16GB = 128GB
Memory Configuration Options by Processor
Memory Supported
Memory Configuration
Xeon 5500 Series
DDR3-1333
Triple-Channel = K3 kits
Xeon 5600 Series
DDR3-1333 / DDR3L
Triple-Channel = K3 kits
Xeon E5 Series
DDR3-1600 / DDR3L-1333
Quad-channel = K4 kits
Triple-channel = K3 kits
Xeon E5 v2 Series
DDR3-1866 / DDR3L-1600
Quad-channel = K4 kits
Triple-channel = K3 kits
Xeon E5 v3 Series
DDR4-2133
Quad-channel = K4 kits
Xeon 6500 / 7500 / E7 Series
DDR3-1066
DDR3L-1066 on E7 Series
Dual-channel = K2 / K4 kits
Xeon E7 v2 Series
DDR3-1600 / DDR3L-1600
Quad-channel = K4 kits
Xeon W3500 / 3600 / E3
Xeon E3 v2 / v3 Series
DDR3-1066 / DDR3L (W3600)
DDR3/L-1600 / DDR3-1866
Dual-channel = K2 / K4 kits
Opteron 8400 / 2400 Series
DDR2-800
Dual-channel = K2 kits
Opteron 6100 - 6300 Series
DDR3 / DDR3L 1333-1866*
Quad-channel = K4 kits
Opteron 4100 - 4300 Series
DDR3 / DDR3L 1333-1866*
Dual-channel = K2 kits
* Subject to motherboard compatibility
27
Kingston Performance Upgrades
• Introduction to Kingston
• Kingston Solutions and Memory Roadmap
• Memory Configurations for Intel server platforms
28
Microarchitecture Roadmap
1600
29
1866
2133
2400
Intel Xeon E5 / E5 v2 / E5 v3 “EP” Series Server Memory Configuration
HP ProLiant Gen 8 / Gen 9 ®
IBM System xSeries M4 ®
30
Picture sources: Google images
Cisco UCS M3 ®
Dell PowerEdge ®
Intel Xeon E5 Series Summary
31
Source: Intel public marketing
Intel Xeon E5-2600/4600 v1/v2/v3 Architecture
Quad Channel – up to 12 DIMMs per CPU
(24 per 2-CPU System, 48 per 4-CPU System)
32
Intel Xeon E5-2400 v1/v2 Architecture
Triple Channel – up to 6 DIMMs per CPU (12 per 2-CPU System)
33
Haswell-EP
Intel Xeon E5-2600 v3 DDR4 Memory Speeds
1.2V
1 DIMM per
Channel
2 DIMMs per
Channel
3 DIMMs per
Channel
1 DPC
2 DPC
3DPC
Registered DIMM (RDIMM)
2133
1866
1600
Load Reduced DIMM (LRDIMM)
2133
2133
1600
Module Type
(No Mixing in a Server)
2133
1866
1600
34
Processor Model
Determines Maximum
Memory Speed
Optimizing Intel Xeon E5 v3 Servers
DDR4
Ch 3
Server
Intel S2600WT
Ch 4
3DPC
1
2
Ch 2
1DPC
3
1
2
Unbalanced
Configuration
(128GB)
3
2
Ch 3
Ch 1
1DPC
3
16 x KVR21R15S4/8
8GB DDR4-2133 RDIMMs 1Rx4 1.2v
128GB
3DPC
1
3
2
Ch 4
3DPC
1
1
2
Ch 2
1DPC
3
1
2
Ch 1
1DPC
3
3
E5-2650v3
E5-2650v3
CPU1
CPU2
2
3DPC
1
3
2
1
Memory Bandwidth
67.464 GB/s
31.41%
Ch 3
(30.89 GB/s)
2DPC
1
Balanced
Configuration
(128GB)
Ch 4
2
Ch 2
2DPC
3
1
2
2DPC
3
3
2
Ch 3
Ch 1
2DPC
1
3
2
Ch 4
2DPC
1
1
2
2DPC
3
1
2
E5-2650v3
CPU1
CPU2
Memory Bandwidth
98.357 GB/s
Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel S2600WT
Ch 1
2DPC
3
3
E5-2650v3
2 DIMMs
Per Channel
35
Ch 2
2
2DPC
1
3
2
1
Performance Gains for Intel DDR4 Servers
Bandwidth comparison using 32GB DDR3 and DDR4 Load Reduced DIMMs
2200
Memory Speed
2000
2133
2133
13%
1866
25%
1800
1600
1600
1400
Xeon E5 v3
1600
35%
Xeon E5 v2
1200
1066
1000
1 DPC
256GB
36
Source: http://www.intel.com/performance
2 DPC
512GB
3 DPC
768GB
Intel Xeon E5 v2 Processors
Processor Model
Determines Maximum
Memory Speed
37
http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-brief.html
Ivybridge-EP
Intel Xeon E5-2600 v2/4600 v2 DDR3 RDIMM Memory Speeds
1.5V
Module Type
(No Mixing in a Server)
Registered
DIMM
(RDIMM)
System
Configuration
Module Ranks
1R/2R=Single/Dual
4R = Quad
1 DIMM per
Channel
2 DIMMs per
Channel
3 DIMMs per
Channel
1 DPC
2 DPC
3DPC
1 Socket Per
Channel
1R/2R
1866
4R
1066
2 or 3 Sockets
Per Channel
1R/2R
1866
1600
1066
4R
1066
800
NO QR
1.35V
Module Type
(No Mixing in a Server)
Registered
DIMM
(RDIMM)
38
System
Configuration
Module Ranks
1R/2R=Single/Dual
4R = Quad
1 DIMM per
Channel
2 DIMMs per
Channel
3 DIMMs per
Channel
1 DPC
2 DPC
3DPC
1 Socket Per
Channel
1R/2R
1600
4R
800
2 or 3 Sockets
Per Channel
1R/2R
1333
1333
800
4R
800
800
NO QR
1DPC 2DPC 3DPC
Ivybridge-EP
Intel Xeon E5-2600 v2/4600 v2 DDR3 LRDIMM & ECC UDIMM Memory Speeds
1.5V / 1.35V
1 DIMM per
Channel
2 DIMMs per
Channel
3 DIMMs per
Channel
1 DPC
2 DPC
3DPC
Module Type
(No Mixing in a Server)
Module Ranks
4R = Quad
Load Reduced
DIMM
(LRDIMM)
4R at 1.5V
1866
1600
1066
4R at 1.35V
1600
1600
1066
1DPC 2DPC 3DPC
1.5V
Module Type
(No Mixing in a Server)
ECC
Unbuffered
DIMM
(ECC UDIMM)
39
1.35V
1 DIMM per
Channel
2 DIMMs
per Channel
3 DIMMs
per Channel
1 DIMM per
Channel
2 DIMMs per
Channel
3 DIMMs per
Channel
System Configuration
1 DPC
2 DPC
3DPC
1 DPC
2 DPC
3DPC
1 Socket Per
Channel
1866
2 or 3 Sockets
Per Channel
1866
1333
NO
UDIMM
1600
1600
NO
UDIMM
1333
Optimizing Intel Xeon E5 v1/v2 Servers
256GB
Unbalanced
34.88%
Balanced
40
Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel S2600GZ
16 x KVR16LR11D4/16
16GB DDR3-1600 RDIMMs 2Rx4 1.5v
Kingston Summary
• Top buyer of memory chips/only the best chips
• Ranked #1 in DRAM
• Ranked #1 in USB drives
• Listed on America’s largest private companies
• Worldwide presence/financially solid
• Channel leadership – OEM, channel, Retail, eTail
• Established relationships with technology leaders
• Value-Add to Flash products
• Major investments in technology
• Most extensive testing in the industry
• Guaranteed compatibility/certifications
• Managing the quality process – from wafer to module
41
Questions?
42
43