L1CTT DFEA2 - Boston University

L1CTT DFEA2
Completion Review
E. Hazen – FNAL – July 29 2005
Outline
●
Introduction / Scope
●
Changes since DFEA (v1)
●
DFEA2 Design Overview
●
DSAT Tester
●
Prototype Testing
●
Production Testing
●
Maintenance / Other Issues
E. Hazen – FNAL – July 29 2005
Introduction / Scope
●
v1 → v2 upgrade provides:
–
●
More FPGA resources, (much) faster downloads,
simpler design (fewer parts), cleaner cabling
Scope of this review:
–
Is the DFEA2 system ready to install?
●
Does it meet the spec?
●
Has appropriate testing been done?
●
Does the firmware perform appropriately?
–
What problems must be solved before installation?
–
How will the DFEA2 system be maintained?
E. Hazen – FNAL – July 29 2005
DFEA update changes
●
LVDS inputs drive only one FPGA (point­to­point)
–
●
No buffers between LVDS receivers and FPGAs
Eliminate rear transition modules... –
Direct LVDS output from rear of module
●
Two SLDBs on module for L1 outputs
●
New DC power scheme – 48V bulk power
●
Full access to big FPGAs from backplane R/W bus
–
Complete flexibility for inject / readout of test data
●
Separate programming of individual FPGAs
●
SCL clock/control via LVDS on backplane
E. Hazen – FNAL – July 29 2005
DFEA2 Module
Simplified Diagram
IsoBits
28+CLK
LVDS 1
SLDB
LVDS 2
XC2V6000
LVDS 3
(DDR)
L1Muon
XC2V6000
“up front” (U7)
“up back” (U8)
L1CTT
LVDS 4
STT
FPGA config
Local bus
Flash
DFE Backplane
LEDs
Backplane
Interface
FPGA
JTAG
LVDS Timing
TTL R/W Bus
Channel-Link
Receivers
28+CLK
LVDS 5
LVDS 6
LVDS 7
SLDB
XC2V6000
“down front” (U9)
(DDR)
L1Muon
XC2V6000
“down back” (U10)
L1CTT
LVDS 8
STT
IsoBits
Channel-Link Transmitters
E. Hazen – FNAL – July 29 2005
DFEA2 Module
Data Paths
IsoBits
28+CLK
LVDS 1
SLDB
LVDS 2
XC2V6000
LVDS 3
ch 1-4
“up front” (U7)
L1Muon
XC2V6000
“up back” (U8)
L1CTT
LVDS 4
STT
ch 5,6
ch 5,6
DDR links (point-to-point)
between FPGAs
IsoBits
ch 3,4
ch 3,4
Channel-Link
Receivers
28+CLK
LVDS 5
LVDS 6
LVDS 7
SLDB
XC2V6000
“down front” (U9)
ch 5-8
L1Muon
XC2V6000
“down back” (U10)
L1CTT
LVDS 8
STT
Channel-Link Transmitters
IsoBits
E. Hazen – FNAL – July 29 2005
Front Panel
LED Display
Test Connector
JTAG Connector
1.5V
Backplane
DTACK
3.3V
FPGAs
ready
Page
Display
Page
Select
Page 0
FPGA, DCM status
Page
Page
Page
Page
LVDS Rx clock error
LVDS Rx sync error
LVDS Rx period error
LVDS Rx pattern error
1
2
3
4
Page 5
Page 6
Page 7
20 test signals
Many functions depending
on programmed setting
JTAG Access to
all FPGAs
Altera “Byte Blaster”
pinout/connector(!)
(FPGAs are all Xilinx)
SCL status
Board Serial Number
Display Register bits 0-7
E. Hazen – FNAL – July 29 2005
DFEA2 Mechanics
Press-fit Hard Metric
Connectors (315 total pins)
HL-TYPE
NOTE: EDGE THICKNESS
1.6mm (0.064")
HANDLE
F
DFE R/W Bus
Serial Encoded SCL
LEDS
various options
for detailed
status display
E
D
C
B
A
1
1.50
11
15
DFE UPGRADE
Iso in/out
LVDS in 1-8
25
1
BOARD
19
6U x 320mm
Logic Analyzer
Connector
COMPONENT
3.50
SIDE VIEW
Coax feed-thru
for SLDB out
48V power
DRAWING REV. 2
5 MARCH 2004
JTAG
LVDS out 1-4
1
19
HL-TYPE
HANDLE
NOTE: EDGE THICKNESS
1.6mm (0.064")
Detailed Pinout
in backplane spec
E. Hazen – FNAL – July 29 2005
DFEA2 PCB Layout
E. Hazen – FNAL – July 29 2005
Layout Guidelines
●
●
●
●
●
Only point­to­point connections for all high­speed signals
Minimize trace lengths for fast I/Os
All LVDS routed as differential striplines, with careful length matching within channel links
Fabrication
Dwg notes
“Easy” design rules: .005/.005 line/space, .024/.012 traditional vias
Strictly follow mfgr guidelines for bypass C's
E. Hazen – FNAL – July 29 2005
PCB Stackup
L1 Signal
62 ohm single-ended (inter-FPGA only)
.006
L2 1.5V
.002
L3 GND
.012
L4 Signal
100 ohm differential striplines (LVDS)
.012
L5 GND
.006
L6 GND
.012
100 ohm differential striplines (LVDS)
L7 Signal
.012
L8 3.3V
.002
L9 GND
.006
L10 Signal
62 ohm single-ended (inter-FPGA only)
First run of PCBs warped due to asymmetrical stack-up...
revised stack-up (shown above) was okay.
E. Hazen – FNAL – July 29 2005
LVDS Inputs
LVDS cable shield
R/C to GND
Backplane
Connector
Terminators
at pair end
LVDS receiver
Differential Striplines
Equal Length
Bypass caps on
every Vcc pin
E. Hazen – FNAL – July 29 2005
Backplane Bus Interface
LVT buffers
for control signals
Short stubs
for LVDS clk, nrzc
LVT buffers
for data, address
XC2V250 FPGA
Backplane interface
Backplane
Connector
E. Hazen – FNAL – July 29 2005
LVDS outputs / 48V in
48V in, return
LVDS cable shield
R/C to GND
48V DC/DC
converter
Bypass caps on
every Vcc pin
Differential Striplines
Equal Length
E. Hazen – FNAL – July 29 2005
Power Supplies
Ripple Filter
4A Fuse
48V Converter
Outputs: 1.5V 3.3V
TVS 400W 85V
Input filter
3.9nF 100V X7R
330uF 2.5V
180uF 6.3V
E. Hazen – FNAL – July 29 2005
JTAG Chain
JTAG
74LVT16245
51Ω
Chain includes all FPGAs and PROM
All signals are buffered and terminated
TCK
TMS
TDI
U10
U8
U6
U22
U7
U9
XC2V6000
XC2V6000
XC2V250
18V04
XC2V6000
XC2V6000
TDO
E. Hazen – FNAL – July 29 2005
DSAT (DFEA Stand­Alone Tester)
E. Hazen – FNAL – July 29 2005
Introduction
●
What is it?
–
●
A single board which can simulate all hardware which connects directly to DFEA2
Why did we build it?
–
New Backplane / Controller not ready in time to test new DFEA2
–
Other infrastructure (SCL etc) not easily reproduced off­
site
–
Very useful for debugging and production testing
(complete control over all I/Os)
E. Hazen – FNAL – July 29 2005
53MHz
Hard Metric
Backplane Connectors
74ABT Buffers
R/W bus
PC Interface
(Parallel port)
FPGA
(Virtex-II)
Flash
Memory
SCL
simulation
LVDS
LVDS Feed-Thru
Connectors
LVDS Channel-Link I/O
JTAG
SLDB Receivers
SLDB
Receiver
E. Hazen – FNAL – July 29 2005
DSAT Layout
Board format same as
6U Backplane
202,2 [7,961]
9,16 [0,361]
20,32 [0,800]
DFEA
Slot
1,55 [0,061]
A
B
C
D
E
1
2
3
A
4
B
C
D
E
5
1
1
6
2
2
7
3
3
8
4
9
5
10
6
1
11
7
2
8
3
LVDS serial
Link I/O
9
10
15
11
16
12
17
1
13
2
18
14
3
19
15
20
16
21
17
1
22
18
2
23
19
3
24
25
C
D
E
1
3
2
2
4
3
3
5
4
6
5
7
6
1
8
7
2
3
9
8
10
9
11
10
12
11
12
1
14
13
2
15
14
3
16
15
17
16
18
17
1
19
18
2
19
3
SLDB Receivers
184,28 [7,255]
262,05 [10,317]
13
C
D
1
2
3
4
3
5
4
6
1
5
7
2
6
8
3
7
9
8
10
9
11
10
12
1
11
13
2
12
14
3
13
15
14
16
15
17
1
16
18
2
17
19
3
18
19
SCL Receiver
(not used)
E
3
94,28 [3,712]
B
2
2
68,28 [2,688]
A
1
1
255,85 [10,073]
B
1
234,28 [9,223]
A
1
2
SCL RECEIVER
DAUGHTER BOARD
BOSTON UNIVERSITY
Electronics Design Facility
3,1 [0,122]
Drawn By:
E. Hazen
DSAT Tester
Suggested Layout
...Tester/mech.dwg
2004-05-03
REV:
A
E. Hazen – FNAL – July 29 2005
DSAT in Operation
Front View
Rear View
E. Hazen – FNAL – July 29 2005
DSAT Summary
●
●
(5) Rev B DSAT boards assembled (so spares are available)
Requires only +3.3V, +5V and 48V power plus a linux computer with parallel port to operate
–
●
(could be converted to USB if there is a compelling reason to do so)
Firmware is well­documented VHDL code so enhancements can be added if needed
E. Hazen – FNAL – July 29 2005
Prototype Testing at BU
●
Backplane interface verified with DSAT, DFEC
●
LVDS links tested for BER to 1e16 bits with 0 errors
●
Isolated track I/O verified with DSAT
●
Basic functionality with data in/out using DSAT
–
●
Fake data sent on inputs; Level 1 and Level 2
outputs verified
Random vector test of 1.5M events (more...)
E. Hazen – FNAL – July 29 2005
Random Vector Test
●
One (C) program which performs the following:
–
Read in MCS (hex) files and configure DFEA2 FPGAs
with normal operating firmware
–
Load and parse VHDL equation files
–
Perform basic read/write test and automatically adjust timing registers
–
For each test cycle:
–
●
Generate 73 random events and send to DFEA2
●
Capture inputs from CTOC and SLDBs
●
Calculate expected outputs using equations
●
Report differences and count errors
Total of ~ 1.5M events run in 3 days (6 sec/event)
E. Hazen – FNAL – July 29 2005
BERT (Bit Error Rate Test)
●
Dedicated (C) Program and Firmware:
–
Configure DFEA2 FPGAs with BERT firmware
–
Set DSAT to fanout CTOC outputs to DFEA2
–
Reset DFEA2, enable error counters, start pseudo­
random sequence generator
–
Random data output through 4 outputs on DFEA2, fanned out via DSAT to 8 DFEA2 inputs.
–
Data pass through same data paths as in normal DFEA2 operation and compare with original data for errors
–
Run for 6 days with no errors (short cables)
total of ~ 10**16 bits
–
Test underway starting 7/26 with one long cable
(no errors so far)
E. Hazen – FNAL – July 29 2005
Production Testing at BU
●
●
Basic functionality
–
Power­up, load firmware, cable all I/Os
–
Check all I/Os with DSAT
–
Check LEDs, diagnostic connector
–
Brief BER test (10**13 bits or so), require zero errors
–
Random track test (how long?), require zero errors
Experience
–
Most boards worked immediately
–
A few (5 or so) needed minor repair due to assembly problems
–
Final tally: 57 good boards, 3 bad ones
E. Hazen – FNAL – July 29 2005
Other Issues...
●
●
Recompiling Xilinx firmware:
–
Each firmware update requires recompiling Xilinx design from VHDL for 160 chips
–
Compiling new firmware takes 1 hr per chip on 2.8GHz Xeon (FC3 linux).
–
This can easily be set up to use a cluster of machines with a common file­system mounted over NFS
–
Are the necessary resources available at FNAL??
If not, look for alternatives (dedicated cluster?)
BU Personnel at the lab
–
Norik Khalatyan, Monica Pangilinan
E. Hazen – FNAL – July 29 2005
Maintenance / Upgrades
●
●
Firmware updates/maintenance
–
BU (Wu) can certainly do this throughout the run (through 2008).
–
Engineering is available at BU on an as­needed basis for upgrades.
Hardware Maintenance
–
Currently ~15 working spares (+3 being repaired) exist. Is this sufficient?
–
Should additional parts be stockpiled to protect against obsolescence? [probably not needed]
–
Diagnosis and repair of hardware problems should be straightforward using DSAT. The number of components on the DFEA2 is relatively small.
E. Hazen – FNAL – July 29 2005
E. Hazen – FNAL – July 29 2005
E. Hazen – FNAL – July 29 2005