The Hl silicon tracker data acquisition system

ELSEYIER
Nuclear Instruments and Methods in Physics Research A 403 1199X1313-375
NUCLEAR
INSTRUMENTS
8 METHODS
IN PMVSICS
RESEARCH
SectionA
The Hl silicon tracker data acquisition system
W.J. Hayne?*,
P. Burchb, R. Halsallb, W. Melottib* ‘, G. Noyes’,
M. Kauschc, P. Kostkad, D. Pitzl”
Received 7 August 1997
Abstract
The H 1 experiment consists of some half million electronics channels in an environment of 96 ns bunch crossings at the
HERA e-p collider at DESY, Hamburg. The data acquisition system built to read out the silicon tracking devices is
described. Several technologies are employed in integrating ASK devices, large numbers of FADCs, fibre-optic interconnections and the latest commercial microprocessor
architectures
in modular bus standards such as VMEbus and the
recent PC1 definition. The experiences gained in system development
are highlighted, which provide significant pointers
to the problems posed by the next generation of hadron colliders. ‘cm 1998 Elsevier Science B.V. All rights reserved.
1. Introduction
HI is one of the experiments
operating
at the
Hadron-Elektron
Ring Anlage (HERA), the large
particle physics proton-electron
storage complex
buried beneath the suburbs of Hamburg at DESY.
A total of nearly one-half million analogue electronic channels are read out and digitised in tune
with successive electron-proton
bunch crossings of
96.064 ns. The early implementation
of the data
acquisition
system has been described elsewhere
[l,?]. Various levels of hardware triggering, soft-
ware filtering and digital compression
are employed in a real-time data acquisition environment
consisting of several hundred processing elements
embedded within the IEEE VMEbus standard [3].
The recent addition of silicon tracking devices [4]
has prompted significant upgrades to its modular
architecture,
taking advantage of the technological
trends since the system’s first inception [S].
Some key parameters of the H 1 data acquisition
system are summarised
in Table 1.
2. Overview of the HI silicon trackers
* C‘orrespondmg author. Tel.: + 44 1235 44 6864; e-mail:
[email protected].
’ Present address: Cognito Ltd. Newbury. UK.
016X-9007.98i$19.00 (‘ 199XElsevier Science B.V. All rights reserved
PII SOl6H-9002(97)01
125-X
Solid state devices are now commonly integrated
into particle physics detectors to measure charged
particle trajectories with the highest spatial resolution
314
Table 1
Breakdown
of the Hl Data
W.J. Hqmes
et al. lNuc1. Ins@. and M&h. in Ph.vs. Res. A 403 (1998) 313-3.?5
Acquisition
System
already been achieved
ment of 1996.
Parameter
Hl
1996
HERA e-p bunch crossing interval
Level-l trigger rate (2.2 us)
Level-2 trigger rate ( < 20 us)
Total number of electronics channels
(1996)
Total number of Si channels (1996)
Total number of Si channels (1998 prov)
Raw data size per event
Raw data from Si trackers per event
(I 996)
Raw data from Si trackers per event
(1998 prov)
Average total event size (formatted)
Average event size from all tracking
detectors
Average event size from Si trackers (1996)
Number of readout branch units
Event builder bandwidth
Event filter computing
power
Data production
Total number of readout/DAQ
crates
Number of electronics boards
Total cost DAQ + readout electronics
96 ns
c: 300 Hz
SO-IOOHz
= 400 000
122880
P 250 000
z 4 Mbytes
0.2 Mbytes
0.4 Mbytes
100 kbytes
60-80 kbytes
4 kbytes
12
25 Mbytes/s
1500 MIPS
z 10 Gbytes/d
Z 150
h 2000
=$15M
achievable
[6,7]. At Hl, silicon detectors
are
located inside the innermost jet chamber of the Hl
central drift chamber tracking device within a 1.2 T
magnetic field. The Central Silicon Tracker (CST)
provides high resolution tracking and vertex reconstruction in the central region, and the Backward
Silicon Tracker (BST) extends this coverage to
small backward angles, The beam pipe radius of
45 mm is determined
by the need to shield the
detectors against synchrotron
radiation.
2. I. Hl central silicon tracker
The CST consists of two layers of double sided
silicon strip detectors [8]. The inner layer (radius
57.5 mm) is a regular prism with 12 faces; the outer
layer (radius 97 mm) is similar with 20 faces (Fig. 1).
Each face consists of six silicon detectors, of total
length 360 mm, with readout at both ends. The
strip pitch in r-o is 25 urn and in z is 88 urn, offering
an intrinsic resolution of 10 urn in r-0 and 20 pm in
z. Resolutions
of 15 and 25 urn, respectively, have
with the preliminary
align-
2.2. HI backward silicon tracker
The BST consists of 8 planes of silicon detector
discs (four in phase 1) mounted perpendicular
to
the beam line at distances between 280 and 860 mm
from the nominal interaction
point in the electron
direction
[9]. The discs have an inner hole of
55 mm radius and are segmented into 16 wedgeshaped, azimuthal sectors of four inch wafers with
an outer radius of 125 mm (Fig. 2).
Each disc is built as a composite
of different
silicon layers. One has a pad segmentation
for
triggering
on deep inelastic electrons
down to
1 GeV. The second layer has circular strips at constant radii (r-strips with a pitch of 48 urn) to
measure the polar angle of backward
scattered
charged particles. The angular resolution of the rstrips is 0.5 mrad. A third layer, with strips at constant azimuthal angles (o-strips with a pitch between
58 and 133 pm depending on the radius) to determine the transverse momentum of charged particles
(resolution of 10% at 1 GeV/c) is also catered for.
As of publication,
four planes are actively used in
the detector with the remaining planes due in 1998.
The possibility of a similar device in the forward
region is also under discussion. The CST is fully
commissioned,
with complete track reconstruction
matched to the rest of the detector.
3. HZ silicon tracker readout electronics
The CST and the full BST have 230 000 channels
which are read out by custom-built
chips and circuit VMEbus boards. A purpose-built
“pipelined”
triggering system (22 bunch-crossings
long, deadtime free) is used to select initial candidates for data
processing
from the background
of lo4 events/s.
A requirement
on HI is that, after an early level
hardware triggering decision, all analogue signals
should be read out within 800 us.
The complete readout chain breaks down into
the following 6 basic elements:
l
the custom on-detector
fulfill the requirement
front-end chip (APC) to
of a pipeline buffer to
315
CST : Double-sided Si strip detector
192 si detectors. 640 chips, 81920 channels
Inner radius (beam pipe) = 45 mm
Radius of first silicon layer = 57.5 mm
Radius of second silicon layer = 97 mm
Fig.
I. H 1 central silicon tracker
HYBRID
WATER
N
iPEATER
-
SICICON SENSOR
Fig. 2. HI backward
accommodate
the HERA bunch repetition
rate
of 10.4 MHz.
l a decoder
chip.
l a line-driver.
for conventional
copper cable or
fibre-optics,
o a control module (OnSiRoC),
silicon tracker
a readout
and processor
module (PowerPCADC PMC)
l a master data acquisition
controller (VMEtaxi).
The master controller supervises the readout and
merges the data-stream
into the VMEbus-base
H1
data acquisition
system via fibre-optic links.
l
316
W.J. Hoynes rr al. JNucl. Instr’. and Meth. in Phw. Rrs. A 403 (IYYK) 313 -325
3. I. HI anzpl~fer pipeline
chip, APC
The 230 000 electronic channels (122 880 in phase
1) are read out by purpose-built
front-end amplifier
chips which contain switched capacitor analoguepipeline buffers [l&12].
A single chip has 128
low-noise channels storing 32 consecutive
bunch
crossings. Pipeline control is synchronised
to the
96 ns bunch crossing period. Thus the pipeline
depth is well matched to the first level trigger decision time of 2.5 ps. The schematics of the front-end
readout chip are shown in Fig. 3.
Each chip is fabricated
in a 1 pm SACMOS
(self-aligned
CMOS) process and is capable of
a 12.5 MHz switching rate and a readout speed of
up to 4 MHz. This allows a 2000 : 1 multiplexing
The
total
within
the 800 ps readout
time.
power consumption
of a single chip, including the
pipeline running at 10 MHz, is less than 40 mW,
whilst functionality
with 2 pm CMOS prototypes
has been maintained
up to 100 krad (1 kGy) integrated dose from a Co6’ source. the equivalent to
10 y of operation at Hl; the SACMOS process further improves the radiation
hardness and gives
larger gain and improved stability. A signal-to-noise
ratio of 18 has been achieved for a load capacitance
of 27 pF.
A total of 14 signals are necessary to control the
pipeline and the shift register of the sequential read-
out. These signals are generated by a “Decoder”
chip which needs just 4 input lines. In both the
“sample” mode, when the information
per 96 ns
bunch crossing is stored into the pipeline capacitors, and in the “readout” mode, when the information of a given time slice is read out sequentially at
the readout speed, the two phases of the clock are
directly passed to the APC.
3.2. On-line rradout controller
For APC control an external, custom-designed,
VMEbus
unit (OnSiRoC)
is used [13,14]. Each
module has a programmable
sequencer unit for
communication
with the decoder chip and four
independent
sets each of three power supplies (strip
detector bias voltage together with APC analogue
and digital voltage).
The sequencer is used to store the intricate command sequences
for controlling
the APC. Sequences are arranged so that each instruction
has
a unique successor. with the possibility of endless
loop command sequences interrupted
by a trigger.
e.g. when sampling. Thus, once the sequencer is
loaded, a single command issued over the VMEbus
can instigate the functional mode of operation on
the APC.
4. The implementation of the Hl silicon tracker
data acquisition system
In order to realise fully the integration
of
the silicon tracker readout electronics
into the
Hl data acquisition,
a coherent architecture
has
been established
consistent
with the rest of the
system.
4.1. Overview, of‘ the HI data acquisition
Fig. 3. Hl amplifier-pipeline
readout
chip.
system
The Hl data acquisition
system consists of
a modular multiprocessing
environment
designed
around the IEEE VMEbus standard
in the late
eighties [l]. Several subdetector
assemblies
are
read out in parallel by arrays of FADCs. DSPs and
microprocessors
before being merged into a central
coordinating
framework. Altogether there are over
150 electronics crates. To avoid saturation
on global VMEbusses, extensive use is made of the local
VSB specification
[IS]. To minimise any further
potential dead-time losses. memory buffering is introduced at all key processing stages.
Currently events of size 100 kbyte can be coordinated with 12 branch-partitions
at rates in excess
of 100 Hz via a VMEtaxi dual optical fibre ring
[ 161. A parallel-filter
farm of around 40 RISC processors (R3000 [17] and PowerPC
[18]), with
a performance
equivalent of 70 IBM-3090 units in
just 3 VMEbus crates, performs online reconstruction as well as providing a final level of full-event
triggering [19]. The complete experiment is archilectured so that graphics-orientated
Macintosh stations provide a platform for both system operation
and software development.
As such. the Hl system is capable of running
entirely in VMEbus; dedicated processors executing dedicated
tasks in real-time so that the net
result is one of a conventional
multi-tasking
system. A well-defined. but essentially simple, framework is defined to enable external control and
monitoring
from commercially
available
computers
and
workstations.
The graphics-based
philosophy
of these devices ensure that the operator is presented with, arguably, one of the most
human interactive interfaces to a complex real-time
system.
The architecture
of the silicon tracker readout
follows that of the central data acquisition
system
by using fibre-optics
and fast integer processing
within a modular framework. Parasitic monitoring
and reconstruction
tasks can be executed in parallel
using
commercially
available
VMEbus
RISC
boards. Graphics-oriented
workstations
then cater
for software development
and operator intervention. However it soon became apparent, during the
first evaluation
of the silicon devices, that a large
amount of additional
real-time processing capability would be required at the very front-end in order
to deal with the intricate problem of hit-finding and
pedestal adjustment
on an event by event basis.
Due to the huge strides consistently
made in the
microprocessor
industry
in recent years, in mid-
1995 it became financially viable to develop a concept of large-scale ADC integration
coupled closely
with commercial
processor cards to satisfy our requirements.
In order to blend in homogeneously
with the VMEbus environment.
yet still provide
enough bandwidth
and buffering capacity to cater
for the digital flow, the final ingredient was suitably
provided by the PC1 specification
[20]. The advantages of adaptation
are now well versed; PC1 is
now a popular standard within workstations,
there
is a defined PMC (PC1 mezzanine)
form facto1
adopted
by all the major VMEbus
producers,
a planned evolution
towards 66 MHz 64-bit and
even an internal chassis definition
of its own in
Compact PCI.
As a consequence,
it was decided to develop an
ADC mezzanine
in the double-sized
PMC form
factor and plug directly into the latest commercially available PowerPC [21] VMEbus cards, which
also come equipped with up to 96 MBytes system
memory [22]. The photograph,
of Fig. 4. shows
a 100 MHz PowerPC 604 VMEbus card equipped
with the RAL developed
mezzanine
for the Hl
silicon readout. Each mezzanine
contains
8 x 12bit, 20 MHz ADCs [23]. Each ADC converts the
analogue
information
from up to 16 amplifier
pipeline chips. under management
of the Xilinx
FPGA. The output data from the ADCs are buffered into the local system memory of the PowerPC
via PCI. The PowerPC executes data processing,
hit- and cluster-finding
and pedestal monitoring
together with the external system i-o communication to the rest of the DAQ. The data are then Iread
out over the 25 Mbyte/s fibre-optic
links of the
VMEtaxi. Thus each PowerPC VMEbus board can
handle 16 384 detector channels, allowing I :4 million H 1 silicon tracker channels to be read out and
processed
within one single 6U VMEbus
crutc
(Fig. 5).
The complete mezzanine readout system was designed. built and commissioned.
with a full complement of system software, within less than a year.
The initial idea was gelled during the summer of
1995, with approval and serious design work commencing that autumn.
Already early in 1996 the
first boards were fully tested in Hl yielding an
effective resolution of 1l-bits for the CST readout.
By summer 1996 the full system was installed and
W.J. Huytws et al. /Nd.
Instr. and Meth. it1Phj,s. REX A 4113 (IYYK) 31% 325
Fig. 4. (a) PowerPC
HI Silicon Tracker
Data Acquisiton (SiDAQ) Readouf Crate
VMEbus
board:
(b). ADC PMC card
CST + BST Readout
VMEtaxi
l
12-16 units
PowerPC + ADC PMC boards
t
I
t
central control room area.
/I$
MacVEE
E Subsystem Trigger Card
Battery-backup constants store
SiDAQ VMEtaxi front-end VMEbus crate
Fig. 5. HI silicon tracker
front-end
readout
crate
1
operational.
Half-way
through
the project the
initial Chief Engineer departed for pastures new
and yet. owing to the adoption
of IS09000 principles. his replacement
was able to continue and
still keep to schedule.
The PMC module itself is a double height PC1
Mezzanine Card conforming
to the IEEE P1386.2
specification [23]. It can digitise 8 input channels in
parallel at up to 20 MHz with 12-bit Comlinear
CLC949 ADCs. Each channel
is buffered with
4/i x 24-bits of true dual-ported
memory, enabling
signals to be digitised and input into the memory
whilst buffered data are transferred to the PC1 bus
via DMA (Fig. 6). The depth of buffering behind
each channel is then determined
by the system
memory of the host processor
board. which on
some commercial VMEbus boards today can be up
to 256 Mbytes.
Two 20-way input connectors
are available on
the front-panel,
providing inputs to the eight lowpower ADCs with differential signals in the range
i. I V (IOOSZ. 2 V peak to peak). A LEMO-00
connector
is also made available to receive NIM
signals to synchronise
data capture with external
signals. A third 64-way PMC connector is used to
route ECL control signals to the on-board
Xilinx
4006E, with an open-collector
output to the backplane. On Hl this was actually routed to the 64 i/o
of the VMEbus P2 connector,
in accordance
with
the IEEE P1382.2 specification. The Xilinx itself is
booted from EPROM and is therefore easily reconfigured.
4.4. Integration
VMEbUS PZ
A0
32.bit
Each OPRAM effectwely double buffers the AOC mput
The system memory of the PCI host mulfkbulfers rhe oufpul dafa
Fig. 6. Functional
s\:stm
The core of the silicon tracker readout is embedded into one VMEbus crate thanks to the compactness of the PMC solution and the benefits of
modern IC integration.
A VMEtaxi system then
provides
the overall
coordination
and
interconnectivity.
VMEtaxi modules can connect VMEbus crates
over several kilometres with multimode 1001140 pm.
graded index. optic fibres [ 16,241. Fig. 7 illustrates
the basic philosophy. The software protocol is purpose-written and optimised for speed and efficiency
in a data acquisition
environment
[25]. There are
many interesting analogies with the future international standard SC1 [26]. During the early operation of Hl. 25 MHz 68020 based boards were used
with 125 MHz taxi chips. The upgraded “Mark-?”
1960
Signalshorn
into the HI data acyuisitkw
block diagram
of the PMC card
320
W.J. Huynus et al.~Nucl. Instr. and Mrth. in Phtts. Rrs. A 403 (1998) 313-3-75
VME
The dual optic-hbre rings loop around all
VMEfaxr crates
LL
Data sent out by one tax, IransmUter are
received at the next VMEtawr receiver ,n
the r,ng
Direct connecllons can be enabled
between any frvo dewces usng By-pass
reg!srers
VSB
Mark-2 boards are equipped w/h 50 MHz 68030 muoprocessors
and can exp/oU
2550 MHz fax! chips worku?g !n parallel wfh double /Inks.
X/i/NX gate arrays are used for VMEM and VSB block transfers.
Separate program memory and up lo 2 MByies of dual-ported extension cam IS also
prowded. The exfens~on ram ensures fhaf the dafa Iransferpedormance
IS
mcreased ,n lfne w/h fhe ch/p bequenoes (50 MBylesisec w/h 250 MHz lax! ch1p.s).
Fig. 7. VMEtaxi
modules (VMExi2) are able to exploit 50 MHz
68030 processors and 250 MHz taxi chips so that,
by using double fibre-optic links in parallel, transfer rates of over 50 Mbyte/s are possible. The link
reliability has been tested to a bit error rate of less
than 1 in 1013. Program memory is provided for by
128 kbyte of on-board
dual-ported
static ram,
whereas EPROM
and EEPROM
cater for firmware storage and configuration
parameters.
In
addition
the on-board
extension
ram provides
2 Mbyte of data memory. As a result the VMExi2
can be regarded as “3 boards in one” providing
processing power, optic-fibre interconnectivity
and
data memory on a single-width
VMEbus card.
The VMExi2 was commissioned
into the Hl
data acquisition
system during the Spring of 1993,
using 175 MHz taxi chips to achieve 33 Mbyte/s
peak transfer rates.
Further readout and reconstruction
monitoring
is carried out in a master monitoring
crate (Fig. 8)
by RISC-based cards similar to the filter-farm [27].
Memory boards provide the necessary buffers for
readout and monitoring
purposes [28]. In addition
some 16 Mbyte are required for the fast access of
stored pedestal and sequencer
RAM values to-
overview.
gether with readout configurations.
Macintosh stations are employed to remain compatible with the
rest of the Hl system for software development
and
operator control. One device is primarily responsible for the global operation
of the data acquisition. a second provides
event displays
and
histogram monitoring
and a third takes care of the
monitoring
of slow control information
using LabVIEW [29-311. Macintosh personal computer access is through MacVEE [32-341 or. if 32-bit access
is required, through MAC 7212 [35]. Existing software development
tools [36] and techniques [37]
as used in the rest of the data acquisition system are
exploited. Data quality can be monitored via event
displays and histograms
of, for example, pedestal
and noise values, signal to noise ratios, clusters.
radiation ageing (via the APC), etc.. all with purpose-written
programs
layered
on top of the
existing base software packages of the HI data
acquisition
system. Much of the binning
and
monitoring
is carried out in dedicated tasks which
execute
permanently
within
the cost-effective
VMEbus processors, thereby leaving the Macintosh as the graphical user interface. Such a solution
has the added advantage
that the VMEbus part
SiDAQ VMEtaxi crate number 0
VMEtaxi
Hl DAQ VMEtaxi branch number 12
,vMEtaxi
Fibre-optic VMEtaxi ring
connected to front-end
SiDAQ readout crates
Fibre-optic VMEtaxi ring
connected to HI Central
Data Acquisiton System
MacVEE
VSB I
Pedestal DPM,
I 16 MB
Master SiDAO and
data monitoring crates
located in control room
= IOOm from front-end
readout crates
MONlTORlNG CRATE
VME64
$7
f”
Mac7212lVlC
NC 6230
f”
%ow Ctrls D/W
Slow Controls MacVEf ?interface
RAID 8239
or PowerPC
Data Monitoring Processors
runs as a self-contained
entity. according
to the
protocol of the complete experiment_ and is able to
update the monitoring
data sent to the rest of the
acquisition system independent
of the functionality
of a local interface station. Finally a Subsystem
Trigger Control Crate handles the HI triggering
protocol [3S].
5. HI silicon tracker data acquisition software
There is a well-ordered
structure defined by the
modular architecture of the hardware which defines
how the software is written and developed. Dedicated tasks run on dedicated processors throughout the VMEbus system with operator intervention
provided for via graphics-orientated
workstations
[39.40]. Devices such as the Macintosh
provide
a convenient
integral component
in both software
development
and operator-control.
In order to embrace the framework of VMEbus, extra tools have
been developed
to ease the integration
into the
VMEbus
[36] and exhaustive
software libraries
cater for external control by providing a full set of
routines for system configuration,
testing. systemstatus and monitoring
[35,37.41.43].
These are
based on relatively simple memory-mapped
mailbox concepts for inter-processor
communication
(command blocks with arguments as illustrated in
Fig. 9) and a well layered modularity
which clearly
separates the different levels, such as the dedicated
package
(assembler)
written
for the VMEtaxi
(Fig. 10). The object-orientated
world of the workstation thus becomes disentangled
from the complex firmware code executing
on the hardware
except at clean and well-defined boundaries.
For the development
of flexible algorithms which
need to be embedded into the firmware task, external hooks are provided for within the tirmware
code itself. e.g. for hit detection. These algorithms
(as with the object-orientated
workstation
modules) can be developed
platform independent
in
the high-level environment
of choice. The resulting
object modules can then bc loaded and stored in
322
W.J. Haynes et al.lNucl.
Instr. and Meth. in Phys. Rex A 403 (1998) 313-325
battery backed-up memory. Upon the next system
reset, the firmware task automatically re-loads and
links the module into its system memory and calls
appropriately [41,43].
The firmware task on the PowerPC itself, once
developed, debugged and established, remains fairly static over the lifetime of the system. As such it
can either be developed in C or even in Assembler
SimpS memory-mapped mailboxes
Pass input & output arguments via Command Blocks
SIMPLE RULES
The Command Block is defined at the The Command Address
Each Command has a unique Command Code Identifier (I.D.)
Parameters are returned by re-settmg the Command I.D.
The Command identifier is always written last
Fig. 9. Inter-processor
communication
mailbox
structure.
since its role is to remain a tight kernel managing
the data throughput protocol between front-end
readout and later data merging with the rest of the
DAQ. In turn, this dispenses with the need for
a potentially performance depriving operating system executing on the VMEbus processors at the
heart of the system. Within each Hl silicon readout
task, 0.5 Mbyte of the on-board system memory is
reserved for 16 event buffers of raw front-end data
input from a maximum of 16 384 channels,
0.5 Mbyte for the hit data (after execution of the
search algorithm described in the following section), 1 Mbyte for the pedestals and up to 4 Mbyte
for further calibration management. The communication protocol itself, between each PowerPC and
its VMEtaxi readout coordinator, demands little
memory resources for an event indicator area passing event number, size, pipeline counter and any
possible error in 16 byte blocks. Once the VMEtaxi
has taken a particular event, its indicator number is
cleared over the VMEbus port of the PowerPC’s
system memory so freeing the corresponding buffer
for subsequent data. The PowerPC processor then
communicates free input buffers to the front-end
ADCs via the Xilinx fifo communication table.
Such a relatively simple protocol mechanism still
leaves ample space for program storage and any
additional features, such as test patterns and simulation data.
VMExi & SiPPCvmexi Software Package comprises several layers of protocol
Basic firmware driver primitives for VMEtaxt & PowerPC control
Communication protocol for VAlEtaxi-VMEtaxlfibre-optic ring
XIDAQ data acquisition & SiDAQ readout protocols
Standalone menues for direct user intervention
External control via “Inter-Processor Communication” mailbox protocol
X/USER & S/USER libraries existing as separate object finkable modules
l
l
l
l
l
l
Fig. 10. Software
structure
of the SiPPCvmexi
package.
W.J. Havnes et al.lNucl.
5.2. Hl silicon tracker hit detection
processing in real-time
323
Instr. and Meth. in Phys. Rex A 403 (1998) 313-325
and data
It is imperative to reduce the massive flow of raw
data volume as early as possible in the readout chain
of any data acquisition system, which in the case of
silicon trackers means the correct coding of hits and
cluster searches. Thus an important consideration
for the silicon tracker system is the ability to marry
flexible physics-based algorithms directly into the
fixed core of systems firmware executing on the
PowerPC microprocessors. As discussed in the previous subsection, the acquisition task is essentially
split between the main readout routine executing the
data transport according to well-defined protocol
mechanisms and the specific hit detection algorithm
which processes the digitised output. By allocating
dedicated memory blocks in calling separated object
code modules. these latter algorithms can then be
developed independently by the detector physicists,
on the platforms of choice. Communication between the two processes is established via input and
return arguments, as with standard subroutine conventions, and prudent memory mapping within the
physical address spaces.
For the Hl silicon trackers, these specific algorithms must update pedestals and continually provide calibration monitoring in order to perform
reliable hit searching and cluster searching. It is due
to the capability of modern microprocessors that
such a task can indeed be performed at rates of
100 Hz in real-time without adding to the burden on
dead-time during readout and acquisition. Nevertheless, for performance reasons, these algorithms
have been implemented in 32-bit integer arithmetic.
The processor architecture, itself, permits pipelined
instruction processing with three integer units, with
the code written in C and the use of appropriate
PowerPC optimising compilers. Math library functions, such as sqrt, are avoided and divisions implemented as bit shift operations whenever possible.
As an example, we consider here the code for the
CST. The hit detection algorithm is a two-pass
approach relying on correctly determined pedestals
(PED) and variances (VAR), as follows:
(i) The first loop over the raw data calculates the
common mode per readout chip as the mean of the
raw data values. Outliers and noisy strips are masked out.
(ii) The second loop determines the pulse-height
for each strip as:
PH = RAW
-
CommonMode
-
PED.
(1)
It then applies a single strip noise cut:
PH’ > VAR x StripCut
(2)
where StripCut = 1. A cluster is defined as a contiguous group of strips for which condition given
by Eq. (2) holds. The cluster is completed with the
first strip below the cut. It is accepted if the summed
pulse-height is larger than the cluster cut:
Z(PH)2 x ClusterWidth
> ZVAR x ClusterCut
(3)
where ClusterCut = 16. If the cluster passes condition given by Eq. (3) one 32-bit word containing the
pulse-height, the address and the variance is written
out for each strip belonging to it. Optionally a few
strips to the left and right of the cluster can also be
transferred.
The time needed for processing the data from all
strips (with the 100 MHz PowerPC board) is currently around 8 ms, permitting deadtime free operation at Hl at rates below 100 Hz.
After a system reset, new pedestals and variances
have to be determined first. For the former an
individual value for each strip and each of the 32
front-end pipeline buffers is stored giving 327680
pedestals per readout board (presently only 1280
channels are multiplexed per ADC). The variances
are, however, not pipeline buffer dependent.
Pedestals are calculated as RAW - CommonMode and averaged over 64 events per pipeline
buffer. Variances are initialised by accumulating
the sum and the sum of squares of the pulse-heights
(Eq. (1)) over 1024 events and then taking the difference of the mean squared quantities:
VAR = ave (CPH’)
- (aveCPH)‘.
(4)
During data taking, the pedestals are updated with
a weighted value from the current event:
PED,,,
= (1 - weight)PEDOld
+ weight(RAW
- CM). curre\ent
(5)
324
W.J. Haynes et al./Nucl.
Instr. and Meth. in Phys. Rex A 403 (1998) 313-325
where weight = l/64. For the variances, the
above introduced pulse-height sums are averaged
over a multiple of 1024 events enabling fresh variances to be applied at that frequency. During the
accumulation, strips which have been marked
by the hit detection routine are excluded except
for “hot” strips that have shown an increased
occupancy.
6. Observations and conclusion
The impact of the mid-nineties style technological approach is immediately visual when one sees
the existing Hl system in action. A quarter of
a million silicon tracker channels can be read out
into one double-height Euro-sized (6U) VMEbus
crate, compared to over a hundred crates reading
out a comparable number of channels from the rest
of the detector, as designed in the eighties. With the
advent of faster modular-based busses such as PCI,
and the increasing trend of ever more transistorised
componentry at the chip level, we can now build
our modularity around interchangeable PMC-like
form factors. The continuing price-performance improvements of microprocessor technology, together
with increasing memory chip densities, are likely to
result in even more PMC flexibility. In the shorter
term, consideration is being given to the adoption
of PMC principles in other areas of the Hl system
to add more flexibility for planned HERA luminosity increases from the year 2000, including PCIbased interconnectivity between front-end crates
and workstations [44,45]. Expansion of the silicon
system is already foreseen for the 1998 data-taking
period when the BST will be extended to 8 detector
planes as well as incorporating the readout of the
GaAs strip detectors of the Very Low Q2 detector
C4f-d.
Tracking devices have long been a major factor
to the problem of data acquisition in large particle
physics experiments. The experiences gained from
implementing the Hl silicon tracking system show
that by using modern, Aexible, bus-based architectural concepts, the demands placed on data
throughput, and the powerful computing required
in real-time, can indeed be met with the latest
technology.
Acknowledgements
We would like to acknowledge the members of
our collaboration who helped in the system integration, notably A. Campbell and S. Bourov of
DESY and D. Clarke of RAL and the encouragement given us by R. Eichler of ETH, Zurich and
M. Klein of IfH-DESY, Zeuthen. The whole
readout chain is indebted to R. Horisberger of PSI
who designed and implemented the APC. We also
wish to acknowledge the excellent cooperation of
the companies involved in the development and
implementation phases of the Hl system, notably
Micro-Research of Helsinki and C.E.S. of Geneva
and the support given by the various funding agencies of the Hl collaboration.
References
[l]
W.J. Haynes. Bus-based
architectures
in the Hl Data
Acquisition
System, in: Proc. VITA Int. Conf. Open Bus
Systems 1992, Zurich, CH, October 1992, ISBN 90-725771l-6, 1992. pp. 27-36. RAL-92-048, August 1992.
[2] W.J. Haynes. Experiences
at HERA with the Hl Data
Acquisition
System, in: Proc. Int. Conf. Computing
in
High Energy Physics ‘92, Annecy. France, September 1992.
CERN 92-07, ISBN 92-9083-049-2,
1992, pp. 151-161.
[3] The VMEbus specification,
IEEE standard
1014.
[4] Hl Collaboration,
Technical
proposal
to build silicon
tracking detectors for Hl, DESY PRC 92/01, July 1992,
DESY PRC 93/02 pp 97799, March 1993 and Upgrade of
the Hl Backward Silicon Tracker, DESY PRC 96/02.
[S] W.J. Haynes, Silicon Tracker Data Acquisition, in: Proc.
6th LeCroy Int. Conf. Electronics
for Particle Physics.
Chestnut Ridge, New York, 28-29 May 1997. RAL-P-97002, May 1997.
C61G. Hall, Rep. Prog. Phys. 57 (1994) 481-531.
c71C.J.S. Damerell. Vertex Detectors: The state of the art and
future prospects, RAL-P-95-008,
December 1995.
PI J. Burger et al., The silicon tracking detector system of
Hl, IEEE Nuclear Science Symp., San Francisco,
USA,
November 1993; Nucl. Instr. and Meth. A 367 (1995) 422.
c91E. Peppel et al., Development of the Hl Backward Silicon
Strip Detector, DESY 96-217, Proc. 5th Int. Workshop on
Vertex Detectors, Chia, Sardinia, June 1996. Nucl. Instr.
and Meth.
Silicon tracking
detectors
for Hl. in:
Cl01R. Horisberger,
Proc. Internal Europhysics
Conf. High Energy Physics,
Marseille, 1993, pp. 366-367.
1111R. Horisberger et al.. A silicon vertex detector for Hl at
HERA. in: Proc. IEEE Nuclear
Science Symposium,
Orlando.
Florida.
October
1992, ISBN o-7803-0883-2
(1992) 222.
W.J. Haynes et al./Nucl.
Instr. and Meth. in Phvs. Rex A 40.3 (1998) 313-325
[12] R. Horisberger,
D. Pitzl, Nucl. Instr. and Meth. A 326
(1993) 92.
[ 131 W. Zimmermann,
OnSiRoC VME Controller,
DESY H 1
Internal Technical Note, July 1993.
1143 J. Burger et al., Nucl. Instr. and Meth. A 386 (1997) 269.
[15] The VME Subsystem Bus (VSB) specification, IEEE standard 1096.
1161 E. Pietarinen, VMEbus cross interface processor module
with high speed fibre optic links: VMExi, Univ. of Helsinki
Report HU-SEFT-1991-14,
Helsinki, Finland, June 1991.
1171 Creative
Electronics
Systems SA, RAID 8235, R3000
VMEbus controller, Geneva, CH, 1990.
[18] Creative Electronics
Systems SA, RTPC 8067 PowerPC
Single VMEbus board computer, Geneva, CH, 1995.
[19] A. Campbell, A RISC multiprocessor
event trigger for the
data acquisition
system of the Hl experiment at HERA,
Conf. Real Time ‘91, Jiilich, FRG, June 1991. RAL-91-060.
[20] PC1 SIG, /www.pcisig.com/,
Hillsboro, OR, 1997.
[21] IBM, Motorola,
PowerPC Microprocessor
Family, 1994.
[22] Creative Electronics
Systems SA, RIO2 8060 PowerPC
based RISC I/O VMEbus board, Geneva, CH, 1996.
[23] G. Noyes, P. Burch, RAL-PC3050
RIO2 FADC PMC,
RAL Technology,
January 1997.
[24] Micro-Research
Oy, VMExi/M2,
A fibre-optic
readout
system, Helsinki, Finland, January 1993.
[25] W.J. Haynes, VMExilSSP,
VMEtaxi
Mark-2
System
Software Package, DESY, Version 4.2, July 1993.
[26] The Scalable Coherent
Interface (SCI), IEEE standard
1596.
[27] G. Noyes, Hl Silicon Tracker Farm, DESY Hl Internal
Technical Note, 1996.
[28] Creative Electronics Systems SA, VME/VSB Dual Ported
Memory DPM 8242, Geneva, CH, 1989.
[29] National Instruments
Corp., LabVIEW
3 Scientific software for the Macintosh,
Austin, TX, 1993.
[30] D. Clarke,
LabView
control
of Hl Silicon Trackers,
RAL/DESY Hl Internal Technical Note.
[-?!I H.C. Reuss. Controller
Area Network (CAN). A powerful
solution for Fieldbus applications,
Philips Semiconductors
Report. July 1992.
[32]
325
B.G. Taylor, MICRON:
VMEbus and CAMAC access
from Macintosh
II, ESONE Conf. VMEbus in Research,
Zurich. CH, October 1988.
[33] B.G. Taylor, The MacVEE User Manual, CERN, Geneva,
1989.
[34] B.G. Taylor,
The MICRON
User Manual.
CERN.
Geneva, 1989.
[35] Creative Electronics Systems SA, MAC 7212, Mac II NuBus to VMVjVME Interface, Geneva, 1990.
1361 W.J. Haynes, VME-TOOLS,
VMEbus interaction
from
the MPW shell, DESY, Version 3.2, February
1994.
[37] W.J. Haynes, SiVMExi, Hl Silicon Trackers
VMEtaxi
Data Acquisition
Package, DESY. Ver. 2.1, March 1994.
[38] H. Krehbiel, The Hl trigger control system, DESY Hl
Internal Technical Note, September 1989.
[39] M. Zimmer, Graphics-orientated
operator
interfaces at
Hl, in: Int. Conf. Computing in High Energy Physics ‘91,
Tsukuba, Japan, March 1991, Universal Academy Press, Inc.
[40] J. Coughlan,
D. Diillmann,
M. Savitski, M. Zimmer,
Object orientated programming
for online systems at H I,
in: Proc. Int. Conf. Computing in High Energy Physics ‘92,
Annecy, France, September 1992. CERN 92-07. ISBN 929083-049-2 (1992) 519-522.
[41] W.J. Haynes. SiPPCvmexi, Hl Silicon Trackers PowerPC
Readout via VMEtaxi. DESY/‘RAL. Version 2.0, January
1997.
[42] G. Noyes, SiMac Library and Test Application.
DESY H 1
internal Technical Note, February
1994.
1433 M. Kausch et al., A PowerPC-based
readout system for
the silicon trackers of the Hl experiment.
Elba Detector
Conference, May 1997; Nucl. Instr. and Meth. A. (to appear).
[44] J. Bovier. PVIC The new generation
inter-crate
connection, in: Proc. VITA Europe Congress, Brussels. Belgium.
October 1996.
[45] P. Burch, PVIC Detailed Description,
RAL Technology,
November
1996.
[46] Hl Collaboration,
Technical proposal
to build a spectrometer covering very small momentum
transfers, DESY
Internal Report, May 1996.