Using the GNI and DMAPP APIs

Transcription

R
S–2446–51
© 2010–2013 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form
unless permitted by contract or by written permission of Cray Inc.
U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE
The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR
252.227-7014.
All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided
with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described
in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable.
Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or
disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48
CFR 252.227-7013, as applicable.
Cray and Sonexion are federally registered trademarks and Active Manager, Cascade, Cray Apprentice2,
Cray Apprentice2 Desktop, Cray C++ Compiling System, Cray CS300, Cray CX, Cray CX1, Cray CX1-iWS,
Cray CX1-LC, Cray CX1000, Cray CX1000-C, Cray CX1000-G, Cray CX1000-S, Cray CX1000-SC,
Cray CX1000-SM, Cray CX1000-HN, Cray Fortran Compiler, Cray Linux Environment, Cray SHMEM, Cray X1,
Cray X1E, Cray X2, Cray XC30, Cray XD1, Cray XE, Cray XEm, Cray XE5, Cray XE5m, Cray XE6, Cray XE6m,
Cray XK6, Cray XK6m, Cray XK7, Cray XMT, Cray XR1, Cray XT, Cray XTm, Cray XT3, Cray XT4, Cray XT5,
Cray XT5h, Cray XT5m, Cray XT6, Cray XT6m, CrayDoc, CrayPort, CRInform, ECOphlex, LibSci, NodeKARE,
RapidArray, The Way to Better Science, Threadstorm, Urika, UNICOS/lc, and YarcData are trademarks of Cray Inc.
Aries and Gemini are trademarks of Intel Corporation in the United States and/or other countries. CUDA is a
trademark of NVIDIA Corporation. Linux is a trademark of Linus Torvalds. Windows is a trademark of Microsoft
Corporation. All other trademarks are the property of their respective owners.
RECORD OF REVISION
S–2446–51 Published September 2013 Supports the Cray Linux Environment (CLE) 5.1.UP00 release.
S–2446–5001 Published November 2012 Supports the Cray Linux Environment (CLE) 5.0.UP01 Limited
Availability (LA) release.
S–2446–41 Published August 2012 Supports the Cray Linux Environment (CLE) 4.1.UP00 release.
S–2446–4003 Published March 2012 Supports the Cray Linux Environment (CLE) 4.0.UP03 release.
S–2446–4002 Published December 2011 Supports the Cray Linux Environment (CLE) 4.0.UP02 release.
S–2446–3103 Published March 2011 Supports the Cray Linux Environment (CLE) 3.1.UP03 release.
S–2446–3102 Published December 2010 Supports the Cray Linux Environment (CLE) 3.1.UP02 release.
S–2446–3101 Published September 2010 Supports the Cray Linux Environment (CLE) 3.1.UP01 release.
3.1 Published June 2010 Supports the Cray Linux Environment (CLE) 3.1 release and the System Management
Workstation (SMW) 5.1 release.
Changes to this Document
S–2446–51
S–2446–51
This revision of the Using the GNI and DMAPP APIs guide supports the CLE 5.1.UP00 release.
•
On Aries™, maximum transaction size is 1 MB. Requests for larger size now return
GNI_RC_SIZE_ERROR. See PostFma on page 106.
•
New return code GNI_RC_INVALID_STATE from RebuildMemHndl on page 104.
•
Additional CQ attribute, GNI_CQ_PHYS_PAGES. Also, the CQ modes/attributes are no longer defined by
an enumeration; they are defined by #define GNI_CQ_* bitmasks. See CqCreate on page 68.
•
Additional CDM mode, GNI_CDM_MODE_FMA_SMALL_WINDOW. See CdmCreate on page 52.
•
Updates to descriptions of possible values for dlvr_mode in gni_post_descriptor on page 172.
•
New function, dmapp_lock_release on page 217.
•
Updates to dmapp_rma_attrs on page 205 and dmapp_rma_attrs_ext on page 207.
•
Updates to guidance regarding the usage of symmetric/asymmetric memory segments. See Virtual
Memory Domain Handles on page 95.
•
See INST_ID Decreased to 24 Bits on page 49, Macros which Extract User Data From CQ Entry on
page 166, and EpSetEventData on page 72.
•
Update to guidance regarding attaching CD to NIC when forking a child process. See Communication
Domain on page 51.
•
Update to guidance regarding setting cqwrite_value in gni_post_descriptor. See PostCqWrite
on page 139.
S–2446–42
This revision of the Using the GNI and DMAPP APIs guide supports the CLE 4.2.UP00 release.
•
New API provides NIC statistics. See GetNicStat on page 150, ResetNicStat on page 151, and
gni_statistic on page 162.
•
Added information regarding the usage of the cqwrite_value field in the gni_post_descriptor
structure. See gni_post_descriptor on page 172.
•
Added information about maximum transaction size for RDMA_GET and RDMA_PUT operations
to description of length field in gni_post_descriptor structure. See length member in
gni_post_descriptor on page 172.
•
List available CQ macros for extracting event data. See Other CQ Macros on page 167.
•
Corrections to CqErrorStr() return codes. See Return Codes on page 140.
S-2446-5002
This rewrite of the Using the GNI and DMAPP APIs guide supports the CLE 5.0.UP02 release.
•
New function SmsgSetDeliveryMode on page 117 sets delivery mode for SMSG transactions.
•
New macro to extract data from the CQ entry structure GNI_CQ_GET_REM_INST_ID(). See Macros
which Extract User Data From CQ Entry on page 166.
•
Changes to data alignment restrictions for PostFma and PostRdma operations on Aries systems. See
PostFma on page 106 and PostRdma on page 129.
•
Additional details about GNI_POST_FMA_PUT_W_SYNCFLAG. See gni_post_type on page 158.
•
New environment variables available for DMAPP debugging. See DMAPP_DEBUG_CATEGORIES on
page 319, DMAPP_DEBUG_FILE on page 319, and DMAPP_DEBUG_LEVEL on page 319.
•
New routine obtains symmetric heap usage information. See dmapp_sheap_stats on page 311 and
dmapp_sheap_stats_t on page 203.
S–2446–5001
This rewrite of the Using the GNI and DMAPP APIs guide supports the CLE 5.0.UP01 Limited Availability
(LA) release.
•
This release supports the Aries-based, system interconnection network. See Chapter 1, Introduction
on page 37.
•
See Implementation Differences for the Aries ASIC on page 49.
•
New mode GNI_CDM_MODE_USE_PCI_IOMMU may be used to specify that IOMMU memory translation
be used by a communication domain. See modes parameter in CdmCreate on page 52.
•
The Aries NIC contains a collective engine (CE) which provides hardware support for collective
operations. New collective engine command types defined. See Create Logical Endpoints on page 46,
Virtual Collective Engine (VCE) Channel on page 87, and gni_ce_cmd_type on page 156.
•
The Aries NIC requires changes to memory region attributes. See flags parameter used by MemRegister
on page 94.
See MemHndlQueryAttr on page 104. See gni_mem_handle_attr on page 157.
•
New function RebuildMemHndl on page 104.
•
New function CqInterruptMask on page 132 and CqVectorMonitor on page 134.
•
Deadlock prevention logic affects PostFma on page 106 and gni_post_descriptor on page 172.
Added Aries specific delivery modes used in gni_post_descriptor.
•
Aries does not support msg_type GNI_SMSG_TYPE_MBOX. See SmsgSend on page 111 and
gni_smsg_type on page 162.
•
New function MemQueryHndls on page 105 .
•
New function GetPtag on page 151.
•
The default value for the PI_ordering memory registration flag is DMAPP_PI_ORDERING_STRICT
on Gemini™ Systems and DMAPP_PI_ORDERING_RELAXED on Aries Systems. See
dmapp_rma_attrs_ext on page 207.
•
Added dmapp_lock_acquire on page 214, dmapp_lock_try on page 216, and
dmapp_lock_release on page 217.
•
Added dmapp_put_flag_nb on page 226 and dmapp_put_flag_nbi on page 227.
•
Dynamic memory registration supported. See dmapp_mem_reserve on page 313 and
dmapp_mem_update on page 314.
S–2446–41
•
New FMA descriptor flags GNI_CDM_MODE_FMA_SHARED and GNI_CDM_MODE_FMA_DEDICATED
control CDM sharing. See Parameters on page 52.
•
Added return code GNI_RC_NOT_DONE to GNI_CdmAttach. See Return Codes on page 57.
•
Added GNI_MEM_CUDA memory attribute to specify GPU-resident memory. See
MemRegisterSegments Parameters on page 100.
•
Added new structure and API for querying library version GetVersionInformation on page 58 and
gni_version_info on page 181.
•
Added second-generation Aries-only atomic memory operations. See gni_fma_cmd_type on
page 153.
•
New example code in Processing SMSG Messages on page 109.
S–2446–4003
•
Describe protection domain flexibility. See Protection Models on page 44.
•
A new error code (GNI_RC_INVALID_PARAM) is returned by function MsgqProgress when a
timeout value is specified for a non-blocking message queue. See MsgqProgress on page 128.
S–2446–4002
•
Sample uGNI test programs, Makefile, and README are installed in
/opt/cray/ugni/version_string/examples.
•
New GNI functions provide hardware resource information about the NIC. As hardware resources may
change between successive iterations of the NIC, this API is designed to insulate software functionality
from hardware modifications for forward compatibility. See GetDevResInfo on page 147 and
GetJobResInfo on page 148.
•
A new return code has been added to SmsgGetNext. See SmsgGetNext on page 115.
•
There are new meanings for function MsgqProgress return codes GNI_RC_ERROR_RESOURCE and
GNI_RC_NOT_DONE. See MsgqProgress on page 128.
•
New DMAPP Atomic Memory Operations (AMO) are available in blocking and non-blocking put and
get styles. See dmapp_amo_put[_nb | _nbi] on page 298.
•
New DMAPP function dmapp_explain_error provides explanations for DMAPP return values. See
dmapp_explain_error on page 210.
•
DMAPP_ROUTING_LAST was renamed to DMAPP_ROUTING_NOT_USED. See
dmapp_routing_type on page 198.
•
New function to test whether DMAPP has been initialized or not. See dmapp_initialized on
page 212.
•
To redirect error messages issued by the DMAPP library to stdout, set the DMAPP_ERROR_FILE
environment variable to stdout. See DMAPP_ERROR_FILE [STDERR | STDOUT] on page 319.
S–2446–3103
Added information
•
•
•
•
MemHndlQueryAttr on page 104
Additional CDM Modes (see Parameters on page 52)
DMAPP collective operations (see Collective Operations on page 193)
Extended RMA attribute structure and supporting interface.
–
dmapp_pi_reg_type on page 200
–
dmapp_rma_attrs_ext on page 207
–
dmapp_init_ext on page 211
–
dmapp_get_rma_attrs_ext on page 213
–
dmapp_set_rma_attrs_ext on page 220
Revised information
•
gni_cq_entry on page 166
Contents
Page
Introduction [1]
37
1.1 Software Stack
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
38
Part I: The GNI API
About the GNI API [2]
2.1 Functional Overview
.
43
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
44
2.1.1.2 ALPS Manages Domain Properties and the Balanced Injection Configuration
.
.
.
.
.
44
2.1.1.3 Obtain Job Attributes Through PMI
2.1.1 Establish Communication Domain
2.1.1.1 Protection Models
.
2.1.1.4 Create the Domain
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
44
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
45
2.1.2 Create Completion Queue (CQ)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
45
2.1.3 Register Memory
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
45
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
46
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
46
2.1.5.1 Fast Memory Access (FMA)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
46
2.1.5.2 Block Transfer Engine (BTE)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
47
.
.
2.1.4 Create Logical Endpoints
2.1.5 Transfer Data
.
2.1.5.3 Datagrams
.
.
.
.
.
.
2.1.6 Process Completion Queue
2.1.7 Release Resources
2.2 Restrictions
2.3 Compiling
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
48
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
48
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
49
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
49
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
49
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
49
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
49
.
.
.
.
.
.
.
.
.
.
.
.
.
50
2.4 Implementation Differences for the Aries ASIC
2.4.1 INST_ID Decreased to 24 Bits
.
.
.
2.4.2 Deadlock Avoidance Logic for FMA Transactions
2.4.2.1 gni_post_descriptor
2.4.2.2 gni_smsg_attr
2.4.3 Alignment Restrictions
S–2446–51
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
50
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
50
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
50
9
Page
GNI API Reference [3]
3.1 Naming Conventions
.
51
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
51
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
51
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
3.2.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
3.2.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
54
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
54
3.2.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
3.2.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
3.2.3 CdmGetNicAddress
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
3.2 Communication Domain
3.2.1 CdmCreate
.
3.2.1.3 Return Codes
3.2.2 CdmDestroy
3.2.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
3.2.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
55
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
56
3.2.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
56
3.2.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
56
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
3.2.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
3.2.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
58
3.2.6 GetVersionInformation
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
58
3.2.4 CdmAttach
.
3.2.5 GetVersion
3.2.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
58
3.2.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
58
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
58
3.2.7 ConfigureNTT
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
58
3.2.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
59
3.2.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
59
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
59
3.2.8 ConfigureJob
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
60
3.2.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
60
3.2.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
61
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
62
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
62
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
63
3.2.9 ConfigureNTTandJob
3.2.9.1 Synopsis
10
.
.
.
S–2446–51
Contents
Page
3.2.9.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
63
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
64
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
64
3.3 Balanced Injection (BI) Configuration
3.3.1 GetBIConfig
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
3.3.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
3.3.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
3.3.2 SetBIConfig
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
3.3.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
3.3.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
66
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
3.3.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
3.3.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
3.4 Completion Queue Management
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
68
3.3.3 BISyncWait
3.4.1 CqCreate
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
68
3.4.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
69
3.4.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
69
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
3.4.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
3.4.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
3.4.2 CqDestroy
.
3.5 Logical Endpoint
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
71
3.5.1 EpCreate
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
71
3.5.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
71
3.5.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
71
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
72
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
72
3.5.2 EpSetEventData
3.5.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
72
3.5.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
72
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
72
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
3.5.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
3.5.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
3.5.3 EpBind
.
.
S–2446–51
11
Page
3.5.4 EpUnbind
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
3.5.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
3.5.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
3.5.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
3.5.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
74
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
3.5.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
3.5.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
76
3.5.5 EpDestroy
.
3.5.6 EpPostData
3.5.7 EpPostDataWId
3.5.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
76
3.5.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
76
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
77
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
77
3.5.8 EpPostDataTest
3.5.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
77
3.5.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
77
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
78
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
78
3.5.9 EpPostDataTestById
3.5.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
78
3.5.9.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
79
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
79
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
80
3.5.10 EpPostDataWait
3.5.10.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
80
3.5.10.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
80
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
81
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
81
3.5.11 EpPostDataWaitById
3.5.11.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
81
3.5.11.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
82
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
3.5.12 EpPostDataCancel
3.5.12.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
3.5.12.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
3.5.13 EpPostDataCancelById
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
12
S–2446–51
Contents
Page
3.5.13.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
3.5.13.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
3.5.14 PostDataProbe
3.5.14.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
3.5.14.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
85
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
85
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
85
3.5.15 PostDataProbeById
3.5.15.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
85
3.5.15.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
86
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
86
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
86
3.5.16 PostdataProbeWaitById
3.5.16.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
86
3.5.16.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
3.6 Virtual Collective Engine (VCE) Channel
3.6.1 CeCreate
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
88
3.6.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
88
3.6.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
88
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
3.6.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
3.6.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
3.6.2 CeGetId
.
3.6.3 EpSetCeAttr
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
90
3.6.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
91
3.6.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
91
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
91
3.6.4 CeConfigure
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
91
3.6.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
91
3.6.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
3.6.5 CeCheckResult
3.6.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
3.6.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
93
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
93
3.6.6 CeDestroy
S–2446–51
.
.
13
Page
3.6.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
93
3.6.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
93
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
93
3.7 Memory Registration
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
94
3.7.1 MemRegister
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
94
3.7.1.1 Virtual Memory Domain Handles
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
95
3.7.1.2 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
95
3.7.1.3 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
96
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
99
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
99
3.7.2 MemRegisterSegments
3.7.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
100
3.7.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
100
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
102
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
102
3.7.3 SetMddResources
3.7.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
3.7.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
3.7.4 MemDeregister
3.7.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
3.7.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
104
3.7.5 MemHndlQueryAttr
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
104
3.7.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
104
3.7.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
104
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
104
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
104
3.7.6 RebuildMemHndl
3.7.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
105
3.7.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
105
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
105
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
105
3.7.7 MemQueryHndls
3.7.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
105
3.7.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
3.8 FMA DM
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
3.8.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
107
3.8.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
108
3.8.1 PostFma
14
S–2446–51
Contents
Page
.
.
3.9 FMA Short Messaging (SMSG)
3.9.1 Sending SMSG Messages
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
108
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
108
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
108
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
109
3.9.2 Processing SMSG Messages
3.9.3 SmsgInit
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
110
3.9.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
111
3.9.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
111
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
111
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
111
3.9.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
112
3.9.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
113
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
113
3.9.5 SmsgSendWTag
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
113
3.9.4 SmsgSend
.
3.9.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
114
3.9.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
114
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
114
3.9.6 SmsgGetNext
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
115
3.9.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
115
3.9.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
115
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
115
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
115
3.9.7 SmsgGetNextWTag
3.9.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
116
3.9.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
116
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
116
3.9.8 SmsgRelease
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
116
3.9.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
3.9.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
3.9.9 SmsgSetDeliveryMode
3.9.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
3.9.9.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
118
3.9.10 SmsgSetMaxRetrans
3.9.10.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
118
3.9.10.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
118
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
118
3.9.11 SmsgBufferSizeNeeded
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
118
S–2446–51
15
Page
3.9.11.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
118
3.9.11.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
119
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
119
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
119
3.10 Shared Message Queue (MSGQ)
3.10.1 MsgqInit
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
120
3.10.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
121
3.10.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
121
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
122
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
122
3.10.2 MsgqRelease
3.10.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
122
3.10.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
122
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
123
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
123
3.10.3 MsgqGetConnAttrs
3.10.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
123
3.10.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
123
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
124
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
124
3.10.4 MsgqConnect
3.10.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
124
3.10.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
124
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
125
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
125
3.10.5 MsgqConnRelease
3.10.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
125
3.10.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
125
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
126
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
126
3.10.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
126
3.10.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
127
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
127
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
127
3.10.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
128
3.10.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
128
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
128
3.10.8 MsgqProgress
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
128
3.10.6 MsgqSend
.
3.10.7 MsgqSize
.
3.10.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
128
3.10.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
128
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
129
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
129
3.11 RDMA (BTE)
16
.
.
S–2446–51
Contents
Page
3.11.1 PostRdma
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
129
3.11.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
129
3.11.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
130
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
130
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
130
3.12 Completion Queue Processing
3.12.1 CqGetEvent
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
131
3.12.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
131
3.12.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
131
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
132
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
132
3.12.2 CqInterruptMask
3.12.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
132
3.12.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
132
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
133
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
133
3.12.3 CqInterruptUnmask
3.12.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
133
3.12.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
133
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
133
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
134
3.12.4 CqTestEvent
3.12.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
134
3.12.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
134
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
134
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
134
3.12.5 CqVectorMonitor
3.12.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
134
3.12.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
135
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
135
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
135
3.12.6 CqVectorWaitEvent
3.12.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
136
3.12.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
136
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
137
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
137
3.12.7 CqWaitEvent
3.12.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
137
3.12.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
138
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
138
3.12.8 GetCompleted
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
138
3.12.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
138
3.12.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
S–2446–51
17
Page
3.12.9 PostCqWrite
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
3.12.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
3.12.9.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
3.12.10 CqErrorStr
3.12.10.1 Synopsis
.
3.12.10.2 Parameters
3.12.11 CqErrorRecoverable
3.12.11.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
141
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
141
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
141
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
141
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
141
3.13 Error Handling
.
.
.
3.13.1 SubscribeErrors
3.13.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
141
3.13.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
142
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
142
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
142
3.13.2 ReleaseErrors
3.13.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
142
3.13.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
142
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
3.13.3 GetErrorMask
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
3.13.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
3.13.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
3.13.4 SetErrorMask
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
3.13.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
3.13.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
144
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
144
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
144
3.13.5 GetErrorEvent
3.13.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
144
3.13.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
144
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
144
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
145
3.13.6 WaitErrorEvents
3.13.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
145
3.13.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
145
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
145
18
S–2446–51
Contents
Page
3.13.7 SetErrorPtag
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
3.13.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
3.13.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
3.14.1 GetDeviceType
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
3.14 Resource Reporting
3.14.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
3.14.1.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
146
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
147
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
147
3.14.2 GetDevResInfo
3.14.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
147
3.14.2.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
147
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
147
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
148
3.14.3 GetJobResInfo
3.14.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
148
3.14.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
148
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
148
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
149
3.14.4 GetLocalDeviceIds
3.14.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
149
3.14.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
149
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
149
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
149
3.14.5 GetNumLocalDevices
3.14.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
149
3.14.5.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
150
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
150
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
150
3.14.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
150
3.14.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
150
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
150
3.14.7 ResetNicStat
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
3.14.6 GetNicStat
3.14.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
3.14.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
3.14.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
3.14.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
151
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
152
3.14.8 GetPtag
.
S–2446–51
19
Page
3.15 Enumerations
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
152
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
152
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
152
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
3.15.3 gni_nic_device_t
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
3.15.4 gni_fma_cmd_type
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
153
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
156
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
156
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
157
3.15.1 gni_dev_res_t
3.15.1.1 Synopsis
.
.
3.15.2 gni_job_res_t
3.15.2.1 Synopsis
.
3.15.3.1 Synopsis
.
3.15.4.1 Synopsis
.
.
.
.
.
3.15.5 gni_ce_cmd_type
3.15.5.1 Synopsis
.
.
.
3.15.6 gni_mem_handle_attr
3.15.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
157
3.15.6.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
157
3.15.7 gni_post_state
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
158
3.15.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
158
3.15.7.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
158
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
158
3.15.8 gni_post_type
3.15.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
159
3.15.8.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
159
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
160
3.15.9 gni_return
3.15.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
160
3.15.9.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
160
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
162
3.15.10 gni_smsg_type
3.15.10.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
162
3.15.10.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
162
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
162
3.15.11 gni_statistic
3.15.11.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
162
3.15.11.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
163
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
163
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
163
3.16 Structures
.
.
3.16.1 gni_bi_desc
3.16.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
163
3.16.1.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
164
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
164
3.16.2 gni_error_event
20
3.16.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
164
3.16.2.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
164
3.16.3 gni_error_mask
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
165
S–2446–51
Contents
Page
3.16.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
166
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
166
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
166
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
166
.
.
.
.
.
.
.
.
.
.
.
.
166
3.16.4 gni_cq_entry
3.16.4.1 Synopsis
.
3.16.4.2 Event types
3.16.4.3 Macros which Extract User Data From CQ Entry
3.16.4.4 Other CQ Macros
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
167
3.16.4.5 Status
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
169
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
169
.
.
.
3.16.5 gni_dev_res_desc_t
3.16.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
169
3.16.6 gni_job_limits
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
169
3.16.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
170
3.16.6.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
170
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
3.16.7 gni_job_res_desc_t
3.16.7.1 Synopsis
.
.
.
3.16.8 gni_mem_segment
3.16.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
3.16.8.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
3.16.9 gni_ntt_descriptor
3.16.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
171
3.16.9.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
172
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
172
3.16.10 gni_post_descriptor
3.16.10.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
173
3.16.10.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
173
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
178
3.16.11 gni_smsg_attr
3.16.11.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
178
3.16.11.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
178
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
179
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
179
3.16.12 gni_smsg_handle
3.16.13 gni_msgq_attr
3.16.13.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
179
3.16.13.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
180
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
180
3.16.14 gni_msgq_rem_inst
3.16.14.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
180
3.16.14.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
180
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
180
3.16.15 gni_msgq_ep_attr
3.16.15.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
180
3.16.15.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
181
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
181
3.16.16 gni_version_info
S–2446–51
21
Page
3.16.16.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
181
3.16.16.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
181
Part II: The DMAPP API
About the DMAPP API [4]
185
4.1 DMAPP Programming Model
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
185
4.2 DMAPP Applications and Fork
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
186
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
186
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
186
4.5 DMAPP Application Intra-node Communication
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
186
4.6 Compiling and Launching DMAPP Applications
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
186
4.3 DMAPP Applications and Threads
4.4 DMAPP Applications and File Descriptors
4.7 Resiliency
.
.
.
.
.
.
.
4.8 DMAPP Remote Memory Access
4.9 DMAPP API
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
187
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
187
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
189
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
189
4.9.1 Initialization and Query Functions
4.9.2 One-sided RMA Functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
189
4.9.2.1 Contiguous Functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
190
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
190
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
191
4.9.2.2 Strided Functions
.
4.9.2.3 SCATTER/GATHER Functions
4.9.2.4 PE-strided Functions
.
4.9.2.5 DMAPP AMO Functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
191
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
192
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
193
4.9.2.6 DMAPP Synchronization Functions
4.9.2.7 Collective Operations
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
193
4.9.3 Symmetric Heap Functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
195
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
195
4.9.4 Checkpoint Restart Functions
DMAPP API Reference [5]
5.1 DMAPP Enumerations
5.1.1 dmapp_return
197
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
197
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
197
5.1.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
197
5.1.2 dmapp_type
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
197
5.1.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
198
5.1.2.2 Constants
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
198
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
198
5.1.3 dmapp_routing_type
22
5.1.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
198
5.1.3.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
199
5.1.4 dmapp_pi_reg_type
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
200
5.1.5 dmapp_c_op
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
201
.
.
.
S–2446–51
Contents
Page
5.1.6 dmapp_c_type
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
201
5.2 DMAPP Structures
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
201
5.2.1 dmapp_amo
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
201
5.2.1.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
202
5.2.1.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
202
5.2.1.3 Supported AMOs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
202
5.2.2 dmapp_seg_desc
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
203
5.2.2.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
203
5.2.2.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
203
5.2.3 dmapp_sheap_stats_t
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
203
5.2.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
204
5.2.3.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
204
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
204
5.2.4 dmapp_jobinfo
5.2.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
205
5.2.4.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
205
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
205
5.2.5 dmapp_rma_attrs
5.2.5.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
205
5.2.5.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
205
5.2.6 dmapp_rma_attrs_ext
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
207
5.2.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
207
5.2.6.2 Members
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
207
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
209
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
209
5.2.8 dmapp_c_pset_desc_t
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
209
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
209
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
209
5.2.7 dmapp_syncid
5.2.7.1 Synopsis
5.2.8.1 Synopsis
.
.
.
.
.
5.2.9 dmapp_c_pset_handle_t
5.2.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
209
5.3 DMAPP Functions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
210
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
210
5.3.1 dmapp_explain_error
5.3.1.1 Synopsis
5.3.2 dmapp_init
5.3.2.1 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
210
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
210
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
210
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
211
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
211
5.3.3 dmapp_init_ext
5.3.3.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
211
5.3.3.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
211
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
211
S–2446–51
23
Page
5.3.4 dmapp_initialized
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
5.3.4.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
5.3.4.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
5.3.5 dmapp_finalize
5.3.5.1 Synopsis
.
5.3.6 dmapp_get_jobinfo
5.3.6.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
212
5.3.6.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
5.3.7 dmapp_get_rma_attrs
5.3.7.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
5.3.7.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
213
5.3.8 dmapp_get_rma_attrs_ext
5.3.8.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
5.3.8.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
5.3.9 dmapp_get_version
5.3.9.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
5.3.9.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
214
5.3.10 dmapp_lock_acquire
5.3.10.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
215
5.3.10.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
215
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
216
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
216
5.3.11 dmapp_lock_try
5.3.11.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
216
5.3.11.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
216
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
217
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
217
5.3.12 dmapp_lock_release
5.3.12.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
217
5.3.12.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
217
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
218
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
218
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
218
5.3.13 dmapp_lock_release_addr
5.3.13.1 Synopsis
24
.
.
.
.
.
.
S–2446–51
Contents
Page
5.3.13.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
218
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
219
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
219
5.3.14 dmapp_set_rma_attrs
5.3.14.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
219
5.3.14.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
219
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
220
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
220
5.3.15 dmapp_set_rma_attrs_ext
5.3.15.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
220
5.3.15.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
220
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
220
5.3.16 dmapp_put_nb
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
220
5.3.16.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
221
5.3.16.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
221
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
221
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
222
5.3.17 dmapp_put_nbi
5.3.17.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
222
5.3.17.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
222
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
223
5.3.18 dmapp_put
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
223
5.3.18.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
223
5.3.18.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
224
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
224
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
224
5.3.19 dmapp_put_flag
5.3.19.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
225
5.3.19.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
225
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
225
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
226
5.3.20 dmapp_put_flag_nb
5.3.20.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
226
5.3.20.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
226
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
227
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
227
5.3.21 dmapp_put_flag_nbi
5.3.21.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
227
5.3.21.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
228
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
228
5.3.22 dmapp_get_nb
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
229
5.3.22.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
229
5.3.22.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
229
S–2446–51
25
Page
5.3.23 dmapp_get_nbi
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
230
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
230
5.3.23.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
230
5.3.23.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
231
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
231
5.3.24 dmapp_get
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
231
5.3.24.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
232
5.3.24.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
232
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
232
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
233
5.3.25 dmapp_get_flag_nbi
5.3.25.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
233
5.3.25.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
233
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
234
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
234
5.3.26 dmapp_iput_nb
5.3.26.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
234
5.3.26.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
235
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
235
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
235
5.3.27 dmapp_iput_nbi
5.3.27.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
236
5.3.27.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
236
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
236
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
236
5.3.28.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
237
5.3.28.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
237
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
237
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
238
5.3.28 dmapp_iput
5.3.29 dmapp_iget_nb
5.3.29.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
238
5.3.29.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
238
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
239
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
239
5.3.30 dmapp_iget_nbi
5.3.30.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
239
5.3.30.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
240
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
240
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
240
5.3.31.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
241
5.3.31.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
241
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
242
5.3.31 dmapp_iget
26
S–2446–51
Contents
Page
5.3.32 dmapp_ixput_nb
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
242
5.3.32.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
242
5.3.32.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
243
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
243
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
243
5.3.33 dmapp_ixput_nbi
5.3.33.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
244
5.3.33.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
244
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
244
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
244
5.3.34 dmapp_ixput
5.3.34.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
245
5.3.34.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
245
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
245
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
245
5.3.35 dmapp_ixget_nb
5.3.35.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
246
5.3.35.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
246
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
247
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
247
5.3.36 dmapp_ixget_nbi
5.3.36.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
247
5.3.36.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
248
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
248
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
248
5.3.37 dmapp_ixget
5.3.37.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
249
5.3.37.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
249
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
250
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
250
5.3.38 dmapp_put_ixpe_nb
5.3.38.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
250
5.3.38.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
251
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
251
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
251
5.3.39 dmapp_put_ixpe_nbi
5.3.39.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
252
5.3.39.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
252
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
252
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
253
5.3.40 dmapp_put_ixpe
5.3.40.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
253
5.3.40.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
253
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
254
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
254
5.3.41 dmapp_scatter_ixpe_nb
S–2446–51
27
Page
5.3.41.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
254
5.3.41.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
255
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
255
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
255
5.3.42 dmapp_scatter_ixpe_nbi
5.3.42.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
256
5.3.42.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
256
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
257
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
257
5.3.43 dmapp_scatter_ixpe
5.3.43.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
257
5.3.43.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
258
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
258
5.3.44 dmapp_gather_ixpe_nb
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
259
5.3.44.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
259
5.3.44.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
259
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
260
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
260
5.3.45 dmapp_gather_ixpe_nbi
5.3.45.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
260
5.3.45.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
261
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
261
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
261
5.3.46 dmapp_gather_ixpe
5.3.46.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
262
5.3.46.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
262
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
263
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
263
5.3.47 dmapp_aadd_qw_nb
5.3.47.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
263
5.3.47.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
263
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
264
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
264
5.3.48 dmapp_aadd_qw_nbi
5.3.48.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
264
5.3.48.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
264
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
265
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
265
5.3.49 dmapp_aadd_qw
5.3.49.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
265
5.3.49.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
265
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
266
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
266
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
266
5.3.50 dmapp_aand_qw_nb
5.3.50.1 Synopsis
28
.
.
.
S–2446–51
Contents
Page
5.3.50.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
266
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
267
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
267
5.3.51 dmapp_aand_qw_nbi
5.3.51.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
267
5.3.51.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
267
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
268
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
268
5.3.52 dmapp_aand_qw
5.3.52.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
268
5.3.52.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
268
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
269
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
269
5.3.53 dmapp_aor_qw_nb
5.3.53.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
269
5.3.53.2 Parameter
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
269
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
270
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
270
5.3.54 dmapp_aor_qw_nbi
5.3.54.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
270
5.3.54.2 Parameter
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
270
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
271
5.3.55 dmapp_aor_qw
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
271
5.3.55.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
271
5.3.55.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
271
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
272
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
272
5.3.56 dmapp_axor_qw_nb
5.3.56.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
272
5.3.56.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
272
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
273
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
273
5.3.57 dmapp_axor_qw_nbi
5.3.57.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
273
5.3.57.2 Parameter
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
273
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
274
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
274
5.3.58 dmapp_axor_qw
5.3.58.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
274
5.3.58.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
274
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
275
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
275
5.3.59 dmapp_afadd_qw_nb
5.3.59.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
275
5.3.59.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
276
S–2446–51
29
Page
.
.
.
5.3.60 dmapp_afadd_qw_nbi
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
276
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
276
5.3.60.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
276
5.3.60.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
277
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
277
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
277
5.3.61 dmapp_afadd_qw
5.3.61.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
277
5.3.61.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
278
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
278
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
278
5.3.62 dmapp_afand_qw_nb
5.3.62.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
279
5.3.62.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
279
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
279
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
280
5.3.63 dmapp_afand_qw_nbi
5.3.63.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
280
5.3.63.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
280
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
280
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
281
5.3.64 dmapp_afand_qw
5.3.64.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
281
5.3.64.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
281
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
281
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
282
5.3.65 dmapp_afxor_qw_nb
5.3.65.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
282
5.3.65.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
282
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
283
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
283
5.3.66 dmapp_afxor_qw_nbi
5.3.66.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
283
5.3.66.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
283
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
284
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
284
5.3.67 dmapp_afxor_qw
5.3.67.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
284
5.3.67.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
284
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
285
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
285
5.3.68 dmapp_afor_qw_nb
5.3.68.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
285
5.3.68.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
286
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
286
30
.
S–2446–51
Contents
Page
5.3.69 dmapp_afor_qw_nbi
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
286
5.3.69.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
287
5.3.69.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
287
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
287
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
287
5.3.70 dmapp_afor_qw
5.3.70.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
288
5.3.70.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
288
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
288
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
289
5.3.71 dmapp_afax_qw_nb
5.3.71.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
289
5.3.71.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
289
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
290
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
290
5.3.72 dmapp_afax_qw_nbi
5.3.72.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
290
5.3.72.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
291
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
291
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
291
5.3.73 dmapp_afax_qw
5.3.73.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
292
5.3.73.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
292
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
293
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
293
5.3.74 dmapp_acswap_qw_nb
5.3.74.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
293
5.3.74.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
294
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
294
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
294
5.3.75 dmapp_acswap_qw_nbi
5.3.75.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
295
5.3.75.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
295
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
295
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
296
5.3.76 dmapp_acswap_qw
5.3.76.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
296
5.3.76.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
296
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
297
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
297
5.3.77 dmapp_amo_flush
5.3.77.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
297
5.3.77.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
297
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
298
5.3.78 dmapp_amo_put[_nb | _nbi]
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
298
S–2446–51
31
Page
5.3.78.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
298
5.3.78.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
298
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
299
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
299
5.3.79 dmapp_amo_get[_nb|_nbi]
5.3.79.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
299
5.3.79.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
300
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
300
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
300
5.3.80 dmapp_syncid_test
5.3.80.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
300
5.3.80.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
5.3.81 dmapp_syncid_wait
5.3.81.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
5.3.81.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
5.3.82 dmapp_gsync_test
5.3.82.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
5.3.82.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
5.3.83 dmapp_gsync_wait
5.3.83.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
302
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
303
5.3.84 dmapp_c_pset_create
5.3.84.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
303
5.3.84.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
303
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
304
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
304
5.3.85 dmapp_c_pset_export
5.3.85.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
304
5.3.85.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
304
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
305
5.3.86 dmapp_c_barrier_join
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
305
5.3.86.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
305
5.3.86.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
305
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
305
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
5.3.87 dmapp_c_pset_cancel_op
32
5.3.87.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
5.3.87.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
S–2446–51
Contents
Page
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
5.3.88 dmapp_c_pset_destroy
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
5.3.88.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
5.3.88.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
306
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
307
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
307
5.3.89 dmapp_c_greduce_start
5.3.89.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
307
5.3.89.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
308
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
308
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
308
5.3.90 dmapp_c_greduce_nelems_max
5.3.90.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
308
5.3.90.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
309
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
309
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
309
5.3.91 dmapp_c_pset_test
5.3.91.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
309
5.3.91.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
309
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
309
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
5.3.92 dmapp_c_pset_wait
5.3.92.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
5.3.92.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
5.3.93 dmapp_sheap_malloc
5.3.93.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
5.3.93.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
310
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
5.3.94 dmapp_sheap_realloc
5.3.94.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
5.3.94.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
5.3.95 dmapp_sheap_free
5.3.95.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
5.3.95.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
5.3.96 dmapp_sheap_stats
5.3.96.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
311
5.3.96.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
312
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
312
5.3.97 dmapp_mem_register
5.3.97.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
312
5.3.97.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
312
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
312
S–2446–51
33
Page
5.3.98 dmapp_mem_unregister
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
313
5.3.98.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
313
5.3.98.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
313
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
313
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
313
5.3.99 dmapp_mem_reserve
5.3.99.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
314
5.3.99.2 Parameters
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
314
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
314
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
314
5.3.100 dmapp_mem_update
5.3.100.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
315
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
315
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
315
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
315
5.3.101 dmapp_segdesc_compare
5.3.101.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
315
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
316
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
316
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
316
5.3.102 dmapp_register_process_cb
5.3.102.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
316
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
316
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
317
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
317
5.3.103 dmapp_deregister_progress_cb
5.3.103.1 Synopsis
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
317
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
317
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
317
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
318
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
5.3.104 dmapp_checkpoint
5.3.104.1 Synopsis
.
.
5.3.105 dmapp_restart
5.3.105.1 Synopsis
.
5.4 Environment Variables Which Affect DMAPP
5.4.1 XT_SYMMETRIC_HEAP_SIZE
5.4.2 DMAPP_ABORT_ON_ERROR
.
5.4.3 DMAPP_DEBUG_CATEGORIES
34
5.4.4 DMAPP_DEBUG_FILE
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
5.4.5 DMAPP_DEBUG_LEVEL
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
S–2446–51
Contents
Page
5.4.6 DMAPP_ERROR_FILE [STDERR | STDOUT]
5.4.7 DMAPP_PUT_NBI_CHAIN_OFF
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
319
.
.
.
.
.
.
.
.
.
.
.
.
319
Appendix A Sample Code
A.1 uGNI
.
A.2 DMAPP
321
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
321
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
321
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
321
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
323
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
59
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
202
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
39
A.2.1 dmapp_sample_put.c
A.2.2 dmapp_sample_get.c
Tables
Table 1.
gni_ntt_descriptor
Table 2.
Supported DMAPP AMOs
.
Figures
Figure 1.
S–2446–51
GNI and DMAPP Software Layers
35
Introduction [1]
This guide includes reference information for the Generic Network Interface
(GNI) and Distributed Shared Memory Application (DMAPP) APIs. The intended
audience are programmers who are developing system software such as Partitioned
Global Address Space (PGAS) compilers and communication libraries that use the
Cray network application-specific integrated circuit (ASIC) to transfer data between
processors on a Cray System.
The Cray network application-specific integrated circuit (ASIC) provides an
interface between the processors and the interconnection network. The ASIC
provides the address translation mechanism, communication modes, and low-latency
synchronization necessary to support the abstraction of a global, shared address space
across the entire machine.
Each Aries ASIC includes four network interface controllers (NICs), and 48
embedded router tiles. Each NIC is an independent, addressable endpoint in the
network, therefore a single ASIC supports 4 nodes. Each Gemini ASIC includes two
network interface controllers (NICs), therefore a single ASIC supports 2 nodes.
The Cray network application-specific integrated circuit (ASIC) and its associated
software provides the following features:
S–2446–51
•
Support for message passing, one-sided operations, and global address space
programming models.
•
Global synchronization. Global timing information is passed through the
high-speed network to synchronize the scheduler interrupts and time-of-day
clocks in all the processors.
•
Gather/scatter performance. A symmetric address translation mode allows access
to all nodes in a job without needing to modify the fast memory access (FMA)
window. A windowing mechanism allows for processors with limited physical
addressing capabilities to efficiently access remote memory. Network packet
overhead is reduced so that network efficiency is high during these operations.
•
Flat collectives. Support for atomic memory operations plus efficient scatters
allows collectives to be programmed in a vector-like manner to scale much better
than typical message-based algorithms.
•
End-to-end data protection. Hardware support is provided so that all packets
37
between the sender and receiver receive a cyclic redundancy check (CRC) to
detect data corruption. Further, link-level data is resent if an error occurs while
data is transiting a link.
•
Network routing allows you to add and delete nodes from the network while it is
running.
•
Adaptive routing may be used for most network data, reducing sensitivity to
network hot spots.
1.1 Software Stack
uGNI and DMAPP provide low-level communication services to user-space software.
uGNI directly exposes the communications capabilities of the Cray network
application-specific integrated circuit (ASIC), and is extensively described in Part I,
The GNI API. DMAPP implements a logically shared, distributed memory (DM)
programming model, and is extensively described in Part II, The DMAPP API.
The uGNI and DMAPP APIs allow system software to realize as much of the
hardware performance of the Cray network application-specific integrated circuit
(ASIC) as possible while being reasonably portable to its successors.
38
S–2446–51
Introduction [1]
Figure 1. GNI and DMAPP Software Layers
Applications
GAMESS
NWChem
F90/C/C++
Compilers
Hardware
Independent
Libraries
Global Arrays
DDI
MPICH2
F08/UPC
SHMEM
libpgas
ARMCI
libonesided
Hardware
Dependent
Libraries
uGNI
DMAPP
I
O
C
T
L
Kernel Level GNI
(kGNI)
GNI Core
D
i
r
e
c
t
D
i
r
e
c
t
A
c
c
e
s
s
A
c
c
e
s
s
Generic Hardware Abstraction Layer
(GHAL)
Hardware
kGNI is a kernel module that presents to kernel-space code an API similar to that of
uGNI. The GNI Core provides low-level services to both uGNI and kGNI. kGNI and
GNI Core are both in the kGNI module.
The Generic Hardware Abstraction Layer (GHAL) isolates all software from the
hardware specifics of the Cray network application-specific integrated circuit (ASIC).
These components are not described further in this book.
S–2446–51
39
Layered on top of uGNI and DMAPP are portable communication libraries (such
as MPICH and Cray SHMEM) and the Partitioned Global Address Space (PGAS)
compilers (such as UPC and Coarray Fortran, labeled F08, in Figure 1). These
software components are extensively described in other books available from Cray
Inc.
uGNI and DMAPP are packaged as libraries available with the Cray Linux
Environment (CLE) release and are installed in /opt/cray/ugni and
/opt/cray/dmapp.
40
S–2446–51
Part I: The GNI API
The GNI API includes two sets of function calls. User-level high-performance
applications use uGNI functions while kernel-level drivers use kGNI functions. This
chapter describes the functionality of the uGNI set of function calls, focusing on their
direct interaction with the NIC.
Sample uGNI test programs, Makefile and README files are installed in
/opt/cray/ugni/version_string/examples.
2.1 Functional Overview
A high-performance user-level application would use the uGNI API to accomplish the
following tasks in order to establish communication among its instances:
•
Establish a communication domain and attach it to an NIC device
•
Create one or more completion queues (CQs)
•
Register memory for use by the NIC
•
Create logical endpoints
•
Use Fast Memory Access (FMA) or Remote Direct Memory Access (RDMA) to
communicate between endpoints.
•
Release resources
Each of these tasks comprises a category of API functions as described in the
following sections.
2.1.1 Establish Communication Domain
The uGNI hardware security protection model uses communication domains to
establish secure network communication. The communication domain allows an
application to enforce a protection scheme across all of its network transactions.
S–2446–51
43
2.1.1.1 Protection Models
Applications or sets of applications may use one of the following protection models,
all of which are implemented using the communication domains:
•
A single communication domain per application prevents one application from
accessing another application's memory regions. This is the most common
implementation of a communication domain and is also referred to as a private
protection domain.
•
Multiple cooperating applications owned by the same user may share a
communication domain. This prevents non-cooperating applications from
accessing another application's memory regions. A communication domain that is
shared between cooperating applications is also referred to as a shared protection
domain.
•
Multiple users of several applications may share the use of a system service which
uses a system dedicated protection domain.
Flexible communication domains and their configuration are further described in
Managing System Software for the Cray Linux Environment.
2.1.1.2 ALPS Manages Domain Properties and the Balanced Injection Configuration
The properties associated with a communication domain include a 32-bit cookie
and 8-bit protection tag (pTag). ALPS assigns cookies and pTags and manages
them according to the configured protection model, which by default is the private
protection domain model.
ALPS calls GNI_ConfigureJob to manage cookie and pTag assignment. See
ConfigureJob on page 60.
ALPS also manages the balanced injection settings and ALPS calls GetBIConfig
and SetBIConfig to do so. For more information regarding these interfaces, see
Balanced Injection (BI) Configuration on page 64. For more information regarding
the usage and configuration of the balanced injection feature, see Using Balanced
Injection in Cray Systems.
2.1.1.3 Obtain Job Attributes Through PMI
The Process Management Interface (PMI) has provided an interface between ALPS
and OS/Programming Environment software across several generations of Cray
systems. So too, the uGNI application obtains the job attributes assigned to it by
ALPS using PMI.
Sample uGNI code provided in this release uses the PMI API to inquire as to what
cookie value(s), pTag, and ntt base/grp_size can be used for GNI resource
initialization.
44
S–2446–51
2.1.1.4 Create the Domain
Each process in an application creates a local instance of the communication domain
and obtains an identifier for future reference. The application links the local instance
of the domain to a local NIC.
See Communication Domain on page 51.
2.1.2 Create Completion Queue (CQ)
Completion Queues (CQ) provide an event notification mechanism. For example, an
application may use them to track the progress of Fast Memory Access (FMA) or
Block Transfer Engine (BTE) requests, or to notify a remote node that data has been
delivered to local memory.
An application must first initialize a CQ to obtain a completion queue handle, which
is used for subsequent CQ references. An application then associates the CQ with
the logical endpoints and with registered memory segments to be used for future
data transactions. After initiating transactions between endpoints, an application
references the associated CQ to track various events that are related to transaction
completion, messaging notifications, and errors.
Completion queues have a fixed size which is specified when they are created. When
the queue is full, it is said to be in the overrun state. CQEs received when the CQ is
full are discarded.
Local completion queues track the completion of operations initiated on local
endpoints. They are linked to these endpoints by being specified as a parameter to
EpCreate(). See EpCreate on page 71.
Receive completion queues notify the application of completion of operations
initiated on remote endpoints targeting local registered memory. They are linked
with this memory by being specified as a parameter to MemRegister(). See
MemRegister on page 94.
See Completion Queue Management on page 68.
2.1.3 Register Memory
Memory allocated by an application must be registered before it can be given to a
peer as a destination buffer or used as a source buffer for most uGNI transactions.
Registration associates a specific memory segment (described by a pointer and a size)
with an NIC that will be performing transactions from/to this memory. Memory can
be registered with multiple NICs at the same time; it is up to the application to ensure
that the NICs do not use the memory simultaneously.
S–2446–51
45
When an application registers a memory segment, it receives a memory handle for
subsequent references to that segment. The application attaches that segment to an
NIC and must keep track of the handles for each attached NIC and deregister the
associated memory when no longer needed. See Memory Registration on page 94.
2.1.4 Create Logical Endpoints
Before instances of an application can start communicating with each other, the
logical local and remote endpoints have to be created within a communication
domain. Endpoint properties include the handle of the NIC device used for this
connection, the remote PE, and the local CQ. Applications usually synchronize
among instances before attempting to bind endpoints. When no longer needed, the
application must unbind the endpoint explicitly through a function call or implicitly
by destroying the endpoint. See Logical Endpoint on page 71.
The Aries NIC contains a Collective Engine (CE) which provides hardware support
for gather/scatter operations. In order to use the CE, once a logical endpoint exits, it
must then be associated with a Virtual Collective Engine (VCE) channel. See Virtual
Collective Engine (VCE) Channel on page 87.
2.1.5 Transfer Data
There are two hardware mechanisms for accessing remote memory on another
node — Fast Memory Access (FMA), and Block Transfer Engine (BTE).
For some transfer operations, the GNI kernel driver first exchanges datagrams, which
contain messaging parameters, to initialize communication between PEs.
2.1.5.1 Fast Memory Access (FMA)
Use FMA primarily for the efficient transfer of small, possibly non-contiguous blocks
of data between local and remote memory. For example, use FMA for the short
inter-process data transfers typical of one-sided communication in models like Cray
SHMEM, UPC or Coarray Fortran.
GNI implements FMA through a set of memory windows that enable data to be
moved by the processor directly from user space, through the ASIC to the network.
Stores into an FMA window are used to generate remote memory reference requests.
46
S–2446–51
Several forms of FMA transactions are available:
•
Use the short message (SMSG) interface to send and receive point-to-point short
messages. Each connection uses a dedicated buffer allocated on the remote peer
called a Mailbox. The maximum size of a SMSG message is 65,535 bytes.
Each connection requires a mailbox on the remote peer; for large jobs, memory
constraints limit the use of short messaging interface for all-to-all communication
patterns. See FMA Short Messaging (SMSG) on page 108.
•
Use the Shared Message Queue (MSGQ) interface to send and receive
point-to-point short messages for large jobs (exceeding 200,000 ranks). MSGQ
uses the SMSG facilities for sending and receiving messages. For increased
scalability, the MSGQ interface allows all job instances on a node to share the
message buffer resources required for an SMSG connection. Keeps SMSG control
information and mailbox buffer space in a shared buffer for all job instances
on a node to access. Greater scalability is achieved because the number of
point-to-point connections needed in a job will scale by the node count rather than
by the number of ranks. See Shared Message Queue (MSGQ) on page 119.
•
Use FMA DM to access Distributed Memory (DM), moving user data between
local and remote memory. An application prepares a Transaction Request
Descriptor, which has properties such as type (PUT/GET), CQ, data source and
destination, and length. To post the transaction, an application passes the pointer
to a Transaction Request Descriptor to the PostFma function. See FMA DM
on page 106.
•
To execute an atomic memory operation (AMO), an application prepares a
Transaction Request Descriptor by specifying the remote node, the AMO
command to execute, operands, and other fields, depending on the syntax of the
AMO command. To post the transaction, the application passes the pointer to a
Transaction Request Descriptor to the PostFma function. See FMA DM on
page 106.
2.1.5.2 Block Transfer Engine (BTE)
The BTE functionality, which is implemented on the ASIC, is intended primarily for
large asynchronous data transfers between nodes. More time is required to set up
data for a transfer, than for an FMA transfer, but once initiated, there is no further
involvement by the processor core.
S–2446–51
47
An application can direct the Block Transfer Engine (BTE) to perform an RDMA
PUT operation, which instructs the BTE to move data from local to a remote memory,
or an RDMA GET operation, which instructs the BTE to move data from remote
to local memory and to notify a source and destination upon completion. These
functions write block transfer descriptors to a queue in the NIC, and the BTE services
the requests asynchronously. Block transfer descriptors use privileged state, so access
to the BTE is gated through the kernel. Due to the overhead of accessing the BTE
through the operating system, the BTE mechanism is more appropriate for larger
messages.
PUT/GET transactions require a pointer and a memory domain handle to identify
the data source and destination, data length and return a transaction ID. These
operations use several modes, some of which are appropriate for kernel-level
applications because they bypass memory registration. Other modes are targeted for
user applications which control data ordering, event notification, and synchronization.
See RDMA (BTE) on page 129.
2.1.5.3 Datagrams
Datagrams are used by the GNI library for initial communication to set up SMSG
or MSGQ connections. The datagram is used by the GNI library to initialize
communication between PEs, before the send message transaction can occur. The
receiving process posts a wildcard datagram on a local endpoint which does not know
where connections will be arriving from (it is in an unbound state). The wildcard
on the unbound endpoint will match incoming datagrams for the local hardware
address and instance identifier. The peer initiating the connection posts a datagram
to its own endpoint, which already has local and a remote hardware address and
instance identifier established (it is said to be in the bound state, in that it knows
its destination). The initiating peer sends the outgoing packet to the receiving node
where it matches the hardware address and instance id to one of the posted wildcards
and sends the reply packet back to the originating node. Both nodes are then able to
see the datagram as matched; the receiving buffer for the SMSG or MSGQ is also
referred to as a mailbox.
2.1.6 Process Completion Queue
The calling process must poll a completion queue for a completion entry to discover
information about events, such as transaction completions, message notifications, and
errors, which are generated by the NIC device. If a new completion entry is found,
the application processes status information and event data.
Any error in the network that leads to data loss will result in the NIC's generating an
interrupt and delivering an error to the completion queue associated with the logical
endpoints of the transaction within that communication domain.
48
S–2446–51
To avoid dropped completion notifications, applications should ensure that the
number of operations posted on endpoints attached to a src_cq_handle does not
exceed the completion queue capacity at any time. See Completion Queue Processing
on page 130.
2.1.7 Release Resources
After a process completes the processing of transactions, before the process
terminates, it should release its resources back to the system in the reverse order of
setup. Wait for all the processes to complete before cleaning up and exiting. Free any
allocated memory buffers. Deregister any registered memory. Unbind endpoints
from remote addresses. Destroy endpoints. Destroy completion queues. Destroy the
communication domain. Clean up any PMI information.
2.2 Restrictions
The total number of GNI application processes running on a given node should be
limited to the number of CPU cores of the node.
Applications must do their own locking around uGNI API calls. uGNI is not
thread-safe.
2.3 Compiling
To compile a routine that makes GNI calls, include the gni_pub.h header, which is
contained in the gni-headers module. Other required headers are stdint.h and
sys/types.h.
pmi.h is required to access the Process Management Interface (PMI) API.
To link a routine that makes GNI calls, link with the ugni library, contained in the
ugni module.
2.4 Implementation Differences for the Aries ASIC
2.4.1 INST_ID Decreased to 24 Bits
In order to increase the amount of information passed in the completion queue event,
the INST_ID was decreased to 24 bits on Aries systems. The INST_ID on Gemini
systems is 32 bits. However, note that the remote instance id can still use 32 bits on
both Aries and Gemini Systems. See EpSetEventData on page 72.
S–2446–51
49
2.4.2 Deadlock Avoidance Logic for FMA Transactions
2.4.2.1 gni_post_descriptor
The cq_mode member in gni_post_descriptor_t must include
GNI_CQMODE_GLOBAL_EVENT. Failure to set GNI_CQMODE_GLOBAL_EVENT
for a PostFMA/PostRDMA transaction will result in the runtime error
GNI_RC_INVALID_PARAM.
cq_mode GNI_CQMODE_SILENT is deprecated.
There are new routing modes on Aries hardware. Existing Gemini routing modes map
onto Aries routing modes, however it is an error to specify multiple routing modes.
There are several new routing modes introduced for the Aries NIC. See dlvr_mode
in gni_post_descriptor on page 172.
2.4.2.2 gni_smsg_attr
For short messages (SMSG), the msg_type must be
GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT. The msg_type
GNI_SMSG_TYPE_MBOX is deprecated. See gni_smsg_attr on page 178.
2.4.3 Alignment Restrictions
There are changes to data alignment restrictions for PostFma and PostRdma
operations on Aries systems. See PostFma on page 106 and PostRdma on
page 129.
50
S–2446–51
This chapter contains reference information for functions, structures, and
enumerations contained in the GNI API. Your application must include the
gni_pub.h file when using this API.
3.1 Naming Conventions
The GNI API defines four types of entities: functions, types, return codes and
constants. User-level functions start with GNI_ and use mixed upper and lower
case. Kernel-level functions start with gni_ and use lower case with underscores
to separate words.
Only user-level GNI functions (uGNI) are documented in this guide.
3.2 Communication Domain
After obtaining a cookie and ptag from the Process Management Interface (PMI), a
uGNI application may establish a communication domain which links the unique
domain properties (ptag and cookie) to an identifier called cdm_handle.
The 8-bit pTag is used by the NIC to enforce a protection scheme for an application's
network transactions involving access to remote memory regions. A cookie is used by
kGNI to verify incoming datagram packets. The NIC requires and enforces that the
same pTag is used by both the sending and receiving sides of a remote memory access
request. A pTag is used in FMA descriptors and BTE TX descriptors.
Each process in an application that requires participation in a communication domain
calls CdmCreate to create an instance of the communication domain and to obtain a
cdm_handle for future reference.
CdmAttach links the local instance of the communication domain to a local NIC,
associating cdm_handle to a local nic_handle. In parallel, other processes in the
application may be associated with the same domain by calling CdmCreate with the
same unique ptag and cookie, allowing them to associate their own local NICs to the
domain. CdmGetNicAddress obtains local NIC addresses. The user is responsible
for distributing its local address among the peers and obtaining remote addresses
of other nodes participating in the job.
An application can create multiple Communication Domains using the same pTag and
cookie, but each CdmCreate will require a unique instance ID.
S–2446–51
51
When a Communication Domain exists and the process forks a child, the
Communication Domain will be destroyed in the child process. The application will
need to create a Communication Domain and attach the Communication Domain to
the NIC.
CdmDestroy returns resources back to the system when they are no longer used.
3.2.1 CdmCreate
The GNI_CdmCreate function creates an instance of the communication domain.
3.2.1.1 Synopsis
gni_return_t GNI_CdmCreate (
IN uint32_t inst_id,
IN uint8_t ptag,
IN uint32_t cookie,
IN uint32_t modes,
OUT gni_cdm_handle_t *cdm_handle)
3.2.1.2 Parameters
inst_id
Rank of the instance in the job.
ptag
Protection tag.
cookie
Unique identifier generated by the system. Along with ptag, the
cookie identifies the communication domain.
modes
The modes bit mask. The following flags are used for this parameter:
•
•
One of the following mutually exclusive flags:
–
GNI_CDM_MODE_FORK_NOCOPY
–
GNI_CDM_MODE_FORK_PARTCOPY
–
GNI_CDM_MODE_FORK_FULLCOPY
One of the following flags:
–
GNI_CDM_MODE_FMA_SHARED
–
GNI_CDM_MODE_FMA_DEDICATED
By default, newly created CDM's are assigned a dedicated
FMA descriptor. If no FMA descriptors are available during the
creation of a dedicated FMA CDM, the operation will fail. The
GNI_CDM_MODE_FMA_SHARED flag allows applications to
share FMA descriptors between (CDM's) within a protection
domain. This enables a user to allocate more CDM's than there
are FMA descriptors on a node. By default, FMA_SHARING
disables DLA forwarding.
52
S–2446–51
•
GNI_CDM_MODE_DLA_ENABLE_FORWARDING and
GNI_CDM_MODE_DLA_DISABLE_FORWARDING
Alter the FMA_SHARED behavior regarding DLA forwarding.
•
GNI_CDM_MODE_CACHED_AMO_ENABLED
Enable the use of cached AMO operations.
•
GNI_CDM_MODE_CQ_NIC_LOCAL_PLACEMENT
Allow a request for placement of the CQ in host memory closest
to the NIC, which increases small message injection rate for some
applications.
•
GNI_CDM_MODE_FLBTE_DISABLE
•
GNI_CDM_MODE_SMALL_WINDOW
By default, the entire FMA window is mapped to a process's
address space. This mode prevents mapping the entire FMA
window into a process's address space. Reduce size of
FMA window, reducing memory footprint and initialization
overhead. FMA throughput unaffected while using this
mode with FMA transactions under the size configured in
/sys/class/gni/kgni0/fma_sm_win_sz:
•
GNI_CDM_MODE_USE_PCI_IOMMU
(Aries) User space may specify this mode to be used for all
memory transactions. Setting this will always attempt to use the
root complex's address translation in the PCI bridge. If this can
not be enabled, but is requested, all memory registrations will
error.
•
GNI_CDM_MODE_DUAL_EVENTS
Must be used when local and global completion events are
needed for RDMA post operations. By default, users may post
transactions with either local or global completion notification,
not both. If receipt of both local and global events is requested
users must set this flag. Performing a post operation with local
and global events enabled without this flag set will yield an error
GNI_RC_INVALID_PARAM. In addition, during an EpBind
in default mode, transfer requests are allocated equal in size to
the number of events in the associated source CQ. When this
flag is set transfer requests are allocated 1 per 2 CQ event slots.
Therefore, a user is limited to posting half as many transactions
as CQ events when this flag is set. Exceeding this limit will yield
an error GNI_RC_ERROR_RESOURCE.
S–2446–51
53
•
GNI_CDM_MODE_FAST_DATAGRAM_POLL
Enable fast polling for
GNI_[Ep]PostData[Test|Probe]* functions. Using
this option may result in loss of intermediate state information
for datagram transactions.
•
GNI_CDM_MODE_DUAL_EVENTS
Allocate transfer requests 1 per 2 event slots. User is limited to
posting half as many transactions.
•
GNI_CDM_MODE_BTE_SINGLE_CHANNEL
Enable RDMA posts through one BTE channel, instead of
defaulting to using all available channels.
•
•
One of the following mutually exclusive flags:
–
GNI_CDM_MODE_ERR_NO_KILL
–
GNI_CDM_MODE_ERR_ALL_KILL
GNI_CDM_MODE_USE_PCI_IOMMU
Attempt to use root complex's address translation in the PCI
bridge. If this is requested, but cannot be enabled, all memory
registrations will error.
cdm_handle
Returns a pointer to a handle for the communication domain object.
The handle is used by other functions to specify a particular instance
of the communication domain.
GNI_RC_SUCCESS
The operation completed successfully.
GNI_RC_INVALID_PARAM
One of the input parameters was invalid.
GNI_RC_ERROR_NOMEM
Insufficient memory to complete the operation.
3.2.2 CdmDestroy
The GNI_CdmDestroy function destroys the instance of the communication
domain and removes associations between the calling process and the NIC devices
that were established by the corresponding GNI_CdmAttach function.
54
S–2446–51
3.2.2.1 Synopsis
gni_return_t GNI_CdmDestroy (
IN gni_cdm_handle_t cdm_handle)
3.2.2.2 Parameters
cdm_handle
The communication domain handle.
GNI_RC_SUCCESS
The caller specified an invalid communication domain handle.
3.2.3 CdmGetNicAddress
The CdmGetNicAddress function returns the PE address of the GNI device with
ID device_id and the ID of its most closely connected CPU.
3.2.3.1 Synopsis
gni_return_t GNI_CdmGetNicAddress (
IN uint32_t device_id,
OUT uint32_t *address,
OUT uint32_t *cpu_id )
3.2.3.2 Parameters
device_id
The device identifier. For example, the
NIC /dev/kgni1 has the device_id=
DEVICE_MINOR_NUMBER-GEMINI_BASE_MINOR_NUMBER=1.
address
PE address of the GNI device.
cpu_id
ID of the first CPU in the slot directly connected to the GNI device.
GNI_RC_SUCCESS
GNI_RC_NO_MATCH
The specified device_id does not exist.
S–2446–51
55
3.2.4 CdmAttach
The CdmAttach function associates the communication domain with a NIC and
provides a NIC handle to the upper layer protocol. A process cannot attach a single
communication domain instance to a NIC more than once, but it can attach multiple
communication domains to a single NIC.
If NTT is used, the local_address contains the virtual address of the PE, rather than
its physical address.
3.2.4.1 Synopsis
gni_return_t GNI_CdmAttach (
IN gni_cdm_handle_t cdm_handle,
OUT uint32_t *local_address,
OUT gni_nic_handle_t *nic_handle)
3.2.4.2 Parameters
cdm_handle
Communication domain handle.
device_id
Device identifier for the NIC to which the communication domain
attaches. The device id is the minor number for the device that is
assigned to the device by the system when the device is created. To
determine the device number, look in the /dev directory, which
contains a list of devices. For a NIC, the device is listed as kgniX,
where X is the device number. Setting to -1 will attach to nearest
NIC.
local_address
Returns a pointer to the PE address for the NIC that this function
has attached to the communication domain. If NTT is used, the
local_address contains the virtual address of the PE, rather than its
physical address.
nic_handle
56
Returns a pointer to a handle for the NIC. The handle is used by the
API to specify an instance of a NIC.
S–2446–51
GNI_RC_SUCCESS
The caller specified an invalid communication domain handle.
GNI_RC_NO_MATCH
GNI_RC_ERROR_RESOURCE
The operation failed due to insufficient resources. To resolve this,
verify that the FMA descriptors are available on the given NIC. The
most likely cause of this error is that too many CDM domains got
attached to the given NIC on that node.
GNI_RC_ERROR_NOMEM
GNI_RC_INVALID_STATE
The caller attempted to attach a communication domain instance to
the NIC device more than once. When device_id = -1, this return
value indicates that there are no more devices left for this CDM to
attach to.
GNI_RC_PERMISSION_ERROR
Insufficient permissions to perform the operation.
GNI_RC_NOT_DONE
The process was interrupted.
3.2.5 GetVersion
The GetVersion function returns the version number of the uGNI library.
3.2.5.1 Synopsis
gni_return_t GNI_GetVersion(
OUT uint32_t *version)
3.2.5.2 Parameters
version
S–2446–51
GNI version number.
57
GNI_RC_SUCCESS
Operation completed successfully.
The version is undefined.
3.2.6 GetVersionInformation
The GetVersionInformation function returns the version number of the uGNI
and kGNI libraries.
3.2.6.1 Synopsis
gni_return_t GNI_GetVersionInformation(
OUT gni_version_info_t *version_info)
3.2.6.2 Parameters
version_info
Points to structure containing version information. See
gni_version_info on page 181.
GNI_RC_SUCCESS
Invalid parameter.
3.2.7 ConfigureNTT
The Node Translation Table (NTT) works in conjunction with the FMA mechanism
to allow applications to employ logical network endpoints when addressing remote
nodes. This facilitates efficient user-level access to FMA, as well as simplifying
checkpoint/restart operations, etc. There are 8192 entries in the NTT for each NIC
on the Gemini network ASIC. Each entry contains 18 bits of data which is used to
convert an application virtual PE into a 16-bit Network Endpoint ID and a 2-bit core
ID (DstID). Bit 17 of the entry specifies bit 1 of the DstID field. The NTTConfig
register controls the granularity for NTT addressing.
The GNI_ConfigureNTT function sets up entries in the NTT associated with a
particular /dev/kgni device.
58
S–2446–51
If the table field of the input ntt_desc is set to NULL, the NTT entries starting from
ntt_base up to and including ntt_base+ntt_desc->group_size–1 are reset to 0.
If the ntt_base is -1 and ntt_desc->group_size is -1, and the table field of ntt_desc is
NULL, all entries of NTT allocations not currently in use will be reset to 0.
3.2.7.1 Synopsis
gni_return_t GNI_ConfigureNTT (
IN gni_ntt_descriptor_t *ntt_desc,
OUT uint32_t ntt_base )
3.2.7.2 Parameters
device_id
The device identifier, for example, for /dev/kgni1
it is device_id = DEVICE_MINOR_NUMBER –
GEMINI_BASE_MINOR_NUMBER = 1.
ntt_desc
NTT configuration descriptor. Descriptions are set using the
gni_ntt_descriptor structure which has the types found in
Table 1.
Table 1. gni_ntt_descriptor
ntt_base
Type
Option
Description
uint32_t
group_size
Size of the NTT group to
be configured.
uint8_t
granularity
NTT granularity.
uint32_t
table
Pointer to the array of new
NTT values.
uint8_t
flags
Configuration flags.
On return, set to the base NTT entry allocated by the driver.
GNI_RC_SUCCESS
The process has insufficient permission to set up NTT resources.
S–2446–51
59
A hardware resource limitation prevents NTT setup.
GNI_RC_ERROR_NOMEM
GNI_RC_NO_MATCH
3.2.8 ConfigureJob
The GNI_ConfigureJob function sets the configuration options for the job, which
include the device ID, the job ID, the protection tag, cookie, and limit values for the
job. The user (ALPS) can call this function multiple times for the same Cray network
application-specific integrated circuit (ASIC) interface. The driver looks up a triplet
(job_id+ptag+cookie) and then adds a new entry into the list it maintains for each
physical NIC, for every unique triplet. Each entry may have a non-unique job_id or
ptag or cookie. Using the same ptag with a different job_id is illegal and such calls
fail. This function must be called before GNI_CdmAttach for the CDM with the
same ptag+cookie. Calling GNI_ConfigureJob for the same triplet has no effect,
unless limits is non-NULL.
An application may also use this function to associate NTT resources with
a job. The NTT resources would have been previously allocated by a call to
GNI_ConfigureNTT. In this case, the application sets the ntt_base and ntt_size
fields in the limits input. If the application expects the driver to clean up the NTT
resources upon termination of the job, the application sets the ntt_ctrl field in the
limits input to GNI_JOB_CTRL_NTT_CLEANUP. The application should not
subsequently attempt to change ntt_base or ntt_size by calling ConfigureJob
with different NTT parameters.
3.2.8.1 Synopsis
gni_return_t GNI_ConfigureJob (
IN uint64_t job_id,
IN uint8_t ptag,
IN uint32_t cookie,
IN gni_job_limits_t *limits )
60
S–2446–51
3.2.8.2 Parameters
S–2446–51
device_id
has device_id = DEVICE_MINOR_NUMBER –
job_id
Job container identifier.
ptag
Protection tag to be used by all applications in the given job
container.
cookie
Unique identifier. Assigned to all applications within the job
container along with ptag.
limits
When this argument is non-NULL, the driver takes all the limit
values that are not set to GNI_JOB_INVALID_LIMIT and stores
them into the table indexed by the ptag. These limits are imposed on
all applications running within the given job container. If you set
different limits for the same ptag, the driver overwrites previously
set limits.
61
GNI_RC_SUCCESS
The process has insufficient permission to configure job or no NTT
entries exist for input ntt_base/ntt_size fields in the limits argument.
GNI_RC_NO_MATCH
The specified device_id does not exist or there are no NTT entries.
The caller attempted to use the same ptag with a different job_id or
a different cookie.
GNI_RC_ILLEGAL_OP
The application is attempting to resize the NTT resources.
A resource allocation error was encountered while trying to configure
the job resources.
GNI_RC_ERROR_NOMEM
3.2.9 ConfigureNTTandJob
This function combines the GNI_ConfigureNTT and GNI_ConfigureJob
functions; it sets up entries in the NTT associated with a particular /dev/kgni
device, and then sets up configuration options of the job in a single system
call. Setting ntt_desc to NULL will makes this function equivalent to
GNI_ConfigureJob.
This function cannot be used to clear the NTT table; GNI_ConfigureNTT should
be used.
This function or GNI_ConfigureJob must be called before GNI_CdmAttach
for the CDM with the same ptag+cookie.
Calling GNI_ConfigureNTTandJob for the same triplet has no effect, unless
limits is non-NULL.
62
S–2446–51
If the application expects the driver to clean up the NTT resources upon
termination of the job, the application sets the ntt_ctrl field in the limits input to
GNI_JOB_CTRL_NTT_CLEANUP. The application should not attempt to change
ntt_base or ntt_size by calling ConfigureNTTandJob subsequently with different
NTT parameters.
3.2.9.1 Synopsis
gni_return_t GNI_ConfigureNTTandJob (
IN uint64_t job_id,
IN uint8_t ptag,
IN uint32_t cookie,
IN gni_job_limits_t *limits,
IN gni_ntt_descriptor_t *ntt_desc,
OUT uint32_t ntt_base )
3.2.9.2 Parameters
S–2446–51
device_id
has device_id = DEVICE_MINOR_NUMBER –
job_id
Job container identifier.
ptag
Protection tag to be used by all applications in the given job
container.
cookie
Unique identifier. Assigned to all applications within the job
container along with ptag.
limits
The driver takes all the limit values that are not set to
GNI_JOB_INVALID_LIMIT and stores them into the table
indexed by the ptag. These limits are imposed on all applications
running within the given job container. If you set different limits for
the same ptag, the driver overwrites previously set limits.
ntt_desc
NTT configuration descriptor. Descriptions are set using the
gni_ntt_descriptor structure which has the types found in
Table 1.
ntt_base
On return, set to the base NTT entry allocated by the driver.
63
GNI_RC_SUCCESS
The process has insufficient permission to configure job or no NTT
entries exist for input ntt_base/ntt_size fields in the limits argument.
GNI_RC_NO_MATCH
The specified device_id does not exist or there are no NTT entries.
The caller attempted to use the same ptag with a different job_id or
a different cookie.
GNI_RC_ILLEGAL_OP
The application is attempting to resize the NTT resources.
A resource allocation error was encountered while trying to configure
the job resources.
GNI_RC_ERROR_NOMEM
3.3 Balanced Injection (BI) Configuration
While some network congestion patterns can be detected and addressed in an
automated way (HSS congestion protection, for example), certain congestion
patterns are more difficult to detect, namely those patterns demanding high local
bandwidth (link bandwidth) which may be impacting remote node communication
(injection bandwidth). Reducing compute node injection bandwidth may have the
effect of improving application performance for certain communication patterns.
The GNI_SetBIConfig function allows the application to reduce the injection
bandwidth. GNI_GetBIConfig allows the application to query a node's BI
configuration. GNI_SetBIConfig returns before a new BI configuration is
committed. If necessary, GNI_BISyncWait may be used to block until all prior BI
updates are committed to HSS (delays should be less than 1 second).
64
S–2446–51
Note that ALPS exposes balanced injection to the user on an application wide basis,
but application programmers may achieve more fine-grained balanced injection
using the functionality provided by the uGNI interfaces. For additional information
regarding ALP's interface to this capability, see Using Balanced Injection in Cray
Systems.
3.3.1 GetBIConfig
The GetBIConfig interface returns a pointer to a structure describing a
node's balanced injection configuration. For a synopsis of this structure, refer to
gni_bi_desc on page 163.
3.3.1.1 Synopsis
gni_return_t GNI_GetBIConfig(
OUT gni_bi_desc_t *desc)
3.3.1.2 Parameters
device_id
The device identifier of the GNI device to query, typically 0.
desc
A pointer to a structure describing the current balanced injection
configuration.
GNI_RC_SUCCESS
One of the input parameters is invalid.
3.3.2 SetBIConfig
The GNI_SetBIConfig interface configures a node's balanced injection settings.
GNI_SetBIConfig returns before a new BI configuration is committed.
3.3.2.1 Synopsis
gni_return_t GNI_SetBIConfig(
IN uint16_t bw,
IN uint16_t aot_bw,
IN uint16_t modes,)
S–2446–51
65
3.3.2.2 Parameters
device_id
bw
The updated apply-now injection bandwidth setting. Must be
set to an integer between 0 and 100. The bw value is mapped
to some number of outstanding request buffer (ORB) entries
between 1 and GNI_BI_INJECT_BW_ORB_MAX. 0 and 100
both mean "wide-open", or no BI. Requires that modes is set to
GNI_BI_FLAG_APPLY_NOW.
aot_bw
The updated apply-on-throttle injection bandwidth setting. This
bandwidth setting is effective after the first instance of congestion
protection until the node is reset after application termination. Must
be set to an integer between 0 and 100. The aot_bw value is mapped
to some number of outstanding request buffer (ORB) entries between
1 and GNI_BI_INJECT_BW_ORB_MAX. aot_bw must be less
than bw if modes is set to both GNI_BI_FLAG_APPLY_NOW and
GNI_BI_FLAG_APPLY_AFTER_THROTTLE. If bw is not set but
aot_bw is, aot_bw must be less than the currently instantiated BI
setting.
Requires that modes is set to
GNI_BI_FLAG_APPLY_AFTER_THROTTLE.
modes
•
GNI_BI_FLAG_APPLY_NOW
Update the apply-now bandwidth setting.
•
GNI_BI_FLAG_APPLY_AFTER_THROTTLE
Update the apply-on-throttle bandwidth setting.
•
GNI_BI_FLAG_USE_DEFAULT_SETTINGS
Apply HSS provided system default. All other fields and flags are
ignored when this is set.
•
GNI_BI_FLAG_VALUE_IS_NUM_ORB_ENTRIES
Internal use only.
66
S–2446–51
GNI_RC_SUCCESS
Insufficient permissions to perform the operation.
3.3.3 BISyncWait
Blocks until the most recent balanced injection configuration update is committed.
3.3.3.1 Synopsis
gni_return_t GNI_BISyncWait(
OUT uint32_t timeout)
3.3.3.2 Parameters
device_id
desc
A pointer to the current balanced injection configuration.
GNI_RC_SUCCESS
GNI_RC_TIMEOUT
The timeout expired.
S–2446–51
67
3.4 Completion Queue Management
Once a communication domain has been established, a process running on node in
the domain (or job) may create completion queues which will store events associated
with transactions through the local nic_handle. Transactions involve an initiator and
a recipient; likewise, transactions may generate events related to the initiation or to
the receipt of a transaction request. A local completion queue (CQ) must exist for the
initiation of a transaction via the local nic_handle. A peer running on a different
node in the same communication domain may create a destination CQ to receive
completion events from the initiating process.
Other completion queues (associated with the same nic_handle as one used to initiate
events) may be created to receive destination events (receive CQs) initiated from
other processes in the communication domain. Receive CQs notify the local process
of completion of operations initiated on remote endpoints which target registered
memory on the local node. Memory registration turns a CQ into a receive CQ (also
called a destination CQ) by associating it with registered memory and creating a
memory domain handle (MDH) for that CQ.
3.4.1 CqCreate
The CqCreate function creates a new completion queue. The caller must specify
the minimum number of completion entries that the queue must contain in the
entry_count parameter. To avoid dropped completion notifications, you should set up
your application to verify that the number of operations posted on endpoints attached
to a cq_handle does not exceed the completion queue entry_count capacity at
any time.
The event_handler function, if specified, is called if (and only if)
CqGetEvent or CqWaitEvent return with either GNI_RC_SUCCESS
or GNI_RC_TRANSACTION_ERROR. The handler is invoked at some time
between the time that the CQ entry arrives in the CQ, and the successful return of
GNI_CqGetEvent or GNI_CqWaitEvent.
If an error occurs while using the GNI_CQ_PHYS_PAGES mode, the request
will be retried without the GNI_CQ_PHYS_PAGES mode set. This retry is done
automatically for the user. If an error occurs after the GNI_CQ_PHYS_PAGES mode
has been cleared, then this error will be reported back to the user.
The user must call GNI_CqGetEvent or GNI_CqWaitEvent for each event
deposited into the CQ, regardless of whether an event_handler is used.
Completion queues may be used for the receipt of locally generated events, such
as those arising from GNI_Post style transactions or may be used for the receipt of
remote events, but not both.
68
S–2446–51
3.4.1.1 Synopsis
gni_return_t GNI_CqCreate (
IN gni_nic_handle_t nic_handle,
IN uint32_t entry_count,
IN uint32_t delay_ count,
IN uint32_t mode,
IN void (*event_handler)(gni_cq_entry_t *, void *),
IN void *context,
OUT gni_cq_handle_t *cq_handle)
3.4.1.2 Parameters
nic_handle
The handle of the associated NIC.
entry_count
The number of completion entries that this completion queue holds.
delay_count
The number of events the NIC allows before generating an interrupt.
Setting this parameter to zero results in interrupt delivery with every
event. When using this parameter, the mode parameter must be set
to GNI_CQ_BLOCKING.
mode
The mode of operation for the new completion queue. The following
modes are used by this parameter:
•
•
•
•
GNI_CQ_BLOCKING
Create CQ with blocking enabled.
GNI_CQ_NOBLOCK
Create CQ with blocking disabled.
GNI_CQ_EMULATED
Internal use only.
GNI_CQ_PHYS_PAGES
Use physical pages instead of memory mapped space. By default,
memory mapped space is used. The use of physical pages
mode can be combined with either GNI_CQ_BLOCKING or
GNI_CQ_NOBLOCK modes.
event_handler
User-supplied callback function to be run for each CQ entry received
in the CQ. The handler is supplied with two arguments: a pointer to
the CQ entry, and a pointer to the context provided at CQ creation.
S–2446–51
context
User-supplied pointer that is passed to the handler callback function.
cq_handle
Returns a pointer to the handle of the newly created completion
queue.
69
GNI_RC_SUCCESS
A new completion queue was successfully created.
One or more of the parameters was invalid.
The completion queue could not be created due to insufficient
resources.
GNI_RC_ERROR_NOMEM
3.4.2 CqDestroy
The CqDestroy function destroys a specified completion queue. If any endpoints
are associated with the completion queue, the completion queue is not destroyed
and an error is returned.
3.4.2.1 Synopsis
gni_return_t GNI_CqDestroy (
IN gni_cq_handle_t cq_handle)
3.4.2.2 Parameters
cq_handle
The handle for the completion queue to be destroyed.
GNI_RC_SUCCESS
The completion queue was successfully destroyed.
The cq_handle was invalid.
The completion queue could not be destroyed because one or more
endpoint instances are still associated with it. Use EpDestroy to
destroy the endpoint instance, then try calling this function again.
70
S–2446–51
3.5 Logical Endpoint
Once a communication domain has been created and completion queues have been
created within it, logical endpoints may be created using EpCreate. Each endpoint
associates the local nic_handle and its associated cq_handle with a local endpoint
handle, ep_handle. Then the process calls EpBind to associate local ep_handle
to the physical address of a remote NIC; the application creates logical endpoints
locally, then binds them to remote NIC addresses. Each endpoint and its bound
address constitute an endpoint pair.
A process must EpUnbind, and EpDestroy for each endpoint pair when they
are no longer used.
Note that it is legal to unbind an endpoint and then bind it again to another remote
NIC without destroying it and creating a new one. EpBind and EpUnbind are
very inexpensive operations and user may want to consider maintaining a pool of
endpoints, which can be bound and unbound dynamically.
3.5.1 EpCreate
The EpCreate function creates an instance of a logical endpoint. A new instance
is always created in a non-bound state. A non-bound endpoint is able to exchange
posted data with any bound remote endpoint within the same communication domain.
While an endpoint is in a non-bound state, it cannot be used to post RDMA or FMA
transactions or to send short messages.
3.5.1.1 Synopsis
gni_return_t GNI_EpCreate (
IN gni_cq_handle_t src_cq_handle,
OUT gni_ep_handle_t *ep_handle)
3.5.1.2 Parameters
nic_handle
Handle of the associated NIC.
src_cq_handle
Handle of the completion queue that is used by default to deliver
events related to the transactions initiated by the local node.
ep_handle
S–2446–51
Returns a pointer to the handle of the newly created endpoint
instance.
71
GNI_RC_SUCCESS
GNI_RC_ERROR_NOMEM
3.5.2 EpSetEventData
The EpSetEventData function enables the user to define the values that the
EpBind function uses to generate CQ events. By default, EpBind uses the
Communication Domain's inst_id as the event data for generating global and remote
CQ events and the Endpoint's remote_id for generating local CQ events.
On Aries systems, inst_id is 24 bits. On Gemini systems, inst_id is 32 bits. On both
Aries and Gemini systems, remote_id is 32 bits. For information on extracting this
data, see Macros which Extract User Data From CQ Entry on page 166.
3.5.2.1 Synopsis
gni_return_t GNI_EpSetEventData (
IN gni_ep_handle_t ep_handle,
IN uint32_t local_event,
IN uint32_t remote_event)
3.5.2.2 Parameters
ep_handle
The handle of the endpoint instance to define event data.
local_event
The value to use when generating local CQ events.
remote_event
The value to use when generating global and remote CQ events.
GNI_RC_SUCCESS
An invalid endpoint handle was specified.
72
S–2446–51
3.5.3 EpBind
The EpBind function binds a logical endpoint to a specific remote address and
remote instance within the communication domain. Once bound, the endpoint can be
used to post RDMA and FMA transactions.
3.5.3.1 Synopsis
gni_return_t GNI_EpBind (
IN uint32_t remote_addr,
IN uint32_t remote_id)
3.5.3.2 Parameters
ep_handle
Handle of the endpoint instance to be bound.
remote_addr
Physical address of the NIC at the remote peer or NTT index, when
NTT is enabled for the given communication domain.
remote_id
User-specified ID of the remote instance in the job or the unique
identifier of the remote instance within the upper layer protocol
domain.
GNI_RC_SUCCESS
Failed due to insufficient resources.
GNI_RC_ERROR_NOMEM
3.5.4 EpUnbind
The EpUnbind function unbinds a logical endpoint from the specific address and
remote instance, and releases any internal short message resources, if applicable.
While it is in a non-bound state, an endpoint cannot be used to post RDMA, FMA
transactions, or send short messages.
S–2446–51
73
3.5.4.1 Synopsis
gni_return_t GNI_EpUnbind (
IN gni_ep_handle_t ep_handle)
3.5.4.2 Parameters
ep_handle
The handle of the endpoint instance to be unbound.
GNI_RC_SUCCESS
GNI_RC_NOT_DONE
The endpoint still has outstanding transaction requests or pending
datagrams and cannot be unbound at this time. Retry unbinding later.
3.5.5 EpDestroy
The EpDestroy function destroys an endpoint, cancels any outstanding requests,
and releases short messaging resources.
3.5.5.1 Synopsis
gni_return_t GNI_EpDestroy (
3.5.5.2 Parameters
ep_handle
The handle of the endpoint instance to be destroyed.
GNI_RC_SUCCESS
74
S–2446–51
3.5.6 EpPostData
The EpPostData function posts a datagram to be exchanged from a bound
endpoint to a remote, bound or unbound endpoint in the communication domain. The
maximum size of a datagram is 128 bytes.
3.5.6.1 Synopsis
gni_return_t GNI_EpPostData (
IN void *in_data,
IN uint16_t data_len,
IN void *out_buf,
IN uint16_t buf_size)
3.5.6.2 Parameters
ep_handle
Handle of the local endpoint.
in_data
Pointer to the data to send.
data_len
Size of the data to send, in bytes.
out_buf
Pointer to the buffer that receives the incoming datagram.
buf_size
Size of the buffer for the incoming datagram, in bytes.
GNI_RC_SUCCESS
The posted datagram is queued.
The specified endpoint handle is invalid.
The system allows only a single outstanding datagram transaction for
each endpoint. There is already a pending datagram for the specified
endpoint
GNI_RC_ERROR_NOMEM
Insufficient memory to complete transaction.
GNI_RC_SIZE_ERROR
The size of the datagram is too large.
S–2446–51
75
3.5.7 EpPostDataWId
The EpPostDataWId function posts a datagram with a user-specified datagram_id
to be exchanged with a remote endpoint in the communication domain. If the local
endpoint is unbound, a datagram can be exchanged with any bound endpoint within
the communication domain.
It is required that datagrams posted on unbound endpoints be associated with a
datagram_id.
3.5.7.1 Synopsis
gni_return_t GNI_EpPostDataWId (
IN void *in_data,
IN uint16_t data_len,
IN void *out_buf,
IN uint16_t buf_size,
IN uint64_t datagram_id)
3.5.7.2 Parameters
ep_handle
in_data
Pointer to the data to send.
data_len
Size of the data to send.
out_buf
Pointer to the buffer that receives the incoming datagram.
buf_size
Size of the buffer for the incoming datagram.
datagram_id
Id associated with the datagram.
76
S–2446–51
GNI_RC_SUCCESS
The posted datagram is queued.
The specified endpoint handle is invalid, or an invalid value for the
datagram_id was specified.
The system allows only a single outstanding datagram transaction
for each endpoint.
GNI_RC_ERROR_NOMEM
Insufficient memory to complete transaction.
GNI_RC_SIZE_ERROR
The size of the datagram is too large.
3.5.8 EpPostDataTest
The EpPostDataTest function returns the state of the EpPostData transaction.
3.5.8.1 Synopsis
gni_return_t GNI_EpPostDataTest (
OUT gni_ep_post_state_t *post_state,
OUT uint32_t *remote_address,
OUT uint32_t *remote_id)
3.5.8.2 Parameters
ep_handle
post_state
Returns a pointer to the state of the transaction. The following states
are used for this parameter:
•
•
•
S–2446–51
GNI_POST_PENDING
GNI_POST_COMPLETED
GNI_POST_ERROR
77
•
•
•
GNI_POST_TIMEOUT
GNI_POST_TERMINATED
GNI_POST_REMOTE_DATA
remote_address
Returns a pointer to the physical address of the being used by the
remote peer. The address is valid only if the post_state returns
GNI_POST_COMPLETE.
remote_id
Returns a pointer to the user specific ID of the remote instance
in the job. The ID is valid only if the post_state returns
GNI_POST_COMPLETE.
GNI_RC_SUCCESS
GNI_RC_NO_MATCH
No matching datagram was found.
GNI_RC_SIZE_ERROR
The size of the output buffer is too small for the received datagram.
GNI_RC_ERROR_NOMEM
3.5.9 EpPostDataTestById
The EpPostDataTestById function returns the state of the EpPostData
transaction with the specified datagram_id.
3.5.9.1 Synopsis
gni_return_t GNI_EpPostDataTestById (
IN uint64_t datagram_id,
OUT gni_ep_post_state_t *post_state,
78
S–2446–51
3.5.9.2 Parameters
ep_handle
Handle of the local endpoint. Must be the same as that used when
posting the datagram using EpPostDataWId.
datagram_id
Id of the datagram to test for.
post_state
•
•
•
•
•
•
GNI_POST_PENDING
GNI_POST_COMPLETED
GNI_POST_ERROR
GNI_POST_TIMEOUT
GNI_POST_TERMINATED
remote_address
Returns a pointer to the physical address of the NIC being used by
the remote peer. The address is only valid if the post_state returns
GNI_POST_COMPLETE.
remote_id
in the job. The ID is only valid if the post_state returns
GNI_POST_COMPLETE.
GNI_RC_SUCCESS
GNI_RC_NO_MATCH
GNI_RC_SIZE_ERROR
GNI_RC_ERROR_NOMEM
S–2446–51
79
3.5.10 EpPostDataWait
The EpPostDataWait function is used to determine the result of a previously
posted EpPostData call on the specified endpoint, blocking the calling thread until
the completion of the posted transaction or until the specified timeout expires.
3.5.10.1 Synopsis
gni_return_t GNI_EpPostDataWait(
IN uint32_t timeout,
OUT gni_post_state_t *post_state,
3.5.10.2 Parameters
ep_handle
timeout
The length of time (in milliseconds) that this function waits for
the transaction to complete. If a timeout is not needed, set this
parameter to -1. If the timeout is set to zero, the system returns the
GNI_RC_INVALID_PARAM error.
post_state
State of the transaction.
remote_address
Physical address of the NIC at the remote peer. Valid only if
post_state returned GNI_POST_COMPLETE.
remote_id
80
User specific ID of the remote instance in the job (user).
Unique address of the remote instance within the upper layer
protocol domain (kernel). Valid only if post_state returned
GNI_POST_COMPLETE.
S–2446–51
GNI_RC_SUCCESS
The transaction completed successfully.
The specified endpoint handle is invalid or timeout was set to zero.
GNI_RC_TIMEOUT
The timeout expired before a successful completion of the
transaction.
GNI_RC_SIZE_ERROR
Output buffer is too small for the size of the received datagram.
GNI_RC_NO_MATCH
GNI_RC_ERROR_NOMEM
3.5.11 EpPostDataWaitById
The EpPostDataWaitById function determines the result of a previously posted
EpPostDataWId call on the specified endpoint, blocking the calling thread until
the transaction involving datagram_id has completed, or until the specified timeout
expires.
3.5.11.1 Synopsis
gni_return_t GNI_EpPostDataWaitById (
IN uint64_t datagram_id,
OUT gni_post_state_t *post_state,
S–2446–51
81
3.5.11.2 Parameters
ep_handle
datagram_id
The datagram ID of the first datagram found.
timeout
The length of time (in milliseconds) that this function waits for
the transaction to complete. If a timeout is not needed, set this
parameter to -1. If the timeout is set to zero, the system returns the
GNI_RC_INVALID_PARAM error.
post_state
•
•
•
•
•
•
GNI_POST_PENDING
GNI_POST_COMPLETED
GNI_POST_ERROR
GNI_POST_TIMEOUT
GNI_POST_TERMINATED
remote_address
Returns a pointer to the physical address of the NIC being used by
the remote peer. The address is only valid if the post_state returns
GNI_POST_COMPLETE.
remote_id
82
in the job. The ID is only valid if the post_state returns
GNI_POST_COMPLETE.
S–2446–51
GNI_RC_SUCCESS
The transaction completed successfully.
An invalid endpoint handle was specified or timeout was set to zero,
or invalid datagram id was specified.
GNI_RC_TIMEOUT
The timeout expired before a successful completion of the
transaction.
GNI_RC_SIZE_ERROR
GNI_RC_NO_MATCH
No datagram was found.
GNI_RC_ERROR_NOMEM
3.5.12 EpPostDataCancel
The EpPostDataCancel function cancels an outstanding post data transaction.
3.5.12.1 Synopsis
gni_return_t GNI_EpPostDataCancel (
3.5.12.2 Parameters
ep_handle
GNI_RC_SUCCESS
The transaction cancellation was successful.
The ep_handle parameter is invalid.
GNI_RC_NO_MATCH
No active post data transaction on the ep_handle.
S–2446–51
83
3.5.13 EpPostDataCancelById
The EpPostDataCancelById function cancels an outstanding post data
transaction with the specified datagram Id.
3.5.13.1 Synopsis
gni_return_t GNI_EpPostDataCancelById (
IN uint64_t datagram_id)
3.5.13.2 Parameters
ep_handle
datagram_id
Id of the datagram to cancel.
GNI_RC_SUCCESS
The transaction cancellation was successful.
One of the input parameters are invalid.
GNI_RC_NO_MATCH
No active post data transaction with the specified Id on the
ep_handle.
3.5.14 PostDataProbe
The PostDataProbe function returns the remote ID and remote address of
the first datagram found in completed, timed out, or canceled state for the CDM
associated with the input NIC handle. This function must be used in conjunction with
GNI_EpPostDataTestById or GNI_EpPostDataWaitById to obtain data
exchanged in the datagram transaction.
3.5.14.1 Synopsis
gni_return_t GNI_PostDataProbe (
84
S–2446–51
3.5.14.2 Parameters
nic_handle
Handle of the NIC associated with the CDM for which the datagram
status is being probed.
remote_address
Physical address of the NIC at the remote peer with a datagram in the
completed, timed-out or cancelled state. Valid only if the return value
is GNI_RC_SUCCESS.
remote_id
User specific ID of the remote instance in the upper layer protocol
domain with a datagram in the completed, timed-out or cancelled
state. Valid only if the return value is GNI_RC_SUCCESS.
GNI_RC_SUCCESS
A datagram in the completed, timed-out or cancelled state was found.
An invalid nic_handle, remote_addr or remote_id was specified.
GNI_RC_NO_MATCH
No datagram in the completed, timed-out or cancelled state was
found.
3.5.15 PostDataProbeById
The PostDataProbeById function returns the datagram_id of the first datagram
found in completed, timed out, or canceled state for the CDM associated with the
input NIC handle. This function should be used for probing for completion of
datagrams that were previously posted using the GNI_EpPostDataWId function.
This function must be used in conjunction with GNI_EpPostDataTestById
or GNI_EpPostDataWaitById to obtain the data exchanged in the datagram
transaction.
3.5.15.1 Synopsis
gni_return_t GNI_PostDataProbeById (
OUT uint64_t *datagram_id)
S–2446–51
85
3.5.15.2 Parameters
nic_handle
Handle of the NIC that is associated with the CDM for which the
datagram status is being probed.
datagram_id
The datagram ID of the first datagram found in the completed,
timed-out or cancelled state. Valid only if the return value is
GNI_RC_SUCCESS.
GNI_RC_SUCCESS
A datagram in the completed, timed-out or cancelled state was found.
An invalid nic_handle, datagram_id was specified.
GNI_RC_NO_MATCH
No datagram with the specified ID found in the completed, timed-out
or cancelled state.
3.5.16 PostdataProbeWaitById
The PostdataProbeWaitById function returns the datagram ID of the first
posted datagram ID found in completed, timed out, or canceled state for the CDM
associated with the input nic_handle. This function must be used in conjunction with
gni_ep_postdata_test_by_id or gni_ep_postdata_wait_by_id to
obtain data exchanged in the datagram transaction.
3.5.16.1 Synopsis
gni_return_t GNI_PostdataProbeWaitById (
OUT uint64_t *datagram_id )
86
S–2446–51
3.5.16.2 Parameters
nic_handle
Handle of the NIC that is associated with the CDM for which the
datagram status is being probed.
timeout
The count (in milliseconds) that this function waits, for transaction to
complete. Set to -1 if no timeout is desired.
datagram_id
The first datagram ID found in the completed, timed-out or cancelled
state. Valid only if the return value is GNI_RC_SUCCESS.
GNI_RC_SUCCESS
A datagram with the specified id was found in the completed,
timed-out or cancelled state.
An invalid nic_handle, or timeout was specified.
GNI_RC_TIMEOUT
No datagram in the completed, timed-out or cancelled state was
found before the timeout expired.
3.6 Virtual Collective Engine (VCE) Channel
On Aries systems, GNI supports the offload of collective operations (gather, scatter)
to a specialized physical component of the Aries NIC called a collective engine (CE).
Each participating process in a collective operation must associate its local endpoints
with a virtual collective channel (VCE). Each participating process is configured with
information about parent and children VCEs.
uGNI routines CeCreate on page 88, CeGetId on page 89, EpSetCeAttr on
page 90, and CeConfigure on page 91 manage the association between a logical
endpoint and the virtual collective engine (VCE) channel.
S–2446–51
87
Collective operations may be launched on a set of processes organized into a
hierarchical tree structure. A process may participate in this GNI CE tree as a leaf
node. In order for software to use the collective engine, all participating instances
must first allocate any VCE channels required to build a tree. Once acquired, a job's
CE resoures must be configured with the information about any edges stemming from
the local tree node. This information includes a PE address, PE core ID, and a CE
ID for the parent and child of each edge stemming from the local node. A job will
be responsible for communicating address information between its instances. Once
all participating VCEs are configured into a GNI CE tree, the transaction may be
initiated.
DMAPP also supports collective operations which use the collective engine hardware.
It is recommended that users utilize the DMAPP interface for collective operations
rather than the uGNI version of this capability.
3.6.1 CeCreate
The CeCreate function allocates a hardware VCE channel resource. Before the
VCE resource can be used, it must be associated with a local endpoint. CeCreate
returns ce_handle which is used by functions CeGetId, SetCeAttr, and CeConfigure to
create this association.
3.6.1.1 Synopsis
gni_return_t GNI_CeCreate (
OUT gni_ce_handle_t *ce_handle)
3.6.1.2 Parameters
88
nic_handle
Handle of the associated NIC.
ce_handle
Returns a pointer to the handle of the newly created VCE channel.
S–2446–51
GNI_RC_SUCCESS
A resource allocation error was encountered.
GNI_RC_ERROR_NOMEM
GNI_RC_ILLEGAL_OP
This operation is not supported on this NIC type.
3.6.2 CeGetId
The CeGetId function retrieves the ID of the VCE channel.
3.6.2.1 Synopsis
gni_return_t GNI_CeCreate (
IN gni_ce_handle_t ce_handle
OUT uint32_t *ce_id)
3.6.2.2 Parameters
ce_handle
The VCE channel to use
cd_id
The ID of the VCE channel.
GNI_RC_SUCCESS
GNI_RC_ILLEGAL_OP
S–2446–51
89
3.6.3 EpSetCeAttr
The EpSetCeAttr function stores CE tree attributes to an endpoint VCE channel.
A VCE channel is configured using a set of endpoints with CE attributes set. Each
endpoint used for VCE channel configuration represents a node directly connected
to the channel. Additionally, endpoints used to initiate CE operations (leaf nodes in
the collective tree) must have EP CE attributes set. Endpoints used for CE channel
configuration represent either a child PE, child VCE or parent VCE. Each of these
endpoint types is configured using a different set of EP CE attributes:
•
If end point represents a child PE, set:
ce_id = unused
child_id = index in [0,GNI_CE_MAX_CHILDREN] that the local VCE channel
uses to refer to this child.
child_type = GNI_CE_CHILD_PE
•
If end point represents a child VCE, set:
ce_id = child VCE .
child_id = index in [0,GNI_CE_MAX_CHILDREN] that the local VCE channel
uses to refer to this child.
child_type = GNI_CE_CHILD_VCE
•
If endpoint represents a parent VCE, set:
ce_id = CE ID of the parent VCE channel
child_id = index in [0,GNI_CE_MAX_CHILDREN] that the remote VCE
channel uses to refer to this child.
child_type = GNI_CE_CHILD_VCE
•
Endpoints used to initiate CE operations using GNI_PostFma() must also be
configured with CE attributes. Note that endpoints used for CE operations (either
configuration of a VCE channel or as a leaf endpoint must be bound using remote
address and instance ID information. These leaf endpoints are configured with:
ce_id = CE ID of the parent VCE channel
child_id = index in [0,GNI_CE_MAX_CHILDREN] that the remote VCE
channel uses to refer to this child.
child_type = GNI_CE_CHILD_PE
90
S–2446–51
3.6.3.1 Synopsis
gni_return_t GNI_EpSetCeAttr (
IN gni_ep_handle_t ep_handle
IN uint32_t ce_id,
IN uint32_t child_id
IN gni_ce_child_t child_type)
3.6.3.2 Parameters
ep_handle
The VCE channel to use
ce_id
The ID of the VCE channel.
GNI_RC_SUCCESS
GNI_RC_ILLEGAL_OP
3.6.4 CeConfigure
The CeConfigure function associates a VCE channel to a set of endpoints which
are participating in a collective operation.
3.6.4.1 Synopsis
gni_return_t GNI_CeConfigure (
IN gni_ce_handle_t ce_handle,
IN gni_ep_handle_t *child_eps
IN uint32_t num_child_eps,
IN gni_ep_handle_t parent_ep,
IN gni_cq_handle_t cq_handle
IN uint32_t modes
)
S–2446–51
91
3.6.4.2 Parameters
ce_handle
VCE channel to configure.
child_eps
Array of endpoints of child connections.
num_child_eps
Number of child connections.
parent_ep
EP representing VCE parent.
cq_handle
The CQ to associate with the VCE channel.
modes
VCE channel configuration modes.
GNI_RC_SUCCESS
GNI_RC_ILLEGAL_OP
3.6.5 CeCheckResult
The CeCheckResult function reads control information in the CE result structure
to determine the status of a CE operation.
3.6.5.1 Synopsis
gni_return_t GNI_CeCheckResult (
IN gni_ce_result_t *result,
IN uint32_t length
)
3.6.5.2 Parameters
92
result
Pointer to CE result structure used for the operation.
length
Unused.
S–2446–51
GNI_RC_SUCCESS
GNI_RC_NOT_DONE
Operation not completed.
GNI_RC_TRANSACTION_ERROR
Operation completed with error.
GNI_RC_ILLEGAL_OP
3.6.6 CeDestroy
The CeDestroy function frees a VCE channel, freeing associated resources.
3.6.6.1 Synopsis
gni_return_t GNI_CeDestroy (
IN gni_ce_result_t ce_handle)
3.6.6.2 Parameters
ce_handle
CE channel to free.
GNI_RC_SUCCESS
GNI_RC_ILLEGAL_OP
S–2446–51
93
3.7 Memory Registration
After a process allocates a memory region to be used as a source or destination buffer
for data transfers, it must register this memory with the NIC. Only memory registered
with the local NIC is accessible from a remote NIC. Memory registration supports
the address translation and data protection mechanism.
FMA and BTE transactions all require remote memory to be registered with that
memory's attached NIC to support the remote address translation and data protection
mechanism.
The need for local memory registration depends on the transaction. In the case of
FMA PUTs and non-fetching AMOs, the host processor writes local data buffers
directly into the local NIC, so the local NIC is not required to do memory translation;
memory translation only occurs at the destination by the remote NIC to access its
own memory.
FMA GETs, fetching AMOs, BTE GETs and BTE PUTs do require local memory
registration.
SMSG transactions are a special class of FMA PUT operations. Similarly, they
require remote buffer memory registration, but not local memory registration.
3.7.1 MemRegister
The MemRegister function allows a process to register a region of memory
with the NIC. It requires the specification of a handle for the currently open NIC,
the starting address of the local memory region to be registered, its length, a flag
identifying the region as read or read-write. A completion queue handle may be
optionally specified. A new memory handle is generated for each region of memory
that is registered by a process. The content of the memory region being registered
is not altered.
Incoming network packets referencing memory use a network logical address which
includes a memory domain handle (MDH) and a memory domain offset. The MDH
indexes into the table of memory domain descriptors (MDDs). Each MDD provides
the base address and bounds of the memory domain within a translation context. The
memory domain offset is added to the base specified in the MDD. In other words,
the memory domain offset is used as an offset into a local memory window defined
by the MDD. This allows the local node to place the memory associated with a
given MDD in any location in its local memory space using the associated address
translation mechanisms (IOMMU for Aries, GART and MRT for Gemini).
MDHs are partially specified by the user level software and cannot be trusted by the
NIC driver, so each MDD also contains a protection tag (PTag) which is assigned by
the operating system and cannot be modified by the user. The PTag in an incoming
memory reference is checked against a PTag stored in the referenced MDD, to verify
that the reference is permitted to use that MDD.
94
S–2446–51
3.7.1.1 Virtual Memory Domain Handles
The memory registration mechanism also supports a limited capability to use virtual
Memory Domain Handles (vMDH), which supports Distributed Memory (DM)
programming models.
For this discussion, a DM programming model is defined as a parallel job consisting
of multiple independent processes distributed across one or more nodes of a system.
The processes may be executing the same application or different ones.
vMDH is intended for registration of symmetric memory segments, which start at the
same virtual address and have an equal size. Unequal segments can be registered
with vMDH, but use of such segments is more complicated and its implementation is
left to a user.
At least one memory segment on each node is made remotely accessible by all of the
processes in the job. To implement this model, the Virtual Memory Domain Handle
Table (vMDHT) creates a relationship between the virtual MDH in the incoming
network request and the actual memory domain handle to use in looking up the MDD
associated with the incoming reference.
3.7.1.2 Synopsis
gni_return_t GNI_MemRegister (
IN uint64_t address,
IN uint64_t length,
IN gni_cq_handle_t dst_cq_handle,
IN uint32_t flags,
IN uint32_t vmdh_index,
INOUT gni_mem_handle_t *mem_handle)
S–2446–51
95
3.7.1.3 Parameters
nic_handle
Handle of a currently open NIC.
address
Starting address of the memory region to be registered.
length
Length of the memory region to be registered, in bytes. A length
parameter of zero will result in a GNI_RC_INVALID_PARAM error.
dst_cq_handle
If this value is not NULL, it specifies the completion queue to receive
events related to the transactions initiated by the remote node into
this memory region.
flags
Attributes of the memory region. A combination of the following
flags are used for this parameter:
GNI_MEM_READWRITE
The read/write attribute is associated with the
memory region.
GNI_MEM_READ_ONLY
The read only attribute is associated with the
memory region.
GNI_MEM_USE_VMDH
Directive to use virtual MDH while
registering this memory region. If the
GNI_MEM_USE_VMDH flag is set, this function
will fail if GNI_SetMddResources has not
been called to specify the size of the MDD block
to be used. If the GNI_MEM_USE_VMDH
flag is set, this function will fail with
GNI_RC_ERROR_RESOURCE return code if
the vMDH entry specified by vmdh_index is already
in use.
GNI_MEM_USE_GART
Directive to use GART while registering the memory
region.
GNI_MEM_PHYS_CONT
Directive to not use GART or MRT while registering
the memory region. Memory is physically
contiguous.
96
S–2446–51
GNI_MEM_PHYS_SEGMENTS
Valid for gni_mem_register segments.
Segments are 4KB each, described by physical
addresses.
GNI_MEM_STRICT_PI_ORDERING
Instructs the NIC to enforce strict Processor Interface
(PI) ordering for the memory region.
On an Aries based system, this flag instructs the
NIC to ignore the relaxed ordering routing mode on
received network requests and responses. This flag
instructs the NIC to disable relaxed ordering rules on
its processor interface for Write, Read, and Response
transactions.
On Gemini based systems, this flag disables the
Non-Posted Pass Posted Writes rule.
GNI_MEM_PI_FLUSH
Instructs the NIC to issue a HT FLUSH command
prior to sending network responses for the memory
region.
GNI_MEM_MDD_CLONE
Instructs the NIC to clone a MDD.
GNI_MEM_RELAXED_PI_ORDERING
Instructs the NIC to allow relaxed PI ordering.
On Aries systems, this flag enables reordering of
requests not originated in the network.
On Gemini systems, enables reordering of
Non-Posted and Posted write requests into the
processor by enabling rules for both Non-posted
pass posted writes and Posted pass posted writes.
Non-posted pass posted writes are enabled by
default.
Use this attribute with caution as it may affect
transaction ordering. It is overridden by
GNI_MEM_STRICT_PI_ORDERING.
S–2446–51
97
GNI_MEM_RESERVE_REGION
Only reserve the PTE range for this block of
memory.
GNI_MEM_UPDATE_REGION
Update the PTE range for the provided block of
memory. The first call with this flag will make the
MDH live. The application may receive page faults
if it does not call update region before sending to
an address. This will only fill in new pages, and
compare old pages to make sure there are not any
changes.
GNI_MEM_CUDA
Indicates the memory region is on the GPU.
GNI_EXMEM_FLAGS
External memory, or resident memory in other PCI
devices. Helper macro, as the different types of
external memory have bits assigned to them via the
above memory flags.
GNI_MEM_IS_EXTERNAL
External memory, or resident memory in other PCI
devices. Helper macro, as the different types of
external memory have bits assigned to them via the
above memory flags.
98
vmdh_index
Specifies the index within the preallocated memory domain
descriptor block that must be used for the registration. For example,
when this parameter is set to 0, it uses the first entry of the memory
domain descriptor block. If set to -1, it relies on the GNI library to
allocate the next available entry from the memory domain descriptor
block.
mem_handle
The new memory handle for the region.
S–2446–51
GNI_RC_SUCCESS
The memory region was successfully registered.
The registration operation failed due to insufficient resources.
GNI_RC_ERROR_NOMEM
The user's buffer read/write permissions conflict with the flags
argument.
3.7.2 MemRegisterSegments
The MemRegisterSegments function enables a process to register a memory
region that is composed of multiple memory segments with the NIC.
Multiple segment registration should be reserved for special cases. Single segment
memory registration is the preferred method for memory registration. To register a
single segment, use GNI_MemRegister.
The user may specify an arbitrary size region of memory, with arbitrary alignment,
but the actual area of memory registered will be registered on MRT block granularity
(or physical page granularity for non-MRT registrations).
If an application registers multiple segments, it must use an offset within the
registered memory region instead of a virtual address in all future transactions
where registered region is aligned to MRT block size (or page size for non-MRT
registrations). This is because a single memory domain is used for the registration
of multiple segments and transactions must access memory for these segments as if
it was contiguous.
A new memory handle is generated for each region of memory that is registered by
a process. The contents of the memory region being registered are not altered. The
memory must be previously allocated by an application. If failure is returned, the
contents of mem_handle are untouched.
S–2446–51
99
3.7.2.1 Synopsis
gni_return_t GNI_MemRegisterSegments (
IN gni_mem_segment_t *mem_segments,
IN uint32_t segments_cnt,
IN gni_cq_handle_t dst_cq_handle,
IN uint32_t flags,
INOUT gni_mem_handle_t *mem_handle)
3.7.2.2 Parameters
nic_handle
mem_segments
List of segments to register. Each element of the list consists of the
starting address of the memory region and the length, in bytes. The
list elements are set using the gni_mem_segment structure.
segment_cnt
Number of segments in the mem_segments list.
dst_cq_handle
If this value is not NULL, it specifies the completion queue to receive
events related to the transactions initiated by the remote node into
this memory region.
flags
Attributes of the memory region. A combination of the following
flags are used for this parameter:
GNI_MEM_READWRITE
The read/write attribute is associated with the
memory region.
GNI_MEM_READ_ONLY
The read only attribute is associated with the
memory region.
GNI_MEM_USE_VMDH
Directive to use virtual MDH while
registering this memory region. If set, and
100
S–2446–51
GNI_SetMddResources has not been called
to specify the size of the MDD block to be used,
or if the vMDH entry specified by vmdh_index
is already in use, this function will return the
GNI_RC_ERROR_RESOURCE error code.
GNI_MEM_USE_GART
Directive to use GART while registering the memory
region.
GNI_MEM_PHYS_CONT
Do not use GART or MRT as memory is physically
contiguous.
GNI_MEM_PHYS_SEGMENTS
Segments are 4KB each, described by physical
addresses.
GNI_MEM_RELAXED_PI_ORDERING
Instructs the NIC to allow relaxed HT ordering
for non-posted and posted write requests into the
processor by enabling rules for both Non-posted
pass posted writes and Posted pass posted writes.
Non-posted pass posted writes are enabled by
default.
GNI_MEM_STRICT_PI_ORDERING
Instructs the NIC to enforce strict Processor Interface
(PI) ordering for the memory region.
On an Aries based system, this flag instructs the
NIC to ignore the relaxed ordering routing mode on
received network requests and responses. This flag
instructs the NIC to disable relaxed ordering rules on
its processor interface for Write, Read, and Response
transactions.
S–2446–51
101
On Gemini based systems, this flag disables the
Non-Posted Pass Posted Writes rule.
GNI_MEM_PI_FLUSH
Instructs the NIC to issue a HT FLUSH command
prior to sending network responses for the memory
region.
GNI_MEM_CUDA
Memory is GPU-resident.
vmdh_index
Specifies the index within the preallocated memory domain
descriptor block that must be used for the registration. For example,
when this parameter is set to 0, it uses the first entry of the memory
domain descriptor block. If set to -1, it relies on the GNI library to
allocate the next available entry from the memory domain descriptor
block.
mem_handle
The new memory handle for the region.
GNI_RC_SUCCESS
The memory region was successfully registered.
One on the parameters was invalid.
The registration operation failed due to insufficient resources.
GNI_RC_ERROR_NOMEM
The user's buffer read/write permissions conflict with the flags
argument.
3.7.3 SetMddResources
The SetMddResources function specifies the size of a contiguous block of MDD
entries that can be used for future memory registrations.
102
S–2446–51
3.7.3.1 Synopsis
gni_return_t GNI_SetMddResources (
IN uint32_t num_entries
3.7.3.2 Parameters
nic_handle
The handle for the NIC.
num_entries
Number of MDD entries in the block.
GNI_RC_SUCCESS
The block size was successfully specified.
GNI_RC_ERROR_NOMEM
3.7.4 MemDeregister
The MemDeregister function deregisters memory that was previously registered
and unlocks the associated pages from physical memory. An application may decide
to deregister memory upon transaction completion or keep memory for future
transactions. The contents and attributes of the deregistered region of memory are
not altered in any way.
3.7.4.1 Synopsis
gni_return_t GNI_MemDeregister (
IN gni_mem_handle_t *mem_handle
3.7.4.2 Parameters
S–2446–51
nic_handle
The handle for the NIC that owns the memory region being
deregistered.
mem_handle
Memory handle for the region; obtained from a previous call to
MemRegister.
103
GNI_RC_SUCCESS
The memory region was successfully deregistered.
3.7.5 MemHndlQueryAttr
Query for memory handle attributes while testing the memory handle for correctness.
Only one attribute may be tested at a time.
3.7.5.1 Synopsis
gni_return_t GNI_MemHndlQueryAttr (
IN gni_mem_handle_t *mem_handle,
IN gni_mem_handle_attr_t attr
OUT int *yesno);
3.7.5.2 Parameters
mem_handle
Memory handle for the region, obtained from a previous call to
MemRegister.
attr
Attribute of mem_handle to test for. See gni_mem_handle_attr
on page 157.
yesno
Returns 1 if mem_handle is described by the attribute, otherwise 0.
GNI_RC_SUCCESS
The memory region was successfully queried.
3.7.6 RebuildMemHndl
Given one src_mem_hndl, this function builds a new one that contains the same
address and length but with a different VMDH. The memory handle exchange does
not need to occur when an instance knows the remote memory layout.
104
S–2446–51
3.7.6.1 Synopsis
gni_return_t GNI_RebuildMemHndl (
IN gni_mem_handle_t *src_mem_handle,
OUT gni_mem_handle_t *dst_mem_handle
);
3.7.6.2 Parameters
src_mem_handle
Memory handle for a registered region.
vmdh_index
New VMDH index.
dst_mem_handle
New memory handle for the region on a different node.
GNI_RC_SUCCESS
The memory region was successfully queried.
The src_mem_handle has never been updated.
3.7.7 MemQueryHndls
The MemQueryHndls function will query the memory registered for the specified
NIC handle or NIC file descriptor and return the memory handle, address and length
of the registered memory. Debuggers may use this API to obtain a list of currently
registered memory regions.
3.7.7.1 Synopsis
GNI_MemQueryHndls(
IN gni_nic_handle_t nic_hndl,
IN int file_desc,
INOUT gni_mem_handle_t *mem_hndl,
OUT uint64_t *address,
OUT uint64_t *length
);
S–2446–51
105
3.7.7.2 Parameters
nic_handle
file_desc
The file descriptor for a currently open NIC.
mem_hndl
On input, this is the current memory handle to find. If the current
memory handle's qword1 and qword2 are both zero, then return
the first memory handle found for the supplied NIC. If the current
memory handle is not NULL, then return the next memory handle
found after the current memory handle. On output, if the return status
is GNI_RC_SUCCESS, this will contain the next available memory
handle. If the return status is not GNI_RC_SUCCESS, then it will
contain the input memory handle.
address
The address of the first physical page that contains the registered
memory.
length
The length in bytes for all of the physical pages that contain the
registered memory.
GNI_RC_SUCCESS
A memory handle was successfully found and returned.
GNI_RC_NO_MATCH
A memory handle was not found for the supplied NIC or a memory
handle was not found after the supplied memory handle.
The supplied memory handle was invalid or not found.
3.8 FMA DM
3.8.1 PostFma
The PostFma function executes a data transaction (PUT, GET, or AMO) by storing
into the directly mapped FMA window to initiate a series of FMA requests. It returns
before the transaction is confirmed by the remote NIC.
On Aries systems, the maximum transaction size is 1 MB. Requests for larger
transactions will return GNI_RC_SIZE_ERROR.
106
S–2446–51
On Gemini systems, PUTs have no alignment restrictions and GETs require 4-byte
alignment for local address, remote address, and the length.
Aries systems differ in that GETs do not require a 4-byte aligned local address.
On Gemini systems, Non-fetching (PUT) AMOs require the remote address and
length to be 8-byte aligned. Non-fetching (PUT) 2-operand AMOs are not supported.
Fetching (GET) AMOs require the local address, remote address and length to be
8-byte aligned.
Aries systems differ in that the non-fetching (PUT) 2-operand AMO operations (AAX
and ACSWAP) have alignment restrictions: 32-bit operands (DMAPP_DW) must be
8-byte aligned and 64-bit operands (DMAPP_QW) must be 16-byte aligned. To avoid
this restriction, use the fetching (GET) AMO AAX or ACSWAP operation instead.
The remote addresses and local addresses used by PUTs and GETs
should be initialized, and MemRegistered with the NIC. A PostFma
request can use a source memory region that has been registered using the
GNI_MemRegisterSegments.
Deadlock prevention logic on the Aries NIC requires that the source buffer remains
untouched until the global completion event is received so that the contents of the
source buffer may be resent if necessary. Since the transaction may fail, the user may
wish to retain the source buffer until global completion event has been received. (The
Gemini NIC assumes the source buffer becomes immediately reusable upon return
from PostFma.)
A FMA PUT request can use a source memory region that has been registered using
GNI_MemRegisterSegments. However, when these segmented memory regions
are used, the user must specify a virtual address for the source buffer. To transfer the
data from these memory regions, the user must issue a separate FMA PUT request
for each memory region with the length for the request being equal to or less than the
length of each memory region. If an absolute offset is used for the source buffer, a
segmentation fault will occur because the FMA PUT request uses this absolute offset
as a virtual address for the source buffer.
Zero-length FMA PUT operations are supported. Zero-length FMA GET and
zero-length FMA AMO operations are not supported.
3.8.1.1 Synopsis
gni_return_t GNI_PostFma (
IN gni_post_descriptor_t *post_descr)
S–2446–51
107
3.8.1.2 Parameters
ep_handle
Instance of a local endpoint.
post_descr
Pointer to a descriptor to be posted. See gni_post_descriptor
on page 172.
GNI_RC_SUCCESS
The descriptor was successfully posted.
The endpoint handle was invalid.
GNI_RC_ALIGNMENT_ERROR
The posted source or destination data pointers or data length are not
properly aligned. There are no alignment restrictions on PUTs. GETs
require 4 byte-alignment. AMOs require 8 byte-alignment, except
AAX which requires 16 byte-alignment.
GNI_RC_SIZE_ERROR
Maximum transaction size exceeded. Maximum on Aries is 1 MB.
The transaction request failed due to insufficient resources.
3.9 FMA Short Messaging (SMSG)
SMSG enables users to send short messages between endpoints on separate nodes.
An SMSG message is a special form of FMA PUT transaction, using a messaging
type sometimes referred to as a mailbox. Each connection has a dedicated destination
buffer called a mailbox where messages are delivered to. A process can not set
multiple endpoints to use the same SMSG destination buffer.
3.9.1 Sending SMSG Messages
An application must first initialize a local endpoint with communication parameters
and preregistered buffers required for performing FMA transactions and bind it to a
remote address. The application determines the necessary buffer size needed for each
mailbox using SmsgBufferSizeNeeded.
An application sets the message attributes in the gni_smsg_attr_t structure and
sends the attributes and control information to the remote node using EpPostData.
This function sends datagrams containing control information which establish the
point-to-point connection. See EpPostData on page 75.
108
S–2446–51
EpPostDataTest may be used to determine if the remote nodes were properly
configured and are ready to receive short messages.
Then initialize the endpoints using the SmsgInit function. See SmsgInit on
page 110 and gni_smsg_attr on page 178.
The application may then call SmsgSend to send a message to a remote peer. An
application calls SmsgSend function with pointer, length and control information.
Note that the msg_id provided to GNI_Smsgsend() is used in the global
completion event associated with the transaction. See SmsgSend on page 111.
3.9.2 Processing SMSG Messages
Process the local completion queue to verify that the message request has been
sent and remove the event from the queue so it does not fill up, which will cause
subsequent SmsgSend requests to fail.
A receiving peer processes its completion queue when data has been received. A
completion event is received from the CQ that was registered with the memory
used for creating the SMSG connection. A completion queue processing call (e.g.,
GNI_CqGetEvent) will be used to dequeue events from the completion queue.
Macros may be used to extract information from the event structure. For example,
the macro GNI_CQ_GET_MSG_ID(event) retrieves the msg_id, from the
event structure and it should only be used for local events. Also, if the CQ is being
used for several types of messaging, a user could use GNI_CQ_GET_TYPE() to
determine if this was a SMSG event, in a situation where the CQ is being used for.
GNI_CQ_GET_TYPE macros should only be used for local events.
An application should use a dedicated CQ for remote events and cannot rely on
GNI_CQ_GET_TYPE to determine the type of the incoming remote event. A
receiving peer should use GNI_CQ_GET_REM_INST_ID(event) to obtain the
sender's instance ID. See gni_cq_entry on page 166.
Use SmsgGetNext to obtain a pointer to the header of the next available message
for a given connection. Either the application process the messages immediately or
copies them to another buffer. The application must release the message buffer when
it is no longer needed. See SmsgGetNext on page 115.
Depending on the size of the message and the number of messages sent, GNI
sometimes needs to send an event to the destination CQ for accounting purposes.
This accounting event is used to notify GNI that some back credits can be released.
Therefore, the user needs to process these accounting CQ events so that the CQ will
not overrun. The following example processes the destination CQ and the received
SMSG message. You need to use the instance id of the event as the index into the
endpoints array and the mailbox.
S–2446–51
109
Example 1. Process the destination CQ and the received SMSG message
/* This pseudo code will process all of the received CQ events, including
the accounting events, and SMSG messages. Because of the potential
accounting CQ events, the number of CQ events that are processed can
be greater than tne number of received SMSG messages. There is no
way to distinguish a normal CQ event from an accounting CQ event.
An accounting CQ event does not have a SMSG message associated with it.
Therefore, for each CQ events received you must check to see if there is
a SMSG message available to process. If no SMSG message has arrived,
get the next CQ event. Continue processing all of the CQ events and
SMSG messages until you receive all of the expected SMSG messages.
Assumes that the number_of_required_messages has been set
to the number of smsg messages which you have sent and
are waiting to process.
*/
do {
status = GNI_CqGetEvent(destination_cq, &event );
if (status == GNI_RC_SUCCESS) {
event_inst_id = GNI_CQ_GET_INST_ID(event);
/* Retrieve the SMSG message. */
status = GNI_SmsgGetNext(endpoints[event_inst_id],
(void **)&message_header );
if (status == GNI_RC_SUCCESS) {
received_messages++;
/* Release the SMSG message. */
status = GNI_SmsgRelease(endpoints[event_inst_id]);
}
} else if (status != GNI_RC_NOT_DONE) {
if (GNI_CQ_OVERRUN(event)) {
fprintf(stdout, "GNI_CQ_OVERRUN destination,
status: %s\n", gni_err_str[status] );
} else {
fprintf(stdout, "GNI_CqGetEvent destination.
status: %s\n", gni_err_str[status] );
}
} else if (received_messages >= number_of_required_messages) {
/* At this point, there are no more messages left in the CQ
and you have processed the expected number of SMSG events,
or greater, depending on the number of accounting CQ
events which may have been interspersed. */
break;
}
} while (1 == 1);
3.9.3 SmsgInit
The SmsgInit function configures the short messaging protocol on the given
endpoint. Short messaging buffers must be zeroed before calling SmsgInit.
110
S–2446–51
3.9.3.1 Synopsis
gni_return_t GNI_SmsgInit (
IN gni_smsg_attr_t *local_smsg_attr,
IN gni_smsg_attr_t *remote_smsg_attr)
3.9.3.2 Parameters
ep_handle
Instance of an endpoint.
local_smsg_attr
Pointer to a list of local parameters used for short messaging.
Parameter values are defined using the gni_smsg_attr structure.
remote_smsg_attr
Pointer to a list of remote parameters that are used for short
messaging, provided by a peer. Parameter values are defined using
the gni_smsg_attr structure.
GNI_RC_SUCCESS
Endpoint is not bound.
GNI_RC_ERROR_NOMEM
Insufficient memory to allocate short message internal structures.
3.9.4 SmsgSend
The SmsgSend function sends a message to the remote peer by copying it into the
preallocated remote buffer space, using the FMA mechanism. The function returns
before the delivery is confirmed by the remote NIC; SmsgSend is a non-blocking
call.
S–2446–51
111
Before using this function, the message type will have been configured by defining
msg_type in the gni_smsg_attr_t structure passed to SmsgInit.
Note: msg_type GNI_SMSG_TYPE_MBOX is not supported on Aries. Therefore,
on Aries systems, user provided header and data buffers should be kept until a
global completion event for the SMSG transaction is received, regardless of the
completion status.
On Gemini systems, when msg_type is GNI_SMSG_TYPE_MBOX, the connection
does not attempt to recover from any network errors. When SmsgSend returns, its
header and data buffers can be freed.
When the msg_type is GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT, an SMSG
connection automatically detects recoverable network errors and attempts to re-send
any failed messages. Therefore, when using the auto-retransmitting mode, user
provided buffers must be kept until the receipt of a global completion event.
The msg_id provided to SmsgSend() is used in the global completion
event associated with the transaction. It can be extracted from an event
dequeued using GNI_CqGetEvent() (or similar functions) using
GNI_CQ_GET_MSG_ID(event).
The number of GNI SMSG send credits equates to the maximum number of messages
the SMSG mailbox can hold at one time. Send credits are independent of buffer space
available, so when sending messages smaller than the max message size specified
to GNI_SmsgInit, the mailbox space is under-utilized. Each SMSG connection
will set the SMSG send credits count equal to the smsg_q_sz by default. See
gni_smsg_attr on page 178.
Note: The SMSG interface uses the FMA mechanism with adaptive routing. This
allows SMSG messages to arrive out of order at the target node. Therefore, it is
possible for completion events to be delivered to the remote completion queue
while GNI_SmsgGetNext reports that no new messages are available. To handle
this case when using remote events to detect the arrival of SMSG sends, be sure to
clear all messages from an endpoint using GNI_SmsgGetNext after receiving
each remote completion event.
3.9.4.1 Synopsis
gni_return_t GNI_SmsgSend (
IN void *header,
IN uint32_t header_length,
IN void *data,
IN uint32_t data_length,
IN uint32_t *msg_id)
112
S–2446–51
3.9.4.2 Parameters
ep_handle
An instance of an endpoint.
header
A pointer to the header of a message.
header_length
The length of the header in bytes.
data
A pointer to the payload of the message.
data_length
The length of the payload in bytes.
msg_id
Identifier for the application to track transaction. Only valid for short
messaging using GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT
type, otherwise ignored. Used in the global completion event
associated with the transaction.
GNI_RC_SUCCESS
The transmission has been initiated.
The endpoint handle was invalid or the endpoint is not initialized
for short messaging.
GNI_RC_NOT_DONE
No credits available to send the message.
The size of the header plus data exceeds the maximum short message
size given in GNI_SmsgInit.
3.9.5 SmsgSendWTag
The GNI_SmsgSendWTag() function sends a tagged message to the remote peer,
by copying it into the preallocated remote buffer space, using the FMA mechanism.
The function returns before the delivery is confirmed by the remote NIC. When the
endpoint is set with GNI_SMSG_MBOX_AUTO_RETRANSMIT type, the system
attempts to retransmit for certain transaction failures. This is a non-blocking call.
S–2446–51
113
3.9.5.1 Synopsis
gni_return_t GNI_SmsgSendWTag(
IN gni_ep_handle_t ep_hndl,
IN void *header,
IN void *data,
IN uint32_t data_length,
IN uint32_t msg_id,
IN uint8_t tag)
3.9.5.2 Parameters
ep_hndl
An instance of an endpoint.
header
header_length
data
A pointer to the payload of the message.
data_length
The length of the payload in bytes.
msg_id
Identifier for application to track transaction. Only valid for short
messaging using the short message type (gni_smsg_type_t)
GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT type, otherwise
ignored.
tag
Tag associated with the short message.
GNI_RC_SUCCESS
The transmission was initiated.
GNI_RC_NOT_DONE
The operation failed due to insufficient memory.
114
S–2446–51
3.9.6 SmsgGetNext
The SmsgGetNext function returns a pointer to the header of the newly arrived
message and makes this message current. You can set up your application to copy the
message out of the mailbox or process it immediately. This is a non-blocking call.
It is required to call GNI_SmsgRelease after every call to GNI_SmsgGetNext
on the same endpoint. If the GNI_SmsgRelease is not called, then the
mailbox read pointer will not be incremented, which will cause every succeeding
SmsgGetNext call to return the same SMSG message.
3.9.6.1 Synopsis
gni_return_t GNI_SmsgGetNext (
OUT void **header)
3.9.6.2 Parameters
ep_handle
header
Pointer to the header of the newly arrived message.
GNI_RC_SUCCESS
The new message arrived successfully.
GNI_RC_NOT_DONE
There are no new messages available.
The connection has entered an invalid state. The user provided
SMSG mailbox buffer has been corrupted. A user should reinitialize
the connection.
3.9.7 SmsgGetNextWTag
The SmsgGetNextWTag function returns a pointer to the header of the newly
arrived message and makes this message current if the input tag matches the tag of the
newly arrived message. An application may decide to copy the message header out of
the mailbox or process the header immediately. This is a non-blocking call.
S–2446–51
115
3.9.7.1 Synopsis
gni_return_t GNI_SmsgGetNextWTag (
IN gni_ep_handle_t ep_hndl,
OUT void **header,
INOUT uint8_t *tag)
3.9.7.2 Parameters
ep_handle
header
Pointer to the header of the newly arrived message.
tag
On input, a pointer to the value of the remote event to be matched.
A wildcard value of GNI_SMSG_ANY_TAG is used to match any
tag value of the incoming message. The value is set to that of the
matching remote event on output.
GNI_RC_SUCCESS
The new message arrived successfully.
GNI_RC_NOT_DONE
There are no new messages available.
GNI_RC_NO_MATCH
The message is available, but the tag of the message does not match
the value supplied in the tag argument.
The connection has entered an invalid state. The user provided
SMSG mailbox buffer has been corrupted. A user should reinitialize
the connection.
3.9.8 SmsgRelease
The SmsgRelease function releases the current message buffer. It must be called
only after GetNext has returned GNI_RC_SUCCESS. This is a non-blocking call.
The message returned by the GetNext function must be copied out or processed
before making this call.
116
S–2446–51
3.9.8.1 Synopsis
gni_return_t GNI_SmsgRelease (
3.9.8.2 Parameters
ep_handle
GNI_RC_SUCCESS
The current message is successfully released.
GNI_RC_NOT_DONE
There is no current message. The GetNext function must return
GNI_RC_SUCCESS before calling this function.
3.9.9 SmsgSetDeliveryMode
Set SMSG delivery mode for the SMSG transactions.
3.9.9.1 Synopsis
gni_return_t GNI_SmsgSetSetDeliveryMode (
IN gni_nic_handle_t nic_handle
IN uint16_t dlvr_mode);
3.9.9.2 Parameters
nic_handle
NIC handle to change.
dlvr_mode
The new SMSG delivery mode. Delivery modes are defined in
gni_pub.h and are named GNI_DLVMODE_*.
GNI_RC_SUCCESS
Invalid NIC handle or delivery mode.
S–2446–51
117
3.9.10 SmsgSetMaxRetrans
Configure SMSG maximum retransmit count. Endpoints associated with the
NIC handle provided will give up retransmitting SMSG transactions and return
GNI_RC_TRANSACTION_ERROR when the retransmit count has been reached.
3.9.10.1 Synopsis
gni_return_t GNI_SmsgSetMaxRetrans (
IN gni_nic_handle_t nic_handle
IN uint16_t max_retrans);
3.9.10.2 Parameters
nic_handle
NIC handle to change.
max_retrans
Maximum retransmit count.
GNI_RC_SUCCESS
Invalid NIC handle.
GNI_RC_NOT_DONE
There is no current message. The GetNext function must return
GNI_RC_SUCCESS before calling this function.
3.9.11 SmsgBufferSizeNeeded
Determine the amount of memory needed to be allocated for short messaging
resources using a gni_smsg_attr_t structure with the mbox_max
credit, msg_maxsize and msg_type fields set. Given this set of parameters,
GNI_SmsgBufferSizeNeeded() returns the required size of SMSG buffers.
End users need to use this value to allocate their SMSG buffers and fill in the
buff_size field of the gni_smsg_attr_t before passing it to GNI_SmsgInit().
3.9.11.1 Synopsis
gni_return_t GNI_SmsgBufferSizeNeeded(
IN gni_smsg_attr_t *smsg_attr,
OUT unsigned int
*size);
118
S–2446–51
3.9.11.2 Parameters
smsg_attr
The msg_buffer, buff_size, mem_hndl, and mbox_offset
fields in the input smsg_attr structure do not need to be defined.
size
The minimum size of the SMSG buffer.
GNI_RC_SUCCESS
3.10 Shared Message Queue (MSGQ)
The MSGQ permits applications based on programming models which exchange
short control messages between nodes, such as MPICH, to achieve greater scalability.
Message buffers used for a MSGQ reside in shared memory used by all participating
software instances on a node.
The MSGQ does not support passing messages between processes on the same node.
MSGQ does not allow connections from multiple processes on the same node to the
same shared message queue using the same NIC address. This prevents applications
from passing messages from one process to another process on the same node.
MSGQ is intended as a message passing mechanism between different nodes.
S–2446–51
119
To create a shared message queue system, each job instance in a job must:
1. Create and fill a gni_msgq_attr_t structure.
2. Call GNI_MsgqInit to attach to a shared message queue.
3. Perform a global barrier.
The leader instance is the one that first attempts to attach to the shared message queue
and thereby creates the shared buffer resources. Any one instance on each node can
act as the leader and perform the following steps:
1. Call GNI_MsgqGetConnAttrs with the message queue handle obtained
during initialization, the remote node's PE address and a pointer to a
gni_msgq_ep_attr_t structure. This fills the gni_msgq_ep_attr_t
structure with the attributes needed on the remote node to establish the internode
connection.
2. Trade the gni_msgq_ep_attr_t structure with the node with the PE
address specified in previous step.
3. Call GNI_MsgqConnect with the received endpoint attribute structure.
4. Repeat from step 1 for each remote node participating in the message queue
system.
5. Perform a global barrier.
Job instances will send messages using the GNI_MsgqSend function. If successful
this function delivers a completion event to the completion queue attached to the
endpoint provided in the send. Job instances will receive messages using the
GNI_MsgqProgress function.
3.10.1 MsgqInit
Create and initialize the resources required for the shared message queue, including
the shared memory buffer and the receive completion queue. Receive CQ's will be
created per job instance on each node.
Each job instance will attach to the shared buffer during initialization. The first
job instance to attach to the shared buffer will create shared buffer resources and
initialize all shared variables according to initialization attributes described in
gni_msgq_attr on page 179.
Register the shared region with a private receive completion queue and store the
provided message queue attributes as control information in the shared area. Map
connection information stored in the shared buffer to its remote PE address.
Job attributes must be equal for all instances in the job and node attributes must be
equal for each instance on a node. Instance attributes may be unique for each job
instance.
120
S–2446–51
The number of connections that the shared message queue will support will be
specified at initialization time. Once all job instances complete their call to
GNI_MsgqInit, each node designates one of its job instances to be the leader,
responsible for each remote node connection in the message queue.
3.10.1.1 Synopsis
GNI_MsgqInit (
IN gni_msgq_rcv_cb_func *rcv_cb,
IN void *cb_data,
IN gni_cq_handle_t snd_cq,
IN gni_msgq_attr_t *attrs,
OUT gni_msgq_handle_t *msgq_handle)
3.10.1.2 Parameters
nic_handle
The handle of the NIC device to attach to shared message queue.
rcv_cb
Callback function to handle received messages.
cb_data
User data to pass to the receive callback function specified by rcv_cb.
snd_cq
Send completion queue to use with message queue.
attrs
Attributes for message queue initialization.
msgq_handle
A handle for the created message queue resources.
S–2446–51
121
GNI_RC_SUCCESS
Message Queue initialization succeeded.
GNI_RC_ERROR_NOMEM
There was insufficient memory available to attach to the shared
memory region.
The attributes provided do not match the existing message queue
attributes or all instances were not ready to attach to the shared
memory area.
The hugetlbfs file system was not available.
3.10.2 MsgqRelease
Releases all resources created during GNI_MsgqInit. All transactions must be
completed (or all endpoints destroyed) before calling GNI_MsgqRelease. The
user is unable to allocate new connections while this function is executing. Each
connection is locked and disabled unless there are outstanding transactions on that
connection. If there are outstanding transactions, returns GNI_RC_NOT_DONE.
3.10.2.1 Synopsis
GNI_MsgqRelease (
IN gni_msgq_handle_t msgq_hndl)
3.10.2.2 Parameters
msgq_hndl
122
The handle for the message queue to use for the operation.
S–2446–51
GNI_RC_SUCCESS
Message Queue resources successfully released.
GNI_RC_NOT_DONE
There are outstanding message queue transactions; those connections
are not released, leaving a partially disabled message queue.
3.10.3 MsgqGetConnAttrs
Assigns connection resources to a remote endpoint address and returns attributes for
completing the connection. The remote PE address provided is assigned to an SMSG
control structure and mailbox for use in an internode connection. The attributes must
be traded with the remote endpoint to establish the connection.
3.10.3.1 Synopsis
GNI_MsgqGetConnAttrs (
IN gni_msgq_handle_t msgq_hndl,
IN uint32_t pe_addr,
OUT gni_msgq_ep_attr_t *attrs,
OUT uint32_t *attrs_size)
3.10.3.2 Parameters
S–2446–51
msgq_hndl
The handle of the message queue to use for the operation.
pe_addr
The PE address of the remote endpoint to assign connection
resources to. If NTT is enabled, it is the virtual address.
attrs
The attribute structure needed to establish a message queue
connection on the remote endpoint.
attrs_size
If non-null, returns the size of the attrs structure.
123
GNI_RC_SUCCESS
Connection resources were assigned to the PE address.
Connection resources have already been assigned to the PE address
provided.
All connection resources have already been assigned.
Message queue initialization has not completed or MsgqRelease
has been started.
3.10.4 MsgqConnect
Connect to a shared message queue. Use the remote PE address, pe_addr
to look up the shared connection resources that were assigned during
GNI_MsgqGetConnAttrs. Complete the internode message queue connection by
adding the remote endpoint attributes attrs, to the connection resources. A connection
to a shared message queue will not be allowed if the shared message queue has not
been initialized or the shared message queue has been released.
3.10.4.1 Synopsis
GNI_MsgqConnect (
IN gni_msgq_handle_t msgq_handle,
IN uint32_t pe_addr,
IN gni_msgq_ep_attr_t *attrs)
3.10.4.2 Parameters
msgq_handle
124
pe_addr
The PE address of the remote endpoint to assign connection
resources to. If NTT is enabled, it is the virtual address.
attrs
The attribute structure received from the remote node.
S–2446–51
GNI_RC_SUCCESS
The connection was established.
GNI_RC_NO_MATCH
The associated connection resources could not be found.
A connection to the PE specified by the attribute structure has already
been established.
Message queue initialization has not completed or MsgqRelease
has been started.
3.10.5 MsgqConnRelease
Release connection resources from a remote PE, assigned via
GNI_MsgqGetConnAttrs. All outstanding transactions on the
connection must be completed before calling GNI_MsgqConnRelease.
Connection resources released in this call may be reassigned with a call to
GNI_MsgqGetConnAttrs.
3.10.5.1 Synopsis
GNI_MsgqConnRelease (
IN uint32_t pe_addr)
3.10.5.2 Parameters
msgq_handle
pe_addr
S–2446–51
The PE address of the remote endpoint to release.
125
GNI_RC_SUCCESS
The connection resources were released.
GNI_RC_NO_MATCH
The associated message queue connection for the specified PE could
not be found.
GNI_RC_NOT_DONE
There are outstanding message queue transactions; the connection
was not released.
3.10.6 MsgqSend
Send a message to a remote message queue by using the message queue handle and
the endpoint to determine the remote node.
If successful, a completion event will be delivered to the endpoint's completion
queue. Event data will have type GNI_CQ_EVENT_TYPE_MSGQ. If the send was
successful, the lower 32 bits of the event data will contain the message ID used for
the send. If the event has an error status, the lower 32 bits of the event data will
contain the remote instance ID in the send endpoint. This matches the behavior of
GNI_SmsgSend.
3.10.6.1 Synopsis
gni_return_t GNI_MsgqSend (
IN void *header,
IN void *msg,
IN uint32_t msg_length,
IN uint32_t msg_id
IN uint8_t msg_tag)
126
S–2446–51
3.10.6.2 Parameters
msgq_handle
The handle for the message queue to use for the operation.
ep_handle
The endpoint describing the target for the send.
header
header_length
msg
A pointer to the message.
msg_length
The length of the message in bytes.
msg_id
Message identifier. Returned in a local completion event.
msg_tag
The message tag sent with message data.
GNI_RC_SUCCESS
The send completed successfully.
An invalid input parameter was provided.
GNI_RC_NO_MATCH
No message queue connection for the endpoint was found.
GNI_RC_NOT_DONE
GNI_RC_SIZE_ERROR
The message size exceeds the maximum message size.
Connection resources exist but are inactive (not yet connected).
3.10.7 MsgqSize
Returns the size of the allocated MSGQ shared buffer. The size is specified in bytes.
The size is always a multiple of the configured hugetlbfs hugepage size.
S–2446–51
127
3.10.7.1 Synopsis
GNI_MsgqSize (
IN gni_msgq_attr_t *attrs,
OUT uint32_t *size)
3.10.7.2 Parameters
attrs
The attributes for the message queue system initialization.
size
The size, in bytes, required to create the MSGQ with the given set
of parameters.
GNI_RC_SUCCESS
The input parameter was invalid.
3.10.8 MsgqProgress
Polls the shared message queue until an event is received or the timeout expires. If
(-1) is specified as the timeout value, the function will return immediately if there are
no messages available.
When an event is received, the registered receive callback function is called
with the message data. If the user provided callback function returns true,
GNI_MsgqProgress will attempt to process another message. If the callback
returns false, GNI_MsgqProgress will return immediately.
3.10.8.1 Synopsis
GNI_MsgqProgress (
IN uint32_t timeout)
3.10.8.2 Parameters
msgq_handle
timeout
128
The number of milliseconds to block waiting for each
message. If a positive timeout value is specified, the mode
GNI_MSGQ_MODE_BLOCKING must be used during MSGQ
creation, otherwise an GNI_RC_INVALID_PARAM error results.
S–2446–51
GNI_RC_SUCCESS
All messages were processed.
Invalid msgq_handle or incorrect timeout value.
GNI_RC_NOT_DONE
Messages may still be available for processing.
The send CQ is full.
An unexpected completion queue event was received.
GNI_RC_ERROR_NOMEM
Insufficient memory was available to complete the operation.
3.11 RDMA (BTE)
3.11.1 PostRdma
The PostRdma function adds a descriptor to the tail of the RDMA queue and returns
immediately. The maximum RDMA post length is ((2^32)-1). Note that DMAPP
supports RDMA transfers of ((2^32)-1) and larger.
PUTs have no alignment restrictions.
On Gemini systems, GETs require 4-byte alignment for local address, remote
address, and the length. Aries systems differ in that GETS do not require 4-byte
aligned local address.
The remote addresses and local addresses used by PUTs and GETs should be
initialized, and MemRegister'ed with the NIC.
A PostRdma request can use a source memory region that has been registered using
the GNI_MemRegisterSegments.
3.11.1.1 Synopsis
gni_return_t GNI_PostRdma (
S–2446–51
129
3.11.1.2 Parameters
ep_handle
post_descr
Pointer to the descriptor to be posted to the queue.
GNI_RC_SUCCESS
Posted source, destination data pointers, or data length are not
properly aligned.
The transaction request could not be posted due to insufficient
resources.
GNI_RC_ERROR_NOMEM
The user's buffer R/W permissions conflict with the access type.
3.12 Completion Queue Processing
A uGNI application monitors the completion queue in order to track events, and
process event data.
The CqGetEvent, CqWaitEvent, and CqVectorWaitEvent functions
return a completion queue entry which may be used as input to event processing
functions and error decoding functions. See gni_cq_entry on page 166 for more
detailed information about completion queue entries.
The CqVectorMonitor and CqVectorWaitEvent functions allow a user to
monitor or receive a CQ event from an array of completion queues.
130
S–2446–51
The CqVectorMonitor function will return the index value of the completion
queue that has a CQ event waiting to be processed. This index value is used to get the
completion queue handle from the array of completion queues. CqVectorMonitor
does not process events; when it returns, the user needs to call CqGetEvent to
retrieve an event from the CQ identified by CqVectorMonitor. The user does the
appropriate processing on this completion queue.
The CqVectorWaitEvent function will return both the index of the completion
queue and the CQ event.
When monitoring an array of completion queues via CqVectorMonitor and
CqVectorWaitEvent functions, the CqInterruptMask function allows
the application to disable receiving CQ events for a specific completion queue.
CqInterruptUnmask is subsequently used to clear the mask, enabling CQ events
for a specific completion queue.
3.12.1 CqGetEvent
The CqGetEvent function returns information about the next event by polling the
specified completion queue for a completion entry. If a completion entry is found,
it returns the event data stored in the entry. CqGetEvent is a non-blocking call.
CqGetEvent only de-queues the completion entry from the completion queue.
If necessary, the calling process must invoke CqCompleted function to dequeue
the completed descriptor.
3.12.1.1 Synopsis
gni_return_t GNI_CqGetEvent (
IN gni_cq_handle_t cq_handle,
OUT gni_cq_entry_t *event_data)
3.12.1.2 Parameters
S–2446–51
cq_handle
The handle for the completion queue.
event_data
Event entry data is placed at the address pointed to by event_data.
The contents of an event entry is dictated by the status and type of
the transaction associated with the event. See gni_cq_entry on
page 166.
131
GNI_RC_SUCCESS
A completion entry was found in the completion queue.
GNI_RC_NOT_DONE
No new completion entries are in the completion queue.
The completion queue handle was invalid.
The completion queue is in an overrun (full) state and completion
queue events may have been lost.
A network error was encountered while processing a transaction.
3.12.2 CqInterruptMask
This function increments the interrupt mask for the specified completion queue.
By incrementing the interrupt mask, this will disable the current process from
receiving CQ events for the specified completion queue. If you increment this
interrupt mask multiple times, then you will need to decrement it the same number
of times, using CqInterruptUnmask, before you will be allowed to process CQ
events for the specified completion queue. The CQ Interrupt mask is only applicable
when using the CqVectorMonitor and CqVectorWaitEvent functions.
3.12.2.1 Synopsis
gni_return_t GNI_CqInterruptMask (
3.12.2.2 Parameters
cq_handle
132
Completion queue handle.
S–2446–51
GNI_RC_SUCCESS
The interrupt mask was incremented successfully.
The interrupt mask was not allocated for completion queue.
GNI_RC_NOT_DONE
The interrupt mask was not incremented.
3.12.3 CqInterruptUnmask
This function decrements the interrupt mask for the specified completion queue.
You will need to decrement the interrupt mask the same number of times that you
incremented it, before the process will be able to receive CQ events from the specified
completion queue.
3.12.3.1 Synopsis
gni_return_t GNI_CqInterruptUnmask (
3.12.3.2 Parameters
cq_handle
Completion queue handle.
GNI_RC_SUCCESS
The interrupt mask was decremented successfully.
The interrupt mask was not allocated for completion queue.
GNI_RC_NOT_DONE
The interrupt mask was not decremented.
S–2446–51
133
3.12.4 CqTestEvent
The CqTestEvent function monitors the specified completion queue for a
completion entry. If a completion entry is found, it returns GNI_RC_SUCCESS,
unless the CQ is full, in which case it returns GNI_RC_ERROR_RESOURCE. If no
completion entry is found, GNI_RC_NOT_DONE is returned. No processing of new
entries is performed by this function.
3.12.4.1 Synopsis
GNI_CqTestEvent (
3.12.4.2 Parameters
cq_handle
GNI_RC_SUCCESS
A completion entry was found on the completion queue.
GNI_RC_NOT_DONE
No new completion entries are on the completion queue.
CQ is full and CQ entries may have been lost.
3.12.5 CqVectorMonitor
The CqVectorMonitor function polls the specified completion queues for a
completion entry. If it finds a completion entry, it immediately returns the array
index for the CQ.
If no completion entry is found, the caller is blocked until a completion entry is
generated, or until the timeout value expires. The completion queues must be created
with the GNI_CQ_BLOCKING mode set in order to be able to block on it.
3.12.5.1 Synopsis
gni_return_t GNI_CqVectorMonitor (
IN gni_cq_handle_t *cq_handles,
IN uint32_t num_cqs,
OUT uint32_t *which)
134
S–2446–51
3.12.5.2 Parameters
cq_handles
Array of completion queue handles.
num_cqs
Number of completion queue handles
timeout
The number of milliseconds to block before returning to the caller.
Set to -1 to specify no timeout.
which
Array index for the CQ which returned an event or error.
GNI_RC_SUCCESS
A completion entry was found in the completion queue.
GNI_RC_TIMEOUT
The request timed out and no completion entry was found.
GNI_RC_NOT_DONE
The completion queue handle had the interrupt mask set and no event
was processed.
One of the completion queue handles was invalid.
One of the completion queues was not created in the
GNI_CQ_BLOCKING mode.
GNI_RC_ERROR_NOMEM
No memory available for the allocation of the CQ descriptor or event
pointers.
3.12.6 CqVectorWaitEvent
The CqVectorWaitEvent function polls the specified completion queues
for a completion entry. If CqVectorWaitEvent finds a completion entry, it
immediately returns event data.
generated, or until the timeout value expires. The completion queues must be created
S–2446–51
135
3.12.6.1 Synopsis
gni_return_t GNI_CqVectorWaitEvent (
IN gni_cq_handle_t *cq_handls,
IN uint32_t num_cqs
OUT gni_cq_entry_t *event_data,
OUT uint32_t *which)
3.12.6.2 Parameters
136
cq_handls
Array of handles for the completion queues.
num_cqs
Number of completion queue handles.
timeout
The number of milliseconds to block before returning to the caller;
set this to -1 if no timeout is desired.
event_data
Event entry data is always placed at the address pointed to by
event_data. The contents of an event entry is dictated by the
status and type of the transaction associated with the event. See
gni_cq_entry on page 166.
which
Returns the index of the CQ within the cq_handls array which
returned event_data on success. Undefined otherwise.
S–2446–51
GNI_RC_SUCCESS
GNI_RC_TIMEOUT
GNI_NOT_DONE
The CQ handle had the interrupt mask set and no event was
processed.
One of the completion queue handles was invalid.
One of the completion queues was not created in the
GNI_CQ_BLOCKING mode.
GNI_RC_ERROR_NOMEM
No memory was available for the allocation of the CQ descriptor or
event pointers.
3.12.7 CqWaitEvent
The CqWaitEvent function polls the specified completion queue for a completion
entry. If CqWaitEvent finds a completion entry, it immediately returns the event
data.
generated, or until the timeout value expires. The completion queue must be created
3.12.7.1 Synopsis
gni_return_t GNI_CqWaitEvent (
OUT gni_cq_entry_t *event_data)
S–2446–51
137
3.12.7.2 Parameters
cq_handle
timeout
The number of milliseconds to block before returning to the caller;
set this to -1 if no timeout is desired.
event_data
Event entry data is always placed at the address pointed to by
event_data. The contents of an event entry is dictated by the
status and type of the transaction associated with the event. See
gni_cq_entry on page 166.
GNI_RC_SUCCESS
GNI_RC_TIMEOUT
The completion queue was not created in the GNI_CQ_BLOCKING
mode.
3.12.8 GetCompleted
The GetCompleted function gets the next completed post descriptor from the
specified completion queue. The descriptor is removed from the head of the queue
and the address of the descriptor is returned.
GetCompleted is a non-blocking call.
See gni_post_descriptor on page 172.
3.12.8.1 Synopsis
gni_return_t GNI_GetCompleted (
IN gni_cq_entry_t event_data,
OUT gni_post_descriptor _t **post_descr)
138
S–2446–51
3.12.8.2 Parameters
cq_handle
Handle for the completion queue.
event_data
The event returned by the CqGetEvent function.
post_desc
Returns a pointer to the address of the descriptor that has completed.
GNI_RC_SUCCESS
A completed descriptor was returned with a successful completion
status.
GNI_RC_DESCRIPTOR_ERROR
If the corresponding post queue (FMA, RDMA or AMO) is empty,
the descriptor pointer is set to NULL.
The CQ handle was invalid.
A completed descriptor was returned with a network error status.
3.12.9 PostCqWrite
The PostCqWrite function executes a CQ write transaction to a remote CQ. It
returns before the transaction is confirmed by the remote NIC.
Send up to 6 bytes in the CQ event by setting cqwrite_value in the
gni_post_descriptor — only the 6 least significant bytes will be delivered.
The receiver should use GNI_CQ_GET_DATA to retrieve data from the event.
3.12.9.1 Synopsis
gni_return_t GNI_PostCqWrite (
3.12.9.2 Parameters
S–2446–51
ep_handle
post_descr
Pointer to a descriptor to be posted.
139
GNI_RC_SUCCESS
GNI_RC_RESOURCE_ERROR
Insufficient were resources available to initialize the endpoint.
3.12.10 CqErrorStr
The CqErrorStr function decodes the error status encoded in a CQ entry by the
hardware.
3.12.10.1 Synopsis
gni_return_t GNI_CqErrorStr (
IN gni_cq_entry_t entry,
OUT void *buffer,
IN uint32_t length)
entry
CQ entry with error status to decode.
buffer
Pointer to the buffer where the error code is returned.
length
Length of the buffer in bytes.
GNI_RC_SUCCESS
The entry was successfully decoded.
Invalid input parameter.
GNI_RC_SIZE_ERROR
The supplied buffer is too small to contain the error code.
3.12.11 CqErrorRecoverable
The CqErrorRecoverable function translates any error status, encoded by the
hardware in a completion queue entry, into a recoverable or unrecoverable flag for
application usage.
140
S–2446–51
3.12.11.1 Synopsis
gni_return_t GNI_CqErrorRecoverable (
IN gni_cq_entry_t cq_handle,
OUT uint32_t *recoverable)
entry
Completion queue entry with error status to be decoded.
recoverable
Pointer to the integer flag that will contain the decoded result.
GNI_RC_SUCCESS
The entry was successfully decoded.
Invalid input parameter.
The completion queue entry translates to an undefined state.
3.13 Error Handling
3.13.1 SubscribeErrors
The SubscribeErrors function creates an error event queue. When this function
returns, events will start being delivered immediately. The error mask, mask,
determines which errors are reported. See gni_error_mask on page 165.
Privileged users, such as superusers, can pass in NULL for nic_handle which causes
the passed in device_id to be used instead. This allows privileged users to subscribe to
errors without a CDM being attached. By default, if no nic_handle is passed in, then
errors are captured for all ptags.
3.13.1.1 Synopsis
gni_return_t GNI_SubscribeErrors(
IN gni_error_mask_t mask,
IN uint32_t EEQ_size,
OUT gni_err_handle_t *err_handle)
S–2446–51
141
3.13.1.2 Parameters
nic_handle
The handle of the associated NIC.
device_id
The device identifier, for privileged mode (when NULL is passed
in for nic_handle).
mask
The error mask with corresponding bits set for notification.
EEQ_size
Size of the EEQ. The queue size uses a default of 64 entries if a value
of 0 is passed in.
err_handle
This handle is returned to identify the instance in uGNI.
GNI_RC_SUCCESS
One of the input parameters is invalid, or a non-privileged user is
trying to subscribe without a communication domain.
GNI_RC_NO_MATCH
Specified device_id does not exist.
The event queue could not be created due to insufficient resources.
GNI_RC_ERROR_NOMEM
3.13.2 ReleaseErrors
The ReleaseErrors function releases the error event notification and cleans up
the memory resources for the event queue.
3.13.2.1 Synopsis
gni_return_t GNI_ReleaseErrors(
IN gni_err_handle_t err_handle)
3.13.2.2 Parameters
err_handle
142
The handle of the subscribed error events.
S–2446–51
GNI_RC_SUCCESS
GNI_RC_NOT_DONE
A thread is still waiting on the event queue.
3.13.3 GetErrorMask
The GetErrorMask function returns the error mask associated with an
error handle. The mask determines which error events are delivered. See
gni_error_mask on page 165.
3.13.3.1 Synopsis
gni_return_t GNI_GetErrorMask(
IN gni_err_handle_t err_handle,
OUT gni_error_mask_t *mask)
3.13.3.2 Parameters
err_handle
mask
The pointer to copy the mask value to.
GNI_RC_SUCCESS
3.13.4 SetErrorMask
The SetErrorMask function sets a new error mask for matching events.
3.13.4.1 Synopsis
gni_return_t GNI_SetErrorMask(
IN gni_error_mask_t mask_in,
IN gni_error_mask_t *mask_out)
S–2446–51
143
3.13.4.2 Parameters
err_handle
mask_in
The error mask with corresponding bits set for notification.
mask_out
The pointer to copy the preset mask value to.
GNI_RC_SUCCESS
3.13.5 GetErrorEvent
The GetErrorEvent function gets an error event, if available.
3.13.5.1 Synopsis
gni_return_t GNI_GetErrorEvent(
IN gni_error_event_t *event)
3.13.5.2 Parameters
err_handle
event
The pointer to the buffer to copy the event into.
GNI_RC_SUCCESS
A completed descriptor was returned with a successful completion
status.
GNI_INVALID_PARAMETER
GNI_RC_NOT_DONE
No event was found in the event queue.
144
S–2446–51
3.13.6 WaitErrorEvents
The WaitErrorEvents function waits for an error event to occur. When the first
error event is triggered, it delays in returning to coalesce additional error events. The
timeout value is specified in number of milliseconds. The number of events copied
are stored in the num_events structure.
3.13.6.1 Synopsis
gni_return_t GNI_WaitErrorEvents(
IN gni_error_event_t *events,
IN uint32_t events_size,
OUT uint32_t *num_events)
3.13.6.2 Parameters
err_handle
events
The pointer to an array of events structures that will be filled in on a
successful return. This pointer must be a valid memory location since
the events will be copied from the EEQ.
events_size
The size of the array passed in from the events pointer.
timeout
After first event is triggered, time to wait for subsequent events.
num_events
The number of events copied into the events buffer.
GNI_RC_SUCCESS
GNI_RC_TIMEOUT
The request timed out and the event array was not filled all the way.
GNI_RC_NOT_DONE
The wait was interrupted by the system.
The events pointer cannot be written to.
S–2446–51
145
3.13.7 SetErrorPtag
The SetErrorPtag function sets the protection tag for an error handler. This is
a privileged operation.
3.13.7.1 Synopsis
gni_return_t GNI_SetErrorPtag(
IN uint8_t ptag)
3.13.7.2 Parameters
err_handle
ptag
The protect tag to set for matching error events.
GNI_RC_SUCCESS
Only the superuser can set ptag to something other than the
communication domain.
3.14 Resource Reporting
3.14.1 GetDeviceType
The GetDeviceType function returns the NIC type of the GNI device.
3.14.1.1 Synopsis
gni_return_t GNI_GetDeviceType (
OUT gni_nic_device_t *dev_type)
3.14.1.2 Parameters
dev_type
146
Device types enumerated by gni_nic_device_t. See
gni_nic_device_t on page 153.
S–2446–51
GNI_RC_SUCCESS
Device type was returned successfully.
GNI device does not exist.
3.14.2 GetDevResInfo
The GetDevResInfo function populates the gni_dev_res_desc_t structure if
a particular resource type res_type is implemented on the NIC specified by device_id,
and res_desc is not null.
Possible resource types are enumerated by gni_dev_res_t. See
gni_dev_res_t on page 152.
3.14.2.1 Synopsis
gni_return_t GNI_GetDevResInfo(
IN gni_dev_res_t res_type,
INOUT gni_dev_res_desc_t *res_desc);
3.14.2.2 Parameters
device_id
NIC device ID. See GetLocalDeviceIds on page 149.
res_type
Query for existence of dev_type. Must be one of those enumerated by
gni_nic_device_t. See gni_nic_device_t on page 153.
res_desc
Points to user created gni_dev_res_desc_t structure. See
gni_dev_res_desc_t on page 169.
GNI_RC_SUCCESS
res_type was returned successfully.
dev_id does not exist, res_type not legal value, or res_desc pointer
value is NULL.
res_type not implemented by the NIC type on running system.
S–2446–51
147
3.14.3 GetJobResInfo
The GetJobResInfo function returns information about the job resource res_type,
for the job with protection tag ptag on the device_id, to the structure pointed to by
res_desc.
Possible resource types are enumerated by gni_job_res_t. See
gni_job_res_desc_t on page 171.
3.14.3.1 Synopsis
gni_return_t GNI_GetJobResInfo(
IN uint8_t ptag,
IN gni_job_res_t res_type,
INOUT gni_job_res_desc_t *res_desc);
3.14.3.2 Parameters
device_id
NIC device ID. See GetLocalDeviceIds on page 149.
ptag
Specifies job.
res_type
Resource type enumerated by gni_job_res_t. See
gni_job_res_t on page 153.
res_desc
Points to user created gni_job_res_desc_t (job resource
description) structure. See gni_job_res_desc_t on page 171.
GNI_RC_SUCCESS
Returned if res_type is implemented on the NIC specified by
device_id, and res_desc is not null. Each field in the res_desc
structure is set as follows:
148
•
The used field is the count of resource elements currently
allocated by the job.
•
The limit field is the maximum number of resource elements
the job is allowed to allocate. The limit field may contain the
reserved value (uint64_t)-1. This value indicates that the
job was not configured with a limit (allocation of the resource
is unlimited).
S–2446–51
dev_id does not exist, res_type not legal value, or res_desc pointer
value is NULL.
res_type not implemented by the NIC type on running system.
3.14.4 GetLocalDeviceIds
The GetLocalDeviceIds function returns an array of local NIC devices.
3.14.4.1 Synopsis
gni_return_t GNI_GetLocalDeviceIds (
IN int len
OUT int *device_id)
3.14.4.2 Parameters
len
Number of entries in device_ids array.
device_ids
Pointer to array of local NIC devices.
GNI_RC_SUCCESS
Number of devices was returned successfully.
Cray network application-specific integrated circuit (ASIC) support
missing from kernel.
3.14.5 GetNumLocalDevices
The GetNumLocalDevices function returns the number of NIC devices.
3.14.5.1 Synopsis
gni_return_t GNI_GetNumLocalDevices
OUT int *num_devices)
S–2446–51
(
149
3.14.5.2 Parameters
num_devices
Number of NICs on node.
GNI_RC_SUCCESS
Number of devices was returned successfully.
Cray network application-specific integrated circuit (ASIC) support
missing from kernel.
3.14.6 GetNicStat
The GetNicStat function returns the value of the statistic counter specified by stat.
3.14.6.1 Synopsis
gni_return_t GNI_GetNicStat (
IN gni_statistic_t stat,
OUT uint32_t *value);
)
3.14.6.2 Parameters
nic_hndl
NIC handle (returned by CdmAttach).
stat
NIC statistic to get.
value
Value of the specified NIC statistic counter.
GNI_RC_SUCCESS
One or more of the parameters invalid.
150
S–2446–51
3.14.7 ResetNicStat
The ResetNicStat function sets the value of the statistic counter specified by
stat to zero.
3.14.7.1 Synopsis
gni_return_t GNI_ResetNicStat (
IN gni_statistic_t stat
);
)
3.14.7.2 Parameters
nic_hndl
NIC handle (returned by CdmAttach).
stat
NIC statistic to reset.
GNI_RC_SUCCESS
One or more of the parameters invalid.
3.14.8 GetPtag
The GetPtag function returns the ptag associated with the cookie for the GNI
device identified by device_id.
3.14.8.1 Synopsis
gni_return_t GNI_GetPtag(
IN uint32_t cookie,
IN uint8_t ptag);
3.14.8.2 Parameters
S–2446–51
device_id
The ID of the GNI device to query.
cookie
The cookie associated with ptag.
ptag
The ptag associated with the cookie.
151
GNI_RC_SUCCESS
Operation completed successfully
GNI_RC_NO_MATCH
Could not find associated ptag or device_id is invalid.
A resource allocation error occurred while trying to configure job
resources.
3.15 Enumerations
3.15.1 gni_dev_res_t
3.15.1.1 Synopsis
typedef enum gni_dev_res {
GNI_DEV_RES_FIRST = 0,
GNI_DEV_RES_MDD,
GNI_DEV_RES_MRT,
GNI_DEV_RES_CQ,
GNI_DEV_RES_FMA,
GNI_DEV_RES_CE,
GNI_DEV_RES_DLA,
GNI_DEV_RES_LAST
} gni_dev_res_t;
152
S–2446–51
3.15.2 gni_job_res_t
3.15.2.1 Synopsis
typedef enum gni_job_res {
GNI_JOB_RES_FIRST = 0,
GNI_JOB_RES_MDD,
GNI_JOB_RES_MRT,
GNI_JOB_RES_IOMMU,
GNI_JOB_RES_GART,
GNI_JOB_RES_CQ,
GNI_JOB_RES_FMA,
GNI_JOB_RES_RMDA,
GNI_JOB_RES_CE,
GNI_JOB_RES_DLA,
GNI_JOB_RES_LAST
} gni_job_res_t;
3.15.3 gni_nic_device_t
3.15.3.1 Synopsis
typedef enum gni_nic_device {
GNI_DEVICE_GEMINI,
GNI_DEVICE_ARIES,
GNI_DEVICE_PISCES,
GNI_DEVICE_LAST
} gni_nic_device_t;
3.15.4 gni_fma_cmd_type
The gni_fma_cmd_type enumeration defines the various FMA AMO requests.
The amo_cmd member of the gni_post_descriptor structure holds the FMA
AMO request. The amo_cmd member is processed by the GNI_PostFma function.
First generation atomic memory operations contain the string _ATOMIC_
and will run on both Gemini and Aries systems. GNI_FMA_ATOMIC_AX
and GNI_FMA_ATOMIC_AX_C are not supported and will return
GNI_RC_INVALID_PARAM.
Second generation commands contain the string _ATOMIC2_ and run on Aries
systems; they will generate an error if run on Gemini systems.
3.15.4.1 Synopsis
typedef enum gni_fma_cmd_type {
/************ AMOs with GET semantics **************/
GNI_FMA_ATOMIC_FADD
= 0x008,
/* atomic FETCH and ADD */
GNI_FMA_ATOMIC_FADD_C = 0x018,
/* cached atomic FETCH and ADD */
GNI_FMA_ATOMIC_FAND
= 0x009,
/* atomic FETCH and AND */
GNI_FMA_ATOMIC_FAND_C = 0x019,
/* cached atomic FETCH and AND */
GNI_FMA_ATOMIC_FOR
= 0x00A,
/* atomic FETCH and OR */
S–2446–51
153
GNI_FMA_ATOMIC_FOR_C
GNI_FMA_ATOMIC_FXOR
GNI_FMA_ATOMIC_FXOR_C
GNI_FMA_ATOMIC_FAX
GNI_FMA_ATOMIC_FAX_C
GNI_FMA_ATOMIC_CSWAP
GNI_FMA_ATOMIC_CSWAP_C
=
=
=
=
=
=
=
0x01A,
0x00B,
0x01B,
0x00C,
0x01C,
0x00D,
0x01D,
/* Second generation commands ( GET
GNI_FMA_ATOMIC2_FAND_S
= 0x240,
GNI_FMA_ATOMIC2_FAND
= 0x041,
GNI_FMA_ATOMIC2_FAND_SC
= 0x260,
GNI_FMA_ATOMIC2_FAND_C
= 0x061,
GNI_FMA_ATOMIC2_FOR_S
= 0x242,
GNI_FMA_ATOMIC2_FOR
= 0x043,
GNI_FMA_ATOMIC2_FOR_SC
= 0x262,
GNI_FMA_ATOMIC2_FOR_C
= 0x063,
GNI_FMA_ATOMIC2_FXOR_S
= 0x244,
GNI_FMA_ATOMIC2_FXOR
= 0x045,
GNI_FMA_ATOMIC2_FXOR_SC
= 0x264,
GNI_FMA_ATOMIC2_FXOR_C
= 0x065,
GNI_FMA_ATOMIC2_FSWAP_S
= 0x246,
GNI_FMA_ATOMIC2_FSWAP
= 0x047,
GNI_FMA_ATOMIC2_FSWAP_SC = 0x266,
GNI_FMA_ATOMIC2_FSWAP_C
= 0x067,
GNI_FMA_ATOMIC2_FAX_S
= 0x248,
GNI_FMA_ATOMIC2_FAX
= 0x049,
GNI_FMA_ATOMIC2_FAX_SC
= 0x268,
GNI_FMA_ATOMIC2_FAX_C
= 0x069,
GNI_FMA_ATOMIC2_FCSWAP_S = 0x24A,
GNI_FMA_ATOMIC2_FCSWAP
= 0x04B,
GNI_FMA_ATOMIC2_FCSWAP_SC = 0x26A,
GNI_FMA_ATOMIC2_FCSWAP_C = 0x06B,
GNI_FMA_ATOMIC2_FIMIN_S
= 0x250,
GNI_FMA_ATOMIC2_FIMIN
GNI_FMA_ATOMIC2_FIMIN_SC
= 0x051,
= 0x270,
GNI_FMA_ATOMIC2_FIMIN_C
GNI_FMA_ATOMIC2_FIMAX_S
= 0x071,
= 0x252,
GNI_FMA_ATOMIC2_FIMAX
GNI_FMA_ATOMIC2_FIMAX_SC
= 0x053,
= 0x272,
GNI_FMA_ATOMIC2_FIMAX_C
GNI_FMA_ATOMIC2_FIADD_S
= 0x073,
= 0x254,
GNI_FMA_ATOMIC2_FIADD
GNI_FMA_ATOMIC2_FIADD_SC
= 0x055,
= 0x274,
GNI_FMA_ATOMIC2_FIADD_C
GNI_FMA_ATOMIC2_FFPMIN_S
= 0x075,
= 0x258,
GNI_FMA_ATOMIC2_FFPMIN
= 0x059,
GNI_FMA_ATOMIC2_FFPMIN_SC = 0x278,
GNI_FMA_ATOMIC2_FFPMIN_C
GNI_FMA_ATOMIC2_FFPMAX_S
= 0x079,
= 0x25A,
GNI_FMA_ATOMIC2_FFPMAX
= 0x05B,
GNI_FMA_ATOMIC2_FFPMAX_SC = 0x27A,
GNI_FMA_ATOMIC2_FFPMAX_C
GNI_FMA_ATOMIC2_FFPADD_S
154
= 0x07B,
= 0x25C,
/*
/*
/*
/*
/*
/*
/*
cached
atomic
cached
atomic
cached
atomic
cached
atomic FETCH and OR */
FETCH and XOR */
atomic FETCH and XOR */
FETCH AND exclusive OR */
atomic FETCH AND exclusive OR */
COMPARE and SWAP */
atomic COMPARE and SWAP */
semantics )
/* atomic FETCH logical AND (32-bit operands) */
/* atomic FETCH and AND */
/* cached atomic FETCH logical AND (32-bit operands) */
/* cached atomic FETCH and AND */
/* atomic FETCH logical OR (32-bit operands) */
/* atomic FETCH and OR */
/* cached atomic FETCH logical OR (32-bit operands) */
/* cached atomic FETCH and OR */
/* atomic FETCH logical Exclusive OR (32-bit operands) */
/* atomic FETCH exclusive OR */
/* cached atomic FETCH logical Exclusive OR (32-bit operands) */
/* cached atomic FETCH exclusive OR */
/* atomic FETCH Swap (32-bit operands) */
/* atomic FETCH and SWAP */
/* cached atomic FETCH Swap (32-bit operands) */
/* cached atomic FETCH and SWAP */
/* atomic FETCH logical AND Exclusive OR (32-bit operands) */
/* atomic FETCH AND exclusive OR */
/* cached atomic FETCH logical AND Exclusive OR (32-bit operands) */
/* cached atomic FETCH AND exclusive OR */
/* atomic FETCH Compare and Swap (32-bit operands) */
/* atomic FETCH COMPARE and SWAP */
/* cached atomic FETCH Compare and Swap (32-bit operands) */
/* cached atomic FETCH COMPARE and SWAP */
/* atomic FETCH integer signed twos complement Min
(32-bit operands) */
/* atomic FETCH integer signed two's complement Min */
/* cached atomic FETCH int signed twos complement Min
/* cached atomic FETCH integer signed two's complement Min */
/* atomic FETCH integer signed twos complement Max
/* atomic FETCH integer signed two's complement Max */
/* cached atomic FETCH int signed twos complement Max
/* cached atomic FETCH integer signed two's complement Max */
/* atomic FETCH integer twos complement Addition
/* atomic FETCH integer two's complement Addition */
/* cached atomic FETCH integer twos complement Addition
/* cached atomic FETCH integer two's complement Addition */
/* atomic FETCH floating point Min (single precision)
/* atomic FETCH floating point Min (double precision) */
/* cached atomic FETCH floating point Min (single precision)
(32-bit ) */
/* cached atomic FETCH floating point Min (double precision) */
/* atomic FETCH floating point Max (single precision)
/* atomic FETCH floating point Max (double precision) */
/* cached atomic FETCH floating point Max (single precision)
(32-bit) */
/* cached atomic FETCH floating point Max (double precision) */
/* atomic FETCH floating point Addition (single precision)
S–2446–51
(32-bit) */
GNI_FMA_ATOMIC2_FFPADD
= 0x05D, /* atomic Fetching floating point Addition (double precision) */
GNI_FMA_ATOMIC2_FFPADD_SC = 0x27C, /* cached atomic FETCH floating point Addition (single precision)
(32-bit) */
GNI_FMA_ATOMIC2_FFPADD_C = 0x07D, /* cached atomic Fetching floating point Addition
(double precision) */
/************ AMOs with PUT semantics ***************/
GNI_FMA_ATOMIC_ADD
= 0x108,
/* atomic ADD */
GNI_FMA_ATOMIC_ADD_C
= 0x118,
/* cached atomic ADD */
GNI_FMA_ATOMIC_AND
= 0x109,
/* atomic AND */
GNI_FMA_ATOMIC_AND_C
= 0x119,
/* cached atomic AND */
GNI_FMA_ATOMIC_OR
= 0x10A,
/* atomic OR */
GNI_FMA_ATOMIC_OR_C
= 0x11A,
/* cached atomic OR */
GNI_FMA_ATOMIC_XOR
= 0x10B,
/* atomic exclusive OR */
GNI_FMA_ATOMIC_XOR_C
= 0x11B,
/* cached atomic exclusive OR */
GNI_FMA_ATOMIC_AX
= 0x10C,
/* atomic AND exclusive OR */
GNI_FMA_ATOMIC_AX_C
= 0x11C,
/* cached atomic AND exclusive OR */
/* Second generation commands ( PUT sematics ) */
GNI_FMA_ATOMIC2_AND_S
= 0x340, /* atomic AND (32-bit operands) */
GNI_FMA_ATOMIC2_AND
= 0x141, /* atomic AND */
GNI_FMA_ATOMIC2_AND_SC
= 0x360, /* cached atomic AND (32-bit operands) */
GNI_FMA_ATOMIC2_AND_C
= 0x161, /* cached atomic AND */
GNI_FMA_ATOMIC2_OR_S
= 0x342, /* atomic OR (32-bit operands) */
GNI_FMA_ATOMIC2_OR
= 0x143, /* atomic OR */
GNI_FMA_ATOMIC2_OR_SC
= 0x362, /* cached atomic OR (32-bit operands) */
GNI_FMA_ATOMIC2_OR_C
= 0x163, /* cached atomic OR */
GNI_FMA_ATOMIC2_XOR_S
= 0x344, /* atomic Exclusive OR (32-bit operands) */
GNI_FMA_ATOMIC2_XOR
= 0x145, /* atomic exclusive OR */
GNI_FMA_ATOMIC2_XOR_SC
= 0x364, /* cached atomic Exclusive OR (32-bit operands) */
GNI_FMA_ATOMIC2_XOR_C
= 0x165, /* cached atomic exclusive OR */
GNI_FMA_ATOMIC2_SWAP_S
= 0x346, /* atomic Swap (Store) (32-bit operands) */
GNI_FMA_ATOMIC2_SWAP
= 0x147, /* atomic SWAP */
GNI_FMA_ATOMIC2_SWAP_SC = 0x366, /* cached atomic Swap (Store) (32-bit operands) */
GNI_FMA_ATOMIC2_SWAP_C
= 0x167, /* cached atomic SWAP */
GNI_FMA_ATOMIC2_AX_S
= 0x348, /* atomic AND Exclusive OR (32-bit operands),
not valid for FMA_LAUNCH */
GNI_FMA_ATOMIC2_AX
= 0x149, /* atomic AND exclusive OR */
GNI_FMA_ATOMIC2_AX_SC
= 0x368, /* cached atomic AND Exclusive OR (32-bit),
GNI_FMA_ATOMIC2_AX_C
= 0x169, /* cached atomic AND exclusive OR */
GNI_FMA_ATOMIC2_CSWAP_S = 0x34A, /* atomic Compare and Swap (Conditional Store) (32-bit),
GNI_FMA_ATOMIC2_CSWAP
= 0x14B, /* atomic COMPARE and SWAP */
GNI_FMA_ATOMIC2_CSWAP_SC = 0x36A, /* cached atomic Compare and Swap (Conditional Store) (32-bit),
GNI_FMA_ATOMIC2_CSWAP_C = 0x16B, /* cached atomic COMPARE and SWAP */
GNI_FMA_ATOMIC2_IMIN_S
= 0x350, /* atomic integer signed twos complement Min (32-bit operands) */
GNI_FMA_ATOMIC2_IMIN
= 0x151, /* atomic integer signed two's complement Min */
GNI_FMA_ATOMIC2_IMIN_SC = 0x370, /* cached atomic integer signed twos complement Min
GNI_FMA_ATOMIC2_IMIN_C
= 0x171, /* cached atomic integer signed two's complement Min */
GNI_FMA_ATOMIC2_IMAX_S
= 0x352, /* atomic integer signed twos complement Max (32-bit operands) */
GNI_FMA_ATOMIC2_IMAX
= 0x153, /* atomic integer signed two's complement Max */
GNI_FMA_ATOMIC2_IMAX_SC = 0x372, /* cached atomic integer signed twos complement Max
GNI_FMA_ATOMIC2_IMAX_C
= 0x173, /* cached atomic integer signed two's complement Max */
GNI_FMA_ATOMIC2_IADD_S
= 0x354, /* atomic integer twos complement Addition (32-bit operands) */
GNI_FMA_ATOMIC2_IADD
= 0x155, /* atomic integer two's complement Addition */
GNI_FMA_ATOMIC2_IADD_SC = 0x374, /* cached atomic integer twos complement Addition
GNI_FMA_ATOMIC2_IADD_C
= 0x175, /* cached atomic integer two's complement Addition */
GNI_FMA_ATOMIC2_FPMIN_S = 0x358, /* atomic floating point Min (single precision)
GNI_FMA_ATOMIC2_FPMIN
= 0x159, /* atomic floating point Min (double precision) */
S–2446–51
155
GNI_FMA_ATOMIC2_FPMIN_SC = 0x378,
GNI_FMA_ATOMIC2_FPMIN_C
GNI_FMA_ATOMIC2_FPMAX_S
GNI_FMA_ATOMIC2_FPMAX
GNI_FMA_ATOMIC2_FPMAX_SC
=
=
=
=
0x179,
0x35A,
0x15B,
0x37A,
GNI_FMA_ATOMIC2_FPMAX_C
GNI_FMA_ATOMIC2_FPADD_S
= 0x17B,
= 0x35C,
GNI_FMA_ATOMIC2_FPADD
= 0x15D,
GNI_FMA_ATOMIC2_FPADD_SC = 0x37C,
GNI_FMA_ATOMIC2_FPADD_C
} gni_fma_cmd_type_t;
= 0x17D,
/* cached atomic floating point Min (single precision)
/* cached atomic floating point Min (double precision) */
/* atomic floating point Max (single precision) (32-bit operands) */
/* atomic floating point Max (double precision) */
/* cached atomic floating point Max (single precision)
/* cached atomic floating point Max (double precision) */
/* atomic floating point Addition (single precision)
/* atomic floating point Addition (double precision) */
/* cached atomic floating point Addition (single precision)
(32-bit) */
/* cached atomic floating point Addition (double precision) */
3.15.5 gni_ce_cmd_type
The gni_ce_cmd_type enumeration defines the various collective engine (CE)
requests. The ce_cmd member of the gni_post_descriptor structure holds
the CE request. The ce_cmd member is processed by the GNI_PostFma function.
Commands contain the string _CE_ and run on Aries systems; they will generate an
error if run on Gemini systems.
3.15.5.1 Synopsis
typedef enum gni_ce_cmd_type {
GNI_FMA_CE_AND_S
= 0x0ull,
GNI_FMA_CE_AND
= 0x1ull,
GNI_FMA_CE_OR_S
= 0x2ull,
GNI_FMA_CE_OR
= 0x3ull,
GNI_FMA_CE_XOR_S
= 0x4ull,
GNI_FMA_CE_XOR
= 0x5ull,
GNI_FMA_CE_IMIN_LIDX_S = 0x10ull,
GNI_FMA_CE_IMIN_LIDX
= 0x11ull,
GNI_FMA_CE_IMAX_LIDX_S = 0x12ull,
GNI_FMA_CE_IMAX_LIDX
= 0x13ull,
GNI_FMA_CE_IADD_S
= 0x14ull,
GNI_FMA_CE_IADD
= 0x15ull,
GNI_FMA_CE_FPMIN_LIDX_S = 0x18ull,
GNI_FMA_CE_FPMIN_LIDX
= 0x19ull,
GNI_FMA_CE_FPMAX_LIDX_S = 0x1aull,
GNI_FMA_CE_FPMAX_LIDX
= 0x1bull,
GNI_FMA_CE_FPADD_S
= 0x1cull,
GNI_FMA_CE_FPADD
= 0x1dull,
GNI_FMA_CE_IMIN_GIDX_S = 0x30ull,
GNI_FMA_CE_IMIN_GIDX
= 0x31ull,
GNI_FMA_CE_IMAX_GIDX_S = 0x32ull,
GNI_FMA_CE_IMAX_GIDX
= 0x33ull,
GNI_FMA_CE_FPMIN_GIDX_S = 0x38ull,
GNI_FMA_CE_FPMIN_GIDX
= 0x39ull,
GNI_FMA_CE_FPMAX_GIDX_S = 0x3aull,
GNI_FMA_CE_FPMAX_GIDX
= 0x3bull,
} gni_ce_cmd_type_t;
156
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
/*
Logical AND, short */
Logical AND */
Logical OR, short */
Logical OR */
Logical XOR, short */
Logical XOR */
Integer signed two's complement min, short (lowest index returned) */
Integer signed two's complement min (lowest index returned) */
Integer signed two's complement max, short (lowest index returned) */
Integer signed two's complement max (lowest index returned) */
Integer two's complement ADD, short */
Integer two's complement ADD */
Floating point minimum, short (lowest index returned) */
Floating point minimum (lowest index returned) */
Floating point maximum, short (lowest index returned) */
Floating point maximum (lowest index returned) */
Floating point ADD, short */
Floating point ADD */
Integer signed two's complement min, short (greatest index returned) */
Integer signed two's complement min (greatest index returned) */
Integer signed two's complement max, short (greatest index returned) */
Integer signed two's complement max (greatest index returned) */
Floating point minimum, short (greatest index returned) */
Floating point minimum (greatest index returned) */
Floating point maximum, short (greatest index returned) */
Floating point maximum (greatest index returned) */
S–2446–51
3.15.6 gni_mem_handle_attr
The gni_mem_handle_attr enumeration defines memory mapping mechanism
used by a memory segment associated with a particular memory handle. The
MemHndlQueryAttr function tests a memory handle for a specific memory handle
attribute.
3.15.6.1 Synopsis
typedef enum gni_mem_handle_attr {
GNI_MEMHNDL_ATTR_READONLY = 1,
GNI_MEMHNDL_ATTR_VMDH,
GNI_MEMHNDL_ATTR_MRT,
GNI_MEMHNDL_ATTR_GART,
GNI_MEMHHDL_ATTR_IOMMU,
GNI_MEMHHDL_ATTR_PCI_IOMMU,
GNI_MEMHHDL_ATTR_CLONE,
} gni_mem_handle_attr_t;
3.15.6.2 Constants
GNI_MEMHNDL_ATTR_READONLY
Initialized to 1.
GNI_MEMHNDL_ATTR_VMDH
Indicates that the memory handle is a virtual domain handle. See
Virtual Memory Domain Handles on page 95.
GNI_MEMHNDL_ATTR_MRT
Indicates that the registered memory segment identified by the handle
is mapped to local memory using the MRT.
GNI_MEMHNDL_ATTR_GART
Indicates that the memory segment identified by the handle is
mapped to local memory using the GART.
GNI_MEMHNDL_ATTR_IOMMU
mapped to local memory using the I/O memory management unit.
GNI_MEMHNDL_ATTR_PCI_IOMMU
mapped to local memory using the PCI I/O memory management
unit.
GNI_MEMHNDL_ATTR_CLONE
Indicates that the memory handle is a clone.
S–2446–51
157
3.15.7 gni_post_state
The gni_post_state enumeration defines the flags for the post state of
datagram transactions between the endpoints on a local and a remote peer that are
in the same communication domain. A pointer to the post state is returned by the
EpPostDataTest, EpPostDataWait, EpPostDataWaitByIdfunctions
when testing the success of an EpPostData operation.
3.15.7.1 Synopsis
typedef enum gni_post_state{
GNI_POST_PENDING,
GNI_POST_COMPLETED,
GNI_POST_ERROR,
GNI_POST_TIMEOUT,
GNI_POST_TERMINATED,
} gni_post_state_t;
3.15.7.2 Constants
GNI_POST_PENDING
Indicates the post is pending.
GNI_POST_COMPLETED
Indicates that the data exchange completed successfully.
GNI_POST_ERROR
Indicates the post did not complete due to an error.
GNI_POST_TIMEOUT
Indicates the post did not complete and timed out.
GNI_POST_TERMINATED
Indicates the post did not complete because it was terminated.
Indicates receipt of the remote data, but the remote peer did not
acknowledge getting the data from the local side.
3.15.8 gni_post_type
The gni_post_type enumeration defines the values to use for the post
transaction. The constant values for this enumeration are used by the type member
of the gni_post_descriptor structure, which is used by the GNI_PostRdma,
GNI_PostFma, GNI_PostCqWrite, and GNI_GetCompleted functions.
158
S–2446–51
Remote FMA events are automatically subscribed to when the
GNI_POST_FMA_PUT_W_SYNCFLAG type option is used. This is
similar to a FMA PUT, however, in addition to sending the PUT data, a user
specified 8-byte flag is delivered to a user specified offset of the target memory
when the operation is complete. The flag data and offset values are fields of the
gni_post_descriptor_t. The flag is delivered into local memory at the
receiver after all other data is sent, prior to the delivery of the CQE to the receiver's
CQ. If an error occurred for any of the data that was sent, the flag is not written; the
error is reported in the CQE. The remote event is delivered after the data and the flag
have been written into the target host's memory. The local endpoint also receives an
event after the data and the flag were written into the target host memory. So, the
local EP and the remote EP receive an event after both data and the flag are written
into the target host memory. Both events triggered by the same condition, but we
cannot predict order between those events as they get delivered at different physical
nodes.
3.15.8.1 Synopsis
typedef enum gni_post_type {
GNI_POST_RDMA_PUT = 1,
GNI_POST_RDMA_GET,
GNI_POST_FMA_PUT,
GNI_POST_FMA_PUT_W_SYNCFLAG,
GNI_POST_FMA_GET,
GNI_POST_AMO,
GNI_POST_CQWRITE,
GNI_POST_CE,
GNI_POST_FMA_GET_W_FLAG,
GNI_POST_AMO_W_FLAG,
} gni_post_type_t;
3.15.8.2 Constants
GNI_POST_RDMA_PUT
Indicates an RDMA PUT transaction.
GNI_POST_RDMA_GET
Indicates an RDMA GET transaction.
GNI_POST_FMA_PUT
Indicates an FMA PUT transaction.
GNI_POST_FMA_PUT_W_SYNCFLAG
Indicates an FMA PUT transaction with a synchronization flag.
Remote events are automatically subscribed to with the use of
this flag. Similar to a regular FMA PUT. However, in addition to
sending the PUT data, a user specified 8-byte flag is delivered to
a user specified offset of the target memory when the operation
S–2446–51
159
is complete. The flag data and offset values are fields of the
gni_post_descriptor_t. The flag is delivered into local
memory at the receiver after all other data sent, prior to delivery of
the CQE to the receiver's CQ. If an error occurred for any of the data
that was sent in the SSID sequence, the flag is not written; the error is
reported in the CQE.
GNI_POST_FMA_GET
Indicates an FMA GET transaction.
GNI_POST_AMO
Indicates an AMO transaction.
GNI_POST_CQWRITE
GNI_POST_CE
Indicates a collective transaction posted to collective engine (CE).
GNI_POST_FMA_GET_W_FLAG
GNI_POST_AMO_W_FLAG
3.15.9 gni_return
The gni_return enumeration defines the values to use for return values.
3.15.9.1 Synopsis
typedef enum gni_return {
GNI_RC_SUCCESS = 0,
GNI_RC_NOT_DONE,
GNI_RC_INVALID_PARAM,
GNI_RC_ERROR_RESOURCE,
GNI_RC_TIMEOUT,
GNI_RC_PERMISSION_ERROR,
GNI_RC_DESCRIPTOR_ERROR,
GNI_RC_ALIGNMENT_ERROR,
GNI_RC_INVALID_STATE,
GNI_RC_NO_MATCH,
GNI_RC_SIZE_ERROR,
GNI_RC_TRANSACTION_ERROR,
GNI_RC_ILLEGAL_OP,
GNI_RC_ERROR_NOMEM
} gni_return_t;
3.15.9.2 Constants
GNI_RC_SUCCESS
The operation was successful.
160
S–2446–51
GNI_RC_NOT_DONE
The operation is not permitted.
Typically, this error means that sufficient resources or the correct
resources are not available to complete the operation.
GNI_RC_TIMEOUT
The request timed out.
The process does not have the correct permissions to complete the
operation.
GNI_RC_DESCRIPTOR_ERROR
If the corresponding post queue (FMA, RDMA or AMO) is empty,
the descriptor pointer is set to NULL, otherwise, a completed
descriptor is returned with an error completion status.
Posted source or destination data pointers or data length are not
properly aligned.
The caller attempted to attach a communication domain instance to
the NIC device more than once.
GNI_RC_NO_MATCH
The requested item and available items do not coincide.
GNI_RC_SIZE_ERROR
The supplied buffer is too small to contain the error code.
Error in processing post data transaction.
GNI_RC_ILLEGAL_OP
The operation being attempted is illegal.
S–2446–51
161
GNI_RC_ERROR_NOMEM
3.15.10 gni_smsg_type
The gni_smsg_type enumeration defines the values to use for the short messaging
type. The constant values for this enumeration are used in the msg_type member of
the gni_smsg_attr structure.
3.15.10.1 Synopsis
typedef enum gni_smsg_type {
GNI_SMSG_TYPE_INVALID = 0,
GNI_SMSG_TYPE_MBOX,
GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT
} gni_smsg_type_t;
3.15.10.2 Constants
GNI_SMSG_TYPE_INVALID
Indicates that the short message type is invalid.
GNI_SMSG_TYPE_MBOX
Indicates the MBOX short messaging type.
GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT
Indicates that the system attempts to retransmit the message for
certain transaction failures.
3.15.11 gni_statistic
The gni_statistic enumeration defines the types of statistics that can be
extracted from the NIC.
3.15.11.1 Synopsis
Only Gemini related statistics are listed here.
typedef enum gni_statistic {
GNI_STAT_SMSG_BUFF_CREDITS_STALL = 0,
GNI_STAT_SMSG_DLA_STALL,
GNI_STAT_SMSG_MBOX_CREDITS_STALL,
GNI_STAT_SMSG_REQ_STALL,
GNI_STAT_SMSG_RETRANS_COUNT,
GNI_STAT_SMSG_RETRANS_DLA_COUNT,
GNI_STAT_SMSG_RETRANS_STALL,
GNI_NUM_STATS
} gni_statistic_t;
162
S–2446–51
3.15.11.2 Constants
GNI_STAT_SMSG_BUFF_CREDITS_STALL
GNI_STAT_SMSG_DLA_STALL
GNI_STAT_SMSG_MBOX_CREDITS_STALL
GNI_STAT_SMSG_REQ_STALL
GNI_STAT_SMSG_RETRANS_COUNT
GNI_STAT_SMSG_RETRANS_DLA_COUNT
GNI_STAT_SMSG_RETRANS_STALL
3.16 Structures
3.16.1 gni_bi_desc
Contains information about a node's balanced injection (BI) setting.
3.16.1.1 Synopsis
typedef struct gni_bi_desc {
uint16_t current_bw;
uint16_t current_aot_bw;
uint16_t current_norbs;
uint16_t flags;
uint16_t sys_def_bw;
uint16_t sys_def_aot_bw;
uint16_t cle_seqnum;
uint16_t hss_seqnum;
} gni_bi_desc_t;
S–2446–51
163
3.16.1.2 Members
current_bw Identifies current balanced injection (BI) setting. Must be an integer
between 0 and 100. Mapped to some number of outstanding request
buffers (ORB).
current_aot_bw
Identifies the current apply-on-throttle bandwidth. This is the
bandwidth setting which would be applied after the first instance
of congestion protection. Must be an integer between 0 and 100.
Mapped to some number of outstanding request buffer (ORB) entries
by HSS algorithm.
current_norbs
Current number of ORB entries configured by HSS.
flags
Only valid setting is GNI_BI_FLAG_USE_DEFAULT_SETTINGS.
sys_def_bw System default BI value.
sys_def_aot_bw
System default apply-on-throttle value. Applied after first instance
of congestion protection.
cle_seqnum Sequence number of the last BI update.
hss_seqnum Sequence number of the last BI update read by HSS.
3.16.2 gni_error_event
3.16.2.1 Synopsis
typedef struct gni_error_event {
uint16_t error_code;
uint8_t error_category;
uint8_t ptag;
uint32_t serial_number;
uint64_t timestamp;
uint64_t info_mmrs[4];
} gni_error_event_t;
3.16.2.2 Members
error_code Identifies the error which caused the event. Used by GNI for problem
reporting. Codes will not be interpreted by uGNI user.
error_category
Errors are divided into 6 categories:
164
S–2446–51
•
CRITICAL_ERR
Caused by uncorrectable memory errors, an invalid hardware
configuration, or other hardware issues. In most cases, future use
of the NIC is unreliable and a node reboot may be required.
•
TRANSACTION_ERR
Caused by errors in a specific transaction sequence, likely due to
a software issue. A node reboot is not required.
•
ADDR_TRANS_ERR
There were errors in the node address translation and/or memory
address translation for a specific transaction. A node reboot is
not required.
•
TRANSIENT_ERR
There may be transient issues with network, memory, or resource
availability (for example, no free descriptors). Software should
often be able to recover from these errors by reissuing the
transaction.
•
CORRECTABLE_MEM_ERR
Benign from a system perspective, but should be monitored by
HSS and accounted for.
•
INFO_ERR
An event occurred which is not necessarily an error condition.
ptag
PTag responsible for error, when applicable.
serial_number
This is a semi-unique identifier for the error. An application can use
this to match errors entered into the HSS logs. However, some OS
errors come outside the normal error reporting path, so they will have
a zero for a serial number.
timestamp
Time the error was reported.
info_mmrs
Some errors gather additional information from other registers in the
hardware which may be useful information in problem reports. Not
used by the uGNI user.
3.16.3 gni_error_mask
The mask value can be a bitwise OR of the error categories as defined by the
ERRMASK flags found in gni_pub.h.
S–2446–51
165
3.16.3.1 Synopsis
typedef uint8_t gni_error_mask_t;
#define
#define
#define
#define
#define
#define
#define
GNI_ERRMASK_CORRECTABLE_MEMORY
GNI_ERRMASK_CRITICAL
GNI_ERRMASK_TRANSACTION
GNI_ERRMASK_ADDRESS_TRANSLATION
GNI_ERRMASK_TRANSIENT
GNI_ERRMASK_INFORMATIONAL
GNI_ERRMASK_DIAG_ONLY
(1
(1
(1
(1
(1
(1
(1
<<
<<
<<
<<
<<
<<
<<
0)
1)
2)
3)
4)
5)
6)
3.16.4 gni_cq_entry
The completion queue entry (CQE) contains event type, status, and user data
components which may be extracted using macros, which operate on receive CQs
associated with registered memory. See Create Completion Queue (CQ) on page 45.
3.16.4.1 Synopsis
typedef uint64_t gni_cq_entry_t;
3.16.4.2 Event types
uGNI sets the event type regardless of the result of the transaction. The
GNI_CQ_GET_TYPE() macro returns the type of transaction associated with the
provided completion event. The event type is only set for local CQs. See The valid
event types defined in gni_pub.h are:
#define
#define
#define
#define
GNI_CQ_EVENT_TYPE_POST
GNI_CQ_EVENT_TYPE_SMSG
GNI_CQ_EVENT_TYPE_DMAPP
GNI_CQ_EVENT_TYPE_MSGQ
0x0ULL
0x1ULL
0x2ULL
0x3ULL
3.16.4.3 Macros which Extract User Data From CQ Entry
The user data component of a CQ event can be extracted using either the
GNI_CQ_GET_INST_ID() or GNI_CQ_GET_MSG_ID() macro, which are
functionally equivalent. GNI_CQ_GET_INST_ID() allows a user to retrieve the
instance ID (inst_id) field from an event dequeued from a source completion queue
(attached to a GNI endpoint). The instance ID field of a source completion queue
event is 24 bits in length on a Gemini system and 32 bits in length on an Aries
system.
GNI_CQ_GET_REM_INST_ID() allows a user to retrieve the instance ID field
from an event dequeued from a destination completion queue (attached to a GNI
registered memory). The instance ID field of a destination completion queue event is
32 bits in length.
166
S–2446–51
The user data component in a local CQ entry is set depending on the event type as
follows:
•
GNI_CQ_EVENT_TYPE_POST
Contains data value from the associated local endpoint. Defaults to the
instance ID of receiver's CDM, but can be changed per endpoint through the
GNI_EpSetEventData interface.
•
GNI_CQ_EVENT_TYPE_SMSG or GNI_CQ_EVENT_TYPE_MSGQ
If the transaction was successful, contains the message ID for that SMSG/MSGQ
transaction.
If the transaction failed, contains the data value from the associated local
endpoint. This value defaults to the instance ID of receiver's CDM, but can be
changed per endpoint through the GNI_EpSetEventData interface.
The user data component in a receive CQ entry defaults to the instance ID of the
sender's CDM, regardless of the transaction type or result; it can be extracted using
GNI_CQ_GET_REM_INST_ID(). This value can be changed per endpoint through
the EpSetEventData() interface.
3.16.4.4 Other CQ Macros
•
GNI_CQ_GET_DATA
Returns the bits associated with the CQ event's data field.
•
GNI_CQ_GET_SOURCE
Returns the bits associated with the CQ event's source field. The source
information identifies the component that created the CQ event.
•
GNI_CQ_GET_STATUS
Returns the bits associated with the CQ event's status field. If these bits are zero,
then this CQ event has not encountered any errors. Otherwise, it will contain
information about the error encountered.
•
GNI_CQ_GET_INFO
Returns the bits associated with the CQ event's info field. This information
provides additional information about the CQ event.
•
GNI_CQ_GET_TID
Returns the bits associated with the CQ event's tid field
•
GNI_CQ_GET_MSG_ID
Returns the bits associated with the CQ event's msg_id field. The msg_id
identifies the sender of a SMSG message.
S–2446–51
167
•
GNI_CQ_GET_TYPE
Returns the bits associated with the CQ event's type field. This information
identifies the type of request that caused this CQ event to be created.
•
GNI_CQ_GET_OVERRUN
Returns true if the provided CQ event caused the completion queue to become
full. This indicates that completion events may have been lost.
•
GNI_CQ_REM_OVERRUN
Returns true if the transaction associated with the source CQ event caused the
completion queue associated with the target memory registration to become full.
This indicates that completion events may have been lost.
•
GNI_CQ_GET_BLOCK_ID
Returns the bits associated with the CQ event's block_id field.
•
GNI_CQ_GET_UNSUCCESSFUL_CNT
Returns the bits associated with the CQ event's unsuccessful_cnt field.
•
GNI_CQ_GET_MARKER_ID
Returns the bits associated with the CQ event's marker_id field.
•
GNI_CQ_GET_FAILED_ENQUEUE_CNT
Returns the bits associated with the CQ event's failed_enqueue_cnt field.
This information indicates the number of consecutive BTE requests that have not
been successfully enqueued.
•
GNI_CQ_GET_CE_ID
Returns the bits associated with the CQ event's ce_id field. The ce_id value
identifies the virtual collective engine that generated the CQ event.
•
GNI_CQ_GET_REDUCTN_ID
Returns the bits associated with the CQ event's reductn_id field
•
GNI_CQ_GET_TRANS_TYPE
Returns the bits associated with the CQ event's trans_type field.
•
GNI_CQ_SET_TID(entry,val)
Sets the bits associated with the CQ event's tid field.
•
GNI_CQ_SET_MSG_ID(entry,val)
Sets the bits associated with the CQ event's msg_id field.
•
168
GNI_CQ_SET_TYPE(entry,val)
S–2446–51
Sets the bits associated with the CQ event's type field.
•
GNI_CQ_CLR_STATUS(entry)
Clears the bits associated with the CQ event's status field.
•
GNI_CQ_STATUS_DLA_OVERFLOW(entry)
Tests if the bits associated with the CQ event's status field have the
DLA_OVERFLOW bit set.
•
GNI_CQ_BTE_ENQ_STATUS(entry)
Tests if the bits associated with the CQ event's status field have the BTE
ENQUEUE bit set.
3.16.4.5 Status
The GNI_CQ_STATUS_OK() macro returns true if the transaction associated with
the provided CQ event was successful.
The GNI_CQ_OVERRUN() macro returns true if the provided CQ event caused the
completion queue to become full. This indicates that completion queue events may
have been lost.
3.16.5 gni_dev_res_desc_t
3.16.5.1 Synopsis
typedef struct gni_dev_res_desc {
uint64_t available;
uint64_t reserved;
uint64_t held;
uint64_t total;
} gni_dev_res_desc_t;
3.16.6 gni_job_limits
The gni_job_limits structure defines job parameters and limits. This structure is
used by the GNI_ConfigureJob function.
S–2446–51
169
3.16.6.1 Synopsis
typedef struct gni_job_limits {
int32_t mdd_limit;
union {
int32_t mrt_limit;
struct {
uint8_t ce_limit;
uint8_t res_byte1;
uint8_t res_byte2;
uint8_t res_byte3;
} m;
} a;
union {
int32_t gart_limit;
int32_t dla_limit;
} b;
int32_t fma_limit;
int32_t bte_limit;
int32_t cq_limit;
int32_t ntt_ctrl;
int32_t ntt_base;
int32_t ntt_size;
} gni_job_limits_t;
3.16.6.2 Members
mdd_limit
Number of MDDs associated with the given ptag.
mrt_limit
Gemini: Number of MRT entries used by MDDs with the given ptag.
ce_limit
Aries: Number of CE entries used by MDDs with the given ptag.
gart_limit Gemini: Number of GART entries used by MDDs with the given
ptag.
170
dla_limit
Aries: Number of DLA entries available with the given ptag.
fma_limit
Number of FMA descriptors associated with the given ptag.
bte_limit
Number of outstanding BTE descriptors with the given source ptag.
cq_limit
Number of CQ descriptors associated with the given ptag.
ntt_ctrl
NTT control flag. The only flag that can be used for this parameter is
GNI_JOB_CTRL_NTT_CLEANUP which is a directive for the driver
to cleanup NTT at the end of the job.
ntt_base
Base entry into NTT.
ntt_size
Size of the NTT.
S–2446–51
3.16.7 gni_job_res_desc_t
3.16.7.1 Synopsis
typedef struct gni_job_res_desc {
uint64_t used;
uint64_t limit;
} gni_job_res_desc_t;
3.16.8 gni_mem_segment
The gni_mem_segment structure defines the address and length of a memory
segment. The MemRegisterSegments function uses this structure.
3.16.8.1 Synopsis
typedef struct gni_mem_segment {
uint64_t
address;
uint64_t
length;
} gni_mem_segment_t;
3.16.8.2 Members
address
Address of the segment.
length
Size of the segment in bytes.
3.16.9 gni_ntt_descriptor
The gni_ntt_descriptor structure defines configuration options that can be set
in NTT. This structure is used by the GNI_CdmCreate and GNI_ConfigureNTT
functions.
3.16.9.1 Synopsis
typedef struct gni_ntt_descriptor {
uint32_t
group_size;
uint8_t
granularity;
uint32_t
*table;
uint8_t
flags;
} gni_ntt_descriptor_t;
S–2446–51
171
3.16.9.2 Members
group_size Size of the NTT group to configure.
granularity
NTT granularity.
table
Pointer to the array of new NTT values.
flags
Configuration flags.
3.16.10 gni_post_descriptor
The gni_post_descriptor structure defines the transaction descriptors.
This structure is used by the GetCompleted, PostFMA, PostCqWrite, and
PostRDMA functions.
Set the post_id field in the gni_post_descriptor_t structure to track a
FMA request. Initialize the post_id field to a unique value that can identify this
FMA request. After the posted FMA request has completed, a completion queue
event will be received in the local completion queue. To identify which FMA request
has completed, get the event from the completion queue and complete the event via
the GNI_GetCompleted() API. The GNI_GetCompleted() API will return
a gni_post_descriptor_t structure for the completed FMA request. The
post_id from the completed event will match one of the previously posted FMA
requests. The user should not reuse the post_id from the time that the request is
posted until it is completed by the GNI_GetCompleted call. Otherwise, they will
not be able to identify the completed request.
Set the cq_mode in the gni_post_descriptor to
GNI_CQMODE_GLOBAL_EVENT to ensure that the source FMA buffer is
not reused until after the global completion event has been received from
GNI_CqGetEvent. This setting is required by the Aries NIC. Failure to set
GNI_CQMODE_GLOBAL_EVENT for a PostFMA/PostRDMA transaction will
result in the runtime error GNI_RC_INVALID_PARAM. The user may wish to
retain original source buffer until GLOBAL event completion. If the user sets
cq_mode in the gni_post_descriptor to GNI_CQMODE_GLOBAL_EVENT it
is necessary to ensure that the source buffer is not reused. GNI cannot ensure the
integrity of the user's buffer.
172
S–2446–51
3.16.10.1 Synopsis
typedef struct gni_post_descriptor {
void
*next_descr;
void
*prev_descr;
uint64_t
post_id;
uint64_t
status;
uint16_t
cq_mode_complete;
gni_post_type_t
type;
uint16_t
cq_mode;
uint16_t
dlvr_mode;
uint64_t
local_addr;
gni_mem_handle_t
local_mem_hndl;
uint64_t
remote_addr;
gni_mem_handle_t
remote_mem_hndl;
uint64_t
length;
uint16_t
rdma_mode;
gni_cq_handle_t
src_cq_hndl;
uint64_t
sync_flag_value;
uint64_t
sync_flag_addr;
gni_fma_cmd_type_t
amo_cmd;
uint64_t
first_operand;
uint64_t
second_operand;
uint64_t
cqwrite_value;
gni_ce_cmd_type_t
ce_cmd;
uint32_t
ce_mode;
uint64_t
ce_red_id;
} gni_post_descriptor_t;
3.16.10.2 Members
next_descr Reserved for use by GNI.
prev_descr Reserved for use by GNI.
post_id
Used to track an FMA request.
status
Reserved for use by GNI.
cq_mode_complete
Reserved for use by GNI.
type
Required. The type of transaction. The following types are used for
this member:
•
•
•
•
•
•
•
S–2446–51
GNI_POST_RDMA_PUT
GNI_POST_RDMA_GET
GNI_POST_FMA_PUT
GNI_POST_FMA_PUT_W_SYNCFLAG
GNI_POST_FMA_GET
GNI_POST_AMO
GNI_POST_CQWRITE
173
cq_mode
Required. Instructs the NIC to generate completion
events. Only GNI_CQMODE_GLOBAL_EVENT and
GNI_CQMODE_REMOTE_EVENT can be requested for FMA_PUT,
FMA_GET and AMO transactions. The following modes are used for
this member:
•
GNI_CQMODE_LOCAL_EVENT
Can be used only for BTE transactions, and causes an event to be
delivered to the local endpoint's CQ when the local BTE engine
has finished handling that descriptor.
•
GNI_CQMODE_GLOBAL_EVENT
Can be specified for FMA and BTE transactions, and causes an
event to be delivered to the local endpoint's CQ when the data
successfully arrives at its destination (either local or remote,
depending on the operation).
With the Aries ASIC NIC, this mode is required to support
deadlock avoidance logic for FMA transactions. If not
specified, PostFma will fail with the runtime error
GNI_RC_INVALID_PARAM. This mode prevents the source
FMA buffer from being reused until after the Global Completion
Event has been received from GNI_CqGetEvent
•
GNI_CQMODE_REMOTE_EVENT
Can be used for FMA and BTE transactions, and causes an event
to be delivered to the CQ associated with the remote memory
registration when the transaction completes.
•
GNI_CQMODE_SILENT
Deprecated on Aries ASIC systems. Generate no completion
events to any associated CQ (local or remote).
•
dlvr_mode
GNI_CQMODE_DUAL_EVENTS(
GNI_CQMODE_LOCAL_EVENT |
GNI_CQMODE_GLOBAL_EVENT )
Required. Applications must reset the delivery mode to zero before
using a default mode when adaptive routing and hashing are enabled.
Behavior is NIC dependent, as indicated. In general, the NIC
uses strict ordering. It is an error to specify contradictory (both
adaptive and non-adaptive, for example) delivery modes on the same
transaction.
174
S–2446–51
GNI_DLVMODE_PERFORMANCE
Gemini: Instructs router to adaptively route
transaction on a per-packet basis.
Aries: Equivalent to GNI_DLVMODE_ADAPTIVE0.
GNI_DLVMODE_NO_ADAPT
Gemini: Instructs routers to not use adaptive
routing for network request packets and route
them in-order, deterministically instead. If
GNI_DLVMODE_NO_HASH is not set, then for a
given source and destination endpoint pair, strict
ordering is maintained only amongst packets
targeting the same destination memory cacheline.
Aries: If only the GNI_DLVMODE_NO_ADAPT
mode is specified, it is equivalent to
GNI_DLVMODE_MIN_HASH. If both
GNI_DLVMODE_NO_ADAPT and
GNI_DLVMODE_NO_HASH delivery modes
are specified, then the routing behavior will be set to
GNI_DLVMODE_MNON_HASH.
GNI_DLVMODE_NO_HASH
Gemini: Instructs routers to not use 'address'
component of the request in the packet routing
algorithm.
Aries: Equivalent to GNI_DLVMODE_MNON_HASH.
GNI_DLVMODE_NO_RADAPT
Gemini: Instructs routers to not use adaptive routing
for network response packets and route them
in-order, deterministically, instead.
Aries: No effect.
GNI_DLVMODE_IN_ORDER
Gemini: For a given source and destination
endpoint pair, strict ordering is maintained
amongst all packets issued with this same
delivery mode (GNI_DLVMODE_NO_ADAPT +
GNI_DLVMODE_NO_HASH).
Aries: Equivalent to GNI_DLVMODE_MNON_HASH.
S–2446–51
175
GNI_DLVMODE_NMON_HASH
Aries only. Routing is minimal. For a given source
and destination endpoint pair, strict ordering is
maintained amongst all packets issued with this same
delivery mode.
GNI_DLVMODE_NMIN_HASH
Aries only. Routing is non-minimal. The route used
to reach destination may not be the most direct.
For a given source and destination endpoint pair,
strict ordering is maintained only amongst packets
targeting the same destination memory cacheline.
GNI_DLVMODE_MIN_HASH
Aries only. Routing is minimal. The route used to
reach the destination is chosen from the set of routes
that are the most direct (use the least number of
hops). For a given source and destination endpoint
pair, strict ordering is maintained only amongst
packets targeting the same destination memory
cacheline.
GNI_DLVMODE_ADAPTIVE0
Aries only. This mode is configured as adaptive,
no-bias routing.
increasing minimal bias.
increasing minimal bias.
Aries only. Adaptive, non-minimal bias.
176
S–2446–51
local_addr The address of the region on the local node. This is the source for
PUT and the target for GET operations. It must be a 4-byte aligned
for GET operations and 8-byte aligned for AMOs.
local_mem_hndl
The local memory handle. This member is not required for FMA
PUT and AMOs with PUT semantics.
remote_addr
The address of the remote region. This is the target for PUTs and
source for GETs. Must be 4-byte aligned for GET operations and
8-byte aligned for AMOs.
remote_mem_hndl
Remote memory handle.
length
Number of bytes to move. Must be a multiple of 4-bytes for GETs
and multiple of 8-bytes for AMOs.
For GNI_POST_RDMA_PUT and GNI_POST_RDMA_GET, the
maximum post length is ((2^32)-1).
rdma_mode
There are two modes used for this member:
•
GNI_RDMAMODE_PHYS_ADDR
If set, the kernel-level application uses a physical address for the
local_addr field.
•
GNI_RDMAMODE_FENCE
If set, causes the completion processing of the transaction
descriptor to be delayed until all network responses associated
with the current descriptor as well as all responses associated
with previously processed descriptors of the same BTE channel,
have been received. Processing of the next descriptor for the
channel does not start until the write-back of the current transmit
transaction descriptor is issued.
src_cq_hndl
If set, the NIC delivers the source completion events related to this
transaction to the specified completion queue instead of the default
one.
sync_flag_value
Synchronization value.
S–2446–51
177
sync_flag_addr
Local to deliver synchronization value.
amo_cmd
AMO command for the transaction.
first_operand
First operand required by the AMO command.
second_operand
Second operand required by the AMO command.
cqwrite_value
Value to use for a CQ write. Only six least significant bytes is
available to software.
3.16.11 gni_smsg_attr
The gni_smsg_attr structure defines the attributes for short messaging. This
structure is used by the SmsgInit function.
3.16.11.1 Synopsis
typedef struct gni_smsg_attr {
gni_smsg_type_t
msg_type;
void
*msg_buffer;
uint32_t
buff_size;
gni_mem_handle_t
mem_hndl;
uint32_t
mbox_offset;
uint16_t
mbox_maxcredit;
uint32_t
msg_maxsize;
} gni_smsg_attr_t;
3.16.11.2 Members
msg_type
The type of short message buffering method
to use. On Aries systems, must be set to
GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT. On Gemini
systems, this member uses the following message types:
•
GNI_SMSG_TYPE_MBOX (Deprecated on Aries systems)
•
GNI_SMSG_TYPE_MBOX_AUTO_RETRANSMIT supports
automatic retransmission of short messages by the GNI library in
the event of transient network faults.
For both of these types, the buffer space for incoming messages is
associated with a single remote endpoint.
178
S–2446–51
msg_buffer A pointer to the beginning of the memory region used for message
buffers. Individual message buffers may be associated with different
endpoints.
buff_size
Size of the message buffer in bytes for this endpoint.
mem_hndl
Memory handle for the memory region used for message buffers.
mbox_offset
Offset from msg_buffer in bytes indicating the base address for
the message buffer associated with this endpoint.
mbox_maxcredit
The maximum number of messages that can be buffered in the
message buffer.
msg_maxsize
The maximum size of the short message which can be received for
this endpoint.
3.16.12 gni_smsg_handle
The gni_smsg_handle structure is reserved for use by the GNI infrastructure.
3.16.13 gni_msgq_attr
The gni_msgq_attr structure defines the attributes for the shared message queue.
This structure is used by the MsgqInit function.
3.16.13.1 Synopsis
typedef struct gni_msgq_attr {
uint32_t
max_msg_sz;
uint32_t
smsg_q_sz;
uint32_t
rcv_pool_sz;
uint32_t
num_msgq_eps;
uint32_t
nloc_insts;
uint8_t
modes;
uint32_t
rcv_cq_sz;
} gni_msgq_attr_t;
S–2446–51
179
3.16.13.2 Members
max_msg_sz Maximum message size in bytes.
smsg_q_sz
SMSG queue size in units of max_msg_sz. Each SMSG connection
sets the SMSG send credits count equal to the smsg_q_sz, by
default.
rcv_pool_sz
Number of message buffers in the receive pool.
num_msgq_eps
The number of nodes using the message queue system.
nloc_insts The number of local instances attaching to the shared message queue
on the node.
modes
Mode flags specifying message queue properties.
rcv_q_sz
The number of entries in the receive completion queue.
3.16.14 gni_msgq_rem_inst
3.16.14.1 Synopsis
typedef struct gni_msgq_rem_inst {
uint32_t
id;
gni_mem_handle_t mdh;
uint64_t
mdh_off;
} gni_msgq_rem_inst_t;
3.16.14.2 Members
id
Instance ID.
mdh
MDH for the SHMEM region.
mdh_off
Offset into the MDH for the SMSG mail box.
3.16.15 gni_msgq_ep_attr
3.16.15.1 Synopsis
struct gni_msgq_ep_attr {
uint32_t
pe_addr;
uint32_t
max_msg_sz;
uint32_t
smsg_q_sz;
uint32_t
num_insts;
gni_msgq_rem_inst_t insts[GNI_MSGQ_NODE_INSTS_MAX];
} gni_msgq_ep_attr_t;
180
S–2446–51
3.16.15.2 Members
pe_addr
The PE address of the remote endpoint (virtual, if NTT is enabled).
max_msg_sz Maximum message size in bytes.
smsg_q_sz
SMSG queue size in units of max_msg_sz. Each SMSG connection
sets the SMSG send credits count equal to the smsg_q_sz, by
default.
num_insts
The number of local instances attaching to the shared message queue
on the node.
insts
An array of local instance descriptions.
3.16.16 gni_version_info
3.16.16.1 Synopsis
struct gni_version_info {
uint32_t
ugni_version;
uint32_t
ugni_svn_revision;
char
ugni_build_string[MAX_BUILD_STRING_LENGTH];
uint32_t
kgni_version;
uint32_t
kgni_svn_revision;
char
kgni_build_string[MAX_BUILD_STRING_LENGTH];
} gni_version_info_t;
3.16.16.2 Members
ugni_version
uGNI version is incremented
ugni_svn_revision
SVN revision incremented with each modification to library
ugni_build_string
Build string incremented with each development build.
kgni_version
kgni version incremented each release.
kgni_svn_revision
kgni svn version incremented with each mod.
kgni_build_string
kgni build string incremented with each development build.
S–2446–51
181
182
S–2446–51
Part II: The DMAPP API
DMAPP is a communication library which supports a logically shared, distributed
memory (DM) programming model. DMAPP provides remote memory access
(RMA) between processes within a job in a one-sided manner. One-sided remote
memory access requests require no active participation by the process at the remote
node; synchronization functions may be used to determine when side-effects of
locally initiated requests are available.
DMAPP is typically not used directly within user application software. The
DMAPP API allows one-sided communication libraries (such as Cray SHMEM),
and PGAS compilers (such as Coarray Fortran and UPC), implemented on top of
DMAPP, to realize much of the hardware performance of the current Cray network
application-specific integrated circuit (ASIC) while being reasonably portable to
its successors.
In this discussion, a DMAPP application is an application that directly or indirectly
uses the DMAPP library.
4.1 DMAPP Programming Model
Cray has supported various forms of logically shared, distributed memory (DM)
programming models since the introduction of the Cray T3D. In this model, a group
of processes typically run the same executable in parallel. For purposes of this
discussion, such a group of related processes is termed a job. Although each process
in a job executes in its own address space, it can access certain memory segments
of other processes in the same job. This parallel model is sometimes referred to as
Single Program Multiple Data (SPMD). DMAPP only supports SPMD style parallel
jobs.
Usually the number of processes executing the application does not change over
the course of a job. Processes are sometimes termed PEs (processing elements).
Although each PE executes in its own address space, it can access certain
memory segments of other PEs in a one-sided (PUT/GET) manner using PGAS
language constructs or by invoking function calls to libraries supporting one-sided
programming models (such as Cray SHMEM).
S–2446–51
185
4.2 DMAPP Applications and Fork
The behavior of DM applications with respect to fork depends on which application
memory segments are selected for export at link time. Fork should be avoided if the
application’s stack and/or local heap are exported. The static data segment is shared
between a PE and any forked children if this segment is exported. The symmetric
heap is shared between PEs and any forked children.
Other DMAPP resources are also shared between a PE and any forked children. Child
processes are not new PEs in the DM job. As with multi-threaded PEs, the application
is responsible for setting up mutual exclusion regions around DMAPP calls.
4.3 DMAPP Applications and Threads
PEs within a DM application can be multi-threaded. However, unless the user
specifies a concurrency level larger than one during DMAPP initialization, none
of the functions in the DMAPP API should be considered thread-safe. Even if a
concurrency level larger than one is specified, none of the functions in the DMAPP
API are reentrant. For instance, they cannot be called from within a signal handler.
4.4 DMAPP Applications and File Descriptors
Handling of file descriptors in a DM application is similar to that on Cray X2 system.
Each PE maintains its own private file descriptors.
4.5 DMAPP Application Intra-node Communication
Data exchange between PEs on a node use the network interface. It is up to the DM
model implementation to optimize for intra-node communication, if desirable.
4.6 Compiling and Launching DMAPP Applications
Source files that call DMAPP functions must include the dmapp.h header, which is
included in the dmapp module.
Compile and link DMAPP applications on Cray systems using Cray-supplied
compiler/linker driver scripts. To link a routine that makes DMAPP calls, the dmapp
library is needed, which is included in the dmapp module. Use the following
commands to compile dmapp_sample_get.c on page 323:
module load dmapp
cc -dynamic -o test dmapp_sample_get.c -ldmapp
186
S–2446–51
By default, DMAPP mmaps the symmetric heap to huge pages and it maps the
static data and private heap to base pages. It is advisable to map the static data
and private heap onto huge pages. To use huge pages, link the application with
libhugetlbfs. See the intro_hugepages(1) man page for more detailed
information.
Use the ALPS aprun command to launch DMAPP applications.
4.7 Resiliency
DMAPP does not support error recovery in the presence of link failures. It is up to the
application to deal with such error, if so desired.
4.8 DMAPP Remote Memory Access
Remotely accessible memory segments in a PE can be classified as either symmetric
or non-symmetric.
The address of an object within a symmetric memory segment of a PE[X] has a
known relationship to the address of this same object in the address space of another
PE[Y] in the same job. Objects within these symmetric memory segments on PE[X]
can be accessed in a one-sided manner by PE[Y] using address information generated
locally on PE[Y].
Objects within non-symmetric memory segments on PE[X], can only be accessed
in a one-sided manner by a second PE[Y], using address information generated by
PE[X] and communicated to PE[Y].
The DMAPP implementation does not guarantee symmetry of the symmetric heap. It
is up to the DM model implementation to guarantee the symmetry of the symmetric
heap or any other symmetric regions other than the statically linked data segment.
For most DM model implementations, symmetric regions are the statically linked data
segment and a symmetric heap segment.
Preparing memory segments of a DM application for remote memory access is
handled by DMAPP startup code. Segments which may be exported include the
static data segment. The symmetric heap is always exported. At runtime, application
software can determine which segments of the address space are exported using query
functions. See dmapp_get_jobinfo on page 212.
Each exported memory segment has an associated dmapp_seg_desc_t. See
dmapp_seg_desc on page 203.
You should be aware that there are trade-offs in requesting that various program
segments be exported.
S–2446–51
187
Since the AMD64 processor cannot be used effectively to directly load/store from
exported memory on remote nodes, DMAPP provides an API for interfacing to the
remote memory access (RMA) hardware mechanisms. The DMAPP RMA functions
can be divided into the following categories:
•
•
One-sided RMA functions
RMA synchronization functions
The effects of one of these RMA functions are said to be globally visible when all of
the effects of that request are assured be visible to all observers (including processors)
of the request. It also means that all side effects of the request have completed.
All one-sided RMA functions (PUT type, GET type and atomic memory operations)
belong to one of the following three categories:
blocking (no suffix)
The process returns from the function only after the side-effects of
the remote memory access are globally visible in the system.
non-blocking explicit (_nb suffix)
A synchronization ID (syncid) is returned to the process.
The effects of the remote memory access are only assured
to be globally visible in the system after the application has
determined via a synchronization call (dmapp_syncid_test or
dmapp_syncid_wait) whether the syncid has been retired.
non-blocking implicit (_nbi suffix)
No synchronization ID is returned to the process, the effects of
the remote memory access are only assured to be globally visible
in the system following a call to dmapp_gsync_test or
dmapp_gsync_wait. For performance reasons, this mode is
recommended for applications with many small messages, where
blocking calls or using individual syncids would be expensive.
One-sided remote memory access requests require no active participation by PEs at
the remote node. Remote memory segments which are targets of operations with
put semantics or sources of operations with get semantics must have been exported
at job startup.
The maximum number of concurrent, non-blocking (explicit and implicit) requests
allowable can be set by the application at DMAPP initialization. If the application
attempts to initiate more non-blocking (explicit and implicit) requests than this
maximum, DMAPP returns an error.
There are no ordering guarantees as to completion of different non-blocking RMA
requests initiated by a PE.
188
S–2446–51
4.9 DMAPP API
DMAPP provides a C interface for applications. Most DMAPP functions return a
status value indicating success or failure of the call. In the case of non-blocking RMA
functions, this status does not indicate whether or not the remote memory access
request completed successfully; it simply indicates whether the transfer request was
initiated successfully. See Chapter 5, DMAPP API Reference on page 197.
4.9.1 Initialization and Query Functions
Before using any other DMAPP functions, an application must call an initialization
function to request and initialize resources.
DMAPP clients should use dmapp_init_ext which provides greater control over
Processor Interface (PI) used by RMA requests than dmapp_init provides.
dmapp_init is preserved for backward compatibility.
See dmapp_init on page 210 and dmapp_init_ext on page 211.
After the last call to any other DMAPP functions, an application must call
dmapp_finalize to return resources. See dmapp_finalize on page 212.
The query function dmapp_get_jobinfo returns general information about the
job, such as the DMAPP version, number of PEs in the job, and symmetric heap and
data segment locations. See dmapp_get_jobinfo on page 212.
A process can set RMA attributes to control the way that DMAPP handles
various RMA requests. Some attributes can be set only during initialization.
They will be referred to as static attributes. Others can be set multiple times
over the course of the job, and will be referred to as dynamic attributes.
Setting dynamic attributes does not affect RMA requests previously issued
by the PE, only subsequent RMA requests. Dynamic attributes include when
to switch from CPU-based mechanisms for handling RMA requests to using
CPU offload mechanisms. See dmapp_set_rma_attrs on page 219,
dmapp_set_rma_attrs_ext on page 220, dmapp_get_rma_attrs on
page 213, and dmapp_get_rma_attrs_ext on page 213.
4.9.2 One-sided RMA Functions
RMA functions share some common arguments.
The remote address for either PUT or GET style operations is specified by a virtual
address, a segment descriptor, and the remote PE.
S–2446–51
189
When the virtual address is generated locally by the initiating PE, as is the case
when working with symmetric data objects, the segment descriptor supplied in the
job_info structure returned by dmapp_get_job_info may be used. If the
virtual address was obtained from the remote PE via some external pointer-passing
mechanism, the segment descriptor from the remote PE must be used.
In addition to some common arguments, each of the data motion functions can
operate on 1, 4 (DWORD), 8 (QWORD), or 16 (DQWORD) byte data types.
Hardware will work most efficiently with requests that are at least DWORD aligned.
For non-blocking explicit functions, the syncid argument supplies a pointer to a
location in local memory which will be used by DMAPP for storing synchronization
related information.
4.9.2.1 Contiguous Functions
The PUT functions store a contiguous block of data, starting at local memory address
source_addr, into a contiguous block at a remote address.
For more detailed information, see dmapp_put_nb on page 220,dmapp_put_nbi
on page 222, and dmapp_put on page 223.
The GET functions load a contiguous block of data starting from a remote source
address to a contiguous block starting at local memory address target_addr. Note
that zero-length GETs are not supported.
For both PUT and GET functions, the remote address is specified by the triplet
consisting of a virtual address, segment descriptor, and processor. The nelems
parameter specifies the number of elements of type type to transfer. The memory
region described by the remote address and nelems must reside in an exported
memory of the remote PE.
For further information, see dmapp_get_nb on page 229, dmapp_get_nbi on
page 230, and dmapp_get on page 231.
4.9.2.2 Strided Functions
The strided PUT functions store data starting at a local memory address, using a
specified stride, to a remote target address, using a separately specified stride.
The strided GET functions load data starting from a remote memory address using
a specified stride and copy the data to a local memory address using a separately
specified stride. For more information, see dmapp_iget_nb on page 238,
dmapp_iget_nbi on page 239, and dmapp_iget on page 240.
For both strided PUT and GET functions, the remote address is specified by the triplet
consisting of a virtual address, segment descriptor, and PE. The remote memory
region described by the remote address, stride, and the number of elements to transfer
must reside in an exported segment of remote memory.
190
S–2446–51
For more information, see dmapp_iput_nb on page 234, dmapp_iput_nbi on
page 235, and dmapp_iput on page 236.
4.9.2.3 SCATTER/GATHER Functions
The SCATTER functions store elements of a contiguous block of data in local
memory to a remote memory location using multiple offset values.
The remote address for each element is specified by the triplet consisting of the
virtual address, segment descriptor and target process, plus an offset value.
The memory region defined by the remote address, the largest offset in the array, and
the number of elements transferred must be within an exported memory segment
of the remote memory.
These functions are also referred to as indexed PUT functions and begin with the
string dmapp_ixput. For more detail, see dmapp_ixput_nb on page 242,
dmapp_ixput_nbi on page 243, and dmapp_ixput on page 244.
The GATHER functions GET separate elements from remote memory locations
placed at various offsets, to contiguous local memory. The remote address for each
element is specified by the triplet consisting of the virtual address, segment descriptor
and remote process, plus an offset value.
The memory region defined by the remote address, the largest offset in the array, and
the number of elements transferred must be within an exported memory segment
of the remote memory.
The Indexed GET functions are dmapp_ixget_nb on page 245,
dmapp_ixget_nbi on page 247, dmapp_ixget on page 248.
4.9.2.4 PE-strided Functions
The following functions provide PUT (broadcast), GATHER, and SCATTER to
remotely accessible addresses across a set of PEs in a DMAPP job. Note that none of
these are collective operations. These routines are best used when a small amount of
data needs to be broadcast, scattered to, or collected from a set of PEs.
The PUT (broadcast) functions with indexed PE stride store data from local memory
to the remote memory of multiple PEs within a DMAPP job. When the transfer is
complete, each remote PE will have a copy of the contents of the original source
buffer. See dmapp_put_ixpe_nb on page 250, dmapp_put_ixpe on page 253,
dmapp_put_ixpe_nbi on page 251.
S–2446–51
191
The SCATTER functions with indexed PE stride store data starting at an address
in local memory to multiple remote PEs within a DMAPP job. Each target PE
receives a different portion of the local source data. The remote address ranges
must be exported for each PE and the remote addresses must be symmetric. These
functions are not collective and best used when a small amount of data needs to
be transferred to a set of PEs. See dmapp_scatter_ixpe_nb on page 254,
dmapp_scatter_ixpe_nbi on page 255, and dmapp_scatter_ixpe on
page 257.
The GATHER functions with indexed PE stride load data from the remote memory
of multiple PEs within a DMAPP job, and concatenate it in local memory. The
remote address ranges must be exported for each PE and the remote addresses must
be symmetric. These functions are not collective and best used when a small amount
of data needs to be transferred to a set of PEs. See dmapp_gather_ixpe_nb on
page 259, dmapp_gather_ixpe_nbi on page 260, and dmapp_gather_ixpe
on page 261.
For all PE-strided functions, the remote address must be a symmetric address; it must
lie in the statically linked data segment or the symmetric heap.
4.9.2.5 DMAPP AMO Functions
In addition to PUT and GET RMA functionality, DMAPP also provides support for
using atomic memory operation (AMO) RMA requests. The set of AMO functions
are modeled on the set provided on the Cray X2 systems. AMOs can be used both for
synchronization and at-memory style operations. AMOs are restricted to operating
on 8-byte (also referred to as qword, qw, or quad word) data types, located in a
remote PE.
As with RMA functions, the remote memory location must reside in an exported
memory segment of remote PE.
The scalar-type blocking and non-blocking Atomic Memory Operations
(AMO) functions where no result is returned are named dmapp_op_qw,
dmapp_op_qw_nb, dmapp_op_qw_nbi, where op is either AADD, AAND, AOR,
or AXOR.
For more detail on all AMO functions, see dmapp_aadd_qw_nb on page 263
through dmapp_acswap_qw on page 296.
The scalar-type blocking and non-blocking Atomic Memory Operations (AMO)
functions in which the result is returned in the response are named dmapp_op_qw,
dmapp_op_qw_nb, dmapp_op_qw_nbi, where op is either ACSWAP, AFADD,
AFAND, AFAX, AFOR, or AFXOR.
AMO functions which return a result are also referred to as AMOs with GET
semantics of fetching AMOs. AMOs which do not return a result are also referred to
as AMO with PUT semantics or non-fetching AMOs.
192
S–2446–51
It is not safe to mix DMAPP access (DMAPP AMO PUT or GET) with an AMO
variable with local load/store access. See the flags member in Members on page 202.
4.9.2.6 DMAPP Synchronization Functions
DMAPP applications use synchronization functions to determine when locally
initiated, non-blocking RMA requests have completed.
A process can determine when the effects of a non-blocking explicit RMA function
call are globally visible in the system by using dmapp_sync_test() or
dmapp_sync_wait(), both of which return information about the specified
syncid.
dmapp_sync_test() immediately returns a value indicating whether all RMA
requests associated with syncid have completed. dmapp_sync_wait(), only
returns after all RMA requests associated with syncid have completed. See
dmapp_syncid_test on page 300, and dmapp_syncid_wait on page 301.
A process can determine when the side effects of one or more non-blocking
implicit RMA function calls are globally visible in the system by using
dmapp_gsync_test() or dmap_gsync_wait() functions, both of which
refer to all remote memory accesses associated with previously issued non-blocking
implicit RMA requests; therefore, a syncid is not relevant.
dmapp_gsync_test() immediately returns with a value indicating whether all
remote memory accesses associated with previously issued non-blocking implicit
RMA function calls are globally visible in the system. dmap_gsync_wait()
only returns after all non-blocking implicit RMA requests are globally visible in
the system. See dmapp_gsync_test on page 302 and dmapp_gsync_wait
on page 302.
4.9.2.7 Collective Operations
DMAPP provides a set of routines to perform global reduction operations across
multiple processes. These collective operations are intended to be used for small
quantities of data such as reductions on 4-7 double precision numbers, broadcast of
up to 64 bytes, etc.
Processes associated with each other, in the context of collective operations, are
referred to as a process set, or pset.
Prior to pset creation, the DMAPP application must first initialize a
dmapp_c_pset_desc_t structure to describe the processes contained in the
pset. See dmapp_c_pset_desc_t on page 209.
S–2446–51
193
A pset is created using a two step process. First, each rank which will be
participating in DMAPP collective operations locally defines the ranks involved in
the pset using the dmapp_c_pset_create function. The ranks then use an
out-of-band synchronization mechanism to insure that all ranks contained in the
pset have completed the call to dmapp_c_pset_create, at which point the
pset can be exported by each rank using the dmapp_c_pset_export function.
Rank information can be obtained from the dmapp_get_job_info function, or
from the PMI PMI_Get_rank_in_app function. See dmapp_c_pset_create
on page 303 and dmapp_c_pset_export on page 304.
To initiate a global reduction operation over a previously exported process set,
an application will use the dmapp_c_greduce_start() function. The
dmapp_c_barrier_join() function initiates a barrier join operation on a
previously exported pset.
The interfaces are non-blocking, with the exception of dmapp_c_pset_wait(),
which instructs an application to wait for completion of a collective operation for a
given pset.
Note that only one outstanding, non-blocking collective operation is allowed on a
given pset.
An application may use dmapp_c_pset_cancel_op() to cancel an outstanding
collective operation on a pset.
See dmapp_c_greduce_start on page 307 ,dmapp_c_barrier_join
on page 305, dmapp_c_pset_wait on page 310, and
dmapp_c_pset_cancel_op on page 306.
The DMAPP application must use dmapp_c_pset_destroy() to free internal
resources associated with the pset. See dmapp_c_pset_destroy on page 306.
Note: DMAPP makes no distinction between ranks in a pset which are
collocated on the same node and those that are not. It is very likely that upper-level
software can more efficiently handle on-node components of, for example, barrier
operations than DMAPP can. It is expected that for best performance, such
software will use DMAPP only for inter-node components of collective operations,
and user lower-latency, shared memory methods for intra-node stages of these
operations.
194
S–2446–51
4.9.3 Symmetric Heap Functions
DMAPP provides routines for allocating and releasing symmetric heap memory.
The DMAPP application is responsible for preserving symmetry of this heap
memory. This is achieved by ensuring that all PEs in a job make the same calls to the
symmetric heap management functions in the same sequence, involving the same
amount of memory. The DMAPP application controls the size of the symmetric heap
at startup. DMAPP also provides a routine which returns symmetric heap usage
information for debugging purposes.
void *dmapp_sheap_malloc(IN size_t size)
This function allocates size bytes memory from the symmetric heap. Equality of
addresses across PEs is not guaranteed.
void *dmapp_sheap_realloc(IN void *ptr,IN size_t size)
This function changes the size of the block to which ptr points to the size (in bytes)
specified by size. Equality of addresses across PEs is not guaranteed.
void dmapp_sheap_free(IN void *ptr)
This function frees a block of memory previously allocated by
dmapp_sheap_malloc or dmapp_sheap_realloc.
dmapp_return_t dmapp_sheap_stats(dmapp_sheap_stats_t *stats);
This function returns the DMAPP Symmetric heap usage information in a structure of
type dmapp_sheap_stats_t.
4.9.4 Checkpoint Restart Functions
These functions allow a BLCR callback function to prepare an application
for checkpointing and to perform the checkpoint/restart operation. See
dmapp_checkpoint on page 318 and dmapp_restart on page 318.
S–2446–51
195
196
S–2446–51
This chapter contains reference information for enumerations, structures, and
functions contained in the DMAPP API. Your application must include the dmapp.h
file when using this API.
5.1 DMAPP Enumerations
Function return codes.
5.1.1 dmapp_return
5.1.1.1 Synopsis
typedef enum dmapp_return {
DMAPP_RC_SUCCESS = 0,
DMAPP_RC_INVALID_PARAM,
DMAPP_RC_ALIGNMENT_ERROR,
DMAPP_RC_TRANSACTION_ERROR,
DMAPP_RC_RESOURCE_ERROR,
DMAPP_RC_PERMISSION_ERROR,
DMAPP_RC_NO_SPACE, /* transaction could not be completed due */
/* to insufficient resources; */
/* user should synchronize more often or */
/* increase max_outstanding_nb */
DMAPP_RC_NOT_DONE, /* used in _dmapp_nbi_syncid->status */
DMAPP_RC_NOT_SUPPORTED,
DMAPP_RC_NOT_FOUND,
DMAPP_RC_BUSY,
DMAPP_RC_NOT_USED /* not used, serves as a delimiter */
} dmapp_return_t;
5.1.2 dmapp_type
The dmapp_type_t enumeration defines the valid types supplied by the type input
parameter to all data motion functions.
S–2446–51
197
5.1.2.1 Synopsis
typedef enum dmapp_type {
DMAPP_DQW = 0,
DMAPP_QW,
DMAPP_DW,
DMAPP_BYTE
} dmapp_type_t;
5.1.2.2 Constants
DMAPP_DQW
Indicates a double quad (16 byte) word.
DMAPP_QW
Indicates a quad (8 byte) word.
DMAPP_DW
Indicates a double (4 byte) word.
DMAPP_BYTE Indicates a byte. This option does not provide good performance.
5.1.3 dmapp_routing_type
The dmapp_routing_type_t enumeration defines the valid network routing
modes to be supplied to the relaxed_ordering fields of the RMA attributes
structures dmapp_rma_attrs on page 205 and dmapp_rma_attrs_ext on
page 207.
The following options are available on both aries and gemini platforms unless
otherwise noted. If an option is provided to dmapp_set_rma_attrs() which is
not valid for a given platform, the previous routing mode setting remains unchanged.
5.1.3.1 Synopsis
typedef enum dmapp_routing_type {
DMAPP_ROUTING_IN_ORDER = 0, /* hash off, adapt off */
DMAPP_ROUTING_DETERMINISTIC, /* hash on, adapt off */
DMAPP_ROUTING_ADAPTIVE,
/* hash off, adapt on */
DMAPP_ROUTING_DETERMINISTIC_1,
DMAPP_ROUTING_ADAPTIVE_1,
DMAPP_ROUTING_NOT_USED
} dmapp_routing_type_t;
198
S–2446–51
5.1.3.2 Members
DMAPP_ROUTING_IN_ORDER
Ensure strict in order delivery of all packets targeting a given PE.
Note that this setting will result in poor use of available network
bandwidth and is generally not recommended.
DMAPP_ROUTING_DETERMINISTIC
Guarantee in-order delivery of packets from a source PE to a
destination PE if the packets target the same cache line. On Aries,
this corresponds to deterministic non-minimal hashed. Packets
are hashed over all available paths, so most traffic will route
non-minimally. This doubles the load on the network, but is safe
because it avoids hot-spots even for non uniform traffic distributions.
On Gemini-based systems, there is no concept of minimal verses
non-minimal hashed and the discussion of this topic does not apply.
DMAPP_ROUTING_ADAPTIVE
Enable adaptive routing of packets. Packets from the same source to
the same target are not ordered.
DMAPP_ROUTING_DETERMINISTIC_1
Aries only. Deterministic minimal hashed. Packets are hashed over
minimal paths. Packets from the same source to the same target with
the same optional hash value (i.e., the same address) are ordered.
This type of routing will provide the best performance for known
uniform random traffic, but is susceptible to worst-case performance
for non-uniform traffic distributions. Deterministic minimal hashed
routing is dangerous, as it could lead to worst-case behavior and
is generally not recommended. In cases where the global traffic is
known to be uniform, deterministic minimal routing could be used to
force minimal routing and reduce overall network contention.
S–2446–51
199
DMAPP_ROUTING_ADAPTIVE_1
Aries only. Enable adaptive routing of packets. Packets from the
same source to the same target are not ordered.
DMAPP_ROUTING_NOT_USED
Not used. Serves as delimiter.
5.1.4 dmapp_pi_reg_type
The dmapp_pi_reg_type_t enumeration defines the modes of PI access
ordering to be used by DMAPP during memory registration with uGNI; therefore,
these modes apply to the data and symmetric heap and any user or dynamically
mapped regions. See PI_ordering field of the extended RMA attributes structure
dmapp_rma_attrs_ext on page 207.
These modes do not affect GET operations, only PUT operations.
STRICT ordering ensures that posted and non-posted writes arrive at the target in
strict order. Use STRICT for small message throughput when bandwidth is not a
priority (for example, when using _nbi interfaces).
DEFAULT and RELAXED ordering impose no ordering constraints. In these modes,
DMAPP does perform remote synchronization during its sync/wait for some
non-blocking implicit PUT operations thus adding some overhead. Therefore,
DEFAULT/RELAXED ordering gives better overall performance when increased
bandwidth is a priority as is the case when many ranks are communicating.
typedef enum dmapp_pi_reg_type {
DMAPP_PI_ORDERING_STRICT = 0, /* Strict PI (P_PASS_PW=0, NP_PASS_PW=0) */
DMAPP_PI_ORDERING_DEFAULT,
/* Default GNI PI (P_PASS_PW=0, NP_PASS_PW=1) */
DMAPP_PI_ORDERING_RELAXED
/* Relaxed PI ordering (P_PASS_PW=1, NP_PASS_PW=1) */
} dmapp_pi_reg_type_t;
200
S–2446–51
5.1.5 dmapp_c_op
The dmapp_c_op enumeration defines the valid op input parameters required by
dmapp_c_greduce_start().
typedef enum dmapp_c_op {
DMAPP_C_SUM = 201,
DMAPP_C_MAX,
DMAPP_C_MIN,
DMAPP_C_PROD,
DMAPP_C_LAND,
DMAPP_C_BAND,
DMAPP_C_LOR,
DMAPP_C_BOR,
DMAPP_C_LXOR,
DMAPP_C_BXOR,
DMAPP_C_MINLOC,
DMAPP_C_MAXLOC,
DMAPP_C_OP_LAST
} dmapp_c_op_t;
5.1.6 dmapp_c_type
The dmapp_c_type_t enumeration defines the valid types required
by the type input parameter to dmapp_c_greduce_start() and
dmapp_c_greduce_nelems_max().
typedef enum dmapp_c_type {
DMAPP_C_INT32 = 101,
DMAPP_C_UINT32,
DMAPP_C_INT64,
DMAPP_C_UINT64,
DMAPP_C_FLOAT,
DMAPP_C_DOUBLE,
DMAPP_C_COMPLEX8,
DMAPP_C_COMPLEX16,
DMAPP_C_FLOAT_UINT64,
DMAPP_C_DOUBLE_UINT64,
DMAPP_C_INT32_UINT64,
DMAPP_C_INT64_UINT64,
DMAPP_C_TYPE_LAST
} dmapp_c_type_t;
5.2 DMAPP Structures
5.2.1 dmapp_amo
The dmapp_amo structure is an atomic memory operator (AMO) descriptor which
defines the operator and operands used as input to the generic put and get functions.
Aries systems only support 32-bit floating point ADD AMOs.
S–2446–51
201
Gemini systems support 64-bit (DMAPP_QW) operands. See Supported AMOs on
page 202.
5.2.1.1 Synopsis
typedef struct dmapp_amo {
dmapp_amo_cmd_t
dmapp_type_t
dmapp_amo_operand_t
dmapp_amo_operand_t
uint64_t
} dmapp_amo_t;
cmd;
type;
operand;
operand2;
flags;
typedef union {
int32_t op_int32;
int64_t op_int64;
float
op_float;
double
op_double;
} dmapp_amo_operand_t;
5.2.1.2 Members
cmd
Specific AMO command to execute.
type
Type of result.
operand1
Type of operand1. Only int64 is supported.
operand2
Type of operand2. Only int64 is supported.
flags
The only valid flag currently supported is DMAPP_AMO_CACHED;
the caching variant of the AMO operation is used, which means that
the value in the AMO cache is only written back when a DMAPP
PUT or GET is executed on the AMO variable or the AMO cache
is full.
5.2.1.3 Supported AMOs
Table 2. Supported DMAPP AMOs
202
Operation
Put
Get
DMAPP_AMO_AAND
Yes
Yes
DMAPP_AMO_AAX
No
Yes
DMAPP_AMO_AOR
Yes
Yes
DMAPP_AMO_AXOR
Yes
Yes
DMAPP_AMO_ASWAP
No
No
DMAPP_AMO_ACSWAP
No
Yes
DMAPP_AMO_AIMIN
No
No
S–2446–51
Operation
Put
Get
DMAPP_AMO_AIMAX
No
No
DMAPP_AMO_AIADD
Yes
Yes
DMAPP_AMO_AFPMIN
No
No
DMAPP_AMO_AFPMAX
No
No
DMAPP_AMO_AFPADD
No
No
5.2.2 dmapp_seg_desc
The dmapp_seg_desc structure is a memory segment descriptor, with an address
and length.
5.2.2.1 Synopsis
typedef struct dmapp_seg_desc {
void
*addr;
size_t
len;
gni_mem_handle_t memhndl;
uint16_t
flags;
uint8_t
type;
uint8_t
seg;
void
*reserved;
} dmapp_seg_desc_t;
5.2.2.2 Members
addr
A pointer to the address for the memory segment.
len
The currently mapped size of the segment, in bytes.
memhndl
Memory handle for the region; automatically obtained at
initialization or obtained from a previous call to MemRegister.
flags
For internal use only.
type
seg
reserved
5.2.3 dmapp_sheap_stats_t
The dmapp_sheap_stats_t structure contains symmetric heap usage
information and is returned by dmapp_sheap_stats.
S–2446–51
203
5.2.3.1 Synopsis
typedef struct {
uint64_t size;
uint64_t usage;
uint64_t hwm;
uint64_t biggest_used;
uint64_t biggest_free;
uint64_t used_blks;
uint64_t free_blks;
uint64_t allocs;
uint64_t frees;
uint64_t fails;
uint64_t reserved[8];
} dmapp_sheap_stats_t;
5.2.3.2 Members
size
Total heap size in bytes.
usage
Current usage in bytes.
hwm
High water mark in bytes
biggest_used
Size of largest allocator block in use
biggest_free
Size of largest free allocator block.
used_blks
Number of allocator blocks in use.
free_blks
Number of free allocator blocks.
allocs
Running total number of alloc calls.
frees
Running total number of free calls.
fails
Running total number of failed alloc calls.
reserved[8]
Reserved for future use.
5.2.4 dmapp_jobinfo
The dmapp_jobinfo structure contains general information relevant to the job.
204
S–2446–51
5.2.4.1 Synopsis
typedef struct dmapp_jobinfo {
int
version;
int
hw_version;
int
npes;
dmapp_pe_t
pe;
dmapp_seg_desc_t data_seg;
dmapp_seg_desc_t sheap_seg;
} dmapp_jobinfo_t;
5.2.4.2 Members
version
The version of DMAPP that this job uses.
hw_version The hardware version of the system.
npes
The number of processing elements in use for the entire job.
pe
The processing element number, in [0, npes-1].
data_seg
The data segment in memory that this job is using.
sheap_seg
The symmetric heap memory that this job is using.
5.2.5 dmapp_rma_attrs
The dmapp_rma_attrs structure sets RMA attributes to control the way in
which DMAPP handles various RMA requests. Some attributes can be set during
initialization only, others can be set multiple times over the course of a job.
5.2.5.1 Synopsis
typedef struct dmapp_rma_attrs {
uint32_t max_outstanding_nb;
uint32_t offload_threshold;
uint8_t put_relaxed_ordering;
uint8_t get_relaxed_ordering;
uint8_t max_concurrency;
} dmapp_rma_attrs_t;
5.2.5.2 Members
max_outstanding_nb
The maximum number of outstanding non-blocking (explicit and
implicit) requests supported. You can only specify this flag during
initialization. The following is the range of valid values to be
supplied:
[DMAPP_MIN_OUTSTANDING_NB, .., DMAPP_MAX_OUTSTANDING_NB]
Setting the value to one of the extremes may lead to a slowdown. The
S–2446–51
205
recommended value is DMAPP_DEF_OUTSTANDING_NB. Users
can experiment with the value to find the optimal setting for their
application.
offload_threshold
The threshold, in bytes, for switching between CPU-based
mechanisms and CPU offload mechanisms. This value can be
specified at any time and can use any value. The default setting is
DMAPP_OFFLOAD_THRESHOLD. Very small or very large settings
may lead to suboptimal performance. The default value is 4k bytes.
Consider how to best set this threshold. While a threshold decrease
may increase CPU availability, it may also increase transfer latency
due to BTE involvement.
put_relaxed_ordering
Flags which indicate whether relaxed ordering of requests is allowed
and if so, which routing option to use.
Applies to RMA requests with PUT semantics and all AMOs.
Routing mode can be controlled separately for PUTs and GETs.
Valid options described by dmapp_routing_type on page 198.
On Gemini, the default for PUTs is
DMAPP_ROUTING_DETERMINISTIC and the default for GETs is
DMAPP_ROUTING_ADAPTIVE.
On Aries, the default for PUTs and GETs is
The value can be specified at any time.
get_relaxed_ordering
Specifies the type of routing to be used. Applies to RMA requests
with GET semantics. See dmapp_routing_type on page 198.
The default is DMAPP_ROUTING_ADAPTIVE. The value can be
specified at any time.
max_concurrency
The maximum number of threads that can access DMAPP. You can
only use this when thread-safety is enabled. The default is 1. You can
only specify this during initialization and it must be >= 1. If the
user does not know how many threads in a multi-threaded application
will access DMAPP, set to DMAPP_MAX_THREADS_UNLIMITED.
See DMAPP Applications and Threads on page 186.
206
S–2446–51
5.2.6 dmapp_rma_attrs_ext
The dmapp_rma_attrs_ext structure sets extended RMA attributes to control
the way in which DMAPP handles various RMA requests. Some attributes can be set
during initialization only, others can be set multiple times over the course of a job.
5.2.6.1 Synopsis
typedef struct dmapp_rma_attrs_ext {
uint32_t max_outstanding_nb;
uint32_t offload_threshold;
uint8_t put_relaxed_ordering;
uint8_t get_relaxed_ordering;
uint8_t max_concurrency;
uint8_t PI_ordering;
unint32_t queue_depth;
unint32_t queue_nelems;
uint8_t unused[32];
} dmapp_rma_attrs_t;
5.2.6.2 Members
max_outstanding_nb
The maximum number of outstanding non-blocking (explicit and
implicit) requests supported. You can only specify this flag during
initialization. The following is the range of valid values to be
supplied:
[DMAPP_MIN_OUTSTANDING_NB, .., DMAPP_MAX_OUTSTANDING_NB]
Setting the value to one of the extremes may lead to a slowdown. The
recommended value is DMAPP_DEF_OUTSTANDING_NB. Users
can experiment with the value to find the optimal setting for their
application.
offload_threshold
The threshold, in bytes, for switching between CPU-based
mechanisms and CPU offload mechanisms. This value can be
specified at any time and can use any value. The default setting is
DMAPP_OFFLOAD_THRESHOLD. Very small or very large settings
may lead to suboptimal performance. The default value is 4k bytes.
Consider how to best set this threshold. While a threshold decrease
may increase CPU availability, it may also increase transfer latency
due to BTE involvement.
put_relaxed_ordering
Flags which indicate whether relaxed ordering of requests is allowed
and if so, which routing option to use.
S–2446–51
207
Applies to RMA requests with PUT semantics and all AMOs.
Routing mode can be controlled separately for PUTs and GETs.
Valid options described by dmapp_routing_type on page 198.
On Gemini, the default for PUTs is
DMAPP_ROUTING_DETERMINISTIC and the default for GETs is
On Aries, the default for PUTs and GETs is
The value can be specified at any time.
get_relaxed_ordering
Specifies the type of routing to be used. Applies to
RMA requests with GET semantics. The default is
DMAPP_ROUTING_ADAPTIVE. The value can be specified at any
time. Note that DMAPP_ROUTING_IN_ORDER may result in poor
performance.
max_concurrency
The maximum number of threads that can access DMAPP. You can
only use this when thread-safety is enabled. The default is 1. You
can only specify this during initialization and it must be >= 1. See
DMAPP Applications and Threads on page 186.
PI_ordering
Defines the PI ordering registration flags used by DMAPP when
registering all memory regions with GNI. Applies to the data,
symmetric heap, and user or dynamically mapped regions. Possible
values are defined by dmapp_pi_reg_type on page 200. The
defaults are DMAPP_PI_ORDERING_STRICT on Gemini Systems
and DMAPP_PI_ORDERING_RELAXED on Aries Systems.
queue_depth
Number of slots each DMAPP queue.
queue_nelems
Number of 64-bit sized elements in each DMAPP queue slot.
queue_flags
Modifier flags for queue creation. Only
DMAPP_QUEUE_ASYNC_PROGRESS currently supported.
208
S–2446–51
5.2.7 dmapp_syncid
The dmapp_syncid structure contains a pointer to the synchronization ID that is
used by a non-blocking explicit RMA function.
5.2.7.1 Synopsis
typedef struct dmapp_syncid *dmapp_syncid_handle_t;
5.2.8 dmapp_c_pset_desc_t
The DMAPP application initializes a dmapp_c_pset_desc structure prior to
pset creation.
A symmetric heap address and length (returned from dmapp_sheap_malloc())
must be supplied in the concat_buf and concat_buf_size fields. Generally
the larger the concat_buf supplied at pset creation, the better the performance of
the concatenate function, especially as the number of ranks increases.
5.2.8.1 Synopsis
typedef struct {
uint32_t n_pes;
uint32_t *vec_pes;
} dmapp_c_pset_delimiter_vec_t;
typedef struct {
uint32_t n_pes;
uint32_t base_pe;
uint32_t stride_pe;
} dmapp_c_pset_delimiter_strided_t;
typedef struct {
void *concat_buf;
uint64_t concat_buf_size;
dmapp_c_pset_delimiter_type_t type;
union {
dmapp_c_pset_delimiter_vec_t vec_type;
dmapp_c_pset_delimiter_strided_t stride_type;
} u;
} dmapp_c_pset_desc_t;
5.2.9 dmapp_c_pset_handle_t
5.2.9.1 Synopsis
typedef struct dmapp_c_pset *dmapp_c_pset_handle_t;
S–2446–51
209
5.3 DMAPP Functions
5.3.1 dmapp_explain_error
Provides explanations for DMAPP return values. Upon completion, expl_ptr
contains the explanation of the return code provided as input argument error_code.
Recommendations of what to do in case of an error is left to upper level software.
5.3.1.1 Synopsis
extern void dmapp_explain_error(
IN dmapp_return_t error_code,
IN/OUT const char **expl_ptr);
5.3.2 dmapp_init
Superseded by dmapp_init_ext. The dmapp_init function initializes
resources for a DMAPP job. All DMAPP applications must call this function before
using other DMAPP functions. In a threaded application, this function should only
be called once.
Sets the network routing mode. See dmapp_routing_type on page 198 and
dmapp_rma_attrs on page 205.
If an invalid routing mode is specified as part of the RMA attributes input parameter,
the default routing modes are selected and a warning message is issued.
After the last call to any other DMAPP functions, an application must call a
finalization function: dmapp_return_t dmapp_finalize(void).
5.3.2.1 Parameters
requested_attrs
Pointer to the desired job attributes. See dmapp_rma_attrs_ext
on page 207.
actual_attrs
The actual job attributes.
210
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_INVALID_PARAM
One or more parameters are invalid.
DMAPP_RC_RESOURCE_ERROR
An error occurred during initialization.
5.3.3 dmapp_init_ext
The dmapp_init_ext function initializes resources for a DMAPP job. All
DMAPP applications must call this function before using other DMAPP functions. In
a threaded application, this function should only be called once.
This function replaces dmapp_init and provides greater control over the Processor
Interface (PI) used by RMA requests. dmapp_init is preserved for backward
compatibility.
5.3.3.1 Synopsis
dmapp_return_t dmapp_init_ext(
IN dmapp_rma_attrs_ext_t *requested_attrs,
OUT dmapp_rma_attrs_ext_t *actual_attrs);
5.3.3.2 Parameters
requested_attrs
Pointer to the desired job attributes. See dmapp_rma_attrs on
page 205.
actual_attrs
The actual job attributes.
DMAPP_RC_SUCCESS
An error occurred during initialization.
S–2446–51
211
5.3.4 dmapp_initialized
Test whether DMAPP has been initialized or not.
5.3.4.1 Synopsis
dmapp_return_t dmapp_initialized(OUT int *flag);
5.3.4.2 Parameters
flag
Set to 1 if DMAPP has been initialized, otherwise 0.
DMAPP_RC_SUCCESS
5.3.5 dmapp_finalize
The dmapp_finalize function synchronizes and cleans up DMAPP resources.
All DMAPP applications must call this function when it has finished using all
DMAPP functions.
5.3.5.1 Synopsis
dmapp_return_t dmapp_finalize(void);
DMAPP_RC_SUCCESS
5.3.6 dmapp_get_jobinfo
The dmapp_get_jobinfo function returns a pointer to the data structure
dmapp_jobinfo on page 204 which contains general information about the job.
5.3.6.1 Synopsis
dmapp_return_t dmapp_get_jobinfo(
OUT dmapp_jobinfo_t *info);
212
S–2446–51
5.3.6.2 Parameters
info
Returns a pointer to the current information about the job.
DMAPP_RC_SUCCESS
The input parameter is invalid.
5.3.7 dmapp_get_rma_attrs
The dmapp_get_rma_attrs function returns RMA attributes of a DMAPP
thread.
5.3.7.1 Synopsis
dmapp_return_t dmapp_get_rma_attrs(
OUT dmapp_rma_attrs_t *attrs);
5.3.7.2 Parameters
attrs
Current RMA attributes of the thread in local storage.
DMAPP_RC_SUCCESS
Input parameter is invalid.
5.3.8 dmapp_get_rma_attrs_ext
The dmapp_get_rma_attrs_ext function returns extended RMA attributes of
a DMAPP thread.
In order to facilitate forward compatibility using the extended capabilities
in the extended RMA attributes structures, the user should call
dmapp_get_rma_attrs_ext() in order to sample the default RMA
attributes and then modify the parameters they wish to change, before calling
dmapp_init_ext() or dmapp_set_rma_ttrs_ext().
S–2446–51
213
5.3.8.1 Synopsis
dmapp_return_t dmapp_get_rma_attrs_ext(
OUT dmapp_rma_attrs_ext_t *attrs);
5.3.8.2 Parameters
attrs
Current extended RMA attributes of the thread in local storage.
DMAPP_RC_SUCCESS
5.3.9 dmapp_get_version
Sets *version to the current DMAPP version. This value can then be decoded further
using DMAPP_GET_MAJOR, DMAPP_GET_MINOR, and DMAPP_GET_SVNREV
macros.
5.3.9.1 Synopsis
extern dmapp_return_t dmapp_get_version(OUT int *version);
5.3.9.2 Parameters
version
Value representing current DMAPP version.
DMAPP_RC_SUCCESS
5.3.10 dmapp_lock_acquire
This set of dmapp_lock_* interfaces provide a scalable, exclusive locking
mechanism. By default, 128 concurrent locks per rank are supported and this can be
increased to a maximum of 1023 by setting the DMAPP_MAX_LOCKS environment
variable.
214
S–2446–51
The dmapp_lock_acquire function acquires an exclusive lock for the
calling thread. If there is contention for the lock, the function will block waiting
until the lock is released. Once a lock is acquired, it can be released by calling
dmapp_lock_release or dmapp_lock_release_addr.
The scalable lock implementation requires a memory location of size
sizeof(dmapp_lock_desc_t) to be allocated in an exported memory segment
of a hosting PE. This memory location must be initialized to zero before its first use.
5.3.10.1 Synopsis
dmapp_return_t dmapp_lock_acquire(
IN
dmapp_lock_desc_t *lock_addr,
IN
dmapp_seg_desc_t *lock_seg,
IN
dmapp_pe_t lock_host,
IN
uint64_t flags,
INOUT dmapp_lock_handle_t *handle);
5.3.10.2 Parameters
S–2446–51
lock_addr
Memory lock location.
lock_seg
Segment descriptor of memory lock location.
lock_pe
Lock host PE.
flags
Modifier flags.
handle
Returns lock handle.
215
DMAPP_RC_SUCCESS
Locks acquired successfully.
DMAPP_RC_BUSY
Lock already held by this rank.
A resource error occurred.
DMAPP_RC_NO_SPACE
The lock request could not be completed due to insufficient
resources.
DMAPP_RC_TRANSACTION_ERROR
A transaction error occurred.
5.3.11 dmapp_lock_try
This function attempts to acquire an exclusive lock for the calling thread. On success
it returns DMAPP_RC_SUCCESS. If the lock is contended, returns immediately with
the status code of DMAPP_RC_NOT_DONE.
5.3.11.1 Synopsis
dmapp_return_t dmapp_lock_try(
IN
IN
dmapp_seg_desc_t *lock_seg,
IN
dmapp_pe_t lock_host,
IN
uint64_t flags,
INOUT dmapp_lock_handle_t *handle);
5.3.11.2 Parameters
216
lock_addr
Memory lock location.
lock_seg
Segment descriptor of memory lock location.
lock_pe
Lock host PE.
flags
Modifier flags.
handle
Returns allocated lock handle.
S–2446–51
DMAPP_RC_SUCCESS
Locks acquired successfully.
DMAPP_RC_NOT_DONE
Lock is contended.
DMAPP_RC_BUSY
Lock already held by this thread.
DMAPP_RC_NO_SPACE
resources.
5.3.12 dmapp_lock_release
This function releases a lock held by the calling process.
5.3.12.1 Synopsis
dmapp_return_t dmapp_lock_release(
IN
dmapp_lock_handle_t *lock_handle,
IN
uint64_t flags);
5.3.12.2 Parameters
lock_handle
Previously acquired lock handle.
flags
S–2446–51
Modifier flags.
217
DMAPP_RC_SUCCESS
Locks released successfully.
DMAPP_RC_NOT_FOUND
Lock not held by this rank.
DMAPP_RC_NO_SPACE
resources.
5.3.13 dmapp_lock_release_addr
This function releases a previously acquired DMAPP lock.
5.3.13.1 Synopsis
dmapp_return_t dmapp_lock_release_addr(
IN
IN
dmapp_pe_t *lock_host,
IN
uint64_t flags);
5.3.13.2 Parameters
218
lock_addr
Previously acquired lock memory location.
lock_host
Lock host PE
flags
Modifier flags.
S–2446–51
DMAPP_RC_SUCCESS
Locks released successfully.
DMAPP_RC_NOT_FOUND
Lock not held by this thread.
DMAPP_RC_NO_SPACE
resources.
5.3.14 dmapp_set_rma_attrs
The dmapp_set_rma_attrs function sets dynamic RMA attributes for a
DMAPP thread using local storage. This allows each thread to independently set
the RMA attributes.
5.3.14.1 Synopsis
dmapp_return_t dmapp_set_rma_attrs(
IN dmapp_rma_attrs_t *requested_attrs,
OUT dmapp_rma_attrs_t *actual_attrs);
5.3.14.2 Parameters
requested_attrs
Pointer to desired job attributes.
actual_attrs
Returns a pointer to the actual job attributes.
S–2446–51
219
DMAPP_RC_SUCCESS
One or more input parameters is invalid.
5.3.15 dmapp_set_rma_attrs_ext
The dmapp_set_rma_attrs_ext function sets dynamic, extended RMA
attributes for a DMAPP thread using local storage. This allows each thread to
independently set the extended RMA attributes.
5.3.15.1 Synopsis
dmapp_return_t dmapp_set_rma_attrs_ext(
IN dmapp_rma_attrs_ext_t *requested_attrs,
OUT dmapp_rma_attrs_ext_t *actual_attrs);
5.3.15.2 Parameters
requested_attrs
Pointer to desired job attributes.
actual_attrs
Returns a pointer to the actual job attributes.
DMAPP_RC_SUCCESS
5.3.16 dmapp_put_nb
The dmapp_put_nb function is a non-blocking explicit PUT. dmapp_put_nb
stores a contiguous block of data, starting at the address indicated by source_addr,
from local memory into a contiguous block at a remote address. The remote address
is specified by the triplet virtual address target_addr, segment descriptor target_seg,
and the target process target_pe. nelems specifies the number of elements of type
type to be transferred. The memory region defined by target_addr and nelems must
be within an exported memory segment of target_pe.
220
S–2446–51
5.3.16.1 Synopsis
dmapp_return_t dmapp_put_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type,
OUT dmapp_syncid_handle_t *syncid);
5.3.16.2 Parameters
target_addr
Pointer to the address of the target buffer.
target_seg
Pointer to the segment descriptor of the target buffer.
target_pe
The target processing element.
source_addr
Pointer to the address of the source buffer.
nelems
Number of elements to transfer.
type
The type of elements to transfer.
syncid
Pointer to the synchronization ID.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
The transaction request could not be completed due to insufficient
resources. To resolve this error, synchronize more often, or if
possible, increase the value for max_outstanding_nb in the
job attributes.
S–2446–51
221
5.3.17 dmapp_put_nbi
The dmapp_put_nbi function is a non-blocking implicit PUT. dmapp_put_nbi
stores a contiguous block of data, starting at the address indicated by source_addr,
from local memory into a contiguous block at a remote address. The remote address
is specified by the triplet virtual address target_addr, segment descriptor target_seg,
and the target process target_pe. nelems specifies the number of elements of type
type to be transferred. The memory region defined by target_addr and nelems must
be within an exported memory segment of target_pe.
5.3.17.1 Synopsis
dmapp_return_t dmapp_put_nbi(
IN void
*target_addr,
IN dmapp_seg_desc_t *target_seg,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.17.2 Parameters
target_addr
Pointer to the address of a target buffer.
target_seg
Pointer to a segment descriptor of a target buffer.
target_pe
source_addr
Address of the source buffer.
222
nelems
The number of elements to transfer.
type
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.18 dmapp_put
The dmapp_put function is a blocking PUT. dmapp_put stores a contiguous
block of data, starting at the address indicated by source_addr, from local memory
into a contiguous block at a remote address. The remote address is specified by the
triplet virtual address target_addr, segment descriptor target_seg, and the target
process target_pe. nelems specifies the number of elements of type type to be
transferred. The memory region defined by target_addr and nelems must be within an
exported memory segment of target_pe.
5.3.18.1 Synopsis
dmapp_return_t dmapp_put(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type);
S–2446–51
223
5.3.18.2 Parameters
target_addr
target_seg
target_pe
source_addr
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
A transaction error has occurred.
5.3.19 dmapp_put_flag
The dmapp_put_flag function is a blocking PUT + FLAG.
224
S–2446–51
5.3.19.1 Synopsis
dmapp_return_t dmapp_put_flag(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type,
IN void
*target_flag
IN uint64_t
flag_value);
5.3.19.2 Parameters
target_addr
target_seg
target_pe
source_addr
nelems
type
target_flag
Address of the 8-byte target flag.
flag_value
64-bit user defined flag value.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
S–2446–51
225
5.3.20 dmapp_put_flag_nb
The dmapp_put_flag_nb function is a non-blocking PUT + FLAG. The
target_flag address must reside within the same memory segment as that of the
target_addr buffer, as identified by the target_seg parameter.
5.3.20.1 Synopsis
dmapp_return_t dmapp_put_flag_nb(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type,
IN void
*target_flag
IN uint64_t
flag_value,
OUT dmapp_sync_id_handle_t *syncid);
5.3.20.2 Parameters
target_addr
target_seg
target_pe
source_addr
nelems
type
target_flag
226
flag_value
sync_id
Synchronization ID.
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.21 dmapp_put_flag_nbi
The dmapp_put_flag_nbi function is a non-blocking implicit PUT + FLAG.
The target_flag address must reside within the same memory segment as that of the
target_addr buffer, as identified by the target_seg parameter.
5.3.21.1 Synopsis
dmapp_return_t dmapp_put_flag_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type,
IN void
*target_flag
IN uint64_type_t
flag_value);
S–2446–51
227
5.3.21.2 Parameters
target_addr
target_seg
target_pe
source_addr
nelems
type
target_flag
flag_value
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
228
S–2446–51
5.3.22 dmapp_get_nb
The dmapp_get_nb function is a non-blocking explicit GET. dmapp_get_nb
loads from a contiguous block of data starting from a remote source address and
returning the data into a contiguous block starting at address target_addr in local
memory. The remote address is specified by the triplet virtual address source_addr,
segment descriptor source_seg and source process source_pe. The nelems parameter
specifies the number of elements of type type to transfer. The memory region
described by the remote address and nelems must reside in an exported memory of
source_pe. Note that zero-length GETs are not supported.
5.3.22.1 Synopsis
dmapp_return_t dmapp_get_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN uint64_t
nelems,
IN dmapp_type_t
type,
5.3.22.2 Parameters
target_addr
source_addr
S–2446–51
source_seg
Pointer to a segment descriptor of a source buffer.
source_pe
The source processing element.
nelems
type
syncid
Pointer to a synchronization ID.
229
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.23 dmapp_get_nbi
The dmapp_get_nbi function is a non-blocking implicit GET. dmapp_get_nbi
loads from a contiguous block of data starting from a remote source address and
returning the data into a contiguous block starting at address target_addr in local
memory. The remote address is specified by the triplet virtual address source_addr,
segment descriptor source_seg and source process source_pe. The nelems parameter
specifies the number of elements of type type to transfer. The memory region
described by the remote address and nelems must reside in an exported memory of
source_pe. Note that zero-length GETs are not supported.
5.3.23.1 Synopsis
dmapp_return_t dmapp_get_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t *source_seg,
IN dmapp_pe_t
source_pe,
IN uint64_t
nelems,
IN dmapp_type_t
type);
230
S–2446–51
5.3.23.2 Parameters
target_addr
source_addr
source_seg
Pointer to the segment descriptor of the source buffer.
source_pe
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.24 dmapp_get
The dmapp_get function is a blocking GET. dmapp_get loads from a contiguous
block of data starting from a remote source address and returning the data into a
contiguous block starting at address target_addr in local memory. The remote address
is specified by the triplet virtual address source_addr, segment descriptor source_seg
and source process source_pe. The nelems parameter specifies the number of
elements of type type to transfer. The memory region described by the remote address
and nelems must reside in an exported memory of source_pe. Note that zero-length
GETs are not supported.
S–2446–51
231
5.3.24.1 Synopsis
dmapp_return_t dmapp_get(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.24.2 Parameters
target_addr
source_addr
source_seg
source_pe
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
232
S–2446–51
5.3.25 dmapp_get_flag_nbi
The dmapp_get_flag_nbi function is a non-blocking implicit GET + FLAG.
A 32-bit response flag word is appended to the returned data when it is written
into the local memory domain. A DMAPP client can poll on the location where
this flag will be written to determine when the requested data has been received.
The flag word immediately follows the last dword of fetched data. Its value is
DMAPP_GET_FLAG_VALUE.
Note that only a non-blocking implicit version of this function is supported. Some
restrictions apply. The target_flag address must reside within the same memory
segment as that of the target_addr buffer, as identified by the target_seg parameter.
The data plus flag cannot cross a local cacheline boundary. Data cannot cross a
remote cacheline boundary. Local and remote address and length need to be Dword
aligned. Type DMAPP_BYTE is not supported.
5.3.25.1 Synopsis
dmapp_return_t dmapp_get_flag_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.25.2 Parameters
target_addr
source_addr
S–2446–51
source_seg
source_pe
nelems
type
233
DMAPP_RC_SUCCESS
DMAPP_RC_ALIGNMENT_ERROR
Source or target buffer or length not properly aligned.
DMAPP_RC_NO_SPACE
job attributes.
5.3.26 dmapp_iput_nb
The dmapp_iput_nb function is a non-blocking explicit strided PUT.
5.3.26.1 Synopsis
dmapp_return_t dmapp_iput_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN ptrdiff_t
tst,
IN ptrdiff_t
sst,
IN uint64_t
nelems,
IN dmapp_type_t
type,
234
S–2446–51
5.3.26.2 Parameters
target_addr
target_seg
Pointer to the segment descriptor of a target buffer.
target_pe
Target processing element.
source_addr
tst
Target stride (>= 1).
sst
Source stride (>= 1).
nelems
type
Type of elements to transfer.
syncid
Returns the synchronization ID.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.27 dmapp_iput_nbi
The dmapp_iput_nbi function is a non-blocking implicit strided PUT.
S–2446–51
235
5.3.27.1 Synopsis
dmapp_return_t dmapp_iput_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN ptrdiff_t
tst,
IN ptrdiff_t
sst,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.27.2 Parameters
target_addr
target_seg
target_pe
source_addr
tst
Target stride.
sst
Source stride.
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.28 dmapp_iput
The dmapp_iput function is a blocking strided PUT.
236
S–2446–51
5.3.28.1 Synopsis
dmapp_return_t dmapp_iput(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN ptrdiff_t
tst,
IN ptrdiff_t
sst,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.28.2 Parameters
target_addr
target_seg
target_pe
source_addr
tst
Target stride.
sst
Source stride.
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
S–2446–51
237
5.3.29 dmapp_iget_nb
The dmapp_iget_nb function is a non-blocking explicit strided GET.
5.3.29.1 Synopsis
dmapp_return_t dmapp_iget_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN ptrdiff_t
tst,
IN ptrdiff_t
sst,
IN uint64_t
nelems,
IN dmapp_type_t
type,
5.3.29.2 Parameters
target_addr
source_addr
238
source_seg
source_pe
Source processing element.
tst
Target stride.
sst
Source stride.
nelems
type
syncid
S–2446–51
DMAPP_RC_SUCCESS
The source or target buffer or length is not properly Dword (4 byte)
aligned.
DMAPP_RC_NO_SPACE
job attributes.
5.3.30 dmapp_iget_nbi
The dmapp_iget_nbi function is a non-blocking implicit strided GET.
5.3.30.1 Synopsis
dmapp_return_t dmapp_iget_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN ptrdiff_t
tst,
IN ptrdiff_t
sst,
IN uint64_t
nelems,
IN dmapp_type_t
type);
S–2446–51
239
5.3.30.2 Parameters
target_addr
source_addr
source_seg
source_pe
tst
Target stride.
sst
Source stride.
nelems
type
DMAPP_RC_SUCCESS
aligned.
DMAPP_RC_NO_SPACE
job attributes.
5.3.31 dmapp_iget
The dmapp_iget function is a blocking strided GET.
240
S–2446–51
5.3.31.1 Synopsis
dmapp_return_t dmapp_iget(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN ptrdiff_t
tst,
IN ptrdiff_t
sst,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.31.2 Parameters
target_addr
source_addr
S–2446–51
source_seg
source_pe
tst
Target stride (>=1).
sst
Source stride (>=1).
nelems
type
241
DMAPP_RC_SUCCESS
Source or target buffer or length not properly Dword (4 byte) aligned.
DMAPP_RC_NO_SPACE
job attributes.
5.3.32 dmapp_ixput_nb
The dmapp_ixput_nb function is a non-blocking explicit Indexed PUT.
PUT contiguous data starting at address source_addr in local memory to a remote
target address using offsets specified by the tidx array. The remote address is
specified by the triplet virtual address target_addr, segment descriptor target_seg
and target process target_pe. nelems specifies the number of elements of type type
to be transferred. Offsets in the tidx array are in units of type. The memory region
described by the remote address, tidx and nelems must reside in an exported memory
of target_pe.
5.3.32.1 Synopsis
dmapp_return_t dmapp_ixput_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN ptrdiff_t
*tidx,
IN uint64_t
nelems,
IN dmapp_type_t
type,
242
S–2446–51
5.3.32.2 Parameters
target_addr
target_seg
target_pe
source_addr
tidx
Pointer to an array of positive offsets into target buffer. Offsets in the
tidx array are in units of type. Each element to be transferred i, where
i=1,nelems is transferred to target_addr + tidx(i). Note that the
length of the array tidx should be equal to nelems, or a segmentation
fault may occur.
nelems
Number of elements to be transferred.
type
Type of elements to be transferred.
syncid
Synchronization ID.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.33 dmapp_ixput_nbi
The dmapp_ixput_nbi function is a non-blocking implicit Indexed PUT.
S–2446–51
243
5.3.33.1 Synopsis
dmapp_return_t dmapp_ixput_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN ptrdiff_t
*tidx,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.33.2 Parameters
target_addr
target_seg
target_pe
source_addr
tidx
Pointer to an array of positive offsets into the target buffer.
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.34 dmapp_ixput
The dmapp_ixput function is a blocking Indexed PUT.
244
S–2446–51
5.3.34.1 Synopsis
dmapp_return_t dmapp_ixput(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN void
*source_addr,
IN ptrdiff_t
*tidx,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.34.2 Parameters
target_addr
target_seg
Pointer to a segment descriptor of the target buffer.
target_pe
source_addr
tidx
Pointer to an array of positive offsets into the target buffer.
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.35 dmapp_ixget_nb
The dmapp_ixget_nb function is a non-blocking explicit Indexed GET.
S–2446–51
245
GET data starting from a remote source address using offsets specified by the sidx
array and returning the data into a contiguous block starting at address target_addr
in local memory. The remote address is specified by the triplet virtual address
source_addr, segment descriptor source_seg and source process source_pe. nelems
specifies the number of elements of type type to be transferred. Offsets in the sidx
array are in units of type. The memory region described by the remote address, sidx
and nelems must reside in an exported memory of source_pe.
5.3.35.1 Synopsis
dmapp_return_t dmapp_ixget_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN ptrdiff_t
*sidx,
IN uint64_t
nelems,
IN dmapp_type_t
type,
5.3.35.2 Parameters
target_addr
source_addr
246
source_seg
source_pe
sidx
Pointer to an array of positive offsets into the source buffer.
nelems
type
Type of elements to transfer. The DMAPP_BYTE type is not
supported.
syncid
S–2446–51
DMAPP_RC_SUCCESS
aligned.
DMAPP_RC_NO_SPACE
job attributes.
5.3.36 dmapp_ixget_nbi
The dmapp_ixget_nbi function is a non-blocking implicit Indexed GET.
5.3.36.1 Synopsis
dmapp_return_t dmapp_ixget_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN ptrdiff_t
*sidx,
IN uint64_t
nelems,
IN dmapp_type_t
type);
S–2446–51
247
5.3.36.2 Parameters
target_addr
source_addr
source_seg
source_pe
sidx
nelems
type
Type of elements to transfer. The type DMAPP_BYTE is not
supported.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.37 dmapp_ixget
The dmapp_ixget function is a blocking indexed GET.
248
S–2446–51
5.3.37.1 Synopsis
dmapp_return_t dmapp_ixget(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN ptrdiff_t
*sidx,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.37.2 Parameters
target_addr
source_addr
S–2446–51
source_seg
source_pe
sidx
nelems
type
Type of elements to transfer. The DMAPP_BYTE type is not
supported.
249
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.38 dmapp_put_ixpe_nb
The dmapp_put_ixpe_nb function is a non-blocking explicit PUT with indexed
PE stride. It delivers data starting at source_addr in local memory to a list of target
PEs target_pe_list starting at target_addr in their memories. nelems specifies the
number of elements of type type to be PUT into each target PE. When the transfer is
complete, each target PE will have a copy of the contents of the original source buffer.
The address range specified by target_addr and nelems must reside in an exported,
symmetric memory segment in each PE in target_pe_list.
This function is not a collective operation. It is best used when a small amount of data
needs to be transferred to a set of PEs.
5.3.38.1 Synopsis
dmapp_return_t dmapp_put_ixpe_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
*target_pe_list,
IN uint32_t
num_target_pes,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type,
250
S–2446–51
5.3.38.2 Parameters
target_addr
target_seg
target_pe_list
Pointer to the list of target processing elements.
num_target_pes
Number of target processing elements.
source_addr
nelems
type
syncid
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.39 dmapp_put_ixpe_nbi
The dmapp_put_ixpe_nbi function is a non-blocking implicit PUT with indexed
PE stride.
It delivers data starting at source_addr in local memory to a list of target PEs
target_pe_list starting at target_addr in their memories. nelems specifies the number
of elements of type type to be PUT into each target PE. When the transfer is
The address range specified by target_addr and nelems must reside in an exported
memory segment in each PE in target_pe_list.
S–2446–51
251
5.3.39.1 Synopsis
dmapp_return_t dmapp_put_ixpe_nbi(
IN void
*target_addr,
IN dmapp_pe_t
*target_pe_list,
IN uint32_t
num_target_pes,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.39.2 Parameters
IN target_addr
target_seg
target_pe_list
Pointer to a list of target processing elements.
num_target_pes
source_addr
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
252
S–2446–51
5.3.40 dmapp_put_ixpe
The dmapp_put_ixpe function is a blocking PUT with indexed PE stride.
It delivers data starting at source_addr in local memory to a list of target PEs
of elements of type type to be PUT into each target PE. When the transfer is
The address range specified by target and nelems must reside in an exported memory
segment in each PE in target_pe_list. The remote address is specified by the target
virtual address target_addr and the segment descriptor target_seg. The address range
specified by target_addr and nelems must reside in an exported, symmetric memory
segment in each PE in target_pe_list.
5.3.40.1 Synopsis
dmapp_return_t dmapp_put_ixpe(
IN void
*target_addr,
IN dmapp_pe_t
*target_pe_list,
IN uint32_t
num_target_pes,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.40.2 Parameters
target_addr
target_seg
target_pe_list
num_target_pes
source_addr
S–2446–51
nelems
type
253
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.41 dmapp_scatter_ixpe_nb
The dmapp_scatter_ixpe_nb function is a non-blocking explicit scatter
with indexed PE stride. The function delivers data to a list of target PEs in the
of elements of type to be PUT into each target PE. A remote PE at some index I in the
target_pe_list will receive elements I * nelems to (I+1)*nelems - 1.
Unlike the dmapp_put_ixpe function, the source_addr array specifies a
num_target_pes * nelems * sizeof (type) array.
The remote address is specified by the virtual address target_addr and segment
descriptor target_seg. The address range specified by target_addr and nelems must
reside in an exported, symmetric memory segment in each PE in target_pe_list.
5.3.41.1 Synopsis
dmapp_return_t dmapp_scatter_ixpe_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
*target_pe_list,
IN uint32_t
num_target_pes,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type,
254
S–2446–51
5.3.41.2 Parameters
target_addr
Pointer to an address of the target buffer.
target_seg
Pointer to a segment descriptor of the target buffer.
target_pe_list
Pointer to a list of target processing elements.
num_target_pes
source_addr
nelems
type
syncid
DMAPP_RC_SUCCESS
A resource error has occurred.
DMAPP_RC_NO_SPACE
job attributes.
5.3.42 dmapp_scatter_ixpe_nbi
The dmapp_scatter_ixpe_nbi function is a non-blocking implicit scatter
with indexed PE stride. The function delivers data to a list of target PEs in the
of elements of type to be PUT into each target PE. A remote PE at some index I in the
target_pe_list will receive elements I * nelems to (I+1) * nelems - 1.
S–2446–51
255
5.3.42.1 Synopsis
dmapp_return_t dmapp_scatter_ixpe_nbi(
IN void
*target_addr,
IN dmapp_pe_t
*target_pe_list,
IN uint32_t
num_target_pes,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.42.2 Parameters
target_addr
target_seg
target_pe_list
num_target_pes
source_addr
256
nelems
type
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.43 dmapp_scatter_ixpe
The dmapp_scatter_ixpe function is a blocking scatter with indexed PE stride.
The function delivers data to a list of target PEs in the target_pe_list starting at
target_addr in their memories. nelems specifies the number of elements of type
to be PUT into each target PE.
A remote PE at some index I in the target_pe_list will receive elements I * nelems to
(I+1) *nelems-1. The address range specified by target_addr and nelems must reside
in an exported memory segment in each PE specified in target_pe_list.
5.3.43.1 Synopsis
dmapp_return_t dmapp_scatter_ixpe(
IN void
*target_addr,
IN dmapp_pe_t
*target_pe_list,
IN uint32_t
num_target_pes,
IN void
*source_addr,
IN uint64_t
nelems,
IN dmapp_type_t
type);
S–2446–51
257
5.3.43.2 Parameters
target_addr
target_seg
target_pe_list
num_target_pes
source_addr
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
258
S–2446–51
5.3.44 dmapp_gather_ixpe_nb
The dmapp_gather_ixpe_nb function is a non-blocking explicit gather with
indexed PE stride. Gather data starting at source_addr from the list of PEs specified
by source_pe_list and concatenate the returned data in a buffer in local memory
specified by target_addr. nelems specifies the number of elements of type collected
from each PE.
The address range specified by source_addr and nelems must reside in an exported,
symmetric memory segment in each PE in source_pe_list.
5.3.44.1 Synopsis
dmapp_return_t dmapp_gather_ixpe_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
*source_pe_list,
IN uint32_t
num_source_pes,
IN uint64_t
nelems,
IN dmapp_type_t
type,
5.3.44.2 Parameters
target_addr
source_addr
source_seg
source_pe_list
Pointer to the list of source processing elements.
num_source_pes
Number of source processing elements.
S–2446–51
nelems
type
syncid
Returns a pointer to the synchronization ID.
259
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.45 dmapp_gather_ixpe_nbi
The dmapp_gather_ixpe_nbi function is a non-blocking implicit gather with
indexed PE stride. Gather data starting at source_addr from the list of PEs specified
from each PE.
symmetric memory segment in each PE in source_pe_list.
5.3.45.1 Synopsis
dmapp_return_t dmapp_gather_ixpe_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
*source_pe_list,
IN uint32_t
num_source_pes,
IN uint64_t
nelems,
IN dmapp_type_t
type);
260
S–2446–51
5.3.45.2 Parameters
target_addr
source_addr
source_seg
source_pe_list
num_source_pes
nelems
type
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.46 dmapp_gather_ixpe
The dmapp_gather_ixpe function is a blocking gather with indexed processing
element stride. Gather data starting at source_addr from the list of PEs specified
from each PE.
S–2446–51
261
symmetric memory segment in each PE listed in source_pe_list.
5.3.46.1 Synopsis
dmapp_return_t dmapp_gather_ixpe(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
*source_pe_list,
IN uint32_t
num_source_pes,
IN uint64_t
nelems,
IN dmapp_type_t
type);
5.3.46.2 Parameters
target_addr
source_addr
source_seg
source_pe_list
num_source_pes
262
nelems
type
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.47 dmapp_aadd_qw_nb
The dmapp_aadd_qw_nb function is a non-blocking explicit atomic ADD.
5.3.47.1 Synopsis
dmapp_return_t dmapp_aadd_qw_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN int64_t
operand,
5.3.47.2 Parameters
target_addr
Pointer to the address of the target buffer (for Qword only).
S–2446–51
target_seg
Pointer to the segment descriptor for the target buffer.
target_pe
operand
Value to be added.
syncid
263
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.48 dmapp_aadd_qw_nbi
The dmapp_aadd_qw_nbi function is a non-blocking implicit atomic ADD.
5.3.48.1 Synopsis
dmapp_return_t dmapp_aadd_qw_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.48.2 Parameters
target_addr
Pointer to the address of the target buffer (Qword only).
264
target_seg
target_pe
operand
Value to be added.
S–2446–51
DMAPP_RC_SUCCESS
Target buffer not properly (Qword, 8 byte) aligned.
DMAPP_RC_NO_SPACE
job attributes.
5.3.49 dmapp_aadd_qw
The dmapp_aadd_qw function is a blocking atomic ADD.
5.3.49.1 Synopsis
dmapp_return_t dmapp_aadd_qw(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.49.2 Parameters
target_addr
S–2446–51
target_seg
target_pe
operand
Value to be added.
265
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.50 dmapp_aand_qw_nb
The dmapp_aand_qw_nb function is a non-blocking explicit atomic AND.
5.3.50.1 Synopsis
dmapp_return_t dmapp_aand_qw_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN int64_t
operand,
5.3.50.2 Parameters
target_addr
266
target_seg
target_pe
operand
Operand for the AND operation.
syncid
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.51 dmapp_aand_qw_nbi
The dmapp_aand_qw_nbi function is a non-blocking implicit atomic AND.
5.3.51.1 Synopsis
dmapp_return_t dmapp_aand_qw_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.51.2 Parameters
target_addr
S–2446–51
target_seg
target_pe
operand
267
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.52 dmapp_aand_qw
The dmapp_aand_qw function is a blocking atomic AND.
5.3.52.1 Synopsis
dmapp_return_t dmapp_aand_qw(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.52.2 Parameters
target_addr
Pointer the address of the target buffer (Qword only).
268
target_seg
target_pe
operand
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.53 dmapp_aor_qw_nb
The dmapp_aor_qw_nb function is a non-blocking explicit atomic OR.
5.3.53.1 Synopsis
dmapp_return_t dmapp_aor_qw_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN int64_t
operand,
5.3.53.2 Parameter
target_addr
S–2446–51
target_seg
target_pe
operand
Operand for the OR operation.
syncid
269
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.54 dmapp_aor_qw_nbi
The dmapp_aor_qw_nbi function is a non-blocking implicit atomic OR.
5.3.54.1 Synopsis
dmapp_return_t dmapp_aor_qw_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.54.2 Parameter
target_addr
270
target_seg
target_pe
operand
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.55 dmapp_aor_qw
The dmapp_aor_qw function is a blocking atomic OR.
5.3.55.1 Synopsis
dmapp_return_t dmapp_aor_qw(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.55.2 Parameters
target_addr
S–2446–51
target_seg
target_pe
operand
271
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.56 dmapp_axor_qw_nb
The dmapp_axor_qw_nb function is a non-blocking explicit atomic XOR.
5.3.56.1 Synopsis
dmapp_return_t dmapp_axor_qw_nb(
IN void
*target_addr,
IN dmapp_seg_desc_t
*target_seg,
IN dmapp_pe_t
target_pe,
IN int64_t
operand,
5.3.56.2 Parameters
target_addr
272
target_seg
target_pe
operand
Operand for the XOR operation.
syncid
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.57 dmapp_axor_qw_nbi
The dmapp_axor_qw_nbi function is a non-blocking implicit atomic XOR.
5.3.57.1 Synopsis
dmapp_return_t dmapp_axor_qw_nbi(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.57.2 Parameter
target_addr
S–2446–51
target_seg
target_pe
operand
Operand for the XOR operation.
273
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.58 dmapp_axor_qw
The dmapp_axor_qw function is a blocking atomic XOR.
5.3.58.1 Synopsis
dmapp_return_t dmapp_axor_qw(
IN void
*target_addr,
IN dmapp_pe_t
target_pe,
IN int64_t
operand);
5.3.58.2 Parameters
target_addr
274
target_seg
target_pe
operand
Operand for an XOR operation.
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.59 dmapp_afadd_qw_nb
The dmapp_afadd_qw_nb function is a non-blocking explicit atomic FADD.
5.3.59.1 Synopsis
dmapp_return_t dmapp_afadd_qw_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN int64_t
operand,
S–2446–51
275
5.3.59.2 Parameters
target_addr
Pointer to the address of the target buffer where the result is returned
(Qword only).
source_addr
Pointer to the address of the source buffer (Qword only).
source_seg
source_pe
operand
Operand for the FADD operation.
syncid
DMAPP_RC_SUCCESS
5.3.60 dmapp_afadd_qw_nbi
The dmapp_afadd_qw_nbi function is a non-blocking implicit atomic FADD.
5.3.60.1 Synopsis
dmapp_return_t dmapp_afadd_qw_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
276
S–2446–51
5.3.60.2 Parameters
target_addr
Pointer to the address of a target buffer where the result is returned
(Qword only).
source_addr
source_seg
source_pe
operand
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.61 dmapp_afadd_qw
The dmapp_afadd_qw function is a blocking atomic FADD.
5.3.61.1 Synopsis
dmapp_return_t dmapp_afadd_qw(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
S–2446–51
277
5.3.61.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
operand
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.62 dmapp_afand_qw_nb
The dmapp_afand_qw_nb function is a non-blocking explicit atomic FAND.
278
S–2446–51
5.3.62.1 Synopsis
dmapp_return_t dmapp_afand_qw_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN int64_t
operand,
5.3.62.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
operand
Operand for the FAND operation.
syncid
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
S–2446–51
279
5.3.63 dmapp_afand_qw_nbi
The dmapp_afand_qw_nbi function is a non-blocking implicit atomic FAND.
5.3.63.1 Synopsis
dmapp_return_t
dmapp_afand_qw_nbi(
IN void
IN void
IN dmapp_seg_desc_t
IN dmapp_pe_t
IN int64_t
*target_addr,
*source_addr,
*source_seg,
source_pe,
operand);
5.3.63.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
Segment descriptor of source buffer.
source_pe
operand
Operand for an FAND operation.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
280
S–2446–51
5.3.64 dmapp_afand_qw
The dmapp_afand_qw function is a blocking atomic FAND.
5.3.64.1 Synopsis
dmapp_return_t dmapp_afand_qw(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
5.3.64.2 Parameters
target_add
(Qword only).
source_addr
source_seg
source_pe
operand
Operand for an FAND operation.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
S–2446–51
281
5.3.65 dmapp_afxor_qw_nb
The dmapp_afxor_qw_nb function is a non-blocking explicit atomic FXOR.
5.3.65.1 Synopsis
dmapp_return_t dmapp_afxor_qw_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN int64_t
operand,
5.3.65.2 Parameters
target_addr
(Qword only).
source_addr
282
source_seg
source_pe
operand
Operand for an FXOR operation.
syncid
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.66 dmapp_afxor_qw_nbi
The dmapp_afxor_qw_nbi function is a non-blocking implicit atomic FXOR.
5.3.66.1 Synopsis
dmapp_return_t dmapp_afxor_qw_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
5.3.66.2 Parameters
target_addr
(Qword only).
source_addr
S–2446–51
source_seg
source_pe
operand
283
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.67 dmapp_afxor_qw
The dmapp_afxor_qw function is a blocking atomic FXOR.
5.3.67.1 Synopsis
dmapp_return_t dmapp_afxor_qw(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
5.3.67.2 Parameters
target_addr
(Qword only).
source_addr
284
source_seg
source_pe
operand
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.68 dmapp_afor_qw_nb
The dmapp_afor_qw_nb function is a non-blocking explicit atomic FOR.
5.3.68.1 Synopsis
dmapp_return_t dmapp_afor_qw_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN int64_t
operand,
S–2446–51
285
5.3.68.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
operand
Operand for a FOR operation.
syncid
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.69 dmapp_afor_qw_nbi
The dmapp_afor_qw_nbi function is a non-blocking implicit atomic FOR.
286
S–2446–51
5.3.69.1 Synopsis
dmapp_return_t dmapp_afor_qw_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
5.3.69.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
operand
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.70 dmapp_afor_qw
The dmapp_afor_qw function is a blocking atomic FOR.
S–2446–51
287
5.3.70.1 Synopsis
dmapp_return_t dmapp_afor_qw(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
operand);
5.3.70.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
operand
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
288
S–2446–51
5.3.71 dmapp_afax_qw_nb
The dmapp_afax_qw_nb function is a non-blocking explicit atomic FAX.
5.3.71.1 Synopsis
dmapp_return_t dmapp_afax_qw_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN int64_t
andMask,
IN int64_t
xorMask,
5.3.71.2 Parameters
target_addr
(Qword only).
source_addr
S–2446–51
source_seg
source_pe
andMask
Mask for an AND operation.
xorMask
Mask for an XOR operation.
syncid
289
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.72 dmapp_afax_qw_nbi
The dmapp_afax_qw_nbi function is a non-blocking implicit atomic FAX.
5.3.72.1 Synopsis
dmapp_return_t dmapp_afax_qw_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
andMask,
IN int64_t
xorMask);
290
S–2446–51
5.3.72.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
andMask
xorMask
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.73 dmapp_afax_qw
The dmapp_afax_qw function is a blocking atomic FAX.
S–2446–51
291
5.3.73.1 Synopsis
dmapp_return_t dmapp_afax_qw(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
andMask,
IN int64_t
xorMask);
5.3.73.2 Parameters
target_addr
(Qword only).
source_addr
292
source_seg
source_pe
andMask
xorMask
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.74 dmapp_acswap_qw_nb
The dmapp_acswap_qw_nb function is a non-blocking explicit atomic CSWAP.
5.3.74.1 Synopsis
dmapp_return_t dmapp_acswap_qw_nb(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_seg_desc_t
*source_seg,
IN dmapp_pe_t
source_pe,
IN int64_t
comperand,
IN int64_t
swaperand,
S–2446–51
293
5.3.74.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
comperand
Operand against which to compare.
swaperand
Operand to swap in.
syncid
Pointer to a synchronization ID.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.75 dmapp_acswap_qw_nbi
The dmapp_acswap_qw_nbi function is a non-blocking implicit atomic CSWAP.
294
S–2446–51
5.3.75.1 Synopsis
dmapp_return_t dmapp_acswap_qw_nbi(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
comperand,
IN int64_t
swaperand);
5.3.75.2 Parameters
target_addr
(Qword only).
source_addr
source_seg
source_pe
comperand
swaperand
Operand to swap in.
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
S–2446–51
295
5.3.76 dmapp_acswap_qw
The dmapp_acswap_qw function is a blocking atomic CSWAP.
5.3.76.1 Synopsis
dmapp_return_t dmapp_acswap_qw(
IN void
*target_addr,
IN void
*source_addr,
IN dmapp_pe_t
source_pe,
IN int64_t
comperand,
IN int64_t
swaperand);
5.3.76.2 Parameters
target_addr
(Qword only).
source_addr
296
source_seg
source_pe
comperand
swaperand
Operand to swap in.
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
job attributes.
5.3.77 dmapp_amo_flush
dmapp_amo_flush ensures that the AMO cache is flushed so data is coherent in
memory.
5.3.77.1 Synopsis
extern dmapp_return_t
dmapp_amo_flush(
IN dmapp_pe_t
IN uint64_t
target_pe,
flags);
5.3.77.2 Parameters
S–2446–51
target_pe
Target PE.
flags
Modifier flags.
297
DMAPP_RC_SUCCESS
DMAPP_RC_NOT_SUPPORTED
Operation not supported.
5.3.78 dmapp_amo_put[_nb | _nbi]
dmapp_amo_put is a generic blocking atomic put function.
dmapp_amo_put_nb is a generic non-blocking atomic put function and
dmapp_amo_put_nbi is a generic non-blocking implicit atomic put function.
The specific AMO to be performed and operands are defined in the dmapp_amo_t
structure. These functions have the same parameter definitions and possible return
values. For brevity, only the blocking form synopsis is shown below.
5.3.78.1 Synopsis
dmapp_amo_put(IN void
IN dmapp_seg_desc_t
IN dmapp_pe_t
IN dmapp_amo_t
*target_addr,
*target_seg,
target_pe,
*amo);
5.3.78.2 Parameters
*target_addr
Pointer to the target buffer.
*target_seg
Segment descriptor for target buffer.
298
target_pe
Target PE.
*amo
Pointer to amo descriptor.
S–2446–51
DMAPP_RC_SUCCESS
Target buffer not properly aligned.
DMAPP_RC_NO_SPACE
resources; increase max_outstanding_nb or sync more
frequently.
5.3.79 dmapp_amo_get[_nb|_nbi]
dmapp_amo_get is a generic blocking atomic get function.
dmapp_amo_get_nb is a generic non-blocking atomic get function and
dmapp_amo_get_nbi is a generic non-blocking implicit atomic get function.
The specific AMO to be performed and operands are defined in the dmapp_amo_t
structure. These functions have the same parameter definitions and possible return
values. For brevity, only the blocking form synopsis is shown below.
Result is returned to target_addr. source refers to operands.
5.3.79.1 Synopsis
dmapp_amo_get(IN void
IN void
IN dmapp_seg_desc_t
IN dmapp_pe_t
IN dmapp_amo_t
S–2446–51
*target_addr,
*source_addr,
*source_seg,
source_pe,
*amo);
299
5.3.79.2 Parameters
*target_addr
Pointer to the target buffer.
*source_addr
Pointer to the source buffer.
*source_seg
Segment descriptor for source buffer.
source_pe
Source PE.
*amo
Pointer to amo descriptor.
DMAPP_RC_SUCCESS
Target buffer not properly aligned.
DMAPP_RC_NO_SPACE
resources; increase max_outstanding_nb or sync more
frequently.
5.3.80 dmapp_syncid_test
The dmapp_syncid_test function tests syncid for completion. It sets flag to 1 if
the remote memory accesses associated with syncid are globally visible in the system.
If the RMA request associated with the syncid has not completed, flag is set to 0.
5.3.80.1 Synopsis
dmapp_return_t dmapp_syncid_test(
INOUT dmapp_syncid_handle_t *syncid,
OUT
int
*flag);
300
S–2446–51
5.3.80.2 Parameters
syncid
Pointer to the syncid to test for completion.
flag
Pointer to the flag indicating global visibility.
DMAPP_RC_SUCCESS
5.3.81 dmapp_syncid_wait
The dmapp_syncid_wait function is a wait for completion of request associated
with syncid.
5.3.81.1 Synopsis
dmapp_return_t dmapp_syncid_wait(
INOUT dmapp_syncid_handle_t *syncid);
5.3.81.2 Parameters
syncid
The syncid to test for completion.
DMAPP_RC_SUCCESS
S–2446–51
301
5.3.82 dmapp_gsync_test
The dmapp_gsync_test function is a test for completion of issued non-blocking
implicit requests. It sets flag to 1 if remote memory accesses associated with
previously issued non-blocking implicit RMA requests are globally visible in the
system. Otherwise, flag is set to 0.
5.3.82.1 Synopsis
dmapp_return_t dmapp_gsync_test(
OUT int *flag);
5.3.82.2 Parameters
flag
Pointer to a flag indicating global visibility.
DMAPP_RC_SUCCESS
5.3.83 dmapp_gsync_wait
The dmapp_gsync_wait function forces a wait for completion of
issued non-blocking implicit requests. This is the blocking version of
dmapp_gsync_test. The function only returns when all remote memory accesses
associated with previously issued non-blocking implicit RMA requests are globally
visible in the system.
5.3.83.1 Synopsis
dmapp_return_t dmapp_gsync_wait(void);
DMAPP_RC_SUCCESS
302
S–2446–51
5.3.84 dmapp_c_pset_create
5.3.84.1 Synopsis
dmapp_return_t dmapp_c_pset_create(
IN dmapp_c_pset_desc_t *pdesc,
IN uint64_t identifier,
IN uint64_t modes,
IN dmapp_c_pset_attrs_t *attrs,
OUT dmapp_c_pset_handle_t *pset_handle);
5.3.84.2 Parameters
pdesc
Pointer to a dmapp_c_pset_desc_t structure previously
initialized to describe the ranks contained in the pset.
identifier
Unique, nonzero 64-bit identifier to be associated with this pset.
The same value must be supplied by all ranks creating the pset.
modes
Specifies additional information about the pset.
Concatenate operations on a are allowed when the
DMAPP_C_PSET_MODE_CONCAT mode bit is set. Use
DMAPP_C_PSET_MODE_FAULT_TOLERANT when collective
algorithms are tolerant to network faults (may be slower
than non-fault tolerant algorithms). Modes which direct
the usage of available hardware for off-loading collective
operations are DMAPP_C_PSET_MODE_CE_OFFLOAD_TRY,
DMAPP_C_PSET_MODE_CE_OFFLOAD_MUST,
DMAPP_C_PSET_MODE_CE_NO_FPADD,
DMAPP_C_PSET_MODE_CE_NO_SPLIT_FPADD.
attrs
Pointer to a previously initialized attribute structure. Can be NULL.
pset_handle
Pointer to an opaque dmapp_c_pset_handle_t structure to be
used for subsequent DMAPP collective calls.
S–2446–51
303
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
Too many psets have already been created.
DMAPP_RC_BUSY
The specified identifier is already in use.
5.3.85 dmapp_c_pset_export
Export a pset before calling DMAPP collective functions. The application
must use some out-of-band synchronization mechanism after calling
dmapp_c_pset_create and before invoking this function. Without this
synchronization, there is a high probability that for one or more ranks, this function
will return DMAPP_RC_NOT_FOUND.
This function will be called by all ranks in a job which are going to be used in the
collective operations involving the pset.
All ranks must provide the same unique identifier value to be associated with the
pset. The structure is designed to allow for expansion in the type of possible pset
delimiters. An attrs attribute argument is included to allow for more detailed
specification of how collective operations should be handled. Examples would
include using a non-default radix, etc.
Note: This operation should not be invoked frequently by the application.
5.3.85.1 Synopsis
dmapp_return_t dmapp_c_pset_export(
IN dmapp_c_pset_handle_t *pset_handle);
5.3.85.2 Parameters
pset_handle
Pointer to an opaque dmapp_c_pset_handle_t structure,
returned from previous call to dmapp_c_pset_create() to be
used for subsequent DMAPP collective calls.
304
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NOT_FOUND
One or more of the processes associated with the pset handle has
not invoked dmapp_c_pset_create.
5.3.86 dmapp_c_barrier_join
Initiate a barrier join operation on a pset. A successful return from this operation
means the barrier operation is proceeding, not that it is complete.
5.3.86.1 Synopsis
dmapp_return_t dmapp_c_barrier_join(
IN dmapp_c_pset_handle_t *pset_handle);
5.3.86.2 Parameters
pset_handle
Structure from a previously exported pset.
DMAPP_RC_SUCCESS
The operation completed successfully. A successful return from
this operation means the barrier operation is proceeding, not that it
is complete.
DMAPP_RC_BUSY
The specified handle is currently in use for another collective
operation.
S–2446–51
305
5.3.87 dmapp_c_pset_cancel_op
Cancel an outstanding collective operation on a pset. This function has no effect if
there is no outstanding collective operation. After a pset is canceled, it cannot be
used for any further collective operations. The only allowed operation on a cancelled
pset is dmapp_c_pset_destroy().
5.3.87.1 Synopsis
dmapp_return_t dmapp_c_pset_cancel_op(
IN dmapp_c_pset_handle_t pset_handle);
5.3.87.2 Parameters
pset_handle
Structure from a previously created pset.
DMAPP_RC_SUCCESS
5.3.88 dmapp_c_pset_destroy
Destroy a previously created pset, freeing DMAPP internal resources associated
with the pset. This is a local operation, not an explicitly synchronizing function.
5.3.88.1 Synopsis
dmapp_return_t dmapp_c_pset_destroy(
IN dmapp_c_pset_handle_t pset_handle);
5.3.88.2 Parameters
pset_handle
Structure from a previously created pset.
306
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_BUSY
The pset is still involved in a collective operation, in which case
an application should invoke this function at a later time to ensure
resources are released.
5.3.89 dmapp_c_greduce_start
Initiate a global reduction operation over the previously exported pset. A successful
return from this operation only means the operation is proceeding, not that it is
complete.
5.3.89.1 Synopsis
dmapp_return_t dmapp_c_greduce_start(
IN dmapp_c_pset_handle_t *pset_handle,
IN void *in,
OUT void *out,
IN uint32_t nelems,
IN dmapp_c_type_t type,
IN dmapp_c_op_t op);
S–2446–51
307
5.3.89.2 Parameters
pset_handle
Pointer to an opaque dmapp_c_pset_handle_t structure from a
previously exported pset.
in
Pointer to buffer containing this PE's contribution to the global
reduction over the pset.
out
Pointer to buffer where the result of the global reduction operation is
to be returned.
nelems
Number of elements in the global operation. The number
of elements specified cannot exceed the value returned by
dmapp_c_greduce_nelems_max for the given data type and
operation type.
type
Data type of the elements in the global operation. See
dmapp_c_type on page 201.
op
Type of global reduction operation. See dmapp_c_op on page 201.
DMAPP_RC_SUCCESS
DMAPP_RC_BUSY
The specified handle is currently in use for another collective
operation.
5.3.90 dmapp_c_greduce_nelems_max
Return the maximum number of elements of a given data type which can be used for
a single global reduction operation. This function may be used before a DMAPP
initialization call is made.
5.3.90.1 Synopsis
dmapp_return_t dmapp_c_greduce_nelems_max(
IN dmapp_c_type_t type,
OUT uint32_t *nelems_max);
308
S–2446–51
5.3.90.2 Parameters
type
Data type of the elements in the global operation. See
dmapp_c_type on page 201.
nelems_max
Maximum number of elements of the supplied data type which can
be used for a single global reduction operation.
DMAPP_RC_SUCCESS
5.3.91 dmapp_c_pset_test
Test for completion of a collective operation for a given pset in a non-blocking
manner.
5.3.91.1 Synopsis
dmapp_c_pset_test(IN dmapp_c_pset_handle_t pset_handle)
5.3.91.2 Parameters
pset_handle
pset for which a collective operation has been initiated.
DMAPP_RC_SUCCESS
The collective operation completed locally. Local completion of a
previously initiated collective operation does not mean all ranks have
completed the collective operation, but that all ranks have initiated
the operation.
DMAPP_RC_NOT_DONE
Collective operation has not completed locally.
An unrecoverable network error was encountered.
S–2446–51
309
5.3.92 dmapp_c_pset_wait
Wait for completion of a collective operation for a given pset. This function is the
blocking equivalent of dmapp_c_pset_test.
5.3.92.1 Synopsis
dmapp_c_pset_wait(IN dmapp_c_pset_handle_t pset_handle);
5.3.92.2 Parameters
pset_handle
pset for which a collective operation has been initiated.
DMAPP_RC_SUCCESS
The collective operation completed successfully.
DMAPP_RC_NOT_DONE
The collective operation has not completed.
An unrecoverable network error was encountered.
5.3.93 dmapp_sheap_malloc
The dmapp_sheap_malloc function allocates size bytes of memory from the
symmetric heap. The space returned is left uninitialized. It cannot be assumed that
the memory returned is zeroed out. There are no address equality guarantees across
ranks.
The function returns NULL when an allocation request is too large or when the
symmetric heap is out of space.
5.3.93.1 Synopsis
void *dmapp_sheap_malloc(
IN size_t size);
5.3.93.2 Parameters
size
310
The size, in bytes, of memory to allocate from the symmetric heap.
S–2446–51
5.3.94 dmapp_sheap_realloc
The dmapp_sheap_realloc function changes the size of the block to which
ptr points to the size, in bytes, specified by size. The contents of the block are
unchanged up to the lesser of the new and old sizes. If the new size is larger, the
value of the newly allocated portion of the block is indeterminate. If ptr is a null
pointer, dmapp_sheap_realloc behaves like dmapp_sheap_malloc for the
specified size. If size is 0 and ptr is not a null pointer, the block to which it points is
freed. Otherwise, if ptr does not match a pointer earlier returned by a symmetric heap
function, or if the space has already been deallocated, dmapp_sheap_realloc
returns a null pointer. If the space cannot be allocated, the block to which ptr points
is unchanged.
5.3.94.1 Synopsis
void *dmapp_sheap_realloc(
IN void
*ptr,
IN size_t size);
5.3.94.2 Parameters
ptr
Pointer to the block.
size
The size, in bytes, of which to change the block.
5.3.95 dmapp_sheap_free
The dmapp_sheap_free function frees a block of memory previously allocated
by dmapp_sheap_malloc or dmapp_sheap_realloc.
5.3.95.1 Synopsis
void dmapp_sheap_free(
IN void
*ptr);
5.3.95.2 Parameters
ptr
Pointer to the block of memory previously allocated with
dmapp_sheap_malloc or dmapp_sheap_realloc.
5.3.96 dmapp_sheap_stats
The dmapp_sheap_stats function returns DMAPP symmetric heap usage
information.
5.3.96.1 Synopsis
dmapp_return_t dmapp_sheap_stats(
INOUT dmapp_sheap_stats_t *stats);
S–2446–51
311
5.3.96.2 Parameters
stats
Pointer to structure containing symmetric heap usage information.
See dmapp_sheap_stats_t on page 203.
5.3.97 dmapp_mem_register
The dmapp_mem_register function dynamically registers a memory region,
other than statically linked data segment or the symmetric heap, with the NIC.
The memory region is described by starting address addr and length. This memory
could have been allocated from the private heap or using mmap. Memory registered
by a call to dmapp_mem_register becomes remotely accessible and is assumed
to be non-symmetric.
The function updates the content of the segment descriptor to reflect the actual
starting address and length of the region which was registered. These values can
differ from the input values due to rounding, for instance. Dynamically registered
memory can only be remotely accessed by point-to-point RMA functions, not by
PE-strided RMA functions.
5.3.97.1 Synopsis
dmapp_return_t dmapp_mem_register(
IN void
*addr,
IN uint64_t
length,
INOUT dmapp_seg_desc_t *seg_desc);
5.3.97.2 Parameters
addr
Points to starting address of the memory region to be registered.
Must be non-NULL.
length
Length in bytes of the memory region in bytes. Must be > 0.
seg_desc
On input, points to segment descriptor and must be non-NULL.
Function updates it to the actual starting address and length of the
registered region.
DMAPP_RC_SUCCESS
Unsuccessful memory registration or invalid memory handle.
312
S–2446–51
5.3.98 dmapp_mem_unregister
The dmapp_mem_unregister function unregisters a memory region, other
than statically linked data segment or the symmetric heap, on the fly, from the
NIC. The memory region must previously have been registered by a call to
dmapp_mem_register.
5.3.98.1 Synopsis
dmapp_return_t dmapp_mem_register(
IN dmapp_seg_desc_t *seg_desc);
5.3.98.2 Parameters
seg_desc
Points to segment descriptor to deregister, which must have been
registered with a call to dmapp_mem_register.
DMAPP_RC_SUCCESS
5.3.99 dmapp_mem_reserve
(Aries only) The dmapp_mem_reserve function reserves address space, other than
statically linked data segment or the symmetric heap, with the NIC.
The memory address range described by starting address addr and length is reserved
with the NIC. The content of the segment descriptor is updated to reflect actual
starting address and length of the region which was reserved. These values can differ
from the input values due to rounding required by page sizes.
A modifier flag setting of DMAPP_MEM_OS_MMAP can be supplied to this call, which
will cause the library to map the reserved NIC virtual address range with virtual
memory from the OS (using the huge page filing system).
The modifier flag DMAPP_MEM_SYMMETRIC can be supplied to this call to indicate
to the library that a symmetric segment reservation is being created. Symmetric
segments must be created at the same virtual address in all ranks in the job and also
have the same size. The library will not validate that this is the case. Symmetric
segments can only be created if all previous reservations have been flagged as
symmetric, and will fail and return DMAPP_RC_RESOURCE_ERROR if this is not
the case or if an on demand memory registration has been performed (e.g. during
data transmission).
S–2446–51
313
Currently only SMP (block cyclic) and cyclic process layouts are supported for
DMAPP_MEM_SYMMETRIC segments. Attempting to create a symmetric segment
from other process layouts, will result in DMAPP_RC_NOT_SUPPORTED being
returned.
5.3.99.1 Synopsis
dmapp_return_t dmapp_mem_reserve(
IN void
*addr,
IN uint64_t
length,
IN uint64_t
flags,
INOUT dmapp_seg_desc_t *seg_desc);
5.3.99.2 Parameters
addr
Points to starting address of the memory region to be reserved. Must
be non-NULL.
length
flags
Modifier flags.
seg_desc
reserved region.
DMAPP_RC_SUCCESS
An error occurred due to insufficient resources.
Operation not supported.
5.3.100 dmapp_mem_update
(Aries only) The memory address range described by starting address addr and
length is updated with the NIC. The seg_desc provided must be one previously
created by a call to dmapp_mem_reserve.
314
S–2446–51
5.3.100.1 Synopsis
dmapp_return_t dmapp_mem_update(
IN void
*addr,
IN uint64_t
length,
IN uint64_t
flags,
IN dmapp_seg_desc_t *seg_desc);
addr
Points to starting address of the memory region to be updated. Must
be non-NULL.
length
flags
Modifier flags.
seg_desc
reserved region.
DMAPP_RC_SUCCESS
An error occurred due to insufficient resources.
The requested operation is not supported.
5.3.101 dmapp_segdesc_compare
The dmapp_segdesc_compare function compares two segment descriptors. If
they describe the same memory region, flag is set to 1. If they describe different
memory regions, flag is set to 0.
5.3.101.1 Synopsis
dmapp_return_t dmapp_segdesc_compare(
IN dmapp_seg_desc_t *seg_desc1,
OUT dmapp_seg_desc_t *seg_desc2,
INOUT int *flag);
S–2446–51
315
seg_desc1
Pointer to segment descriptor 1.
seg_desc2
Pointer to segment descriptor 2.
flag
Set to 1 if both segment descriptors describe the same memory
region, otherwise set to 0.
DMAPP_RC_SUCCESS
5.3.102 dmapp_register_process_cb
Register a process callback function. This function is used when there are progress
requirements that require the use of DMAPP even when using blocking DMAPP
calls, e.g., dmapp_get. A progress callback function should return 0 upon success,
-1 upon failure.
5.3.102.1 Synopsis
dmapp_return_t dmapp_register_process_cb(
IN int
*progress_cb,
IN void
*data);
progress_cb
Pointer to callback function to be registered.
data
316
Pointer to optional data supplied as an argument to the callback
function. Can be null.
S–2446–51
DMAPP_RC_SUCCESS
DMAPP_RC_NO_SPACE
Too many callback functions already registered.
5.3.103 dmapp_deregister_progress_cb
Deregister a progress callback function previously registered using
dmapp_register_progress_cb.
5.3.103.1 Synopsis
dmapp_return_t dmapp_deregister_progress_cb(
IN int
*progress_cb);
progress_cb
Pointer to callback function to be deregistered.
DMAPP_RC_SUCCESS
DMAPP_RC_NOT_FOUND
Supplied callback function not found in callback list.
S–2446–51
317
5.3.104 dmapp_checkpoint
The dmapp_checkpoint function destroys GNI resources and DMAPP internal
resources that cannot be checkpointed. This function should not be called outside of
the checkpoint-restart context. In a threaded environment, this function should only
be called once per process.
5.3.104.1 Synopsis
dmapp_return_t dmapp_checkpoint(void);
DMAPP_RC_SUCCESS
DMAPP_RC_NOT_DONE
Upper-level software has not quiesced the network.
5.3.105 dmapp_restart
This is a checkpoint-restart specific function and should not be called outside the
context of that feature. Reinitialize GNI and DMAPP internal resources which could
not be checkpointed. In a threaded environment, this function should only be called
once per process.
5.3.105.1 Synopsis
dmapp_return_t dmapp_restart(
IN uint32_t restart_modes);
restart_modes
DMAPP_RC_SUCCESS
DMAPP_RC_NOT_DONE
Upper-level software has not restarted the network.
318
S–2446–51
5.4 Environment Variables Which Affect DMAPP
5.4.1 XT_SYMMETRIC_HEAP_SIZE
Controls the size (in bytes) of the symmetric heap. One Mbyte is allocated for
internal use only.
Default: 0 bytes for the user
5.4.2 DMAPP_ABORT_ON_ERROR
Allows a user to force a core dump upon error. This is supported during initialization
and memory handling operations.
Default: not set
5.4.3 DMAPP_DEBUG_CATEGORIES
When DMAPP_DEBUG_CATEGORIES is set to "memory", and
DMAPP_DEBUG_LEVEL is set to 1, DMAPP displays symmetric heap usage.
5.4.4 DMAPP_DEBUG_FILE
Causes a log file per rank to be created when DMAPP_DEBUG_LEVEL is set.
5.4.5 DMAPP_DEBUG_LEVEL
When set to a value between 1 and , Allows a user to see debugging information.
Setting DMAPP_DEBUG_FILE causes a log file per rank to be created.
Default: not set
5.4.6 DMAPP_ERROR_FILE [STDERR | STDOUT]
To redirect error messages issued by the DMAPP library to stdout, set this variable
to stdout.
5.4.7 DMAPP_PUT_NBI_CHAIN_OFF
By default, message rate optimization for non-blocking implicit PUTs and
non-fetching AMOs is on. Defining this environment variable turns it off.
S–2446–51
319
320
S–2446–51
Sample Code [A]
A.1 uGNI
The /opt/cray/ugni/version_string/examples directory contains test
programs, Makefile and README to demonstrate the usage of several uGNI
functions.
A.2 DMAPP
A.2.1 dmapp_sample_put.c
#include
#include
#include
#include
<stdio.h>
<unistd.h>
"pmi.h"
"dmapp.h"
#define MAX_NELEMS (128L*1024L)
/* If necessary, run the job with fewer than the maximum number of cores
* per node so that enough memory is available for each PE. */
int main(int argc,char **argv)
{
int
pe = -1;
int
npes = -1;
int
target_pe;
int
fail_count = 0;
long
nelems = MAX_NELEMS;
long
*source;
long
*target;
long
i;
dmapp_return_t
status;
dmapp_rma_attrs_t actual_args = {0}, rma_args = {0};
dmapp_jobinfo_t
job;
dmapp_seg_desc_t *seg = NULL;
/* Set the RMA parameters. */
rma_args.put_relaxed_ordering = DMAPP_ROUTING_ADAPTIVE;
rma_args.max_outstanding_nb = DMAPP_DEF_OUTSTANDING_NB;
rma_args.offload_threshold = DMAPP_OFFLOAD_THRESHOLD;
rma_args.max_concurrency = 1;
/* Initialize DMAPP. */
status = dmapp_init(&rma_args, &actual_args);
if (status != DMAPP_RC_SUCCESS) {
S–2446–51
321
fprintf(stderr," dmapp_init FAILED: %d\n", status);
exit(1);
}
/* Allocate and initialize the source and target arrays. */
source = (long *)dmapp_sheap_malloc(nelems*sizeof(long));
target = (long *)dmapp_sheap_malloc(nelems*sizeof(long));
if ((source == NULL) || (target == NULL)) {
fprintf(stderr," sheap_malloc'd failed src 0x%lx targ 0x%lx\n",
(long)source, (long)target);
exit(1);
}
for (i=0; i<nelems; i++) {
source[i] = i;
target[i] = -9L;
}
/* Wait for all PEs to complete array initialization. */
PMI_Barrier();
/* Get job related information. */
status = dmapp_get_jobinfo(&job);
fprintf(stderr," dmapp_get_jobinfo FAILED: %d\n", status);
exit(1);
}
pe = job.pe;
npes = job.npes;
seg = &(job.sheap_seg);
/* Send my data to my target PE. */
target_pe = npes - pe -1;
status = dmapp_put(target, seg, target_pe, source, nelems, DMAPP_QW);
fprintf(stderr," dmapp_put FAILED: %d\n", status);
exit(1);
}
/* Wait for all PEs to complete their PUT. */
PMI_Barrier();
/* Check the results. */
if ((target[i] != i) && (fail_count<10)) {
fprintf(stderr," PE %d: target[%ld] is %ld, should be %ld\n",
pe, i, target[i], (long)i);
fail_count++;
}
}
322
S–2446–51
Sample Code [A]
if (fail_count == 0) {
fprintf(stderr," dmapp_put PASSED for PE %04d\n", pe);
}
else {
fprintf(stderr," dmapp_put FAILED for PE %04d: "
"%d or more wrong values\n", pe, fail_count);
}
/* Finalize. */
status = dmapp_finalize();
return(0);
}
A.2.2 dmapp_sample_get.c
/*
* Copyright (c) 2009 Cray Inc. All Rights Reserved.
*
* The contents of this file is proprietary information of Cray Inc.
* and may not be disclosed without prior written consent.
*
* $HeadURL$
* $LastChangedRevision$
*
* Simple DMAPP Get test - blocking version
* This test uses the external job launch package PMI.
*
*/
#include
#include
#include
#include
<stdio.h>
<stdlib.h>
"pmi.h"
"dmapp.h"
int main(int argc,char **argv)
{
char
*expl = NULL;
int
nelems = 128;
int
i;
int
pe = -1;
int
npes = -1;
int
fail_count = 0;
long
*source = NULL;
long
*target = NULL;
dmapp_return_t
status;
dmapp_rma_attrs_t actual_args;
dmapp_jobinfo_t
job;
dmapp_seg_desc_t *seg = NULL;
/* Initialize DMAPP resources before executing any other DMAPP calls. */
status = dmapp_init(NULL, &actual_args);
dmapp_explain_error(status, (const char **)&expl);
fprintf(stderr," dmapp_init returned %d - %s - %s \n",
status, dmapp_err_str[status], expl);
S–2446–51
323
exit(1);
}
/* Allocate remotely accessible memory for source and target buffers.
Only memory in the data segment or the sheap is remotely accessible.
Here we allocate from the sheap. */
source = (long *)dmapp_sheap_malloc(nelems*sizeof(long));
target = (long *)dmapp_sheap_malloc(nelems*sizeof(long));
if ((source == NULL) || (target == NULL)) {
fprintf(stderr," dmapp_sheap_malloc FAILED \n");
exit(1);
}
source[i] = i;
target[i] = -9L;
}
/* Synchronize to make sure everyone's buffers are initialized before
data transfer is started. */
PMI_Barrier();
/* Retrieve information about job details such as PE id and number of PEs. */
status = dmapp_get_jobinfo(&job);
fprintf(stderr," dmapp_get_jobinfo FAILED w/ %d - %s -%s\n",
exit(1);
}
pe = job.pe;
npes = job.npes;
/* Specify in which segment the remote memory region (the source) lies.
In this case, it is the sheap (see above). */
seg = &(job.sheap_seg);
fprintf(stderr," Hello from PE %d of %d, segment start %p, segment size 0x%lx\n",
pe, npes, seg->addr, (unsigned long)seg->len);
fprintf(stderr," PE %d getting %d nelems from addr %p on PE %d to local addr %p\n",
pe, nelems, (void *)source, npes - pe -1, (void *)source);
/* Execute GET operation from remote memory region source on PE Y
into local memory region target on PE X. */
status = dmapp_get(target, source, seg, npes - pe - 1, nelems, DMAPP_QW);
fprintf(stderr," dmapp_get FAILED w/ %d - %s -%s \n",
exit(1);
}
/* Synchronize before verifying the data. */
PMI_Barrier();
/* Verify data received in target buffer. */
if (target[i] != i) {
324
S–2446–51
Sample Code [A]
fprintf(stderr," PE %d: target[%d] is %ld, should be %ld\n",
pe, i, target[i], (long)i);
fail_count++;
}
}
if (fail_count == 0)
fprintf(stderr," dmapp_sample_get PASSED\n");
else
fprintf(stderr," dmapp_sample_get FAILED: %d wrong values\n",
fail_count);
/* Free buffers allocated from sheap. */
dmapp_sheap_free(target);
dmapp_sheap_free(source);
/* Release DMAPP resources. This is a mandatory call. */
status = dmapp_finalize();
fprintf(stderr," dmapp_finalize FAILED w/ %d - %s -%s\n",
exit(1);
}
return(0);
}
S–2446–51
325

Using the GNI and DMAPP APIs

Transcription

Similar documents

GNI Employee Newsletter

" iull-tit"a plan. lf you are intrigued by the cha

9 December 2013 Senator Dato` Sri Idris Jala Presentation to

JAMISON WILLIAM KING

Senator Dato` Sri Idris Jala - ETP

The Gay Naturist E-former - November 2014

EU Budget Milestones: From Fundamental Systemic Reforms to

mpk klang

Delivering Transformation - ETP