wdiff rfc5046.original rfc5046.txt

INTERNET DRAFT Mike
Network Working Group M. Ko
draft-ietf-ips-iser-06.txt
Request for Comments: 5046 IBM Corporation
Mallikarjun
Category: Standards Track M. Chadalapaka
Hewlett-Packard Company
John
J. Hufferd
Brocade, Inc.
Uri
U. Elzur
Hemal
H. Shah
Patricia
P. Thaler
Broadcom Corporation

Expires: May,
October 2007

iSCSI

Internet Small Computer System Interface (iSCSI) Extensions
for RDMA the Remote Direct Memory Access (RDMA) Specification

Status of this This Memo

By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of

This document specifies an Internet standards track protocol for the
Internet Engineering
Task Force (IETF), its areas, community, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.

Internet-Drafts are draft documents valid requests discussion and suggestions for a maximum
improvements. Please refer to the current edition of six
months the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."

The list status of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html.

The list this protocol. Distribution of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html. this memo is unlimited.

Abstract

iSCSI

Internet Small Computer System Interface (iSCSI) Extensions for RDMA
Remote Direct Memory Access (RDMA) provides the RDMA data transfer
capability to iSCSI by layering iSCSI on top of an RDMA-Capable Protocol
Protocol, such as the iWARP protocol suite. An RDMA-Capable Protocol
provides RDMA Read and Write services, which enable data to be
transferred directly into SCSI I/O Buffers without intermediate data
copies. This document describes the extensions to the iSCSI protocol
to

Ko et al. Expires May 2007 1 support RDMA services as provided by an RDMA-Capable Protocol Protocol,
such as the iWARP protocol suite.

Ko et al. Expires May 2007 2

Table of Contents

1 Definitions and Acronyms....................................7
1.1 Definitions.................................................7
1.2 Acronyms...................................................13
1.3 Conventions................................................15
2 Introduction...............................................16
2.1 Motivation.................................................16
2.2

1. Introduction ....................................................5
1.1. Motivation .................................................5
1.2. Architectural Goals........................................17
2.3 Goals ........................................6
1.3. Protocol Overview..........................................18
2.4 Overview ..........................................7
1.4. RDMA services Services and iSER.....................................19
2.4.1 STag......................................................19
2.4.2 Send......................................................20
2.4.3 iSER .....................................8
1.4.1. STag ................................................8
1.4.2. Send ................................................9
1.4.3. RDMA Write................................................20
2.4.4 Write ..........................................9
1.4.4. RDMA Read.................................................20
2.5 Read ...........................................9
1.5. SCSI Read Overview.........................................21
2.6 Overview ........................................10
1.6. SCSI Write Overview........................................21
2.7 Overview .......................................10
1.7. iSCSI/iSER Layering........................................22
3 Layering .......................................10
2. Definitions and Acronyms .......................................11
2.1. Definitions ...............................................11
2.2. Acronyms ..................................................17
2.3. Conventions ...............................................19
3. Upper Layer Interface Requirements.........................23
3.1 Requirements .............................19
3.1. Operational Primitives offered Offered by iSER.....................23
3.1.1 Send_Control..............................................24
3.1.2 Put_Data..................................................24
3.1.3 Get_Data..................................................24
3.1.4 Allocate_Connection_Resources.............................25
3.1.5 Deallocate_Connection_Resources...........................25
3.1.6 Enable_Datamover..........................................25
3.1.7 Connection_Terminate......................................26
3.1.8 Notice_Key_Values.........................................26
3.1.9 Deallocate_Task_Resources.................................26
3.2 iSER ....................20
3.1.1. Send_Control .......................................20
3.1.2. Put_Data ...........................................20
3.1.3. Get_Data ...........................................21
3.1.4. Allocate_Connection_Resources ......................21
3.1.5. Deallocate_Connection_Resources ....................22
3.1.6. Enable_Datamover ...................................22
3.1.7. Connection_Terminate ...............................22
3.1.8. Notice_Key_Values ..................................23
3.1.9. Deallocate_Task_Resources ..........................23
3.2. Operational Primitives used Used by iSER........................27
3.2.1 Control_Notify............................................27
3.2.2 Data_Completion_Notify....................................27
3.2.3 Data_ACK_Notify...........................................28
3.2.4 Connection_Terminate_Notify...............................28
3.3 iSER .......................23
3.2.1. Control_Notify .....................................24
3.2.2. Data_Completion_Notify .............................24
3.2.3. Data_ACK_Notify ....................................24
3.2.4. Connection_Terminate_Notify ........................25
3.3. iSCSI Protocol Usage Requirements..........................28
4 Requirements .........................25
4. Lower Layer Interface Requirements.........................30
4.1 Requirements .............................26
4.1. Interactions with the RCaP Layer...........................30
4.2 Layer ..........................26
4.2. Interactions with the Transport Layer......................31
5 Layer .....................27
5. Connection Setup and Termination...........................32
5.1 Termination ...............................27
5.1. iSCSI/iSER Connection Setup................................32
5.1.1 Setup ...............................27
5.1.1. Initiator Behavior........................................33
5.1.2 Behavior .................................29
5.1.2. Target Behavior...........................................35
5.1.3 Behavior ....................................30
5.1.3. iSER Hello Exchange.......................................37
5.2 Exchange ................................32
5.2. iSCSI/iSER Connection Termination..........................38
5.2.1 Termination .........................33
5.2.1. Normal Connection Termination at the Initiator............38
5.2.2 Initiator .....33
5.2.2. Normal Connection Termination at the Target...............38
5.2.3 Target ........34
5.2.3. Termination without Logout Request/Response PDUs..........39

Ko et al. Expires May 2007 3
6 PDUs ...34
6. Login/Text Operational Keys................................41
6.1 Keys ....................................35
6.1. HeaderDigest and DataDigest................................41
6.2 MaxRecvDataSegmentLength...................................41
6.3 RDMAExtensions.............................................41
6.4 TargetRecvDataSegmentLength................................42
6.5 InitiatorRecvDataSegmentLength.............................43
6.6 DataDigest ...............................35
6.2. MaxRecvDataSegmentLength ..................................36
6.3. RDMAExtensions ............................................36
6.4. TargetRecvDataSegmentLength ...............................37
6.5. InitiatorRecvDataSegmentLength ............................38
6.6. OFMarker and IFMarker......................................43
6.7 MaxOutstandingUnexpectedPDUs...............................44
7 IFMarker .....................................38
6.7. MaxOutstandingUnexpectedPDUs ..............................38
7. iSCSI PDU Considerations...................................45
7.1 Considerations .......................................39
7.1. iSCSI Data-Type PDU........................................45
7.2 PDU .......................................39
7.2. iSCSI Control-Type PDU.....................................46
7.3 PDU ....................................40
7.3. iSCSI PDUs.................................................46
7.3.1 PDUs ................................................40
7.3.1. SCSI Command..............................................46
7.3.2 Command .......................................40
7.3.2. SCSI Response.............................................48
7.3.3 Response ......................................42
7.3.3. Task Management Function Request/Response.................49
7.3.4 Request/Response ..........44
7.3.4. SCSI Data-out.............................................51
7.3.5 Data-Out ......................................45
7.3.5. SCSI Data-in..............................................51
7.3.6 Data-In .......................................46
7.3.6. Ready To to Transfer (R2T)...................................54
7.3.7 (R2T) ............................48
7.3.7. Asynchronous Message......................................56
7.3.8 Message ...............................50
7.3.8. Text Request & and Text Response..............................56
7.3.9 Response .....................50
7.3.9. Login Request & and Login Response............................56
7.3.10 Response ...................50
7.3.10. Logout Request & and Logout Response........................56
7.3.11 Response ................51
7.3.11. SNACK Request...........................................57
7.3.12 Reject..................................................57
7.3.13 Request .....................................51
7.3.12. Reject ............................................51
7.3.13. NOP-Out & NOP-In........................................57
8 and NOP-In ................................51
8. Flow Control and STag Management...........................58
8.1 Management ...............................52
8.1. Flow Control for RDMA Send Message Types...................58
8.1.1 Types ..................52
8.1.1. Flow Control for Control-Type PDUs from the Initiator.....58
8.1.2
Initiator ..........................................52
8.1.2. Flow Control for Control-Type PDUs from the Target........61
8.2
Target .............................................55
8.2. Flow Control for RDMA Read Resources.......................62
8.3 Resources ......................56
8.3. STag Management............................................62
8.3.1 Management ...........................................56
8.3.1. Allocation of STags.......................................63
8.3.2 STags ................................57
8.3.2. Invalidation of STags.....................................63
9 STags ..............................57
9. iSER Control and Data Transfer.............................65
9.1 Transfer .................................58
9.1. iSER Header Format.........................................65
9.2 Format ........................................58
9.2. iSER Header Format for the iSCSI Control-Type PDU..............65
9.3 PDU .........59
9.3. iSER Header Format for the iSER Hello Message..................67
9.4 Message .............60
9.4. iSER Header Format for the iSER HelloReply Message.............68
9.5 Message ........61
9.5. SCSI Data Transfer Operations..............................69
9.5.1 Operations .............................62
9.5.1. SCSI Write Operation......................................69
9.5.2 Operation ...............................62
9.5.2. SCSI Read Operation.......................................70
9.5.3 Operation ................................63
9.5.3. Bidirectional Operation...................................70
10 Operation ............................64
10. iSER Error Handling and Recovery...........................71
10.1 Recovery ..............................64
10.1. Error Handling............................................71
10.1.1 Handling ...........................................64
10.1.1. Errors in the Transport Layer...........................71
10.1.2 Layer .....................64
10.1.2. Errors in the RCaP Layer................................72
10.1.3 Layer ..........................65
10.1.3. Errors in the iSER Layer................................72

Ko et al. Expires May 2007 4
10.1.4 Layer ..........................66
10.1.4. Errors in the iSCSI Layer...............................74
10.2 Layer .........................67
10.2. Error Recovery............................................76
10.2.1 Recovery ...........................................69
10.2.1. PDU Recovery............................................76
10.2.2 Recovery ......................................69
10.2.2. Connection Recovery.....................................77
11 Recovery ...............................70
11. Security Considerations....................................78
12 IANA Considerations........................................79
13 References.................................................80
13.1 Considerations .......................................71
12. References ....................................................71
12.1. Normative References......................................80
13.2 References .....................................71
12.2. Informative References....................................80
14 References ...................................72
Appendix A.................................................82
14.1 A. iWARP Message Format for iSER.............................82
14.1.1 iSER .........................73
A.1. iWARP Message Format for iSER Hello Message.............82
14.1.2 Message ...............73
A.2. iWARP Message Format for iSER HelloReply Message........83
14.1.3 Message ..........74
A.3. iWARP Message Format for SCSI Read Command PDU..........84
14.1.4 PDU ............75
A.4. iWARP Message Format for SCSI Read Data.................85
14.1.5 Data ...................76
A.5. iWARP Message Format for SCSI Write Command PDU.........86
14.1.6 PDU ...........77
A.6. iWARP Message Format for RDMA Read Request..............87
14.1.7 Request ................78
A.7. iWARP Message Format for Solicited SCSI Write Data......88
14.1.8 Data ........79
A.8. iWARP Message Format for SCSI Response PDU..............89
15 PDU ................80
Appendix B.................................................90
15.1 B. Architectural discussion Discussion of iSER over InfiniBand..........90
15.2 InfiniBand ......81
B.1. The Host side Side of the iSCSI & and iSER connections Connections
in Infiniband90
15.3 InfiniBand .............................................81
B.2. The Storage side Side of the iSCSI & and iSER mixed network environment91
15.4 Mixed
Network Environment .......................................82
B.3. Discovery processes Processes for an InfiniBand Host................91
15.5 Host ................82
B.4. IBTA Connection specifications............................92
16 Author's Address...........................................93
17 Acknowledgments............................................94
18 Full Copyright Statement...................................95

Ko et al. Expires May 2007 5 Specifications ............................83
Acknowledgments ...................................................83

Table of Figures

Figure 1 1. Example of iSCSI/iSER Layering in Full Feature Phase...22 Phase ....11
Figure 2 2. iSER Header Format.....................................65 Format ......................................58
Figure 3 3. iSER Header Format for iSCSI Control-Type PDU..........66 PDU ...........59
Figure 4 4. iSER Header Format for iSER Hello Message..............67 Message ...............60
Figure 5 5. iSER Header Format for iSER HelloReply Message.........68 Message ..........61
Figure 6 6. SendSE Message containing an iSER Hello Message........82 Message .........72
Figure 7 7. SendSE Message containing an iSER HelloReply Message...83 Message ....74
Figure 8 8. SendSE Message containing a SCSI Read Command PDU......84 PDU .......75
Figure 9 9. RDMA Write Message containing SCSI Read Data...........85 Data ............76
Figure 10 10. SendSE Message containing a SCSI Write Command PDU....86 PDU .....77
Figure 11 11. RDMA Read Request Message.............................87 Message ..............................78
Figure 12 12. RDMA Read Response Message containing SCSI Write Data.88 Data ..79
Figure 13 13. SendInvSE Message containing SCSI Response PDU........89 PDU .........80
Figure 14 14. iSCSI and iSER on IB..................................90 IB ...................................81
Figure 15 15. Storage Controller with TCP, iWARP, and IB Connections91

Ko et al. Expires May 2007 6

1 Definitions and Acronyms

1.1 Definitions

Advertisement (Advertised, Advertise, Advertisements, Advertises) - Connections .82

1. Introduction

1.1. Motivation

The act of informing a remote iSER Layer that a local node's
buffer iSCSI protocol [RFC3720] is available to it. A Node makes a buffer available for
incoming RDMA Read Request Message or incoming RDMA Write
Message access by informing the remote iSER Layer of the Tagged
Buffer identifiers (STag, TO, and buffer length). Note that
this Advertisement mapping of Tagged Buffer information is the
responsibility of SCSI Architecture
Model (see [SAM2]) over the iSER Layer on either end TCP protocol. SCSI commands are carried
by iSCSI requests, and is not
defined by the RDMA-Capable Protocol. A typical method would be
for the iSER Layer to embed the Tagged Buffer's STag, TO, SCSI responses and
buffer length in a Send Message destined for the remote iSER
Layer.

Completion (Completed, Complete, Completes) - Completion is defined
as the process status are carried by the RDMA-Capable Protocol layer to inform the
iSER Layer, that a particular RDMA Operation has performed all
functions specified for the RDMA Operation.

Connection - A connection is a logical circuit between the initiator
and the target, e.g., a TCP connection. Communication between
the initiator iSCSI
responses. Other iSCSI protocol exchanges and the target occurs over one or more
connections. The connections carry control messages, SCSI
commands, parameters, and data within are also
transported in iSCSI Protocol Data Units
(iSCSI PDUs).

Connection Handle - An information element that identifies (PDUs).

Out-of-order TCP segments in the
particular Traditional iSCSI connection model have to be
stored and is unique for a given iSCSI-iSER
pair. Every invocation of reassembled before the iSCSI protocol layer within an Operational Primitive is qualified
with end
node can place the Connection Handle.

Data Sink - The peer receiving a data payload. Note that in the Data
Sink can be iSCSI buffers. This reassembly is
required because not every TCP segment is likely to both send and receive RCaP Messages to
transfer a data payload.

Data Source - The peer sending a data payload. Note that the Data
Source can be required contain an iSCSI
header to both send enable its placement, and receive RCaP Messages to
transfer TCP itself does not have a data payload.

Datamover Interface (DI) - The interface between the iSCSI Layer
built-in mechanism for signaling Upper Level Protocol (ULP) message
boundaries to aid placement of out-of-order segments. This TCP
reassembly at high network speeds is quite counter-productive for the
following reasons: wasted memory bandwidth in data copying, the need
for reassembly memory, wasted CPU cycles in data copying, and the Datamover Layer
general store-and-forward latency from an application perspective.
TCP reassembly was recognized as described a serious issue in [DA].

Datamover Layer - A layer [RFC3720], and
the notion of a "sync and steering layer" was introduced that is directly below the iSCSI Layer
optional to implement and
above the underlying transport layers. This layer exposes use. One specific sync and

Ko et al. Expires May 2007 7
uses a set steering
mechanism, called "markers", was defined in [RFC3720], which provides
an application-level way of transport independent Operational Primitives for framing iSCSI PDUs within the communication between TCP data
stream even when the iSCSI Layer and itself. The
Datamover layer, operating TCP segments are not yet reassembled to be in-
order.

With these defined techniques in conjunction with the transport
layers, moves [RFC3720], a Network Interface
Controller customized for iSCSI (SNIC) could offload the control TCP/IP
processing and support direct data information on the placement, but most iSCSI
connection.
implementations do not support iSCSI "markers", making SNIC marker-
based direct data placement unusable in practice.

The iWARP protocol stack provides direct data placement functionality
that is usable in practice. In this specification, the iSER Layer addition, there is interest in using
iSCSI with other RDMA protocol stacks that support direct data
placement, such as the
Datamover layer.

Datamover one provided by InfiniBand. The generic term
RDMA-Capable Protocol - A Datamover protocol (RCaP) is used to refer to the wire-protocol that
is defined to realize the Datamover layer functionality. In
this specification, the iSER protocol is the Datamover protocol.

Event - An indication RDMA
functionality provided by such protocol stacks.

With the availability of RDMA-Capable Protocol layer Controllers within a host
system, which does not have SNICs, it is appropriate for iSCSI to
the iSER Layer be
able to indicate a Completion or exploit the direct data placement function of the RDMA-
Capable Controller like other condition
requiring immediate attention.

Inbound applications.

iSCSI Extensions for RDMA Read Queue Depth (IRD) - The maximum number (iSER) is designed precisely to take
advantage of incoming
outstanding generic RDMA Read Requests that the RDMA-Capable Controller
can handle on technologies -- iSER's goal is to permit
iSCSI to employ direct data placement and RDMA capabilities using a particular RCaP Stream at the Data Source. For
some
generic RDMA-Capable Protocol layers, the term "IRD" may be known
by a different name. For example, for InfiniBand, Controller. In summary, the
equivalent for IRD iSCSI/iSER protocol
stack is the Responder Resources.

Invalidate STag - A mechanism used designed to prevent the Remote Peer from
reusing enable scaling to high speeds by relying on a previous explicitly Advertised STag, until the
generic data placement process and RDMA technologies and products,
which enable direct data placement of both in-order and out-of-order
data.

This document describes iSER
Layer at the local node makes as a protocol extension to iSCSI, both
for convenience of description and because it available through is true in a subsequent
explicit Advertisement.

I/O Buffer - A buffer very
strict protocol sense. However, note that iSER is used in a SCSI Read or Write operation
so SCSI data may be sent from or received into that buffer. reality
extending the connectivity of the iSCSI - The protocol defined in
[RFC3720], and the name iSER reflects this reality.

When the iSCSI protocol as defined in [RFC3720] (i.e., without the
iSER enhancements) is a mapping of intended in the SCSI Architecture Model rest of SAM-2 over TCP.

iSCSI control-type PDU - Any iSCSI PDU that the document, the term
"Traditional iSCSI" is not used to make the intention clear.

1.2. Architectural Goals

This section summarizes the architectural goals that guided the
design of iSER.

1. Provide an iSCSI data-
type PDU and also not a SCSI Data-out PDU carrying solicited RDMA data is defined as an transfer model for iSCSI control-type PDU. Specifically, it
is to be noted that enables direct
in-order or out-of-order data placement of SCSI Data-out PDUs for unsolicited data are
defined as iSCSI control-type PDUs.

iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI
PDU that causes into pre-
allocated SCSI buffers while maintaining in-order data transfer, transparent delivery.

2. Not require any major changes to the remote SCSI Architecture Model
[SAM2] and SCSI command set standards.

3. Utilize existing iSCSI
Layer, infrastructure (sometimes referred to take place between the peer iSCSI nodes on as
"iSCSI ecosystem") including but not limited to MIB,
bootstrapping, negotiation, naming and discovery, and security.

4. Require a full
feature phase iSCSI connection. An iSCSI data-type PDU, when
requested for transmission by the sender iSCSI Layer, results session to operate in the associated Traditional iSCSI data
transfer without the participation of the
remote iSCSI Layer, i.e. the PDU itself mode if iSER is not delivered as-is

Ko et al. Expires May 2007 8
to supported by either the remote iSCSI Layer. The following iSCSI PDUs constitute initiator or
the set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU. target (i.e., not require iSCSI Layer - A layer Full Feature Phase
interoperability between an end node operating in the protocol stack implementation within Traditional
iSCSI mode, and an end node that implements the operating in iSER-assisted mode).

5. Allow initiator and target implementations to utilize generic
RDMA-Capable Controllers such as RDMA-enabled Network Interface
Controllers (RNICs), or to implement iSCSI protocol and interfaces with
the iSER Layer via in software
(not require iSCSI- or iSER-specific assists in the Datamover Interface.

iSCSI PDU (iSCSI Protocol Data Unit) - The iSCSI Layer RCaP
implementation or RDMA-Capable Controller).

6. Require full and only generic RCaP functionality at both the
initiator and the iSCSI Layer at the target divide their
communications into messages. The term "iSCSI target.

7. Implement a lightweight Datamover protocol data
unit" (iSCSI PDU) is used for these messages.

iSCSI/iSER Connection - An iSER-assisted iSCSI connection.

iSCSI/iSER Session - An iSER-assisted iSCSI session.

iSCSI-iSER Pair - The iSCSI Layer and with minimal
state maintenance.

1.3. Protocol Overview

Consistent with the underlying iSER Layer.

iSER - iSCSI Extensions for RDMA, architectural goals stated in Section 2.2, the
iSER protocol defined does not require changes in this
document.

iSER-assisted - A term generally used to describe the operation of iSCSI when the ecosystem or any
related SCSI specifications. The iSER functionality is also enabled below protocol defines the mapping
of iSCSI Layer for the specific iSCSI/iSER connection PDUs to RCaP Messages in question.

iSER-IRD - This variable represents the maximum number of incoming
outstanding RDMA Read Requests that the iSER Layer at the
initiator declares on such a particular RCaP Stream.

iSER-ORD - This variable represents the maximum number of
outstanding RDMA Read Requests way that the iSER Layer can initiate
on a particular RCaP Stream. This variable it is maintained only
by the iSER Layer at the target.

iSER Layer - entirely
feasible to realize iSCSI/iSER implementations that are based on
generic RDMA-Capable Controllers. The iSER protocol layer that implements the iSCSI Extensions for RDMA
(iSER) protocol.

iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and
[MPA] when layered above [TCP]. [RDMAP] and [DDP] may be
layered above SCTP or other transport protocols.

Local Mapping - A task requires
minimal state record maintained by the iSER Layer
that associates the Initiator Task Tag maintenance to assist an iSCSI Full Feature Phase
connection, besides being oblivious to the Local STag(s).
The specifics notion of the record structure are implementation
dependent.

Ko et al. Expires May 2007 9
Local Peer - an iSCSI
session. The implementation of the RDMA-Capable Protocol on the
local end crucial protocol aspects of iSER may be summarized
thus:

1. iSER-assisted mode is negotiated during the connection. Used to refer to the local entity
when describing protocol exchanges or other interactions between
two Nodes.

Node - A computing device attached to iSCSI login for each
session, and an entire iSCSI session can only operate in one or more links of mode
(i.e., a
network. A Node connection in this context does not refer to a specific
application or protocol instantiation running on the computer.
A Node may consist of one or more RDMA-Capable Controllers
installed session cannot operate in iSER-assisted
mode if a host computer.

Operational Primitive - An Operational Primitive is an abstract
functional interface procedure that requests another layer to
perform a specific action on different connection of the requestor's behalf or notifies same session is already in
Full Feature Phase in the other layer of some event. The Datamover Interface between
an Traditional iSCSI Layer and a Datamover layer within an mode).

2. Once in iSER-assisted mode, all iSCSI end node
uses a set of Operational Primitives to define the functional
interface between the two layers. Note interactions on that not every
invocation of an Operational Primitive may elicit a response
from the requested layer.
connection use RCaP Messages.

3. A full discussion of the Operational
Primitive types and request-response semantics available to Send Message Type is used for carrying an iSCSI and control-type PDU
preceded by an iSER can be found in [DA].

Outbound header. See Section 7.2 for more details on
iSCSI control-type PDUs.

4. RDMA Write, RDMA Read Queue Depth (ORD) - The maximum number of
outstanding Request, and RDMA Read Requests that Response Messages are
used for carrying control and all data information associated with
the RDMA-Capable Controller
can initiate iSCSI data-type PDUs. See Section 7.1 for more details on a particular RCaP Stream at the Data Sink. For
some RDMA-Capable Protocol layer,
iSCSI data-type PDUs.

5. Target drives all data transfer (with the term "ORD" may be known exception of iSCSI
unsolicited data) for SCSI writes and SCSI reads, by
a different name. For issuing RDMA
Read Requests and RDMA Writes, respectively.

6. RCaP is responsible for ensuring data integrity. (For example,
iWARP includes a CRC-enhanced framing layer called MPA on top of
TCP; and for InfiniBand, the equivalent
for ORD is the Initiator Depth.

Phase-Collapse - Refers to CRCs are included in the optimization Reliable
Connection mode). For this reason, iSCSI header and data digests
are negotiated to "None" for iSCSI/iSER sessions.

7. The iSCSI error recovery hierarchy defined in [RFC3720] is fully
supported by iSER. (However, see Section 7.3.11 on the handling
of SNACK Request PDUs.)

8. iSER requires no changes to iSCSI where authentication, security, and
text mode negotiation mechanisms.

Note that Traditional iSCSI implementations may have to be adapted to
employ iSER. It is expected that the SCSI
status adaptation when required is transferred along
likely to be centered around the upper layer interface requirements
of iSER (Section 3).

1.4. RDMA Services and iSER

iSER is designed to work with software and/or hardware protocol
stacks providing the final SCSI Data-in PDU from
a target. See section 3.2 protocol services defined in [RFC3720]. RCaP Message - One or more packets documents such
as [RDMAP], [IB], etc. The following subsections describe the key
protocol elements of RCaP services that iSER relies on.

1.4.1. STag

An STag is the identifier of an I/O Buffer unique to an RDMA-Capable
Controller that the network iSER layer comprising a
single RDMA operation or Advertises to the remote iSCSI/iSER
node in order to complete a part SCSI I/O.

In iSER, Advertisement is the act of informing the target by the
initiator that an I/O Buffer is available at the initiator for RDMA
Read Operation of or RDMA Write access by the
RDMA-Capable Protocol. For iWARP, target. The initiator Advertises
the I/O Buffer by including the STag in the header of an RCaP iSER Message
containing the SCSI Command PDU to the target. The base Tagged
Offset is known as
an RDMAP Message.

RCaP Stream - A single bidirectional association between not explicitly specified, but the peer
RDMA-Capable Protocol layers on two Nodes over a single
transport-level stream. For iWARP, an RCaP Stream target must always assume
it as zero. The buffer length is known as
an RDMAP Stream, and specified in the association is created when SCSI Command
PDU.

The iSER layer at the
connection transitions initiator Advertises the STag for the I/O
Buffer of each SCSI I/O to iSER-assisted mode following a
successful Login Phase during which the iSER support is negotiated.

Ko et al. Expires May 2007 10
RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that
provides a reliable RDMA transport functionality, e.g., iWARP,
InfiniBand, etc.

RDMA-Capable Controller - A network layer at the target in the iSER
header of the SendSE Message containing the SCSI Command PDU, unless
the I/O adapter or embedded
controller with RDMA functionality. For example, for iWARP,
this could can be an RNIC, and completely satisfied by unsolicited data alone.

The iSER layer at the target provides the STag for InfiniBand, this could be a HCA
(Host Channel Adapter) or TCA (Target Channel Adapter).

RDMA-enabled Network Interface Controller (RNIC) - A network the I/O
adapter or embedded controller with iWARP functionality. Buffer
that is the Data Sink of an RDMA Read Operation - A sequence of RCaP Messages, including control
Messages, (Section 2.4.4) to transfer data from a Data Source
the RCaP layer on the initiator node -- i.e., this is completely
transparent to a Data Sink. the iSER layer at the initiator.

The following RDMA Operations are iSER protocol is defined - RDMA Write
Operation, RDMA Read Operation, Send Operation, Send with
Invalidate Operation, Send with Solicited Event Operation, Send
with Solicited Event and Invalidate Operation, and Terminate
Operation.

RDMA Protocol (RDMAP) - A wire protocol so that supports RDMA
Operations to transfer ULP data between a Local Peer and the
Remote Peer as described in [RDMAP].

RDMA Read Operation - An RDMA Operation used by the Data Sink to
transfer the contents of a Data Source buffer from the Remote
Peer to Advertised STag is
automatically invalidated upon a Data Sink buffer at the Local Peer. An RDMA Read
operation consists normal completion of a single RDMA Read Request Message and a
single RDMA Read Response Message.

RDMA Read Request - An RCaP Message used by the Data Sink to request the Data Source to transfer associated
task. This automatic invalidation is realized via the contents of a buffer. The RDMA
Read Request SendInvSE
Message describes both the Data Source and carrying the Data
Sink buffers.

RDMA Read SCSI Response - An RCaP Message used by the Data Source PDU. There are two exceptions to
transfer the contents
this automatic invalidation -- bidirectional commands, and abnormal
completion of a buffer command. The iSER layer at the initiator is required
to explicitly invalidate the Data Sink, STag in response these cases, in addition to an RDMA Read Request. The RDMA Read Response Message only
describes
sanity checking the automatic invalidation even when that does
happen.

1.4.2. Send

Send is the Data Sink buffer.

RDMA Write Operation - An RDMA Operation used by the Data Source that is not addressed to
transfer the contents of a Data Source an Advertised
buffer from by the Local
Peer to a Data Sink buffer at sending side, and thus uses Untagged buffers on the Remote Peer.
receiving side.

The RDMA Write
Message only describes iSER layer at the Data Sink buffer.

Remote Direct Memory Access (RDMA) - A method of accessing memory on
a remote system in which initiator uses the local system specifies Send Operation to transmit
any iSCSI control-type PDU to the remote

Ko et al. Expires May 2007 11
location of target. As an example, the data
initiator uses Send Operations to be transferred. Employing transfer iSER Messages containing
SCSI Command PDUs to the iSER layer at the target.

An iSER layer at the target uses the Send Operation to transmit any
iSCSI control-type PDU to the initiator. As an RDMA-
Capable Controller in example, the remote system allows target
uses Send Operations to transfer iSER Messages containing SCSI
Response PDUs to the access iSER layer at the initiator.

1.4.3. RDMA Write

RDMA Write is the RDMA Operation that is used to take place without interrupting data into an
Advertised buffer on the processing of receiving side. The sending side addresses
the CPU(s) Message using an STag and a Tagged Offset that are valid on the
system.

Remote Mapping - A task state record maintained by the
Data Sink.

The iSER Layer
that associates layer at the Initiator Task Tag target uses the RDMA Write Operation to
transfer the Advertised STag(s).
The specifics contents of a local I/O Buffer to an Advertised I/O
Buffer at the record structure are implementation
dependent.

Remote Peer - initiator. The implementation of iSER layer at the RDMA-Capable Protocol on target uses the
opposite end RDMA
Write to transfer whole or part of the connection. Used to refer data required to the remote
entity when describing protocol exchanges or other interactions
between two Nodes. complete a
SCSI Layer - This read command.

The iSER layer builds/receives SCSI CDBs (Command
Descriptor Blocks) and sends/receives them with at the remaining
command execute [SAM2] parameters to/from initiator does not employ RDMA Writes.

1.4.4. RDMA Read

RDMA Read is the iSCSI Layer.

Send - An RDMA Operation that transfers the contents of a Buffer
from the Local Peer is used to retrieve data from an
Advertised buffer on a Buffer at remote node. The sending side of the RDMA
Read Request addresses the Remote Peer.

Send Message Type - A Send Message, Send with Invalidate Message,
Send with Solicited Event Message, or Send with Solicited Event
and Invalidate Message.

SendInvSE Message - A Send with Solicited Event using an STag and Invalidate
Message.

SendSE Message - A Send with Solicited Event Message

Sequence Number (SN) - DataSN for a SCSI Data-in PDU and R2TSN for
an R2T PDU. The semantics for both types of sequence numbers Tagged Offset
that are as defined valid on the Data Source in [RFC3720].

Session, iSCSI Session - addition to providing a valid
local STag and Tagged Offset that identify the Data Sink.

The group iSER layer at the target uses the RDMA Read Operation to transfer
the contents of Connections that link an Advertised I/O Buffer at the initiator SCSI port with to a local
I/O Buffer at the target. The iSER layer at the target SCSI port form an iSCSI
session (equivalent uses the RDMA
Read to a SCSI I-T nexus). Connections can be
added fetch whole or part of the data required to and removed from complete a session even while SCSI
write command.

The iSER layer at the I-T nexus is
intact. Across all connections within a session, an initiator
sees one and the same target.

Solicited Event (SE) - A facility by which an does not employ RDMA Operation sender
may cause an Event to be generated Reads.

1.5. SCSI Read Overview

The iSER layer at the recipient, if initiator receives the
recipient is configured to generate such SCSI Command PDU from
the iSCSI layer. The iSER layer at the initiator generates an Event, when a Send
with Solicited Event or Send with Solicited Event and Invalidate
Message is received.

Ko et al. Expires May 2007 12
Steering Tag (STag) - An identifier of a Tagged STag
for the I/O Buffer on a Node
(Local or Remote) as defined in [RDMAP] of the SCSI Read and [DDP]. For other
RDMA-Capable Protocols, Advertises the Steering Tag may be known buffer by
different names but will be herein referred to as STags. For
example, for Infiniband, a Remote STag is known as an R-Key, and
a Local
including the STag is known as an L-Key, and both will be considered
STags.

Tagged Buffer - A buffer that is explicitly Advertised to part of the iSER
Layer at the remote node through header for the exchange of an STag, Tagged
Offset, and length.

Tagged Offset (TO) - PDU. The offset within a Tagged Buffer.

Traditional iSCSI - Refers to the iSCSI protocol as defined in
[RFC3720] (i.e. without the iSER enhancements).

Untagged Buffer - A buffer that
Message is not explicitly Advertised transferred to the target using a SendSE Message.

The iSER Layer layer at the remode node.

1.2 Acronyms

Acronym Definition

--------------------------------------------------------------

AHS Additional Header Segment

BHS Basic Header Segment

CO Connection Only

CRC Cyclic Redundancy Check

DDP Direct Data Placement Protocol

DI Datamover Interface

HCA Host Channel Adapter

IANA Internet Assigned Numbers Authority

IB Infiniband

IETF Internet Engineering Task Force

I/O Input - Output

Ko et al. Expires May 2007 13
IO Initialize Only

IP Internet Protocol

IPoIB IP over Infiniband

IPsec Internet Protocol Security

iSER iSCSI Extensions for target uses one or more RDMA

ITT Initiator Task Tag

LO Leading Only

MPA Marker Writes to transfer
the data required to complete the SCSI Read.

The iSER layer at the target uses a SendInvSE Message to transfer the
SCSI Response PDU Aligned Framing for TCP

NOP No Operation

NSG Next Stage (during back to the iSER layer at the initiator. The iSER
layer at the initiator notifies the iSCSI Login Phase)

OS Operating System

PDU Protocol Data Unit

R2T Ready To Transfer

R2TSN Ready To Transfer Sequence Number

RDMA Remote Direct Memory Access

RDMAP Remote Direct Memory Access Protocol

RFC Request For Comments

RNIC RDMA-enabled Network Interface Controller

SAM2 layer of the availability
of the SCSI Architecture Model - 2 Response PDU.

1.6. SCSI Small Computer Systems Interface

SNACK Selective Negative Acknowledgment - also

Sequence Number Acknowledgement for Write Overview

The iSER layer at the initiator receives the SCSI Command PDU from
the iSCSI layer. If solicited data transfer is involved, the iSER
layer at the initiator generates an STag Steering Tag

SW Session Wide

Ko et al. Expires May 2007 14
TCA Target Channel Adapter

TCP Transmission Control Protocol

TMF Task Management Function

TTT Target Transfer Tag

TO Tagged Offset

ULP Upper Level Protocol

1.3 Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", for the I/O Buffer of the
SCSI Write and "OPTIONAL" in this
document are to be interpreted Advertises the buffer by including the STag as described in RFC 2119.

Ko et al. Expires May 2007 15

2 Introduction

2.1 Motivation part of
the iSER header for the PDU. The iSCSI protocol ([RFC3720]) iSER Message is transferred to the
target using a mapping of SendSE Message.

The iSER layer at the SCSI Architecture
Model (see [SAM2]) over initiator may optionally send one or more non-
immediate unsolicited data PDUs to the target using Send Message
Types.

If solicited data transfer is involved, the iSER layer at the target
uses one or more RDMA Reads to transfer the data required to complete
the TCP protocol. SCSI commands are carried
by iSCSI requests and Write.

The iSER layer at the target uses a SendInvSE Message to transfer the
SCSI responses and status are carried by iSCSI
responses. Other Response PDU back to the iSER layer at the initiator. The iSER
layer at the initiator notifies the iSCSI protocol exchanges and layer of the availability
of the SCSI Data are also
transported in Response PDU.

1.7. iSCSI/iSER Layering

iSCSI PDUs.

Out-of-order TCP segments in Extensions for RDMA (iSER) is layered between the Traditional iSCSI model have to be
stored layer
and reassembled before the iSCSI protocol RCaP layer. Note that the RCaP layer within may be composed of one
or more distinct protocol layers depending on the specifics of the
RCaP. Figure 1 shows an end
node can place example of the data in relationship between SCSI,
iSCSI, iSER, and the iSCSI buffers. This reassembly different RCaP layers. For TCP, the RCaP is
required because not every TCP segment
iWARP. For InfiniBand, the RCaP is likely to contain an the Reliable Connected Transport
Service. Note that the iSCSI
header to enable its placement and layer as described here supports the
RDMA Extensions as used in iSER.

Figure 1. Example of iSCSI/iSER Layering in mechanism for signaling ULP message boundaries to aid placement Full Feature Phase

2. Definitions and Acronyms

2.1. Definitions

Advertisement (Advertised, Advertise, Advertisements, Advertises) -
The act of out-of-order segments. This TCP reassembly at high network
speeds informing a remote iSER layer that a local node's
buffer is quite counter-productive available to it. A Node makes a buffer available for
incoming RDMA Read Request Message or incoming RDMA Write Message
access by informing the following reasons: wasted
memory bandwidth in data copying, need for reassembly memory, wasted
CPU cycles in data copying, and remote iSER layer of the general store-and-forward
latency from an application perspective. TCP reassembly was
recognized as a serious issue in [RFC3720], Tagged Buffer
identifiers (STag, TO, and buffer length). Note that this
Advertisement of Tagged Buffer information is the notion responsibility
of a
"sync the iSER layer on either end and steering layer" was introduced that is optional to
implement and use. One specific sync and steering mechanism, called
"markers", was not defined in [RFC3720] which provides an application-
level way of framing iSCSI PDUs within by the TCP data stream even when RDMA-
Capable Protocol. A typical method would be for the TCP segments are not yet reassembled iSER layer to be in-order.

With these defined techniques
embed the Tagged Buffer's STag, TO, and buffer length in [RFC3720], a Network Interface
Controller customized Send
Message destined for iSCSI (SNIC) could offload the TCP/IP
processing and support direct data placement, but most iSCSI
implementations do not support iSCSI "markers", making SNIC
marker-based direct data placement unusable in practice.

The iWARP protocol stack provides direct data placement
functionality that is usable in practice, and in addition, there is
also interest in using iSCSI with other RDMA protocol stacks that
support direct data placement, such remote iSER layer.

Completion (Completed, Complete, Completes) - Completion is defined
as the one provided process by
InfiniBand. The generic term the RDMA-Capable Protocol (RCaP) is used
to refer layer to inform the
iSER layer, that a particular RDMA functionality provided by such protocol stacks.

With Operation has performed all
functions specified for the availability of RDMA-Capable Controllers within a host
system, which does not have SNICs, it RDMA Operation.

Connection - A connection is appropriate for iSCSI to be
able to exploit a logical circuit between the direct data placement function of initiator
and the RDMA-
Capable Controller like other applications.

Ko et al. Expires May 2007 16 target, e.g., a TCP connection. Communication between the
initiator and the target occurs over one or more connections. The
connections carry control messages, SCSI commands, parameters, and
data within iSCSI Extensions for RDMA (iSER) is designed precisely to take
advantage of generic RDMA technologies Protocol Data Units (iSCSI PDUs).

Connection Handle - iSER's goal is to permit An information element that identifies the
particular iSCSI to employ direct data placement connection and RDMA capabilities using is unique for a
generic RDMA-Capable Controller. In summary, iSCSI/iSER protocol
stack given iSCSI-iSER
pair. Every invocation of an Operational Primitive is designed to enable scaling to high speeds by relying on qualified
with the Connection Handle.

Data Sink - The peer receiving a
generic data placement process and RDMA technologies and products,
which enable direct data placement of payload. Note that the Data
Sink can be required to both in-order send and out-of-order
data.

This document describes iSER as receive RCaP Messages to
transfer a protocol extension data payload.

Data Source - The peer sending a data payload. Note that the Data
Source can be required to iSCSI, both
for convenience of description send and also because it is true in a very
strict protocol sense. However, it is receive RCaP Messages to be noted
transfer a data payload.

Datamover Interface (DI) - The interface between the iSCSI layer and
the Datamover layer as described in [DA].

Datamover Layer - A layer that iSER is in
reality extending the connectivity of directly below the iSCSI protocol defined in
[RFC3720], layer and
above the name iSER reflects this reality.

When underlying transport layers. This layer exposes and
uses a set of transport independent Operational Primitives for the
communication between the iSCSI protocol as defined layer and itself. The Datamover
layer, operating in [RFC3720] (i.e. without conjunction with the
iSER enhancements) is intended in transport layers, moves
the rest of control and data information on the document, iSCSI connection. In this
specification, the term
"Traditional iSCSI" iSER layer is used to make the intention clear.

2.2 Architectural Goals

This section summarizes the architectural goals that guided Datamover layer.

Datamover Protocol - A Datamover protocol is the
design of iSER.

1. Provide an RDMA data transfer model for iSCSI wire-protocol that enables direct
in order or out of order data placement of SCSI data into pre-
allocated SCSI buffers while maintaining in order data delivery.

2. Not require any major changes to SCSI Architecture Model [SAM2]
and SCSI command set standards.

3. Utilize existing iSCSI infrastructure (sometimes referred to as
"iSCSI ecosystem") including but not limited to MIB,
bootstrapping, negotiation, naming & discovery, and security.

4. Require a session
is defined to operate in realize the Datamover layer functionality. In this
specification, the Traditional iSCSI data
transfer mode if iSER protocol is not supported by either the initiator or Datamover protocol.

Event - An indication provided by the target (not require iSCSI full feature phase interoperability
between an end node operating in Traditional iSCSI mode, and an
end node operating in iSER-assisted mode).

5. Allow initiator and target implementations to utilize generic RDMA-Capable Controllers such as RNICs, or implement iSCSI and Protocol layer to
the iSER in software (not require iSCSI layer to indicate a Completion or iSER specific assists in other condition
requiring immediate attention.

Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming
outstanding RDMA Read Requests that the RCaP implementation or RDMA-Capable Controller).

Ko et al. Expires May 2007 17
6. Require full and only generic Controller
can handle on a particular RCaP functionality Stream at both the
initiator and Data Source. For
some RDMA-Capable Protocol layers, the target.

7. Implement term "IRD" may be known by
a light weight Datamover protocol different name. For example, for iSCSI with minimal
state maintenance.

2.3 Protocol Overview

Consistent with InfiniBand, the architectural goals stated in section 2.2, equivalent for
IRD is the Responder Resources.

Invalidate STag - A mechanism used to prevent the Remote Peer from
reusing a previous explicitly Advertised STag, until the iSER protocol does not require changes in
layer at the iSCSI ecosystem local node makes it available through a subsequent
explicit Advertisement.

I/O Buffer - A buffer that is used in a SCSI Read or any
related Write operation
so SCSI specifications. iSER data may be sent from or received into that buffer.

iSCSI - The iSCSI protocol defines the as defined in [RFC3720] is a mapping of
the SCSI Architecture Model of SAM-2 over TCP.

iSCSI control-type PDU - Any iSCSI PDUs to RCaP Messages in such a way PDU that is not an iSCSI data-
type PDU and also not a SCSI Data-out PDU carrying solicited data
is defined as an iSCSI control-type PDU. Specifically, it is entirely
feasible to realize iSCSI/iSER implementations
be noted that SCSI Data-out PDUs for unsolicited data are based on
generic RDMA-Capable Controllers. The iSER protocol layer requires
minimal state maintenance to assist defined
as iSCSI control-type PDUs.

iSCSI data-type PDU - An iSCSI data-type PDU is defined as an iSCSI full feature phase
connection, besides being oblivious
PDU that causes data transfer, transparent to the notion of an remote iSCSI
session. The crucial protocol aspects of iSER may be summarized
thus:

1. iSER-assisted mode is negotiated during
layer, to take place between the peer iSCSI login nodes on a Full
Feature Phase iSCSI connection. An iSCSI data-type PDU, when
requested for each
session, and an entire transmission by the sender iSCSI session can only operate in one mode
(i.e. a connection in a session cannot operate layer, results in iSER-assisted
mode if a different connection
the associated data transfer without the participation of the same session
remote iSCSI layer, i.e. the PDU itself is already in
full feature phase in not delivered as-is to
the Traditional remote iSCSI mode).

2. Once in iSER-assisted mode, all layer. The following iSCSI interactions on that
connection use RCaP Messages.

3. PDUs constitute the
set of iSCSI data-type PDUs - SCSI Data-In PDU and R2T PDU.

iSCSI Layer - A Send Message Type is used for carrying layer in the protocol stack implementation within an
end node that implements the iSCSI protocol and interfaces with
the iSER layer via the Datamover Interface.

iSCSI control-type PDU preceded by an iSER header. See section 7.2 for more details
on (iSCSI Protocol Data Unit) - The iSCSI control-type PDUs.

4. RDMA Write, RDMA Read Request, layer at the
initiator and RDMA Read Response Messages
are the iSCSI layer at the target divide their
communications into messages. The term "iSCSI protocol data unit"
(iSCSI PDU) is used for carrying control these messages.

iSCSI/iSER Connection - An iSER-assisted iSCSI connection.

iSCSI/iSER Session - An iSER-assisted iSCSI session.

iSCSI-iSER Pair - The iSCSI layer and all data information associated
with the underlying iSER layer.

iSER - iSCSI data-type PDUs. See section 7.1 Extensions for more details
on iSCSI data-type PDUs.

5. Target drives all data transfer (with RDMA, the exception protocol defined in this
document.

iSER-assisted - A term generally used to describe the operation of
iSCSI
unsolicited data) when the iSER functionality is also enabled below the iSCSI
layer for SCSI writes and SCSI reads, by issuing the specific iSCSI/iSER connection in question.

iSER-IRD - This variable represents the maximum number of incoming
outstanding RDMA Read Requests and RDMA Writes respectively.

6. RCaP is responsible for ensuring data integrity. (For example,
iWARP includes a CRC-enhanced framing that the iSER layer called MPA at the
initiator declares on top of
TCP; and for Infiniband, a particular RCaP Stream.

iSER-ORD - This variable represents the CRCs are included in maximum number of outstanding
RDMA Read Requests that the Reliable
Connection mode). For this reason, iSCSI header and data digests
are negotiated to "None" for iSCSI/iSER sessions.

Ko et al. Expires May 2007 18
7. The iSCSI error recovery hierarchy defined in [RFC3720] iSER layer can initiate on a
particular RCaP Stream. This variable is fully
supported maintained only by iSER. (However, see section 7.3.11 on the handling
of SNACK Request PDUs.)

8.
iSER requires no changes to iSCSI authentication, security, and
text mode negotiation mechanisms.

Note layer at the target.

iSER Layer - The layer that Traditional implements the iSCSI implementations Extensions for RDMA
(iSER) protocol.

iWARP - A suite of wire protocols comprising of [RDMAP], [DDP], and
[MPA] when layered above [TCP]. [RDMAP] and [DDP] may have to be adapted
to employ iSER. It is expected layered
above SCTP or other transport protocols.

Local Mapping - A task state record maintained by the iSER layer that
associates the adaptation when required is
likely Initiator Task Tag to be centered around the upper layer interface requirements local STag(s). The
specifics of iSER (section 3).

2.4 RDMA services and iSER

iSER is designed to work with software and/or hardware protocol
stacks providing the protocol services defined in RCaP documents
such as [RDMAP], [IB], etc. record structure are implementation dependent.

Local Peer - The following subsections describe the
key protocol elements implementation of RCaP services that iSER relies on.

2.4.1 STag

An STag is the identifier RDMA-Capable Protocol on the
local end of an I/O Buffer unique the connection. Used to refer to an RDMA-Capable
Controller that the iSER Layer Advertises local entity
when describing protocol exchanges or other interactions between
two Nodes.

Node - A computing device attached to one or more links of a network.
A Node in this context does not refer to a specific application or
protocol instantiation running on the remote iSCSI/iSER
node computer. A Node may
consist of one or more RDMA-Capable Controllers installed in order to complete a SCSI I/O.

In iSER, Advertisement
host computer.

Operational Primitive - An Operational Primitive is the act of informing the target by the
initiator that an I/O Buffer is available at abstract
functional interface procedure that requests that another layer
perform a specific action on the initiator for RDMA
Read requestor's behalf or RDMA Write access by the target. The initiator Advertises
the I/O Buffer by including the STag in notifies
the header other layer of an iSER
Message containing the SCSI Command PDU to the target. The base
Tagged Offset is not explicitly specified, but the target must
always assume it as zero. The buffer length is as specified in the
SCSI Command PDU. some event. The iSER Layer at the initiator Advertises the STag for the I/O
Buffer Datamover Interface between an
iSCSI layer and a Datamover layer within an iSCSI end node uses a
set of each SCSI I/O Operational Primitives to define the iSER Layer at the target in functional interface
between the iSER
header two layers. Note that not every invocation of an
Operational Primitive may elicit a response from the SendSE Message containing the SCSI Command PDU, unless requested
layer. A full discussion of the I/O Operational Primitive types and
request-response semantics available to iSCSI and iSER can be completely satisfied by unsolicited data alone.
found in [DA].

Outbound RDMA Read Queue Depth (ORD) - The iSER Layer at the target provides the STag for the I/O Buffer
that is the Data Sink maximum number of an
outstanding RDMA Read Operation (section 2.4.4) to
the RCaP layer on the initiator node - i.e. this is completely
transparent to Requests that the iSER Layer RDMA-Capable Controller
can initiate on a particular RCaP Stream at the initiator.

The iSER protocol is defined so that Data Sink. For
some RDMA-Capable Protocol layer, the Advertised STag is
automatically invalidated upon term "ORD" may be known by a normal completion of
different name. For example, for InfiniBand, the associated
task. This automatic invalidation equivalent for
ORD is realized via the SendInvSE

Ko et al. Expires May 2007 19
Message carrying the SCSI Response PDU. There are two exceptions to
this automatic invalidation Initiator Depth.

Phase-Collapse - bidirectional commands, and abnormal
completion of a command. The iSER Layer at the initiator is
required Refers to explicitly invalidate the STag in these cases, optimization in
addition to sanity checking iSCSI where the automatic invalidation even when
that does happen.

2.4.2 Send

Send SCSI
status is transferred along with the final SCSI Data-in PDU from a
target. See section 3.2 in [RFC3720].

RCaP Message - One or more packets of the network layer comprising a
single RDMA Operation that is not addressed to or a part of an Advertised
buffer by the sending side, and thus uses Untagged buffers on the
receiving side.

The iSER Layer at the initiator uses the Send RDMA Read Operation to transmit
any iSCSI control-type PDU to of the target. As
RDMA-Capable Protocol. For iWARP, an example, the
initiator uses Send Operations to transfer iSER Messages containing
SCSI Command PDUs to the iSER Layer at the target.

An iSER layer at the target uses the Send Operation to transmit any
iSCSI control-type PDU to RCaP Message is known as an
RDMAP Message.

RCaP Stream - A single bidirectional association between the initiator. As peer
RDMA-Capable Protocol layers on two Nodes over a single
transport-level stream. For iWARP, an example, RCaP Stream is known as an
RDMAP Stream, and the target
uses Send Operations to transfer iSER Messages containing SCSI
Response PDUs to association is created when the connection
transitions to iSER-assisted mode following a successful Login
Phase during which iSER Layer at the initiator.

2.4.3 support is negotiated.

RDMA-Capable Protocol (RCaP) - The protocol or protocol suite that
provides a reliable RDMA Write transport functionality, e.g., iWARP,
InfiniBand, etc.

RDMA-Capable Controller - A network I/O adapter or embedded
controller with RDMA Write is the functionality. For example, for iWARP, this
could be an RNIC, and for InfiniBand, this could be a HCA (Host
Channel Adapter) or TCA (Target Channel Adapter).

RDMA-enabled Network Interface Controller (RNIC) - A network I/O
adapter or embedded controller with iWARP functionality.

RDMA Operation that is used - A sequence of RCaP Messages, including control
Messages, to place transfer data into an
Advertised buffer on the receiving side. The sending side addresses
the Message using an STag and from a Data Source to a Tagged Offset that are valid on the Data Sink. The iSER Layer at the target uses the
following RDMA Operations are defined - RDMA Write Operation Operation, RDMA
Read Operation, Send Operation, Send with Invalidate Operation,
Send with Solicited Event Operation, Send with Solicited Event and
Invalidate Operation, and Terminate Operation.

RDMA Protocol (RDMAP) - A wire protocol that supports RDMA Operations
to transfer the contents of ULP data between a local I/O Buffer to an Advertised I/O
Buffer at the initiator. The iSER Layer at the target uses Local Peer and the Remote Peer as
described in [RDMAP].

RDMA
Write Read Operation - An RDMA Operation used by the Data Sink to
transfer whole or part the contents of a Data Source buffer from the data required Remote Peer
to complete a
SCSI Read command.

The iSER Layer Data Sink buffer at the initiator does not employ Local Peer. An RDMA Writes.

2.4.4 Read operation
consists of a single RDMA Read Request Message and a single RDMA
Read is the Response Message.

RDMA Operation that is Read Request - An RCaP Message used by the Data Sink to retrieve data from
an Advertised buffer on request
that the Data Source transfer the contents of a remote node. buffer. The sending side of the RDMA
Read Request addresses the Message using an STag and a Tagged Offset
that are valid on describes both the Data Source in addition to providing a valid
local STag and Tagged Offset that identify the Data Sink.

Ko et al. Expires May 2007 20
The iSER Layer at the target uses the
Sink buffers.

RDMA Read Operation Response - An RCaP Message used by the Data Source to
transfer the contents of an Advertised I/O Buffer at the initiator
to a local I/O Buffer at the target. The iSER Layer at the target
uses the RDMA Read buffer to fetch whole or part of the data required Data Sink, in response to
complete a SCSI Write Command.

The iSER Layer at the initiator does not employ
an RDMA Reads.

2.5 SCSI Read Overview

The iSER Layer at the initiator receives the SCSI Command PDU from
the iSCSI Layer. Request. The iSER Layer at the initiator generates an STag
for the I/O Buffer of the SCSI RDMA Read and Advertises Response Message only
describes the buffer Data Sink buffer.

RDMA Write Operation - An RDMA Operation used by
including the STag as part Data Source to
transfer the contents of a Data Source buffer from the iSER header for Local Peer
to a Data Sink buffer at the PDU. Remote Peer. The iSER RDMA Write Message is transferred to
only describes the target using Data Sink buffer.

Remote Direct Memory Access (RDMA) - A method of accessing memory on
a SendSE Message.

The iSER Layer at remote system in which the target uses one or more RDMA Writes to
transfer local system specifies the remote
location of the data required to complete be transferred. Employing an RDMA-
Capable Controller in the SCSI Read.

The iSER Layer at remote system allows the target uses a SendInvSE Message access to transfer take
place without interrupting the SCSI Response PDU back to processing of the iSER Layer at CPU(s) on the initiator. The
iSER Layer at
system.

Remote Mapping - A task state record maintained by the initiator notifies iSER layer
that associates the iSCSI Layer of Initiator Task Tag to the
availability Advertised STag(s).
The specifics of the SCSI Response PDU.

2.6 SCSI Write Overview record structure are implementation
dependent.

Remote Peer - The iSER Layer at implementation of the initiator receives RDMA-Capable Protocol on the SCSI Command PDU from
opposite end of the iSCSI Layer. If solicited data transfer is involved, connection. Used to refer to the iSER remote
entity when describing protocol exchanges or other interactions
between two Nodes.

SCSI Layer at the initiator generates an STag for the I/O Buffer of the - This layer builds/receives SCSI Write CDBs (Command Descriptor
Blocks) and Advertises sends/receives them with the buffer by including remaining command execute
[SAM2] parameters to/from the STag as part iSCSI layer.

Send - An RDMA Operation that transfers the contents of a Buffer from
the iSER header for the PDU. The iSER Message is transferred Local Peer to
the target using a SendSE Message.

The iSER Layer Buffer at the initiator may optionally send one or more non-
immediate unsolicited data PDUs to the target using Remote Peer.

Send Message
Types.

If solicited data transfer is involved, the iSER Layer at the target
uses one Type - A Send Message, Send with Invalidate Message,
Send with Solicited Event Message, or more RDMA Reads to transfer the data required to
complete the Send with Solicited Event
and Invalidate Message.

SendInvSE Message - A Send with Solicited Event and Invalidate
Message.

SendSE Message - A Send with Solicited Event Message.

Sequence Number (SN) - DataSN for a SCSI Write. Data-in PDU and R2TSN for an
R2T PDU. The iSER Layer at the target uses semantics for both types of sequence numbers are as
defined in [RFC3720].

Session, iSCSI Session - The group of connections that link an
initiator SCSI port with a SendInvSE Message target SCSI port form an iSCSI session
(equivalent to transfer
the a SCSI Response PDU back I-T nexus). Connections can be added to and
removed from a session even while the iSER Layer at I-T nexus is intact. Across
all connections within a session, an initiator sees one and the initiator. The
iSER Layer
same target.

Solicited Event (SE) - A facility by which an RDMA Operation sender
may cause an Event to be generated at the initiator notifies the iSCSI Layer of recipient, if the
availability
recipient is configured to generate such an Event, when a Send
with Solicited Event or Send with Solicited Event and Invalidate
Message is received.

Steering Tag (STag) - An identifier of a Tagged Buffer on a Node
(Local or Remote) as defined in [RDMAP] and [DDP]. For other
RDMA-Capable Protocols, the SCSI Response PDU.

Ko et al. Expires May 2007 21

2.7 iSCSI/iSER Layering

iSCSI Extensions Steering Tag may be known by different
names but will be herein referred to as STags. For example, for RDMA (iSER)
InfiniBand, a Remote STag is layered between the iSCSI layer known as an R-Key, and the RCaP layer. Note a local STag
is known as an L-Key, and both will be considered STags.

Tagged Buffer - A buffer that is explicitly Advertised to the RCaP iSER
layer may be composed of one
or more distinct protocol layers depending on at the specifics of remote node through the
RCaP. Figure 1 shows an example exchange of the relationship between SCSI,
iSCSI, iSER, an STag, Tagged
Offset, and length.

Tagged Offset (TO) - The offset within a Tagged Buffer.

Traditional iSCSI - Refers to the different RCaP layers. For TCP, the RCaP is
iWARP. For Infiniband, iSCSI protocol as defined in
[RFC3720] (i.e. without the RCaP iSER enhancements).

Untagged Buffer - A buffer that is not explicitly Advertised to the Reliable Connected Transport
Service. Note that
iSER layer at the remode node.

2.2. Acronyms

Acronym Definition
--------------------------------------------------------------

AHS Additional Header Segment

BHS Basic Header Segment

CO Connection Only

CRC Cyclic Redundancy Check
DDP Direct Data Placement Protocol

DI Datamover Interface

HCA Host Channel Adapter

IANA Internet Assigned Numbers Authority

IB InfiniBand

IETF Internet Engineering Task Force

I/O Input - Output

IO Initialize Only

IP Internet Protocol

IPoIB IP over InfiniBand

IPsec Internet Protocol Security

iSER iSCSI layer as described here supports the
RDMA Extensions as used in iSER.

+-------------------------------------+
| SCSI |
+-------------------------------------+
| for RDMA

ITT Initiator Task Tag

LO Leading Only

MPA Marker PDU Aligned Framing for TCP

NOP No Operation

NSG Next Stage (during the iSCSI |
DI ------> +-------------------------------------+
| iSER |
+---------+--------------+------------+
| Login Phase)

OS Operating System

PDU Protocol Data Unit

R2T Ready To Transfer

R2TSN Ready To Transfer Sequence Number

RDMA Remote Direct Memory Access

Figure 1 Example of iSCSI/iSER Layering Interface Controller

SAM2 SCSI Architecture Model - 2

SCSI Small Computer Systems Interface

SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for data

STag Steering Tag

SW Session Wide

TCA Target Channel Adapter

TCP Transmission Control Protocol

TMF Task Management Function

TTT Target Transfer Tag

TO Tagged Offset

ULP Upper Level Protocol

2.3. Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in Full Feature Phase

Ko et al. Expires May 2007 22

3 this
document are to be interpreted as described in [RFC2119].

3. Upper Layer Interface Requirements

This section discusses the upper layer interface requirements in the
form of an abstract model of the required interactions between the
iSCSI Layer layer and the iSER Layer. layer. The abstract model used here is
derived from the architectural model described in [DA]. [DA] also
provides a functional overview of the interactions between the iSCSI
Layer
layer and the datamover Datamover layer as intended by the Datamover
Architecture.

The interface requirements are specified by Operational Primitives.
An Operational Primitive is an abstract functional interface
procedure between the iSCSI Layer layer and the iSER Layer layer that requests
one layer to perform a specific action on behalf of the other layer
or notifies the other layer of some event. Whenever an Operational
Primitive in invoked, the Connection_Handle qualifier is used to
identify a particular iSCSI connection. For some Operational
Primitives, a Data_Descriptor is used to identify the iSCSI/SCSI data
buffer associated with the requested or completed operation.

The abstract model and the Operational Primitives defined in this
section facilitate the description of the iSER protocol. In the rest
of the iSER specification, the compliance statements related to the
use of these Operational Primitives are only for the purpose of the
required interactions between the iSCSI Layer layer and the iSER
Layer. layer.
Note that the compliance statements related to the Operational
Primitives in the rest of this specification only mandate functional
equivalence on implementations, but do not put any requirements on
the implementation specifics of the interface between the iSCSI Layer layer
and the iSER Layer. layer.

Each Operational Primitive is invoked with a set of qualifiers which that
specify the information context for performing the specific action
being requested of the Operational Primitive. While the qualifiers
are required, the method of realizing the qualifiers (e.g., by
passing synchronously with invocation, or by retrieving from task
context, or by retrieving from shared memory, etc.) is implementation
dependent.

3.1

3.1. Operational Primitives offered Offered by iSER

The iSER protocol layer MUST support the following Operational
Primitives to be used by the iSCSI protocol layer.

Ko et al. Expires May 2007 23

3.1.1

3.1.1. Send_Control

Input qualifiers: Connection_Handle, BHS and AHS (if any) of the
iSCSI PDU, PDU-specific qualifiers

Return results: Not specified

This is used by the iSCSI Layers layers at the initiator and the target to
request the outbound transfer of an iSCSI control-type PDU (see
section
Section 7.2). Qualifiers that only apply for a particular control-
type PDU are known as PDU-specific qualifiers, e.g.,
ImmediateDataSize for a SCSI Write write command. For details on PDU-
specific qualifiers, see section Section 7.3. The iSCSI Layer layer can only
invoke the Send_Control Operational Primitive when the connection is
in iSER-assisted mode.

3.1.2

3.1.2. Put_Data

Input qualifiers: Connection_Handle, content of a SCSI Data-in
PDU header, Data_Descriptor, Notify_Enable
Return results: Not specified

This is used by the iSCSI Layer layer at the target to request the outbound
transfer of data for a SCSI Data-in PDU from the buffer identified by
the Data_Descriptor qualifier. The iSCSI Layer layer can only invoke the
Put_Data Operational Primitive when the connection is in iSER-assisted iSER-
assisted mode.

The Notify_Enable qualifier is used to indicate to the iSER Layer layer
whether or not it should generate an eventual local completion
notification to the iSCSI Layer. layer. See section Section 3.2.2 on
Data_Completion_Notify for details.

3.1.3

3.1.3. Get_Data

Input qualifiers: Connection_Handle, content of an R2T PDU,
Data_Descriptor, Notify_Enable

Return results: Not specified

This is used by the iSCSI Layer layer at the target to request the inbound
transfer of solicited data requested by an R2T PDU into the buffer
identified by the Data_Descriptor qualifier. The iSCSI Layer layer can
only invoke the Get_Data Operational Primitive when the connection is
in iSER-assisted mode.

Ko et al. Expires May 2007 24

The Notify_Enable qualifier is used to indicate to the iSER Layer layer
whether or not it should generate the eventual local completion
notification to the iSCSI Layer. layer. See section Section 3.2.2 on
Data_Completion_Notify for details.

3.1.4

3.1.4. Allocate_Connection_Resources

Input qualifiers: Connection_Handle, Resource_Descriptor
(optional)

Return results: Status

This is used by the iSCSI Layers layers at the initiator and the target to
request the allocation of all connection resources necessary to
support RCaP for an operational iSCSI/iSER connection. The iSCSI
Layer
layer may optionally specify the implementation-specific resource
requirements for the iSCSI connection using the Resource_Descriptor
qualifier.

A return result of Status=success means that the invocation
succeeded, and a return result of Status=failure means that the
invocation failed. If the invocation is for a Connection_Handle for
which an earlier invocation succeeded, the request will be ignored by
the iSER Layer layer and the result of Status=success will be returned.
Only one Allocate_Connection_Resources Operational Primitive
invocation can be outstanding for a given Connection_Handle at any
time.

3.1.5

3.1.5. Deallocate_Connection_Resources

Input qualifiers: Connection_Handle

Return results: Not specified

This is used by the iSCSI Layers layers at the initiator and the target to
request the deallocation of all connection resources that were
allocated earlier as a result of a successful invocation of the
Allocate_Connection_Resources Operational Primitive.

3.1.6

3.1.6. Enable_Datamover

Input qualifiers: Connection_Handle,
Transport_Connection_Descriptor, Final Login_Response_PDU
(optional)

Return results: Not specified

Ko et al. Expires May 2007 25

This is used by the iSCSI Layers layers at the initiator and the target to
request that a specified iSCSI connection be transitioned to iSER-
assisted mode. The Transport_Connection_Descriptor qualifier is used
to identify the specific connection associated with the
Connection_Handle. The iSCSI layer can only invoke the
Enable_Datamover Operational Primitive when there was is a corresponding
prior resource allocation.

The Final_Login_Response_PDU input qualifier is applicable only for a
target, and contains the final Login Response PDU that concludes the
iSCSI Login Phase. If the underlying transport is TCP, the final
Login Response PDU must be sent as a byte stream as expected by the
iSCSI Layer layer at the initiator. When this qualifier is used, the iSER Layer
layer at the target MUST transmit this final Login Response PDU
before transitioning to iSER-assisted mode.

3.1.7

3.1.7. Connection_Terminate

Input qualifiers: Connection_Handle

Return results: Not specified
This is used by the iSCSI Layers layers at the initiator and the target to
request that a specified iSCSI/iSER connection be terminated and all
associated connection and task resources be freed. When this
Operational Primitive invocation returns to the iSCSI layer, the
iSCSI layer may assume full ownership of all iSCSI-level resources,
e.g.
e.g., I/O Buffers, associated with the connection.

3.1.8

3.1.8. Notice_Key_Values

Input qualifiers: Connection_Handle, number of keys, list of
Key-Value pairs

Return results: Not specified

This is used by the iSCSI Layers layers at the initiator and the target to
request that the iSER Layer to layer take note of the specified Key-Value
pairs
which that were negotiated by the iSCSI peers for the connection.

3.1.9

3.1.9. Deallocate_Task_Resources

Input qualifiers: Connection_Handle, ITT

Return results: Not specified

Ko et al. Expires May 2007 26

This is used by the iSCSI Layers layers at the initiator and the target to
request the deallocation of all RCaP-specific resources allocated by
the iSER Layer layer for the task identified by the ITT qualifier. The
iSER Layer layer may require a certain number of RCaP-specific resources
associated with the ITT for each new iSCSI task. In the normal
course of execution, these task-level resources in the iSER Layer layer are
assumed to be transparently allocated on each task initiation and
deallocated on the conclusion of each task as appropriate. In
exception scenarios where the task does not conclude with a SCSI
Response PDU, the iSER Layer layer needs to be notified of the individual
task terminations to aid its task-level resource management. This
Operational Primitive is used for this purpose, and is not needed
when a SCSI Response PDU normally concludes a task. Note that RCaP-
specific task resources are deallocated by the iSER Layer layer when a SCSI
Response PDU normally concludes a task, even if the SCSI Status status was
not success.

3.2

3.2. Operational Primitives used Used by iSER

The iSER layer MUST use the following Operational Primitives offered
by the iSCSI protocol layer when the connection is in iSER-assisted
mode.

3.2.1

3.2.1. Control_Notify

Input qualifiers: Connection_Handle, an iSCSI control-type PDU

Return results: Not specified

This is used by the iSER Layers layers at the initiator and the target to
notify the iSCSI Layer layer of the availability of an inbound iSCSI
control-type PDU. A PDU is described as "available" to the iSCSI
Layer
layer when the iSER Layer layer notifies the iSCSI Layer layer of the reception
of that inbound PDU, along with an implementation-specific indication
as to where the received PDU is.

3.2.2

3.2.2. Data_Completion_Notify

Input qualifiers: Connection_Handle, ITT, SN

Return results: Not specified

This is used by the iSER Layer layer to notify the iSCSI Layer layer of the
completion of outbound data transfer that was requested by the iSCSI
Layer
layer only if the invocation of the Put_Data Operational Primitive
(see section Section 3.1.2) was qualified with Notify_Enable set. SN refers
to the DataSN associated with the SCSI Data-In Data-in PDU.

Ko et al. Expires May 2007 27

This is used by the iSER Layer layer to notify the iSCSI Layer layer of the
completion of inbound data transfer that was requested by the iSCSI
Layer
layer only if the invocation of the Get_Data Operational Primitive
(see section Section 3.1.3) was qualified with Notify_Enable set. SN refers
to the R2TSN associated with the R2T PDU.

3.2.3

3.2.3. Data_ACK_Notify

Input qualifier: Connection_Handle, ITT, DataSN

Return results: Not specified

This is used by the iSER Layer layer at the target to notify the iSCSI
Layer
layer of the arrival of the data acknowledgement (as defined in
[RFC3720]) requested earlier by the iSCSI Layer layer for the outbound data
transfer via an invocation of the Put_Data Operational Primitive
where the A-bit in the SCSI Data-in PDU is set to 1. See
section Section
7.3.5. DataSN refers to the expected DataSN of the next SCSI Data-in PDU
PDU, which immediately follows the SCSI Data-in PDU with the A-bit
set to which this notification corresponds, with semantics as defined
in [RFC3720].

3.2.4

3.2.4. Connection_Terminate_Notify

Input qualifiers: Connection_Handle

Return results: Not specified

This is used by the iSER Layers layers at the initiator and the target to
notify the iSCSI Layer layer of the unsolicited termination or failure of
an iSCSI/iSER connection. The iSER Layer layer MUST deallocate the
connection and task resources associated with the terminated
connection before the invocation of this Operational Primitive. Note
that the Connection_Terminate_Notify Operational Primitive is not
invoked when the termination of the connection was is earlier requested
by the local iSCSI Layer.

3.3 layer.

3.3. iSCSI Protocol Usage Requirements

To operate in an iSER-assisted mode, the iSCSI Layers layers at both the
initiator and the target MUST negotiate the RDMAExtensions key (see
section
Section 6.3) to "Yes" on the leading connection. If the
RDMAExtensions key is not negotiated to "Yes", then iSER-assisted
mode MUST NOT be used. If the RDMAExtensons RDMAExtensions key is negotiated to
"Yes" but the invocation of the Allocate_Connection_Resources
Operational Primitive to the iSER layer fails, the iSCSI layer MUST

Ko et al. Expires May 2007 28
fail the iSCSI Login process or terminate the connection as
appropriate. See section Section 10.1.3.1 for details.

If the RDMAExtensions key is negotiated to "Yes", the iSCSI Layer layer
MUST satisfy the following protocol usage requirements from the iSER
protocol:

1. The iSCSI Layer layer at the initiator MUST set ExpDataSN to 0 in Task
Management Function Requests for Task Allegiance Reassignment for
read/bidirectional commands, so as to cause the target to send
all unacknowledged read data.

2. The iSCSI Layer layer at the target MUST always return the SCSI status
in a separate SCSI Response PDU for read commands, i.e., there
MUST NOT be a "phase collapse" in concluding a SCSI Read
Command. read command.

3. The iSCSI Layers layers at both the initiator and the target MUST
support the keys as defined in section Section 6 on Login/Text
Operational Keys. If used as specified, these keys MUST NOT be
answered with NotUnderstood NotUnderstood, and the semantics as defined MUST be
followed for each iSER-assisted connection.

4. The iSCSI Layer layer at the initiator MUST NOT issue SNACKs for PDUs.

Ko et al. Expires May 2007 29

4. Lower Layer Interface Requirements

4.1

4.1. Interactions with the RCaP Layer

The iSER protocol layer is layered on top of an RCaP layer (see
Figure 1) and the following are the key features that are assumed to
be supported by any RCaP layer:

* The RCaP layer supports all basic RDMA operations, including RDMA
Write Operation, RDMA Read Operation, Send Operation, Send with
Invalidate Operation, Send with Solicited Event Operation, Send
with Solicited Event & and Invalidate Operation, and Terminate
Operation.

* The RCaP layer provides reliable, in-order message delivery and
direct data placement.

* When the iSER Layer layer initiates an RDMA Read Operation following an
RDMA Write Operation on one RCaP Stream, the RDMA Read Response
Message processing on the remote node will be started only after
the preceding RDMA Write Message payload is placed in the memory
of the remote node.

* The RCaP layer encapsulates a single iSER Message into a single
RCaP Message on the Data Source side. The RCaP layer decapsulates
the iSER Message before delivering it to the iSER
Layer layer on the
Data Sink side.

* When the iSER Layer layer provides the STag to be remotely invalidated
to the RCaP layer for a SendInvSE Message, the RCaP layer uses
this STag as the STag to be invalidated in the SendInvSE Message.

* The RCaP layer uses the STag and Tagged Offset provided by the
iSER Layer layer for the RDMA Write and RDMA Read Request Messages.

* When the RCaP layer delivers the content of an RDMA Send Message
Type to the iSER Layer, layer, the RCaP layer provides the length of the
RDMA Send message. This ensures that the iSER Layer layer does not have
to carry a length field in the iSER header.

* When the RCaP layer delivers the SendSE or SendInvSE Message to
the iSER Layer, layer, it notifies the iSER Layer layer with the mechanism
provided on that interface.

* When the RCaP layer delivers a SendInvSE Message to the iSER
Layer,
layer, it passes the value of the STag that was invalidated.

Ko et al. Expires May 2007 30

* The RCaP layer propagates all status and error indications to the
iSER Layer. layer.

* For a transport layer that operates in byte stream mode such as
TCP, the RCaP implementation supports the enabling of the RDMA
mode after Connection connection establishment and the exchange of Login
parameters in byte stream mode. For a transport layer that
provides message delivery capability such as [IB], the RCaP
implementation supports the use of the messaging capability by the
iSCSI Layer layer directly for the Login phase Phase after connection
establishment before enabling iSER-assisted mode.

* Whenever the iSER Layer layer terminates the RCaP Stream, the RCaP layer
terminates the associated Connection.

4.2 connection.

4.2. Interactions with the Transport Layer

The iSER Layer layer does not directly setup the transport layer connection
(e.g., TCP, or [IB]). During Connection connection setup, the iSCSI
Layer layer is
responsible for setting up the Connection. connection. If the login is
successful, the iSCSI Layer layer invokes the Enable_Datamover Operational
Primitive to request the iSER Layer layer to transition to the iSER-
assisted mode for that iSCSI connection. See section Section 5.1 on
iSCSI/iSER Connection Setup. connection setup. After transitioning to iSER-assisted
mode, the RCaP layer and the underlying transport layer are
responsible for maintaining the Connection connection and reporting to the iSER
Layer
layer any Connection connection failures.

Ko et al. Expires May 2007 31

5. Connection Setup and Termination

5.1

5.1. iSCSI/iSER Connection Setup

During connection setup, the iSCSI Layer layer at the initiator is
responsible for establishing a connection with the target. After the
connection is established, the iSCSI Layers layers at the initiator and the
target enter the Login Phase using the same rules as outlined in
[RFC3720]. Transition to iSER-assisted mode occurs when the
connection transitions into the iSCSI full feature phase Full Feature Phase following a
successful login negotiation between the initiator and the target in
which iSER-assisted mode is negotiated and the connection resources
necessary to support RCaP have been allocated at both the initiator
and the target. The same connection MUST be used for both the iSCSI
Login phase Phase and the subsequent iSER-assisted full feature phase. Full Feature Phase.

iSER-assisted mode MUST be enabled only if it is negotiated on the
leading connection during the LoginOperationalNegotiation Stage stage of
the iSCSI Login Phase. iSER-assisted mode is negotiated using the
RDMAExtensions=<boolean-value> key. Both the initiator and the
target MUST exchange the RDMAExtensions key with the value set to
"Yes" to enable iSER-assisted mode. If both the initiator and the
target fail to negotiate the RDMAExtensions key set to "Yes", then
the connection MUST continue with the login semantics as defined in
[RFC3720]. If the RDMAExtensions key is not negotiated to Yes, then
for some RCaP implementation (such as [IB]), the connection may need
to be re-established in TCP capable mode. (For InfiniBand this will
require an [IPoIB] type connection.)

iSER-assisted mode is defined for a Normal session only and the
RDMAExtensions key MUST NOT be negotiated for a Discovery session.
Discovery sessions are always conducted using the transport layer as
described in [RFC3720].

An iSER enabled node is not required to initiate the RDMAExtensions
key exchange if its preference is for the Traditional iSCSI mode.
The RDMAExtensions key, if offered, MUST be sent in the first
available Login Response or Login Request PDU in the
LoginOperationalNegotiation stage. This is due to the fact that the
value of some login parameters might depend on whether iSER-assisted
mode is enabled or not. enabled.

iSER-assisted mode is a session-wide attribute. If both the
initiator and the target negotiated negotiate RDMAExtensions="Yes" on the
leading connection of a session, then all subsequent connections of
the same session MUST enable iSER-assisted mode without having to
exchange an RDMAExtensions key during the iSCSI Login Phase.

Ko et al. Expires May 2007 32

Conversely, if both the initiator and the target failed fail to negotiate
RDMAExtensions to "Yes" on the leading connection of a session, then
the RDMAExtensions key MUST NOT be negotiated further on any
additional subsequent connection of the session.

When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest
and the DataDigest keys MUST be negotiated to "None" on all
iSCSI/iSER connections participating in that iSCSI session. This is
because, for an iSCSI/iSER connection, RCaP is responsible for
providing error detection that is at least as good as a 32-bit CRC
for all iSER Messages. Furthermore, all SCSI Read data are sent
using RDMA Write Messages instead of the SCSI Data-in PDUs, and all
solicited SCSI write data are sent using RDMA Read Response Messages
instead of the SCSI Data-out PDUs. HeaderDigest and DataDigest
which that
apply to iSCSI PDUs PDUs, would not be appropriate for RDMA Read and RDMA
Write operations used with iSER.

5.1.1

5.1.1. Initiator Behavior

If the outcome of the iSCSI negotiation is to enable iSER-assisted
mode, then on the initiator side, prior to sending the Login Request
with the T (Transit) bit set to 1 and the NSG (Next Stage) field set
to FullFeaturePhase, the iSCSI Layer layer MUST request that the iSER Layer to layer
allocate the connection resources necessary to support RCaP by
invoking the Allocate_Connection_Resources Operational Primitive.
The connection resources required are defined by implementation and
are outside the scope of this specification. The iSCSI Layer layer may
invoke the Notice_Key_Values Operational Primitive before invoking
the Allocate_Connection_Resources Operational Primitive to request
that the iSER Layer to layer take note of the negotiated values of the iSCSI
keys for the Connection. connection. The specific keys to be passed in as input
qualifiers are implementation dependent. These may include, but are
not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.

To minimize the potential for a denial of denial-of service attack, the iSCSI
Layer
layer MUST NOT request that the iSER Layer to layer allocate the connection
resources necessary to support RCaP until the iSCSI layer is
sufficiently far along in the iSCSI Login Phase that it is reasonably
certain that the peer side is not an attacker. In particular, if the
Login Phase includes a SecurityNegotiation stage, the iSCSI Layer layer
MUST defer the connection resource allocation (i.e. (i.e., invoking the
Allocate_Connection_Resources Operational Primitive) to the
LoginOperationalNegotiation stage ([RFC3720]) [RFC3720] so that the resource
allocation occurs after the authentication phase is completed.

Ko et al. Expires May 2007 33

Among the connection resources allocated at the initiator is the
Inbound RDMA Read Queue Depth (IRD). As described in section Section 9.5.1,
R2Ts are transformed by the target into RDMA Read operations. IRD
limits the maximum number of simultaneously incoming outstanding RDMA
Read Requests per an RCaP Stream from the target to the initiator.
The required value of IRD is outside the scope of the iSER
specification. The iSER Layer layer at the initiator MUST set IRD to 1 or
higher if R2Ts are to be used in the connection. However, the iSER Layer
layer at the initiator MAY set IRD to 0 based on implementation
configuration
configuration, which indicates that no R2Ts will be used on that
connection. Initially, the iSER-IRD value at the initiator SHOULD be
set to the IRD value at the initiator and MUST NOT be more than the
IRD value.

On the other hand, the Outbound RDMA Read Queue Depth (ORD) MAY be
set to 0 0, since the iSER Layer layer at the initiator does not issue RDMA
Read Requests to the target.

Failure to allocate the requested connection resources locally
results in a login failure and its handling is described in section Section
10.1.3.1.

If the iSER Layer layer at the initiator is successful in allocating the
connection resources necessary to support RCaP, the following events
MUST occur in the specified sequence:

1. The iSER Layer layer MUST return a success status to the iSCSI Layer layer in
response to the Allocate_Connection_Resources Operational
Primitive.

2. After the target returns the Login Response with the T bit set to
1 and the NSG field set to FullFeaturePhase, and a status class
of 0 (Success), the iSCSI Layer layer MUST request that the iSER
Layer to layer
transition to iSER-assisted mode by invoking the Enable_Datamover
Operational Primitive with the following qualifiers. (See section
Section 10.1.4.6 for the case when the status class is not
Success.):

a. Connection_Handle that identifies the iSCSI connection.

b. Transport_Connection_Descriptor which that identifies the specific
transport connection associated with the Connection_Handle.

3. If necessary, the iSER Layer layer should enable RCaP and transition
the connection to iSER-assisted mode. When the RCaP is iWARP,

Ko et al. Expires May 2007 34
then this step MUST be done. Not all RCaPs may need it depending
on the RCaP Stream start-up state.

4. The iSER Layer layer MUST send the iSER Hello Message as the first iSER
Message. See Section 5.1.3 on iSER Hello Exchange.

5.1.2

5.1.2. Target Behavior

If the outcome of the iSCSI negotiation is to enable iSER-assisted
mode, then on the target side, prior to sending the Login Response
with the T (Transit) bit set to 1 and the NSG (Next Stage) field set
to FullFeaturePhase, the iSCSI Layer layer MUST request that the iSER Layer to layer
allocate the resources necessary to support RCaP by invoking the
Allocate_Connection_Resources Operational Primitive. The connection
resources required are defined by implementation and are outside the
scope of this specification. Optionally, the iSCSI Layer layer may invoke
the Notice_Key_Values Operational Primitive before invoking the
Allocate_Connection_Resources Operational Primitive to request that
the iSER Layer to layer take note of the negotiated values of the iSCSI keys
for the Connection. connection. The specific keys to be passed in as input
qualifiers are implementation dependent. These may include, but are
not limited to, MaxOutstandingR2T, ErrorRecoveryLevel, etc.

To minimize the potential for a denial of service denial-of-service attack, the iSCSI
Layer
layer MUST NOT request that the iSER Layer to layer allocate the connection
resources necessary to support RCaP until the iSCSI layer is
sufficiently far along in the iSCSI Login Phase that it is reasonably
certain that the peer side is not an attacker. In particular, if the
Login Phase includes a SecurityNegotiation stage, the iSCSI Layer layer
MUST defer the connection resource allocation (i.e. (i.e., invoking the
Allocate_Connection_Resources Operational Primitive) to the
LoginOperationalNegotiation stage ([RFC3720]) [RFC3720] so that the resource
allocation occurs after the authentication phase is completed.

Among the connection resources allocated at the target is the
Outbound RDMA Read Queue Depth (ORD). As described in section Section 9.5.1,
R2Ts are transformed by the target into RDMA Read operations. The
ORD limits the maximum number of simultaneously outstanding RDMA Read
Requests per RCaP Stream from the target to the initiator.
Initially, the iSER-ORD value at the target SHOULD be set to the ORD
value at the target.

On the other hand, the IRD at the target MAY be set to 0 since the
iSER Layer layer at the target does not expect RDMA Read Requests to be
issued by the initiator.

Ko et al. Expires May 2007 35

Failure to allocate the requested connection resources locally
results in a login failure and its handling is described in section Section
10.1.3.1.

If the iSER Layer layer at the target is successful in allocating the
connection resources necessary to support RCaP, the following events
MUST occur in the specified sequence:

1. The iSER Layer layer MUST return a success status to the iSCSI Layer layer in
response to the Allocate_Connection_Resources Operational
Primitive.

2. The iSCSI Layer layer MUST request that the iSER Layer to layer transition to
iSER-assisted mode by invoking the Enable_Datamover Operational
Primitive with the following qualifiers:

a. Connection_Handle that identifies the iSCSI connection.

b. Transport_Connection_Descriptor which that identifies the specific
transport connection associated with the Connection_Handle.

c. The final transport layer (e.g. (e.g., TCP) message containing the
Login Response with the T bit set to 1 and the NSG field set
to FullFeaturePhase FullFeaturePhase.

3. The iSER Layer layer MUST send the final Login Response PDU in the
native transport mode to conclude the iSCSI Login Phase. If the
underlying transport is TCP, then the iSER Layer layer MUST send the
final Login Response PDU in byte stream mode.

4. After sending the final Login Response PDU, the iSER Layer layer should
enable RCaP if necessary and transition the connection to
iSER-assisted iSER-
assisted mode. When the RCaP is iWARP, then this step MUST be
done. Not all RCaPs may need it depending on the RCaP Stream
start-up state.

5. After receiving the iSER Hello Message from the initiator, the
iSER Layer layer MUST respond with the iSER HelloReply Message to be
sent as the first iSER Message. See section 5.1.3 on iSER Hello
Exchange for more details.

Note: In the above sequence, the operations as described in bullets 3
and 4 MUST be performed atomically for iWARP connections. Failure to
do this may result in race conditions.

Ko et al. Expires May 2007 36

5.1.3

5.1.3. iSER Hello Exchange

After the connection transitions into the iSER-assisted mode, the first
iSER Message sent by the iSER Layer layer at the initiator to the target
MUST be the iSER Hello Message. The iSER Hello Message is used by
the iSER Layer layer at the initiator to declare iSER parameters to the
target. See section Section 9.3 on iSER Header Format for the iSER Hello
Message.

In response to the iSER Hello Message, the iSER Layer layer at the target
MUST return the iSER HelloReply Message as the first iSER Message
sent by the target. The iSER HelloReply Message is used by the iSER
Layer
layer at the target to declare iSER parameters to the initiator. See section
Section 9.4 on iSER Header Format for the iSER HelloReply Message.

In the iSER Hello Message, the iSER Layer layer at the initiator declares
the iSER-IRD value to the target.

Upon receiving the iSER Hello Message, the iSER Layer layer at the target
MUST set the iSER-ORD value to the minimum of the iSER-ORD value at
the target and the iSER-IRD value declared by the initiator. The
iSER Layer layer at the target MAY adjust (lower) its ORD value to match
the iSER-ORD value if the iSER-ORD value is smaller than the ORD
value at the target in order to free up the unused resources.

In the iSER HelloReply Message, the iSER Layer layer at the target declares
the iSER-ORD value to the initiator.

Upon receiving the iSER HelloReply Message, the iSER Layer layer at the
initiator MAY adjust (lower) its IRD value to match the iSER-ORD
value in order to free up the unused resources, if the iSER-ORD value
declared by the target is smaller than the iSER-IRD value declared by
the initiator.

It is an iSER level negotiation failure if the iSER parameters
declared in the iSER Hello Message by the initiator are unacceptable
to the target. This includes the following:

* The initiator-declared iSER-IRD value is greater than 0 and the
target-declared iSER-ORD value is 0.

* The initiator-supported and the target-supported iSER protocol
versions do not overlap.

See section Section 10.1.3.2 on for the handling of the error situation.

Ko et al. Expires May 2007 37

5.2

5.2. iSCSI/iSER Connection Termination

5.2.1

5.2.1. Normal Connection Termination at the Initiator

The iSCSI Layer layer at the initiator terminates an iSCSI/iSER connection
normally by invoking the Send_Control Operational Primitive qualified
with the Logout Request PDU. The iSER Layer layer at the initiator MUST
use a SendSE Message to send the Logout Request PDU to the target.
After the iSER Layer layer at the initiator receives the SendSE Message
containing the Logout Response PDU from the target, it MUST notify
the iSCSI Layer layer by invoking the Control_Notify Operational Primitive
qualified with the Logout Response PDU.

After the iSCSI logout process is complete, the iSCSI layer at the
target is responsible for closing the iSCSI/iSER connection as
described in Section 5.2.2. After the RCaP layer at the initiator
reports that the Connection connection has been closed, the iSER Layer layer at the
initiator MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local Mapping(s)
(if any) that associate the ITT(s) used on that connection to the
local STag(s) before notifying the iSCSI Layer layer by invoking the
Connection_Terminate_Notify Operational Primitive.

5.2.2

5.2.2. Normal Connection Termination at the Target

Upon receiving the SendSE Message containing the Logout Request PDU,
the iSER Layer layer at the target MUST notify the iSCSI Layer layer at the
target by invoking the Control_Notify Operational Primitive qualified
with the Logout Request PDU. The iSCSI Layer layer completes the logout
process by invoking the Send_Control Operational Primitive qualified
with the Logout Response PDU. The iSER Layer layer at the target MUST use
a SendSE Message to send the Logout Response PDU to the initiator.
After the iSCSI logout process is complete, the iSCSI Layer layer at the
target MUST request that the iSER Layer layer at the target
to terminate the
RCaP Stream by invoking the Connection_Terminate Operational
Primitive.

As part of the termination process, the RCaP layer MUST close the
Connection.
connection. When the RCaP layer notifies the iSER Layer layer after the
RCaP Stream and the associated Connection connection are terminated, the iSER
Layer
layer MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local and Remote
Mapping(s) (if any) that associate the ITT(s) used on that connection
to the local STag(s) and the Advertised STag(s) respectively.

Ko et al. Expires May 2007 38

5.2.3

5.2.3. Termination without Logout Request/Response PDUs

5.2.3.1

5.2.3.1. Connection Termination Initiated by the iSCSI Layer

The Connection_Terminate Operational Primitive MAY be invoked by the
iSCSI Layer layer to request that the iSER Layer to layer terminate the RCaP Stream
without having previously exchanged the Logout Request and Logout
Response PDUs between the two iSCSI/iSER nodes. As part of the
termination process, the RCaP layer will close the Connection. connection. When
the RCaP layer notifies the iSER Layer layer after the RCaP Stream and the
associated Connection connection are terminated, the iSER Layer layer MUST perform the
following actions.

If the Connection_Terminate Operational Primitive is invoked by the
iSCSI Layer layer at the target, then the iSER Layer layer at the target MUST
deallocate all connection and task resources (if any) associated with
the connection, and invalidate the Local and Remote Mappings (if any)
that associate the ITT(s) used on the connection to the local STag(s)
and the Advertised STag(s) STag(s), respectively.

If the Connection_Terminate Operational Primitive is invoked by the
iSCSI Layer layer at the initiator, then the iSER Layer layer at the initiator
MUST deallocate all connection and task resources (if any) associated
with the connection, and invalidate the Local Mapping(s) (if any)
that associate the ITT(s) used on the connection to the local
STag(s).

5.2.3.2

5.2.3.2. Connection Termination Notification to the iSCSI Layer

If the iSCSI/iSER connection is terminated without the invocation of
Connection_Terminate from the iSCSI Layer, layer, the iSER Layer layer MUST notify
the iSCSI Layer layer that the iSCSI/iSER connection has been terminated by
invoking the Connection_Terminate_Notify Operational Primitive.

Prior to invoking Connection_Terminate_Notify, the iSER Layer layer at the
target MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local and Remote
Mappings (if any) that associate the ITT(s) used on the connection to
the local STag(s) and the Advertised STag(s) STag(s), respectively.

Prior to invoking Connection_Terminate_Notify, the iSER Layer layer at the
initiator MUST deallocate all connection and task resources (if any)
associated with the connection, and invalidate the Local Mappings (if
any) that associate the ITT(s) used on the connection to the local
STag(s).

Ko et al. Expires May 2007 39

If the remote iSCSI/iSER node initiated the closing of the
Connection connection
(e.g., by sending a TCP FIN or TCP RST), the iSER Layer layer MUST notify
the iSCSI Layer layer after the RCaP layer reports that the
Connection connection is
closed by invoking the Connection_Terminate_Notify Operational
Primitive.

Another example of a Connection connection termination without a preceding
logout is when the iSCSI Layer layer at the initiator does an implicit
logout (connection reinstatement).

Ko et al. Expires May 2007 40

6. Login/Text Operational Keys

Certain iSCSI login/text operational keys have restricted usage in
iSER, and additional keys are used to support the iSER protocol
functionality. All other keys defined in [RFC3720] and not discussed
in this section may be used on iSCSI/iSER connections with the same
semantics.

6.1

6.1. HeaderDigest and DataDigest

Irrelevant when: RDMAExtensions=Yes

Negotiations resulting in RDMAExtensions=Yes for a session implies
HeaderDigest=None and DataDigest=None for all connections in that
session and overrides both the default and an explicit setting.

6.2

6.2. MaxRecvDataSegmentLength

For an iSCSI connection belonging to a session in which
RDMAExtensions=Yes was negotiated on the leading connection of the
session, MaxRecvDataSegmentLength need not be declared in the Login
Phase. Instead Instead, InitiatorRecvDataSegmentLength (as described in
section
Section 6.5) and TargetRecvDataSegmentLength (as described in
section Section
6.4) keys are negotiated. The values of the local and remote
MaxRecvDataSegmentLength are derived from the
InitiatorRecvDataSegmentLength and TargetRecvDataSegmentLength keys
even if the MaxRecvDataSegmentLength was is declared during the login
phase. Login
Phase.

In the full feature phase, Full Feature Phase, the initiator MUST consider the value of
its local MaxRecvDataSegmentLength (that it would have declared to
the target) as having the value of InitiatorRecvDataSegmentLength,
and the value of the remote MaxRecvDataSegmentLength (that would have
been declared by the target) as having the value of
TargetRecvDataSegmentLength. Similarly, the target MUST consider the
value of its local MaxRecvDataSegmentLength (that it would have
declared to the initiator) as having the value of
TargetRecvDataSegmentLength, and the value of the remote
MaxRecvDataSegmentLength (that would have been declared by the
initiator) as having the value of InitiatorRecvDataSegmentLength.

The MaxRecvDataSegmentLength key is applicable only for iSCSI
control-type PDUs.

6.3

6.3. RDMAExtensions

Use: LO (leading only)

Ko et al. Expires May 2007 41

Senders: Initiator and Target

Scope: SW (session-wide)

RDMAExtensions=<boolean-value>

Irrelevant when: SessionType=Discovery

Default is No

Result function is AND

This key is used by the initiator and the target to negotiate the support
for iSER-assisted mode. To enable the use of iSER-assisted mode,
both the initiator and the target MUST exchange RDMAExtensions=Yes.

iSER-assisted mode MUST NOT be used if either the initiator or the
target offers RDMAExtensions=No.

An iSER-enabled node is not required to initiate the RDMAExtensions
key exchange if it prefers to operate in the Traditional iSCSI mode.
However, if the RDMAExtensions key is to be negotiated, an initiator
MUST offer the key in the first Login Request PDU in the
LoginOperationalNegotiation stage of the leading connection, and a
target MUST offer the key in the first Login Response PDU with which
it is allowed to do so (i.e., the first Login Response PDU issued
after the first Login Request PDU with the C bit set to 0) in the
LoginOperationalNegotiation stage of the leading connection. In
response to the offered key=value pair of RDMAExtensions=yes, an
initiator MUST respond in the next Login Request PDU with which it is
allowed to do so, and a target MUST respond in the next Login
Response PDU with which it is allowed to do so.

Negotiating the RDMAExtensions key first enables a node to negotiate
the optimal value for other keys. Certain iSCSI keys such as
MaxBurstLength, MaxOutstandingR2T, ErrorRecoveryLevel, InitialR2T,
ImmediateData, etc., may be negotiated differently depending on
whether the connection is in Traditional iSCSI mode or iSER-assisted
mode.

6.4

6.4. TargetRecvDataSegmentLength

Use: IO (Initialize only)

Senders: Initiator and Target

Scope: CO (connection-only)

Ko et al. Expires May 2007 42 (connection-only)

Irrelevant when: RDMAExtensions=No

TargetRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>

Default is 8192 bytes

Result function is minimum

This key is relevant only for the iSCSI connection of an iSCSI
session if RDMAExtensions=Yes was is negotiated on the leading connection
of the session. It is used by the initiator and the target to negotiate
the maximum size of the data segment that an initiator may send to
the target in an iSCSI control-type PDU in the
full feature phase. Full Feature Phase.
For SCSI Command PDUs and SCSI Data-out PDUs containing non-immediate
unsolicited data to be sent by the initiator, the initiator MUST send
all non-Final PDUs with a data segment size of exactly
TargetRecvDataSegmentLength whenever the PDUs constitute a data
sequence whose size is larger than TargetRecvDataSegmentLength.

6.5

6.5. InitiatorRecvDataSegmentLength

Use: IO (Initialize only)

Senders: Initiator and Target

Scope: CO (connection-only)

Irrelevant when: RDMAExtensions=No

InitiatorRecvDataSegmentLength=<numerical-value-512-to-(2**24-1)>

Default is 8192 bytes

Result function is minimum

6.6 Full Feature Phase.

6.6. OFMarker and IFMarker

Irrelevant when: RDMAExtensions=Yes

Ko et al. Expires May 2007 43

Negotiations resulting in RDMAExtensions=Yes for a session implies
OFMarker=No and IFMarker=No for all connections in that session and
overrides both the default and an explicit setting.

6.7

6.7. MaxOutstandingUnexpectedPDUs

Use: LO (leading only), Declarative

Senders: Initiator and Target

Scope: SW (session-wide)

Irrelevant when: RDMAExtensions=No

MaxOutstandingUnexpectedPDUs=<numerical-value-from-2-to-(2**32-1) |
0>

Default is 0
This key is used by the initiator and the target to declare the
maximum number of outstanding "unexpected" iSCSI control-type PDUs
that it can receive in the full feature phase. Full Feature Phase. It is intended to
allow the receiving side to determine the amount of buffer resources
needed beyond the normal flow control mechanism available in iSCSI.
An initiator or target should select a value such that it would not
impose an unnecessary constraint on the iSCSI Layer layer under normal
circumstances. The value of 0 is defined to indicate that the
declarer has no limit on the maximum number of outstanding
"unexpected" iSCSI control-type PDUs that it can receive. See
sections
Sections 8.1.1 and 8.1.2 for the usage of this key. Note that iSER
Hello and HelloReply Messages are not iSCSI control-type PDUs and are
not affected by this key.

Ko et al. Expires May 2007 44

7. iSCSI PDU Considerations

When a connection is in the iSER-assisted mode, two types of message
transfers are allowed between the iSCSI Layer layer at the initiator and
the iSCSI Layer layer at the target. These are known as the iSCSI data-
type PDUs and the iSCSI control-type PDUs PDUs, and these terms are
described in the following sections.

7.1

7.1. iSCSI Data-Type PDU

An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
transfer, transparent to the remote iSCSI layer, to take place
between the peer iSCSI nodes in the full feature phase of an
iSCSI/iSER connection. An iSCSI data-type PDU, when requested for
transmission by the iSCSI Layer layer in the sending node, results in the
data being transferred without the participation of the iSCSI Layers layers
at the sending and the receiving nodes. This is due to the fact that
the PDU itself is not delivered as-is to the iSCSI Layer layer in the
receiving node. Instead, the data transfer operations are
transformed into the appropriate RDMA operations which that are handled by
the RDMA-Capable Controller. The set of iSCSI data-type PDUs
consists of SCSI Data-in PDUs and R2T PDUs.

If the invocation of the Operational Primitive by the iSCSI Layer layer to
request that the iSER Layer to layer process an iSCSI data-type PDU is
qualified with Notify_Enable set, then upon completing the RDMA
operation, the iSER Layer layer at the target MUST notify the iSCSI Layer layer
at the target by invoking the Data_Completion_Notify Operational
Primitive qualified with ITT and SN. There is no data completion
notification at the initiator since the RDMA operations are
completely handled by the RDMA-Capable Controller at the initiator
and the iSER Layer layer at the initiator is not involved with the data
transfer associated with iSCSI data-type PDUs.

If the invocation of the Operational Primitive by the iSCSI Layer layer to
request that the iSER Layer to layer process an iSCSI data-type PDU is
qualified with Notify_Enable cleared, then upon completing the RDMA
operation, the iSER Layer layer at the target MUST NOT notify the iSCSI
Layer
layer at the target and MUST NOT invoke the Data_Completion_Notify
Operational Primitive.

If an operation associated with an iSCSI data-type PDU fails for any
reason, the contents of the Data Sink buffers associated with the
operation are considered indeterminate.

Ko et al. Expires May 2007 45

7.2

7.2. iSCSI Control-Type PDU

Any iSCSI PDU that is not an iSCSI data-type PDU and also not a SCSI
Data-out PDU carrying solicited data is defined as an iSCSI control-
type PDU. The iSCSI Layer layer invokes the Send_Control Operational
Primitive to request that the iSER Layer to layer process an iSCSI control-type control-
type PDU. iSCSI control-type PDUs are transferred using Send Message
Types of RCaP. Specifically, it is to be noted note that SCSI Data-Out Data-out PDUs carrying
unsolicited data are defined as iSCSI control-type PDUs. See section Section
7.3.4 on the treatment of SCSI Data-out PDUs.

When the iSER Layer layer receives an iSCSI control-type PDU, it MUST
notify the iSCSI Layer layer by invoking the Control_Notify Operational
Primitive qualified with the iSCSI control-type PDU.

7.3

7.3. iSCSI PDUs

This section describes the handling of each of the iSCSI PDU types by
the iSER Layer. layer. The iSCSI Layer layer requests that the iSER Layer to layer process
the iSCSI PDU by invoking the appropriate Operational Primitive. A
Connection_Handle MUST qualify each of these invocations. In
addition, BHS and the optional AHS of the iSCSI PDU as defined in
[RFC3720] MUST qualify each of the invocations. The qualifying
Connection_Handle, the BHS BHS, and the AHS are not explicitly listed in
the subsequent sections.

7.3.1

7.3.1. SCSI Command

Type: control-type PDU

PDU-specific qualifiers (for SCSI Write or bidirectional command):
ImmediateDataSize, UnsolicitedDataSize, DataDescriptorOut

PDU-specific qualifiers (for SCSI Read read or bidirectional command):
DataDescriptorIn
The iSER Layer layer at the initiator MUST send the SCSI command in a
SendSE Message to the target.

For a SCSI Write or bidirectional command, the iSCSI Layer layer at the
initiator MUST invoke the Send_Control Operational Primitive as
follows:

* If there is immediate data to be transferred for the SCSI write Write or
bidirectional command, the qualifier ImmediateDataSize MUST be
used to define the number of bytes of immediate unsolicited data

Ko et al. Expires May 2007 46
to be sent with the write Write or bidirectional command, and the
qualifier DataDescriptorOut MUST be used to define the initiator's
I/O Buffer containing the SCSI Write data.

* If there is unsolicited data to be transferred for the SCSI Write
or bidirectional command, the qualifier UnsolicitedDataSize MUST
be used to define the number of bytes of immediate and non-
immediate unsolicited data for the command. The iSCSI Layer layer will
issue one or more SCSI Data-out PDUs for the non-immediate
unsolicited data. See Section 7.3.4 on SCSI Data-out.

* If there is solicited data to be transferred for the SCSI Write write or
bidirectional command, as indicated by the Expected Data Transfer
Length in the SCSI Command PDU exceeding the value of
UnsolicitedDataSize, the iSER Layer layer at the initiator MUST do the
following:

a. It MUST allocate a Write STag for the I/O Buffer defined by
the qualifier DataDescriptorOut. The DataDescriptorOut
describes the I/O buffer starting with the immediate
unsolicited data (if any), followed by the non-immediate
unsolicited data (if any) and solicited data. This means
that the BufferOffset for the SCSI Data-out for this
command is equal to the TO. This implies that a zero TO
for this STag points to the beginning of this I/O Buffer.

b. It MUST establish a Local Mapping that associates the
Initiator Task Tag (ITT) to the Write STag.

c. It MUST Advertise the Write STag to the target by sending
it as the Write STag in the iSER header of the iSER Message
(the payload of the SendSE Message of RCaP) containing the
SCSI Write write or bidirectional command PDU. See section Section 9.2
on iSER Header Format for the iSCSI Control-Type PDU.

For a SCSI Read read or bidirectional command, the iSCSI Layer layer at the
initiator MUST invoke the Send_Control Operational Primitive
qualified with DataDescriptorIn DataDescriptorIn, which defines the initiator's I/O
Buffer for receiving the SCSI Read data. The iSER Layer layer at the
initiator MUST do the following:

a. It MUST allocate a Read STag for the I/O Buffer.

b. It MUST establish a Local Mapping that associates the
Initiator Task Tag (ITT) to the Read STag.

Ko et al. Expires May 2007 47

c. It MUST Advertise the Read STag to the target by sending it
as the Read STag in the iSER header of the iSER Message
(the payload of the SendSE Message of RCaP) containing the
SCSI
Read read or bidirectional command PDU. See section Section 9.2 on
iSER Header Format for the iSCSI Control-Type PDU.

If the amount of unsolicited data to be transferred in a SCSI
Command command
exceeds TargetRecvDataSegmentLength, then the iSCSI Layer layer at the
initiator MUST segment the data into multiple iSCSI control-type
PDUs, with the data segment length in all PDUs generated except the
last one having exactly the size TargetRecvDataSegmentLength. The
data segment length of the last iSCSI control-type PDU carrying the
unsolicited data can be up to TargetRecvDataSegmentLength.

When the iSER Layer layer at the target receives the SCSI Command, command, it MUST
establish a Remote Mapping that associates the ITT to the Advertised
Write STag and the Read STag if present in the iSER header. The
Write STag is used by the iSER Layer layer at the target in handling the
data transfer associated with the R2T PDU(s) as described in section Section
7.3.6. The Read STag is used in handling the SCSI Data-in PDU(s)
from the iSCSI Layer layer at the target as described in section Section 7.3.5.

7.3.2

7.3.2. SCSI Response

Type: control-type PDU

PDU-specific qualifiers: DataDescriptorStatus

The iSCSI Layer layer at the target MUST invoke the Send_Control
Operational Primitive qualified with DataDescriptorStatus DataDescriptorStatus, which
defines the buffer containing the sense and response information.
The iSCSI Layer layer at the target MUST always return the SCSI status for
a SCSI command in a separate SCSI Response PDU. "Phase collapse" for
transferring SCSI status in a SCSI Data-in PDU MUST NOT be used. The
iSER Layer layer at the target sends the SCSI Response PDU according to the
following rules:

* If no STags were are Advertised by the initiator in the iSER Message
containing the SCSI command PDU, then the iSER Layer layer at the target
MUST send a SendSE Message containing the SCSI Response PDU.

* If the initiator Advertised a Read STag in the iSER Message
containing the SCSI Command PDU, then the iSER Layer layer at the target
MUST send a SendInvSE Message containing the SCSI Response PDU.
The header of the SendInvSE Message MUST carry the Read STag to be
invalidated at the initiator.

Ko et al. Expires May 2007 48

* If the initiator Advertised only the Write STag in the iSER
Message containing the SCSI command Command PDU, then the iSER Layer layer at
the target MUST send a SendInvSE Message containing the SCSI
Response PDU. The header of the SendInvSE Message MUST carry the
Write STag to be invalidated at the initiator.

When the iSCSI Layer layer at the target invokes the Send_Control
Operational Primitive to send the SCSI Response PDU, the iSER Layer layer
at the target MUST invalidate the Remote Mapping that associates the
ITT to the Advertised STag(s) before transferring the SCSI Response
PDU to the initiator.

Upon receiving the SendInvSE Message containing the SCSI Response PDU
from the target, the RCaP layer at the initiator will invalidate the
STag specified in the header. The iSER Layer layer at the initiator MUST
ensure that the correct STag is invalidated. If both the Read and
the Write STags were are Advertised earlier by the initiator, then the
iSER Layer layer at the initiator MUST explicitly invalidate the Write STag
upon receiving the SendInvSE Message because the header of the
SendInvSE Message can only carry one STag (in this case case, the Read
STag) to be invalidated.

The iSER Layer layer at the initiator MUST ensure the invalidation of the
STag(s) used in a command before notifying the iSCSI Layer layer at the
initiator by invoking the Control_Notify Operational Primitive
qualified with the SCSI Response. This precludes the possibility of
using the STag(s) after the completion of the command command, thereby
causing data corruption.

When the iSER Layer layer at the initiator receives the SendSE or the
SendInvSE Message containing the SCSI Response PDU, it SHOULD
invalidate the Local Mapping that associates the ITT to the local
STag(s). The iSER Layer layer MUST ensure that all local STag(s)
associated with the ITT are invalidated before notifying the iSCSI
Layer
layer of the SCSI Response PDU by invoking the Control_Notify
Operational Primitive qualified with the SCSI Response PDU.

7.3.3

7.3.3. Task Management Function Request/Response

Type: control-type PDU

PDU-specific qualifiers (for TMF Request): DataDescriptorOut,
DataDescriptorIn

The iSER Layer layer MUST use a SendSE Message to send the Task Management
Function Request/Response PDU.

Ko et al. Expires May 2007 49

For the Task Management Function Request with the TASK REASSIGN
function, the iSER Layer layer at the initiator MUST do the following:

* It MUST use the ITT as specified in the Referenced Task Tag from
the Task Management Function Request PDU to locate the existing
STag(s), if any, in the Local Mapping(s) that associates the ITT
to the local STag(s).

* It MUST invalidate the existing STag(s), if any, and the Local
Mapping(s) that associates the ITT to the local STag(s).

* It MUST allocate a Read STag for the I/O Buffer as defined by the
qualifier DataDescriptorIn if the Send_Control Operational
Primitive invocation is qualified with DataDescriptorIn.

* It MUST allocate a Write STag for the I/O Buffer as defined by the
qualifier DataDescriptorOut if the Send_Control Operational
Primitive invocation is qualified with DataDescriptorOut.

* If STags are allocated, it MUST establish a new Local Mapping(s)
that associate the ITT to the allocated STag(s).

* It MUST Advertise the STags, if allocated, to the target in the
iSER header of the SendSE Message carrying the iSCSI PDU, as
described in section Section 9.2.

For the Task Management Function Request with the TASK REASSIGN
function for a SCSI Read read or bidirectional command, the iSCSI Layer layer at
the initiator MUST set ExpDataSN to 0 since the data transfer and
acknowledgements happen transparently to the iSCSI Layer layer at the
initiator. This provides the flexibility to the iSCSI Layer layer at the
target to request transmission of only the unacknowledged data as
specified in [RFC3720].

When the iSER Layer layer at the target receives the Task Management
Function Request with the TASK REASSIGN function, it MUST do the
following:

* It MUST use the ITT as specified in the Referenced Task Tag from
the Task Management Function Request PDU to locate the mappings
that associate the ITT to the Advertised STag(s) and the local
STag(s), if any.

* It MUST invalidate the local STaq(s), STag(s), if any, associated with the
ITT.

Ko et al. Expires May 2007 50

* It MUST replace the Advertised STag(s) in the Remote Mapping that
associates the ITT to the Advertised STag(s) with the Write STag
and the Read STag if present in the iSER header. The Write STag
is used in the handling of the R2T PDU(s) from the iSCSI Layer layer at
the target as described in section Section 7.3.6. The Read STag is used
in the handling of the SCSI Data-in PDU(s) from the iSCSI Layer layer at
the target as described in section Section 7.3.5.

7.3.4

7.3.4. SCSI Data-out Data-Out

Type: control-type PDU

PDU-specific qualifiers: DataDescriptorOut

The iSCSI Layer layer at the initiator MUST invoke the Send_Control
Operational Primitive qualified with DataDescriptorOut DataDescriptorOut, which defines
the initiator's I/O Buffer containing unsolicited SCSI Write data.

If the amount of unsolicited data to be transferred as SCSI Data-out
exceeds TargetRecvDataSegmentLength, then the iSCSI Layer layer at the
initiator MUST segment the data into multiple iSCSI control-type
PDUs, with the DataSegmentLength having the value of
TargetRecvDataSegmentLength in all PDUs generated except the last
one. The DataSegmentLength of the last iSCSI control-type PDU
carrying the unsolicited data can be up to
TargetRecvDataSegmentLength. The iSCSI Layer layer at the target MUST
perform the reassembly function for the unsolicited data.

For unsolicited data, if the F bit is set to 0 in a SCSI Data-out
PDU, the iSER Layer layer at the initiator MUST use a Send Message to send
the SCSI Data-out PDU. If the F bit is set to 1, the iSER Layer layer at
the initiator MUST use a SendSE Message to send the SCSI Data-out
PDU.

Note that for solicited data, the SCSI Data-out PDUs are not used
since R2T PDUs are not delivered to the iSCSI layer at the initiator; instead
instead, R2T PDUs are transformed by the iSER layer at the target
into RDMA Read operations. (See section Section 7.3.6.)

7.3.5

7.3.5. SCSI Data-in Data-In

Type: data-type PDU

PDU-specific qualifiers: DataDescriptorIn

When the iSCSI Layer layer at the target is ready to return the SCSI Read
data to the initiator, it MUST invoke the Put_Data Operational

Ko et al. Expires May 2007 51
Primitive qualified with DataDescriptorIn DataDescriptorIn, which defines the SCSI
Data-in buffer. See section Section 7.1 on the general requirement on the
handling of iSCSI data-type PDUs. SCSI Data-in PDU(s) are used in
SCSI Read data transfer as described in section Section 9.5.2.

The iSER Layer layer at the target MUST do the following for each
invocation of the Put_Data Operational Primitive:

1. It MUST use the ITT in the SCSI Data-in PDU to locate the remote
Read STag in the Remote Mapping that associates the ITT to
Advertised STag(s). The Remote Mapping was established earlier
by the iSER Layer layer at the target when the SCSI Read Command read command was
received from the initiator.

2. It MUST generate and send an RDMA Write Message containing the
read data to the initiator.

a. It MUST use the remote Read STag as the Data Sink STag of the
RDMA Write Message.

b. It MUST use the Buffer Offset from the SCSI Data-in PDU as
the Data Sink Tagged Offset of the RDMA Write Message.

c. It MUST use DataSegmentLength from the SCSI Data-in PDU to
determine the amount of data to be sent in the RDMA Write
Message.

3. It MUST associate DataSN and ITT from the SCSI Data-in PDU with
the RDMA Write operation. If the Put_Data Operational Primitive
invocation was qualified with Notify_Enable set, then when the
iSER Layer layer at the target receives a completion from the RCaP
layer for the RDMA Write Message, the iSER Layer layer at the target
MUST notify the iSCSI Layer layer by invoking the
Data_Completion_Notify Operational Primitive qualified with
DataSN and ITT. Conversely, if the Put_Data Operational
Primitive invocation was qualified with Notify_Enable cleared,
then the iSER Layer layer at the target MUST NOT notify the iSCSI
Layer layer
on completion and MUST NOT invoke the Data_Completion_Notify
Operational Primitive.

When the A-bit is set to 1 in the SCSI Data-in PDU, the iSER Layer layer at
the target MUST notify the iSCSI Layer layer at the target when the data
transfer is complete at the initiator. To perform this additional
function, the iSER Layer layer at the target can take advantage of the
operational ErrorRecoveryLevel if previously disclosed by the iSCSI Layer
layer via an earlier invocation of the Notice_Key_Values Operational
Primitive. There are two approaches that can be taken:

Ko et al. Expires May 2007 52

1. If the iSER Layer layer at the target knows that the operational
ErrorRecoveryLevel is 2, or if the iSER Layer layer at the target does
not know the operational ErrorRecoveryLevel, then the iSER Layer layer
at the target MUST issue a zero-length RDMA Read Request Message
following the RDMA Write Message. When the iSER Layer layer at the
target receives a completion for the RDMA Read Request Message
from the RCaP layer, implying that the RDMA-Capable Controller at
the initiator has completed processing the RDMA Write Message due
to the completion ordering semantics of RCaP, the iSER Layer layer at
the target MUST notify the iSCSI Layer layer at the target by invoking
the Data_Ack_Notify Operational Primitive qualified with ITT and
DataSN (see section Section 3.2.3).

2. If the iSER Layer layer at the target knows that the operational
ErrorRecoveryLevel is 1, then the iSER Layer layer at the target MUST
do one of the following:

a. It MUST notify the iSCSI Layer layer at the target by invoking the
Data_Ack_Notify Operational Primitive qualified with ITT and
DataSN (see section Section 3.2.3) when it receives the local
completion from the RCaP layer for the RDMA Write Message.
This is allowed since digest errors do not occur in iSER (see section
Section 10.1.4.2) and a CRC error will cause the connection
to be terminated and the task to be terminated anyway. The
local RDMA Write completion from the RCaP layer guarantees
that the RCaP layer will not access the I/O Buffer again to
transfer the data associated with that RDMA Write operation.

b. Alternatively, it MUST use the same procedure for handling
the data transfer completion at the initiator as for
ErrorRecoveryLevel 2.

It should be noted

Note that the iSCSI Layer layer at the target cannot set the A-bit to 1 if
the ErrorRecoveryLevel=0.

The SCSI status MUST always be returned in a separate SCSI Response
PDU. The S bit in the SCSI Data-in PDU MUST always be set to 0.
There MUST NOT be a "phase collapse" in the SCSI Data-in PDU.

Since the RDMA Write Message only transfers the data portion of the
SCSI Data-in PDU but not the control information in the header, such
as ExpCmdSN, if timely updates of such information is are crucial, the
iSCSI Layer layer at the initiator MAY issue NOP-Out PDUs to request that
the iSCSI Layer layer at the target to respond with the information using NOP-
In PDUs.

Ko et al. Expires May 2007 53

7.3.6

7.3.6. Ready To to Transfer (R2T)

Type: data-type PDU

PDU-specific qualifiers: DataDescriptorOut

In order to send an R2T PDU, the iSCSI Layer layer at the target MUST
invoke the Get_Data Operational Primitive qualified with
DataDescriptorOut
DataDescriptorOut, which defines the I/O Buffer for receiving the
SCSI Write data from the initiator. See section Section 7.1 on the general
requirements on the handling of iSCSI data-type PDUs.

The iSER Layer layer at the target MUST do the following for each
invocation of the Get_Data Operational Primitive:

1. It MUST ensure a valid local STag for the I/O Buffer and a valid
Local Mapping that associates the Initiator Task Tag (ITT) to the
local STag. This may involve allocating a valid local STag and
establishing a Local Mapping.

2. It MUST use the ITT in the R2T to locate the remote Write STag in
the Remote Mapping that associates the ITT to Advertised STag(s).
The Remote Mapping was is established earlier by the iSER
Layer layer at
the target when the iSER Message containing the Advertised Write
STag and the SCSI Command PDU for a SCSI Write write or bidirectional
command was is received from the initiator.

3. If the iSER-ORD value at the target is set to 0, the iSER Layer layer
at the target MUST terminate the connection and free up the
resources associated with the connection (as described in Section
5.2.3) if it received receives the R2T PDU from the iSCSI Layer layer at the
target. Upon termination of the connection, the iSER Layer layer at
the target MUST notify the iSCSI Layer layer at the target by invoking
the Connection_Terminate_Notify Operational Primitive.

4. If the iSER-ORD value at the target is set to greater than 0, the
iSER Layer layer at the target MUST transform the R2T PDU into an RDMA
Read Request Message. While transforming the R2T PDU, the iSER Layer
layer at the target MUST ensure that the number of outstanding
RDMA Read Request Messages does not exceed the iSER-ORD value.
To transform the R2T PDU, the iSER Layer layer at the target:

a. MUST derive the local STag and local Tagged Offset from the
DataDescriptorOut that qualified the Get_Data invocation.

b. MUST use the local STag as the Data Sink STag of the RDMA
Read Request Message.

Ko et al. Expires May 2007 54

c. MUST use the local Tagged Offset as the Data Sink Tagged
Offset of the RDMA Read Request Message.

d. MUST use the Desired Data Transfer Length from the R2T PDU as
the RDMA Read Message Size of the RDMA Read Request Message.

e. MUST use the remote Write STag as the Data Source STag of the
RDMA Read Request Message.

f. MUST use the Buffer Offset from the R2T PDU as the Data
Source Tagged Offset of the RDMA Read Request Message.

5. It MUST associate R2TSN and ITT from the R2T PDU with the RDMA
Read operation. If the Get_Data Operational Primitive invocation was
is qualified with Notify_Enable set, then when the iSER Layer layer at
the target receives a completion from the RCaP layer for the RDMA
Read operation, the iSER Layer layer at the target MUST notify the
iSCSI Layer layer by invoking the Data_Completion_Notify Operational
Primitive qualified with R2TSN and ITT. Conversely, if the
Get_Data Operational Primitive invocation was is qualified with
Notify_Enable cleared, then the iSER Layer layer at the target MUST NOT
notify the iSCSI
Layer layer on completion and MUST NOT invoke the
Data_Completion_Notify Operational Primitive.

When the RCaP layer at the initiator receives a valid RDMA Read
Request Message, it will return an RDMA Read Response Message
containing the solicited write data to the target. When the RCaP
layer at target receives the RDMA Read Response Message from the
initiator, it will place the solicited data in the I/O Buffer
referenced by the Data Sink STag in the RDMA Read Response Message.

Since the RDMA Read Request Message from the target does not transfer
the control information in the R2T PDU PDU, such as ExpCmdSN, if timely
updates of such information is are crucial, the iSCSI Layer layer at the
initiator MAY issue NOP-Out PDUs to request that the iSCSI Layer layer at
the target to respond with the information using NOP-In PDUs.

Similarly, since the RDMA Read Response Message from the initiator
only transfers the data but not the control information normally
found in the SCSI Data-out PDU, such as ExpStatSN, if timely updates
of such information is are crucial, the iSCSI Layer layer at the target MAY
issue NOP-In PDUs to request that the iSCSI Layer layer at the initiator to
respond with the information using NOP-Out PDUs.

Ko et al. Expires May 2007 55

7.3.7

7.3.7. Asynchronous Message

Type: control-type PDU

PDU-specific qualifiers: DataDescriptorSense

The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
qualified with DataDescriptorSense DataDescriptorSense, which defines the buffer
containing the sense and iSCSI Event information. The iSER Layer layer
MUST use a SendSE Message to send the Asynchronous Message PDU.

7.3.8

7.3.8. Text Request & and Text Response

Type: control-type PDU

PDU-specific qualifiers: DataDescriptorTextOut (for Text
Request), DataDescriptorIn (for Text Response)

The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
qualified with DataDescriptorTextOut (or DataDescriptorIn) DataDescriptorIn), which
defines the Text Request (or Text Response) buffer. The iSER Layer layer
MUST use SendSE Messages to send the Text Request (or Text Response
PDUs).

7.3.9

7.3.9. Login Request & and Login Response

During the login negotiation, the iSCSI Layer layer interacts with the
transport layer directly and the iSER Layer layer is not involved. See
section
Section 5.1 on iSCSI/iSER Connection Setup. connection setup. If the underlying
transport is TCP, the Login Request PDUs and the Login Response PDUs
are exchanged when the connection between the initiator and the
target is still in the byte stream mode.

The iSCSI Layer layer MUST not send a Login Request (or a Login Response)
PDU during the full feature phase. Full Feature Phase. A Login Request (or a Login
Response) PDU, if used, MUST be treated as an iSCSI protocol error.
The iSER Layer layer MAY reject such a PDU from the iSCSI Layer layer with an
appropriate error code. If a Login Request PDU is received by the
iSCSI Layer layer at the target, it MUST respond with a Reject PDU with a
reason code of "protocol error".

7.3.10

7.3.10. Logout Request & and Logout Response

Type: control-type PDU

PDU-specific qualifiers: None

Ko et al. Expires May 2007 56

The iSER Layer layer MUST use a SendSE Message to send the Logout Request
or Logout Response PDU. Section Sections 5.2.1 and 5.2.2 describe the
handling of the Logout Request and the Logout Response at the
initiator and the target and the interactions between the initiator
and the target to terminate a connection.

7.3.11

7.3.11. SNACK Request

Since HeaderDigest and DataDigest must be negotiated to "None", there
are no digest errors when the connection is in iSER-assisted mode. Also
Also, since RCaP delivers all messages in the order they were sent,
there are no sequence errors when the connection is in iSER-
assisted iSER-assisted
mode. Therefore Therefore, the iSCSI Layer layer MUST NOT send SNACK Request PDUs.
A SNCAK Request PDU, if used, MUST be treated as an iSCSI protocol
error. The iSER Layer layer MAY reject such a PDU from the iSCSI Layer layer
with an appropriate error code. If a SNACK Request PDU is received
by the iSCSI Layer layer at the target, it MUST respond with a Reject PDU
with a reason code of "protocol error".

7.3.12

7.3.12. Reject

Type: control-type PDU

PDU-specific qualifiers: DataDescriptorReject

The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
qualified with DataDescriptorReject DataDescriptorReject, which defines the Reject buffer.
The iSER Layer layer MUST use a SendSE Message to send the Reject PDU.

7.3.13

7.3.13. NOP-Out & and NOP-In

Type: control-type PDU

PDU-specific qualifiers: DataDescriptorNOPOut (for NOP-Out),
DataDescriptorNOPIn (for NOP-In)

The iSCSI Layer layer MUST invoke the Send_Control Operational Primitive
qualified with DataDescriptorNOPOut (or DataDescriptorNOPIn) DataDescriptorNOPIn), which
defines the Ping (or Return Ping) data buffer. The iSER Layer layer MUST
use SendSE Messages to send the NOP-Out (or NOP-In) PDU.

Ko et al. Expires May 2007 57

8. Flow Control and STag Management

8.1

8.1. Flow Control for RDMA Send Message Types

Send Message Types in RCaP are used by the iSER Layer layer to transfer
iSCSI control-type PDUs. Each Send Message Type in RCaP consumes an
Untagged Buffer at the Data Sink. However, neither the RCaP layer
nor the iSER Layer layer provides an explicit flow control mechanism for
the Send Message Types. Therefore, the iSER Layer layer SHOULD provision
enough Untagged buffers for handling incoming Send Message Types to
prevent buffer exhaustion at the RCaP layer. If buffer exhaustion
occurs, it may result in the termination of the connection.

An implementation may choose to satisfy the buffer requirement by
using a common buffer pool shared across multiple connections, with
usage limits on a per connection per-connection basis and usage limits on the buffer
pool itself. In such an implementation, exceeding the buffer usage
limit for a connection or the buffer pool itself may trigger
interventions from the iSER Layer layer to replenish the buffer pool and/or
to isolate the connection causing the problem.

iSER also provides the MaxOutstandingUnexpectedPDUs key to be used by
the initiator and the target to declare the maximum number of
outstanding "unexpected" control-type PDUs that it can receive. It
is intended to allow the receiving side to determine the amount of
buffer resources needed beyond the normal flow control mechanism
available in iSCSI.

The buffer resources required at both the initiator and the target as
a result of control-type PDUs sent by the initiator is described in section
Section 8.1.1. The buffer resources required at both the initiator
and target as a result of control-type PDUs sent by the target is
described in section Section 8.1.2.

8.1.1

8.1.1. Flow Control for Control-Type PDUs from the Initiator

The control-type PDUs that can be sent by an initiator to a target
can be grouped into the following categories:

1. Regulated: Control-type PDUs in this category are regulated by
the iSCSI CmdSN window mechanism and the immediate flag is not
set.

2. Unregulated but Expected: Control-type PDUs in this category are
not regulated by the iSCSI CmdSN window mechanism but are
expected by the target.

Ko et al. Expires May 2007 58

3. Unregulated and Unexpected: Control-type PDUs in this category
are not regulated by the iSCSI CmdSN window mechanism and are
"unexpected" by the target.

8.1.1.1

8.1.1.1. Control-Type PDUs from the Initiator in the Regulated Category

Control-type PDUs that can be sent by the initiator in this category
are regulated by the iSCSI CmdSN window mechanism and the immediate
flag is not set.

The queuing capacity required of the iSCSI layer at the target is
described in section Section 3.2.2.1 of [RFC3720]. For each of the control-
type PDUs that can be sent by the initiator in this category, the
initiator MUST provision for the buffer resources required for the
corresponding control-type PDU sent as a response from the target.
The following is a list of the PDUs that can be sent by the initiator
and the PDUs that are sent by the target in response:

a. When an initiator sends a SCSI Command PDU, it expects a SCSI
Response PDU from the target.

b. When the initiator sends a Task Management Function Request
PDU, it expects a Task Management Function Response PDU from
the target.

c. When the initiator sends a Text Request PDU, it expects a
Text Response PDU from the target.

d. When the initiator sends a Logout Request PDU, it expects a
Logout Response PDU from the target.

e. When the initiator sends a NOP-Out PDU as a ping request with
ITT != 0xffffffff and TTT = 0xffffffff, it expects a NOP-In
PDU from the target with the same ITT and TTT as in the ping
request.

The response from the target for any of the PDUs enumerated here may
alternatively be in the form of a Reject PDU sent instead before the
task is active, as described in section Section 6.3 of [RFC3720].

8.1.1.2

8.1.1.2. Control-Type PDUs from the Initiator in the Unregulated but
Expected Category

For the control-type PDUs in the Unregulated but Expected category,
the amount of buffering resources required at the target can be
predetermined. The following is a list of the PDUs in this category:

Ko et al. Expires May 2007 59

a. SCSI Data-out PDUs are used by the initiator to send
unsolicited data. The amount of buffer resources required by
the target can be determined using FirstBurstLength. Note
that SCSI Data-out PDUs are not used for solicited data since
the R2T PDU which that is used for solicitation is transformed into
RDMA Read operations by the iSER layer at the target. See section
Section 7.3.4.

b. A NOP-Out PDU with TTT != 0xffffffff is sent as a ping
response by the initiator to the NOP-In PDU sent as a ping
request by the target.

8.1.1.3

8.1.1.3. Control-Type PDUs from the Initiator in the Unregulated and
Unexpected Category

PDUs in the Unregulated and Unexpected category are PDUs with the
immediate flag set. The number of PDUs in this category which that can be
sent by an initiator is controlled by the value of
MaxOutstandingUnexpectedPDUs declared by the target. (See section
6.7.) target (see Section
6.7). After a PDU in this category is sent by the initiator, it is
outstanding until it is retired. At any time, the number of
outstanding unexpected PDUs MUST not exceed the value of
MaxOutstandingUnexpectedPDUs declared by the target.

The target uses the value of MaxOutstandingUnexpectedPDUs that it
declared to determine the amount of buffer resources required for
control-type PDUs in this category that can be sent by an initiator.
For the initiator, for each of the control-type PDUs that can be sent
in this category, the initiator MUST provision for the buffer
resources if required for the corresponding control-type PDU that can
be sent as a response from the target.

An outstanding PDU in this category is retired as follows. If the
CmdSN of the PDU sent by the initiator in this category is x, the PDU
is outstanding until the initiator sends a non-immediate control-type
PDU on the same connection with CmdSN = y (where y is at least x) and
the target responds with a control-type PDU on any connection where
ExpCmdSN is at least y+1.

When the number of outstanding unexpected control-type PDUs equals
MaxOutstandingUnexpectedPDUs, the iSCSI Layer layer at the initiator MUST
NOT generate any unexpected PDUs which that otherwise it would have
generated, even if it is intended for immediate delivery.

Ko et al. Expires May 2007 60

8.1.2

8.1.2. Flow Control for Control-Type PDUs from the Target

Control-type PDUs that can be sent by a target and are expected by
the initiator are listed in the Regulated category. (See category (see section
8.1.1.1.)
8.1.1.1).

For the control-type PDUs that can be sent by a target and are
unexpected by the initiator, the number is controlled by
MaxOutstandingUnexpectedPDUs declared by the initiator. (See
section 6.7.) initiator (see Section
6.7). After a PDU in this category is sent by a target, it is
outstanding until it is retired. At any time, the number of
outstanding unexpected PDUs MUST not exceed the value of
MaxOutstandingUnexpectedPDUs declared by the initiator. The
initiator uses the value of MaxOutstandingUnexpectedPDUs that it
declared to determine the amount of buffer resources required for
control-type PDUs in this category that can be sent by a target. The
following is a list of the PDUs in this category and the conditions
for retiring the outstanding PDU:

a. For an Asynchronous Message PDU with StatSN = x, the PDU is
outstanding until the initiator sends a control-type PDU with
ExpStatSN set to at least x+1.

b. For a Reject PDU with StatSN = x which that is sent after a task is
active, the PDU is outstanding until the initiator sends a
control-type PDU with ExpStatSN set to at least x+1.

c. For a NOP-In PDU with ITT = 0xffffffff and StatSN = x, the
PDU is outstanding until the initiator responds with a
control-type PDU on the same connection where ExpStatSN is at
least x+1. But if the NOP-In PDU is sent as a ping request
with TTT != 0xffffffff, the PDU can also be retired when the
initiator sends a NOP-Out PDU with the same ITT and TTT as in
the ping request. Note that when a target sends a NOP-In PDU
as a ping request, it must provision a buffer for the NOP-Out
PDU sent as a ping response from the initiator.

When the number of outstanding unexpected control-type PDUs equals
MaxOutstandingUnexpectedPDUs, the iSCSI Layer layer at the target MUST NOT
generate any unexpected PDUs which that otherwise it would have generated,
even if its intent is to indicate an iSCSI error condition (e.g.,
Asynchronous Message, Reject). Task timeouts timeouts, as in the initiator
waiting for a command completion or other connection and session
level exceptions exceptions, will ensure that correct operational behavior will
result in these cases despite not generating the PDU. This rule
overrides any other requirements elsewhere which that require that a Reject
PDU MUST be sent.

Ko et al. Expires May 2007 61

(Implementation note: A SCSI task timeout and recovery can be a
lengthy process and hence SHOULD be avoided by proper provisioning of
resources.)

(Implementation note: To ensure that the initiator has a means to
inform the target that outstanding PDUs have been retired, the target
should reserve the last unexpected control-type PDU allowable by the
value of MaxOutstandingUnexpectedPDUs declared by the initiator for
sending a NOP-In ping request with TTT != 0xffffffff to allow the
initiator to return the NOP-Out ping response with the current
ExpStatSN.)

8.2

8.2. Flow Control for RDMA Read Resources

The total number of RDMA Read operations that can be active
simultaneously on an iSCSI/iSER connection depends on the amount of
resources allocated as declared in the iSER Hello exchange described
in section Section 5.1.3. Exceeding the number of RDMA Read operations
allowed on a connection will result in the connection being
terminated by the RCaP layer. The iSER Layer layer at the target maintains
the iSER-ORD to keep track of the maximum number of RDMA Read
Requests that can be issued by the iSER Layer layer on a particular RCaP
Stream.

During connection setup (see section Section 5.1), iSER-IRD is known at the
initiator and iSER-ORD is known at the target after the iSER Layers layers
at the initiator and the target have respectively allocated the
connection resources necessary to support RCaP, as directed by the
Allocate_Connection_Resources Operational Primitive from the iSCSI
Layer
layer before the end of the iSCSI Login Phase. In the full feature
phase, Full Feature
Phase, the first message sent by the initiator is the iSER Hello
Message (see section 9.3) Section 9.3), which contains the value of iSER-IRD. In
response to the iSER Hello Message, the target sends the iSER
HelloReply Message (see section 9.4) Section 9.4), which contains the value of
iSER-ORD. The iSER Layer layer at both the initiator and the target MAY
adjust (lower) the resources associated with iSER-IRD and iSER-ORD
respectively to match the iSER-ORD value declared in the HelloReply
Message. The iSER Layer layer at the target MUST flow control the RDMA
Read Request Messages to not exceed the iSER-ORD value at the target.

8.3

8.3. STag Management

An STag, as defined in [RDMAP], is an identifier of a Tagged Buffer
used in an RDMA operation. The allocation and the subsequent
invalidation of the STags are specified in this document if the

Ko et al. Expires May 2007 62 STags
are exposed on the wire by being Advertised in the iSER header or
declared in the header of an RCaP Message.

8.3.1

8.3.1. Allocation of STags

When the iSCSI Layer layer at the initiator invokes the Send_Control
Operational Primitive to request that the iSER Layer layer at the initiator to
process a SCSI Command, command, zero, one, or two STags may be allocated by
the iSER Layer. layer. See section Section 7.3.1 for details. The number of STags
allocated depends on whether the command is unidirectional or
bidirectional and whether or not solicited write data transfer is involved
or not.
involved.

When the iSCSI Layer layer at the initiator invokes the Send_Control
Operational Primitive to request that the iSER Layer layer at the initiator to
process a Task Management Function Request with the TASK REASSIGN
function, besides allocating zero, one, or two STags, the iSER Layer layer
MUST invalidate the existing STags, if any, associated with the ITT.
See section Section 7.3.3 for details.

The iSER Layer layer at the target allocates a local Data Sink STag when
the iSCSI Layer layer at the target invokes the Get_Data Operational
Primitive to request that the iSER Layer to layer process an R2T PDU. See
section
Section 7.3.6 for details.

8.3.2

8.3.2. Invalidation of STags

The invalidation of the STags at the initiator at the completion of a
unidirectional or bidirectional command when the associated SCSI
Response PDU is sent by the target is described in section Section 7.3.2.

When a unidirectional or bidirectional command concludes without the
associated SCSI Response PDU being sent by the target, the iSCSI
Layer
layer at the initiator MUST request that the iSER Layer layer at the
initiator
to invalidate the STags by invoking the
Deallocate_Task_Resources Operational Primitive qualified with ITT.
In response, the iSER
Layer layer at the initiator MUST locate the STag(s)
(if any) in the Local Mapping that associates the ITT to the local
STag(s). The iSER
Layer layer at the initiator MUST invalidate the STag(s)
(if any) and the Local Mapping.

For an RDMA Read operation used to realize a SCSI Write data
transfer, the iSER Layer layer at the target SHOULD invalidate the Data
Sink STag at the conclusion of the RDMA Read operation referencing
the Data Sink STag (to permit the immediate reuse of buffer
resources).

Ko et al. Expires May 2007 63

For an RDMA Write operation used to realize a SCSI Read data
transfer, the Data Source STag at the target is not declared to the
initiator and is not exposed on the wire. Invalidation of the STag
is thus not specified.

When a unidirectional or bidirectional command concludes without the
associated SCSI Response PDU being sent by the target, the iSCSI
Layer
layer at the target MUST request that the iSER Layer layer at the target to
invalidate the STags by invoking the Deallocate_Task_Resources
Operational Primitive qualified with ITT. In response, the iSER
Layer
layer at the target MUST locate the local STag(s) (if any) in the
Local Mapping that associates the ITT to the local STag(s). The iSER Layer
layer at the target MUST invalidate the local STag(s) (if any) and
the mapping.

Ko et al. Expires May 2007 64

9. iSER Control and Data Transfer

For iSCSI data-type PDUs (see section Section 7.1), the iSER Layer layer uses RDMA
Read and RDMA Write operations to transfer the solicited data. For
iSCSI control-type PDUs (see section Section 7.2), the iSER Layer layer uses Send
Message Types of RCaP.

9.1

9.1. iSER Header Format

An iSER header MUST be present in every Send Message Type of RCaP.
The iSER header is located in the first 12 bytes of the message
payload of the Send Message Type of RCaP, as shown in Figure 2.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Opcode Specific Fields |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 2 2. iSER Header Format

Opcode - Operation Code: 4 bits

The Opcode field identifies the type of iSER Messages:

0001b = iSCSI control-type PDU

0010b = iSER Hello Message

0011b = iSER HelloReply Message

All other opcodes are reserved.

9.2

9.2. iSER Header Format for the iSCSI Control-Type PDU

The iSER Layer layer uses Send Message Types of RCaP to transfer iSCSI
control-type PDUs (see section Section 7.2). The message payload of each of
the Send Message Types of RCaP used for transferring an iSER Message
contains an iSER Header followed by an iSCSI control-type PDU.

The iSER header in a Send Message Type of RCaP carrying an iSCSI
control-type PDU MUST have the format as described in Figure 3.

Ko et al. Expires May 2007 65

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |W|R| |
| 0001b |S|S| Reserved |
| |V|V| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Write STag (or N/A) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Read STag (or N/A) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 3 3. iSER Header Format for iSCSI Control-Type PDU

WSV - Write STag Valid flag: 1 bit

This flag indicates the validity of the Write STag field of the
iSER Header. If set to one, the Write STag field in this iSER
Header is valid. If set to zero, the Write STag field in this
iSER Header MUST be ignored at the receiver. The Write STag
Valid flag is set to one when there is solicited data to be
transferred for a SCSI Write write or bidirectional command, or when
there are non-immediate unsolicited and solicited data to be
transferred for the referenced task specified in a Task
Management Function Request with the TASK REASSIGN function.

RSV - Read STag Valid flag: 1 bit

This flag indicates the validity of the Read STag field of the
iSER Header. If set to one, the Read STag field in this iSER
Header is valid. If set to zero, the Read STag field in this
iSER Header MUST be ignored at the receiver. The Read STag Valid
flag is set to one for a SCSI Read read or bidirectional command, or
for a Task Management Function Request with the TASK REASSIGN
function.

Write STag - Write Steering Tag: 32 bits

This field contains the Write STag when the Write STag Valid flag
is set to one. For a SCSI Write write or bidirectional command, the
Write STag is used to Advertise the initiator's I/O Buffer
containing the solicited data. For a Task Management Function
Request with the TASK REASSIGN function, the Write STag is used
to Advertise the initiator's I/O Buffer containing the non-
immediate unsolicited data and solicited data. This Write STag
is used as the Data Source STag in the resultant RDMA Read
operation(s). When the Write STag Valid flag is set to zero,
this field MUST be set to zero.

Ko et al. Expires May 2007 66

Read STag - Read Steering Tag: 32 bits

This field contains the Read STag when the Read STag Valid flag
is set to one. The Read STag is used to Advertise the
initiator's Read I/O Buffer of a SCSI Read read or bidirectional
command, or of a Task Management Function Request with the TASK
REASSIGN function. This Read STag is used as the Data Sink STag
in the resultant RDMA Write operation(s). When the Read STag
Valid flag is zero, this field MUST be set to zero.

Reserved:

Reserved fields MUST be set to zero on transmit and MUST be
ignored on receive.

9.3 reception.

9.3. iSER Header Format for the iSER Hello Message

An iSER Hello Message MUST only contain the iSER header header, which MUST
have the format as described in Figure 4. The iSER Hello Message is
the first iSER Message sent on the RCaP Stream from the iSER Layer layer at
the initiator to the iSER Layer layer at the target.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | | | | |
| 0010b | Rsvd | MaxVer| MinVer| iSER-IRD |
| | | | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 4 4. iSER Header Format for iSER Hello Message
MaxVer - Maximum Version: 4 bits

This field specifies the maximum version of the iSER protocol
supported. It MUST be set to 1 one to indicate the version of the
specification described in this document.

MinVer - Minimum Version: 4 bits

This field specifies the minimum version of the iSER protocol
supported. It MUST be set to 1 one to indicate the version of the
specification described in this document.

Ko et al. Expires May 2007 67

iSER-IRD: 16 bits

This field contains the value of the iSER-IRD at the initiator.

Reserved (Rsvd):

Reserved fields MUST be set to zero on transmit, and MUST be
ignored on receive.

9.4 reception.

9.4. iSER Header Format for the iSER HelloReply Message

An iSER HelloReply Message MUST only contain the iSER header which
MUST have the format as described in Figure 5. The iSER HelloReply
Message is the first iSER Message sent on the RCaP Stream from the
iSER Layer layer at the target to the iSER Layer layer at the initiator.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |R| | | |
| 0011b |Rsvd |E| MaxVer| CurVer| iSER-ORD |
| | |J| | | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 5 5. iSER Header Format for iSER HelloReply Message

REJ - Reject flag: 1 bit

This flag indicates whether the target is rejecting this
connection. If set to one, the target is rejecting the
connection.

MaxVer - Maximum Version: 4 bits

This field specifies the maximum version of the iSER protocol
supported. It MUST be set to 1 one to indicate the version of the
specification described in this document.

CurVer - Current Version: 4 bits

This field specifies the current version of the iSER protocol
supported. It MUST be set to 1 one to indicate the version of the
specification described in this document.

Ko et al. Expires May 2007 68

iSER-ORD: 16 bits

This field contains the value of the iSER-ORD at the target.

Reserved (Rsvd):

Reserved fields MUST be set to zero on transmit, and MUST be
ignored on receive.

9.5 reception.

9.5. SCSI Data Transfer Operations

The iSER Layer layer at the initiator and the iSER Layer layer at the target
handle each SCSI Write, SCSI Read, and bidirectional operation as
described below.

9.5.1

9.5.1. SCSI Write Operation

The iSCSI Layer layer at the initiator MUST invoke the Send_Control
Operational Primitive to request that the iSER Layer layer at the initiator to
send the SCSI Write Command. write command. The iSER Layer layer at the initiator MUST
request that the RCaP layer to transmit a SendSE Message with the
message payload consisting of the iSER header followed by the SCSI
Command PDU and immediate data (if any). If there is solicited data,
the iSER Layer layer MUST Advertise the Write STag in the iSER header of
the SendSE Message, as described in section Section 9.2. Upon receiving the
SendSE Message, the iSER Layer layer at the target MUST notify the iSCSI
Layer
layer at the target by invoking the Control_Notify Operational
Primitive qualified with the SCSI Command PDU. See section Section 7.3.1 for
details on the handling of the SCSI Write Command. write command.

For the non-immediate unsolicited data, the iSCSI Layer layer at the
initiator MUST invoke a Send_Control Operational Primitive qualified
with the SCSI Data-out PDU. Upon receiving each Send or SendSE
Message containing the non-immediate unsolicited data, the iSER
Layer layer
at the target MUST notify the iSCSI Layer layer at the target by invoking
the Control_Notify Operational Primitive qualified with the SCSI
Data-out PDU. See section Section 7.3.4 for details on the handling of the
SCSI Data-out PDU.

For the solicited data, when the iSCSI Layer layer at the target has an I/O
Buffer available, it MUST invoke the Get_Data Operational Primitive
qualified with the R2T PDU. See section Section 7.3.6 for details on the
handling of the R2T PDU.

When the data transfer associated with this SCSI Write operation is
complete, the iSCSI Layer layer at the target MUST invoke the Send_Control
Operational Primitive when it is ready to send the SCSI Response

Ko et al. Expires May 2007 69 PDU.
Upon receiving a SendSE or SendInvSE Message containing the SCSI
Response PDU, the iSER Layer layer at the initiator MUST notify the iSCSI Layer
layer at the initiator by invoking the Control_Notify Operational
Primitive qualified with the SCSI Response PDU. See
section Section 7.3.2
for details on the handling of the SCSI Response PDU.

9.5.2

9.5.2. SCSI Read Operation

The iSCSI Layer layer at the initiator MUST invoke the Send_Control
Operational Primitive to request that the iSER Layer layer at the initiator
to send the SCSI Read Command. read command. The iSER Layer layer at the initiator MUST
request that the RCaP layer to transmit a SendSE Message with the
message payload consisting of the iSER header followed by the SCSI
Command PDU. The iSER Layer layer at the initiator MUST Advertise the Read
STag in the iSER header of the SendSE Message, as described in section
Section 9.2. Upon receiving the SendSE Message, the iSER Layer layer at
the target MUST notify the iSCSI Layer layer at the target by invoking the
Control_Notify Operational Primitive qualified with the SCSI Command
PDU. See section Section 7.3.1 for details on the handling of the SCSI Read
Command. read
command.

When the requested SCSI data is available in the I/O Buffer, the
iSCSI Layer layer at the target MUST invoke the Put_Data Operational
Primitive qualified with the SCSI Data-in PDU. See section Section 7.3.5 for
details on the handling of the SCSI Data-in PDU.

When the data transfer associated with this SCSI Read operation is
complete, the iSCSI Layer layer at the target MUST invoke the Send_Control
Operational Primitive when it is ready to send the SCSI Response PDU.
Upon receiving the SendInvSE Message containing the SCSI Response
PDU, the iSER Layer layer at the initiator MUST notify the iSCSI
Layer layer at
the initiator by invoking the Control_Notify Operational Primitive
qualified with the SCSI Response PDU. See section Section 7.3.2 for details
on the handling of the SCSI Response PDU.

9.5.3

9.5.3. Bidirectional Operation

The initiator and the target handle the SCSI Write and the SCSI Read
portions of this bidirectional operation the same as described in
Section
Sections 9.5.1 and Section 9.5.2 9.5.2, respectively.

Ko et al. Expires May 2007 70

10. iSER Error Handling and Recovery

RCaP provides the iSER Layer layer with reliable in-order delivery.
Therefore, the error management needs of an iSER-assisted connection
are somewhat different than those of a Traditional iSCSI connection.

10.1

10.1. Error Handling

iSER error handling is described in the following sections,
classified loosely based on the sources of errors:

1. Those originating at the transport layer (e.g., TCP).

2. Those originating at the RCaP layer.

3. Those originating at the iSER Layer. layer.

4. Those originating at the iSCSI Layer.

10.1.1 layer.

10.1.1. Errors in the Transport Layer

If the transport layer is TCP, then TCP packets with detected errors
are silently dropped by the TCP layer and result in retransmission at
the TCP layer. This has no impact on the iSER Layer. layer. However,
connection loss (e.g., link failure) and unexpected termination
(e.g., TCP graceful or abnormal close without the iSCSI Logout
exchanges) at the transport layer will cause the iSCSI/iSER
connection to be terminated as well.

10.1.1.1

10.1.1.1. Failure in the Transport Layer Before before RCaP Mode is Is Enabled

If the Connection connection is lost or terminated before the iSCSI Layer layer
invokes the Allocate_Connection_Resources Operational Primitive, the
login process is terminated and no further action is required.

If the Connection connection is lost or terminated after the iSCSI Layer layer has
invoked the Allocate_Connection_Resources Operational Primitive, then
the iSCSI Layer layer MUST request that the iSER Layer to layer deallocate all
connection resources by invoking the Deallocate_Connection_Resources
Operational Primitive.

10.1.1.2

10.1.1.2. Failure in the Transport Layer After after RCaP Mode is Is Enabled

If the Connection connection is lost or terminated after the iSCSI Layer layer has
invoked the Enable_Datamover Operational Primitive, the iSER Layer layer
MUST notify the iSCSI Layer layer of the connection loss by invoking the
Connection_Terminate_Notify Operational Primitive. Prior to

Ko et al. Expires May 2007 71 invoking
the Connection_Terminate_Notify Operational Primitive, the iSER layer
MUST perform the actions described in Section 5.2.3.2.

10.1.2

10.1.2. Errors in the RCaP Layer

The RCaP layer does not have error recovery operations built in. If
errors are detected at the RCaP layer, the RCaP layer will terminate
the RCaP Stream and the associated Connection.

10.1.2.1 connection.

10.1.2.1. Errors Detected in the Local RCaP Layer

If an error is encountered at the local RCaP layer, the RCaP layer
MAY send a Terminate Message to the Remote Peer to report the error
if possible. (For iWARP, see [RDMAP] for the list of errors where a
Terminate Message is sent.) The RCaP layer is responsible for
terminating the Connection. connection. After the RCaP layer notifies the iSER
Layer
layer that the Connection connection is terminated, the iSER Layer layer MUST notify
the iSCSI Layer layer by invoking the Connection_Terminate_Notify
Operational Primitive. Prior to invoking the
Connection_Terminate_Notify Operational Primitive, the iSER layer
MUST perform the actions described in Section 5.2.3.2.

10.1.2.2

10.1.2.2. Errors Detected in the RCaP Layer at the Remote Peer

If an error is encountered at the RCaP layer at the Remote Peer, the
RCaP layer at the Remote Peer may send a Terminate Message to report
the error if possible. If it is unable to send the Terminate
Message, the Connection connection is terminated. This is treated the same as a
failure in the transport layer after RDMA is enabled as described in
section 10.1.1.2.

If an error is encountered at the RCaP layer at the Remote Peer and
it is able to send a Terminate Message, the RCaP layer at the Remote
Peer is responsible for terminating the connection. After the local
RCaP layer notifies the iSER Layer layer that the Connection connection is terminated,
the iSER Layer layer MUST notify the iSCSI Layer layer by invoking the
Connection_Terminate_Notify Operational Primitive. Prior to invoking
the Connection_Terminate_Notify Operational Primitive, the iSER layer
MUST perform the actions described in Section 5.2.3.2.

10.1.3

10.1.3. Errors in the iSER Layer

The error handling due to errors at the iSER Layer layer is described in
the following sections.

Ko et al. Expires May 2007 72

10.1.3.1

10.1.3.1. Insufficient Connection Resources to Support RCaP at
Connection Setup

After the iSCSI Layer layer at the initiator invokes the
Allocate_Connection_Resources Operational Primitive during the iSCSI
login negotiation phase,
Login Negotiation Phase, if the iSER Layer layer at the initiator fails to
allocate the connection resources necessary to support RCaP, it MUST
return a status of failure to the iSCSI Layer layer at the initiator. The
iSCSI Layer layer at the initiator MUST terminate the Connection connection as
described in Section 5.2.3.1.

After the iSCSI Layer layer at the target invokes the
Allocate_Connection_Resources Operational Primitive during the iSCSI
login negotiation phase,
Login Negotiation Phase, if the iSER Layer layer at the target fails to
allocate the connection resources necessary to support RCaP, it MUST
return a status of failure to the iSCSI Layer layer at the target. The
iSCSI Layer layer at the target MUST send a Login Response with a status
class of 3 (Target Error), and a status code of "0302" (Out of
Resources). The iSCSI Layers layers at the initiator and the target MUST
terminate the Connection connection as described in Section 5.2.3.1.

10.1.3.2

10.1.3.2. iSER Negotiation Failures

If the RCaP or iSER related parameters declared by the initiator in
the iSER Hello Message is are unacceptable to the iSER Layer layer at the
target, the iSER Layer layer at the target MUST set the Reject (REJ) flag,
as described in section Section 9.4, in the iSER HelloReply Message. The
following are the cases when the iSER Layer layer MUST set the REJ flag to
1
one in the HelloReply Message:

* The initiator-declared iSER-IRD value is greater than 0 and the
target-declared iSER-ORD value is 0.

* The initiator-supported and the target-supported iSER protocol
versions do not overlap.

After requesting that the RCaP layer to send the iSER HelloReply
Message, the handling of the error situation is the same as that for
iSER format errors as described in section Section 10.1.3.3.

10.1.3.3.

10.1.3.3 iSER Format Errors

The following types of errors in an iSER header are considered format
errors:

* Illegal contents of any iSER header field

Ko et al. Expires May 2007 73

* Inconsistent field contents in an iSER header

* Length error for an iSER Hello or HelloReply Message (see section Section
9.3 and 9.4)

When a format error is detected, the following events MUST occur in
the specified sequence:

1. The iSER Layer layer MUST request that the RCaP layer to terminate the
RCaP Stream. The RCaP layer MUST terminate the associated
Connection.
connection.

2. The iSER Layer layer MUST notify the iSCSI Layer layer of the connection
termination by invoking the Connection_Terminate_Notify
Operational Primitive. Prior to invoking the
Connection_Terminate_Notify Operational Primitive, the iSER layer
MUST perform the actions described in Section 5.2.3.2.

10.1.3.4

10.1.3.4. iSER Protocol Errors

The first iSER Message sent by the iSER Layer layer at the initiator after
transitioning into iSER-assisted mode MUST be the iSER Hello Message
(see section Section 9.3). Likewise, the first iSER Message sent by the iSER Layer
layer at the target after transitioning into iSER-assisted mode MUST
be the iSER HelloReply Message (see section Section 9.4). Failure to send
the iSER Hello or HelloReply Message, as indicated by the wrong
Opcode in the iSER header, is a protocol error. The handling of this
error situation is the same as that for iSER format errors as
described in section Section 10.1.3.3.

If the sending side of an iSER-enabled connection acts in a manner
not permitted by the negotiated or declared login/text operational
key values as described in section Section 6, this is a protocol error error, and
the receiving side MAY handle this the same as for iSER format errors
as described in section Section 10.1.3.3.

10.1.4

10.1.4. Errors in the iSCSI Layer

The error handling due to errors at the iSCSI Layer layer is described in
the following sections. For error recovery, see section Section 10.2.

10.1.4.1

10.1.4.1. iSCSI Format Errors

When an iSCSI format error is detected, the iSCSI Layer layer MUST request
that the iSER Layer to layer terminate the RCaP Stream by invoking the
Connection_Terminate Operational Primitive. For more details on the
connection termination, see Section 5.2.3.1.

Ko et al. Expires May 2007 74

10.1.4.2

10.1.4.2. iSCSI Digest Errors

In the iSER-assisted mode, the iSCSI Layer layer will not see any digest
error because both the HeaderDigest and the DataDigest keys are
negotiated to "None".

10.1.4.3

10.1.4.3. iSCSI Sequence Errors

For Traditional iSCSI, sequence errors are caused by dropped PDUs due
to header or data digest errors. Since digests are not used in
iSER-assisted mode and the RCaP layer will deliver all messages in
the order they were sent, sequence errors will not occur in iSER-
assisted mode.

10.1.4.4

10.1.4.4. iSCSI Protocol Error

When the iSCSI Layer layer handles certain protocol errors by dropping the
connection, the error handling is the same as that for iSCSI format
errors as described in section Section 10.1.4.1.

When the iSCSI Layer layer uses the iSCSI Reject PDU and response codes to
handle certain other protocol errors, no special handling at the iSER Layer
layer is required.

10.1.4.5

10.1.4.5. SCSI Timeouts and Session Errors

This is

SCSI Timeouts and Session Errors are handled at the iSCSI Layer layer and
no special handling at the iSER Layer layer is required.

10.1.4.6

10.1.4.6. iSCSI Negotiation Failures

For negotiation failures that happen during the Login Phase at the
initiator after the iSCSI Layer layer has invoked the
Allocate_Connection_Resources Operational Primitive and before the
Enable_Datamover Operational Primitive has been invoked, the iSCSI
Layer
layer MUST request that the iSER Layer to layer deallocate all connection
resources by invoking the Deallocate_Connection_Resources Operational
Primitive. The iSCSI Layer layer at the initiator MUST terminate the Connection.
connection.

For negotiation failures during the Login Phase at the target, the
iSCSI Layer layer can use a Login Response with a status class other than 0
(success) to terminate the Login Phase. If the iSCSI Layer layer has
invoked the Allocate_Connection_Resources Operational Primitive and
before the Enable_Datamover Operational Primitive has been invoked,
the iSCSI Layer layer at the target MUST request that the iSER Layer layer at the
target to deallocate all connection resources by invoking the

Ko et al. Expires May 2007 75
Deallocate_Connection_Resources Operational Primitive. The iSCSI
Layer
layer at both the initiator and the target MUST terminate the
Connection.
connection.

During the iSCSI Login Phase, if the iSCSI Layer layer at the initiator
receives a Login Response from the target with a status class other
than 0 (Success) after the iSCSI Layer layer at the initiator has invoked
the Allocate_Connection_Resources Operational Primitive, the iSCSI
Layer
layer MUST request the iSER Layer layer to deallocate all connection
resources by invoking the Deallocate_Connection_Resources Operational
Primitive. The iSCSI Layer layer MUST terminate the
Connection connection in this
case.

For negotiation failures during the full feature phase, Full Feature Phase, the error
handling is left to the iSCSI Layer layer and no special handling at the
iSER Layer layer is required.

10.2

10.2. Error Recovery

Error recovery requirements of iSCSI/iSER are the same as that of
Traditional iSCSI. All three ErrorRecoveryLevels as defined in
[RFC3720] are supported in iSCSI/iSER.

* For ErrorRecoveryLevel 0, session recovery is handled by iSCSI and
no special handling by the iSER Layer layer is required.

* For ErrorRecoveryLevel 1, see section Section 10.2.1 on PDU Recovery.

* For ErrorRecoveryLevel 2, see section Section 10.2.2 on Connection
Recovery.

The iSCSI Layer layer may invoke the Notice_Key_Values Operational
Primitive during connection setup to request that the iSER Layer to layer take
note of the value of the operational ErrorRecoveryLevel, as described
in sections Sections 5.1.1 and 5.1.2.

10.2.1

10.2.1. PDU Recovery

As described in sections Sections 10.1.4.2 and 10.1.4.3, digest and sequence
errors will not occur in the iSER-assisted mode. If the RCaP layer
detects an error, it will close the iSCSI/iSER connection, as
described in section Section 10.1.2. Therefore, PDU recovery is not useful
in the iSER-assisted mode.

The iSCSI Layer layer at the initiator SHOULD disable iSCSI timeout-driven
PDU retransmissions.

Ko et al. Expires May 2007 76

10.2.2

10.2.2. Connection Recovery

The iSCSI Layer layer at the initiator MAY reassign connection allegiance
for non-immediate commands which that are still in progress and are
associated with the failed connection by using a Task Management
Function Request with the TASK REASSIGN function. See section Section 7.3.3
for more details.

When the iSCSI Layer layer at the initiator does a task reassignment for a
SCSI Write write command, it MUST qualify the Send_Control Operational
Primitive invocation with DataDescriptorOut DataDescriptorOut, which defines the I/O
Buffer for both the non-immediate unsolicited data and the solicited
data. This allows the iSCSI Layer layer at the target to use recovery R2Ts
to request for data originally sent as unsolicited and solicited from the
initiator.

When the iSCSI Layer layer at the target accepts a reassignment request for
a SCSI Read read command, it MUST request that the iSER Layer to layer process SCSI
Data-in for all unacknowledged data by invoking the Put_Data
Operational Primitive. See section Section 7.3.5 on the handling of SCSI
Data-in.

When the iSCSI Layer layer at the target accepts a reassignment request for
a SCSI Write write command, it MUST request that the iSER Layer to layer process a
recovery R2T for any non-immediate unsolicited data and any solicited
data sequences that have not been received by invoking the Get_Data
Operational Primitive. See section Section 7.3.6 on the handling of Ready To
Transfer (R2T).

The iSCSI Layer layer at the target MUST NOT issue recovery R2Ts on an
iSCSI/iSER connection for a task for which the connection allegiance
was never reassigned. The iSER Layer layer at the target MAY reject such a
recovery R2T received via the Get_Data Operational Primitive
invocation from the iSCSI Layer layer at the target, with an appropriate
error code.

The iSER Layer layer at the target will process the requests invoked by the
Put_Data and Get_Data Operational Primitives for a reassigned task in
the same way as for the original commands.

Ko et al. Expires May 2007 77

11. Security Considerations

When iSER is layered on top of an RCaP layer and provides the RDMA
extensions to the iSCSI protocol, the security considerations of iSER
are the same as that of the underlying RCaP layer. For iWARP, this
is described in [RDMAP] and [RDDPSEC].

Since the iSER-assisted iSCSI protocol is still functionally iSCSI
from a security considerations perspective, all of the iSCSI security
requirements as described in [RFC3720] and [RFC3723] apply. If the
IPsec [IPSEC] mechanism is used, then it MUST be established before
the connection transitions to the iSER-assisted mode. If iSER is
layered on top of a non-IP based RCaP layer, all the security
protocol mechanisms applicable to that RCaP layer is are also applicable
to an iSCSI/iSER connection. If iSER is layered on top of a non-IP
protocol, the IPsec mechanism as specified in [RFC3720] MUST be
implemented at any point where the iSER protocol enters the IP
network (e.g., via gateways), and the non-IP protocol SHOULD
implement (optional to use) a packet by packet-by packet security protocol
equal in strength to the IPsec mechanism specified by [RFC3720].

Ko et al. Expires May 2007 78

12 IANA Considerations

This document has no actions

Note that the IPsec requirements for IANA.

Ko et al. Expires May 2007 79

13 this document are based on the
version of IPsec specified in RFC 2401 [IPSEC] and related RFCs, as
profiled by RFC 3723 [RFC3723], despite the existence of a newer
version of IPsec specified in RFC 4301 [RFC4301] and related RFCs.

12. References

13.1

12.1. Normative References

[RFC3720] J. Satran et al., "iSCSI", Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and
E. Zeidner, "Internet Small Computer Systems Interface
(iSCSI)", RFC 3720, April 2004 2004.

[RFC3723] B. Aboba et al., Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
Travostino, "Securing Block Storage Protocols over IP", RFC
3723, April 2004.

[RDMAP] R. Recio et al., "An RDMA Recio, R., Culley, P., Garcia, D., Hilland, J., and B.
Metzler, "A Remote Direct Memory Access Protocol
Specification", IETF
Internet-draft draft-ietf-rddp-rdmap-07.txt (work in progress),
September 2006 RFC 5040, October 2007.

[DDP] H. Shah et al., Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
Data Placement over Reliable Transports", IETF Internet-draft draft-ietf-rddp-ddp-07.txt
(work in progress), September 2006 RFC 5041, October
2007.

[IPSEC] Kent, S. and R. Atkinson, "Security Architecture for the
Internet Protocol", RFC 2401, November 1998.

[MPA] P. Culley et al., Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP
Specification", IETF Internet-draft draft-ietf-rddp-mpa-08.txt
(work in progress), RFC 5044, October 2006 2007.

[RDDPSEC] Pinkerton, J. Pinkerton et al., "DDP/RDMAP and E. Deleganes, "Direct Data Placement
Protocol (DDP) / Remote Direct Memory Access Protocol
(RDMAP) Security", IETF Internet
Draft draft-ietf-rddp-security-10.txt (work in progress), June
2006 RFC 5042, October 2007.

[TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981 1981.

[RFC2119] Bradner, S., "Key Words words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997

13.2 1997.

12.2. Informative References

[SAM2] T10/1157D, SCSI Architecture Model - 2 (SAM-2)

[DA] M. Chadalapaka et al., Chadalapaka, M., Hufferd, J., Satran, J., and H. Shah,
"Datamover Architecture for iSCSI", IETF
Internet-draft, draft-ietf-ips-da-04.txt (work in progress),
October 2006

[VERBS] J. Hilland et al., "RDMA Protocol Verbs Specification",
RDMAC Consortium Draft Specification draft-hilland-iwarp-verbs-
v1.0-RDMAC, April 2003

[IPSEC] S. Kent et al., "Security Architecture for the Internet
Protocol", iSCSI (DA)", RFC 2401, November 1998

Ko et al. Expires May 2007 80 5047, October
2007.

[IB] InfiniBand Architecture Specification Volume 1 Release 1.2,
October 2004

[IPoIB] H.K. Chu et al, Chu, J. and V. Kashyap, "Transmission of IP over InfiniBand", InfiniBand
(IPoIB)", RFC 4391, March 2006

Ko et al. Expires May 2007 81

14 April 2006.

[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, December 2005.

Appendix A

14.1 A. iWARP Message Format for iSER

This section is for information only and is NOT part of the standard.
It simply depicts the iWARP Message format for the various iSER
Messages when the transport layer is TCP.

14.1.1

A.1. iWARP Message Format for iSER Hello Message

The following figure depicts an iSER Hello Message encapsulated in an
iWARP SendSE Message.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0010b | Zeros | 0001b | 0001b | iSER-IRD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 6 6. SendSE Message containing Containing an iSER Hello Message

Ko et al. Expires May 2007 82

14.1.2

A.2. iWARP Message Format for iSER HelloReply Message

The following figure depicts an iSER HelloReply Message encapsulated
in an iWARP SendSE Message. The Reject (REJ) flag is set to 0.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0011b |Zeros|0| 0001b | 0001b | iSER-ORD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 7 7. SendSE Message containing Containing an iSER HelloReply Message

Ko et al. Expires May 2007 83

14.1.3 Message

A.3. iWARP Message Format for SCSI Read Command PDU

The following figure depicts a SCSI Read Command PDU embedded in an
iSER Message encapsulated in an iWARP SendSE Message. For this
particular example, in the iSER header, the Write STag Valid flag is
set to zero, the Read STag Valid flag is set to one, the Write STag
field is set to all zeros, and the Read STag field contains a valid
Read STag.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |0|1| All zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Read STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Read Command PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 8 8. SendSE Message containing Containing a SCSI Read Command PDU

Ko et al. Expires May 2007 84

14.1.4

A.4. iWARP Message Format for SCSI Read Data

The following figure depicts an iWARP RDMA Write Message carrying
SCSI Read data in the payload:

Figure 9 9. RDMA Write Message containing Containing SCSI Read Data

Ko et al. Expires May 2007 85

14.1.5

A.5. iWARP Message Format for SCSI Write Command PDU

The following figure depicts a SCSI Write Command PDU embedded in an
iSER Message encapsulated in an iWARP SendSE Message. For this
particular example, in the iSER header, the Write STag Valid flag is
set to one, the Read STag Valid flag is set to zero, the Write STag
field contains a valid Write STag, and the Read STag field is set to
all zeros since it is not used.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |1|0| All zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Write STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Write Command PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 10 10. SendSE Message containing Containing a SCSI Write Command PDU

Ko et al. Expires May 2007 86

14.1.6

A.6. iWARP Message Format for RDMA Read Request

An iSCSI R2T is transformed into an iWARP RDMA Read Request Message.
The following figure depicts an iWARP RDMA Read Request Message:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Sink STag (SinkSTag) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Data Sink Tagged Offset (SinkTO) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RDMA Read Message Size (RDMARDSZ) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Source STag (SrcSTag) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Data Source Tagged Offset (SrcTO) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 11 11. RDMA Read Request Message

Ko et al. Expires May 2007 87

14.1.7

A.7. iWARP Message Format for Solicited SCSI Write Data

The following figure depicts an iWARP RDMA Read Response Message
carrying the solicited SCSI Write data in the payload:

Figure 12 12. RDMA Read Response Message containing Containing SCSI Write Data

Ko et al. Expires May 2007 88

14.1.8

A.8. iWARP Message Format for SCSI Response PDU

The following figure depicts a SCSI Response PDU embedded in an iSER
Message encapsulated in an iWARP SendInvSE Message:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA Header | DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Invalidate STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0001b |0|0| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| All Zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCSI Response PDU |
// //
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MPA CRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 13 13. SendInvSE Message containing Containing SCSI Response PDU

Ko et al. Expires May 2007 89

Appendix B

15.1 B. Architectural discussion Discussion of iSER over InfiniBand

This section explains how an InfiniBand network (with Gateways) would
be structured. It is informational only and is intended to provide
insight on how iSER is used in an InfiniBand environment.

15.2

B.1. The Host side Side of the iSCSI & and iSER connections Connections in Infiniband InfiniBand

Figure 14 defines the topologies in which iSCSI and iSER will be able
to operate on an InfiniBand Network.

+---------+ +---------+ +---------+ +---------+ +--- -----+
| Host | | Host | | Host | | Host | | Host |
| | | | | | | | | |
+---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+
|HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA|
+-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+
|----+------|-----+-----|-----+-----|-----+-----|-----+---> To IB
IB| IB | IB | IB | IB | SubNet2 SWTCH
+-v-----------v-----------v-----------v-----------v---------+
| InfiniBand Switch for Subnet1 |
+---+-----+--------+-----+--------+-----+------------v------+
| TCA | | TCA | | TCA | |
+-----+ +-----+ +-----+ | IB
/ IB \ / IB \ / \ +--+--v--+--+
| iSER | | iSER | | IPoIB | | | TCA | |
| Gateway | | Gateway | | Gateway | | +-----+ |
| to | | to | | to | | Storage |
| iSCSI | | iSER | | IP | | Controller|
| TCP | | iWARP | |Ethernet | +-----+-----+
+---v-----| +---v-----| +----v----+
| EN | EN | EN
+--------------+---------------+----> to IP based storage
Ethernet links that carry iSCSI or iWARP

Figure 14 14. iSCSI and iSER on IB

In Figure 14, the Host systems are connected via the InfiniBand Host
Channel Adapters (HCAs) to the InfiniBand links. With the use of IB
switch(es), the InfiniBand links connect the HCA to InfiniBand Target
Channel Adapters (TCAs) located in gateways or Storage Controllers.
An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated
in IB protocols to either standard iSCSI, or iSER Messages for iWARP.
An [IPoIB] Gateway converts the InfiniBand [IPoIB] protocol to IP
protocol, and in the iSCSI case, permits

Ko et al. Expires May 2007 90 iSCSI to be operated on an
IB Network between the Hosts and the [IPoIB] Gateway.

15.3

B.2. The Storage side Side of the iSCSI & and iSER mixed network environment Mixed Network Environment

Figure 15 shows a storage controller that has three different portal
groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP
or iSCSI (TPG-2), and one supporting iSER/IB (TPG-1).

| | |
| | |
+--+--v--+----------+--v--+----------+--v--+--+
| | IB | |iWARP| | EN | |
| | | | TCP | | NIC | |
| |(TCA)| | RNIC| | | |
| +-----| +-----+ +-----+ |
| TPG-1 TPG-2 TPG-4 |
| 9.1.3.3 9.1.2.4 9.1.2.6 |
| |
| Storage Controller |
| |
+---------------------------------------------+

Figure 15 15. Storage Controller with TCP, iWARP, and IB Connections

The normal iSCSI portal group advertising processes (via SLP, iSNS, the Service
Location Protocol (SLP), the Internet Storage Name Service (iSNS), or
SendTargets) are available to a Storage Controller.

15.4

B.3. Discovery processes Processes for an InfiniBand Host

An InfiniBand Host system can gather portal group IP address addresses from
SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via
[IPoIB]. After obtaining one or more remote portal IP addresses, the
Initiator uses the standard IP mechanisms to resolve the IP address
to a local outgoing interface and the destination hardware address
(Ethernet MAC or IB GID of the target or a gateway leading to the
target). If the resolved interface is an [IPoIB] network interface,
then the target portal can be reached through an InfiniBand fabric.
In this case case, the Initiator can establish an iSCSI/TCP or iSCSI/iSER
session with the Target over that InfiniBand interface, using the
Hardware Address (InfiniBand GID) obtained through the standard
Address Resolution (ARP) processes.

If more than one IP address are is obtained through the discovery
process, the Initiator should select a Target IP address that is on
the same IP subnet as the Initiator Initiator, if one exists. This will avoid

Ko et al. Expires May 2007 91
a potential overhead of going through a gateway when a direct path
exists.

In addition addition, a user can configure manual static IP route entries if a
particular path to the target is preferred.

15.5

B.4. IBTA Connection Specifications

The InfiniBand Trade Association (IBTA) connection specifications

It is are
outside the scope of this document, but it is expected that the InfiniBand Trade Association (IBTA) IBTA
has or will define:

* The iSER ServiceID ServiceID.

* A Means for permitting a Host to establish a connection with a
peer InfiniBand end-node, and that peer indicating when that
end-node supports iSER, so the Host would be able to fall back to iSCSI/TCP over [IPoIB].
[IPoIB] if that peer indicates iSER is not supported.

* A Means for permitting the Host to establish connections with IB
iSER connections on storage controllers or IB iSER connected
Gateways in preference to [IPoIB] connected Gateways/Bridges or
connections to Target Storage Controllers that also accept iSCSI
via [IPoIB].

* A Means for combining the IB ServiceID for iSER and the IP port
number such that the IB Host can use normal IB connection
processes, yet ensure that the iSER target peer can actually
connect to the required IP port number.

Ko et al. Expires May 2007 92

Acknowledgments

This protocol was developed by a design team that, in addition to the
authors, included Dwight Barron (HP), John Carrier (formerly from
Adaptec), Ted Compton (EMC), Paul R. Culley (HP), Yaron Haviv
(Voltaire), Jeff Hilland (HP), Mike Krause (HP), Alex Nezhinsky
(Voltaire), Jim Pinkerton (Microsoft), Renato J. Recio (IBM), Julian
Satran (IBM), Tom Talpey (Network Appliance), and Jim Wendt (HP).
Special thanks to David Black (EMC) for his extensive review
comments.

Author's Address

Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA
Phone: +1-916-785-5621
Email:
EMail: cbm@rose.hp.com

Uri Elzur
Broadcom Corporation
16215 Alton Parkway
Irvine, CA 92619-7013, USA
Phone: +1-949-926-6432
Email:
EMail: Uri@Broadcom.com

John Hufferd
Brocade Communications Systems, Inc.
1745 Technology Drive
San Jose, CA 95110, USA
Phone: +1-408-333-5244
Email:
EMail: jhufferd@brocade.com

Mike Ko
IBM Corp.
650 Harry Rd.
San Jose, CA 95120, USA
Phone: +1-408-927-2085
Email:
EMail: mako@us.ibm.com

Hemal Shah
Broadcom Corporation
16215 Alton Parkway
P.O. Box 57013
Irvine, CA 92619, USA
Phone: +1-949-926-6941
Email:
EMail: hemal@broadcom.com

Patricia Thaler
Broadcom Corporation
5025 Keane Dr.
Carmichael, CA 95608, USA
Phone: +1-916-570-2707
email:
EMail: pthaler@broadcom.com

Ko et al. Expires May 2007 93

17 Acknowledgments

This protocol was developed by a design team that, in addition to
the authors, included Dwight Barron (HP), John Carrier (formerly
from Adaptec), Ted Compton (EMC), Paul R. Culley (HP), Yaron Haviv
(Voltaire), Jeff Hilland (HP), Mike Krause (HP), Alex Nezhinsky
(Voltaire), Jim Pinkerton (Microsoft), Renato J. Recio (IBM), Julian
Satran (IBM), Tom Talpey (Network Appliance), and Jim Wendt (HP).
Special thanks to David Black (EMC) for his extensive review
comments.

Ko et al. Expires May 2007 94

Full Copyright Statement

This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.

This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.

Ko et al. Expires May 2007 95