Internet Draft JamesNetwork Working Group J. Pinkertondraft-ietf-rddp-security-10.txtRequest for Comments: 5042 Microsoft Corporation Category: Standards TrackEllenE. DeleganesExpires: December, 2006 Intel Corporation June 2006 DDP/RDMAPSelf October 2007 Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security Status ofthisThis MemoBy submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents ofThis document specifies an Internet standards track protocol for the InternetEngineering Task Force (IETF), its areas,community, andits working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents validrequests discussion and suggestions fora maximumimprovements. Please refer to the current edition ofsix monthsthe "Internet Official Protocol Standards" (STD 1) for the standardization state andmay be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The liststatus ofcurrent Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The listthis protocol. Distribution ofInternet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.this memo is unlimited. Abstract This document analyzes security issues around implementation and use of the Direct Data PlacementProtocol(DDP)Protocol (DDP) and Remote Direct Memory Access Protocol (RDMAP). It first defines an architectural model for an RDMA Network Interface Card (RNIC), which can implement DDP or RDMAP and DDP. The document reviews various attacks against the resources defined in the architectural model and the countermeasures that can be used to protect the system. Attacks are grouped into those that can be mitigated by using secure communication channels across the network, attacks from Remote Peers, and attacks from Local Peers. Attack categories include spoofing, tampering, information disclosure, denial of service, and elevation of privilege.J. Pinkerton, et al. Expires December, 2006 1Table of Contents1 Introduction.................................................4 21. Introduction ....................................................4 2. ArchitecturalModel..........................................7 2.1 Components...................................................8 2.2 Resources...................................................10 2.2.1Model .............................................6 2.1. Components .................................................7 2.2. Resources ..................................................9 2.2.1. Stream ContextMemory.....................................10 2.2.2Memory ...............................9 2.2.2. DataBuffers..............................................10 2.2.3Buffers .......................................10 2.2.3. Page TranslationTables...................................11 2.2.4Tables ............................10 2.2.4. Protection Domain(PD)....................................11 2.2.5(PD) .............................11 2.2.5. STag Namespace andScope..................................12 2.2.6Scope ...........................11 2.2.6. CompletionQueues.........................................13 2.2.7Queues ..................................12 2.2.7. Asynchronous EventQueue..................................13 2.2.8Queue ...........................12 2.2.8. RDMA Read RequestQueue...................................13 2.3Queue ............................13 2.3. RNICInteractions...........................................14 2.3.1Interactions .........................................13 2.3.1. Privileged Control InterfaceSemantics....................14 2.3.2Semantics .............13 2.3.2. Non-Privileged Data InterfaceSemantics...................14 2.3.3Semantics ............13 2.3.3. Privileged Data InterfaceSemantics.......................15 2.3.4Semantics ................14 2.3.4. Initialization of RNIC Data Structures for DataTransfer..15 2.3.5Transfer ......................................14 2.3.5. RNIC Data TransferInteractions...........................16 3Interactions ....................16 3. Trust and ResourceSharing..................................18 4Sharing .....................................17 4. AttackerCapabilities.......................................19 5Capabilities ..........................................18 5. Attacks That CanbeBe MitigatedWithwith End-to-EndSecurity......20 5.1 Spoofing....................................................20 5.1.1 Impersonation.............................................20 5.1.2Security .........18 5.1. Spoofing ..................................................19 5.1.1. Impersonation ......................................19 5.1.2. StreamHijacking..........................................21 5.1.3Hijacking ...................................20 5.1.3. Man-in-the-MiddleAttack..................................21 5.2Attack ...........................20 5.2. Tampering -Network based modificationNetwork-Based Modification ofbuffer content....22 5.3Buffer Content ..21 5.3. Information Disclosure -Network Based Eavesdropping........22 5.4Network-Based Eavesdropping ......21 5.4. Specific Requirements for SecurityServices.................22 5.4.1Services ...............21 5.4.1. Introduction to SecurityOptions..........................23 5.4.2Options ...................21 5.4.2. TLSisIs Inappropriate for DDP/RDMAPSecurity...............23 5.4.3Security ........22 5.4.3. DTLS andRDDP.............................................24 5.4.4RDDP ......................................23 5.4.4. ULPsWhichThat ProvideSecurity...............................24 5.4.5Security .........................23 5.4.5. Requirements for IPsec Encapsulation ofDDP...............25 6DDP ........24 6. Attacks from RemotePeers...................................26 6.1 Spoofing....................................................26 6.1.1Peers ......................................24 6.1. Spoofing ..................................................25 6.1.1. Using an STag on a DifferentStream.......................26 6.2 Tampering...................................................27 6.2.1Stream ................25 6.2. Tampering .................................................26 6.2.1. Buffer Overrun - RDMA Write or ReadResponse..............28 6.2.2Response .......26 6.2.2. Modifying a BufferAfter Indication.......................28 6.2.3after Indication ................27 6.2.3. Multiple STags toaccessAccess thesame buffer..................29 6.3Same Buffer ...........28 6.3. InformationDisclosure......................................29 6.3.1Disclosure ....................................28 6.3.1. Probingmemory outsideMemory Outside of thebuffer bounds...............29 6.3.2Buffer Bounds ........28 6.3.2. Using RDMA Read to Access StaleData......................29 6.3.3Data ...............28 6.3.3. Accessing a BufferAfterafter theTransfer.....................30 6.3.4Transfer ..............29 6.3.4. Accessing Unintended DataWithwith a ValidSTag...............30 6.3.5STag ........29 6.3.5. RDMA Read into an RDMA WriteBuffer.......................30 6.3.6Buffer ................29 6.3.6. Using Multiple STagsWhichThat Alias to the SameBuffer.......31 6.4Buffer .............................................30 6.4. Denial of Service(DOS).....................................31 6.4.1(DOS) ...................................30 6.4.1. RNIC ResourceConsumption.................................32 6.4.2Consumption ..........................31 6.4.2. Resource Consumption by IdleULPs.........................32 6.4.3ULPs ..................32 6.4.3. Resource ConsumptionByby ActiveULPs.......................33 6.4.3.1ULPs ................32 6.4.3.1. Multiple Streams Sharing ReceiveBuffers...............33 6.4.3.2Buffers ..32 6.4.3.2. Remote or Local Peer Attacking a SharedCQ.............35 6.4.3.3CQ .................................34 6.4.3.3. Attacking the RDMA Read RequestQueue..................37 6.4.4Queue .....37 6.4.4. Exercise ofnon-optimal code paths........................38 6.4.5Non-Optimal Code Paths .................37 6.4.5. Remote Invalidate an STag Shared on MultipleStreams......38 6.4.6Streams ...................................38 6.4.6. Remote PeerattackingAttacking an UnsharedCQ......................39 6.5CQ ...............38 6.5. Elevation ofPrivilege......................................39 7Privilege ....................................39 7. Attacks from LocalPeers....................................40 7.1Peers .......................................39 7.1. Local ULP Attacking a SharedCQ.............................40 7.2CQ ...........................39 7.2. Local Peer Attacking the RDMA Read RequestQueue............40 7.3Queue ..........40 7.3. Local ULP Attacking the PTT&and STagMapping..................40 8Mapping ..............40 8. Securityconsiderations.....................................42 9considerations ........................................40 9. IANAConsiderations.........................................43 10 References..................................................44 10.1Considerations ............................................41 10. References ....................................................41 10.1. NormativeReferences......................................44 10.2References .....................................41 10.2. InformativeReferences....................................44 11References ...................................42 AppendixA:A. ULP Issues for RDDP Client/ServerProtocols.....46 12Protocols ...........44 AppendixB:B. Summary of RNIC and ULP ImplementationRequirements.....................................................50 13Requirements ...47 AppendixC:C. Partial TrustTaxonomy..........................52 14 Author's Addresses..........................................54 15 Acknowledgments.............................................55 16 Full Copyright Statement....................................57 Table of Figures Figure 1 - RDMA Security Model....................................8 1Taxonomy ................................48 Acknowledgments ...................................................50 1. Introduction RDMA enables new levels of flexibility when communicating between two parties compared to current conventional networking practice(e.g.(e.g., a stream-based model or datagram model). This flexibility brings new security issues that must be carefully understood when designing Upper Layer Protocols (ULPs) utilizing RDMA and when implementing RDMA-aware NICs (RNICs). Note that for the purposes of this security analysis, an RNIC may implement RDMAP [RDMAP] and DDP [DDP], or just DDP. Also, a ULP may be an application or it may be a middleware library. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.AdditionallyAdditionally, the security terminology defined in[RFC2828][RFC4949] is used in this specification. The document first develops an architectural model that is relevant for the securityanalysis - itanalysis. Section 2 details components, resources, and system properties that may beattacked in Section 2.attacked. The document uses Local Peer to represent the RDMA/DDP protocol implementation on the local end of a Stream (implemented with a transportprotocolprotocol, such as [RFC793] or [RFC2960]). The local Upper-Layer-Protocol (ULP) is used to represent the application or middle-ware layer above the Local Peer. The document does not attempt to differentiate between a Remote Peer and a Remote ULP (an RDMA/DDP protocol implementation on the remote end of a Stream versus the application on the remote end) for several reasons:oftenoften, the source of the attack is difficult to know forsure; andsure and, regardless of the source, the mitigations required of the Local Peer or local ULP are the same.ThusThus, the document generically refers to a Remote Peer rather than trying to further delineate the attacker. The document then defines what resources a local ULP may share across Streams and what resources the local ULP may share with the Remote Peer across Streams in Section 3. Intentional sharing of resources between multiple Streams may imply some level of trust between the Streams. However, some types of resource sharing have unmitigated securityattacksattacks, which would mandate not sharing a specific type of resource unless there is some level of trust between the Streams sharing resources. This document defines a new term, "Partial MutualTrust"Trust", to address this concept: Partial Mutual Trust - a collection of RDMAP/DDP Streams, which represent the local and remote end points of theStream, whichStream that are willing to assume that the Streams from the collection will not perform malicious attacks against any of the other Streams in the collection. ULPs have explicit control of which collection of endpoints is in a Partial Mutual Trust collection through tools discussed inSection 13AppendixC:C, Partial Trust Taxonomy. An untrusted peer relationship is appropriate when a ULP wishes to ensure that it will be robust and uncompromised even in the face of a deliberate attack by its peer. For example, a single ULP that concurrently supports multiple unrelated Streams(e.g.(e.g., a server) would presumably treat each of its peers as an untrusted peer. For a collection of Streamswhichthat share Partial Mutual Trust, the assumption is that any Stream not in the collection is untrusted. For the untrusted peer, a brief list of capabilities is enumerated in Section 4. The rest of the document is focused on analyzing attacks and recommending specific mitigations to the attacks. Attacks are categorized into attacks mitigated by end-to-end security, attacks initiated by Remote Peers, and attacks initiated by Local Peers. For each attack, possible countermeasures are reviewed. ULPs within a host are divided into two categories - Privileged and Non-Privileged. Both ULP types can send and receive data and request resources. The key differences between the two are: The Privileged ULP is trusted by the local systemtonot to maliciously attack the operating environment, but it is not trusted to optimize resource allocation globally. For example, the Privileged ULP could be a kernelULP, thusULP; thus, the kernel presumably has in some way vetted the ULP before allowing it to execute. A Non-Privileged ULP's capabilities are a logical sub-set of the Privileged ULP's. It is assumed by the local system that aNon-PrivilegedNon- Privileged ULP is untrusted. All Non-Privileged ULP interactions with the RNIC Engine that could affect other ULPs need to be done through a trusted intermediary that can verify the Non-Privileged ULP requests. The appendices provide focused summaries of this specification.Section 11AppendixA:A, ULP Issues for RDDP Client/ServerProtocolsProtocols, focuses on implementers of traditional client/server protocols.Section 12AppendixB:B, Summary of RNIC and ULP ImplementationRequirementsRequirements, summarizes all normative requirements in this specification.Section 13AppendixC:C, Partial TrustTaxonomyTaxonomy, provides an abstract model for categorizing trust boundaries. If an RDMAP/DDP protocol implementation uses the mitigations recommended in this document, that implementation should not exhibit additional security vulnerabilities above and beyond those of an implementation of the transport protocol (i.e., TCP or SCTP) and protocols beneath it (e.g., IP) without RDMAP/DDP.22. Architectural Model This section describes an RDMA architectural reference model that is used as security issues are examined. It introduces the components of the model, the resources that can be attacked, the types of interactions possible between components and resources, and the system propertieswhichthat must be preserved. Figure 1 shows the components comprising the architecture and the interfaces where potential security attacks could be launched. External attacks can be injected into the system from a ULP that sits above the RNIC Interface or from the network. The intent here is to describe high level components and capabilitieswhichthat affect threat analysis, and not focus on specific implementation options. Also note that the architectural model is an abstraction, and an actual implementation may choose to subdivide its components along different boundary linesthanfrom those defined here. For example, the Privileged Resource Manager may be partially or completely encapsulated in the Privileged ULP. Regardless, it is expected that the security analysis of the potential threats and countermeasures still apply. Note that the model below is derived from several specific RDMA implementations. A few of note are [VERBS-RDMAC], [VERBS-RDMAC- Overview], and [INFINIBAND]. +-------------+ | Privileged | | Resource | Admin<-+>| Manager | ULP Control Interface | | |<------+-------------------+ | +-------------+ | | | ^ v v | | +-------------+ +-----------------+ +---------------->| Privileged | | Non-Privileged | | | ULP | | ULP | | +-------------+ +-----------------+ | ^ ^ |Privileged |Privileged |Non-Privileged |Control |Data |Data |Interface |Interface |Interface RNIC | | | Interface v v v ================================================================= +--------------------------------------+ | | | RNIC Engine | | | +--------------------------------------+ ^ | v Internet Figure 1 - RDMA Security Model2.12.1. Components The components shown in Figure 1 - RDMA Security Model are: * RDMA Network Interface Controller Engine (RNIC) -theThe component that implements the RDMA protocol and/or DDP protocol. * Privileged Resource Manager -theThe component responsible for managing and allocating resources associated with the RNIC Engine. The Resource Manager does not send or receive data. Note that whether the Resource Manager is an independent component, part of the RNIC, or part of the ULP is implementation dependent. * Privileged ULP - See Section1 Introduction1, Introduction, for a definition of Privileged ULP. The local host infrastructure can enable the Privileged ULP to map adata bufferData Buffer directly from the RNIC Engine to the host through the RNIC Interface, but it does not allow the Privileged ULP to directly consume RNIC Engine resources. * Non-Privileged ULP - See Section1 Introduction1, Introduction, for a definition of Non-Privileged ULP. A design goal of the DDP and RDMAP protocols is to allow, under constrained conditions, Non-Privileged ULP to send and receive data directly to/from the RDMA Engine without Privileged Resource Managerintervention -intervention, while ensuring that the host remains secure. Thus, one of the primary goals of this document is to analyze this usage model for the enforcement that is required in the RNIC Engine to ensure that the system remains secure. DDP provides two mechanisms for transferring data: * Untagged Data Transfer -theThe incoming payload simply consumes the first buffer in a queue of buffers that are in the order specified by the receiving Peer (commonly referred to as the Receive Queue), and * Tagged Data Transfer -theThe Peer transmitting the payload explicitly states which destination buffer is targeted, through use of an STag.STag basedSTag-based transfers allow the receiving ULP to be indifferent to what order (or in what messages) the opposite Peer sent the data, or in what order packets arereceived in.received. Both data transfer mechanisms are also enabled through RDMAP, with additional control semantics.TypicallyTypically, Tagged Data Transfer can be used for payload transfer, while Untagged Data Transfer is best used for control messages. However, eachupper layer protocolUpper Layer Protocol can determine the optimal use oftaggedTagged anduntaggedUntagged messages for itself. See [APPLICABILITY] for more information on application applicability for the two transfer mechanisms. ForDDPDDP, the two forms correspond to Untagged and Tagged DDP Messages, respectively. ForRDMAPRDMAP, the two forms correspond to Send Type Messages and RDMA Messages (either RDMA Read or RDMA Write Messages), respectively. The host interfaces that could be exercised include: * Privileged Control Interface - A Privileged Resource Manager uses the RNIC Interface to allocate and manage RNIC Engine resources, control the state within the RNIC Engine, and monitor various events from the RNIC Engine. It also uses this interface to act as a proxy for some operations that a Non-Privileged ULP may require (after performing appropriate countermeasures). * ULP Control Interface - A ULP uses this interface to the Privileged Resource Manager to allocate RNIC Engine resources. The Privileged Resource Manager implements countermeasures to ensurethatthat, if the Non-Privileged ULP launches anattackattack, it can prevent the attack from affecting other ULPs. * Non-Privileged Data Transfer Interface - A Non-Privileged ULP uses this interface to initiate andtocheck the status of data transfer operations. * Privileged Data Transfer Interface - A superset of the functionality provided by the Non-Privileged Data Transfer Interface. The ULP is allowed to directly manipulate RNIC Engine mapping resources to map an STag to a ULPdata buffer.Data Buffer. If Internet control messages, such as ICMP, ARP, RIPv4, etc. are processed by the RNIC Engine, the threat analyses for those protocols is also applicable, but outside the scope of this document.2.22.2. Resources This section describes the primary resources in the RNIC Engine that could be affected if under attack. For RDMAP, allofthe defined resources apply. For DDP, allofthe resources except the RDMA Read Queue apply.2.2.12.2.1. Stream Context Memory The state information for each Stream is maintained in memory, which could be located in a number of places - on the NIC, inside RAM attached to the NIC, in host memory, or in any combination of the three, depending on the implementation. Stream Context Memory includes state associated with Data Buffers. For Tagged Buffers, this includes how STag names, Data Buffers, and Page Translation Tables (see Section 2.2.3) interrelate. It also includes the list of Untagged Data Buffers posted for reception of Untagged Messages (commonly called the Receive Queue), and a list of operations to perform to send data (commonly called the Send Queue).2.2.22.2.2. Data Buffers As mentioned previously, there are two different ways to expose a local ULP'sdata buffersData Buffers for datatransfer;transfer: Untagged DataTransfer -Transfer, where a buffer can be exposed for receiving RDMAP Send Type Messages (a.k.a. DDP Untagged Messages) on DDP Queuezero -zero, or Tagged DataTransfer -Transfer, where the buffer can be exposed for remote access through STags (a.k.a. DDP Tagged Messages). This distinction is important because the attacks and the countermeasures used to protect against the attack are different depending on the method for exposing the buffer to the network. For the purposes of the security discussion, for Tagged DataTransferTransfer, a single logical Data Buffer is exposed with a singleStagSTag on a given Stream. Actual implementations may support scatter/gather capabilities to enable multiple physical data buffers to be accessed with a single STag, but from a threat analysisperspectiveperspective, it is assumed that a single STag enables access to a single logical Data Buffer. In any event, it is the responsibility of the Privileged Resource Manager to ensure that no STag can be created that exposes memory that the consumer had no authority to expose. Adata bufferData Buffer has specific access rights. The local ULP can control whether adata bufferData Buffer is exposed for local only, or local and remote access, and assign specific access privileges (read, write, read and write) on a per Stream basis. For DDP, when an STag isadvertised,Advertised, the Remote Peer is presumably given write access rights to the data(otherwise(otherwise, therewaswould not be much point to theadvertisement).Advertisement). For RDMAP, when a ULPadvertisesAdvertises an STag, it can enable write-only, read-only, or both write and read access rights. Similarly, some ULPs may wish to provide a single buffer with different access rights on aper-Streamper Stream basis. For example, some Streams may have read-only access, some may have remote read and write access, while on otherStreamsStreams, only the local ULP/Local Peer is allowed access.2.2.32.2.3. Page Translation Tables Page Translation Tables are the structures used by the RNIC to be able to access ULP memory for data transfer operations. Even though these structures are called "Page" Translation Tables, they may not reference a page at all -conceptuallyconceptually, they are used to map a ULP address space representation(e.g.(e.g., a virtual address) of a buffer to the physical addresses that are used by the RNIC Engine to move data.IfIf, on a specificsystemsystem, a mapping is not used, then a subset of the attacks examined may be appropriate. Note that the Page Translation Table may or may not be a shared resource.2.2.42.2.4. Protection Domain (PD) A Protection Domain (PD) is a local construct to the RDMA implementation, and never visible over the wire. Protection Domains are assigned to three of the resources of concern - Stream Context Memory, STags associated with Page Translation Table entries, anddata buffers.Data Buffers. A correct implementation of a Protection Domain requires that resourceswhichthat belong to a given Protection Domaincan notcannot be used on a resource belonging to another Protection Domain, because Protection Domain membership is checked by the RNIC prior to taking any action involving such a resource. Protection Domains are therefore used to ensure that an STag can only be used to access an associateddata bufferData Buffer on one or more Streams that are associated with the same Protection Domain as the specific STag. If an implementation choosestonot to share resources between Streams, it is recommended that each Stream be associated with its own, unique Protection Domain. If an implementation chooses to allow resource sharing, it is recommended that Protection Domain be limited to the collection of Streams that have Partial Mutual Trust with each other. Note that a ULP (either Privileged or Non-Privileged) can potentially have multiple Protection Domains. This could be used, for example, to ensure that multiple clients of a server do not have the ability to corrupt each other. The server would allocate a Protection Domain per client to ensure that resources covered by the Protection Domain could not be used by another (untrusted) client.2.2.52.2.5. STag Namespace and Scope The DDP specification defines a 32-bit namespace for the STag. Implementations may vary in terms of the actual number of STags that are supported. In any case, this is a bounded resource that can come under attack. Depending upon STag namespace allocation algorithms, the actual name space to attack may be significantly less than 2^32. The scope of an STag is the set of DDP/RDMAP Streams on which the STag is valid. If an STag is valid on a particular DDP/RDMAP Stream, then that stream can modify the buffer, subject to the access rights that the stream has for the STag (see Section2.2.22.2.2, DataBuffersBuffers, for additional information). The analysis presented in this document assumes two mechanisms for limiting the scope of Streams for which the STag is valid: * Protection Domain scope. The STag is valid if used on any Stream within a specific Protection Domain, and is invalid if used on any Stream that is not a member of the Protection Domain. * Single Stream scope. The STag is valid on a single Stream, regardless of what the Stream association is to a Protection Domain. If used on any other Stream, it is invalid.2.2.62.2.6. Completion Queues Completion Queues (CQ) are used in this document to conceptually represent how the RNIC Engine notifies the ULP about the completion of the transmission of data, or the completion of the reception of data through the Data Transfer Interface (specifically for Untagged DataTransfer -Transfer; Tagged Data Transfercan notcannot cause a completion to occur). Because there could be many transmissions or receptions in flight at any one time, completions are modeled as a queue rather than as a single event. An implementation may also use the Completion Queue to notify the ULP of otheractivities,activities; for example, the completion of a mapping of an STag to a specific ULP buffer. Completion Queues may be shared by a group of Streams, or may be designated to handle a specific Stream's traffic. Limiting Completion Queue association to one, or a smallnumbernumber, of RDMAP/DDP Streams can prevent several forms of attacks by sharply limiting the scope of the attack's effect. Some implementations may allow this queue to be manipulated directly by both Non-Privileged and Privileged ULPs.2.2.72.2.7. Asynchronous Event Queue The Asynchronous Event Queue is a queue from the RNIC to the Privileged Resource Manager of bounded size. It is used by the RNIC to notify the host of various eventswhichthat might require management action, including protocol violations, Stream state changes, local operation errors, low water marks on receive queues, and possibly other events. The Asynchronous Event Queue is a resource that can be attacked because Remote or Local Peers and/or ULPs can cause events to occurwhichthat have the potential of overflowing the queue. Note that an implementation is at liberty to implement the functions of the Asynchronous Event Queue in a variety of ways, including multiple queues or even simple callbacks. All vulnerabilities identified are intended toapplyapply, regardless of the implementation of the Asynchronous Event Queue. For example, a callback function may be viewedassimply as a very short queue.2.2.82.2.8. RDMA Read Request Queue The RDMA Read Request Queue is the memory that holds state information for one or more RDMA Read Request Messages that have arrived, but for which the RDMA Read Response Messages have not yet been completely sent. Because potentially more than one RDMA Read Request can be outstanding at one time, the memory is modeled as a queue of bounded size. Some implementations may enable sharing of a single RDMA Read Request Queue across multiple Streams.2.32.3. RNIC Interactions With RNIC resources and interfaces defined, it is now possible to examine the interactions supported by the generic RNIC functional interfaces through each of the 3interfaces -interfaces: Privileged Control Interface, Privileged Data Interface, and Non-Privileged Data Interface. As mentioned previously in Section2.12.1, Components, there are two data transfer mechanisms to beexamined -examined, Untagged Data Transfer and Tagged Data Transfer.2.3.12.3.1. Privileged Control Interface Semantics Generically, the Privileged Control Interface controls the RNIC's allocation, de-allocation, and initialization of RNIC global resources. This includes allocation and de-allocation of Stream Context Memory, Page Translation Tables, STag names, Completion Queues, RDMA Read Request Queues, and Asynchronous Event Queues. The Privileged Control Interface is also typically used for managing Non-Privileged ULP resources for the Non-Privileged ULP (and possibly for the Privileged ULP as well). This includes initialization and removal of Page Translation Table resources, and managing RNIC events (possibly managing all events for the Asynchronous Event Queue).2.3.22.3.2. Non-Privileged Data Interface Semantics The Non-Privileged Data Interface enables data transfer (transmit and receive) but does not allow initialization of the Page Translation Table resources. However, once the Page Translation Table resources have been initialized, the interface may enable a specific STag mapping to be enabled and disabled by directly communicating with the RNIC, or create an STag mapping for a buffer that has been previously initialized in the RNIC. For RDMAP, ULP data can be sent by one of the previously described data transfermechanisms -mechanisms: Untagged Data Transfer or Tagged Data Transfer. Two RDMAP data transfer mechanisms are defined, one using Untagged Data Transfer (Send Type Messages), and one using Tagged Data Transfer (RDMA Read Responses and RDMA Writes). ULP data reception through RDMAP can be done by receiving Send Type Messages into buffers that have been posted on the Receive Queue or Shared Receive Queue.ThusThus, a Receive Queue or Shared Receive Queue can only be affected by Untagged Data Transfer. Data reception can also be done by receiving RDMA Write and RDMA Read Response Messages into buffers that have previously been exposed for external write access throughadvertisementAdvertisement of an STag(i.e.(i.e., Tagged Data Transfer). Additionally, to cause ULP data to be pulled (read) across the network, RDMAP uses an RDMA Read Request Message (which only contains RDMAP control information necessary to access the ULP buffer to be read), to cause an RDMA Read Response Message to be generated that contains the ULP data. For DDP, transmitting data means sending DDP Tagged or Untagged Messages. For data reception, DDP can receive Untagged Messages into buffers that have been posted on the Receive Queue or Shared Receive Queue. It can also receive Tagged DDP Messages into buffers that have previously been exposed for external write access throughadvertisementAdvertisement of an STag. Completion of data transmission or reception generally entails informing the ULP of the completed work by placing completion information on the Completion Queue. For data reception, only an Untagged Data Transfer can cause completion information to be put in the Completion Queue.2.3.32.3.3. Privileged Data Interface Semantics The Privileged Data Interface semantics are a superset of theNon-PrivilegedNon- Privileged Data Transfer semantics. The interface can do everything defined in the prior section, as well as create/destroy buffer to STag mappings directly. This generally entails initialization or clearing of Page Translation Table state in the RNIC.2.3.42.3.4. Initialization of RNIC Data Structures for Data Transfer Initialization of the mapping between an STag and a Data Buffer can be viewed in the abstract as two separate operations: a. Initialization of the allocated Page Translation Table entries with the location of the Data Buffer, and b. Initialization of a mapping from an allocated STag name to a set of Page Translation Table entry(s) orpartial-partial entries. Note that an implementation may not have a Page Translation Table(i.e.(i.e., it may support a direct mapping between an STag and a Data Buffer). If there is no Page Translation Table, then attacks based on changing its contents or exhausting its resources are not possible. Initialization of the contents of the Page Translation Table can be done by either the Privileged ULP or by the Privileged Resource Manager as a proxy for the Non-Privileged ULP. Bydefinitiondefinition, the Non-Privileged ULP is not trusted to directly manipulate the Page Translation Table. Ingeneralgeneral, the concern is that theNon-PrivilegedNon- Privileged ULP may try to maliciously initialize the Page Translation Table to access a buffer for which it does not have permission. The exact resource allocation algorithm for the Page Translation Table is outside the scope of this document. It may be allocated for a specific Data Buffer, orbe allocatedas a pooled resource to be consumed by potentially multiple Data Buffers, or be managed in some other way. This document attempts to abstract implementation dependent issues, and group them into higher level securityissuesissues, such as resource starvation and sharing of resources between Streams. The next issue is how an STag name is associated with a Data Buffer. For the case of an Untagged Data Buffer(i.e.(i.e., Untagged Data Transfer), there is no wire visible mapping between an STag and the Data Buffer. Note that there may, in fact, be an STagwhichthat represents the buffer, if an implementation chooses to internally represent Untagged Data Buffer using STags. However, because theSTagSTag, bydefinitiondefinition, is not visible on the wire, this is a localhost implementation specifichost, implementation-specific issuewhichthat should be analyzed in the context of a local hostimplementation specificimplementation-specific security analysis, andthusthus, is outside the scope of this document. For a Tagged Data Buffer(i.e.(i.e., Tagged Data Transfer), either the Privileged ULP or the Privileged Resource Manager acting on behalf of the Non-Privileged ULP may initialize a mapping from an STag to a Page Translation Table, or may have the ability to simply enable/disable an existing STag to Page Translation Table mapping. There may also be multiple STag nameswhichthat map to a specific group of Page Translation Table entries (orsub- entries).sub-entries). Specific security issues with this level of flexibility are examined in Section6.2.36.2.3, Multiple STags toaccessAccess thesame buffer.Same Buffer. There are a variety of implementation options for initialization of Page Translation Table entries and mapping an STag to a group of Page Translation Table entrieswhichthat have security repercussions. This includes support for separation ofMappingmapping an STag versus mapping a set of Page Translation Table entries, and support for ULPs directly manipulating STag to Page Translation Table entry mappings (versus requiring access through the Privileged Resource Manager).2.3.52.3.5. RNIC Data Transfer Interactions RNIC Data Transfer operations can be subdivided into sendoperationsand receive operations. For send operations, there is typically a queue that enables the ULP to post multiple operation requests to send data (referred to as the Send Queue). Depending upon the implementation, Data Buffers used in the operations may or may not have Page Translation Table entries associated with them, and may or may not have STags associated with them. Because this is a local host specific implementation issue rather than a protocol issue, the security analysis of threats and mitigations is left to the host implementation. Receive operations are different for Tagged Data Buffers versus Untagged Data Buffers(i.e.(i.e., Tagged Data Transfer vs. Untagged Data Transfer). For Untagged Data Transfer, if more than one Untagged Data Buffer can be posted by the ULP, the DDP specification requires that they be consumed in sequential order (the RDMAP specification also requires this).ThusThus, the most general implementation is that there is a sequential queue of receive Untagged Data Buffers (Receive Queue). Some implementations may also support sharing of the sequential queue between multiple Streams. In thiscasecase, defining "sequential" becomes non-trivial - ingeneralgeneral, the buffers for a single Stream are consumed from the queue in the order that they were placed on the queue, but there is no consumption order guarantee between Streams. For receive Tagged Data Transfer(i.e.(i.e., Tagged Data Buffers, RDMA Write Buffers, or RDMA Read Buffers), at some time prior to data transfer, the mapping of the STag to specific Page Translation Table entries (if present) and the mapping from the Page Translation Table entries to the Data Buffer must have been initialized (see Section 2.3.4 for interaction details).33. Trust and Resource Sharing It is assumedthatthat, ingeneralgeneral, the Local and Remote Peer are untrusted, and thus attacks by either should have mitigations in place. A separate, but related issue is resource sharing between multiple Streams. If local resources are not shared, the resources are dedicated on a per Stream basis. Resources are defined in Section2.22.2, Resources. The advantage of not sharing resources between Streams is that it reduces the types of attacks that are possible. The disadvantage of not sharing resources is that ULPs might run out of resources.ThusThus, there can be a strong incentive for sharing resources, if the security issues associated with the sharing of resources can be mitigated. It is assumed in this document that the component that implements the mechanism to control sharing of the RNIC Engine resources is the Privileged Resource Manager. The RNIC Engine exposes its resources through the RNIC Interface to the Privileged Resource Manager. All Privileged and Non-Privileged ULPs request resources from the Resource Manager (note that by definition both theNon- PrivilegedNon-Privileged and the Privileged application might try to greedily consume resources, thus creating a potential Denial of Service (DOS) attack). The Resource Manager implements resource management policies to ensure fair access to resources. The Resource Manager should be designed to take into account security attacks detailed in this document. Note that for some systems the Privileged Resource Manager may be implemented within the Privileged ULP. All Non-Privileged ULP interactions with the RNIC Engine that could affect other ULPs MUST be done using the Privileged Resource Manager as a proxy. All ULP resource allocation requests for scarce resources MUST also be done using a Privileged Resource Manager. The sharing of resources across Streams should be under the control of the ULP, both in terms of the trust model the ULP wishes to operate under, as well as the level of resource sharing the ULP wishes to give local processes. For more discussion on types of trust modelswhichthat combine partial trust and sharing of resources, see AppendixC:C, Partial Trust Taxonomy. The Privileged Resource Manager MUST NOT assume that different Streams share Partial Mutual Trust unless there is a mechanism to ensure that the Streams do indeed share Partial Mutual Trust. This can be done in several ways, including explicit notification from the ULP that owns the Streams.44. Attacker Capabilities An attacker's capabilities delimit the types of attacks that the attacker is able to launch. RDMAP and DDP require that the initial LLP Stream (and connection) be set up prior to transferring RDMAP/DDP Messages. This requires at least one round-trip handshake to occur. If the attacker is not the Remote Peer that created the initial connection, then the attacker's capabilities can be segmented into send only capabilities or send and receive capabilities. Attacking with send only capabilities requires the attacker to first guess the current LLP Stream parameters before they can attack RNIC resources(e.g.(e.g., TCP sequence number). If this class of attacker also has receive capabilities and the ability to pose as the receiver to the sender and the sender to the receiver, they are typically referred to as a "man-in-the-middle" attacker [RFC3552]. A man-in-the-middle attacker has a much wider ability to attack RNIC resources. The breadth of attack is essentially the same as that of an attacking Remote Peer(i.e.(i.e., the Remote Peer thatsetupset up the initial LLP Stream).55. Attacks That CanbeBe MitigatedWithwith End-to-End Security This section describes the RDMAP/DDP attacks where the only solution is to implement some form of end-to-end security. The analysis includes a detailed description of each attack, what is being attacked, and a description of the countermeasures that can be taken to thwart the attack. Some forms of attack involve modifying the RDMAP or DDP payload by anetwork basednetwork-based attacker or involve monitoring the traffic to discover private information. An effective tool to ensure confidentiality is to encrypt the data stream throughmechanismsmechanisms, such as IPsec encryption. Additionally, authenticationprotocolsprotocols, such as IPsecauthenticationauthentication, are an effective tool to ensure the remote entity is who they claim tobebe, as well as ensuring that the payload is unmodified as it traverses the network. Note that connection setup andteardowntear down is presumed to be done in stream mode(i.e.(i.e., no RDMA encapsulation of the payload), so there are no new attacks related to connectionsetup/teardownsetup/tear down beyond what is already present in the LLP(e.g.(e.g., TCP or SCTP). Note, however, that RDMAP/DDP parameters may be exchanged in stream mode, and if they are corrupted by an attacker unintended consequences will result. Therefore, any existing mitigations for LLP Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, or Elevation of Privilege continue to apply (and are out of scope of this document).ThusThus, the analysis in this section focuses on attacks that arepresentpresent, regardless of the LLP Stream type. Tampering is any modification of the legitimate traffic (machine internal or network). Spoofing attack is a special case of tampering where the attacker falsifies an identity of the Remote Peer (identity can be an IP address, machine name, ULP levelidentityidentity, etc.).5.15.1. Spoofing Spoofing attacks can be launched by the Remote Peer, or by anetwork basednetwork-based attacker. Anetwork basednetwork-based spoofing attack applies to all Remote Peers. This section analyzes the various types of spoofing attacks applicable to RDMAP&and DDP.5.1.15.1.1. Impersonation Anetwork basednetwork-based attacker can impersonate a legal RDMAP/DDP Peer (by spoofing a legal IP address). This can either be done as a blind attack (see [RFC3552]) or by establishing an RDMAP/DDP Stream with the victim. Because an RDMAP/DDP Stream requires an LLP Stream to be fully initialized(e.g.(e.g., for[RFC793][RFC793], it is in the ESTABLISHED state), existing transport layer protection mechanisms against blind attacks remain in place. For a blind attack to succeed, it requires the attacker to inject a valid transport layer segment(e.g.(e.g., forTCPTCP, it must match at least the 4-tuple as well as guess a sequence number within the window) while also guessing valid RDMAP or DDP parameters. There are many ways to attack the RDMAP/DDP protocol if the transport protocol is assumed to be vulnerable. For example, for Tagged Messages, this entails guessing the STag and TO values. If the attacker wishes to simply terminate the connection, it can do so by correctly guessing the transport&and network layer values, and providing an invalid STag. Per the DDP specification, if an invalid STag is received, the Stream is torn down and the Remote Peer is notified with an error. If an attacker wishes to overwrite an Advertised Buffer, it must successfully guess the correct STag and TO. Given that the TOoftenwill often start at zero, this is straightforward. The value of the STag should be chosen at random, as discussed in Section6.1.16.1.1, Using an STag on a Different Stream. For Untagged Messages, if the MSN is invalid then the connection may be torn down. If it is valid, then the receive buffers can be corrupted. End-to-end authentication(e.g.(e.g., IPsec or ULP authentication) provides protection against either the blind attack or the connected attack.5.1.25.1.2. Stream Hijacking Stream hijacking happens when anetwork basednetwork-based attacker eavesdrops on the LLP connection through the Stream establishment phase, and waits until the authentication phase (if such a phase exists) is completed successfully. The attacker then spoofs the IP address and re-directs the Stream from the victim to its own machine. For example, an attacker can wait until an iSCSI authentication is completed successfully, and then hijack the iSCSI Stream. The best protection against this form of attack is end-to-end integrity protection and authentication, such as IPsec, to prevent spoofing. Another option is to provide a physically segregated network for security. Discussion of physical security is out of scope for this document. Because the connection and/or Stream itself is established by the LLP, some LLPs are more difficult to hijack than others. Please see the relevant LLP documentation on security issues around connection and/or Stream hijacking.5.1.35.1.3. Man-in-the-Middle Attack If anetwork basednetwork-based attacker has the ability to delete or modify packetswhichthat will still be accepted by the LLP (e.g., TCP sequence number iscorrect)correct), then the Stream can be exposed to aman-in-the-middleman-in-the- middle attack. One style of attack is for theman-in- the-middleman-in-the-middle to send Tagged Messages (either RDMAP or DDP). If it can discover a buffer that has been exposed for STag enabled access, then theman-in-the-middleman- in-the-middle can use an RDMA Read operation to read the contents of the associateddata buffer,Data Buffer, perform an RDMA Write Operation to modify the contents of the associateddata buffer,Data Buffer, or invalidate the STag to disable further access to the buffer. The best protection against this form of attack is end-to-end integrity protection and authentication, such as IPsec, to prevent spoofing or tampering. If authentication and integrity protections are not used, then physical protection must be employed to prevent man-in-the-middle attacks. Because the connection/Stream itself is established by the LLP, some LLPs are more exposed to man-in-the-middle attack than others. Please see the relevant LLP documentation on security issues around connection and/or Stream hijacking. Another approach is to restrict access to only the local subnet/link, and provide some mechanism to limit access, such as physical security or 802.1.x. This model is an extremely limited deployment scenario, and will not be further examined here.5.25.2. Tampering -Network based modificationNetwork-Based Modification ofbuffer contentBuffer Content This is actually aman in the middle attack -man-in-the-middle attack, but only on the content of the buffer, as opposed to theman in the middleman-in-the-middle attack presented above, where both the signaling and content can be modified. See Section5.1.35.1.3, Man-in-the-Middle Attack.5.35.3. Information Disclosure -Network BasedNetwork-Based Eavesdropping An attacker that is able to eavesdrop on the network can read the content of all read and write accesses to a Peer's buffers. To prevent information disclosure, the read/written data must be encrypted. See also Section5.1.35.1.3, Man-in-the-Middle Attack. The encryption can be done either by the ULP, or by a protocol that can provide security services to RDMAP&and DDP(e.g.(e.g., IPsec).5.45.4. Specific Requirements for Security Services Generally speaking, Stream confidentiality protects against eavesdropping. Stream and/or session authentication and integrity protection is a counter measurement against various spoofing and tampering attacks. The effectiveness of authentication and integrity against a specific attackdependdepends on whether the authentication is machine level authentication (such as IPsec), or ULP authentication.5.4.15.4.1. Introduction to Security Options The following security services can be applied to an RDMAP/DDP Stream: 1. Session confidentiality -protectsProtects against eavesdropping (Section 5.3). 2. Per-packet data source authentication -protectsProtects against the following spoofing attacks:network basednetwork-based impersonation (Section5.1.1),5.1.1) and Stream hijacking (Section5.1.2), and man in the middle (Section 5.1.3).5.1.2). 3. Per-packet integrity -protectsProtects against tampering done bynetwork basednetwork-based modification of buffer content (Section 5.2) and when combined with authenticaiton, also protects against man-in- the-middle attacks (Section 5.1.3). 4. Packet sequencing - protects against replay attacks, which is a special case of the above tampering attack. If an RDMAP/DDP Stream may be subject to impersonation attacks, or Stream hijacking attacks, it is recommended that the Stream be authenticated, integrity protected, and protected from replay attacks; it may use confidentiality protection to protect from eavesdropping (in case the RDMAP/DDP Stream traverses a public network). IPsec is a protocol suitewhichthat is used to secure communication at the network layer between two peers. The IPsec protocol suite is specified within the IP Security Architecture [RFC2401], IKE [RFC2409], IPsec Authentication Header (AH)[RFC2402][RFC2402], and IPsec Encapsulating Security Payload (ESP) [RFC2406] documents. IKE is the key managementprotocolprotocol, while AH and ESP are used to protect IP traffic. Please see those RFCs for a complete description of the respective protocols. IPsec is capable of providing the above security services for IP and TCPtraffictraffic, respectively. ULP protocols are able to provide only part of the above security services.5.4.25.4.2. TLSisIs Inappropriate for DDP/RDMAP Security TLS[RFC 2246][RFC4346] provides Stream authentication, integrity and confidentiality for TCP based ULPs. TLS supports one-way (server only) or mutual certificates based authentication. If TLS is layered underneath RDMAP,there are at least two limitations that makeTLS's connection orientation makes TLS inappropriate for DDP/RDMAsecurity: 1. The maximum length supported by the TLS record layer protocol is 2^14 bytes - longer packets must be fragmented (as a comparison, the maximum length of an Untagged DDP Message is roughly 2^32). 2. TLS is a connection oriented protocol.security. If a stream cipher or block cipher in CBC mode is used for bulk encryption, then a packet can be decrypted only after all the packets preceding it have already arrived. If TLS is used to protect DDP/RDMAP traffic, then TCP must gather all out-of-order packets before TLS can decrypt them. Only after this is done can RDMAP/DDP place them into the ULP buffer.ThusThus, one of the primary features of DDP/RDMAP - enabling implementations to have a flow-through architecture with little to nobuffering, can notbuffering - cannot be achieved if TLS is used to protect the data stream. If TLS is layered on top of RDMAP or DDP, TLS does not protect the RDMAP and/or DDP headers.ThusThus, a man-in-the-middle attack can still occur by modifying the RDMAP/DDP header toincorrectlyplace the data into the wrong buffer, thus effectively corrupting the data stream. For these reasons, it is not RECOMMENDED that TLS be layered on top of RDMAP or DDP.5.4.35.4.3. DTLS and RDDP DTLS [DTLS] provides security services for datagram protocols, including unreliable datagram protocols. These services include anti-replay based on a mechanism adapted from IPsec that is intended to operate on packets as they are received from the network. For these and other reasons, DTLS is best applied to RDDP by employing DTLS beneath TCP, yielding a layering of RDDP over TCP over DTLS over UDP/IP. Such a layering inserts DTLS at roughly the same level in the protocol stack as IPsec, making DTLS's security services an alternative to IPsec's services from an RDDP standpoint. For RDDP, IPsec is the better choice for a security framework, and hence is mandatory-to-implement (as specified elsewhere in this document). An important contributing factor to the specification of IPsec rather than DTLS is that the non-RDDP versions of two initial adopters of RDDP (iSCSI [iSCSI][iSER] and NFSv4 [NFSv4][NFSv4.1]) are compatible with IPsec but neither of these protocols currently uses either TLS or DTLS. For the specific case of iSCSI, IPsec is the basis formandatory-to- implementmandatory-to-implement security services [RFC3723].ThereforeTherefore, this document and the RDDP protocol specifications contain mandatory implementation requirements for IPsec rather than for DTLS.5.4.45.4.4. ULPsWhichThat Provide Security ULPswhichthat provide integrated security but wish to leveragelower-layerlower- layer protocolsecuritysecurity, should be aware of security concerns around correlating a specific channel's security mechanisms to the authentication performed by the ULP. See [NFSv4CHANNEL] for additional information on a promising approach called "channel binding". From [NFSv4CHANNEL]: "The concept of channel bindings allows applications to prove that the end-points of two secure channels at different network layers are the same by binding authentication at one channel to the session protection at the other channel. The use of channel bindings allows applications to delegate session protection to lower layers, which may significantly improve performance for some applications."5.4.55.4.5. Requirements for IPsec Encapsulation of DDP The IP Storage working group has spent significant time and effort to define the normative IPsec requirements for IP Storage [RFC3723]. Portions of that specification are applicable to a wide variety of protocols, including the RDDP protocol suite. In ordertonot to replicate this effort, an RNIC implementation MUST follow the requirements defined inRFC3723RFC 3723, Section 2.3 and Section 5, including the associated normative references for those sections. Note that this means that support for IPSEC ESP mode is normative. Additionally, since IPsec acceleration hardware may only be able to handle a limited number of active IKE Phase 2 SAs, Phase 2 delete messages may be sent for idleSAs,SAs as a means of keeping the number of active Phase 2 SAs to a minimum. The receipt of an IKE Phase 2 delete message MUST NOT be interpreted as a reason for tearing down a DDP/RDMA Stream. Rather, it is preferable to leave the Stream up, and if additional traffic is sent on it, to bring up another IKE Phase 2 SA to protect it. This avoids the potential for continually bringing Streams up and down. Note that there are serious security issues if IPsec is not implemented end-to-end. For example, if IPsec is implemented as a tunnel in the middle of the network, any hosts between the Peer and the IPsec tunneling device can freely attack the unprotected Stream.6The IPsec requirements for RDDP are based on the version of IPsec specified in RFC 2401 [RFC2401] and related RFCs, as profiled by RFC 3723 [RFC3723], despite the existence of a newer version of IPsec specified in RFC 4301 [RFC4301] and related RFCs. One of the important early applications of the RDDP protocols is their use with iSCSI [iSER]; RDDP's IPsec requirements follow those of IPsec in order to facilitate that usage by allowing a common profile of IPsec to be used with iSCSI and the RDDP protocols. In the future, RFC 3723 may be updated to the newer version of IPsec; the IPsec security requirements of any such update should apply uniformly to iSCSI and the RDDP protocols. 6. Attacks from Remote Peers This section describes remote attacks that are possible against the RDMA system defined in Figure 1 - RDMA Security Model and the RNIC Engine resources defined in Section 2.2. The analysis includes a detailed description of each attack, what is being attacked, and a description of the countermeasures that can be taken to thwart the attack. The attacks are classified into five categories: Spoofing, Tampering, Information Disclosure, Denial of Service (DoS) attacks, and Elevation of Privileges. As mentioned previously, tampering is any modification of the legitimate traffic (machine internal or network). A spoofing attack is a special case of tampering where the attacker falsifies an identity of the Remote Peer (identity can be an IP address, machine name, ULP levelidentityidentity, etc.).6.16.1. Spoofing This section analyzes the various types of spoofing attacks applicable to RDMAP&and DDP. Spoofing attacks can be launched by the RemotePeer,Peer or by anetwork basednetwork-based attacker. For countermeasures against anetwork basednetwork-based attacker, see Section55, Attacks That CanbeBe MitigatedWithwith End-to-End Security.6.1.16.1.1. Using an STag on a Different Stream One style of attack from the Remote Peer is for it to attempt to use STag values that it is not authorized to use. Note that if the Remote Peer sends an invalid STag to the Local Peer, per the DDP and RDMAP specifications, the Stream must be torn down.ThusThus, the threat exists if an STag has been enabled for Remote Access on one Stream and a Remote Peer is able to use it on an unrelated Stream. If the attack is successful, the attacker could potentially be able toperformeither perform RDMA ReadOperationsoperations to read the contents of the associateddata buffer,Data Buffer, perform RDMA WriteOperationsoperations to modify the contents of the associated data buffer, ortoinvalidate the STag to disable further access to the buffer. An attempt by a Remote Peer to access a buffer with an STag on a different Stream in the same Protection Domain may or may not be anattackattack, depending on whether resource sharing is intended(i.e.(i.e., whether the Streams shared Partial MutualTrust or not).Trust). For some ULPs, using an STag on multiple Streams within the same Protection Domain could be desired behavior. For other ULPs, attempting to use an STag on a different Stream could be consideredto bean attack. Since this varies by ULP, a ULP typically would need to be able to control the scope of the STag. In the case where an implementation does not share resources between Streams (including STags), this attack can be defeated by assigning each Stream to a different Protection Domain. Before allowing remote access to the buffer, the Protection Domain of the Stream where the access attempt was made is matched against the Protection Domain of the STag. If the Protection Domains do not match, access to the buffer is denied, an error is generated, and the RDMAP Stream associated with the attacking Stream is terminated. For implementations that share resources between multiple Streams, it may not be practical to separate each Stream into its own Protection Domain. In this case, the ULP can still limit the scope of any of the STags to a single Stream (if it is enabling it for remote access). If the STag scope has been limited to a single Stream, any attempt to use that STag on a different Stream will result in an error, and the RDMAP Stream is terminated.ThusThus, for implementations that do not share STags between Streams, each Stream MUST either be in a separate Protection Domain or the scope of an STag MUST be limited to a single Stream. An RNIC MUST ensure that a specific Stream in a specific Protection Domaincan notcannot access an STag in a different Protection Domain. An RNIC MUST ensurethatthat, if an STag is limited in scope to a single Stream, no other Stream can use the STag. An additional issue may be unintended sharing of STags(i.e.(i.e., a bug in the ULP) or a bug in the Remote Peerwhichthat causes anoff- by-oneoff-by-one STag to be used. For additional protection, an implementation should allocate STags in such a fashion that it is difficult to predict the next allocated STag number, and also ensure that STags are reused at as slow a rate as possible. Any allocation methodwhichthat would lead to intentional or unintentional reuse of an STag by the peer should be avoided(e.g.(e.g., a methodwhichthat always starts with a given STag and monotonically increases it for each new allocation, or a methodwhichthat always uses the same STag for each operation).6.26.2. Tampering A Remote Peer or anetwork basednetwork-based attacker can attempt to tamper with the contents ofdata buffersData Buffers on a Local Peer that have been enabled for remote write access. The types of tampering attacks from a Remote Peer are outlined in the sections that follow. For countermeasures against anetwork basednetwork-based attacker, see Section55, Attacks That CanbeBe MitigatedWithwith End-to-End Security.6.2.16.2.1. Buffer Overrun - RDMA Write or Read Response This attack is an attempt by the Remote Peer to perform an RDMA Write or RDMA Read Response to memory outside of the valid length range of thedata bufferData Buffer enabled for remote write access. This attack can occur even when no resources are shared across Streams. This issue can also arise if the ULP has a bug. The countermeasure for this type of attack must be in the RNIC implementation, leveraging the STag. When the local ULP specifies to the RNIC the base address and thenumberumber of bytes in the buffer that it wishes to make accessible, the RNIC must ensure that the base and bounds check are applied to any access to the buffer referenced by the STag before the STag is enabled for access. When an RDMA data transfer operation (which includes an STag) arrives on a Stream, a base and bounds byte granularity access check must be performed to ensure that the operation accesses only memory locations within the buffer described by that STag. Thus an RNIC implementation MUST ensure that a Remote Peer is not able to access memory outside of the buffer specified when the STag was enabled for remote access.6.2.26.2.2. Modifying a BufferAfterafter Indication This attack can occur if a Remote Peer attempts to modify the contents of an STag referenced buffer by performing an RDMA Write or an RDMA Read Response after the Remote Peer has indicated to the Local Peer or local ULP (by a variety of means) that the STagdata bufferData Buffer contents are ready for use. This attack can occur even when no resources are shared across Streams. Note that a bug in a Remote Peer, ornetwork basednetwork-based tampering, could also result in this problem. For example, assume that the STag referenced buffer contains ULP control information as well as ULP payload, and the ULP sequence of operation is to first validate the control information and then perform operations on the control information. If the Remote Peer can perform an additional RDMA Write or RDMA Read Response(thus(thus, changing the buffer) after the validity checks have been completed but before the control data is operated on, the Remote Peer could force the ULP down operational paths that were never intended. The local ULP can protect itself from this type of attack by revoking remote access when the original data transfer has completed and before it validates the contents of the buffer. The local ULP caneitherdo this either by explicitly revoking remote access rights for the STag when the Remote Peer indicates the operation has completed, or by checking to make sure the Remote Peer invalidated the STag through the RDMAP Remote Invalidatecapability (seecapability. If the Remote Peer did not invalidate the STag, the local ULP then explicitly revokes the STag remote access rights. (See Section6.4.56.4.5, Remote Invalidate an STag Shared on Multiple Streams for a definition of RemoteInvalidate), and if it did not, the local ULP then explicitly revokes the STag remote access rights.Invalidate.) The local ULP SHOULD follow the above procedure to protect the buffer before it validates the contents of the buffer (or uses the buffer in any way). An RNIC MUST ensure that network packets using the STag for a previouslyadvertised bufferAdvertised Buffer can no longer modify the buffer after the ULP revokes remote access rights for the specific STag.6.2.36.2.3. Multiple STags toaccessAccess thesame bufferSame Buffer See Section 6.3.6 Using Multiple STagsWhichThat Alias to the SameBufferBuffer, for this analysis.6.36.3. Information Disclosure The main potential source for information disclosure is through a local buffer that has been enabled for remote access. If the buffer can be probed by a Remote Peer on another Stream, then there is potential for information disclosure. The potential attacks that could result in unintended information disclosure and countermeasures are detailed in the following sections.6.3.16.3.1. Probingmemory outsideMemory Outside of thebuffer boundsBuffer Bounds This is essentially the same attack as described in Section6.2.16.2.1, Buffer Overrun - RDMA Write or Read Response, except that an RDMA Read Request is used to mount the attack. The same countermeasure applies.6.3.26.3.2. Using RDMA Read to Access Stale Data If a buffer is being used for some combination of reads and writes (either remote or local), and is exposed to a Remote Peer with at least remote read access rights before it is initialized with the correct data, there is a potential race condition where the Remote Peer can view the prior contents of the buffer. This becomes a security issue if the prior contents of the buffer were not intended to be shared with the Remote Peer. To eliminate this race condition, the local ULP SHOULD ensure that no stale data is contained in the buffer before remote read access rights are granted (this can be done by zeroing the contents of the memory, for example). This ensures that the Remote Peercan notcannot access the buffer until the stale data has been removed.6.3.36.3.3. Accessing a BufferAfterafter the Transfer If the Remote Peer has remote read access to abuffer, andbuffer and, by somemechanismmechanism, tells the local ULP that the transfer has been completed, but the local ULP does not disable remote access to the buffer before modifying the data, it is possible for the Remote Peer to retrieve the new data. This is similar to the attack defined in Section6.2.26.2.2, Modifying a BufferAfterafter Indication. The same countermeasures apply. In addition, the local ULP SHOULD grant remote read access rights only for the amount of time needed to retrieve the data.6.3.46.3.4. Accessing Unintended DataWithwith a Valid STag If the ULP enables remote access to a buffer using an STag that references the entire buffer, but intends only a portion of the buffer to be accessed, it is possible for the Remote Peer to access the other parts of the buffer anyway. To prevent this attack, the ULP SHOULD set the base and bounds of the buffer when the STag is initialized to expose only the data to be retrieved.6.3.56.3.5. RDMA Read into an RDMA Write Buffer One form of disclosure can occur if the access rights on the buffer enabled remote read, when only remote write access was intended. If the buffer contained ULP data, or data from a transfer on an unrelated Stream, the Remote Peer could retrieve the data through an RDMA Read operation. Note that an RNIC implementation is not required to support STags that have both read and write access. The most obvious countermeasure for this attack is to not grant remote read access if the buffer is intended to be write-only. Then the Remote Peer would not be able to retrieve data associated with the buffer. An attempt to do so would result in an error and the RDMAP Stream associated with the Stream would be terminated.ThusThus, if a ULP only intends a buffer to be exposed for remote write access, it MUST set the access rights to the buffer to only enable remote write access. Note that this requirement is not meant to restrict the use of zero-length RDMA Reads. Zero-length RDMA Reads do not expose ULP data. Because they are intended to be used as a mechanism to ensure that all RDMA Writes have been received, and do not even require a valid STag, their use is permitted even if a buffer has only been enabled for write access.6.3.66.3.6. Using Multiple STagsWhichThat Alias to the Same Buffer Multiple STagswhichthat alias to the same buffer at the same time can result in unintentional information disclosure if the STags are used by different, mutuallyuntrusted,untrusted Remote Peers. This model applies specifically to client/server communication, where the server is communicating with multiple clients, each of which do not mutually trust each other. If only read access is enabled, then the local ULP has complete control over information disclosure.ThusThus, a serverwhichthat intended to expose the same data(i.e.(i.e., buffer) to multiple clients by using multiple STags to the same buffer creates no new security issues beyond what has already been described in this document. Note that if the server did not intend to expose the same data to the clients, it should use separate buffers for each client (and separate STags). When one STag has remote read access enabled and a different STag has remote write access enabled to the same buffer, it is possible for one Remote Peer to view the contents that have been written by another Remote Peer. If both STags have remote write access enabled and the two Remote Peers do not mutually trust each other, it is possible for one Remote Peer to overwrite the contents that have been written by the other Remote Peer.ThusThus, a ULP with multiple Remote Peerswhichthat do not share Partial Mutual Trust MUST NOT grant write access to the same buffer through different STags. A buffer should be exposed to only one untrusted Remote Peer at a time to ensure that no information disclosure or information tampering occurs between peers.6.46.4. Denial of Service (DOS) A DOS attack is one of the primary security risks of RDMAP. This is because RNIC resources are valuable and scarce, and many ULP environments require communication with untrusted Remote Peers. If the Remote Peer can be authenticated or the ULP payload encrypted, clearly, the DOS profile can be reduced. For the purposes of this analysis, it is assumed that the RNIC must be able to operate in untrusted environments, which are open toDOS styleDOS-style attacks. Denial of service attacks against RNIC resources are not the typical unknown party spraying packets at a random host (such as a TCP SYN attack). Because the connection/Stream must be fully established(e.g.(e.g., a3 message3-message transport layer handshake has occurred), the attacker must be able to both send and receive messages over that connection/Stream, or be able to guess a valid packet on an existing RDMAP Stream. This section outlines the potential attacks and the countermeasures available for dealing with each attack.6.4.16.4.1. RNIC Resource Consumption This section covers attacks that fall into the general category of a local ULP attempting to unfairly allocate scarce(i.e.(i.e., bounded) RNIC resources. The local ULP may be attempting to allocate resources on its own behalf, or on behalf of a Remote Peer. Resources that fall into this categoryinclude:include Protection Domains, Stream Context Memory, Translation and Protection Tables, and STag namespace. These can be due to attacks by currently active local ULPs or ones that allocated resourcesearlier,earlier but are now idle. This type of attack can occur regardless of whetheror notresources are shared across Streams. The allocation of all scarce resources MUST be placed under the control of a Privileged Resource Manager. This allows the Privileged Resource Manager to: * prevent a local ULP from allocating more than its fair share of resources. * detect if a Remote Peer is attempting to launch a DOS attack by attempting to create an excessive number of Streams (with associated resources) and take corrective action (such as refusing the request or applying network layer filters against the Remote Peer). This analysis assumes that the Resource Manager is responsible for handing out Protection Domains, and that RNIC implementations will provide enough Protection Domains to allow the Resource Manager to be able to assign a unique Protection Domain for each unrelated, untrusted local ULP (for a bounded, reasonable number of local ULPs). This analysis further assumes that the Resource Manager implements policies to ensure that untrusted local ULPs are not able to consume allofthe Protection Domains through a DOS attack. Note that Protection Domain consumption cannot result from a DOS attack launched by a Remote Peer, unless a local ULP is acting on the Remote Peer's behalf.6.4.26.4.2. Resource Consumption by Idle ULPs The simplest form of a DOSattackattack, given a fixed amount ofresourcesresources, is for the Remote Peer to createaan RDMAP Stream to a Local Peer,andrequest dedicatedresourcesresources, and then do no actual work. This allows the Remote Peer to be very light weight(i.e.(i.e., only negotiate resources, but do no data transfer) and consumes a disproportionate amount of resources at the Local Peer. A general countermeasure for this style of attack is to monitor active RDMAP Streamsandand, if resources are getting low, to reap the resources from RDMAP Streams that are not transferring data and possibly terminate the Stream. This would presumably be under administrative control. Refer to Section 6.4.1 for the analysis and countermeasures for this style of attack on the following RNIC resources: Stream Context Memory, Page TranslationTablesTables, and STag namespace. Note that some RNIC resources are not at risk of this type of attack from a Remote Peer because an attack requires the Remote Peer to send messages in order to consume the resource. Receive Data Buffers, Completion Queue, and RDMA Read Request Queue resources are examples. These resources are, however, at risk from a local ULP that attempts to allocate resources, then goes idle. This could also be created if the ULP negotiates the resource levels with the Remote Peer, which causes the Local Peer to consumeresources, howeverresources; however, the Remote Peer never sends data to consume them. The general countermeasure described in this section can be used to free resources allocated by an idle Local Peer.6.4.36.4.3. Resource ConsumptionByby Active ULPs This section describes DOS attacks from Local and Remote Peers that are actively exchanging messages. Attacks on each RDMA NIC resource are examined and specific countermeasures are identified. Note that attacks on Stream Context Memory, Page Translation Tables, and STag namespace are covered in Section6.4.16.4.1, RNIC Resource Consumption, so they are not included here.6.4.3.16.4.3.1. Multiple Streams Sharing Receive Buffers The Remote Peer can attempt to consume more than its fair share of receivedata buffers (i.e.Data Buffers (i.e., UntaggedbuffersBuffers for DDPareor Send Type Messages for RDMAP) if receive buffers are shared across multiple Streams. If resources are not shared across multiple Streams, then this attack is not possible because the Remote Peer will not be able to consume more buffers than were allocated to the Stream. The worst case scenario is that the Remote Peer can consume more receive buffers than the local ULP allowed, resulting in no buffers being available, which could cause the Remote Peer's Stream to the Local Peer to be torn down, and all allocated resources to be released. If local receivedata buffersData Buffers are shared among multiple Streams, then the Remote Peer can attempt to consume more than its fair share of the receive buffers, causing a different Stream to be short of receive buffers,thusand thus, possibly causing the other Stream to be torn down. For example, if the Remote Peer sent enoughone byteone-byte Untagged Messages, they might be able to consume alllocal sharedlocally shared, receive queue resources with little effort on their part. One method the Local Peer could use is to recognize that a Remote Peer is attempting to use more than its fair share of resources and terminate the Stream (causing the allocated resources to be released). However, if the Local Peer is sufficiently slow, it may be possible for the Remote Peer to still mount a denial of service attack. One countermeasure that can protect against this attack is implementing a low-water notification. The low-water notification alerts the ULP if the number of buffers in the receive queue is less than a threshold. If allofthe following conditions are true, then the Local Peer or local ULP can size the amount of local receive buffers posted on the receive queue to ensure a DOS attack can be stopped. *aA low-water notification is enabled, and *theThe Local Peer is able to bound the amount of time that it takes to replenish receive buffers, and *theThe Local Peer maintains statistics to determine which Remote Peer is consuming buffers. The above conditions enable the low-water notification to arrive before resources aredepleteddepleted, andthusthus, the Local Peer or local ULP can take corrective action (e.g., terminate the Stream of the attacking Remote Peer). A different, butsimilarsimilar, attack is if the Remote Peer sends a significant number of out-of-order packets and the RNIC has the ability to use the ULP buffer(i.e.(i.e., the Untagged Buffer for DDP or the buffer consumed by a Send Type Message for RDMAP) as a reassembly buffer. In thiscasecase, the Remote Peer can consume a significant number of ULP buffers, but never send enough data to enable the ULP buffer to be completed to the ULP. An effective countermeasure is to create a high-water notificationwhichthat alerts the ULP if there is more than a specified number of receive buffers "in process" (partially consumed, but not completed). The notification is generated when more than the specified number of buffers are in process simultaneously on a specific Stream (i.e., packets have started to arrive for the buffer, but the buffer has not yet been delivered to the ULP). A different countermeasure is for the RNIC Engine to provide the capability to limit the Remote Peer's ability to consume receive buffers on a per Stream basis.UnfortunatelyUnfortunately, this requires a large amount of state to be tracked in each RNIC on a per Stream basis. Thus, if an RNIC Engine provides the ability to share receive buffers across multiple Streams, the combination of the RNIC Engine and the Privileged Resource Manager MUST be able to detect if the Remote Peer is attempting to consume more than its fair share of resources so that the Local Peer or local ULP can apply countermeasures to detect and prevent the attack.6.4.3.26.4.3.2. Remote or Local Peer Attacking a Shared CQ For an overview of the shared CQ attack model, see Section 7.1. The Remote Peer can attack a shared CQ by consuming more than its fair share of CQ entries by using one of the following methods: * The ULP protocol allows the Remote Peer to cause the local ULP to reserve a specified number of CQ entries, possibly leaving insufficient entries for other Streams that are sharing the CQ. * If the Remote Peer, Local Peer, or local ULP (or any combination) can attack the CQ by overwhelming the CQ with completions, then completion processing on other Streams sharing that Completion Queue can be affected(e.g.(e.g., the Completion Queue overflows and stops functioning). The first method of attack can be avoided if the ULP does not allow a Remote Peer to reserve CQentriesentries, or if there is a trustedintermediaryintermediary, such as a Privileged Resource Manager.UnfortunatelyUnfortunately, it is often unrealistictonot to allow a Remote Peer to reserve CQentries -entries, particularly if the number of completion entries is dependent on other ULP negotiated parameters, such as the amount of buffering required by the ULP.ThusThus, an implementation MUST implement a Privileged Resource Manager to control the allocation of CQ entries. See Section2.1 Components2.1, Components, for a definition of a Privileged Resource Manager. One way that a Local or Remote Peer can attempt to overwhelm a CQ with completions is by sending minimum length RDMAP/DDP Messages to cause as many completions (receive completions for the Remote Peer, send completions for the Local Peer) per second as possible. If it is the Remote Peer attacking, and we assume that the Local Peer's receive queue(s) do not run out of receive buffers (if they do, then this is a different attack, documented in Section 6.4.3.1 Multiple Streams Sharing Receive Buffers), then it might be possible for the Remote Peer to consume more than its fair share of Completion Queue entries. Depending upon the CQ implementation, this could either cause the CQ to overflow (if it is not large enough to handle allofthe completions generated) or for another Streamtonot to be able to generate CQ entries (if the RNIC had flow control on generation of CQ entries into the CQ). In either case, the CQ will stop functioningcorrectlycorrectly, and any Streams expecting completions on the CQ will stop functioning. This attack can occur regardless of whether allofthe Streams associated with the CQ are in the sameProtection Domainorare indifferent Protection Domains - the key issue is that the number of Completion Queue entries is less than the number of all outstanding operations that can cause a completion. The Local Peer can protect itself from this type of attack using either of the following methods: * Size the CQ to the appropriate level, as specified below (note that if the CQ currentlyexists,exists anditneeds to be resized, resizing the CQ is not required to succeed in all cases, so the CQ resize should be done before sizing the Send Queue and Receive Queue on the Stream), OR * Grant fewer resources than the Remote Peer requested (not supplying the number of Receive Data Buffers requested). The proper sizing of the CQ is dependent on whether the local ULP(s) will post as many resources to the various queues as the size of the queueenables or not.enables. If the local ULP(s) can be trusted to post a number of resources that is smaller than the size of the specific resource's queue, then a correctly sized CQ means that the CQ is large enough to hold completion status for allofthe outstanding Data Buffers (both send and receive buffers), or: CQ_MIN_SIZE = SUM(MaxPostedOnEachRQ) + SUM(MaxPostedOnEachSRQ) + SUM(MaxPostedOnEachSQ) Where: MaxPostedOnEachRQ = the maximum number of requestswhichthat can cause a completion that will be posted on a specific Receive Queue. MaxPostedOnEachSRQ = the maximum number of requestswhichthat can cause a completion that will be posted on a specific Shared Receive Queue. MaxPostedOnEachSQ = the maximum number of requestswhichthat can cause a completion that will be posted on a specific Send Queue. If the local ULP must be able to completely fill the queues, orcan notcannot be trusted to observe a limit smaller than the queues, then the CQ must be sized to accommodate the maximum number of operations that it is possible to post at any one time.ThusThus, the equation becomes: CQ_MIN_SIZE = SUM(SizeOfEachRQ) + SUM(SizeOfEachSRQ) + SUM(SizeOfEachSQ) Where: SizeOfEachRQ = the maximum number of requestswhichthat can cause a completion that can ever be posted on a specific Receive Queue. SizeOfEachSRQ = the maximum number of requestswhichthat can cause a completion that can ever be posted on a specific Shared Receive Queue. SizeOfEachSQ = the maximum number of requestswhichthat can cause a completion that can ever be posted on a specific Send Queue.WhereMaxPosted*OnEach*Q and SizeOfEach*Qvariesvary on a per Stream or per Shared Receive Queue basis. If the ULP is sharing a CQ across multiple Streamswhichthat do not share Partial Mutual Trust, then the ULP MUST implement a mechanism to ensure that the Completion Queuecandoes not overflow. Note that it is possible to share CQs even if the Remote Peers accessing the CQs are untrusted if either of the above two formulas are implemented. If the ULP can be trustedtonot to post more than MaxPostedOnEachRQ, MaxPostedOnEachSRQ, and MaxPostedOnEachSQ, then the first formula applies. If the ULPcan notcannot be trusted to obey the limit, then the second formula applies.6.4.3.36.4.3.3. Attacking the RDMA Read Request Queue The RDMA Read Request Queue can be attacked if the Remote Peer sends more RDMA Read Requests than the depth of the RDMA Read Request Queue at the Local Peer. If the RDMA Read Request Queue is a shared resource, this could corrupt the queue. If the queue is not shared, then the worst case is that the current Stream is no longer functional(e.g.(e.g., torn down). One approach to solving the shared RDMA Read Request Queue would be to create thresholds, similar to those described in Section6.4.3.16.4.3.1, Multiple Streams Sharing Receive Buffers. A simpler approach is to not share RDMA Read Request Queue resources among Streams or to enforce hard limits of consumption per Stream.ThusThus, RDMA Read Request Queue resource consumption MUST be controlled by the Privileged Resource Manager such that RDMAP/DDP Streamswhichthat do not share Partial Mutual Trust do not share RDMA Read Request Queue resources. If the issue is a bug in the Remote Peer's implementation, but not a malicious attack, the issue can be solved by requiring the Remote Peer's RNIC to throttle RDMA Read Requests. By properly configuring the Stream at the Remote Peer through a trusted agent, the RNIC can be madetonot to transmit RDMA Read Requests that exceed the depth of the RDMA Read Request Queue at the Local Peer. If the Stream is correctly configured, and if the Remote Peer submits more requests than the Local Peer's RDMA Read Request Queue can handle, the requests would be queued at the Remote Peer's RNIC until previous requests complete. If the Remote Peer's Stream is not configured correctly, the RDMAP Stream is terminated when more RDMA Read Requests arrive at the Local Peer than the Local Peer can handle (assuming that the prior paragraph's recommendation is implemented).ThusThus, an RNIC implementation SHOULD provide a mechanism to cap the number of outstanding RDMA Read Requests. The configuration of this limit is outside the scope of this document.6.4.46.4.4. Exercise ofnon-optimal code pathsNon-Optimal Code Paths Another form of a DOS attack is to attempt to exercise data paths that can consume a disproportionate amount of resources. An example might be if error cases are handled on a "slow path" (consuming either host or RNIC computational resources), and an attacker generates excessive numbers of errors in an attempt to consume these resources. Note that for most RDMAP or DDP errors, the attacking Stream will simply be torn down.ThusThus, for this form of attack to be effective, the Remote Peer needs to exercise data pathswhichthat do not cause the Stream to be torn down. If an RNIC implementation contains "slow paths"whichthat do not result in the tear down of the Stream, it is recommended that an implementation provide the ability to detect the above condition and allow an administrator to act, including potentially administratively tearing down the RDMAP Stream associated with the Stream that is exercising datapaths consumingpaths, which consume a disproportionate amount of resources.6.4.56.4.5. Remote Invalidate an STag Shared on Multiple Streams If a Local Peer has enabled an STag for remote access, the Remote Peer could attempt toremoteremotely invalidate the STag by using the RDMAP Send with Invalidate or Send with SE and Invalidate Message. If the STag is only valid on the current Stream, then the only side effect is that the Remote Peer can no longer use the STag;thusthus, there are no security issues. If the STag is valid across multiple Streams, then the Remote Peer can prevent other Streams from using that STag by using theremote invalidateRemote Invalidate functionality.ThusThus, if RDDP Streams do not share Partial Mutual Trust(i.e.(i.e., the Remote Peer may attempt toremoteremotely invalidate the STag prematurely), the ULP MUST NOT enable an STagwhichthat would be valid across multiple Streams.6.4.66.4.6. Remote PeerattackingAttacking an Unshared CQ The Remote Peer can attack an unshared CQ if the Local Peer does not size the CQ correctly. For example, if the Local Peer enables the CQ to handle completions of received buffers, and the receive buffer queue is longer than the Completion Queue, then an overflow can potentially occur. The effect on the attacker's Stream is catastrophic.HoweverHowever, if an RNIC does not have the proper protections in place, then an attack to overflow the CQ can also cause corruption and/or termination of an unrelated Stream.ThusThus, an RNIC MUST ensure that if a CQ overflows, any Streamswhichthat do not use the CQ MUST remain unaffected.6.56.5. Elevation of Privilege The RDMAP/DDP Security Architecture explicitly differentiates between three levels ofprivilege -privilege: Non-Privileged, Privileged, and the Privileged Resource Manager. If a Non-Privileged ULP is able to elevate its privilege level to a Privileged ULP, then mapping a physical address list to an STag can provide local and remote access to any physical address location on the node. If a Privileged Mode ULP is able to promote itself to be a Resource Manager, then it is possible for it to perform denial of service type attacks where substantial amounts of local resources could be consumed. In general, elevation of privilege is a local implementation specific issue and is thus outside the scope of this document.77. Attacks from Local Peers This section describes local attacks that are possible against the RDMA system defined in Figure 1 - RDMA Security Model and the RNIC Engine resources defined in Section 2.2.7.17.1. Local ULP Attacking a Shared CQ DOS attacks against a Shared Completion Queue (CQ - see Section2.2.62.2.6, Completion Queues) can be caused by either the local ULP or the Remote Peer if either attempts to cause more completions than its fair share of the number ofentries, thusentries; thus, potentially starving another unrelated ULP such that no Completion Queue entries are available. A Completion Queue entry can potentially be maliciously consumed by a completion from the Send Queue or a completion from the Receive Queue. In the former, the attacker is the local ULP. In the latter, the attacker is the Remote Peer. A form of attack can occur where the local ULPs can consume resources on the CQ. A local ULP that is slow to free resources on the CQ by not reaping the completion status quickly enough could stall all other local ULPs attempting to use that CQ. For these reasons, an RNIC MUST NOT enable sharing a CQ across ULPs that do not share Partial Mutual Trust.7.27.2. Local Peer Attacking the RDMA Read Request Queue If RDMA Read Request Queue resources are pooled across multiple Streams, one attack is if the local ULP attempts to unfairly allocate RDMA Read Request Queue resources for its Streams. For example, a local ULP attempts to allocate all available resources on a specific RDMA Read Request Queue for its Streams, thereby denying the resource to ULPs sharing the RDMA Read Request Queue. The same type of argument applies even if the RDMA Read Request is notshared -shared, but a local ULP attempts to allocate allofthe RNIC's resources when the queue is created.ThusThus, access to interfaces that allocate RDMA Read Request Queue entries MUST be restricted to a trusted Local Peer, such as a Privileged Resource Manager. The Privileged Resource Manager SHOULD prevent a local ULP from allocating more than its fair share of resources.7.37.3. Local ULP Attacking the PTT&and STag Mapping If a Non-Privileged ULP is able to directly manipulate the RNIC Page Translation Tables (which translate from an STag to a host address), it is possible that the Non-Privileged ULP could point the Page Translation Table at an unrelated Stream's or ULP's buffersand therebyand, thereby, be able to gain access to information of the unrelated Stream/ULP. As discussed in Section22, Architectural Model, introduction of a Privileged Resource Manager to arbitrate the mapping requests is an effective countermeasure. This enables the Privileged Resource Manager to ensure that a local ULP can only initialize the Page Translation Table(PTT)to(PTT) to point to its own buffers.ThusThus, if Non-Privileged ULPs are supported, the Privileged Resource Manager MUST verify that the Non-Privileged ULP has the right to access a specific Data Buffer before allowing an STag for which the ULP has access rights to be associated with a specific Data Buffer. This can be done when the Page Translation Table is initialized to access the Data Buffer or when the STag is initialized to point to a group of Page Translation Table entries, or both.88. Security considerations Please see Sections55, Attacks That Can be MitigatedWith End-to- End Security,with End-to-End Security; Section66, Attacks from RemotePeers,Peers; and Section77, Attacks from Local Peers, for a detailed analysis of attacks and normative countermeasures to mitigate the attacks. Additionally, the appendices provide a summary of the security requirements for specific audiences.Section 11AppendixA:A, ULP Issues for RDDP Client/ServerProtocolsProtocols, provides a summary of implementation issues and requirements for applicationswhichthat implement a traditional client/server style of interaction. It provides additional insight and applicability of the normative text in Sections 5, 6, and 7.Section 12,AppendixB:B, Summary of RNIC and ULP ImplementationRequirementsRequirements, provides a convenient summary of normative requirements for implementers.99. IANA Considerations IANA considerations are not addressed by this document. Any IANA considerations resulting from the use of DDP or RDMA must be addressed in the relevant standards.1010. References10.110.1. Normative References [DDP] Shah, H.,J.Pinkerton,R.J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports",Internet-Draft Work in Progress draft-ietf-rddp-ddp-05.txt, July 2005.RFC 5041, October 2007. [RDMAP] Recio, R.,P.Culley,D.P., Garcia, D., and J. Hilland,"An RDMA"A Remote Direct Memory Access Protocol Specification",Internet-Draft Work in Progress draft-ietf-rddp-rdmap-05.txt, July 2005. [RFC2406] Kent, S., Atkinson, R. "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998. [RFC2409] Harkins, D., Carrel, D., "The Internet Key Exchange (IKE)",RFC2409, November 1998.5040, October 2007. [RFC2401] Kent,S., Atkinson,S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998. [RFC2402] Kent,S., Atkinson,S. and R. Atkinson, "IP Authentication Header", RFC 2402, November 1998. [RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998. [RFC2409] Harkins, D. and D. Carrel, "The Internet Key Exchange (IKE)", RFC 2409, November 1998. [RFC3723] Aboba, B.,et al,Tseng, J., Walker, J., Rangan, V., and F. Travostino, "Securing Block Storage Protocols over IP",RFC3723,RFC 3723, April 2004. [RFC2960] Stewart,R. et al.,R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang, L., and V. Paxson, "Stream Control Transmission Protocol", RFC 2960, October 2000. [RFC793] Postel, J., "Transmission ControlProtocol - DARPA Internet Program Protocol Specification",Protocol", STD 7, RFC 793, September 1981.10.210.2. Informative References[RFC2828] Shirley,[RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005. [RFC4346] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.1", RFC 4346, April 2006. [RFC4949] Shirey, R., "Internet SecurityGlossary", FYI 36,Glossary, Version 2", RFC2828, May 2000.4949, August 2007. [APPLICABILITY] Bestler, C., Coene,and L. Coene, "Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement (DDP)",Internet-Draft Work in Progress draft-ietf- rddp-applicability-06.txt, April 2006. [IPv6-Trust] Nikander, P., J.Kempf, E. Nordmark, "IPv6 Neighbor Discovery Trust Models and threats", Informational RFC, RFC3756, May 2004.RFC 5045, October 2007. [NFSv4CHANNEL] Williams, N., "On the Use of Channel Bindings to Secure Channels",Internet-Draft draft-ietf-nfsv4-channel- bindings-02.txt,Work in Progress, July 2004. [VERBS-RDMAC] "RDMA Protocol Verbs Specification", RDMA Consortium standard, April2003. http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs- v1.0-RDMAC.pdf2003, <http://www.rdmaconsortium.org/ home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>. [VERBS-RDMAC-Overview] "RDMA enabled NIC (RNIC) Verbs Overview", slide presentation by Renato Recio, April2003. http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf2003, <http://www.rdmaconsortium.org/home/ RNIC_Verbs_Overview2.pdf>. [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations",Best Current Practice RFC,BCP 72, RFC 3552, July 2003. [INFINIBAND] "InfiniBand Architecture Specification Volume 1", release 1.2, InfiniBand Trade Associationstandard. http://www.infinibandta.org/specs.standard, <http://www.infinibandta.org/specs>. Verbs are documented in chapter 11. [DTLS] Rescorla, E.Rescorlaand N. Modadugu, "Datagram Transport Layer Security", RFC 4347, April 2006. [iSCSI]J.Satran,et al,J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004.[ISER] M.[iSER] Ko,et al, "iSCSIM., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H., and P. Thaler, "Internet Small Computer System Interface (iSCSI) Extensions forRDMAthe Remote Direct Memory Access (RDMA) Specification",Internet-Draft Work in Progress draft-ietf-ips-iser-05.txt,RFC 5046, October2005.2007. [NFSv4]S.Shepler,et al,S., Callaghan, B., Robinson, D., Thurlow, R., Beame, C., Eisler, M., and D. Noveck, "Network File System (NFS) version 4 Protocol", RFC 3530, April 2003. [NFSv4.1]S.Shepler,ed.,S., Ed., Eisler, M., Ed., and D. Noveck, Ed., "NFSv4 Minor Version 1",Internet- Draft draft-ietf-nfsv4-minorversion1-03.txt,Work in Progress,June 2006. 11September 2007. Appendix A: ULP Issues for RDDP Client/Server Protocols This section is a normative appendix to the document that is focused on client/server ULP implementation requirements to ensure a secure server implementation. The prior sections outlined specific attacks and their countermeasures. This section summarizes the attacks and countermeasures that have been defined in the priorsectionsection, which are applicable to creation of a secure ULP(e.g.(e.g., application) server. A ULP server is defined as a ULPwhichthat must be able to communicate with many clientswhichthat do not necessarily have a trust relationship with each other, and to ensure that each clientcan notcannot attack another client through server interactions. Further, the server may wish to use multiple Streams to communicate with a specific client, and those Streams may share mutual trust. Note that this section assumes a compliant RNIC and Privileged Resource Manager implementation -thusthus, it focuses specifically on ULP server(e.g.(e.g., application) implementation issues. All of the prior section's details on attacks and countermeasures apply to theserver, thusserver; thus, requirementswhichthat are repeated in this section use non-normative "must", "should", and "may". In somecasescases, normative SHOULD statements for the ULP from the main body of this document are made MUST statements for the ULP server because the operating conditions can be refined to make the motives for a SHOULD inapplicable. If a prior SHOULD is changed to a MUST in this section, it is explicitly noted and it usesupper-caseuppercase normative statements. The following list summarizes the relevant attacks that clients can mount on the sharedserver,server by re-stating the previous normative statements to be client/server specific. Note that each client/server ULP may employ explicit RDMAoperationsOperations (RDMA Read, RDMA Write) in differing fashions.ThereforeTherefore, where appropriate, "Local ULP", "LocalPeer"Peer", and "Remote Peer" are used in place of "server" or "client", in order to retain full generality of each requirement. * Spoofing * Sections 5.1.1 to 5.1.3. For protection against many forms of spoofing attacks, enable IPsec. * Section6.1.16.1.1, Using an STag on a Different Stream. To ensure that one clientcan notcannot access another client's data via use of the other client's STag, the server ULP must either scope an STag to a single Stream or use a unique Protection Domain per client. If a single client has multiple Streams that share Partial Mutual Trust, then the STag can be shared between the associated Streams by using a single Protection Domain among the associated Streams (see Section5.4.45.4.4, ULPsWhichThat ProvideSecuritySecurity, for additional issues). To prevent unintended sharing of STags within the associated Streams, a server ULP should use STags in such a fashion that it is difficult to predict the next allocated STag number. * Tampering * 6.2.2 Modifying a BufferAfterafter Indication. Before the local ULP operates on a buffer that was written by the Remote Peer using an RDMA Write or RDMA Read, the local ULP MUST ensure the buffer can no longer bemodified,modified by invalidating the STag for remote access (note that this is stronger than the SHOULD in Section 6.2.2). This caneitherbe doneexplicitlyeither by explicitly revoking remote access rights for the STag when the Remote Peer indicates the operation has completed, or by checking to make sure the Remote Peer Invalidated the STag through the RDMAP Invalidatecapability, and if itcapability. If the Remote Peer didnot,not invalidate the STag, the local ULP then explicitlyrevokingrevokes the STag remote access rights. * Information Disclosure *6.3.26.3.2, Using RDMA Read to Access Stale Data. In a general purpose serverenvironmentenvironment, there is no compelling rationaletonot to require a buffer to be initialized before remote read is enabled (and an enormousdown sidedownside of unintentionally sharing data).ThusThus, a local ULP MUST (this is stronger than the SHOULD in Section 6.3.2) ensure that no stale data is contained in a buffer before remote read access rights are granted to a Remote Peer (this can be done by zeroing the contents of the memory, for example). *6.3.36.3.3, Accessing a BufferAfterafter the Transfer. This mitigation is already covered by Section 6.2.2 (above). *6.3.46.3.4, Accessing Unintended DataWithwith a Valid STag. The ULP must set the base and bounds of the buffer when the STag is initialized to expose only the data to be retrieved. *6.3.56.3.5, RDMA Read into an RDMA Write Buffer. If a peer only intends a buffer to be exposed for remote write access, it must set the access rights to the buffer to only enable remote write access. *6.3.66.3.6, Using Multiple STagsWhichThat Alias to the Same Buffer. The requirement in Section 6.1.1 (above) mitigates this attack. A server buffer is exposed to only one client at a time to ensure that no information disclosure or information tampering occurs between peers. *5.3 - Network Based5.3, Network-Based Eavesdropping. Confidentiality services should be enabled by the ULP if this threat is a concern. * Denial of Service *6.4.3.16.4.3.1, Multiple Streams Sharing Receive Buffers. ULP memory footprint size can be important for some server ULPs. If a server ULP is expecting significant network traffic from multiple clients, using a receive buffer queue per Stream where there is a large number of Streams can consume substantial amounts of memory.ThusThus, a receive queue that can be shared by multiple Streams is attractive. However, because of the attacks outlined in this section, sharing a single receive queue between multiple clients must only be done if a mechanism is in place to ensure that one client cannot consume receive buffers in excess of its limits, as defined by each ULP. For multiple Streams within a single client ULP (which presumably shared Partial MutualTrust)Trust), this added overhead may be avoided. * 7.1 Local ULP Attacking a Shared CQ. The normative RNIC mitigations require that the RNICtonot enable sharing of a CQ if the local ULPs do not share Partial Mutual Trust.ThusThus, while the ULP is not allowed to enable this feature in an unsafe mode, if the two local ULPs share Partial Mutual Trust, they must behave in the following manner: 1) The sizing of the completion queue is based on the size of the receive queue and sendqueuesqueues, as documented in6.4.3.26.4.3.2, Remote or Local Peer Attacking a Shared CQ. 2) The local ULP ensures that CQ entries are reaped frequently enough to adhere to Section 6.4.3.2's rules. *6.4.3.26.4.3.2, Remote or Local Peer Attacking a Shared CQ. There are two mitigations specified in this section - one requires a worst-case size of the CQ, and can be implemented entirely within the Privileged Resource Manager. The second approach requires cooperation with the local ULP server(to not(not to post too many buffers), and enables a smaller CQ to be used. In some server environments, partial trust of the server ULP (but not the clients) isacceptable, thusacceptable; thus, the smaller CQ fully mitigates the remote attacker. In other environments, the local server ULP could also contain untrusted elementswhichthat can attack the local machine (or have bugs). In those environments, the worst-case size of the CQ must be used. *6.4.3.36.4.3.3, Attacking the RDMA Read Request Queue. The section requires a server's Privileged Resource Managertonot to allow sharing of RDMA Read Request Queues across multiple Streams that do not share Partial MutualTrust,Trust for a ULPwhichthat performs RDMA Read operations to server buffers. However, because the server ULP knowsbestwhich of its Streams best share Partial Mutual Trust, this requirement can be reflected back to the ULP. The ULP(i.e.(i.e., server)requirementrequirement, in thiscasecase, is that it MUST NOT allow RDMA Read Request Queues to be shared between ULPswhichthat do not have Partial Mutual Trust. *6.4.56.4.5, Remote Invalidate an STag Shared on Multiple Streams. This mitigation is already covered by Section 6.2.2 (above).12Appendix B: Summary of RNIC and ULP Implementation Requirements This appendix is informative. Below is a summary of implementation requirements for the RNIC: * 3 Trust and Resource Sharing * 5.4.5 Requirements for IPsec Encapsulation of DDP * 6.1.1 Using an STag on a Different Stream * 6.2.1 Buffer Overrun - RDMA Write or Read Response * 6.2.2 Modifying a BufferAfterafter Indication * 6.4.1 RNIC Resource Consumption * 6.4.3.1 Multiple Streams Sharing Receive Buffers * 6.4.3.2 Remote or Local Peer Attacking a Shared CQ * 6.4.3.3 Attacking the RDMA Read Request Queue * 6.4.6 Remote PeerattackingAttacking an UnsharedCQ.CQ * 6.5 Elevation of Privilege 39 * 7.1 Local ULP Attacking a Shared CQ * 7.3 Local ULP Attacking the PTT&and STag Mapping Below is a summary of implementation requirements for the ULP above the RNIC: * 5.3 Information Disclosure -Network BasedNetwork-Based Eavesdropping * 6.1.1 Using an STag on a Different Stream * 6.2.2 Modifying a BufferAfterafter Indication * 6.3.2 Using RDMA Read to Access Stale Data * 6.3.3 Accessing a BufferAfterafter the Transfer * 6.3.4 Accessing Unintended DataWithwith a Valid STag * 6.3.5 RDMA Read into an RDMA Write Buffer * 6.3.6 Using Multiple STagsWhichThat Alias to the Same Buffer * 6.4.5 Remote Invalidate an STag Shared on Multiple Streams13Appendix C: Partial Trust Taxonomy This appendix is informative. Partial Trust is defined as when one party is willing to assume that another party will refrain from a specific attack or set of attacks, the parties are said to be in a state of Partial Trust. Note that the partially trusted peer may attempt a different set of attacks. This may be appropriate for many ULPs where any adverse effects of the betrayal is easily confined and does not place other clients or ULPs at risk. The Trust Models described in this section have three primary distinguishing characteristics. The Trust Model refers to a local ULP and Remote Peer, which are intended to be the local and remote ULP instances communicating via RDMA/DDP. * Local Resource Sharing (yes/no) - When local resources are shared, they are shared across a grouping of RDMAP/DDP Streams. If local resources are not shared, the resources are dedicated on a per Stream basis. Resources are defined in Section2.2 -2.2, Resources. The advantage of not sharing resources between Streams is that it reduces the types of attacks that are possible. The disadvantage is that ULPs might run out of resources. * Local Partial Trust (yes/no) - Local Partial Trust is determined based on whether the local grouping of RDMAP/DDP Streams (which typically equates to one ULP or group of ULPs) mutually trust each othertonot to perform a specific set of attacks. * Remote Partial Trust (yes/no) - The Remote Partial Trust level is determined based on whether the local ULP of a specific RDMAP/DDP Stream partially trusts the Remote Peer of the Stream (see the definition of Partial Trust in Section11, Introduction). Not allofthe combinations of the trust characteristics are expected to be used by ULPs. This document specifically analyzes five ULP Trust Models that are expected to be in common use. The Trust Models are as follows: * NS-NT - Non-Shared Local Resources, no Local Trust, no RemoteTrust - typicallyTrust; typically, a server ULP that wants to run in the safest mode possible. All attack mitigations are in place to ensure robust operation. * NS-RT - Non-Shared Local Resources, no Local Trust, Remote PartialTrust - typicallyTrust; typically, a peer-to-peerULP, whichULP that has, by some method outside of the scope of this document, authenticated the Remote Peer. Note that unless some form of key based authentication is used on a per RDMA/DDP Stream basis, it may not be possiblebe possiblefor man-in-the-middle attacks to occur. * S-NT - Shared Local Resources, no Local Trust, no RemoteTrust - typicallyTrust; typically, a server ULP that runs in an untrusted environment where the amount of resources required is either too large or too dynamic to dedicate for each RDMAP/DDP Stream. * S-LT - Shared Local Resources, Local Partial Trust, no RemoteTrust - typicallyTrust; typically, aULP, whichULP that provides a session layer and uses multiple Streams, toprovideprovides additional throughput or fail-over capabilities. Allofthe Streams within the local ULP partially trust each other, but do not trust the Remote Peer. Thistrust modelTrust Model may be appropriate for embedded environments. * S-T - Shared Local Resources, Local Partial Trust, Remote PartialTrust - typicallyTrust; typically, a distributed application, such as a distributed database application oraHigh Performance Computer (HPC) application, which is intended to run on a cluster. Due to extreme resource and performance requirements, the application typically authenticates with all of its peers and then runs in a highly trusted environment. The application peers are all in a single application fault domain and depend on one another to be well-behaved when accessing data structures. If a trusted Remote Peer has an implementation defect that results in poor behavior, the entire application could be corrupted. Models NS-NT andS-NT aboveS-NT, above, are typical for Internet networking - neither the localULPsULP nor the Remote Peer is trusted.SometimesSometimes, optimizations can be done that enable sharing of Page Translation Tables across multiple localULPs, thusULPs; thus, Model S-LT can be advantageous. Model S-T is typically used when resource scaling across a large parallel ULP makes it infeasible to use any other model. Resource scaling issues can either be due to performance around scaling or because there simply are not enough resources. Model NS-RT is probably the least likely model to be used, but is presented for completeness.14 Author's Addresses James Pinkerton Microsoft Corporation One Microsoft Way Redmond, WA. 98052 USA Phone: +1 (425) 705-5442 Email: jpink@windows.microsoft.com Ellen Deleganes Intel Corporation MS JF5-355 2111 NE 25th Ave. Hillsboro, OR 97124 USA Phone: +1 (503) 712-4173 Email: ellen.m.deleganes@intel.com 15Acknowledgments Sara Bitan Microsoft CorporationEmail:EMail: sarab@microsoft.com Allyn Romanow Cisco Systems 170 W Tasman Drive San Jose, CA 95134 USA Phone: +1408 525 8836 Email:(408) 525-8836 EMail: allyn@cisco.com Catherine Meadows Naval Research Laboratory Code 5543 Washington, DC 20375Email:USA EMail: meadows@itd.nrl.navy.mil Patricia Thaler Agilent Technologies, Inc. 1101 Creekside Ridge Drive, #100 M/S-RG10 Roseville, CA 95678 USA Phone:+1-916-788-5662 email:+1 (916) 788-5662 EMail: pat_thaler@agilent.com James Livingston NEC Solutions (America), Inc. 7525 166th Ave. N.E., Suite D210 Redmond, WA 98052-7811 USA Phone: +1 (425) 897-2033Email:EMail: james.livingston@necsam.com John CarrierAdaptec,Cray Inc.691 S. Milpitas Blvd. Milpitas, CA 95035 USA411 First Avenue S, Suite 600 Seattle, WA 98104-2860 Phone:+1 (360) 378-8526 Email: john_carrier@adaptec.com206-701-2090 EMail: carrier@cray.com Caitlin Bestler Broadcom 49 Discovery Irvine, CA 92618Email:EMail: cait@asomi.com Bernard Aboba Microsoft Corporation One Microsoft Way USA Redmond,WA.WA 98052USAPhone: +1 (425) 706-6606Email:EMail: bernarda@windows.microsoft.com16Authors' Addresses James Pinkerton Microsoft Corporation One Microsoft Way Redmond, WA 98052 USA Phone: +1 (425) 705-5442 EMail: jpink@windows.microsoft.com Ellen Deleganes Self P.O. Box 9245 Brooks, OR 97305 Phone: (503) 642-3950 EMail: deleganes@yahoo.com Full Copyright Statement Copyright (C) TheInternet Society (2006).IETF Trust (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNETSOCIETYSOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society.