【Fabric】libfabric

【Fabric】libfabric

A local-area RDMA network is usually referred to as a fabric.

所谓Fabric，就是支持RDMA的局域网(LAN)。所以 libfabric '可以理解为局域网(LAN)（支持RDMA、socket……等协议）的编程库。

​​《For the RDMA novice: libfabric, libibverbs, InfiniBand, OFED, MOFED? 》​​

​​libibverbs '是IB接口的组件(Verbs API)。 ' libibverbs '之于IB，正如' sockets '之于TCP/IP。

IB接口的其他关键软件组件是  用户空间库 和  内核空间模块 ，它们实现了API并能够创建IB资源，这些都是Open Fabric Enterprise Distribution (OFED)来规定的。

OFED的用户空间库在rdma-core存储库中，内核组件在linux树的driver/infiniband 子系统中。

' libfabric '是另一个相当新的API，它打算提供比' libibverbs '更高的抽象级别：

除了InfiniBand，还有其他几个用户级网络接口。通常，它们是专有的和特定于供应商的。Cray有uGNI接口，Intel omnipath有PSM2, Cisco usNIC等。不同接口之间的底层概念(消息队列、完成队列、注册内存等)是相似的，但在功能和语义上有一定的差异。

Open Fabrics Interface (OFI) 打算通过提供一个抽象的API“libfabric”来统一所有可用的接口。然后，每个供应商将通过其“libfabric-provider”来支持OFI，后者将在自己的接口中调用相应的函数。这样，使用“libfabric”API编写的用户级网络应用程序就可以在不同的供应商之间移植。基于应用程序将运行的硬件，正确的libfabric

《​​Libfabric Programmer's Manual》​​

​​Interfaces (OFI)是一个专注于 向 应用程序提供fabric通信服务的框架。最好将OFI描述为提供（导出）fabric服务的 库和  应用程序的集合。

OFI的关键组件有:应用程序接口、provider库、内核服务、守护进程和测试应用程序。（application interfaces, provider libraries, kernel services, daemons, and test applications.）

原文1：

For the RDMA novice: libfabric, libibverbs, InfiniBand, OFED, MOFED?

​​For the RDMA novice: libfabric, libibverbs, InfiniBand, OFED, MOFED? — Rohit Zambre​​​  in ​​User-level Networking​​

Prior to entering the high-performance, user-level networking space in April 2017, I wasn't even aware of the existence of the field of high-performance interconnects. I barely knew the sockets interface. I had heard about the concepts of DMA and RDMA but nothing about RDMA programming (while RDMA is technically a subset of high-performance, user-level networking, the terms are commonly used interchangeably). At SC'16, I had heard about InfiniBand, but only at Mellanox Technology's booths. So my brain subconsciously associated the two. To the RDMA novices: I was wrong; to the RDMA experts: you can imagine the turmoil in my head when the first RDMA interface I was learning to use was the abstract `libfabric`.

While I could further digress from the topic of this post and write about my then confused state of mind, I will instead leave a list of links for those just getting started with the high-performance, user-level networking space:

If you don't know much about `sockets`, I suggest you quickly read about it and write a simple send/receive program using the `sockets` API to understand the implications of maintaining flow control of data transmission/reception in your code.​​OFI guide​​: all sections just before `OFI Architecture` to motivate the existence of this field.​​InfiniBand​​: this RedHat guide officially defines what InfiniBand is.

From what I have read so far, InfiniBand is the first official interface (building off of the ​​Virtual Interface Architecture​​) for high-performance, user-level networking. High-performance because of no kernel involvement (hence, user-level) for operations that involve transmission/reception of data, unlike TCP/IP. The kernel is involved only in the creation of resources used for issuing data transmission/reception. Additionally, unlike TCP/IP, the InfiniBand interface permits RDMA operations (remote reads, writes, atomics, etc.). The InfiniBand (IB) specification has both hardware and software components.

`libibverbs` is the software component (Verbs API) of the IB interface. As `sockets` is to TCP/IP, `libibverbs` is to IB. Your best bet to learn how to code with `libibverbs` is the amazing ​​RDMAmojo​​​ blog written by Dotan Barak, the creator of the man pages for `libibverbs`. You could even solely rely on his blog to learn about the InfiniBand concepts. He writes about the API in excruciating detail with very helpful FAQs. ​​Here's​​​ his big-picture tutorial-style presentation. Other critical software components of the IB interface are the user-space libraries and kernel-space modules that implement the API and enable IB resource creation. This is precisely what the Open Fabric Enterprise Distribution (OFED) is. OFED’ user-space libraries are in the ​​rdma-core​​​ repository and the kernel components are in ​​driver/infiniband​​ subsystem of the linux tree.

The hardware component of IB is where different vendors come into play. The IB interface is abstract; hence, multiple vendors can have different implementations of the IB specification. Mellanox Technologies has been an active, prominent InfiniBand hardware vendor. In addition to meeting the IB hardware specifications in the NIC design, the vendors have to support the `libibverbs` API by providing a user-space driver and a kernel-space driver that actually do the work (of setting up resources on the NIC) when a `libibverbs` function such as `ibv_open_device` is called. These vendor-specific libraries and kernel modules are a standard part of the OFED. The vendor-specific user-space libraries are called ​​providers​​ in rdma-core. These providers span both the IB and other technologies, such as RoCE and iWARP, that implement RDMA over Ethernet adapters (I’ll delve into the convergence between IB and Ethernet in another post). Mellanox OFED (MOFED) is Mellanox's implementation of the OFED libraries and kernel modules. MOFED contains certain optimizations that are targeted towards Mellanox hardware (the mlx4 and mlx5 providers) but haven't been incorporated into OFED yet.

`libfabric` is another, fairly recent API and intends to serve a level of abstraction higher than that of `libibverbs`. Alongside InfiniBand, several other user-level networking interfaces exist. Typically they are proprietary and vendor-specific. Cray has the uGNI interface, Intel Omni-Path has PSM2, Cisco usNIC, etc. The underlying concepts (message queues, completion queues, registered memory, etc.) between the different interfaces are similar with certain differences in capabilities and semantics. The Open Fabrics Interface (OFI) intends to unify all of the available interfaces by providing an abstract API: `libfabric`. Each vendor will then support the OFI through its `libfabric-provider` that will call corresponding functions in its own interface. This way, a user-level networking application written using the `libfabric` API is portable across different vendors. Based on the hardware that the application will be running on, the right `libfabric-provider` can then be selected.

Hope that helps you with an introduction to high-performance, user-level networking. If not, hope it gives you enough search keywords to use in your favorite search engine. All the best!

If you are interested to learn how transmitting a message works under the hood, check out my newer blog post: ​​How are messages transmitted on InfiniBand?​​

Tags: ​​InfiniBand​​​, ​​libibverbs​​​, ​​libfabric​​​, ​​What is InfiniBand​​​, ​​What is OFED​​​, ​​Difference between OFED and MOFED​​​, ​​What is libfabric​​

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。