While the solution worked, Cisco decided FCoE was a better way to unify I/O.

The problem? An additional network to manage: IB. And new switches to learn how to manage.

The problem? New I/O protocols to support, such as IP over InfiniBand (IPoIB), SCSI RDMA Protocol (SRP), and iSCSI over RDMA (iSER).

The problem? New driver stacks to support those protocols. Drivers for Windows, drivers for Red Hat, drivers for SUSE, drivers for Solaris. By far, maintaining drivers is the bigger ongoing engineering effort for any InfiniBand MFIO provider. For this reason Mellanox, the provider all IB switching silicon, and most of the host channel adapter silicon, is exploring FCoIB, which encapsulates FC frames onto IB, rather than converting FC to SRP or iSER.

The problem? New upper-layer drivers to support the new base I/O drivers for things like multipathing, and failover. And making things like IP multicast work over IPoIB.

The problem? New certifications for drivers, from upper layer software like clustering, to storage systems like EMC.

FCoE eliminates this mishmash.

The connection from the host (CNA) to the access switch (Nexus) is Ethernet! There is no new network to manage.

And the protocol used for storage transport on FCoE is Fibre-Channel! There are no new drivers required.

It is Emulex or QLogic FC silicon, and the same Emulex and QLogic drivers work just as before.

It is IP over Ethernet, and multicast just works.

It is Fibre Channel, and unmodified multipathing software (i.e., EMC PowerPath) just works. From the disk drive to the host driver, there is no change to the underlying Fibre Channel frame.

The FC and Ethernet interfaces are unmodified to the host operating system. That means clustering, etc., just works.

There are still some certifications required, but these are just that, certifications no differnet than a new HBA or a new FC switch. Not the kind of work required to support new protocols and new drivers.

As someone who has configured and set up both InfiniBand based multi-fabric I/O and Nexus based FCoE, I can tell you FCoE is easily 10X easier and faster. Why? Because there are no new protocols to configure. No new drivers to worry about. And no new networks or switches to learn how to manage.

While the solution worked, Cisco decided FCoE was a better way to unify I/O.

The problem? An additional network to manage: IB. And new switches to learn how to manage.

The problem? New I/O protocols to support, such as IP over InfiniBand (IPoIB), SCSI RDMA Protocol (SRP), and iSCSI over RDMA (iSER).

The problem? New driver stacks to support those protocols. Drivers for Windows, drivers for Red Hat, drivers for SUSE, drivers for Solaris. By far, maintaining drivers is the bigger ongoing engineering effort for any InfiniBand MFIO provider. For this reason Mellanox, the provider all IB switching silicon, and most of the host channel adapter silicon, is exploring FCoIB, which encapsulates FC frames onto IB, rather than converting FC to SRP or iSER.

The problem? New upper-layer drivers to support the new base I/O drivers for things like multipathing, and failover. And making things like IP multicast work over IPoIB.

The problem? New certifications for drivers, from upper layer software like clustering, to storage systems like EMC.

FCoE eliminates this mishmash.

The connection from the host (CNA) to the access switch (Nexus) is Ethernet! There is no new network to manage.

And the protocol used for storage transport on FCoE is Fibre-Channel! There are no new drivers required.

It is Emulex or QLogic FC silicon, and the same Emulex and QLogic drivers work just as before.

It is IP over Ethernet, and multicast just works.

It is Fibre Channel, and unmodified multipathing software (i.e., EMC PowerPath) just works. From the disk drive to the host driver, there is no change to the underlying Fibre Channel frame.

The FC and Ethernet interfaces are unmodified to the host operating system. That means clustering, etc., just works.

There are still some certifications required, but these are just that, certifications no differnet than a new HBA or a new FC switch. Not the kind of work required to support new protocols and new drivers.

As someone who has configured and set up both InfiniBand based multi-fabric I/O and Nexus based FCoE, I can tell you FCoE is easily 10X easier and faster. Why? Because there are no new protocols to configure. No new drivers to worry about. And no new networks or switches to learn how to manage.

Re IB performance: Duh. However–CEE at 10Gb, and especially at 40 & 100Gb, should change that. The economics of that will be interesting to see.

Re the overhead of 3 separate fabrics: IMHO if you're already already dealing with 1/10GbE & FC, then it isn't particularly onerous to collapse the two into FCoE at the access layer. You're still managing GbE & FC. If the FCoE works as the standards in dev say it should, then the added management overhead of 1 (not 3) additional fabric that works in ways similar to the existing ones isn't going to be significantly more. Of course, it won't work the way it should–that's where you'll suffer.

Re moving IB out from HPC into the rest of the network: now that would be interesting.

Re IB performance: Duh. However–CEE at 10Gb, and especially at 40 & 100Gb, should change that. The economics of that will be interesting to see.

Re the overhead of 3 separate fabrics: IMHO if you're already already dealing1/10GbE & FC, then it isn't particularly onerous to collapse the two into FCoE at the access layer. You're still managing GbE & FC. If the FCoE works as the standards in dev say it should, then the added management overhead of 1 (not 3) additional fabric that works in ways similar to the existing ones isn't going to be significantly more. Of course, it won't work the way it should–that's where you'll suffer.

Re moving IB out from HPC into the rest of the network: now that would be interesting.