had a really interesting experience recently. We are running a VMware environment with Broadcom 57800 nic's with iSCSI hardware offload, but were seeing really high datastore latency in VMware even though the actual storage was reporting normal latency values.
At 1st we though it was the switches, but after several calls with VMware and then the storage vendor we decided to have a play with the NIC's in the servers. 1st thing was to go from the Hardware acceleration on the Broadcom 57800 NIC's to VMware software iSCSI initiator.
Just making this change we went from (using iometer) 10mbps to 21 mbps and 51 ms average latency to 24 ms average latency (100% write work load) and fro, 265 mbps to 402mbps and 1.98ms latency to 1.3 ms latency (100% read workload)! This is on a 10GB Cat 6 network all layer 2, server -> switch -> SAN. This is something we were not expecting as you would think that hardware would be faster than software.
The next change was to swap the NIC's completely to Intel but still using the software VMware initiator. Again we saw a massive improvement with the Broadcom hardware delivering around 33,000 IOPS, Broadcom with software imitator just short of 50,000 IOPS and with Intel NIC but VMware software initiator delivering over 60,000 and peaked at nearly 70,000 IOPS.
This is a staggering result with us improving the performance of our SAN nearly 100% by simply swapping the host network adaptors from Broadcom to Intel. This will defiantly be the last time I use Broadcom as a NIC and shows that even though Intel's are a little bit more expensive, they are certainly worth the extra cash.
Hope this helps
Test done using ESX 5.5 enterprise plus, Dell R720 servers, Juniper 4550 10gb Base-T switches and Tegile Hybrid SAN