Search This Blog

Monday, 13 July 2015

Broadcom v's Intel

Hi all,

had a really interesting experience recently. We are running a VMware environment with Broadcom 57800 nic's with iSCSI hardware offload, but were seeing really high datastore latency in VMware even though the actual storage was reporting normal latency values.

At 1st we though it was the switches, but after several calls with VMware and then the storage vendor we decided to have a play with the NIC's in the servers. 1st thing was to go from the Hardware acceleration on the Broadcom 57800 NIC's to VMware software iSCSI initiator.

Just making this change we went from (using iometer) 10mbps to 21 mbps and 51 ms average latency to 24 ms average latency (100% write work load) and fro, 265 mbps to 402mbps and 1.98ms latency to 1.3 ms latency (100% read workload)! This is on a 10GB Cat 6 network all layer 2, server -> switch -> SAN. This is something we were not expecting as you would think that hardware would be faster than software.

The next change was to swap the NIC's completely to Intel but still using the software VMware initiator. Again we saw a massive improvement with the Broadcom hardware delivering around 33,000 IOPS, Broadcom with software imitator just short of 50,000 IOPS and with Intel NIC but VMware software initiator delivering over 60,000 and peaked at nearly 70,000 IOPS.

This is a staggering result with us improving the performance of our SAN nearly 100% by simply swapping the host network adaptors from Broadcom to Intel. This will defiantly be the last time I use Broadcom as a NIC and shows that even though Intel's are a little bit more expensive, they are certainly worth the extra cash.

Hope this helps


Test done using ESX 5.5 enterprise plus, Dell R720 servers, Juniper 4550 10gb Base-T switches and Tegile Hybrid SAN

Wednesday, 24 June 2015

What is your ESXi storage doing?

Currently working on a cloud platform for an ISP and found a great command line tool for your ESXi hosts to see how your storage is performing. SSH into one of your servers and run ESXTOP and press the D key.
This shows usful information like how long a storage command has spent in the kernel v's the amount of time that it has spent going to the device.

See link below for further information:

Thursday, 20 February 2014

Installing vCenter 5.5 Client on Windows 2012 R2

Hi all,

I had to install vCenter Client on a Windows 2012 R2 server for a customer and what a pain it turns out to be. If you just run the installer you get an error message saying "Internal Error 28173. -2146498298" which in plain English means that it doesn't have access to the Microsoft .Net framework 3.5 as this isn't installed as standard on a windows 2012 R2 box.

So the next step is to install .Net 3.5; you go about this by adding a feature. .Net 3.5 is one of the 1st items on this list, but if you just tick it and next next next all the way through the install, the install will fail. You need the windows installation DVD to actually install this and on the last option for the install at the bottom of the screen there is an option "specify an alternative source path", click this and enter X:\sources\sxs (where X is there letter of your DVD drive). When you next through the remaining sections this will now install correctly.

Talk about making something that should be a 2 minute job completly overly complicated!

Tuesday, 6 August 2013

Cisco VIC 1225 with VMware | Redundant Networking Problem

Hi all,

I have been installing some Cisco C220 servers with the VIC 1225 CNA (converged network adapter) for a local company. As part of the installation these servers were connected to a pair of Nexus 5K's with the CNA used for both fiber channel and 10gbps ethernet for VMware ESXi 5.1.

The connectivity was pretty standard to any installation of this type with one 10 gig link going to one switch and the other going to the second switch, exactly like if this was a 1 gig implementation with catalyst switches. So i installed VMware and as part of the default installation vmware takes the first network card and uses that as its management interface and also uses the MAC address of that NIC for the MAC address of the management console. FINE! But that will come back and trouble me later on.

So i give the management interface a static IP address and connect using the vSphere client and all's going well until the point where i add the second NIC, and all connectivity to the host was lost!  So i connect the CIMC and run the test network connectivity and everything fails. So i remove one of the NIC's and everything comes back up again. So after lots and lots of testing and nothing working when everything looked correct, we resulted in logging a support call with Cisco.

It turns out that there is currently a fault on the VIC 1225 where by, even though all the vNIC's are in promiscuous mode, if one port see's the MAC address of another vNIC it doesn't forward the packet. I have been told that there is a fix on the way and should be available in September.

So the solution, if you get this as a problem is to delete the default VMware management interface and create a brand new one, this is given a VMWare MAC address stating in 00 and is therefore not a MAC address that appears on the VIC. Add your two network to this new management interface and everything works as you would expect.

Thanks for reading and i hope this was of use to someone.

Monday, 13 May 2013

Installing ESXi onto new Dell Servers

Hi all,

just a quick post, i've recently had to install ESXi onto some new Dell server (R720 and the 1u equivalent) for some clients. I normally download the latest release from VMware and burn it to CD before i go to site to save some time, but this has caught me out with the new Dell Servers.

What you will find is that when the ESXi disk starts to boot you get an error message stating that there are no network cards in the server (even though i had on this occasion 2x quad port broadcom NIC's). What you need to do is go to Dell's website, enter in the server tag for the server you have purchased and got to the enterprise solutions part of the support and drivers section. There you can download a Dell customized version of ESXi which includes all the dell drivers you need to successfully install ESXi

Hope this helps and thanks for reading.


Thursday, 7 February 2013

Increase VMware converter performance

Hi All,

I was involved with a P2V migration last week and while the number of servers to migrate were small the amount of data held on these servers were around 250gb - 500gb in size. This conversion was to a vSphere 5.1 infrastructure using the VMware standalone converter running in a VM. When i started the migration off, we were experiencing terrible transfer rates in the region of 2MB/s and it was saying well over 8 hours of the P2V to complete per server.

After a little bit of googling i found the following article on VMwares communities forum which says that from converter 5.0 and onward, the transfer of data has been encrypted by default and that this could slow down the transfer of data.(

I stopped the job and turned off the encryption by editing the converter-worker.xml file and started the job again. The transfer rate, once stabilized went up to around 25MB/s and drastically reduced the amount of time the P2V took!

If you are experiencing slow P2V rate this might be something worth trying.

kind regards and thanks for reading


Tuesday, 4 December 2012

Upgrade to my Cisco Lab

Hi everyone,

I'm spending a lot of spare time working towards extra Cisco certification, in particular the CCIE routing and switching. No matter what exams i take, i always seem to find something enjoyable about seeing networks converge and routers relearning different routes around a network after a failure.

So i've invested in some more kit (off ebay) for my home lab to give me some more flexibility for different Cisco certifications.

My 1/2 rack now looks like this:
The kit list is as follows (top to bottom)
Cisco 1603 router (spare, as no power supply)
GTA Gnatbox 1000 firewall (Uses RIP v1 and V2 for dynamic routing. Can be a BB router for labs or, when i purchase an access server - segmentation for potentially renting this out for other Cisco students)
Cisco 1601 router (used as BB router for BGP/ospf/eigrp etc)
Cisco 861 router (used as BB router for RIP - access to home LAN via NAT)
Cisco 3524 switch
Cisco 2950 switch
Cisco 1841 router (R4 in lab 1x WIC 1T, 2xFE)
Cisco 2621 router (R5 in lab 2x WIC 1T, 2xFE)
Cisco 2621 router (R7 in lab 1x WIC 1T, 1x ISDN WIC, 2xFE)
Cisco 3640 router (Frame Relay Switch. 1x 4 port asyn serial , 1x 4 port sync serial, 1x 10base-t for management)
Cisco 2801 router (R6 in lab 1x WIC 1T, 2xFE)
Cisco 2501 router (R8 in lab 2x serial, 1x ethernet)
Cisco 2611 router (R3 in lab 1x WIC 1T, 1xFE)
Cisco 2501 router (R2 in lab 2x serial, 1x ethernet)
Cisco 1760 router (R1 in lab 1x WIC 1T, 1x vwic 1mft-g703)
Cisco 3550 switch (SW1 in lab with EMI IOS) 

Other than the addition of another 3550, i'm really happy with this setup.  The frame relay switch is connected to every Rx router in the rack, which prevents me from re-cabling, all i have to do is change the DLCI which the router is using, giving me endless possibilities for a topology. Every router is connected to their correspending switch port on SW1 (R1 - FE0/1, R2 - FE0/2 etc) and if they have multiple ethernet interfaces, the 2nd interface goes to SW2 with SW1 and SW2 being ether channeled together.

 Also i run an old desktop PC with VMware ESXi free edition into one of the switches using a trunk port which allows me to put pc clients on different networks and look at the network traffic passing through any topology which i create on here.

For anyone that wants it. Here is the config of the ports on the frame relay switch.

interface Serial0/2
 no ip address
 encapsulation frame-relay
 clockrate 128000
 frame-relay intf-type dce
 frame-relay route 201 interface Serial0/1 102
 frame-relay route 203 interface Serial0/3 302
 frame-relay route 204 interface Serial1/0 402
 frame-relay route 205 interface Serial1/1 502
 frame-relay route 206 interface Serial1/2 602
 frame-relay route 207 interface Serial1/3 702
 no shutdown
interface Serial0/3
 no ip address
 encapsulation frame-relay
 clockrate 128000
 frame-relay intf-type dce
 frame-relay route 302 interface Serial0/2 203
 frame-relay route 301 interface Serial0/1 103
 frame-relay route 304 interface Serial1/0 403
 frame-relay route 305 interface Serial1/1 503
 frame-relay route 306 interface Serial1/2 603
 frame-relay route 307 interface Serial1/3 703
 no shutdown
interface Serial1/0
 no ip address
 encapsulation frame-relay
 clockrate 128000
 frame-relay intf-type dce
 frame-relay route 402 interface Serial0/2 204
 frame-relay route 403 interface Serial0/3 304
 frame-relay route 401 interface Serial0/1 104
 frame-relay route 405 interface Serial1/1 504
 frame-relay route 406 interface Serial1/2 604
 frame-relay route 407 interface Serial1/3 704
 no shutdown
interface Serial1/1
 no ip address
 encapsulation frame-relay
 clockrate 128000
 frame-relay intf-type dce
 frame-relay route 502 interface Serial0/2 205
 frame-relay route 503 interface Serial0/3 305
 frame-relay route 504 interface Serial1/0 405
 frame-relay route 501 interface Serial0/1 105
 frame-relay route 506 interface Serial1/2 605
 frame-relay route 507 interface Serial1/3 705
 no shutdown
interface Serial1/2
 no ip address
 encapsulation frame-relay
 clockrate 128000
 frame-relay intf-type dce
 frame-relay route 602 interface Serial0/2 206
 frame-relay route 603 interface Serial0/3 306
 frame-relay route 604 interface Serial1/0 406
 frame-relay route 605 interface Serial1/1 506
 frame-relay route 601 interface Serial0/1 106
 frame-relay route 607 interface Serial1/3 706
 no shutdown
interface Serial1/3
 no ip address
 encapsulation frame-relay
 clockrate 128000
 frame-relay intf-type dce
 frame-relay route 702 interface Serial0/2 207
 frame-relay route 703 interface Serial0/3 307
 frame-relay route 704 interface Serial1/0 407
 frame-relay route 705 interface Serial1/1 507
 frame-relay route 706 interface Serial1/2 607
 frame-relay route 701 interface Serial0/1 107
 no shutdown