Hyperconverged isn't a magic bullet (what the vendors woan't tell you)

Hi all,

and thank you for reading my blog. In recent years there has been a massive shift from a traditional three tried architecture (SAN Switching and Compute) to a Hyperconvered model (Storage and compute in the same tin). There are many attractions to this, some of the main "Sells" being that when you add compute and add storage, you add IOPS when you do this and the platform scales with you; It also saves on data centre space with the nodes typically being 2U or 4U without the need for a SAN head and disk shelves.

This all seems too good to be true and as a whole its a really sensible model. However, there are some major pitfalls in this design which means some workloads do not perform well on them, and not only that they would typically perform much worse than the SAN you have just replaced.

So what's the problem you ask?

Well let's take a traditional SAN from a vendor; This would have some spinning disks either in RAID 5 or 6 and most of the large vendors would accelerate this with a bank of SSD's for frequently accessed data (something like EMC Fast Cache for example). On a hyper-converged platform, for example, Nutanix, you have 4 JBOD drives with 2 SSD cache disks for frequently access data and typically this is furthermore cached in RAM. For that frequent access data, the Hyperconverege tin would have a lower latency in reading that data (assuming the VM is on the same piece of tin as where the data is) than a traditional SAN which would have to traverse some switching, either fibre channel or TCP/IP with iSCSI.

But what about cold data, and not just cold data; data the is also sequentially read... and here lies the problem. Cold data is absolutely on the spinning disks and a lot of the hyper-converged vendors us an algorithm for data reads which when it sees a sequential read, typically will read that from the spinning disks also as its quicker than reading from SSD (SSD's work best for random reads).

Knowing this, if you think about the architecture of those spinnings disks we described at the start we have a SAN with RAID 5 or RAID 6 which the more spindles you add the faster it gets or the hyper-converged platform which is JBOD (single disks). The fastest a Hyperconverged platform can read cold data or sequential reads is as fast as a single disk can read that data (100 - 140 IOPS) whereas a traditional SAN could be 10x, 20x etc faster depending on the number of spindles used to create that RAID set.

If that disk is also reading or writing data for another VM at the same time, then due to the head moving across that platter of the disk performance could be a lot worse then the 100 - 140 IOPS as well (try running multiple VM's on your desktop PC for example).

So, while hyper-converged provides a great platform for a lot of companies, if your workload is, for example, replicating data between VM's, running data warehousing reports on an infrequent basis or something similar then hyper-converged might not be the right decision for you.

Also just as a final thought. You could absolutely use hyper-converged as you compute platform but also complement this with a traditional SAN for those workloads that don't fit well on them.

Broadcom v's Intel

Hi all,

had a really interesting experience recently. We are running a VMware environment with Broadcom 57800 nic's with iSCSI hardware offload, but were seeing really high datastore latency in VMware even though the actual storage was reporting normal latency values.

At 1st we though it was the switches, but after several calls with VMware and then the storage vendor we decided to have a play with the NIC's in the servers. 1st thing was to go from the Hardware acceleration on the Broadcom 57800 NIC's to VMware software iSCSI initiator.

Just making this change we went from (using iometer) 10mbps to 21 mbps and 51 ms average latency to 24 ms average latency (100% write work load) and fro, 265 mbps to 402mbps and 1.98ms latency to 1.3 ms latency (100% read workload)! This is on a 10GB Cat 6 network all layer 2, server -> switch -> SAN. This is something we were not expecting as you would think that hardware would be faster than software.

The next change was to swap the NIC's completely to Intel but still using the software VMware initiator. Again we saw a massive improvement with the Broadcom hardware delivering around 33,000 IOPS, Broadcom with software imitator just short of 50,000 IOPS and with Intel NIC but VMware software initiator delivering over 60,000 and peaked at nearly 70,000 IOPS.

This is a staggering result with us improving the performance of our SAN nearly 100% by simply swapping the host network adaptors from Broadcom to Intel. This will defiantly be the last time I use Broadcom as a NIC and shows that even though Intel's are a little bit more expensive, they are certainly worth the extra cash.

Hope this helps

Andy

Test done using ESX 5.5 enterprise plus, Dell R720 servers, Juniper 4550 10gb Base-T switches and Tegile Hybrid SAN

What is your ESXi storage doing?

Currently working on a cloud platform for an ISP and found a great command line tool for your ESXi hosts to see how your storage is performing. SSH into one of your servers and run ESXTOP and press the D key.
This shows usful information like how long a storage command has spent in the kernel v's the amount of time that it has spent going to the device.


See link below for further information:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008205

Installing vCenter 5.5 Client on Windows 2012 R2

Hi all,

I had to install vCenter Client on a Windows 2012 R2 server for a customer and what a pain it turns out to be. If you just run the installer you get an error message saying "Internal Error 28173. -2146498298" which in plain English means that it doesn't have access to the Microsoft .Net framework 3.5 as this isn't installed as standard on a windows 2012 R2 box.

So the next step is to install .Net 3.5; you go about this by adding a feature. .Net 3.5 is one of the 1st items on this list, but if you just tick it and next next next all the way through the install, the install will fail. You need the windows installation DVD to actually install this and on the last option for the install at the bottom of the screen there is an option "specify an alternative source path", click this and enter X:\sources\sxs (where X is there letter of your DVD drive). When you next through the remaining sections this will now install correctly.

Talk about making something that should be a 2 minute job completly overly complicated!

Cisco VIC 1225 with VMware | Redundant Networking Problem

Hi all,

I have been installing some Cisco C220 servers with the VIC 1225 CNA (converged network adapter) for a local company. As part of the installation these servers were connected to a pair of Nexus 5K's with the CNA used for both fiber channel and 10gbps ethernet for VMware ESXi 5.1.

The connectivity was pretty standard to any installation of this type with one 10 gig link going to one switch and the other going to the second switch, exactly like if this was a 1 gig implementation with catalyst switches. So i installed VMware and as part of the default installation vmware takes the first network card and uses that as its management interface and also uses the MAC address of that NIC for the MAC address of the management console. FINE! But that will come back and trouble me later on.

So i give the management interface a static IP address and connect using the vSphere client and all's going well until the point where i add the second NIC, and all connectivity to the host was lost!  So i connect the CIMC and run the test network connectivity and everything fails. So i remove one of the NIC's and everything comes back up again. So after lots and lots of testing and nothing working when everything looked correct, we resulted in logging a support call with Cisco.

It turns out that there is currently a fault on the VIC 1225 where by, even though all the vNIC's are in promiscuous mode, if one port see's the MAC address of another vNIC it doesn't forward the packet. I have been told that there is a fix on the way and should be available in September.

So the solution, if you get this as a problem is to delete the default VMware management interface and create a brand new one, this is given a VMWare MAC address stating in 00 and is therefore not a MAC address that appears on the VIC. Add your two network to this new management interface and everything works as you would expect.

Thanks for reading and i hope this was of use to someone.

Installing ESXi onto new Dell Servers

Hi all,

just a quick post, i've recently had to install ESXi onto some new Dell server (R720 and the 1u equivalent) for some clients. I normally download the latest release from VMware and burn it to CD before i go to site to save some time, but this has caught me out with the new Dell Servers.

What you will find is that when the ESXi disk starts to boot you get an error message stating that there are no network cards in the server (even though i had on this occasion 2x quad port broadcom NIC's). What you need to do is go to Dell's website, enter in the server tag for the server you have purchased and got to the enterprise solutions part of the support and drivers section. There you can download a Dell customized version of ESXi which includes all the dell drivers you need to successfully install ESXi

Hope this helps and thanks for reading.

Andy

Increase VMware converter performance

Hi All,

I was involved with a P2V migration last week and while the number of servers to migrate were small the amount of data held on these servers were around 250gb - 500gb in size. This conversion was to a vSphere 5.1 infrastructure using the VMware standalone converter running in a VM. When i started the migration off, we were experiencing terrible transfer rates in the region of 2MB/s and it was saying well over 8 hours of the P2V to complete per server.

After a little bit of googling i found the following article on VMwares communities forum which says that from converter 5.0 and onward, the transfer of data has been encrypted by default and that this could slow down the transfer of data.(http://communities.vmware.com/message/1866091).

I stopped the job and turned off the encryption by editing the converter-worker.xml file and started the job again. The transfer rate, once stabilized went up to around 25MB/s and drastically reduced the amount of time the P2V took!

If you are experiencing slow P2V rate this might be something worth trying.

kind regards and thanks for reading

Andy

Upgrade to my Cisco Lab

Hi everyone,

I'm spending a lot of spare time working towards extra Cisco certification, in particular the CCIE routing and switching. No matter what exams i take, i always seem to find something enjoyable about seeing networks converge and routers relearning different routes around a network after a failure.

So i've invested in some more kit (off ebay) for my home lab to give me some more flexibility for different Cisco certifications.

My 1/2 rack now looks like this:

 

The kit list is as follows (top to bottom)

Cisco 1603 router (spare, as no power supply)

GTA Gnatbox 1000 firewall (Uses RIP v1 and V2 for dynamic routing. Can be a BB router for labs or, when i purchase an access server - segmentation for potentially renting this out for other Cisco students)

Cisco 1601 router (used as BB router for BGP/ospf/eigrp etc)

Cisco 861 router (used as BB router for RIP - access to home LAN via NAT)

Cisco 3524 switch

Cisco 2950 switch

Cisco 1841 router (R4 in lab 1x WIC 1T, 2xFE)

Cisco 2621 router (R5 in lab 2x WIC 1T, 2xFE)

Cisco 2621 router (R7 in lab 1x WIC 1T, 1x ISDN WIC, 2xFE)

Cisco 3640 router (Frame Relay Switch. 1x 4 port asyn serial , 1x 4 port sync serial, 1x 10base-t for management)

Cisco 2801 router (R6 in lab 1x WIC 1T, 2xFE)

Cisco 2501 router (R8 in lab 2x serial, 1x ethernet)

Cisco 2611 router (R3 in lab 1x WIC 1T, 1xFE)

Cisco 2501 router (R2 in lab 2x serial, 1x ethernet)

Cisco 1760 router (R1 in lab 1x WIC 1T, 1x vwic 1mft-g703)

Cisco 3550 switch (SW1 in lab with EMI IOS) 

Other than the addition of another 3550, i'm really happy with this setup.  The frame relay switch is connected to every Rx router in the rack, which prevents me from re-cabling, all i have to do is change the DLCI which the router is using, giving me endless possibilities for a topology. Every router is connected to their correspending switch port on SW1 (R1 - FE0/1, R2 - FE0/2 etc) and if they have multiple ethernet interfaces, the 2nd interface goes to SW2 with SW1 and SW2 being ether channeled together.

 Also i run an old desktop PC with VMware ESXi free edition into one of the switches using a trunk port which allows me to put pc clients on different networks and look at the network traffic passing through any topology which i create on here.

For anyone that wants it. Here is the config of the ports on the frame relay switch.

!

interface Serial0/2

 no ip address

 encapsulation frame-relay

 clockrate 128000

 frame-relay intf-type dce

 frame-relay route 201 interface Serial0/1 102

 frame-relay route 203 interface Serial0/3 302

 frame-relay route 204 interface Serial1/0 402

 frame-relay route 205 interface Serial1/1 502

 frame-relay route 206 interface Serial1/2 602

 frame-relay route 207 interface Serial1/3 702

 no shutdown

!

interface Serial0/3

 no ip address

 encapsulation frame-relay

 clockrate 128000

 frame-relay intf-type dce

 frame-relay route 302 interface Serial0/2 203

 frame-relay route 301 interface Serial0/1 103

 frame-relay route 304 interface Serial1/0 403

 frame-relay route 305 interface Serial1/1 503

 frame-relay route 306 interface Serial1/2 603

 frame-relay route 307 interface Serial1/3 703

 no shutdown

!

interface Serial1/0

 no ip address

 encapsulation frame-relay

 clockrate 128000

 frame-relay intf-type dce

 frame-relay route 402 interface Serial0/2 204

 frame-relay route 403 interface Serial0/3 304

 frame-relay route 401 interface Serial0/1 104

 frame-relay route 405 interface Serial1/1 504

 frame-relay route 406 interface Serial1/2 604

 frame-relay route 407 interface Serial1/3 704

 no shutdown

!

interface Serial1/1

 no ip address

 encapsulation frame-relay

 clockrate 128000

 frame-relay intf-type dce

 frame-relay route 502 interface Serial0/2 205

 frame-relay route 503 interface Serial0/3 305

 frame-relay route 504 interface Serial1/0 405

 frame-relay route 501 interface Serial0/1 105

 frame-relay route 506 interface Serial1/2 605

 frame-relay route 507 interface Serial1/3 705

 no shutdown

!

interface Serial1/2

 no ip address

 encapsulation frame-relay

 clockrate 128000

 frame-relay intf-type dce

 frame-relay route 602 interface Serial0/2 206

 frame-relay route 603 interface Serial0/3 306

 frame-relay route 604 interface Serial1/0 406

 frame-relay route 605 interface Serial1/1 506

 frame-relay route 601 interface Serial0/1 106

 frame-relay route 607 interface Serial1/3 706

 no shutdown

!

interface Serial1/3

 no ip address

 encapsulation frame-relay

 clockrate 128000

 frame-relay intf-type dce

 frame-relay route 702 interface Serial0/2 207

 frame-relay route 703 interface Serial0/3 307

 frame-relay route 704 interface Serial1/0 407

 frame-relay route 705 interface Serial1/1 507

 frame-relay route 706 interface Serial1/2 607

 frame-relay route 701 interface Serial0/1 107

 no shutdown

!

Setting up a new DHCP server

Hi all,

I was setting up some new domain controlers with DHCP and found that the customer had a lot of reservations. Rather than having to retype everything i found this great link to export the scope and import it back into the new server. I tested it from 2003 to 2008 and it worked perfectly.

http://koolbeans.wordpress.com/2007/07/31/howto-import-and-export-dhcp-reservations-in-server-2003/

The dangers of VMware snapshots.

I come across a lot of VMware environments where people have been miss-informed about the use of VMware snapshots and can later on, have a large detrimental effect on storage and performance of their live environments.

First of all, let’s define what a snapshot is. Wikipedia states it to be “Snapshot (computer storage), a set of computer files and directories kept in storage as they were sometime in the past”, or as a lot of people say “a point in time copy”. 

The above has lead people to believe the VMware snapshots can be used as backups or that it’s fine to leave several snapshots on a VM for its entire life, but unfortunately this is NOT how they function. 

VMware snapshots are Delta’s not a true snapshot, when initiated the original virtual disk (vmdk) is locked and made read only and a new delta disk is created which all future changes are made. If you were to then snapshot this again, the original disk is still locked, the 1^st Delta disk is made read only and a 2^nd delta disk is created which all changes are written to.

The 2^nd delta disk is dependent on the 1^st delta disk and the 1^st delta disk is dependent on the original virtual disk of the VM and the more times you snapshot the VM, the more the dependency tree expands.

I have seen cases where by the original disk has been provisioned of 60GB on storage and then a further six 60GB delta drives had been created (all of various sizes on the storage) going back several years, all adding a massive overhead on storage. This also highlights why you cannot use VMware snapshots as backups due to the Delta tree dependency and that they are NOT point in time copies.

There is a well-known case where someone was miss-informed that VMware snapshots can be used as a backup and proceeded to snapshot their main mail system. Several weeks later their storage was reporting that it was nearly out of space and performance was really slow. They hired a VMware expert to investigate who found a 800gb Delta file and due to the amount of data that would be lost had no other options but to commit this snapshot to the VM, which subsequently took nearly a week to roll into the original disk and had a major impact on system performance.

Please use VMware snapshot responsibly; if you need to test a patch, clone the VM, put it on an internal test network in VMware (no physical NIC), patch the clone and test. If the test is successful you can then snapshot the original machine out of hours and patch knowing with confidence that the patch works with your application; after the install commit the snapshot immediately. The only reason I add the snapshot to patching the original VM is if something happens during the install nothing else. Other than this and its use with backup technologies (like backup exec, etc) which snapshot the VM to take a backup and then immediately commit the changes, there is no real reason they should be used.

Thanks for reading, and any comments or questions please feel free to ask.

Cisco UCS / Nexus

I'm just currently running through the Cisco DCUCI certification process, and are really impressed with concepts and business benefits for the high server farm / data centre operations user. Consolidating down the number of connections per rack, saving power on the switch and SAN elements, allowing full data centres to utilise this extra power and space to add extra compute power without the need to expand the physical building or relocate into another building which can be extremely costly.

Check out the Cisco Nexus platform and their UCS (not unified communications server :-)! ) platform.

VNXe NDMP backup Issue

Hi everyone,

I’ve been working with a client installing a VNXe for them along with VMware with Veeam backing up the VMware and Backup exec backing up the file shares on the VNXe and archiving backups to tape.

The setup is a pretty standard one with cisco switches and HP servers, but we ran into a problem when backing up the file shares on the VNXe (using NDMP) which backup exec, where the Backup Exec server would randomly skip a share or two one evening, and then skip a completely different one the following evening. The error that we were getting back was that the VNXe either could not be contacted or that a connection was refused.

The setup of the server was a HP network team in active / active (giving 2 Gbps) and two ports teamed on the VNXe for CIFS. The issues was raised with EMC, and it transpires, that this problem is caused by the HP network team and as soon as this was changed to NFT (active / passive), then the problem went away.

If I get the opportunity, I would like to test this with an Intel network team so see if it is vendor specific, but certainly something that you all might encounter.

Thanks for reading.

vCenter Appliance extra NIC issue

Currently I’m working on a customer install where we are using the new vsphere 5.0 vcenter server appliance. As we were migrating an existing 3.5 environment to a new platform, we had to keep adding and removing servers in and out of vCenter due to the limitations of the appliance.

On one occasion we had to remove the new vCenter appliance from the inventory of one server and add it to the inventory of another. When this happened, the appliance found added a new NIC (eth1) and would not function as it said eth0 was not connected and the appliance is only configured to used eth0.

If you ever get this error, there is a simple fix.

Log onto the vCenter appliance (via the console) and log in as root, for the vCenter appliance the username is root and the password VMware. Edit the file (using vi) /etc/udev/rules.d/70-net-persistent-names.rules.

This file shows all the network interfaces and their associated MAC addresses. Simply remove the entry for eth0 and edit the line for eth1, renaming it to eth0. Save the file with the command :wq and restart the server using shutdown –r now.

Backing up iSCSI targets

One thing you will more than likly need to day when implementing a SAN is backup the content of the iSCSI targets. If you implement something like backup exec on windows, then windows will try and Automount the volumes which will then cause you no end of problems.

To prevent this, before connecting to the iSCSI targets, run the following from the command prompt to prevent this:

diskpart

automount disable

automount scrub

Network IP Error after P2V

Hi All,

Just a quick update to my Blog. For those of you that have VMware and have run Converter on a physical server that has had a static IP address before the Converter was run, you may get an error message saying that the IP Address is already in use when you try and assign the new virtual network adapter with the old IP address.

I personally create a batchfile which looks like this

"set devmgr_show_nonpresent_devices=1
DEVMGMT.MSC"

I then run this on the new virtual machine. This will enable non present devices to be shown in device manager, and then it will open device manager. If you then click View and Show hidden devices, this will show all the old hardware that was present when it was a physical server. Simply right click the old network adapter and uninstall it and this will remove the offending static IP address.

Hope this helps.

Andy

Installation of New EMC VNXe 3300 - Part 2

Following on from setting up the iSCSI connectivity i needed to create some CIFS shares to replace a windows 2003 file server. Connectivity from the VNXe to the switch infrastructure was very simple due to how the VNXe fails over and fails back.

We currently have a pair of Cisco 3750's in a stack for the core connectivty of the data network so it was very simple to take a single RJ45 from SPA into one of the 3750's and a single RJ45 cable from SPB into the other 3750. Again as described in part 1, if the VNXe detects a cable fault or switch failure it willl fail over only the services on that network port affected to the other controler and back again once connectivty is resumed.

To increase network performance the CIFS service was configure on SPB as the primary connection ensure that both controlers were being used in an active / active senario (SPA is being used for iSCSI - from part 1).

That pretty much sums it up. Any questions or comments please feel free to ask and i'll add them to my blog.

Thanks for reading

Andy

Installation of New EMC VNXe 3300

We required a SAN for a company which had the resilience, reliability and performance at a reasonable price point to hold a server estate of around 30 servers including the usual Microsoft suite of Domain controllers, SQL and Exchange which would all be virtualised using VMware along with approximately 1TB of file shares. I Bench marked HP P4000 (left hand), Netapp’s FAS 2040 and the new EMC VNXe 3300 and the combination of great functionality, performance and a really great price point I opted for the EMC VNXe 3300.

The requirement was for 99.99% availability of the hardware so the connectivity between SAN and VMware estate were to be protect by a pair of Cisco 2960G’s powered via twin APC ups and a power transfer switch. This would protect the environment from power, network, server or cable failure.

As the EMC VNXe3300 is so new, the documentation that came with it, while simple to read didn’t cover a more complex environment, only a simple installation into a single switch so after several test scenarios and a learning curve on how the VNXe fails over and fails back (which is the key to the solution described below) I have devised the following example on how to configure a VNXe 3300 for a production environment.

How does the VNXe fail over? The VNXe 3300 is very clever on how it detects and fails over services between it processors (even better than it was sold to me). You set up a service on one or a team of network ports and should complete loss of connectivity occur on those network ports, it will fail over the server associated with those ports to the other processor module. For example, if you set up iSCSI on port 2 on SPA and CIFS on port 3 on SPA, if you pull the network cable out of Port 2 it will fail over iSCSI to SPB but keep the CIFS on SPA (very clever indeed). Once the cable is plugged back into SPA the iSCSI service then fails back over to SPA.

So the configuration I have used and tested is as follows. Two cables teamed together from SPA ports 2 and 3 go into your 1^st switch and two cables teamed together from SPB ports 2 and 3 go into your 2^nd switch. On the Cisco switch a LACP trunk is created for each of these connections. I then created a LACP trunk of two ports to connect the two switches together. The VMware host are connected to the switches via a single network cable to Switch 1 and a single network cable to switch 2. This configuration will allow for any 1 item to fail and connectivity from VMware hosts to SAN will remain operation.

The diagram below shows in more detail how this should be connected.