Monitor VMware ESXi hardware without root (Nagios)

Download and configure the plugin: http://exchange.nagios.org/directory/Plugins/Operating-Systems/*-Virtual-Environments/VMWare/check_esxi_hardware-2Epy/

– Create a new user in ESXi with no access privilege, you need to login to the ESXi directly to do that.

user

esxi_access

– Enable SSH, and add nagios user to root group:
# vi /etc/group
root:x:0:root,nagios

– Check from the command line, if it works
./check_esxi_hardware.py --host https://esxihost:5989 --user file:credentials.txt --pass file:credentials.txt
OK - Server: Cisco Systems Inc.....

– Configure the credentials files to use the nagios user credentials.

Could not connect to vmware console https://vcenter_address:7331/

This usually happens from the vSphere web client while opening a console session with virtual machine.

and the log (/var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log) shows something like:

[ERROR] Thread-42 System.err
INFO:oejsh.ContextHandler:started o.e.j.w.WebApp Context{/console,file:/tmp/jetty-0.0.0.0-7331-console.war-_console-any-/webapp/},/usr/lib/vmware-vsphere-client/server/work/tmp/console-distro/webapps/console.war

To fix this set the environment variable VMWARE_JAVA_HOME to proper path:

– SSH to vcenter
# vi /usr/lib/vmware-vsphere-client/server/wrapper/conf/wrapper.conf

– Under Environment variables add:
set.default.VMWARE_JAVA_HOME=/usr/java/jre-vmware

– Restart vsphere-client
# /etc/init.d/vsphere-client restart
Stopping VMware vSphere Web Client...
Stopped VMware vSphere Web Client.
Starting VMware vSphere Web Client...
Intializing registration provider...
Getting SSL certificates
Service with name was updated.
Return code is: Success
Waiting for VMware vSphere Web Client......
running: PID:

Reference: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2060604

Convert Linux Physical Server to VMware virtual machine

Download and install vCenter Converter on a windows machine.
http://www.vmware.com/products/converter
Unfortunately this tool does not have a Linux / MAC version.

vCenter Converter

vCenter Converter

In case you see an error: “Permission to perform this operation was denied”, right click and run the program as Administrator.

Permission to perform this operation was denied
Provide the source and destination information, the source is the physical server to be converted and the destination vCenter.

vCenter Converter

vCenter Converter

vCenter Converter

vCenter Converter

Follow the steps, to do the conversion , a temporary OS will be started on the destination, by default it try to get an IP address from the DHCP server so that it can connect to the source machine and fetch the files required. But in case if you don’t have DHCP server you might see error like: “Unable to obtain the IP address of the helper virtual machine” . Fix this issue by setting up a static IP to the helper virtual machine during the conversion setup. Basically the helper VM IP should be able to communicate with the source machine which needs to be migrated

vCenter Converter Static IP

vCenter Converter Static IP

Proceed with the conversion, the duration will be based on the size of the VM and the connectivity if it belongs to another site/LAN.

You may need to change the network configuration (eg: HWADDR) and the MAC address mapping (/etc/udev/rules.d) to get it connected.

ESXi host fails with a purple diagnostic screen PSOD

This happened while converting KVM VMs to VMware and power them on (method used: http://arunnsblog.com/2013/06/10/migrate-kvm-virtual-machines-to-vmware-esxi/) . It works for a while but then the ESXi crashes with PSOD.

Version : 5.1.0-799733

There were two sort of PSOD messages observed:
1) Crashed while the VM was running

 VMware NOT_IMPLEMENTED bora/vmkernel/sched/memsched.c:17724
 Code start: 0x41802b200000 VMK uptime: 10:19:25:27.335
 cpu4:8243)0x412200cdbaf0:[0x41802b27abff]PanicvPanicInt@vmkernel#nover+0x56 stack: 0x3000000008
 cpu4:8243)0x412200cdbbd0:[0x41802b27b4a7]Panic@vmkernel#nover+0xae stack: 0x100000000000000
 cpu4:8243)0x412200cdbc50:[0x41802b3d88eb]MemSched_WorldCleanup@vmkernel#nover+0x426 stack: 0x4100018a4fb0
 cpu4:8243)0x412200cdbef0:[0x41802b3033b8]WorldCleanup@vmkernel#nover+0x1cb stack: 0x4700cdbf40
 cpu4:8243)0x412200cdbf60:[0x41802b303829]WorldReap@vmkernel#nover+0x318 stack: 0x0
 cpu4:8243)0x412200cdbff0:[0x41802b2483c8]helpFunc@vmkernel#nover+0x517 stack: 0x0
 cpu4:8243)0x412200cdbff8:[0x0] stack: 0x0
 cpu4:8243)base fs=0x0 gs=0x418041000000 Kgs=0x0
VMWare_ESXi_PSOD
VMWare_ESXi_PSOD

2) Crashed during ESXi reboot.

#PF Exception 14 in world 8243:helper13-1 IP 0x41802b880a1e addr 0x410401503020
VMWare_ESXi_PSOD
VMWare_ESXi_PSOD

This seems to be a known issue in VMware ESXi 5.1 and is resolved in patch ESXi510-201212401-BG (Build 914609).
Ref: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038767

To work around this issue, SSH to the ESXi host and increase the MinZeroCopyBufferLength to 512.

# esxcli system settings advanced set -o /BufferCache/MinZeroCopyBufferLength -i 512

To verify that the setting has been updated, run this command:

# esxcli system settings advanced list --option /BufferCache/MinZeroCopyBufferLength
Before and after change
Before and after change

 

 

Migrate KVM virtual machines to VMware ESXi

– Shutdown the KVM guest
– convert the QCOW2 or RAW format to VMDK format

# qemu-img convert image.img -O vmdk image.vmdk

– Upload this image to datastore

– Create a new virtual machine with this disk image

– There might be issues with network interface mapping, fix the network mapping at /etc/udev/rules.d/70-persistent-net.rules

Create NAS/SAN storage with openfiler, work with VMware ESXi as shared storage

– Download the openfiler installation ISO , download link

I have downloaded the Installation ISO image (x86/64).

Basically we need to create open filer as a virtual machine with say 20 GB thin provisioning, and attach another disk/datastore to the virtual machine to configure it as SAN.

– Download and install  the Installation ISO image (x86/64) as a virtual machine, noting fancy here just do a normal installation.

– Once rebooted you get a web interface to login on port 446, https://< ip > :446/

– Login with username: openfiler, and password: password

Network access configuration:

set up network access configuration, enter the Network/Host , who are allowed to access.

System –> Network Access Configuration

Network Access Config
Network Access Config
Network Access Config
Network Access Config

Create a new physical volume

Volumes –> Block devices

Create Physical Volume
Create Physical Volume

Click on Edit devices on the hard disk where we going to create new physical volume, this is the extra hard drive we added to the virtual machine at the beginning. (/dev/sdb)

 

Create Physical Volume
Create Physical Volume

Select Physical volume as your partition type ( assuming that you are not using RAID), and set the mode to Primary, click create.

Create Physical Volume
Create Physical Volume

Create new Volume Group

Let us create a volume group for the physical volumes

Volumes –> Volume groups , enter a group name and select the physical drive and click Add volume

Create volume group
Create volume group

 

Create volume group
Create volume group

Create the Volume

Volume –> Add Volume

Enter Volume Name , Description, required space and select File System block.

Create Volume
Create Volume
Create Volume
Create Volume

Now Enable and Add  iSCSI Target

Services –> iSCSI Target Enable, Start

Start iSCSI target
Start iSCSI target

Volumes –> iSCSI Targets –> Target Configuration –> Add new iSCSI Target

Add new iSCSI target
Add new iSCSI target

Setup the LUN Mapping and allow access to iSCSI target

Setup LUN Mapping
Setup LUN Mapping

 

Allow access to iSCSI target
Allow access to iSCSI target

Now the open filer is ready to use.

In VMware ESXi

Create new VMKernel network adapter to use for iSCSI connection

VMKernel Adapter
VMKernel Adapter
VMKernel Adapter
VMKernel Adapter
VMKernel Adapter
VMKernel Adapter

Click on storage adapter –> Add new

Add storage adapter
Add storage adapter

Click on properties, add the VMkernel switch and the discover the iSCSI target.

Add VMKernel to iSCSI adapter
Add VMKernel to iSCSI adapter
Dynamic discovery
Dynamic discovery

Now your ESXi will show the openfiler as a datastore :)

Openfiler datastore
Openfiler datastore

More information about Openfiler:

http://www.openfiler.com/products

 

Performance issues with KVM – Redhat

The general performance issue with KVM is due to DISK I/O

– by default the Redhat KVM guest are created with x86_64 architecuture, if you installed 32 bit operating system change this to i686.

<os>
<type arch=’i686′ machine=’rhel5.6.0′>hvm</type>
<boot dev=’hd’/>

– Make sure the hypervisor used is correct in the configuration , either qemu or kvm

<domain type=’kvm’>

or

<domain type=’kvm’>

– Use virtio drivers if the guest is paravirtulized (http://www.ibm.com/developerworks/linux/library/l-virtio/index.html?ca=dgr-lnxw97Viriodth-LX&S_TACT=105AGX59&S_CMP=grlnxw97 http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaat/liaatbpparavirt.htm)

./arun

Converting LVM virtual machine storage to image

To convert the LVM disk to qcow2 formatted disk image,

Use lvdisplay to get the Logical volume name

$ sudo lvdisplay

Use qemu-img to convert to the required image format

# qemu-img convert -O qcow2 /dev/mapper/lv_name <destination_file>.qcow2

eg:

# qemu-img convert -O qcow2 /dev/mapper/disk1 disk1.qcow2

This will be useful to replicate the virtual machines to other hardware.

./arun

IPv6 configuration for KVM guests

It is simple and straight forward to enable IPv6 on KVM guests

Configure the host machine with IPv6 Address on the bridge interface

cat ifcfg-br0

IPV6INIT=yes
IPV6ADDR=xxxx.xx::10
IPV6_DEFAULTGW=xxxx.xx::1
IPV6_AUTOCONF=no

Configure the interface on virutal machines with ipv6 address

cat ifcfg-eth0

IPV6INIT=yes
IPV6ADDR=xxxx.xx::11
IPV6_DEFAULTGW=xxxx.xx::1
IPV6_AUTOCONF=no

Add the the necessary firewall rules to ip6tables on the host machine

-A FORWARD -m physdev –physdev-is-bridged -j ACCEPT.

./arun

NAT with port forwarding on Virtual Box

You can use the host-only-adapter networking, if you require the virtual machine to be accessible only from the host machine. In this case your virtual machine will not have access to anywhere outside the host. Read more about virtual box networking at http://www.virtualbox.org/manual/ch06.html

On the other hand NAT enabled interface can communicate with clients outside the host, but the host cannot access the services on the virtual machine directly. We need to enabled port forwarding with NAT interface to achieve this.

On Linux:
If you need to have ssh accessible from host machine to virtual machine,

$ VBoxManage modifyvm "VM Name" --natpf1 "openssh,tcp,127.0.0.1,2222,,22"

Where –natpf1 is for adapter1, openssh is just a anme, and you can also input the ip address of virtual machine like

$ VBoxManage modifyvm "VM Name" --natpf1 "openssh,tcp,127.0.0.1,2222,10.0.2.20,22"

(assume the virtual machine ip is 10.0.2.20)

Now you can make ssh connection from host like, $ ssh localhost -p 2222

We can use same port number for port number about 1024 , say for a service running on port 8080 we can forward it with

VBoxManage modifyvm "VM Name" --natpf1 "proxy,tcp,127.0.0.1,8080,10.0.2.20,8080"

These rules will be added to the .VirtualBox/Machines/machine_name/machine_name.xml file like:
< Forwarding name="openssh" proto="1" hostip="127.0.0.1" hostport="2222" guestip=10.0.2.20 guestport="2222"/>

You can forward connection to any port on virtual host like this.

Make sure that the virtual machine interface is closed and the vm is not running while you change it, otherwise the changes will not take effect.

On Windows:

VBoxManage setextradata "VM Name" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
VBoxManage setextradata "VM Name" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
VBoxManage setextradata "VM Name" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222

* Replace VM Name with your virtual instance name

./arun