The Syncing Apple An Exploration of Technology and Devices

Xen Domains, RHEL and Networking

At the office I'm deploying some new servers using Xen domains running under RHEL 5.0. We use RHEL 4.4 for our systems, and so re-deploying some of them as RHEL 4.5 (the Xen-kernel enabled version) Domains saves hardware space, money and keeps the server room cooler. Many dev servers are not utilized like production systems and so will benefit from the Xen virtualization.

Setting up the domains is pretty straight forward, RedHat has a nice guide using their virt-manager GUI tool. There are also excellent guides at the Centos wiki.

Two things were not obvious to me, and I'm posting my findings here to help others.

Issue 1 Multiple networks

After building the domain, the first thing that stopped me was that I needed multiple networks. The virtual hosts need to be on three networks, and so I needed three virtual devices. The RedHat guide has a nice section on how to create the necessary br0 (network bridge devices) devices, so modifying the scripts to go from one br0 to three (br0, br2, br3) each corresponding to a real NIC on the host was straightforward. However, how to get the virtual devices to recognize and use these new bridges? There is only one config file for your domain, and it looks like this:

# Automatically generated xen config file
name = "myVirtualPony"
memory = "1024"
disk = [ 'phy:/dev/virt_vg/some_virtdisk,xvda,w', ]
vif = [ 'mac=00:16:3e:3d:11:f7, bridge=xenbr0', ]
vfb = ["type=vnc,vncunused=1"]
uuid = "fff0f6c0-60e2-xxxx-xxxx-xxxxxxxxxx"
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot   = 'restart'
on_crash    = 'restart'

It should be obvious to any Python programmer, but it wasn't to me initially. Eventually I realized you had to add some new attributes to the vif list. The Redhat guide has a nice python snippet to generate new MAC addresses for you. I used that to generate two more MACs, then added them in:

# Automatically generated xen config file
name = "myVirtualPony"
memory = "1024"
disk = [ 'phy:/dev/virt_vg/some_virtdisk,xvda,w', ]
vif = [ 'mac=00:16:3e:3d:11:f7, bridge=xenbr0',
        'mac=00:16:3e:19:11:ed, bridge=xenbr2',
        'mac=00:16:3e:49:11:40, bridge=xenbr3', ]
vfb = ["type=vnc,vncunused=1"]
uuid = "fff0f6c0-60e2-xxxx-xxxx-xxxxxxxxxx"
bootloader="/usr/bin/pygrub"
vcpus=1
on_reboot   = 'restart'
on_crash    = 'restart'

Now just reboot the domain with a quick

$ xm reboot myVirtualPony

and loginto the domain. Copy the /etc/sysconfig/network-scripts/ifcfg-eth0 to ifcfg-eth1, ifcfg-eth2 and edit them, using the new MAC addresses you created and the proper TCP/IP info.

There is an excellent Page on Xen Networking at Xensource Wiki. I wish I had found this when I was banging my head on this problem.

Issue 2 - Virtual Console

After my domains were setup most Xen docs talk about using the virtual console to login to your domain from the host, similar to what you can do with Solaris Zones.

On the master host you should be able to login to the virtual console of the local domains. However, when you build your domains with RedHat's virt-manger they don't setup the virtual console correctly. They attach serial output to the virtual-framebuffer. So if you try and use the virtual tty, you only get the output of the kernel booting, then it stops and your terminal is stuck.

To fix this, you can follow the directions Xen Centos Tips and Tricks page:

You just need to add

co:2345:respawn:/sbin/agetty xvc0 9600 vt100-nav

in /etc/inittab

and add

xvc0

to the file /etc/securetty. Then do a @ telinit q to re-load the /etc/inittab and you should be able to use the virtual console from a tty.

Now with that out of the way I can do this to login into my domain:

xm console myVirtualPony

Xen is fun.