OpenNebula 3.2 cloud setup

OpenNebula enables building a very good cloud setup that even has multi-tenancy features. OpenNebula's distro packaging is sad, at best.Using Xen on the "host nodes" will bring better scalabiltiy but makes the setup initially a lot harder. A dynamic storage layer using LIO and FlashCache would enable supreme flexibility and performance. A networking layer based on OpenVSwitch allows better LACP performance and a basic level of cross-site tunneling. Using a modern Linux kernel gives additional choices with Xen.

Alltogether these tools have so many incompatibilites that it's impossible to just install the stuff - hell, it's so many I can't even memorize them.

If you want to build heaven on a single perfect cloud you'll soon find yourself in hell.

We will have to break problems into small ones!

  1. Deal with OpenNebula setup
  2. Sort out a basic setup
  3. Bring up including KVM until we can deploy
  4. Deal with Xen later on (Xen might be best to run on something other than Ubuntu. Or maybe we can easily integrate it once we're down to this single, last, problem)
  5. include all the Gizmos we'd want to have in our hosts (flashcache, DR:BD, lio ...)

OpenNebula software

Ubuntu only has packages for OpenNebula 2.2.

There are no Ubuntu PPAs with OpenNebula 3.x. has packages for Ubuntu 10.04 only, you can chose either OpenNebula 3.2 or 3.0

The packages are not split into separate packages, so you should only use them for the frontend node!

The OpenNebula tarball does not come with Debian buildscripts.

The packages are fully incompatible with the ones from Ubuntu. You need to avoid any interference.

Some versions try to initialize the database before starting it (die in a fire, dpkg)

Basic setup 

Make oned and Sunstone work

don't be greedy with the storage at first setup

NFS-Export a image store (100GB)

NFS-Export a VM store (100GB)

Add a KVM node (test oned add works and it's visible / reporting in Sunstone)

Ensure /var/lib/one on the node has another 100GB

Make oneaccd work (and ensure it doesnt die soon after)

Successfully deploy a VM

Make noVNC work for vnc access via webbrowser on hosts other than the one frontend node.


Xen is usually utterly broken on Ubuntu. Xen 4.1.2 is a regression of it's own, deprecating many features.  For example, no external storage scripts are supported. They just trigger a message that says something like "uh sorry" and this means integrating things like MooseFS or RADOS will be harder.

The chances are better on 11.10 than on older releases but I *have* tested it and they have broken the dom0_mem option. This means the dom0 autoballooned down from 128GB to something negative, and after 10 minutes or so the OOM killer would kick into action.

It would be good to track which PPA Xen build has which bugs, also I had to manually select the right Xen options in grub2, it is not autoconfigured to be the primary boot option.

Kernel 3.2 is interesting since it has most Xen internal stuff right for the PVOPS support.

As for Xen versions - I have had Opennebula talk to Oracle VM 2.2.3 and Ubuntu 11.10.I also wanna test against Oracle VM 3.0.2 and AlpineLinux.

I don't want to test against CentOS6.2 as I think it will be very painful. I also don't want to try Debian.

tmem support would be a very good thing to have.


Pacemaker would be needed to cluster the frontend node including Sunstone


Kernel 3.2 is not supported in OpenVswitch 1.3.

Don't upgrade to kernel 3.2 via PPA if you need OpenVSwitch


FaceBook Flashcache installs OK on 11.10.

Not sure about 10.04/10.10. It is said to not work on 32bit.

for local storage as a SCSI target:

LIO-Target isn't available as something easlly installable. It's split into a few packages and I'll have a separate doc for it.