Building Enterprise OpenSim Grid on 2nd Birthday

Building enterprise quality virtual environment services out of OpenSim is a complex operation but today you can do it all with open source software. OpenSim still enjoys alpha status and many of the core components are under active development on daily basis. Following the svn head and figuring out stable revisions to baseline your system to is challenging task and not suitable for faint hearted.

First things first: I named my grid "Bubble Cloud" according to the website where we have been drafting out next generation virtual reality protocol MXP with Ben Lindquist and Ryan McDougall :

http://www.bubblecloud.org/

Building

I went for the following architecture:

I reserved one virtual machine (Psi) for shared grid services called UGAIM according to User, Grid, Asset, Inventory and Messaging. Another virtual server went to monitoring use (Omega). Last but not least I configured 2 virtual machines for simulators which run regions (Alpha and Beta).

All virtual machines have 2 Giga bytes memory, 1 Xeon Core and 20 Giga bytes of raid disk.

OS selection was a tough one as .NET is rumored to be very good choice for running OpenSim servers. In the end I went for open source and thus latest stable Ubuntu was the clear choice as it is most used OS for running OpenSim servers.

Database was no brainer as MySQL is the most used database in OpenSim Ubuntu environments. I installed local MySQL instances to all virtual machines to enable fast database access and to save in resource costs.

Mono version was another hard question but I thought to go the easy way and use the standard mono version provided by the Ubuntu release (1.9.1) all though version 2 has many optimizations available. It might be that I need to upgrade soon to newer mono.

For easy administration I was happy to acquire linux init scripts from Sean Dague's Linux Service Helpers project for both ugaim and simulator:

http://forge.opensimulator.org/gf/project/linuxservice/

By copying these scripts to /etc/init.d and linking them to run levels you can have your OpenSim services start and stop automatically with your server.

Monitoring

To be able to measure the resource utilization, receive alerts, diagnose problems and track service availability I chose Zabbix server to be installed to Omega virtual machine and agents to other servers. Zabbix is excellent open source monitoring solution:

http://www.zabbix.com/

Configuring basic zabbix Linux monitoring is easy and can be done in two hours. Monitoring OpenSim servers and regions took a bit more work. I did this with Web Monitoring feature by hooking separate scenarios, triggers and IT services to each component. As a result everyone can follow availability figures of Bubble Cloud by bookmarking this url:

http://monitor.bubblecloud.org/srv_status.php

It is also possible to follow things like processor loads, memory usage, network traffic and disk space:

Cost

Running Enterprise OpenSim Grid is not free but you do not need mega corporation to finance it. One needs to pay for Virtual Machine rents or alternatively hardware costs and server hotel rent.

Renting these virtual machines will cost anything between 100 - 500 us dollars per virtual machine depending where you acquire them and if you pay by use or constant rate.

If you decide to build a server yourself (2 x Quad Core Xeon, 32G memory, raid 1-0 disks...) and take it to good server hotel with 100 Mbps or better internet connections you have to prepare for 5000 us dollar up front cost and 100 hundred dollar monthly cost. With this setup you can get 8 heavy duty virtual machines with 1 Xeon Core and 4G memory. If you divide the cost for period of 3 years you get approximately 30 dollars monthly cost which is not bad at all. The downside is the up front investment, knowledge requirement and time needed.

If you have the knowledge you can build the environment yourself but if you need to hire someone the time required for building this kind of setup is at minimum 1 week for the hardware and 1 week for the software.

Thanks

I want to thank all the nice people in freenode irc channels #opensim, #osgrid and #zabbix for their invaluable effort put in building this software and the help they gave to me in my learning process.

Demo for MXP Virtual Reality Protocol

Interactive bubble bouncer demonstrates MXP in real interactive multiuser environment. MXP is an open distributed virtual environment protocol with some novel ideas builtin. Detailed MXP specification can be found from: http://www.bubblecloud.org/

MXP is one candidate to become standard means of client to server communication and server to server interoperability. Typical applications would be systems like OpenSim, SecondLife, realXtend and their interoperation.

Interactive bubble bouncer demo client can be downloaded from: http://www.bubblecloud.org/

Client requires modern 3d video card (shared profile 2.0), Microsoft Windows with .NET Framework 3.5 and XNA runtime. If you do not have them you can download them install time.

If these requirements are too much you can check the video: http://www.youtube.com/watch?v=SiO7D_HDJR4

Metaverse eXchange Protocol (MXP) - A Practical Virtual Reality Protocol

We have been working closely with Ben Lindquist (http://greenphosphor.com/), Ryan McDougall (http://www.realxtend.org/) and Jon Watte (http://www.forterrainc.com/) to create a practical virtual reality protocol specification. Metaverse eXchange Protocol (MXP) is currently in 0.3 draft phase and a C# reference implementation exists. Detailed specification, source codes and binaries can be found from the following address:

http://www.bubblecloud.org/

MXP is a UDP binary protocol offering structured messages for distributed 3d simulations. MXP has well defined scope but extendability and flexibility has not been forgotten. It carries as payload a genre specific protocol which is used to communicate for example animation sequences and interactions. MXP has well defined domain model and use cases which are good read for those who are interested in this area of expertice. MXP leverages novel ideas enabling unpreceded level of virtual environment scalability and interoperability.

Current proof of concept implementation demonstrates 100 moving objects with client bandwidth use of 75 kbps (bits per second) without optimized version of movement message. Optimized movement messages will theoretically offer 15 kbps bandwidth for 100 objects and 1.5Mbps for 10 000 objects.

MXP supports bubble cloud concept on both client to server and server to server messaging. Bubble cloud concept consists of linked spherical server bubbles which may overlap and be sized and arranged in any 3 dimensional configuration. Clients roam between bubbles seemlessly in analogy with mobile devices and base stations. Bubbles exchange information about objects they contain so that clients connected to one bubble observe objects from the neighbouring bubbles as well. Any client may inject objects to the bubble in the boundaries of access rights.

We are looking for more people willing to join this community effort and projects like realXtend and OpenSim which are able to give us requirements and use cases for the specification work. More details can be found from the following address:

http://www.bubblecloud.org/

Improving Second Life and OpenSim Grid Configuration

Second Life, OpenSim and many other virtual environment implementations apply simple grid configuration consisting of single layer of cubes. This is the most simple way to divide virtual space to separate simulation nodes in other words to do spatial load balancing:

While this configuration is simple and effective it is possible to improve it significantly to provide better performance, redundancy and local load balancing. In redundant grid one simulation node may shutdown and the grid remains accessible in that location. Local load balancing means that the same space is simulated by several simulations which exchange state information. One such configuration is dual cubic lattice:The following page contains example calculation estimating that redundant dual cubic lattice provides 25% more objects and 100% more clients with the same bandwidth usage and same maximum object size as simple cubic lattice. The example assumes that network capacity is the limiting performance factor:

Advanced Grid Configurations at http://wiki.setp.info

25% more objects is not that much but double the amount of clients and simulation redundancy are remarkable improvements.

OpenSim Starts a New Era?

Thanks to Chris Thorne I found the OpenSim project. There is a great promise in this project as there are major corporations behind it and they all have ambitious plans for the future of virtual environments. It is an open source virtual world server written with C#. The server functions with the open source SecondLife client and there are tens of grids in the grid list. Grid is a synonym for cluster or network of servers forming together a virtual world. I want to contribute to the effort myself with a node or two. Now now, which grid to choose?

The functional clients seem to be branches from the open source SecondLife client and as such have the same glitches and usability problems. It would be great to see a bravely redesigned client with good usability and smooth functionality. SecondLife and the OpenSim clients remind me of the good old rule: "Prefer quality over quantity".

Why is it that still even now after all these years the avatar in SecondLife moon walks like a zombie. Creative directors need to appreciate how important immersion is in 3d environments and how fragile it is.

http://opensimulator.org/wiki/Main_Page

Open Themes - Key to Coherent Media

There are great amount of 3d models, sounds and music available as open source in the Internet. Building a good user experience from these incoherent sources is problematic. Open themes: color packs, style guides and examples would be an important catalyst for open virtual reality development. Once a theme exist the open media could be themed accordingly to become an open media pack with unified style and standard media formats like Collada. This would inspire more artists and developers to participate in the effort and give open virtual environments more polished and professional look. If you find such theme projects please let me know.

Virtual Brought to Real World

Polygon Playground is an innovative way to bring virtual phenomenon to real world for children and adults to play with. The video is worth watching. Reminds me of science fiction movies.

http://www.polygon-playground.com/