BOSH on vSphere (BOSH Deployment)

In the previous post we successfully deployed micro Bosh. But this is just an intermediate step, what we need is a "full" BOSH deployment. In this post we'll go through the steps required to deploy BOSH.

Stemcell, Release and Manifest

These three represent components that are needed for any deployment to Bosh. Let's start with stemcell. To get a list of available stemcells use:

$ bosh public stemcells
+---------------------------------+-------------------------------------------------------+
| Name                            | Url                                                   |
+---------------------------------+-------------------------------------------------------+
| bosh-stemcell-0.3.0.tgz         | https://blob.cfblob.com/rest/objects/4e4e78bca41e1... |
| bosh-stemcell-0.4.4.tgz         | https://blob.cfblob.com/rest/objects/4e4e78bca51e1... |
| bosh-stemcell-0.4.7.tgz         | https://blob.cfblob.com/rest/objects/4e4e78bca21e1... |
| bosh-stemcell-0.5.2.tgz         | https://blob.cfblob.com/rest/objects/4e4e78bca31e1... |
| bosh-stemcell-aws-0.5.1.tgz     | https://blob.cfblob.com/rest/objects/4e4e78bca21e1... |
| bosh-stemcell-vsphere-0.6.0.tgz | https://blob.cfblob.com/rest/objects/4e4e78bca41e1... |
| bosh-stemcell-vsphere-0.6.1.tgz | https://blob.cfblob.com/rest/objects/4e4e78bca31e1... |
| micro-bosh-stemcell-0.1.0.tgz   | https://blob.cfblob.com/rest/objects/4e4e78bca51e1... |
+---------------------------------+-------------------------------------------------------+
To download use 'bosh download public stemcell '.For full url use --full.

The latest vSphere stemcell available is of version 0.6.1 so let's download it:

$ cd ~/stemcells
$ bosh download public stemcell bosh-stemcell-vsphere-0.6.1.tgz

and upload it to the micro BOSH so that is is available later for our deployment and verify it's there:

$ bosh upload stemcell bosh-stemcell-vsphere-0.6.1.tgz
$ bosh stemcells

+---------------+---------+-----------------------------------------+
| Name          | Version | CID                                     |
+---------------+---------+-----------------------------------------+
| bosh-stemcell | 0.6.1   | sc-6477c50d-c2df-4783-88ea-b3e88e002445 |
+---------------+---------+-----------------------------------------+

Stemcells total: 1

Release

Release is the next requirement. We'll get one directly from the github repository, upload to micro Bosh and verify it's there:

$ mkdir ~/releases
$ cd ~/releases
$ git clone https://github.com/cloudfoundry/bosh-release.git
$ cd bosh-release
$ bosh upload release releases/bosh-5.yml

Copying packages
----------------
...
Copying jobs
------------
...
Release info
------------
Name:    bosh
Version: 5

Packages
...
Jobs
...
Release uploaded
$ bosh releases

+------+----------+
| Name | Versions |
+------+----------+
| bosh | 5        |
+------+----------+

Releases total: 1

Deployment Manifest

Deployment manifest is the last component needed for our deployment. BOSH deployment consists of 6 VMs:

• Director - 192.168.2.225
• Postgres DB - 192.168.2.223
• Nats - 192.168.2.222
• Redis - 192.168.2.224
• Blobstore - 192.168.2.226
• Health Monitor - 192.168.2.227

There is one provided as an example and we'll modify it a little to fit our configuration. The changes include:

• director_uuid: has to be the same as obtained from $ bosh status
• networks: IP ranges as used in our vSphere infrastructure configuration
• compilation: we'll use 2 workers only (due to resource limits)
• jobs: update IP addresses appropriately
• properties: update IP addresses appropriately
• properties:vcenter: update to match our vSphere configuration

The customized deployment manifest that we'll use can be found here: https://gist.github.com/3017913

$ mkdir -p ~/deployments/igmlab
$ cd ~/deployments/igmlab
$ vim bosh-manifest.yml
... copy and paste content of bosh-manifest.yml

The last step is to perform actual deploy:

$ bosh deployment igmlab/bosh-manifest.yml
Deployment set to '$HOME/deployments/igmlab/bosh-manifest.yml'
$ bosh deploy

Entire process takes about 16 minutes in my case. We can verify deployment afterwards:

$ bosh deployments

+------+
| Name |
+------+
| bosh |
+------+

Deployments total: 1

This is what it looks like now in my vSphere Client:

Now we can target our new Bosh:

$ bosh target 192.168.2.225:25555
$ bosh login
Your username: admin
Enter password: admin
Logged in as 'admin'
$ bosh deployment
No deployments
$ bosh releases
No releases
$ bosh stemcells
No stemcells

Troubleshooting

If anything goes wrong it's important to find out what exactly went wrong and that is not the easiest thing in BOSH at this stage. Very often BOSH CLI just prints "Error" message without any details. So here are few tips.

Each time a task is executed from CLI on Bosh Director it is assigned an ID and this ID is printed on the console: Director task 2 for example. If error happens during the task execution you can find details directly on the target server:

$ ssh root@192.168.2.120
password: vmware

# cd /var/vcap/store/director/tasks
# ls -l
total 8
drwxr-xr-x 2 root root 4096 2012-06-29 15:10 1
drwxr-xr-x 2 root root 4096 2012-06-29 15:13 2

Each task has it's own directory. So for task '2' go:

# cd 2
# ls -la
total 400
-rw-r--r-- 1 root root  19142 2012-06-29 15:18 debug
-rw-r--r-- 1 root root   1693 2012-06-29 15:18 event
-rw-r--r-- 1 root root      0 2012-06-29 15:13 result
-rw-r--r-- 1 root root 384951 2012-06-29 15:18 soap

All the debug level information is stored in "debug" file and that is the best place to find error details. "ERROR" lines are those to look at first. An example of ERROR line is here. The next logical step is to open the script and look directly into the source to see what is going on there:

# vi /var/vcap/packages/director/bosh/director/lib/director/deployment_plan/network_subnet.rb
.. and navigate to line 60

If you are interested in SOAP interaction with vSphere Server all this is traced in "soap" file as Request/Response blocks.

Another source of information can be found in agent log directory:

# cd /var/vcap/bosh/log
# vi current 

 

References

[1] Cloud Foundy Bosh Documentation
[2] Deployment Manifest Samples
[3] End to End on vSphere
[4] Using Bosh on vSphere Series