vRo 7/8 Plugin on vCenter 6.7 missing/do not load after upgrade

There is no GA plugin available for the HTML5 client. This is planed to be included on the 8.2 release.

You may use the beta client as a workaround, However this needs manual installation.. Please ensure that you take a snapshot before you run through the below steps. 

cleanup: Delete/move the contents of the below directory. If the directory do not exist, create them.

ui client:
 Flex client:

H5 plugin:

Download the zip file from


extract the contents of the file to the below path


set appropriate permissions to the directory

chown -R  vsphere-ui:users  /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1

restart vsphere-ui client:

service-control --restart vsphere-ui


Download file:


extract the contents of the zip to


set appropriate permissions

chown -R  vsphere-client:users /etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.vmware.vco-7.3.1

restart vsphere-webclient service:

service-control --restart vsphere-client

log back in and double check , The plugin will take about 3-5 min to pull data from vRo on loading it.

Install Rancher and set up a Kubernetes cluster

Install docker:

sudo curl https://releases.rancher.com/install-docker/19.03.sh | sh
sudo usermod -aG docker $user

install Rancher in a docker container (single node) with persistent data. We’ll map 80 of the container to 8080 of the docker host and 443 of the container to 8443 of the external host, I’m doing this since I will be using the above ports for different containers.

docker run -d --restart=unless-stopped \
  -p 8080:80 -p 8443:443 \
  -v /opt/rancher:/var/lib/rancher \

check the status of the container:

docker ps -l

on a browser, open the IP address of the Docker host with the port specified previously. in my case it was , it should prompt you for a password similar to the below.

Once Done, You are prompted with the below page, click on “add a cluster” and then select “existing node”

Enter the cluster name and then hit next

In the node option, ensure that you select etcd, control plane and worker, copy the script from the below tab and run them on the Nodes that you want to spin up kubernetics.

In my senario since this is a lab setup, I will use the same docker host and 2 other nodes.

Node 1
Node 2
Node 3

Once Setup, wait a while for the nodes to startup (you will see few warnings, do not panic. wait it out.)

once they are all up, it should look like the below:

Congratulations!!, you have now set up rancher and are ready to start deploying Kubernetes workloads!!

Usage meter 4.2 ip/DNS changes back to default after reboot.

Symptom: On new deployment, the Ip/DNS are set correctly but after deployment and power on, the DNS or the IP range is different to that of what was used at the time of deployment

symptom2: When you try to change the Ip/DNS of the usage meter appliance, it returns back to the original values after reboot.

symptom 3: How to change usage meter 4.2 ip address/DNS

cause: usage meter 4.2 relies on ovfenv(a feature of vCenter that leverages IP pool’s to automatically lease out/Set DNS records on compatible ovf). The virtual machine port group used for usage meter is associated to a port group that has an IP pool associated with the incorrect values (values used here is what is cascaded over to usage meter)

Resolution: Log into vCenter flex client> networking>Look for the vm port group>configuration>network pool>associated network, edit this and correct the ip pool and the DNS there and then reboot/retry the deployment

Workaround: to disable (uncheck) ovf environment via vCenter flex client>edit UM VM settings>vapp options> under authoring>ip allocation>ip allocation scheme

Set the correct the IP/mask/gw/DNS in /opt/vmware/etc/vami/ovfEnv.xml and then reboot the VM

Note: This is only applicable if the VM port group at the time of deployment had an ip pool associated. If it did not have one, then you can set the ip/mask/gateway by following the instructions here: http://blog.ikigo.net/?p=472 or using vami_config_net

Replace Usage Meter 4.2 certificates

Note: Before making any changes, it is highly recommended that you take the necessary backup/snapshots of the usage meter VM instances.

Usage meter certificate replacement is rather simple. Usage meter 4.2 Runs on Nginx and the Nginx configuration for the webserver is located here


Let’s take a look at the Nginx configuration for certificate and the key used:

root@is-dhcp35-102 [ / ]# cat ./opt/vmware/cloudusagemetering16222545/conf/nginx.conf | grep crt
        ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
root@is-dhcp35-102 [ / ]# cat ./opt/vmware/cloudusagemetering16222545/conf/nginx.conf | grep key
        ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

In order to replace the certificate, Either rename and replace the above-mentioned files with the new CA-signed certificate, key
create a directory and dump the new, signed certificate, key in there and make necessary corrections to the config file.

Do keep in mind that the permissions will need to be set accordingly. (chown and chgrp)

root@is-dhcp35-102 [ /etc/ssl/certs ]# ls -ltrh nginx-selfsigned.crt
-rw-r--r-- 1 root root 1.3K Jun 25 05:02 nginx-selfsigned.crt

root@is-dhcp35-102 [ /etc/ssl/private ]# ls -ltrh
total 4.0K
-rw------- 1 usagemeter usagemeter 1.7K Jun 25 05:02 nginx-selfsigned.key

Once done, restart the usage meter appliance.

reboot -f

photon 2.0 Network configuration

file: /etc/systemd/network/10-eth0.network





if the DNS= is not specified on the file, resolve.conf will be overwritten with DHCP provided values (if the DHCP flag is not present)

Usage meter 4.x root password reset/unlock

At the photon logo, press ‘e’ and you will see the editable grub menu:
Append rw init=/bin/bash on the line that starts with “linux”

Press ctrl + x or f10 to continue

you should see a screen similar to the below:

To unlock the account, type the below command (if you know the password)

/sbin/pam_tally2 -r -u root

to reset the password, run the below

passwd root

Note: Changing the password does not unlock the account. if the account is locked out, you will need to run the previous command to unlock

Restart the guest of and then boot back into normal appliance and then try logging back in.

vRBC -Exporting a database dump/restoring Database dump

Instructions to backup/restore the vRBC database below

Backup DB:

export PGPASSWORD=`grep 'jdbc.password=\K.*' /usr/local/tomcat/itbm-server/conf/itfm.properties -Po` 

   /opt/vmware/vpostgres/current/bin/pg_dump --data-only -U itfm_cloud_admin -d postgres -t vcac_user_consumer_mapping > /root/backup.sql 

Restore DB

   export PGPASSWORD=`grep 'jdbc.password=\K.*' /usr/local/tomcat/itbm-server/conf/itfm.properties -Po` 

   /opt/vmware/vpostgres/current/bin/psql -U itfm_cloud_admin -d postgres -f /root/backup.sql 

Log in/connect to vRBC vPostgres:


VRA health API via bash with results


curl https://localhost/SAAS/API/1.0/REST/system/health -k

[master] cava-n-80-094:/etc/init.d # curl https://localhost/SAAS/API/1.0/REST/system/health  -k
{"AnalyticsUrl":"http://localhost:8080","EhCacheClusterPeers":"","AuditPollInterval":"1000","EncryptionServiceVersion":"unknown","AnalyticsConnectionOk":"true","EncryptionServiceVerified":"Master Keystore verified","FederationBrokerStatus":"ok","ServiceReadOnlyMode":"false","AuditWorkerThreadAlive":"true","BuildVersion":" Build 12694081","AuditQueueSize":"0","DatabaseStatus":"connection failure","HostName":"cava-n-80-094.eng.vmware.com","EncryptionStatus":"connected","FederationBrokerOk":"true","EncryptionConnectionOk":"true","EncryptionServiceImpl":"Encryption Service DB","ClusterId":"9b545db2-2c45-4950-b8e3-99e0eb3671d3","EhCacheClusterDiagnostics":"","DatabaseConnectionOk":"false","StatusDate":"2020-02-22 13:38:46 UTC","ClockSyncOk":"true","MaintenanceMode":"false","MessagingConnectionOk":"true","fipsModeEnabled":"false","ServiceVersion":"3.1.0","IpAddress":"","AuditDisabled":"false","AllOk":"false"}[master] 

Elastic search

curl localhost:9200/_cluster/health?pretty=true

[master] cava-n-80-094:~ # curl localhost:9200/_cluster/health?pretty=true
  "cluster_name" : "horizon",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0

Recovering from expired certificates on vRo (vRealize orchestrator)- cli methord

So the other day the vRo certificates had expired. We tried to change it from the vco-control center, after importing the certificates, vRo UI would simply stay there stating restarting in 2 min but nothing happens.

to replace the certificate’s via CLI

Grab the Keystore password

cat  /var/lib/vco/keystore.password

Generate CSR using key tool

keytool -certreq -alias dunes -keypass "Pge2Nn366tNBqNavkgg6VZOHJuWmkIHAEPNq1DYu" -keystore "/etc/vco/app-server/security/jssecacerts" -file "/crt/new.csr" -storepass "Pge2Nn366tNBqNavkgg6VZOHJuWmkIHAEPNq1DYu" -ext SAN=DNS:vip.domain.com,DNS:vro1.domain.com,DNS:vro2.domain.com

Grab the /crt/new.csr and get this signed using the CA, Import the signed cert back into vRo and then import the certificate

keytool -importcert -alias dunes -keypass "Pge2Nn366tNBqNavkgg6VZOHJuWmkIHAEPNq1DYu" -file "/crt/casigned.crt" -keystore "/etc/vco/app-server/security/jssecacerts" -storepass "Pge2Nn366tNBqNavkgg6VZOHJuWmkIHAEPNq1DYu"

Restart Services

service vco-server restart && service vco-configurator restart

Now, copy the signed certificate over to node2 and then run the import command (grab the keystore password from /var/lib/vco/keystore.password)

keytool -importcert -alias dunes -keypass "AzW2gI1QJcNcRNzRX3TyrznhKlYNagKje45fTbSB" -file "/crt/casigned.crt" -keystore "/etc/vco/app-server/security/jssecacerts" -storepass "AzW2gI1QJcNcRNzRX3TyrznhKlYNagKje45fTbSB"

Restart services and you are done!!

service vco-server restart && service vco-configurator restart