VMCA Certificates replacement for vCenter, workflow

VMCA is the Default self-signed certificates that is set up at the time of vCenter deployment.

All solution users and machine SSL certificates are signed with this certificate.

VMCA certificates can be regenerated by using option 8 on the certificatae manager.

root@is-dhcp36-107 [ / ]# /usr/lib/vmware-vmca/bin/certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 6.5 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]:

choosing option 8, you are presented with the below options:

Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@vs.lo
Enter password:

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] :

Enter proper value for 'Name' [Default value : CA] :

Enter proper value for 'Organization' [Default value : VMware] :

Enter proper value for 'OrgUnit' [Default value : VMware Engineering] :

Enter proper value for 'State' [Default value : California] :

Enter proper value for 'Locality' [Default value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] :

Enter proper value for 'Email' [Default value : email@acme.com] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : is-dhcp36-107.isl.vmware.com

Enter proper value for VMCA 'Name' :VMCA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y

Below are the .cfg’s created for the certificate

root@is-dhcp36-107 [ /var/tmp/vmware ]# ls -ltrh
total 32K
drwxr-xr-x 3 root root 4.0K Mar 14 00:07 cis-license
-rw-r--r-- 1 root root  191 Apr  2 19:24 certool.cfg
-rw-r--r-- 1 root root  243 Apr  2 19:24 vsphere-webclient.cfg
-rw-r--r-- 1 root root  240 Apr  2 19:24 vpxd-extension.cfg
-rw-r--r-- 1 root root  230 Apr  2 19:24 vpxd.cfg
-rw-r--r-- 1 root root   87 Apr  2 19:24 root.cfg
-rw-r--r-- 1 root root  217 Apr  2 19:24 MACHINE_SSL_CERT.cfg
-rw-r--r-- 1 root root  233 Apr  2 19:24 machine.cfg

Workflow (below are the commands that are run in the background)

/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
1. machine-6368669f-591a-44fa-bfb3-a76b166bfed6
2. vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
3. vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
4. vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6

Create certificate cfg files for the respective services

  • root.cfg
root@is-dhcp36-107 [ /var/tmp/vmware ]# cat root.cfg
Country = US
Name = VMCA
OrgUnit = VMware Engineering
State = California
#IPAddress =
  • Solution users


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat machine.cfg
Country = US
Name = machine-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vsphere-webclient.cfg
Country = US
Name = vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd.cfg
Country = US
Name = vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com



root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd-extension.cfg
Country = US
Name = vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com



  • Machine SSL.cfg
root@is-dhcp36-107 [ /var/tmp/vmware ]# cat MACHINE_SSL_CERT.cfg
Country = US
Name = is-dhcp36-107.isl.vmware.com
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com

Check if a backup store exist.. if not, create one.

/usr/lib/vmware-vmafd/bin/vecs-cli store list
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
vpxd
vpxd-extension
SMS

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name BACKUP_STORE
service-control --start vmafdd
service-control --start vmcad
service-control --start vmdird

Export exiting certificate and import them to the backup store

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/vmware/old_machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp___MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp___MACHINE_CERT --cert /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt --key /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_machine --cert /storage/certmanager/rollback/machine_bkp.crt --key /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vsphere-webclient --cert /storage/certmanager/rollback/vsphere-webclient_bkp.crt --key /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd --cert /storage/certmanager/rollback/vpxd_bkp.crt --key /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd-extension --cert /storage/certmanager/rollback/vpxd-extension_bkp.crt --key /storage/certmanager/rollback/vpxd-extension_bkp.priv

Generate Machine_SSL certificate

/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --selfca --config /var/tmp/vmware/root.cfg --server localhost
/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --pubkey=/storage/certmanager/MACHINE_SSL_CERT.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --cert=/storage/certmanager/MACHINE_SSL_CERT.crt --config=/var/tmp/vmware/MACHINE_SSL_CERT.cfg
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /storage/certmanager/MACHINE_SSL_CERT.crt --key /storage/certmanager/MACHINE_SSL_CERT.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT

Generate machine certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/machine.priv --pubkey=/storage/certmanager/machine.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/machine.priv --cert=/storage/certmanager/machine.crt --config=/var/tmp/vmware/machine.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/machine.crt --name machine-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert /storage/certmanager/machine.crt --key /storage/certmanager/machine.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine

Generate web client certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vsphere-webclient.priv --pubkey=/storage/certmanager/vsphere-webclient.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vsphere-webclient.priv --cert=/storage/certmanager/vsphere-webclient.crt --config=/var/tmp/vmware/vsphere-webclient.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vsphere-webclient.crt --name vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert /storage/certmanager/vsphere-webclient.crt --key /storage/certmanager/vsphere-webclient.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient

Generate vpxd certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd.priv --pubkey=/storage/certmanager/vpxd.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd.priv --cert=/storage/certmanager/vpxd.crt --config=/var/tmp/vmware/vpxd.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd.crt --name vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert /storage/certmanager/vpxd.crt --key /storage/certmanager/vpxd.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd

Generate vpxd-extension certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd-extension.priv --pubkey=/storage/certmanager/vpxd-extension.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd-extension.priv --cert=/storage/certmanager/vpxd-extension.crt --config=/var/tmp/vmware/vpxd-extension.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd-extension.crt --name vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert /storage/certmanager/vpxd-extension.crt --key /storage/certmanager/vpxd-extension.priv

Update EAM and autodeploy with vpxd-extension certificate

/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - administrator@vs.lo -p *****
/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - administrator@vs.lo -p *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
service-control --stop --ignore  --all
service-control --start  --all

VMware Converter Workflow for Linux conversions.

Step 1: Validate source

  • connect to source Linux VM via SSH (port 22 by default/specify port number in the IP address field if using custom ports
  • The account used must be a part of sudoers (must be able to run a sudo command without prompting for a password.
add the below line visudo
nik ALL=(ALL) NOPASSWD:ALL

Note: User "nik" to be replaced with the user have you on your setup.

Here’s an example from my setup

nik@mail:~$ sudo cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d
nik ALL=(ALL) NOPASSWD:ALL

Step 2: Validate Destination

  • connect to ESXi host/vCenter (port 80, 443, 902)
  • Polls for avilable Host (vCenter), Datastore, Networking portgroups, Space.
  • Virtual machine options, Select what Data to copy
  • Hardware resource (like vCPU, Memory, nics
  • Advanced Options like Power off source, Power on destination, Install tools
  • IP details for The Helper VM (if there is no DHCP on the enveronmnet)

A Helper VM is a standby operating environment (live boot) which needs a temporary IP address. The VM helps with the conversion and needs to be able to communicate with the VMware converter server (443) and the source virtual machine (22).

When the job is submitted, Converter creates a dummy virtual machine on the esxi host and boots it via ISO (helper VM iso: converter-helper-vm-x64.iso or converter-helper-vm.iso found at C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\

By Default, the root/user login for the helper VM is disabled. Inorder to enable this, change the config file located at:
C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-worker.xml

C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-worker.xml 

change:    <useSourcePasswordInHelperVm>false</useSourcePasswordInHelperVm>
to 
<useSourcePasswordInHelperVm>true</useSourcePasswordInHelperVm>

Restart all converter related services via services.msc

The root password will now be the same as the password that was used to connect to the source VM

when the Helper VM is on the network, it attempts to ssh into the source linux VM and runs a query like that to the below:

 ssh user@source_linux_IP -p 22 "sudo tar --one-file-system --sparse -C '/' -cf - ." | /us r/bin/tar --numeric-owner --delay-directory-restore -C '/home/p2vtest/' -xf 

Similarly, the other partitions are copied over to the Helper VM (review the helper-VM logs for others)

Once all volume’s are copied over, the filesystem (root and other volume’s, sda, sdb) are re-mapped with the respective path

The bootloader/GRUB is rebuilt (This is native to the version of linux have you, Running custom kernel can break this process)

 /usr/lib/vmware-converter/installGrub.sh

Restricting Plugins on vSphere web-client (Flex and html5)

on Certain occations when the vCenter is upgraded, Plugins like pure, Netapp, NSX, SRM etc can cause incompatibility and prevent the webclient from loading up/Malfunctioning

Starting vSphere 6.7, we have a config file to white list and black list plugins which helps us to flag specific Malicious/incompatible plugins which prevents the web client form loading up

the symptoms to this include
* web client hangs at a blue, loading screen.
* web client loads but does not display any inventory object
* web client does not allow to click on the inventory objects (home button and task and events are sometimes clickable)

The quickest way to sort out these specious plugin is by adding the plugin id to blacklist to the respective UI/flex file comparability config.

UI(html5):

/etc/vmware/vsphere-ui/compatibility-matrix.xml

root@is-dhcp36-107 [ /etc/vmware/vsphere-ui ]# cat compatibility-matrix.xml
<!--
    This file lets you define a WHITE LIST and a BLACK LIST of plugins to control your own setup.
    It overrides the internal black and white lists that are hard-coded in this release.

    Fling Note: until further notice all plugins are disabled by the HTML5 client except SDK samples.
    Use this file to re-enable specific HTML plugins during your testing.
-->
<Matrix>
    <pluginsCompatibility>
        <!--
            WHITE LIST:
            Add this to enable all plugins whose plugin-package id is com.acme.example.myplugin:
                <PluginPackage id="com.acme.myplugin" status="compatible"/>
            Or this to specify all versions greater or equal to 2.1.0:
                <PluginPackage id="com.acme.myplugin" version=[2.1.0,] status="compatible"/>
            Or this to enable all plugins starting with com.acme:
                <PluginPackage id="com.acme.*" status="compatible"/>
        -->

        <!--
            BLACK LIST:
            Add this to disable a plugin whose plugin-package id is com.acme.example.myplugin:
                <PluginPackage id="com.acme.myplugin" status="incompatible"/>
         -->

    </pluginsCompatibility>
</Matrix>

vsphere-client (flex)

/etc/vmware/vsphere-client/compatibility-matrix.xml

root@is-dhcp36-107 [ /etc/vmware/vsphere-client ]# cat compatibility-matrix.xml
<!--
   The 'id' value allows for standard java regular expressions. The actual plugin id
   is matched against this regular expression.

   The 'version' values must be a string in any of the following formats. If skipped
   it indicates any version.

   Version: An exact version, e.g. 6.0.0, 5.5, etc. The format allows for maximum
      of 4 dotted numbers.

   Interval: (, 6.4], [6.5, 7.0), (7.5, ). Use empty strings to mark infinity values.

   Range: Several versions and intervals can be mixed in one single string
      producing a set of value, e.g. ( ,6.4), 6.8.3, 6.9.1, [7.5.2.2, ).

      Each item in the set is separated from the others using comma ",".
      Each item in the set can be either version or interval.

   The 'status' can be any of the following strings: unknown,
      incompatible, compatible, certified
-->
<Matrix>
  <pluginsCompatibility>
     <!--
        'Incompatible' plugins are not loaded. You can use this to 'blacklist'
        plugins if needed. To prevent specific plugin package(s) from
        loading, use the template entry below as guidance how to add new records
        with the actual id and version of the plugin you want to prevent from loading.

        The plugin ids can be taken from the plugin-package.xml file of each plugin.

        A few well known set of plugins locations (on the vCenter Appliance):
           /usr/lib/vmware-vsphere-client/plugin-packages/
           /etc/vmware/vsphere-client/vc-packages
           /etc/vmware/vsphere-client/cm-service-packages

        On Windows the following locations can be checked:
           <INSTALL DRIVE>\ProgramData\vCenterServer\runtime\vsphere-client\plugin-packages
           <INSTALL DRIVE>\ProgramData\vCenterServer\cfg\vsphere-client\vc-packages
           <INSTALL DRIVE>\ProgramData\vCenterServer\cfg\vsphere-client\cm-service-packages
     -->
     <!--
     <PluginPackage id="com.foo.plugin.id" version="1.0.0" status="incompatible"/>
     <PluginPackage id="net.bar.plugin.id" version="(,2.1]" status="incompatible"/>
     -->



     <!--
        The sample section below shows 'whitelist' definition. Compatible plugins
        are loaded. All others are declared as incompatible (due to the id regex),
        thus effectively forbidding them.

        The approach limits the list of plugins loaded only to a small 'white' list.
        This allows for vsphere-client to work in a 'safe-like' mode.

        Below are predefined sets of plugins for your convenience:
        1st-party, core:
          com.vmware.vsphere.client,
          com.vmware.ds,
          com.vmware.ssoadminui,
          com.vmware.vsphere.client.modules,
          com.vmware.license.client,
          com.vmware.opsmgmt.client

        1st-party extended (in addition to the above):
          com.vmware.loganalyzer,
          com.vmware.vsphere.client.telemetry

        2nd-party (basically anything that comes already pre-bundled with the
        vCenter Appliance and is not in the above two sets):
            com.vmware.vca.marketing.ngc.ui,
            com.vmware.vco
     -->
     <!--
     <PluginPackage id="com.vmware.vsphere.(client|ds|ssoadminui|client.modules)" status="compatible"/>
     <PluginPackage id="com.vmware.license.client" status="compatible"/>
     <PluginPackage id="com.vmware.opsmgmt.client" status="compatible"/>

     <PluginPackage id=".*" status="incompatible"/>
     -->
  </pluginsCompatibility>
</Matrix>

The plugin ID can be found on the extension manager under vCenter MOB. https://vc-fqdn/mob

recovering from an unresponsive hostd after a datastore/storage goes PDL

Hostd crashes with the below:

Hostd.log
2018-12-17T22:37:50.138Z info hostd[9130B80] [Originator@6876 sub=Hostsvc] Storage data synchronization policy set to invalidate_change
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: OpenSSL using FIPS_drbg for RAND
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: cipher list !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES
2018-12-17T22:37:50.141Z info hostd[9130B80] [Originator@6876 sub=Libs] GetTypedFileSystems: fstype vfat
2018-12-17T22:37:50.141Z info hostd[9130B80] [Originator@6876 sub=Libs] GetTypedFileSystems: uuid 579bba34-1440dc54-3308-70106f411e18   <-----volume that went offfline
2018-12-17T22:37:51.136Z info hostd[9AB8B70] [Originator@6876 sub=ThreadPool] Thread enlisted
 
VMkernel:
2018-12-17T22:19:15.397Z cpu18:65910)VMW_SATP_LOCAL: satp_local_updatePath:789: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2018-12-17T22:19:18.801Z cpu14:65607)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.00a0504658335330" state in doubt; requested fast path state update...
2018-12-17T22:19:18.801Z cpu14:65607)ScsiDeviceIO: 2968: Cmd(0x439d4981c540) 0x1a, CmdSN 0x6d2dc7 from world 0 to dev "eui.00a0504658335330" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:21.394Z cpu6:5268773)ScsiPath: 5115: Command 0x0 (cmdSN 0x0, World 0) to path vmhba32:C0:T0:L2 timed out: expiry time occurs 1002ms in the past
2018-12-17T22:19:21.394Z cpu6:5268773)VMW_SATP_LOCAL: satp_local_updatePath:789: Failed to update path "vmhba32:C0:T0:L2" state. Status=Transient storage condition, suggest retry
2018-12-17T22:19:22.892Z cpu22:65615)ScsiDeviceIO: 2968: Cmd(0x439d4981c540) 0x1a, CmdSN 0x6d2dc7 from world 0 to dev "eui.00a0504658335330" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:23.400Z cpu34:65627)NMP: nmp_ThrottleLogForDevice:3593: last error status from device eui.00a0504658335330 repeated 1 times
2018-12-17T22:19:23.400Z cpu34:65627)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x1a (0x439d4989a540, 0) to dev "eui.00a0504658335330" on path "vmhba32:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2018-12-17T22:19:23.400Z cpu34:65627)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.00a0504658335330" state in doubt; requested fast path state update...
2018-12-17T22:19:23.400Z cpu34:65627)ScsiDeviceIO: 2968: Cmd(0x439d4989a540) 0x1a, CmdSN 0x6d2dc8 from world 0 to dev "eui.00a0504658335330" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:26.798Z cpu14:65607)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x1a (0x439d4a8ed5c0, 0) to dev "eui.00a0504658335331" on path "vmhba32:C0:T0:L1" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2018-12-17T22:19:26.798Z cpu14:65607)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.00a0504658335331" state in doubt; requested fast path state update...
2018-12-17T22:19:26.798Z cpu14:65607)ScsiDeviceIO: 2968: Cmd(0x439d4a8ed5c0) 0x1a, CmdSN 0x6d2dd8 from world 0 to dev "eui.00a0504658335331" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x31 0x22 0x20.

Scsi Decoder: Link

In my case, The volume appeared to have gone offline because the host was aborting the commands to the HBA.

/etc/init.d/hostd status
hostd is not running.
However,
ps | grep hostd
2098894 2098894 hostdCgiServer
2105175 2105175 hostd
2105176 2105175 hostd-worker
2105177 2105175 hostd-worker
2105178 2105175 hostd-worker
2105179 2105175 hostd-worker
2105180 2105175 hostd-IO
2105181 2105175 hostd-IO
2105182 2105175 hostd-fair
2105183 2105175 hostd-worker
2105184 2105175 hostd-worker
2105185 2105175 hostd-worker
2105187 2105175 hostd-worker
2105191 2105175 hostd-worker
2105192 2105175 hostd-worker
2105193 2105175 hostd-worker
2105194 2105175 hostd-worker
2105251 2105175 hostd-poll
localcli storage core device world list
Device World ID Open Count World Name
------------------------------------------------------------------------------------------------------
mpx.vmhba32:C0:T0:L0 2099479 1 smartd
mpx.vmhba32:C0:T0:L0 2105105 1 vpxa
mpx.vmhba32:C0:T0:L0 2105175 1 hostd
naa.600508b1001c555e5048cfd74e058fdc 2097185 1 idle0
naa.600508b1001c555e5048cfd74e058fdc 2097403 1 OCFlush
naa.600508b1001c555e5048cfd74e058fdc 2098198 1 Res6AffinityMgrWorld
naa.600508b1001c555e5048cfd74e058fdc 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001c555e5048cfd74e058fdc 2099479 1 smartd
naa.600508b1001c555e5048cfd74e058fdc 2105175 1 hostd
naa.6001405d7dc7524f3364522a27b7c508 2097185 1 idle0
naa.6001405d7dc7524f3364522a27b7c508 2099760 1 fdm
naa.6001405d7dc7524f3364522a27b7c508 2099766 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2099771 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2100189 1 J6AsyncReplayManager
naa.6001405d7dc7524f3364522a27b7c508 2100219 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2105175 1 hostd
naa.6001405d7dc7524f3364522a27b7c508 2105286 1 hostd-worker
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2097185 1 idle0
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2097446 1 bcflushd
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2098539 1 J6AsyncReplayManager
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2105105 1 vpxa
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2105175 1 hostd
naa.600508b1001c5a5167700b7ae7160e91 2097185 1 idle0
naa.600508b1001c5a5167700b7ae7160e91 2097403 1 OCFlush
naa.600508b1001c5a5167700b7ae7160e91 2098198 1 Res6AffinityMgrWorld
naa.600508b1001c5a5167700b7ae7160e91 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001c5a5167700b7ae7160e91 2099479 1 smartd
naa.600508b1001c5a5167700b7ae7160e91 2105175 1 hostd
naa.600508b1001ca2c68c28022b4447710f 2097185 1 idle0
naa.600508b1001ca2c68c28022b4447710f 2097403 1 OCFlush
naa.600508b1001ca2c68c28022b4447710f 2098198 1 Res6AffinityMgrWorld
naa.600508b1001ca2c68c28022b4447710f 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001ca2c68c28022b4447710f 2099479 1 smartd
naa.600508b1001ca2c68c28022b4447710f 2105175 1 hostd

This shows that hostd still appears to be stuck as running in a zombie state.

To resolve this, we will need to

  • reset scsi commands to vmhba32 (vmkfstools -B /vmfs/volume/disk/naa.xxxx)
  • Rescan vmhba32 and wait for 3-5 min (esxcfg-rescan vmhba32)
  • confirm that hostd is no longer running for that device  (localcli storage core device world list and ps | grep hostd)
  • start hostd

Worst case scenario, Host reboot.

PostgreSQL: could not open file “/var/lib/pgsql/data/pg_clog/0726”: No such file or directory

when selecting a specific date from a table, at times due to database corruption Postgres might report the file was not found.

hsphere=# select * from TABLE;
ERROR:  could not access status of transaction 1918986094
DETAIL:  could not open file "/var/lib/pgsql/data/pg_clog/0726": No such file or directory

In Such instances, we will can re-create the file using the below dd command (note: data loss on the missing file)

dd if=/dev/zero of=/var/lib/pgsql/data/pg_clog/0726 bs=256k count=1

Note: Ensure the permissions on the file that was re-created are set appropriately. creating the blank file means that the data that was originally on it might actually go missing. The respective records will need to be re-created or its relevant records (constraints etc) will need to be removed from the DB.

joining Esxi to domain fails with LW_ERROR_INVALID_MESSAGE [code 0x00009c46]

Joining Esxi to doamin fails with the below message:

[root@esx:~] ./usr/lib/vmware/likewise/bin/domainjoin-cli join ikigo.net
Joining to AD Domain: ikigo.net
	With Computer DNS Name: esx.
	adm@ikigo.net's password:
	Error: LW_ERROR_INVALID_MESSAGE [code 0x00009c46]
	The Inter Process message is invalid
 

Cat /etc/hosts show below:

Cat /etc/hosts
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost.localdomain localhost
192.168.1.104 esx. esx      <------------------notice the "esx."

In some cases, the hostname on the /etc/hosts file might not match to that of the host.  (look for the message on domainjoin-join “With Computer DNS Name: esx.”

In order to sort this out,  correct the /etc/host file (in my case, I had to suffix the DNS suffix)

Cat /etc/hosts:
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost.localdomain localhost
192.168.1.104 esx.ikigo.net esx
 

restart lwsmd

/etc/init.d/lwsmd start

Re-attempt doamin join CLI

Note: In some cases after performing the change to /etc/hosts, joining the host via powerCLI will fail with error “‘ vmwauth InvalidHostNameException: The current hostname is invalid: hostname cannot be resolved”

Hostd.log:
2019-01-07T19:20:16.288Z error hostd[DEC1B70] [Originator@6876 sub=ActiveDirectoryAuthentication opID=500fc81b-b5-602f user=vpxuser:IKIGO\nik] vmwauth InvalidHostNameException: The current hostname is invalid: hostname cannot be resolved

The management agents/host will need a restart to sort this out.

services.sh restart

Installing VMware PowerCLi Install on windows 10/2012/2016 via Windows Gallery

Before you proceed, ensure there are no older version of VMware PowerCLi installed (check in add or remove programs), uninstall if you do have the older version.

Install PowerCli from Windows Gallery

Open powerCLI as an administrator and then run:

Install-Module -Name VMware.PowerCLI

List available VMware module:

Get-Module -ListAvailable -Name VMware*

Import Module

 Get-Module vmware* | Import-Module

Troubleshooting






Import module’s generally fail with the below error:

PS C:\windows\system32> Import-Module VMware*
>>
Import-Module : File C:\Program
Files\WindowsPowerShell\Modules\VMware.VimAutomation.Sdk\11.0.0.10334495\VMware.VimAutomation.Sdk.psm1 cannot be
loaded because running scripts is disabled on this system. For more information, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ Import-Module VMware.VimAutomation.Core
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : SecurityError: (:) [Import-Module], PSSecurityException
 + FullyQualifiedErrorId : UnauthorizedAccess,Microsoft.PowerShell.Commands.ImportModuleCommand

Set execution policy to remote signed if the importing fails:

Set-ExecutionPolicy RemoteSigned






If vCenter is running on the default self signed certificates: The connect-viserver is likely to fail

If the above command is not run, we would see an error similar to below:
Connect-VIServer : 12/21/2018 2:05:00 AM Connect-VIServer Error: Invalid server certificate. Use
Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you'd like to connect
once or to add a permanent exception for this server.
Additional Information: Could not establish trust relationship for the SSL/TLS secure channel with authority
'vc.ikigo.net'.
At line:1 char:1
+ Connect-VIServer vc.FQDN.Domain
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : SecurityError: (:) [Connect-VIServer], ViSecurityNegotiationException
 + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_CertificateError,VMware.VimAutomation.ViCore.
 Cmdlets.Commands.ConnectVIServer

Set invalid certificate to ignore on the computer to work this around

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore


Connecting to vCenter using PowerCli

connect-viserver vc.ikigo.net

on a successful connect, you should see the below

Passing through a Nvidia GTX970 to a VMware Virtual machine

Create a virtual machine with EFI boot, install windows + tools and power down the VM

Open ssh to the host where the VM is registered to

use vim-cmd to look for the VM

vim-cmd vmsvc/getallvms |   grep *name_of_VM* 

cd to the data store path

cd /vmfs/volumes/NVME/nvidia/

Edit the vmx file using the vi editor and add the below lines

hypervisor.cpuid.v0 = "FALSE"
pciHole.dynStart = "2816"

Save the editor and then reload the VMX (201 is the vmid from the above screenshot)

vim-cmd vmsvc/reload 201

Edit settings on the VM> add new hardware>pci device>add graphic card and its audio device(2 pci pass through device and set the memory reservation to maximum)

Take a snapshot and then power on the VM

Conform the hardware under graphic adapter followed by nvidia drivers installation.

Device manager

Note: The nvidia card and its audio device must already have been marked as under pci device as pass through!!

Creating a Cent OS7 template for VMware Guest Customization and Deployment

To start off with, Download the latest version of CentOs7 from https://www.centos.org/download/

At the time of writing, this is: CentOS-7 (1804)

Lets start by creating a new Virtual machine.  I will select Esxi 6.5 for backward compatibility  with other host.

Mount the ISO,

Power on the VM and begin installing of the OS and begin the installation


Note: During the install we will enable the default nic interface and set this to DHCP

Note: Since the ISO used was from a most recent release, openvm-tools is auto installed along with the linux installer. if you are using an older version of the cent os installer iso, you must install open-vm-tools with the below command (will need the VM connected to the internet).

yum install open-vm-tools

I would recommend updating the tools to the latest release.

then followed by installing pearl (pre-requisites for guest customization)

yum install perl

Once done, power down the VM and convert it to a template.

Test the template by deploying a VM with guest customization

When the VM boots up you should see the host name set to the name of the VM (the spec that I used to customize uses the name as of vsphere inventory as the the virtual machine name)

Looking at the VM that was just deployed, we see the host name has changed as per the specification.

Troubleshooting:
Log file for guest cust:

/var/log/vmware-imc/toolsDeployPkg.log

Generate a memory dump for a frozen VMware, windows virtual machine.

Enable complete memory dump feature by changing following registry keys:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl
CrashDumpEnabled    REG_DWORD    0x1

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
PagingFiles    REG_MULTI_SZ    c:\pagefile.sys 13312 13312


Enable keyboard crash dump feature by adding following registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters
Value Name: CrashOnCtrlScroll
Data Type:    REG_DWORD
Value:    1

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\kbdhid\Parameters
Value Name: CrashOnCtrlScroll
Data Type:    REG_DWORD
Value:    1

Enable NMI crash dump feature by adding following key:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl
Value Name: NMICrashDump
Data Type:  REG_DWORD
Value:   1

  • Restart Server to take effects.
  • Do a memory dump test with below steps:

Capture a kernel memory dump in following ways:

o    Send NMI to Guest OS Link:How to send NMI to Guest OS on ESXi 6.x (2149185)

Or

o   On VM console, press Right Ctrl + Scroll Lock button 2 times.

Machine crashes into blue screen and save a memory dump, restart automatically once dump generation reaches 100%. You should be able to see 12GB (memory allocated to the VM) here: C:\Windows\MEMORY.DMP file.