Fixing the broken/corrupt Locker Partition on Esxi

Start by determining the device backing up the locker partition

commands: 
ls -ltrh / | grep store
 vmkfstools -P /vmfs/volumes/5cdce747-375af1f6-b185-0050569674de
Output: 
[root@is-dhcp41-13:~] ls -ltrh / | grep store
lrwxrwxrwx    1 root     root           6 May 13 23:03 locker -> /store
lrwxrwxrwx    1 root     root          49 May 16 04:29 store -> /vmfs/volumes/5cdce747-375af1f6-b185-0050569674de


[root@is-dhcp41-13:~] vmkfstools -P /vmfs/volumes/5cdce747-375af1f6-b185-0050569674de
vfat-0.04 (Raw Major Version: 0) file system spanning 1 partitions.
File system label (if any):
Mode: private
Capacity 299712512 (36586 file blocks * 8192), 299712512 (36586 blocks) avail, max supported file size 0
Disk Block Size: 512/0/0
UUID: 5cdce747-375af1f6-b185-0050569674de
Partitions spanned (on "disks"):
        mpx.vmhba0:C0:T0:L0:8
Is Native Snapshot Capable: NO
[root@is-dhcp41-13:~]

Make a note of the device under the line Partitions spanned (on “disks”):

Note: The :8 on the above result signifives that this is partition 8 of the disk
Note: On a default install, the locker/tools iso are always stored to partition 8 of the installed disk/drive.

Format the partition with fat filesystem using the below command: Ensure you DO NOT MISS the partition number

vmkfstools -C vfat /dev/disks/mpx.vmhba0:C0:T0:L0:8
eg:
[root@is-dhcp41-13:~] vmkfstools -C vfat /dev/disks/mpx.vmhba0:C0:T0:L0:8
create fs deviceName:'/dev/disks/mpx.vmhba0:C0:T0:L0:8', fsShortName:'vfat', fsName:'(null)'
deviceFullPath:/dev/disks/mpx.vmhba0:C0:T0:L0:8 deviceFile:mpx.vmhba0:C0:T0:L0:8
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vfat file system on "mpx.vmhba0:C0:T0:L0:8" with blockSize 1048576 and volume label "none".
Successfully created new volume: 5cdcf45e-68f98eec-adb0-0050569674de


Note: If the format fails with the resource in use errors, the host will need a reboot.

re-create the symlink for store:

ln -snf /vmfs/volumes/5cdcf45e-68f98eec-adb0-0050569674de /store

Copy contents of the store partition from a working host, same Esxi build

cron jobs on vcsa 6.7

root@is-dhcp40-236 [ /etc/cron.d ]# cat nuke_logs.cron
* /1 * * * *   root . /usr/sbin/nukedns.sh >/dev/null 2>&1

root@is-dhcp40-236 [ /etc/cron.d ]# cat /usr/sbin/nukedns.sh
echo  0 > /var/log/vmware/dnsmasq.log
echo  0 > /var/log/vmware/other_logs_that_that_needs_to_be_nulled


change /1 to x for the min, duration 

permission for cron file must be 666 or 700

an example can be found in the attachment for
https://kb.vmware.com/s/article/54526 (use WinRAR to extract the attachment, the file shows up as corrupt otherwise)

vCenter Webclient logon screen glitches after upgrade

After vCenter upgrade, the Logon screen is improperly formatted and might look like the below:

the text would read like the below:
<img id=\'topSplash\' src=\'..\/..\/resources\/img\/AppBgPattern.png\'><img id=\'brand\' src=\'..\/..\/resources\/img\/vmwareLogoBigger.png\'><span>VMware<sup>®<\/sup> vCloud Automation Center<sup>™<\/sup><\/span><style type=\'text\/css\'>body { background: #3075ab; \/* Old browsers *\/ background: -moz-linear-gradient(top, #3a8dc8 0%, #183a62 100%); \/* FF3.6+ *\/ background: -webkit-gradient(linear, left top, left bottom, color-stop(0%, #3a8dc8), color-stop(100%, #183a62)); \/* Chrome,Safari4+ *\/ background: -webkit-linear-gradient(top, #3a8dc8 0%, #183a62 100%); \/* Chrome10+,Safari5.1+ *\/ background: -o-linear-gradient(top, #3a8dc8 0%, #183a62 100%); \/* Opera 11.10+ *\/ background: -ms-linear-gradient(top, #3a8dc8 0%, #183a62 100%); \/* IE10+ *\/ background: linear-gradient(to bottom, #3a8dc8 0%, #183a62 100%); \/* W3C *\/ filter: progid:DXImageTransform.Microsoft.gradient( startColorstr=\'#3a8dc8\', endColorstr=\'#183a62\', GradientType=0); \/* IE6-9 *\/ background-repeat: no-repeat; margin : 0; font-size : 12px; font-family : Arial, Helvetica, sans-serif; color: #87ceff; margin: 0; font-size: 12px; font-family: Arial, Helvetica, sans-serif;}#topSplash { position: absolute; top: 0; left: 0; z-index: 1;}#brand { position: absolute; top: 55px; left: 44px; z-index: 2;}#tenantBrand { top: 0; left: 0; margin: 0; padding: 0; width: 100%;}#tenantBrand span { position: absolute; top: 345px; left: 424px; color: #FFF; font-size: 21px;}#tenantBrand sup { font-size: 11px;}#loginForm { background-image: url(..\/..\/resources\/img\/divider.png);}.loginLabel { color: #FFFFFF;}#productName { top: 365px;}#response { color: #87CEFF;}#footer { background-color: 090B0D; color: #838689;}<\/style> 

or

 var tenant_brandname="<img id=\'topSplash\' src=\'..\/..\/resources\/img\/AppBgPattern.png\'><img id=\'brand\' src=\'..\/..\/resources\/img\/vmwareLogoBigger.png\'><span>VMware<sup>®<\/sup> vRealize<sup>™<\/sup> Automation<\/span><style type=\'text\/css\'>body {    background: #3075ab; \/* Old browsers *\/    background: -moz-linear-gradient(top, #3a8dc8 0%, #183a62 100%);    \/* FF3.6+ *\/    background: -webkit-gradient(linear, left top, left bottom, color-stop(0%, #3a8dc8),        color-stop(100%, #183a62)); \/* Chrome,Safari4+ *\/    background: -webkit-linear-gradient(top, #3a8dc8 0%, #183a62 100%);    \/* Chrome10+,Safari5.1+ *\/    background: -o-linear-gradient(top, #3a8dc8 0%, #183a62 100%);    \/* Opera 11.10+ *\/    background: -ms-linear-gradient(top, #3a8dc8 0%, #183a62 100%);    \/* IE10+ *\/    background: linear-gradient(to bottom, #3a8dc8 0%, #183a62 100%);    \/* W3C *\/    filter: progid:DXImageTransform.Microsoft.gradient( startColorstr=\'#3a8dc8\',        endColorstr=\'#183a62\', GradientType=0); \/* IE6-9 *\/    background-repeat: no-repeat; margin : 0; font-size : 12px; font-family    : Arial, Helvetica, sans-serif;    color: #87ceff;    margin: 0;    font-size: 12px;    font-family: Arial, Helvetica, sans-serif;}#topSplash {    position: absolute;    top: 0;    left: 0;    z-index: 1;}#brand {    position: absolute;    top: 55px;    left: 44px;    z-index: 2;}#tenantBrand {    top: 0;    left: 0;    margin: 0;    padding: 0;    width: 100%;}#tenantBrand span {    position: absolute;    top: 345px;    left: 499px;    color: #FFF;    font-size: 21px;}#tenantBrand sup {    font-size: 11px;}#loginForm {    background-image: url(..\/..\/resources\/img\/divider.png);}.loginLabel {    color: #FFFFFF;}#productName {    top: 365px;}#response {    color: #87CEFF;}#footer {    background-color: 090B0D;    color: #838689;}<\/style>";

This is because the STS banner flag has an inappropriate data. Inorder to fix this, download and connect to the sso using jxplorer: https://kb.vmware.com/s/article/2077170

Note: Take a snapshot of the PSC/backup the vmdird database (/storage/db/vmware-vmdir/lock.mdb) before proceeding, deleting the wrong object can break the psc/vCenter.

Delete/remove the value on the attribute ‘vmwSTSBrandName’ under the object dn ‘cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local’ using jxplorer
(screenshot below)

Duplicate STS signing certificates can cause authentication failure.

Log:

04-15T23:27:57.946Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.sso.client.exception.TimeSynchronizationException: Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Mon Apr 15 23:27:57 UTC 2019, endTime=Tue Apr 16 01:07:57 UTC 2019] :: Signing certificate is not valid at Mon Apr 15 23:27:57 UTC 2019, cert validity: TimePeriod [startTime=Wed Apr 11 18:17:03 UTC 2018, endTime=Thu Apr 11 18:17:03 UTC 2019]
        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:182)
        at com.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:77)
        at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:54)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.sso.client.exception.TimeSynchronizationException: Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Mon Apr 15 23:27:57 UTC 2019, endTime=Tue Apr 16 01:07:57 UTC 2019] :: Signing certificate is not valid at Mon Apr 15 23:27:57 UTC 2019, cert validity: TimePeriod [startTime=Wed Apr 11 18:17:03 UTC 2018, endTime=Thu Apr 11 18:17:03 UTC 2019]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:1016)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:932)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:856)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:477)
        at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:179)
        ... 12 more

The duplicate certificate needs to be deleted via jxplorer, refer to the screenshot above (from the above example, trustedcertchain-2 and trusted credential-2 had the same certificate which had to be deleted from the system)

Handy commands when working with vPostgres

Copy Tables to CSV

copy vpx_vm to '/tmp/vpx_vm' DELIMITER ',' CSV;

Copy tables from CSVto table

copy table FROM '/tmp/table.csv' DELIMITER ',' CSV;

list Top tables by size:

SELECT
  schema_name,
  relname,
  pg_size_pretty(table_size) AS size,
  table_size

FROM (
       SELECT
         pg_catalog.pg_namespace.nspname           AS schema_name,
         relname,
         pg_relation_size(pg_catalog.pg_class.oid) AS table_size

       FROM pg_catalog.pg_class
         JOIN pg_catalog.pg_namespace ON relnamespace = pg_catalog.pg_namespace.oid
     ) t
WHERE schema_name NOT LIKE 'pg_%'
ORDER BY table_size DESC;

postgres password:

root@is-dhcp34-161 [ / ]# cat ~/.pgpass

localhost:5432:replication:replicator:*v&w1pTkmZY}Q2<z
127.0.0.1:5432:replication:replicator:*v&w1pTkmZY}Q2<z
/var/run/vpostgres:5432:replication:replicator:*v&w1pTkmZY}Q2<z
localhost:5432:postgres:postgres:_ouG|OZ4NUwna0fB
127.0.0.1:5432:postgres:postgres:_ouG|OZ4NUwna0fB
localhost:5432:VCDB:postgres:_ouG|OZ4NUwna0fB
127.0.0.1:5432:VCDB:postgres:_ouG|OZ4NUwna0fB
/var/run/vpostgres:5432:VCDB:postgres:_ouG|OZ4NUwna0fB
/var/run/vpostgres:5432:postgres:postgres:_ouG|OZ4NUwna0f

Rename table

ALTER TABLE tablename RENAME TO new_table

Database backup/Dump using pg_dump

pg_dump  VCDB -U postgres > /tmp/dump

Database backup (excluding a specific corrupted table)

pg_dump  VCDB -U postgres  -T vpx_host > /tmp/dump.excluting.currupt.table

Note: T: Exclude table and grab the rest.
      t: backup specific table only.

Determining broken tables(pg toast)

 for ((i=0; i<"668"; i++ )); do /opt/vmware/vpostgres/current/bin/psql -U "postgres" "VCDB" -c "SELECT * FROM VPX_VM LIMIT 1 offset $i" >/dev/null || echo $i; done

Note: Replace 668 with the highest table count on your setup

Replacing VMware-stsd certificate on 6.5/6.7

If the vCenter was upgraded from 5.5, it retains legacy endpoints for lookup service
IE: https://FQDN:7444/lookupservce/sdk
Running certificate-manager will not replace the certificate and vCenter might actually start to complain about expired certificate although they are valid.

In this can be easily worked around by replacing the certificate on the VECS store:

Start by exporting the STS_INTERNAL_SSL_CERT and MACHINE_SSL_CERT

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /ssl/machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /ssl/machine_ssl.key


/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /ssl/STS_INTERNAL_SSL_CERT-__MACHINE_CERT.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output /ssl/STS_INTERNAL_SSL_CERT-__MACHINE_CERT.key

Delete the contents of STS_INTERNAL_SSL_CERT store

 /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y

import machine_ssl store to the STS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert  /ssl/machine_ssl.crt --key /ssl/machine_ssl.key 

Restart services and confirm service status

Understanding VMCA, Manual VMCA Certificates replacement for vCenter.

VMCA is the Default self-signed certificates that is set up at the time of vCenter deployment.

All solution users and machine SSL certificates are signed with this certificate.

VMCA certificates can be regenerated by using option 8 on the certificatae manager.

root@is-dhcp36-107 [ / ]# /usr/lib/vmware-vmca/bin/certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 6.5 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]:

choosing option 8, you are presented with the below options:

Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@vs.lo
Enter password:

Please configure certool.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] :

Enter proper value for 'Name' [Default value : CA] :

Enter proper value for 'Organization' [Default value : VMware] :

Enter proper value for 'OrgUnit' [Default value : VMware Engineering] :

Enter proper value for 'State' [Default value : California] :

Enter proper value for 'Locality' [Default value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] :

Enter proper value for 'Email' [Default value : email@acme.com] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : is-dhcp36-107.isl.vmware.com

Enter proper value for VMCA 'Name' :VMCA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y

Below are the .cfg’s created for the certificate

root@is-dhcp36-107 [ /var/tmp/vmware ]# ls -ltrh
total 32K
drwxr-xr-x 3 root root 4.0K Mar 14 00:07 cis-license
-rw-r--r-- 1 root root  191 Apr  2 19:24 certool.cfg
-rw-r--r-- 1 root root  243 Apr  2 19:24 vsphere-webclient.cfg
-rw-r--r-- 1 root root  240 Apr  2 19:24 vpxd-extension.cfg
-rw-r--r-- 1 root root  230 Apr  2 19:24 vpxd.cfg
-rw-r--r-- 1 root root   87 Apr  2 19:24 root.cfg
-rw-r--r-- 1 root root  217 Apr  2 19:24 MACHINE_SSL_CERT.cfg
-rw-r--r-- 1 root root  233 Apr  2 19:24 machine.cfg

Workflow (below are the commands that are run in the background)

/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
1. machine-6368669f-591a-44fa-bfb3-a76b166bfed6
2. vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
3. vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
4. vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6

Create certificate cfg files for the respective services

  • root.cfg
root@is-dhcp36-107 [ /var/tmp/vmware ]# cat root.cfg
Country = US
Name = VMCA
OrgUnit = VMware Engineering
State = California
#IPAddress =
  • Solution users


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat machine.cfg
Country = US
Name = machine-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vsphere-webclient.cfg
Country = US
Name = vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com


root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd.cfg
Country = US
Name = vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com



root@is-dhcp36-107 [ /var/tmp/vmware ]# cat vpxd-extension.cfg
Country = US
Name = vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com



  • Machine SSL.cfg
root@is-dhcp36-107 [ /var/tmp/vmware ]# cat MACHINE_SSL_CERT.cfg
Country = US
Name = is-dhcp36-107.isl.vmware.com
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
#IPAddress =
Email = email@acme.com
Hostname = is-dhcp36-107.isl.vmware.com

Check if a backup store exist.. if not, create one.

/usr/lib/vmware-vmafd/bin/vecs-cli store list
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
vpxd
vpxd-extension
SMS

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name BACKUP_STORE
service-control --start vmafdd
service-control --start vmcad
service-control --start vmdird

Export exiting certificate and import them to the backup store

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/vmware/old_machine_ssl.crt
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp___MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp___MACHINE_CERT --cert /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.crt --key /storage/certmanager/rollback/MACHINE_SSL_CERT_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /storage/certmanager/rollback/machine_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_machine --cert /storage/certmanager/rollback/machine_bkp.crt --key /storage/certmanager/rollback/machine_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /storage/certmanager/rollback/vsphere-webclient_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vsphere-webclient --cert /storage/certmanager/rollback/vsphere-webclient_bkp.crt --key /storage/certmanager/rollback/vsphere-webclient_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /storage/certmanager/rollback/vpxd_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd --cert /storage/certmanager/rollback/vpxd_bkp.crt --key /storage/certmanager/rollback/vpxd_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /storage/certmanager/rollback/vpxd-extension_bkp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store BACKUP_STORE --alias bkp_vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias bkp_vpxd-extension --cert /storage/certmanager/rollback/vpxd-extension_bkp.crt --key /storage/certmanager/rollback/vpxd-extension_bkp.priv

Generate Machine_SSL certificate

/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --selfca --config /var/tmp/vmware/root.cfg --server localhost
/usr/lib/vmware-vmca/bin/certool --getrootca --server localhost
/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --pubkey=/storage/certmanager/MACHINE_SSL_CERT.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/MACHINE_SSL_CERT.priv --cert=/storage/certmanager/MACHINE_SSL_CERT.crt --config=/var/tmp/vmware/MACHINE_SSL_CERT.cfg
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store MACHINE_SSL_CERT --alias __MACHINE_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /storage/certmanager/MACHINE_SSL_CERT.crt --key /storage/certmanager/MACHINE_SSL_CERT.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store MACHINE_SSL_CERT --alias __MACHINE_CERT

Generate machine certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/machine.priv --pubkey=/storage/certmanager/machine.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/machine.priv --cert=/storage/certmanager/machine.crt --config=/var/tmp/vmware/machine.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/machine.crt --name machine-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store machine --alias machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert /storage/certmanager/machine.crt --key /storage/certmanager/machine.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store machine --alias machine

Generate web client certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vsphere-webclient.priv --pubkey=/storage/certmanager/vsphere-webclient.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vsphere-webclient.priv --cert=/storage/certmanager/vsphere-webclient.crt --config=/var/tmp/vmware/vsphere-webclient.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vsphere-webclient.crt --name vsphere-webclient-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vsphere-webclient --alias vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert /storage/certmanager/vsphere-webclient.crt --key /storage/certmanager/vsphere-webclient.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vsphere-webclient --alias vsphere-webclient

Generate vpxd certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd.priv --pubkey=/storage/certmanager/vpxd.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd.priv --cert=/storage/certmanager/vpxd.crt --config=/var/tmp/vmware/vpxd.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd.crt --name vpxd-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd --alias vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert /storage/certmanager/vpxd.crt --key /storage/certmanager/vpxd.priv
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd --alias vpxd

Generate vpxd-extension certificate

/usr/lib/vmware-vmca/bin/certool --genkey --privkey=/storage/certmanager/vpxd-extension.priv --pubkey=/storage/certmanager/vpxd-extension.pub --server=localhost
/usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/storage/certmanager/vpxd-extension.priv --cert=/storage/certmanager/vpxd-extension.crt --config=/var/tmp/vmware/vpxd-extension.cfg
/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/dir-cli service update --cert /storage/certmanager/vpxd-extension.crt --name vpxd-extension-6368669f-591a-44fa-bfb3-a76b166bfed6 --login administrator@vs.lo --password *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store vpxd-extension --alias vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert /storage/certmanager/vpxd-extension.crt --key /storage/certmanager/vpxd-extension.priv

Update EAM and autodeploy with vpxd-extension certificate

/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - administrator@vs.lo -p *****
/usr/bin/python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -s is-dhcp36-107.isl.vmware.com -c /storage/certmanager/vpxd-extension.crt -k /storage/certmanager/vpxd-extension.priv - administrator@vs.lo -p *****
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --text --store vpxd-extension --alias vpxd-extension
service-control --stop --ignore  --all
service-control --start  --all

VMware Converter Workflow for Linux conversions.

Step 1: Validate source

  • connect to source Linux VM via SSH (port 22 by default/specify port number in the IP address field if using custom ports
  • The account used must be a part of sudoers (must be able to run a sudo command without prompting for a password.
add the below line visudo
nik ALL=(ALL) NOPASSWD:ALL

Note: User "nik" to be replaced with the user have you on your setup.

Here’s an example from my setup

nik@mail:~$ sudo cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d
nik ALL=(ALL) NOPASSWD:ALL

Step 2: Validate Destination

  • connect to ESXi host/vCenter (port 80, 443, 902)
  • Polls for avilable Host (vCenter), Datastore, Networking portgroups, Space.
  • Virtual machine options, Select what Data to copy
  • Hardware resource (like vCPU, Memory, nics
  • Advanced Options like Power off source, Power on destination, Install tools
  • IP details for The Helper VM (if there is no DHCP on the enveronmnet)

A Helper VM is a standby operating environment (live boot) which needs a temporary IP address. The VM helps with the conversion and needs to be able to communicate with the VMware converter server (443) and the source virtual machine (22).

When the job is submitted, Converter creates a dummy virtual machine on the esxi host and boots it via ISO (helper VM iso: converter-helper-vm-x64.iso or converter-helper-vm.iso found at C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\

By Default, the root/user login for the helper VM is disabled. Inorder to enable this, change the config file located at:
C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-worker.xml

C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-worker.xml 

change:    <useSourcePasswordInHelperVm>false</useSourcePasswordInHelperVm>
to 
<useSourcePasswordInHelperVm>true</useSourcePasswordInHelperVm>

Restart all converter related services via services.msc

The root password will now be the same as the password that was used to connect to the source VM

when the Helper VM is on the network, it attempts to ssh into the source linux VM and runs a query like that to the below:

 ssh user@source_linux_IP -p 22 "sudo tar --one-file-system --sparse -C '/' -cf - ." | /us r/bin/tar --numeric-owner --delay-directory-restore -C '/home/p2vtest/' -xf 

Similarly, the other partitions are copied over to the Helper VM (review the helper-VM logs for others)

Once all volume’s are copied over, the filesystem (root and other volume’s, sda, sdb) are re-mapped with the respective path

The bootloader/GRUB is rebuilt (This is native to the version of linux have you, Running custom kernel can break this process)

 /usr/lib/vmware-converter/installGrub.sh