Restricting Plugins on vSphere web-client (Flex and html5)

on Certain occations when the vCenter is upgraded, Plugins like pure, Netapp, NSX, SRM etc can cause incompatibility and prevent the webclient from loading up/Malfunctioning

Starting vSphere 6.7, we have a config file to white list and black list plugins which helps us to flag specific Malicious/incompatible plugins which prevents the web client form loading up

the symptoms to this include
* web client hangs at a blue, loading screen.
* web client loads but does not display any inventory object
* web client does not allow to click on the inventory objects (home button and task and events are sometimes clickable)

The quickest way to sort out these specious plugin is by adding the plugin id to blacklist to the respective UI/flex file comparability config.

UI(html5):

/etc/vmware/vsphere-ui/compatibility-matrix.xml

root@is-dhcp36-107 [ /etc/vmware/vsphere-ui ]# cat compatibility-matrix.xml
<!--
    This file lets you define a WHITE LIST and a BLACK LIST of plugins to control your own setup.
    It overrides the internal black and white lists that are hard-coded in this release.

    Fling Note: until further notice all plugins are disabled by the HTML5 client except SDK samples.
    Use this file to re-enable specific HTML plugins during your testing.
-->
<Matrix>
    <pluginsCompatibility>
        <!--
            WHITE LIST:
            Add this to enable all plugins whose plugin-package id is com.acme.example.myplugin:
                <PluginPackage id="com.acme.myplugin" status="compatible"/>
            Or this to specify all versions greater or equal to 2.1.0:
                <PluginPackage id="com.acme.myplugin" version=[2.1.0,] status="compatible"/>
            Or this to enable all plugins starting with com.acme:
                <PluginPackage id="com.acme.*" status="compatible"/>
        -->

        <!--
            BLACK LIST:
            Add this to disable a plugin whose plugin-package id is com.acme.example.myplugin:
                <PluginPackage id="com.acme.myplugin" status="incompatible"/>
         -->

    </pluginsCompatibility>
</Matrix>

vsphere-client (flex)

/etc/vmware/vsphere-client/compatibility-matrix.xml

root@is-dhcp36-107 [ /etc/vmware/vsphere-client ]# cat compatibility-matrix.xml
<!--
   The 'id' value allows for standard java regular expressions. The actual plugin id
   is matched against this regular expression.

   The 'version' values must be a string in any of the following formats. If skipped
   it indicates any version.

   Version: An exact version, e.g. 6.0.0, 5.5, etc. The format allows for maximum
      of 4 dotted numbers.

   Interval: (, 6.4], [6.5, 7.0), (7.5, ). Use empty strings to mark infinity values.

   Range: Several versions and intervals can be mixed in one single string
      producing a set of value, e.g. ( ,6.4), 6.8.3, 6.9.1, [7.5.2.2, ).

      Each item in the set is separated from the others using comma ",".
      Each item in the set can be either version or interval.

   The 'status' can be any of the following strings: unknown,
      incompatible, compatible, certified
-->
<Matrix>
  <pluginsCompatibility>
     <!--
        'Incompatible' plugins are not loaded. You can use this to 'blacklist'
        plugins if needed. To prevent specific plugin package(s) from
        loading, use the template entry below as guidance how to add new records
        with the actual id and version of the plugin you want to prevent from loading.

        The plugin ids can be taken from the plugin-package.xml file of each plugin.

        A few well known set of plugins locations (on the vCenter Appliance):
           /usr/lib/vmware-vsphere-client/plugin-packages/
           /etc/vmware/vsphere-client/vc-packages
           /etc/vmware/vsphere-client/cm-service-packages

        On Windows the following locations can be checked:
           <INSTALL DRIVE>\ProgramData\vCenterServer\runtime\vsphere-client\plugin-packages
           <INSTALL DRIVE>\ProgramData\vCenterServer\cfg\vsphere-client\vc-packages
           <INSTALL DRIVE>\ProgramData\vCenterServer\cfg\vsphere-client\cm-service-packages
     -->
     <!--
     <PluginPackage id="com.foo.plugin.id" version="1.0.0" status="incompatible"/>
     <PluginPackage id="net.bar.plugin.id" version="(,2.1]" status="incompatible"/>
     -->



     <!--
        The sample section below shows 'whitelist' definition. Compatible plugins
        are loaded. All others are declared as incompatible (due to the id regex),
        thus effectively forbidding them.

        The approach limits the list of plugins loaded only to a small 'white' list.
        This allows for vsphere-client to work in a 'safe-like' mode.

        Below are predefined sets of plugins for your convenience:
        1st-party, core:
          com.vmware.vsphere.client,
          com.vmware.ds,
          com.vmware.ssoadminui,
          com.vmware.vsphere.client.modules,
          com.vmware.license.client,
          com.vmware.opsmgmt.client

        1st-party extended (in addition to the above):
          com.vmware.loganalyzer,
          com.vmware.vsphere.client.telemetry

        2nd-party (basically anything that comes already pre-bundled with the
        vCenter Appliance and is not in the above two sets):
            com.vmware.vca.marketing.ngc.ui,
            com.vmware.vco
     -->
     <!--
     <PluginPackage id="com.vmware.vsphere.(client|ds|ssoadminui|client.modules)" status="compatible"/>
     <PluginPackage id="com.vmware.license.client" status="compatible"/>
     <PluginPackage id="com.vmware.opsmgmt.client" status="compatible"/>

     <PluginPackage id=".*" status="incompatible"/>
     -->
  </pluginsCompatibility>
</Matrix>

The plugin ID can be found on the extension manager under vCenter MOB. https://vc-fqdn/mob

recovering from an unresponsive hostd after a datastore/storage goes PDL

Hostd crashes with the below:

Hostd.log
2018-12-17T22:37:50.138Z info hostd[9130B80] [Originator@6876 sub=Hostsvc] Storage data synchronization policy set to invalidate_change
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: OpenSSL using FIPS_drbg for RAND
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: cipher list !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES
2018-12-17T22:37:50.141Z info hostd[9130B80] [Originator@6876 sub=Libs] GetTypedFileSystems: fstype vfat
2018-12-17T22:37:50.141Z info hostd[9130B80] [Originator@6876 sub=Libs] GetTypedFileSystems: uuid 579bba34-1440dc54-3308-70106f411e18   <-----volume that went offfline
2018-12-17T22:37:51.136Z info hostd[9AB8B70] [Originator@6876 sub=ThreadPool] Thread enlisted
 
VMkernel:
2018-12-17T22:19:15.397Z cpu18:65910)VMW_SATP_LOCAL: satp_local_updatePath:789: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2018-12-17T22:19:18.801Z cpu14:65607)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.00a0504658335330" state in doubt; requested fast path state update...
2018-12-17T22:19:18.801Z cpu14:65607)ScsiDeviceIO: 2968: Cmd(0x439d4981c540) 0x1a, CmdSN 0x6d2dc7 from world 0 to dev "eui.00a0504658335330" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:21.394Z cpu6:5268773)ScsiPath: 5115: Command 0x0 (cmdSN 0x0, World 0) to path vmhba32:C0:T0:L2 timed out: expiry time occurs 1002ms in the past
2018-12-17T22:19:21.394Z cpu6:5268773)VMW_SATP_LOCAL: satp_local_updatePath:789: Failed to update path "vmhba32:C0:T0:L2" state. Status=Transient storage condition, suggest retry
2018-12-17T22:19:22.892Z cpu22:65615)ScsiDeviceIO: 2968: Cmd(0x439d4981c540) 0x1a, CmdSN 0x6d2dc7 from world 0 to dev "eui.00a0504658335330" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:23.400Z cpu34:65627)NMP: nmp_ThrottleLogForDevice:3593: last error status from device eui.00a0504658335330 repeated 1 times
2018-12-17T22:19:23.400Z cpu34:65627)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x1a (0x439d4989a540, 0) to dev "eui.00a0504658335330" on path "vmhba32:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2018-12-17T22:19:23.400Z cpu34:65627)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.00a0504658335330" state in doubt; requested fast path state update...
2018-12-17T22:19:23.400Z cpu34:65627)ScsiDeviceIO: 2968: Cmd(0x439d4989a540) 0x1a, CmdSN 0x6d2dc8 from world 0 to dev "eui.00a0504658335330" failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:26.798Z cpu14:65607)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x1a (0x439d4a8ed5c0, 0) to dev "eui.00a0504658335331" on path "vmhba32:C0:T0:L1" Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2018-12-17T22:19:26.798Z cpu14:65607)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "eui.00a0504658335331" state in doubt; requested fast path state update...
2018-12-17T22:19:26.798Z cpu14:65607)ScsiDeviceIO: 2968: Cmd(0x439d4a8ed5c0) 0x1a, CmdSN 0x6d2dd8 from world 0 to dev "eui.00a0504658335331" failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x31 0x22 0x20.

Scsi Decoder: Link

In my case, The volume appeared to have gone offline because the host was aborting the commands to the HBA.

/etc/init.d/hostd status
hostd is not running.
However,
ps | grep hostd
2098894 2098894 hostdCgiServer
2105175 2105175 hostd
2105176 2105175 hostd-worker
2105177 2105175 hostd-worker
2105178 2105175 hostd-worker
2105179 2105175 hostd-worker
2105180 2105175 hostd-IO
2105181 2105175 hostd-IO
2105182 2105175 hostd-fair
2105183 2105175 hostd-worker
2105184 2105175 hostd-worker
2105185 2105175 hostd-worker
2105187 2105175 hostd-worker
2105191 2105175 hostd-worker
2105192 2105175 hostd-worker
2105193 2105175 hostd-worker
2105194 2105175 hostd-worker
2105251 2105175 hostd-poll
localcli storage core device world list
Device World ID Open Count World Name
------------------------------------------------------------------------------------------------------
mpx.vmhba32:C0:T0:L0 2099479 1 smartd
mpx.vmhba32:C0:T0:L0 2105105 1 vpxa
mpx.vmhba32:C0:T0:L0 2105175 1 hostd
naa.600508b1001c555e5048cfd74e058fdc 2097185 1 idle0
naa.600508b1001c555e5048cfd74e058fdc 2097403 1 OCFlush
naa.600508b1001c555e5048cfd74e058fdc 2098198 1 Res6AffinityMgrWorld
naa.600508b1001c555e5048cfd74e058fdc 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001c555e5048cfd74e058fdc 2099479 1 smartd
naa.600508b1001c555e5048cfd74e058fdc 2105175 1 hostd
naa.6001405d7dc7524f3364522a27b7c508 2097185 1 idle0
naa.6001405d7dc7524f3364522a27b7c508 2099760 1 fdm
naa.6001405d7dc7524f3364522a27b7c508 2099766 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2099771 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2100189 1 J6AsyncReplayManager
naa.6001405d7dc7524f3364522a27b7c508 2100219 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2105175 1 hostd
naa.6001405d7dc7524f3364522a27b7c508 2105286 1 hostd-worker
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2097185 1 idle0
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2097446 1 bcflushd
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2098539 1 J6AsyncReplayManager
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2105105 1 vpxa
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2105175 1 hostd
naa.600508b1001c5a5167700b7ae7160e91 2097185 1 idle0
naa.600508b1001c5a5167700b7ae7160e91 2097403 1 OCFlush
naa.600508b1001c5a5167700b7ae7160e91 2098198 1 Res6AffinityMgrWorld
naa.600508b1001c5a5167700b7ae7160e91 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001c5a5167700b7ae7160e91 2099479 1 smartd
naa.600508b1001c5a5167700b7ae7160e91 2105175 1 hostd
naa.600508b1001ca2c68c28022b4447710f 2097185 1 idle0
naa.600508b1001ca2c68c28022b4447710f 2097403 1 OCFlush
naa.600508b1001ca2c68c28022b4447710f 2098198 1 Res6AffinityMgrWorld
naa.600508b1001ca2c68c28022b4447710f 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001ca2c68c28022b4447710f 2099479 1 smartd
naa.600508b1001ca2c68c28022b4447710f 2105175 1 hostd

This shows that hostd still appears to be stuck as running in a zombie state.

To resolve this, we will need to

  • reset scsi commands to vmhba32 (vmkfstools -B /vmfs/volume/disk/naa.xxxx)
  • Rescan vmhba32 and wait for 3-5 min (esxcfg-rescan vmhba32)
  • confirm that hostd is no longer running for that device  (localcli storage core device world list and ps | grep hostd)
  • start hostd

Worst case scenario, Host reboot.

PostgreSQL: could not open file “/var/lib/pgsql/data/pg_clog/0726”: No such file or directory

when selecting a specific date from a table, at times due to database corruption Postgres might report the file was not found.

hsphere=# select * from TABLE;
ERROR:  could not access status of transaction 1918986094
DETAIL:  could not open file "/var/lib/pgsql/data/pg_clog/0726": No such file or directory

In Such instances, we will can re-create the file using the below dd command (note: data loss on the missing file)

dd if=/dev/zero of=/var/lib/pgsql/data/pg_clog/0726 bs=256k count=1

Note: Ensure the permissions on the file that was re-created are set appropriately. creating the blank file means that the data that was originally on it might actually go missing. The respective records will need to be re-created or its relevant records (constraints etc) will need to be removed from the DB.

joining Esxi to domain fails with LW_ERROR_INVALID_MESSAGE [code 0x00009c46]

Joining Esxi to doamin fails with the below message:

[root@esx:~] ./usr/lib/vmware/likewise/bin/domainjoin-cli join ikigo.net
Joining to AD Domain: ikigo.net
	With Computer DNS Name: esx.
	adm@ikigo.net's password:
	Error: LW_ERROR_INVALID_MESSAGE [code 0x00009c46]
	The Inter Process message is invalid
 

Cat /etc/hosts show below:

Cat /etc/hosts
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost.localdomain localhost
192.168.1.104 esx. esx      <------------------notice the "esx."

In some cases, the hostname on the /etc/hosts file might not match to that of the host.  (look for the message on domainjoin-join “With Computer DNS Name: esx.”

In order to sort this out,  correct the /etc/host file (in my case, I had to suffix the DNS suffix)

Cat /etc/hosts:
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost.localdomain localhost
192.168.1.104 esx.ikigo.net esx
 

restart lwsmd

/etc/init.d/lwsmd start

Re-attempt doamin join CLI

Note: In some cases after performing the change to /etc/hosts, joining the host via powerCLI will fail with error “‘ vmwauth InvalidHostNameException: The current hostname is invalid: hostname cannot be resolved”

Hostd.log:
2019-01-07T19:20:16.288Z error hostd[DEC1B70] [Originator@6876 sub=ActiveDirectoryAuthentication opID=500fc81b-b5-602f user=vpxuser:IKIGO\nik] vmwauth InvalidHostNameException: The current hostname is invalid: hostname cannot be resolved

The management agents/host will need a restart to sort this out.

services.sh restart

Installing VMware PowerCLi Install on windows 10/2012/2016 via Windows Gallery

Before you proceed, ensure there are no older version of VMware PowerCLi installed (check in add or remove programs), uninstall if you do have the older version.

Install PowerCli from Windows Gallery

Open powerCLI as an administrator and then run:

Install-Module -Name VMware.PowerCLI

List available VMware module:

Get-Module -ListAvailable -Name VMware*

Import Module

 Get-Module vmware* | Import-Module

Troubleshooting






Import module’s generally fail with the below error:

PS C:\windows\system32> Import-Module VMware*
>>
Import-Module : File C:\Program
Files\WindowsPowerShell\Modules\VMware.VimAutomation.Sdk\11.0.0.10334495\VMware.VimAutomation.Sdk.psm1 cannot be
loaded because running scripts is disabled on this system. For more information, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ Import-Module VMware.VimAutomation.Core
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : SecurityError: (:) [Import-Module], PSSecurityException
 + FullyQualifiedErrorId : UnauthorizedAccess,Microsoft.PowerShell.Commands.ImportModuleCommand

Set execution policy to remote signed if the importing fails:

Set-ExecutionPolicy RemoteSigned






If vCenter is running on the default self signed certificates: The connect-viserver is likely to fail

If the above command is not run, we would see an error similar to below:
Connect-VIServer : 12/21/2018 2:05:00 AM Connect-VIServer Error: Invalid server certificate. Use
Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you'd like to connect
once or to add a permanent exception for this server.
Additional Information: Could not establish trust relationship for the SSL/TLS secure channel with authority
'vc.ikigo.net'.
At line:1 char:1
+ Connect-VIServer vc.FQDN.Domain
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : SecurityError: (:) [Connect-VIServer], ViSecurityNegotiationException
 + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_CertificateError,VMware.VimAutomation.ViCore.
 Cmdlets.Commands.ConnectVIServer

Set invalid certificate to ignore on the computer to work this around

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore


Connecting to vCenter using PowerCli

connect-viserver vc.ikigo.net

on a successful connect, you should see the below

Passing through a Nvidia GTX970 to a VMware Virtual machine

Create a virtual machine with EFI boot, install windows + tools and power down the VM

Open ssh to the host where the VM is registered to

use vim-cmd to look for the VM

vim-cmd vmsvc/getallvms |   grep *name_of_VM* 

cd to the data store path

cd /vmfs/volumes/NVME/nvidia/

Edit the vmx file using the vi editor and add the below lines

hypervisor.cpuid.v0 = "FALSE"
pciHole.dynStart = "2816"

Save the editor and then reload the VMX (201 is the vmid from the above screenshot)

vim-cmd vmsvc/reload 201

Edit settings on the VM> add new hardware>pci device>add graphic card and its audio device(2 pci pass through device and set the memory reservation to maximum)

Take a snapshot and then power on the VM

Conform the hardware under graphic adapter followed by nvidia drivers installation.

Device manager

Note: The nvidia card and its audio device must already have been marked as under pci device as pass through!!

Creating a Cent OS7 template for VMware Guest Customization and Deployment

To start off with, Download the latest version of CentOs7 from https://www.centos.org/download/

At the time of writing, this is: CentOS-7 (1804)

Lets start by creating a new Virtual machine.  I will select Esxi 6.5 for backward compatibility  with other host.

Mount the ISO,

Power on the VM and begin installing of the OS and begin the installation


Note: During the install we will enable the default nic interface and set this to DHCP

Note: Since the ISO used was from a most recent release, openvm-tools is auto installed along with the linux installer. if you are using an older version of the cent os installer iso, you must install open-vm-tools with the below command (will need the VM connected to the internet).

yum install open-vm-tools

I would recommend updating the tools to the latest release.

then followed by installing pearl (pre-requisites for guest customization)

yum install perl

Once done, power down the VM and convert it to a template.

Test the template by deploying a VM with guest customization

When the VM boots up you should see the host name set to the name of the VM (the spec that I used to customize uses the name as of vsphere inventory as the the virtual machine name)

Looking at the VM that was just deployed, we see the host name has changed as per the specification.

Troubleshooting:
Log file for guest cust:

/var/log/vmware-imc/toolsDeployPkg.log

Generate a memory dump for a frozen VMware, windows virtual machine.

Enable complete memory dump feature by changing following registry keys:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl
CrashDumpEnabled    REG_DWORD    0x1

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
PagingFiles    REG_MULTI_SZ    c:\pagefile.sys 13312 13312


Enable keyboard crash dump feature by adding following registry keys:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters
Value Name: CrashOnCtrlScroll
Data Type:    REG_DWORD
Value:    1

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\kbdhid\Parameters
Value Name: CrashOnCtrlScroll
Data Type:    REG_DWORD
Value:    1

Enable NMI crash dump feature by adding following key:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl
Value Name: NMICrashDump
Data Type:  REG_DWORD
Value:   1

  • Restart Server to take effects.
  • Do a memory dump test with below steps:

Capture a kernel memory dump in following ways:

o    Send NMI to Guest OS Link:How to send NMI to Guest OS on ESXi 6.x (2149185)

Or

o   On VM console, press Right Ctrl + Scroll Lock button 2 times.

Machine crashes into blue screen and save a memory dump, restart automatically once dump generation reaches 100%. You should be able to see 12GB (memory allocated to the VM) here: C:\Windows\MEMORY.DMP file.

Statement diagnostic data from driver is XX002:0:7:ERROR: index “pk_index_name” contains corrupted page at block xxxx;

on the vpxd.log:
--> Error while executing the query
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] Connection diagnostic data from driver is HY000:0:110:
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] Bind parameters:
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] [0]datatype: 1, size: 4, arraySize: 0
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] value = 172237
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX002:7
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (XX002) - ERROR: index "pk_vpx_alarm_runtime" contains corrupted page at block 5877;
--> Error while executing the query" is returned when executing SQL statement "DELETE FROM VPX_ALARM_RUNTIME WHERE ENTITY_ID=?"
2018-09-25T21:17:09.355Z error vpxd[7F3FC958B800] [Originator@6876 sub=Daemon] Unhandled exception: Error[VdbODBCError] (-1) "ODBC error: (XX002) - ERROR: index "pk_vpx_alarm_runtime" contains corrupted page at block 5877;
--> Error while executing the query" is returned when executing SQL statement "DELETE FROM VPX_ALARM_RUNTIME WHERE ENTITY_ID=?"
2018-09-25T21:17:09.355Z info vpxd[7F3FC958B800] [Originator@6876 sub=SupportMgr] Wrote uptime information
(END)

Resolution:

Take a snapshot of the vcsa
Export affected table:

VCDB=# copy (select * from vpx_alarm_runtime) to '/tmp/vpx_alarm_runtime_select_.csv' with CSV DELIMITER ',';
COPY 3854
Drop affected constrain.
VCDB=# alter table vpx_alarm_runtime drop CONSTRAINT PK_VPX_ALARM_RUNTIME;

Recreate constrain

alter table vpx_alarm_runtime add constraint PK_VPX_ALARM_RUNTIME primary key (ENTITY_ID, ALARM_ID,
EXPRESSION_NAME)
recreate Table: VPX_ALARM_RUNTIME
/*==============================================================*/
/* Table: VPX_ALARM_RUNTIME */
/*==============================================================*/
create table VPX_ALARM_RUNTIME (
ENTITY_ID INTEGER not null,
ALARM_ID INTEGER not null,
ENTITY_TYPE INTEGER,
EXPRESSION_NAME VARCHAR(440) not null,
STATE_VALUE VARCHAR(255),
METRIC_VALUE INTEGER,
CREATED_TIME TIMESTAMP,
STATUS_VALUE VARCHAR(50),
EVENT_KEY INTEGER null,
constraint PK_VPX_ALARM_RUNTIME primary key (ENTITY_ID, ALARM_ID,
EXPRESSION_NAME)
);