] Statement diagnostic data from driver is XX002:0:7:ERROR: index “pk_vpx_alarm_runtime” contains corrupted page at block xxxx;

on the vpxd.log:

–> Error while executing the query
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] Connection diagnostic data from driver is HY000:0:110:
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] Bind parameters:
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] [0]datatype: 1, size: 4, arraySize: 0
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] value = 172237
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX002:7
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from HY000:110
2018-09-25T21:17:09.351Z error vpxd[7F3FC958B800] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: “ODBC error: (XX002) – ERROR: index “pk_vpx_alarm_runtime” contains corrupted page at block 5877;
–> Error while executing the query” is returned when executing SQL statement “DELETE FROM VPX_ALARM_RUNTIME WHERE ENTITY_ID=?”
2018-09-25T21:17:09.355Z error vpxd[7F3FC958B800] [Originator@6876 sub=Daemon] Unhandled exception: Error[VdbODBCError] (-1) “ODBC error: (XX002) – ERROR: index “pk_vpx_alarm_runtime” contains corrupted page at block 5877;
–> Error while executing the query” is returned when executing SQL statement “DELETE FROM VPX_ALARM_RUNTIME WHERE ENTITY_ID=?”
2018-09-25T21:17:09.355Z info vpxd[7F3FC958B800] [Originator@6876 sub=SupportMgr] Wrote uptime information
(END)

 

 

Resolution:

Take a snapshot of the vcsa

Export affected table:
VCDB=# copy (select * from vpx_alarm_runtime) to ‘/tmp/vpx_alarm_runtime_select_.csv’ with CSV DELIMITER ‘,’;
COPY 3854

Drop affected constrain.
VCDB=# alter table vpx_alarm_runtime drop CONSTRAINT PK_VPX_ALARM_RUNTIME;

re-create constrain.


recrate Table: VPX_ALARM_RUNTIME


/*==============================================================*/
/* Table: VPX_ALARM_RUNTIME */
/*==============================================================*/
create table VPX_ALARM_RUNTIME (
ENTITY_ID INTEGER not null,
ALARM_ID INTEGER not null,
ENTITY_TYPE INTEGER,
EXPRESSION_NAME VARCHAR(440) not null,
STATE_VALUE VARCHAR(255),
METRIC_VALUE INTEGER,
CREATED_TIME TIMESTAMP,
STATUS_VALUE VARCHAR(50),
EVENT_KEY INTEGER null,
constraint PK_VPX_ALARM_RUNTIME primary key (ENTITY_ID, ALARM_ID,
EXPRESSION_NAME)
);

 

 

 

Replacing vmdir certificates on vCenter 6.0

vmdir is a vCenter component that Listens on port 636 and 389 (LDAPs/LDAP)

 

  • We will start creating a new configuration file called vmdir.cfg with the below content: (replace the contents under v3_req with the fields appropriate to your enveronment)

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:psc1.domain.com, DNS:psc1, IP: x.x.x.x
[ req_distinguished_name ]
countryName = US
stateOrProvinceName = State
localityName = City
0.organizationName = Company
organizationalUnitName = Department
commonName = psc1.domain.com

 

  • using openssl, create a new CSR file with the above configuration:

“%VMWARE_OPENSSL_BIN%” req -new -out c:\cert\vmdir.csr -newkey rsa:2048 -keyout c:\cert\vmdir.key -config c:\cert\vmdir.cfg

 

 

If the solution user certificates are signed with a CA cert, sign the CSR with the same issueing CA
else, Sign them using VMCA using the instructions below.

Sigining the CSR with the VMCA certificate.

  • Copy root.cer and privatekey.pem from C:\ProgramData\VMware\vCenterServer\data\vmca
    (appliance: /var/lib/vmware/vmca/) to c:\cert\

 

  • Run the beow command to signe the certificate:
    “%VMWARE_OPENSSL_BIN%” x509 -req -days 3650 -in c:\cert\vmdir.csr -out c:\cert\vmdir_signed.crt -CA c:\cert\root.cer -CAkey c:\cert\privatekey.pem -extensions v3_req -CAcreateserial -extfile c:\cert\vmdir.cfg

 

  • Now we have a certificate that can be used to replace the existing vmdir certificates. To proceed with the certificate replacement, Stop all vCenter services

service-control –stop –all

Note: For windows, you must be on path: “C:\Program Files\VMware\vCenter Server\bin”

  • Go into path: C:\ProgramData\VMware\vCenterServer\cfg\vmdird (appliance: ‘/usr/lib/vmware-vmdir/share/config/’)
  • (backup original certificates) vmdircert.pem and vmdirkey.pem to a temp directory
  • rename vmdir_signed.crt to vmdircert.pem  and  vmdir.key to vmdirkey.pem on the above directory
  • Start all services

service-control –start–all

Note: If the services fail to start (most likely inventory) then you it means that the wrong root cert was used when sigining the certificate. Replace the original files on the directory and restart the service to roll back to previous configuraton.

web client service crashes java.lang.OutOfMemoryError: PermGen space and java.lang.OutOfMemoryError

vSphere web client refused to start with memory errors

log location:
Windows: c:\programdata\VMware\vCenter\Logs\vsphere-client
Appliance: /var/log/vmware/vsphere-client

 

wrapper.log

INFO | jvm 1 | 2018/04/03 15:34:25 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “org.springframework.scheduling.timer.TimerFactoryBean#0”
INFO | jvm 1 | 2018/04/03 15:34:33 |
INFO | jvm 1 | 2018/04/03 15:34:33 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-bio-9443-exec-10”
INFO | jvm 1 | 2018/04/03 15:35:12 |
INFO | jvm 1 | 2018/04/03 15:35:12 | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-bio-9443-exec-6”

vsphere_client_virgo.log

[2018-04-03T15:28:45.855-04:00] [ERROR] http-bio-9090-exec-2 com.vmware.vise.util.concurrent.WorkerThread http-bio-9090-exec-2 terminated with exception: java.lang.OutOfMemoryError: PermGen space
[2018-04-03T15:28:46.773-04:00] [ERROR] http-bio-9090-exec-5 com.vmware.vise.util.concurrent.WorkerThread http-bio-9090-exec-5 terminated with exception: java.lang.OutOfMemoryError: PermGen space

 

Cause: insufficient  Web-client heap size/Insufficient PermGen space?configuration change

 

Scenario 1:  Heap-size

Resolution:

  • Ensure there is sufficient free memory on the vCenter

free -m

  • Review and Increase (double) the heap size of the web client

Appliance: cloudvm-ram-size -l vsphere-client
windows: C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat -l

cloudvm-ram-size.bat -C XXX 

  • start vsphere-client and observe if this still crashes.

Scenario 2: PermGen

  • Take a copy of the file service-layout.mfx as service-layout.mfx.bak
    Appliance path: /etc/vmware/
    Windows Path: C:\ProgramData\VMware\vCenterServer\cfg\
  • Edit service-layout.mfx with a text editor
  •  Change MaxPermMB size from 256 to 512 for the row vspherewebclientsvc. (increase accordingly depending on the number of plugins configured with vCenter_webclient)
  • start vsphere-client service

Scenario 3: Problem persists even after increasing/maxing out scenario 1 and scenario 2.

  • backup configuration file before you proceed

    cp /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphere-client /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphereclient.bak

  • Edit the file using a text editor

vi /usr/lib/vmware-vsphere-client/server/wrapper/bin/vsphere-client

  • Look for the line  “RUN_AS_USER=vsphere-client” and hash this
    vi vsphere-client
  • Start vsphere-client service.

vCenter Pre-upgrade fails

Error: Internal error occurred during execution of upgrade process.

Resolution: Send upgrade log files to VMware technical support team for further Assistance.

 

Upgrade logs say:

less /var/log/vmware/upgrade/bootstrap.log
2018-03-23T20:14:34.11Z ERROR transport.guestops Invalid command: “/bin/bash” –login -c ‘/opt/vmware/share/vami/vami_get_network eth0 1>/tmp/vmware-root/exec-vmware47-
stdout 2>/tmp/vmware-root/exec-vmware235-stderr’
None
2018-03-23T20:14:34.12Z ERROR upgrade_commands Unable to execute pre-upgrade checks on host 10.1.0.209
Traceback (most recent call last):
File “/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py”, line 2199, in execute
preupgradeResult = self._executePreupgradeChecks()
File “/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py”, line 2655, in _executePreupgradeChecks
srcIpv4Address, srcIpv4SubnetMask, srcIpv6Address, srcIpv6Prefix = retrieveNetworkingConfiguration(self.opsManager)
File “/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/transfer_network.py”, line 1309, in retrieveNetworkingConfiguration
interface)
File “/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py”, line 188, in _retrieveNetworkIdentity
networkConfig = vamiGetNetwork(processManager, interface)
File “/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py”, line 144, in vamiGetNetwork
output = _execNetworkConfigCommand(processManager, [VAMI_GET_NETWORK_CMD, interface])
File “/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/apply_networking.py”, line 66, in _execNetworkConfigCommand
cr = transport.executeCommand(processManager, cmd)
File “/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/__init__.py”, line 122, in executeCommand
return processManager.pollProcess(processUid, True)
File “/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py”, line 81, in __call__
ret = self.func(*args, **kwargs)
File “/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/guestops.py”, line 1184, in pollProcess
self._checkInvalidCommandError(processInfo, stderr)
File “/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/guestops.py”, line 1123, in _checkInvalidCommandError
raise ExecutionException(error, ErrorCode.INVALID_REQUEST)
ExecutionException: (‘Invalid command: “/bin/bash” –login -c \’/opt/vmware/share/vami/vami_get_network eth0 1>/tmp/vmware-root/exec-vmware47-stdout 2>/tmp/vmware-root/
exec-vmware235-stderr\”, 1)

2018-03-23T20:14:39.442Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log

 

Now look at the source appliance.

VMware VirtualCenter 6.0.0 build-3339084
vCenter:~ # ifconfig
eth0 Link encap:Ethernet HWaddr 00:50:56:AC:53:FD
inet addr:x.x.x.x Bcast:x.x.x.x Mask:255.255.252.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:45028984 errors:0 dropped:28266 overruns:0 frame:0
TX packets:16476384 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:74680502042 (71220.8 Mb) TX bytes:7187692049 (6854.7 Mb)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:147809637 errors:0 dropped:0 overruns:0 frame:0
TX packets:147809637 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:93984509789 (89630.6 Mb) TX bytes:93984509789 (89630.6 Mb)

Running /opt/vmware/share/vami/vami_get_network  less returns an dependency error:

vCenter:~ # /opt/vmware/share/vami/vami_get_network eth0 1 | less
/opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory

 

To resolve this, re-create the link to dependency by running the below commands.

echo “LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/opt/vmware/lib/vami/” >> /etc/profile
echo ‘export LD_LIBRARY_PATH’ >> /etc/profile

 

Re-run the command to confirm if it is returning the IP details

/opt/vmware/share/vami/vami_get_network

vCenter55:~ # /opt/vmware/share/vami/vami_get_network
interface: eth0
config_present: true
config_flags: STATICV4
config_ipv4addr: 10.1.0.209
config_netmask: 255.255.252.0
config_broadcast: 10.1.3.255
config_gatewayv4:
config_ipv6addr:
config_prefix:
config_gatewayv6: 10
autoipv6:
active_ipv4addr: 10.1.0.209
active_netmask: 255.255.252.0
active_broadcast: 10.1.3.255
active_ipv6addr:
active_prefix:
active_gatewayv4: 10.1.0.61
active_gatewayv6:
hasdhcpv6: 1
Traceback (most recent call last):
File “/opt/vmware/share/vami/vami_ovf_process”, line 25, in <module>
import libxml2
File “/usr/lib64/python2.6/site-packages/libxml2.py”, line 1, in <module>
ImportError: No module named libxml2mod
managed:

 

vami_ovf_process and libxml2.py can be ignored

Re-run the upgrade/migration.

Deploy A VM using OVFtool

with vCenter 6.5 the OVF Deployments via the web client  (network traffic) will need to go via the vCenter. This is an expected behavior.

However, this can be worked around by using the OVF tool for deployment.  But I must advise that a small portion of bandwidth is used to send metadata information from the client PC to the vCenter.

Here’s a sample command:

Syntax:

ovftool -ds=”datastore” -n=”VM_Name” -–net:”VM _old_Network”=”VM Network”  c:\path_to_ovf\file.ovf vi://administrator@vsphere.local:password@vCenter_server_name?ip=Esxi_Host_IP

 

Example:

ovftool -ds=”datastore” -n=”VMName” -–net:”VM Network”=”VM Network”  C:\Users\ntitta\Downloads\myth.ovf vi://administrator@vs.lo:P@ssw0rd@is-dhcp39-171.isl.vmware.com/?ip=10.109.10.120

Note:
VM Network = the vmnetwork of the ovf at the time of export

If you are not too sure of the original network, Run the below to query:
C:\Program Files\VMware\VMware OVF Tool>ovftool C:\Users\ntitta\Downloads\myth.ovf
Output:
OVF version:   1.0
VirtualApp:    false
Name:          myth

Download Size:  Unknown

Deployment Sizes:
Flat disks:   16.00 GB
Sparse disks: Unknown

Networks:
Name:        VM Network
Description: The VM Network network

Virtual Machines:
Name:               myth
Operating System:   ubuntu64guest
Virtual Hardware:
Families:         vmx-13
Number of CPUs:   2
Cores per socket: 1
Memory:           1024.00 MB

Disks:
Index:          0
Instance ID:    5
Capacity:       16.00 GB
Disk Types:     SCSI-lsilogic

NICs:
Adapter Type:   VmxNet3
Connection:     VM Network

Link to download OVFtool  https://my.vmware.com/web/vmware/details?productId=614&downloadGroup=OVFTOOL420