VCSA/Insufficient inodes can result in service crash with no space left on device errors

vpxd logs:


019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root
2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07
2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device)
--> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context]
2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down
2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2019-02-08T22:44:53.616Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Alert:false@ bora/vpx/vpxd/util/vdb.cpp:509
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-8024368, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002B5E90]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B1804]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x00178BDB]: Vmacore::Service::Alert(char const*, char const*, int)
--> backtrace[03] vpxd[0x00A367FF]
--> backtrace[04] vpxd[0x0054E418]
--> backtrace[05] vpxd[0x0054FC2F]
--> backtrace[06] vpxd[0x00539F6A]
--> backtrace[07] libc.so.6[0x000205E0]
--> backtrace[08] vpxd[0x00539775]
--> [backtrace end]
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Registry Item DB 5 value is ''
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd.
2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) -
2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database
2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock
2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

Looking at filesystem:

root@cmtolpvctrapp24 [ / ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                   16G     0   16G   0% /dev
tmpfs                                      16G  8.0K   16G   1% /dev/shm
tmpfs                                      16G  656K   16G   1% /run
tmpfs                                      16G     0   16G   0% /sys/fs/cgroup
/dev/sda3                                  11G  8.1G  2.1G  80% /
tmpfs                                      16G  884K   16G   1% /tmp
/dev/sda1                                 120M   28M   87M  24% /boot
/dev/mapper/log_vg-log                     25G  1.9G   22G   9% /storage/log
/dev/mapper/dblog_vg-dblog                 25G  173M   24G   1% /storage/dblog
/dev/mapper/db_vg-db                       25G  396M   23G   2% /storage/db
/dev/mapper/seat_vg-seat                  197G  5.8G  181G   4% /storage/seat
/dev/mapper/netdump_vg-netdump            9.8G   23M  9.2G   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy       25G   45M   24G   1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr         99G  435M   93G   1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder   25G   45M   24G   1% /storage/imagebuilder
/dev/mapper/core_vg-core                   99G  188M   94G   1% /storage/core

 


looking at inode:


root@cmtolpvctrapp24 [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 712704        0  100% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core


Now we determine what is consuming the most of inode:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
    875 /usr/bin
   1417 /opt/vmware/lib/python2.7/test
   1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
   5362 /usr/share/man/man3
607550 /var/spool/mqueue

vCenter Build

 version: VMware VirtualCenter 6.5.0 build-8024368

To resolve this, we run the below to clear the inode

 find /var/spool/mqueue/ -type f -print0 | xargs  -0 rm -f

Check for available node to conform if everything is cleared up.

root@cmtolpvctrapp24 [ / ]# df -i
Filesystem                                 Inodes  IUsed    IFree IUse% Mounted on
devtmpfs                                  4115632    531  4115101    1% /dev
tmpfs                                     4117336      4  4117332    1% /dev/shm
tmpfs                                     4117336    690  4116646    1% /run
tmpfs                                     4117336     16  4117320    1% /sys/fs/cgroup
/dev/sda3                                  712704 105154   607550   15% /
tmpfs                                     4117336     77  4117259    1% /tmp
/dev/sda1                                   32768    305    32463    1% /boot
/dev/mapper/log_vg-log                    1638400  14332  1624068    1% /storage/log
/dev/mapper/dblog_vg-dblog                1638400     22  1638378    1% /storage/dblog
/dev/mapper/db_vg-db                      1638400   2854  1635546    1% /storage/db
/dev/mapper/seat_vg-seat                 13107200   4430 13102770    1% /storage/seat
/dev/mapper/netdump_vg-netdump             655360     11   655349    1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      1638400     13  1638387    1% /storage/autodeploy
/dev/mapper/updatemgr_vg-updatemgr        6553600    273  6553327    1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  1638400     14  1638386    1% /storage/imagebuilder
/dev/mapper/core_vg-core                  6553600     15  6553585    1% /storage/core

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.