Inaccessible VM, Disappearing Datastore, and ATS locking!
I recently learned more than I ever wanted to know about the ATS (Atomic Test and Set) algorithm feature that is used in VAAI (VStorage API for Array Integration).
So what is it? Apparently ATS was added as a feature back with ESXi4.1, but the environment I had the issue on is 5.5. Apparently I have dodge this bullet for a long time. Before ATS was introduced a volume would lock by scsi reservations locking the entire volume during creation and metadata updates, snapshots etc…. So performance was an issue on a volume that has multiple Vm’s on it. What ATS does is lock on sections of the volume, using a heartbeat, instead of the whole volume itself resulting in better performance. In ESXi5 any new VMFS volume is by default set to ATS-Only.
So before 5.5 U2 ATS would only be used when the heartbeat changed states… Acquire, Clear, Replay, or Reclaim a heartbeat.
Apparently after 5.5 U2, 5.5U3, and 6.0 the ATS feature is now used for maintaining the heartbeat using heartbeat I/O. Because the timeout value for the heartbeat I/O is set so low this can cause datastore, VM, and application disconnects.
We shut down the ESXi boxes and the ISCSI attached SAN. When we brought them back up the VM’s showed inaccessible, the datastores where not there. The LUN was showing but I could not see inside of it or detach it as it said “The resource was busy” This was my conclusion
SYMPTOMS:
-Virtual Machine shows inaccessible after a host reboot
-The Volume shows in Devices but not the datastore
-Cannot detach the LUN it says the resource is in use.
– Error in log : info ‘Vimsvc.ha-eventmgr’] Event 1675 : ATS-Only VMFS volume ‘DATASTORENAME’ not mounted. Host does not support ATS or ATS initialization has failed.
FIX:
To work around this you need to disable the VAAI ATS locking mechanism on each host in your cluster.
On ESXi 5 you can run the 3 Commands from ssh
# esxcli system settings advanced set –int-value 0 –option /DataMover/HardwareAcceleratedMove
# esxcli system settings advanced set –int-value 0 –option /DataMover/HardwareAcceleratedInit
# esxcli system settings advanced set –int-value 0 –option /VMFS3/HardwareAcceleratedLocking
Or From VCenter
Log in to vCenter Server 5.0
In the vSphere Client inventory panel, click the ESXi 5.0 host.
Click the Configuration tab.
Under Software, click Advanced Settings.
Click on DataMover
-Change DataMover.HardwareAcceleratedMove to 0
-Change DataMover/HardwareAcceleratedInit to 0
Click VMFS3.
Change HardwareAcceleratedLocking 0
Do this for all your ESXi host the VM / Datastore is connected to. Then rescan you HBA and your Datastore / VM should be back.
I hope you find this post and it saves you more time than it did me trying to figure out why my datastore disappeared.
Murphy
May 11. 2018
Thanks for the post!!!! It worked and very much appreciated!!!
Rahamathulla
Nov 14. 2017
Shellie,
Disabling ATS will also make the VAAI disable. In turn, the environment may face performance issues. So is there any way to improvise this ATS locking feature rather than completely disabling it.
Juhnu
Mar 20. 2017
Just had this pop up after relocating servers and had our WSUS down for a day. This solved the problem, Big thanks!