vSekar Blog

Getting Hands Dirty With Technology

Posts and Views are my Own

Another blog post on APD & PDL

By

This article is Part of Becoming a VMware Storage Expert series

APD & PDL is the most discussed topic when it comes to vSphere Storage but this topic never old. VMware APD & PDL conditions are introduced in 5.x and 6.x

This article is part of How to Handle APD & PDL Series

But what happens exactly when disaster strikes and an APD has occurred? When an APD condition is detected a timer is started. After 140 seconds the APD condition is officially declared and the device is marked as APD time out. When the 140 seconds has passed HA will start counting, the default HA timeout is 3 minutes.

The timeout parameter controls how many seconds the ESXi host will retry nonvirtual machine I/O commands to a storage device in an all paths down (APD) state. If needed, you can change the default timeout value.

The timer starts immediately after the device enters the APD state. When the timeout expires, the host marks the APD device as unreachable and fails any pending or new nonvirtual machine I/O. Virtual machine I/O will continue to be retried.

The default timeout parameter on your host is 140 seconds. You can increase the value of the timeout if, for example, storage devices connected to your ESXi host take longer than 140 seconds to recover from a connection loss.\

Under Advanced System Settings, select the Misc.APDTimeout parameter and change the default value.

You can enter a value between 20 and 99999 seconds

When the 3 minutes have passed HA VMCP can restart the impacted virtual machines, depending on how you configured it

An unplanned permanent device loss (PDL) condition occurs when a storage device becomes permanently unavailable without being properly detached from the ESXi host.

The following items in the vSphere Web Client indicate that the device is in the PDL state:

  • The datastore deployed on the device is unavailable.

  • The operational state of the device changes to Lost Communication.

  • All paths are shown as Dead.

  • A warning about the device being permanently inaccessible appears in the VMkernel log file

This table outlines possible SCSI sense codes that determine if a device is in a PDL state:

SCSI sense code Description
H:0x0 D:0x2 P:0x0 Valid sense data: 0x__ /0x25/0x0 *LOGICAL UNIT NOT SUPPORTED
H:0x0 D:0x2 P:0x0 Valid sense data: 0x__/0x68/0x0 *LOGICAL UNIT NOT CONFIGURED
H:0x0 D:0x2 P:0x0 Valid sense data: 0x4/0x4c/0x0 HARDWARE ERROR/LOGICAL UNIT FAILED SELF-CONFIGURATION
H:0x0 D:0x2 P:0x0 Valid sense data: 0x4/0x3e/0x3 HARDWARE ERROR/LOGICAL UNIT FAILED SELF-TEST
H:0x0 D:0x2 P:0x0 Valid sense data: 0x4/0x3e/0x1 HARDWARE ERROR/LOGICAL UNIT FAILURE
H:0x0 D:0x2 P:0x0 Valid sense data: 0x2/0x4c/0x0 NOT READY/LOGICAL UNIT FAILED SELF-CONFIGURATION
H:0x0 D:0x2 P:0x0 Valid sense data: 0x2/0x3e/0x3 NOT READY/LOGICAL UNIT FAILED SELF-TEST
H:0x0 D:0x2 P:0x0 Valid sense data: 0x2/0x3e/0x1 NOT READY/LOGICAL UNIT FAILURE

When the device enters this PDL state, the vSphere host can take action to prevent directing any further, unnecessary I/O to this device. This alleviates other conditions that might arise on the host as a result of this unnecessary I/O. With vSphere 5.5, a new feature called PDL AutoRemove is introduced. This feature automatically removes a device from a host when it enters a PDL state. Because vSphere ESXi hosts have a limit of 255 disk devices per host, a device that is in a PDL state can no longer accept I/O, but can still occupy one of the available disk device spaces.

PDL AutoRemove occurs only if there are no open handles left on the device. The auto-remove takes place when the last handle on the device closes. If the device recovers, or if it is re-added after having been inadvertently removed, it is treated as a new device

  1. Run this command to disable AutoRemove:esxcli system settings advanced set -o “/Disk/AutoremoveOnPDL” -i 0

Notes:

  • There is no guarantee of data integrity can be given if a device returns from PDL.
  • To re-enable the AutoRemove feature, run this command from a shell session to the ESXi host:esxcli system settings advanced set -o “/Disk/AutoremoveOnPDL” -i 1
In vSphere 6.x, the expectation is that a device in a PDL state will not return. Therefore, the device needs to be removed from the ESXi host, before it can be recovered. If the AutoRemoveOnPDL feature is disabled, a manual rescan is required to remove the device while in a PDL state.
Note: For vSphere Metro Storage Cluster (vMSC) environments, VMware recommends to have AutoremoveOnPDL set to 1 explicitly for vSphere 6.x.
A planned PDL occurs when there is an intent to remove a device presented to the ESXi host. The datastore must first be unmounted, then the device detached before the storage device can be unpresented at the storage array. For more information on how to correctly unpresent a LUN
Sometime we may run into issues when remounting LUN after PDL, follow this article if you face the issue

3 years of VMCP, how well can you implement them

By

Its been 3 years since VMCP has been released, what is it and how to configure it the right way?

Under the vSphere High availability feature, we now have VMCP (VM Component Protection). It provides enhanced protection from All Paths Down (APD) and Permanent Device Loss (PDL) conditions for block and file storage.

VMCP detects APD and PDL conditions and efficiently remediates the situations by restarting the Virtual Machines on fully functional ESXi hosts when compared to its predecessor. If this is enabled vSphere HA can detect datastore accessibility failures and provide automated recovery for affected virtual machines.

How to configure VMCP

On your vSphere client goto configure tab and click vSphere availability -> click on Edit

On Failures and Responses section select Datastore with PDL or Datastore with APD.

The storage protection levels you can choose and the virtual machine remediation
actions available differ depending on the type of database accessibility failure.
PDL Failures Under Datastore with PDL, you can select Issue events or Power off and
restart VMs.
APD Failures The response to APD events is more complex and accordingly the
configuration is more fine-grained. You can select Issue events, Power off
and restart VMs–conservative restart policy, or Power off and restart
VMs–aggressive restart policy

Workflow:

 

 

 

 

 

 

 

 

 

 

 

 

 

Recovery Timeline:

 

 

 

 

 

 

 

 

 

 

 

 

There are different responses that you can set:

  • Power off and restart VMs (conservative)
  • Power off and restart VMs (aggressive)

Conservative and Aggressive refers to the reaction of HA being able to restart VMs. When set to “conservative” HA will only kill the VM that is impacted by the APD if it knows another host can restart it. In the case of “aggressive” HA will always kill the VM even if it doesn’t know the state of the other hosts, which could lead to a situation where your VM is not restarted as there is no host that has access to the datastore the VM is located on.

 

 

 

How to handle APD & PDL – In Detail series

By

This article is Part of Becoming a VMware Storage Expert series

Houston we have a APD

APD and PDL are few general terms you hear when working on a VMware environment, this is yet another blog on how to handle the situation of APD how to avoid those situations and how to get out of PDL situation

All-Paths-Down (APD) situation occurs when all paths to a device are down. As there is no indication whether this is a permanent or temporary device loss, the ESXi host keeps reattempting to establish connectivity. APD-style situations commonly occur when the LUN is incorrectly unpresented from the ESXi/ESX host. The ESXi/ESX host, still believing the device is available, retries all SCSI commands indefinitely. This has an impact on the management agents, as their commands are not responded to until the device is again accessible. This causes the ESXi/ESX host to become inaccessible/not-responding in vCenter Server.

In ESXi 5.x and 6.x we have another situation called PDL.

Esxi considers device loss permanent. It can be caused by making a LUN inaccessible to a host, either by unmapping or deleting it. In this case, the storage array informs the host of a PDL state through a SCSI command response. The removal is considered permanent when all paths have the PDL error. PDL was introduced in 5.1

 In vSphere 6.0, VMware has enhanced APD and PL and introduced vSphere VMCP (VM component protection) feature. If an APD or a PDL condition occurs and VM is running on a host which has got connectivity issue with a datastore, HA will kick in and restart that VM to other host which has not connectivity issue with the same storage.

There are two variants of PDL, planned and unplanned:

Let’s see in detail about the VMCP, how to interpret a PDL and how to resolve it in upcoming articles