Veeam suddenly failed backing up one of my VMs, a VM homed at a datastore with multiple other VMs - these VMs were backed up just fine.

My first thought was that the device backing the datastore, an old piece of spinning rust, was about to fail. The S.M.A.R.T.-info from ESXi was no help. Zero errors and a interesting runtime of 42 hours?

Retrying the backup didn't help. This single VM still failed to be backed up.

I tried booting of a live Ubuntu ISO to run fsck and found some bad sectors on the volume. Surely the physical drive will be replaced, but for now the backup can run without errors.

Details:

fsck arguments: -ccfky

ESXi version: 7.0.0, 16324942

VMFS version: 6

Veeam VBR version: 10.0.1.4854

Use ovftool to deploy the image directly to the ESXi-host instead:

ovftool -dm=thick -ds=<DATASTORENAME> -n=<VMNAME> --net:"VM Network"="<VMNETWORKNAME>" "junos-media-vsrx-x86-64-vmdisk-18.2R1.9.scsi.ova" vi://root@esxi-host.tld

Replace with your own settings

If a snapshot seems stuck, use the console to verify a task is actually running:

1) Run vim-cmd vmsvc/getallvms and note the relevant VM-ID 2) Run vim-cmd vmsvc/get.tasklist <VM-ID> and note the Task-id 3) Run vim-cmd vimsvc/task_info <Task-id> to get task status 4) Browse to the VMs location on the datastore and run watch -d 'ls -lut | grep -E "delta|flat|sesparse"' to monitor the process

Unmap VMFS using esxcli

- Posted in VMware by

First fetch a list of VMFS:

esxcli storage filesystem list

For VMFS' where unmapping is supported, run:

esxcli storage vmfs unmap --volume-label=<label> | --volume-uuid=<uid>  [--reclaim-unit=<blocks>]

ESXTOP xterm, for unsupported terminals

- Posted in VMware by

Set TERM to xterm, before running esxtop to get a usable output, when the terminal/tty is not supported; run the following command to do so:

TERM=xterm esxtop

Get Virtual Machine uptime, with vim-cmd

- Posted in VMware by

Run vim-cmd vmsvc/getallvms to get a list of VM IDs (pipe to grep -i to filter)

With the ID from the second column, use the following command to fetch the uptime (replace 12345 with your VMs ID)

vim-cmd vmsvc/get.summary 12345 |grep "uptimeSeconds"

ESXi 6.5, switch to legacy USB-stack

- Posted in VMware by

Disable vmkusb module in ESXi 6.5 and switch to legacy stack:

esxcli system module set -m=vmkusb -e=FALSE

Reenable vmkusb:

esxcli system module set -m=vmkusb -e=TRUE

Either change requires rebooting of ESXi

Disable native driver and revert to sata-ahci:

esxcli system module set --enabled=false --module=vmw_ahci

Reenable native driver from sata-ahci:

esxcli system module set --enabled=true --module=vmw_ahci