One of my Veeam Backup Copy jobs failed for every VM in the job, reporting IO errors:

10/08/2023 08.29.02 :: Processing vSRX-18.3 Error: File does not exist. File: [vSRX-18.3.1D2023-08-09T020227_4DE4.vib]. Failed to open storage for read access. Storage: [vSRX-18.3.1D2023-08-09T020227_4DE4.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{13faea43-f648-4fee-8abb-630907bd1df7}]. CHMOD mask: [0]. Agent failed to process method {DataTransfer.RestoreText}. ...

The volume holding the VIBs, an external USB-drive forwarded to the Veeam guest within ESXi, was gone. Seemingly a failed drive.

VCSA failed to remove the USB Host device, with the error:

Invalid configuration for device '0'

Not being able to remove a missing device (the USB-controller), even when the VM is powered off, I had no choice but to manually delete it:

Simply remove the VM from the inventory - obviously not deleting it from VMFS.

Locating the device in the .VMX for the VM and removing the line from the configuration:

usb_xhci.autoConnect.device0 = "path:0/1/1/5 host:esxi7.rotteslottet.lan hostId:71 05 57 47 bf 16 10 d6-2c 0e 3c 7c 3f 11 9a f0 autoclean:1 deviceType:remote-host"

After trashing the failed drive, replacing it with a new one, and re-powering the USB-controller, I simply registered the VM via VCSA and started over with the local backup.

I wish it was possible to forcefully remove a virtual device via VCSA

Veeam suddenly failed backing up one of my VMs, a VM homed at a datastore with multiple other VMs - these VMs were backed up just fine.

My first thought was that the device backing the datastore, an old piece of spinning rust, was about to fail. The S.M.A.R.T.-info from ESXi was no help. Zero errors and a interesting runtime of 42 hours?

Retrying the backup didn't help. This single VM still failed to be backed up.

I tried booting of a live Ubuntu ISO to run fsck and found some bad sectors on the volume. Surely the physical drive will be replaced, but for now the backup can run without errors.

Details:

fsck arguments: -ccfky

ESXi version: 7.0.0, 16324942

VMFS version: 6

Veeam VBR version: 10.0.1.4854

Create an encrypted container for storage with LUKS

Change the names and paths to reflect your environment and needs

1) Make sure cryptsetup is installed: sudo apt update && apt install cryptsetup -y

2) Create an empty file for the container: sudo dd if=/dev/zero bs=1M of=/path/to/lukscontainer count=10240 (I prefer using a flat file, instead of a device, for portability)

3) Create the LUKS volume within the flat file: sudo cryptsetup luksOpen /path/to/lukscontainer container_crypt

4) Create a filesystem within the LUKS volume: sudo mkfs.ext4 /dev/mapper/container_crypt

5) Create a mountpoint for the container: sudo mkdir -p /storage/container/

6) Mount the container in your newly created mountpoint: sudo mount /dev/mapper/container_crypt /storage/container/

To easily unmount and mount the container in the future, create 2 simple scripts:

luksUnmountContainer.sh:

#!/bin/sh /usr/bin/umount /dev/mapper/container_crypt /sbin/cryptsetup luksClose /dev/mapper/container_crypt

luksMountContainer.sh:

#!/bin/sh /sbin/cryptsetup luksOpen /path/to/lukscontainer container_crypt /usr/bin/mount /dev/mapper/container_crypt /storage/container/

(the editor in htmly isn't playing nice, insert linebreaks manually)

Make the scrips executable with chmod +x luks*Container.sh and run them with ./

Make sure to upgrade your KDF to argon2id (default for latest version at the time of writing): https://mjg59.dreamwidth.org/66429.html

In case monitoring of vSRX/SRX-licensing isn't available from the official solutions from Juniper, one still might want to be in the know, before Junos stops pushing packets.

Managing a growing number of vSRX'es deployed around the world, I didn't want to manually check licenses. I had to make a quick'n'dirty solution. So I did.

The "solution" is rather simple; create a read-only user in Junos. Run a command via SSH, store the result and repeat. It has been a while, so you'd need some old repo's or rewrite some stuff.

Tested with versions:

php-cli 5.5.9
sed (GNU sed) 4.2.2

0) Create read-only users on each device (assuming 'readonlyuser' in this example) and replace 'SECRETPASSWORD' with your set password for 'readonlyuser'.

0.1) Connect to the devices with ssh to accept their keys. There might be a way to accomplish this blindly, however that is beyond the scope of this post.

1) Create a table for storing license data:

CREATE TABLE `vsrx-licenses` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `timestamp` int(12) DEFAULT NULL,
  `host` varchar(64) DEFAULT NULL,
  `expirationdate` varchar(10) DEFAULT NULL,
  `daystoexpire` int(3) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;

2) Create file parseXML.php in /home/derp/vsrx-fetch-license/

<?php
// Configure database connection
function connectToDatabase($database) {
  $link = mysql_connect("DBHOST","DBUSER","DBPASS");
  $db = mysql_select_db($database, $link);
  mysql_set_charset('utf8',$link);
}

// Define function to calc days to expire
function daysToExpire($expireDate) {
  $todaysDate = date("Y-m-j");
  $origin = new DateTime($todaysDate);
  $target = new DateTime($expireDate);
  $interval = $origin->diff($target);
  return $interval->format('%a');
}

// Execute magics
connectToDatabase('DBNAME');
$timeNow = time();

$xmlFile = '/home/derp/vsrx-fetch-license/'.$argv[1];
$licenses = json_decode(json_encode((array) simplexml_load_file($xmlFile)), 1);
$licenseDetails = array_column($licenses, 'feature-summary');

// 
$deviceNoExt = substr($argv[1], 0, -4);
$deviceClean = substr($deviceNoExt, 34);

// Clear existing count to reduce db-size, optional
#mysql_query("DELETE FROM `vsrx-licenses`");

foreach ($licenseDetails[0] as $element) {
    if ($element['licensed'] != 0 && isset($element['end-date'])) {
    $deviceEndDate = $element['end-date'];
    $deviceDaysToExpire = daysToExpire($deviceEndDate);

    mysql_query("INSERT INTO `vsrx-licenses`
    (id, timestamp, host, expirationdate, daystoexpire)
    VALUES (null, '$timeNow', '$deviceNoExt', '$deviceEndDate', '$deviceDaysToExpire')") or die(mysql_error());
        echo 'License for device '.$deviceNoExt.' expires: '.$deviceEndDate.'
';
    }
}

3) Create file fetch-licenses.sh in /home/derp/vsrx-fetch-license/. Replace vsrx01.domain.tld ... with the hostnames of your devices

#!/bin/bash
# Clear old logs
napTime=3
/bin/rm /home/derp/vsrx-fetch-license/*.xml
/usr/bin/php /home/derp/vsrx-fetch-license/wipeDB.php
vsrxDevices=("vsrx01.domain.tld" "vsrx02.domain.tld" "vsrx03.domain.tld")
echo "Fetching licenses..."
for device in ${vsrxDevices[@]}; do
  echo "Fetching license details for device $device"
  /home/derp/vsrx-fetch-license/vsrx-expect.sh $device > /home/derp/vsrx-fetch-license/$device.xml
  /bin/sed -i -n '2,$p' /home/derp/vsrx-fetch-license/$device.xml
  /bin/sed -i -n '2,$p' /home/derp/vsrx-fetch-license/$device.xml
  /usr/bin/php /home/derp/vsrx-fetch-license/parseXML.php $device.xml
  echo "Napping for $napTime seconds..."
  sleep $napTime
done
echo "All done!"

4) Add execution to cron:

0 * * * * /bin/bash /home/derp/vsrx-fetch-license/fetch-licenses.sh > /home/derp/vsrx-fetch-license/fetch.log 2>&1

5) Create file vsrx-expect.sh in /home/derp/vsrx-fetch-license/ and replace values.

#!/usr/bin/expect -f
set timeout 20000
match_max 100000
set vsrxhost [lindex $argv 0];
spawn ssh -o "StrictHostKeyChecking=no" readonlyuser@$vsrxhost "show system license usage |display xml|no-more"
expect "Password:"
send "SECRETPASSWORD\r"
expect "*>"
expect eof

If all goes well, the database is updated every hour. Each run takes around 5 minutes in my case. Check the logfile /home/derp/vsrx-fetch-license/fetch.log for details after the first run.

I've added a panel in Grafana:

Query is configured as follows:

SELECT host as "Hostname", FROM_UNIXTIME(timestamp-3600) as "Licens opdateret", expirationdate as "Udløbsdato", daystoexpire as "Dage til udløb"
FROM `vsrx-licenses`
ORDER BY daystoexpire asc

Good luck.

To upgrade the PoE controller software in the device, run the following command with your fpc-slot id:

request system firmware upgrade poe fpc-slot 0

To get the status of the upgrade, issue the following command for details:

root> show poe controller
Controller  Maximum   Power         Guard    Management   Status        Lldp
index       power     consumption   band                                Priority
   0**      124W      0.00W           0W                  DOWNLOAD_INIT Disabled

...

root> show poe controller
Controller  Maximum   Power         Guard    Management   Status        Lldp
index       power     consumption   band                                Priority
   0**      124W      0.00W           0W                  SW_DOWNLOAD(43%) Disabled

The download requires no network connectivity, as the sofware is stored on the device.

On my EX2300-C, the download progress went to a halt at 95%, I figured it was simply installing:

Controller  Maximum   Power         Guard    Management   Status        Lldp
index       power     consumption   band                                Priority
   0**      124W      0.00W           0W                  SW_DOWNLOAD(95%) Disabled

a few minutes later, the install process had ended:

Controller  Maximum   Power         Guard    Management   Status        Lldp
index       power     consumption   band                                Priority
   0        124W      0.00W           0W                  AT_MODE       Disabled

As per intructions; Please Reboot the system after Upgrade is complete

Should you for some reason need to install a Wordpress-site, here are the basics:

1) Install packages, run:

apt install libapache2-mod-php apache2 php-gd php-mysql php-zip mariadb-server -y

2) Secure MariaDB, run:

mysql_secure_installation

and go through the wizard:

<Enter> to set password if empty
N for no Unix socket
Y to change root pass
Y to remove anonymous users
Y to disallow remote root login
Y to remove test db
Y to reload privilege table

3) Create a user, a database and set permissions

Connect to MariaDB, run:

mysql -uroot -p

Create a user, run:

CREATE USER 'wordpress'@'localhost' IDENTIFIED BY 'V3RY_S3CR3T_P4SSW0RD';

Create database, run:

CREATE DATABASE wordpress;

Set permissions for database and user, run:

GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'localhost';

Apply changes, run:

FLUSH PRIVILEGES;

Exit from MariaDB, run:

quit

4) Download latest version of Wordpress

cd /var/www/html/ && wget https://wordpress.org/latest.tar.gz

5) Extract Wordpress archive

tar xzf latest.tar.gz

6) Move files from newly created "wordpress"-folder and delete it, including the default index.html from the Apache install

mv wordpress/* . && rm -rf wordpress/

7) Set permissions to Wordpress-install, if not already properly configured

chown -R www-data: /var/www/html/

Browse to the webserver and complete the installation wizard.

Consider the following policy:

security {
    policies {
        from-zone untrust to-zone junos-host {
            policy pub-ping {
                match {
                    source-address any;
                    destination-address any;
                    application junos-icmp-ping;
                }
                then {
                    permit;
                }
            }
        }
    }
}

Security policy details:

Policy: pub-ping, action-type: permit, State: enabled, Index: 20, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 1
  From zone: untrust, To zone: junos-host
  Source addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Destination addresses:
    any-ipv4(global): 0.0.0.0/0
    any-ipv6(global): ::/0
  Application: junos-icmp-ping
    IP protocol: icmp, ALG: 0, Inactivity timeout: 60
      ICMP Information: type=8, code=0
  Per policy TCP Options: SYN check: No, SEQ check: No, Window scale: No

When using match-policies in Junos, a match for icmp ping is not found unless the source-port is 2048.

 

Example matching for source-port 2049 (or any other port):

spiderpig@vsrx-lab> show security match-policies from-zone untrust to-zone junos-host source-ip 1.2.3.4 destination-ip 3.4.5.6 source-port 2049 destination-port 1234 protocol icmp
Policy: deny-all, action-type: deny, State: enabled, Index: 19
0
  Policy Type: Configured
  Sequence number: 4
  From zone: untrust, To zone: junos-host
  Source addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Destination addresses:
    any-ipv4(global): 0.0.0.0/0
    any-ipv6(global): ::/0
  Application: any
    IP protocol: 0, ALG: 0, Inactivity timeout: 0
      Source port range: [0-0]
      Destination port range: [0-0]
  Per policy TCP Options: SYN check: No, SEQ check: No, Window scale: No

The above result matches the last policy in the sequence, not the one permitting icmp ping

 

Example matching for source-port 2048:

spiderpig@vsrx-lab> show security match-policies from-zone untrust to-zone junos-host source-ip 1.2.3.4 destination-ip 3.4.5.6 source-port 2048 destination-port 1234 protocol icmp
Policy: pub-ping, action-type: permit, State: enabled, Index: 20
0
  Policy Type: Configured
  Sequence number: 1
  From zone: untrust, To zone: junos-host
  Source addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Destination addresses:
    any-ipv4(global): 0.0.0.0/0
    any-ipv6(global): ::/0
  Application: junos-icmp-ping
    IP protocol: icmp, ALG: 0, Inactivity timeout: 60
      ICMP Information: type=8, code=0
  Per policy TCP Options: SYN check: No, SEQ check: No, Window scale: No

The above example matches the policy permitting icmp ping.

Junos version: 18.3R1.9

When sending alerts from MegaRAID Storage Manager (MSM) fails, even though the SMTP-server configuration is correct and the network access is permitted, it might be due to what I believe is a bug in MSM.

I've seen this issue in servers with multiple NICs/servers having changes made to the pNICs after configuring MSM.

In short, the configuration utility does not bind to the correct IP-address/NIC when saving the configuration. This setting is nowhere to be seen in MSM, which is why it took me hours to figure out. To check and potentially fix the issue, do the following:

1) Open MSM and navigate to Tools > Monitor Configure Alerts

2) Make sure the settings for the SMTP-server are correct

3) Click Save Backup, store the file monitorconfig.xml on the Desktop and click "OK" to close the Configure Alerts-window

4) Edit monitorconfig.xml with Notepad or another text-editor

5) Find the <nic>-tag in the file and set it to the IP-address of the interface that should be used to access the SMTP-server

Example, see last line:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<monitor-config>
<actions>
<popup/>
<email>
<nic>10.106.22.40</nic>
...

6) Save the file and return to MSM

7) Navigate to Tools > Monitor Configure Alerts

8) Click Load Backup, then Yes, select monitorconfig.xml and click OK

9) Navigate to Tools > Monitor Configure Alerts and test again

Note: The dialog may display "The test email could not be sent. Check the mail server settings and try again." - ignore this and check if the email is delivered within a few minutes.

I don't know the mechanism MSM uses for verifying the email is sent - more often than not, the error displays but the email is delivered anyway.

Versions tested: 16.05.04.01, 17.05.00.02, 17.05.01.03