FreeNAS and ESXi config backup

My FreeNAS was running off a single USB so I grabbed another one to setup a mirror but it was a few blocks smaller then the current one so would not mirror. Instead of spending time to make it work, I just took a backup of the config, re-installed to the mirror and restored the config. Super simple. This made me think I should get some automation in place to backup its config vs me doing it manually when I make changes in the GUI.

Doing some research on the FreeNAS forums you can just copy the config db from /data/freenas-v1.db to somewhere else. Since I'm using v11 its a simple 'cp -a' script. Being FreeNAS I thought using snapshots would be useful for change control etc so I created a new dataset (sysadmin) with max compression and copied the config to that location.

Then I thought someone else did something more eloquent so I went looking around and found a nice script from Spearfoot on GitHub that handles this and had the bonus of backing up the config of a ESXi host which I also needed to get some config backups.

I created the directory /root/bin as its in path so easy to keep track of and put the script there.

I ended up changing the script as it puts a timestamp in the file however due to my snapshots I didnt need it. Here is old vs new.

 #fnconfigdest="${configdir}"/"${freenashost}"-"${fnconfigdest_version}"-"${fnconfigdest_date}".db  
 #fnconfigdest="${configdir}"/"${freenashost}"-"${fnconfigdest_version}".db  

I'll still have to manually cleanup files when the version changes however I wanted to retain the version for compatability reasons. This gives me a nice single daily file and then the snapshot handles revision control.

 -rw-r----- 1 root wheel 933888 Feb 22 03:00 freenas-FreeNAS-11.1-U1-f7e246b8f.db  

For the ESXI host this script will SSH to it and then pulls the config. In order to automate I had to first setup certificates between these two systems. Interactive logins are disabled on all my hosts but I didnt expect these two to talk to each this way so had to setup certificates. You can go here or here for more detail on the process if you are unfamilar. These are the commands.

 #cd /root/.ssh  
 #ssh-keygen -N "" -f id_esxi  
 #cat id_esxi.pub | ssh root@ESXI_HOSTNAME_OR_IP_ADDRESS 'cat >>/etc/ssh/keys-root/authorized_keys'  

First line  just puts you into roots ssh dir. The second generates a passphrase less key pair. Think hard here on whether this works in your environment. Being my homelab and interactice logins disabled everywhere I was ok with the risk. Third line will copy the public key to the ESXi host.

Once the keys are done and on each system you just ssh with this switch:

 ssh -i id_esxi root@ESXI_HOSTNAME_OR_IP_ADDRESS  

For the GitHub script I had to add the -i switch (highlighted above) to use the certificate.

 #esxihostname=$(ssh -i /root/.ssh/id_esxi root@"${esxihost}" hostname)   
 #esxiversion=$(ssh -i /root/.ssh/id_esxi root@"${esxihost}" uname -a | sed -e "s|VMkernel ||;s|$esxihostname ||")   
 #esxiconfig_url=$(ssh -i /root/.ssh/id_esxi root@"${esxihost}" vim-cmd hostsvc/firmware/backup_config | awk '{print $  7}' | sed -e "s|*|$esxihostname|")  

This all works via shell and I now have backups of my two main systems.

 -rw-r--r-- 1 root wheel  20623 Feb 22 03:00 esxi-configBundle.tgz  
 -rw-r----- 1 root wheel 933888 Feb 22 03:00 freenas-FreeNAS-11.1-U1-f7e246b8f.db  

Next up was to get the cronjob going so we head over to the FreeNAS GUI under Tasks | Cron Jobs and create it. I did it for 3AM. For the command I am using:

 sh /root/bin/save_config.sh  

Finally create the snapshot for 4AM.

The script will also send an email which is nice.

 Configuration file saved successfully on Thu Feb 22 03:00:00 MST 2018  
   
 FreeNAS:  
 Server: freenas.mydomain.com  
 Version: FreeNAS-11.1-U1 (f7e246b8f)  
 File: /mnt/Pool/sysadmin/freenas-FreeNAS-11.1-U1-f7e246b8f.db  
   
 ESXi:  
 Server: esxi.mydomain.com
 Version: 6.5.0 #1 SMP Release build-5969303 Jul 6 2017 21:22:25 x86_64 x86_64 x86_64 ESXi  
 File: /mnt/Pool/sysadmin/esxi-configBundle.tgz  

Almost done. RAID is not a backup so I still need to do something for fire/theft etc. To address that, I created a weekly cronjob to copy this from the sysadmin dataset into my main dataset which gets rsynced offsite daily. I may change this to daily so I have the most current configs offsite.

Just having about a weeks worth of backups its looking good

Extras

For a Linux/UN*X host you would put it in the accounts .ssh folder authorized_keys.

 cat id_hostname.pub | ssh username@hostname_or_IP_Address ''cat >>/home/username/.ssh/authorized_keys'  
 cat id_hostname.pub | ssh root@hostname_or_IP_Address ''cat >>/root/.ssh/authorized_keys'  

For other commands that ride SSH such as RSYNC you can tell it to use the certificate.

 rsync -a -h --delete -e "ssh -i /root/.ssh/id_hostname"/path/to/source. username@hostname_or_IP_Address:/path/to/remote  

-Kevin

Dell OneShot Driver Update

Sometimes when there is a problem with a machine an older driver is at fault. For firms that have standardized on Dell they have provided a nice tool call Dell Command  | Update. This is a nice tool Dell made to keep their business lines (OptiPlex, Precision, or Latitude for example) driver sets up to date.

It installs an agent that runs on a schedule (monthly by default) to keep the system up to date on drivers. It queries drivers from Dell directly (or via an on site repository) so you can get the latest drivers easily and automatically. As my firm has very few egress points, I chose not to use it in this capacity however it does have a CLI executable! While we use ConfigMgr to do driver management as issues dictate (needing newer nVidia non ISV driver for example), I put together a package in ConfigMgr for the techs to manually update all drivers on a device when it comes in for service. Known as the "OneShot Driver Update".

You can find install instructions to get Command | Update installed on a PC. Once installed, just copy the install folder to a source folder in SCCM. I created two BAT Files that perform the following:

• Remove logs from a previous run
• Pause Bitlocker
• Update Drivers
• Resume Bitlocker if no restart needed
• Optionally copy the logs to a server location

As you are generally updating drivers that impact the boot chain (yes the video driver at times) you need to pause Bitlocker. Older versions of the tool did not handle Bitlocker, which is what started me down the path of a simple BAT wrapper for the tool. The Update drivers bullet above is handled via a specific BAT and custom XML file though I have though about doing a menu in the BAT at some point to combine them. For the XML, you can create it in the GUI version of the tool then save it out and therefore use it to update only the categories you want.

I have created two file sets. One that does all device drivers yet skips applications. The second only updates the system firmware, aka BIOS or UEFI. You just run the appropriate BAT as an admin. They are even called '_RunAsAdmin.BAT'.

For SCCM we just have a package with both BATs called out as Programs and have them published via our internal Package Downloader Tool (PDT). Otherwise you can simply advertise it as available to be ran from Software Center. Or just put on a network location somewhere.

Download

This Script is provided as-is, no warranty is provided or implied.The author is NOT responsible for any damages or data loss that may occur through the use of this Script.  Always test, test, test before rolling anything into a production environment.

You can find these two scripts here.

-Kevin