Tuesday, January 29, 2019

Tiny Ubiquity Networks UniFi Controller VM

Like many out there, I use Ubiquiti's Enterprise devices at home. I don't do routers but instead use a firewall (PFSense currently) with a separate Access Point for WiFi so I can upgrade them separately. Been this way for two decades now. Currently only have a Unifi AP (UAP-AV-PRO-US)  however if they ever make a 24 port gig switch with two SFP+ and two POE(+) I'm all over that. This would be the ultimate soho/homelab purpose switch IMO.

With that said, Ubiquiti has a controller for their devices to manage firmwares, configurations, and track metrics. Really nice featureset. They have a device you can buy which is pretty slick, especially the gen 2 version. I just cannot justify the price to control only one AP today so I use their controller software. It used to be a Windows 7 VM on ESX (did other stuff as well) so with its demise looming next year, I decided to move it to a Linux Server VM. Glen R, a user on the ubnt forum created a fantastic script to automate the install process once you get the OS ready, so all credit goes to him. I thought I would document how I created a tiny VM to support it. I even ended up creating a 'template' VM that friends are using with thier clients who have Ubiquiti devices.

While this is around VMWare ESX, any hypervisor of choice, such as ProxMox/Hyper-V would work great as well, even on Windows 10. Everything is a VM, even my pi-hole adblocker is a VM even though it was designed for the Raspberry Pi hardware.

Since I moved from gentoo linux to Ubuntu years ago I elected to use that so I downloaded the Ubuntu Server 18.04 LTS version ISO. Set it to autoupdate and forget about it for the most part as it will be supported into 2023.

First was to get on the current Windows 7 controller and backup its configuration and save it out. Stopped the controller service in case I needed to fall back on it.

Created a VM with these settings and mounted the ISO
  • 2 CPU (1 socket, 2 core)
  • 2 GB memory
  • 10GB thin provisioned storage
  • vmxnet3 network
  • paravirtual storage

Start up the VM and boot from the ISO. Install with defaults with automatic security updates. Nothing fancy like /var/usr etc on its own partition. Just one partition.

After first login I modify /etc/fstab to move /tmp to memory by adding this

 tmpfs      /tmp      tmpfs      defaults,nosuid     0 0  

It was unable to find some basic packages. I am guessing this is fixed in 18.04.1 that is available now, however, I had to add the main repositories as they were not in the install ISO media.

 sudo add-apt-repository main universe multiverse restricted  

Then update from the ISO to the latest code

 sudo apt-get update  
 sudo apt-get upgrade  

Restarted it then added some other packages. First was rsnapshot. Used this for many years since I was introduced to the snapshot concept via NetApp Filer. It is a filesystem snapshot utility based on rsync to backup folders on various cycles.

 sudo apt-get install rsnapshot   

 I configured it to back up the /etc directory and /var/lib/unifi where the controller configuration is kept. Lots of guides to set up rsnapshot in more detail. Edit /etc/rsnapshot.conf and set these lines

  snapshot_root /home/rsnapshot/   
  backup /etc/ localhost/   
  backup /var/lib/unifi localhost/   

Then create an /etc/cron.d/rsnapshot file with these below settings. Should be a sample file at this location actually, or put in crontab.

 0 */4      * * *      root  /usr/bin/rsnapshot hourly  
 30 3  * * *      root  /usr/bin/rsnapshot daily  
 0 3  * * 1      root  /usr/bin/rsnapshot weekly  
 30 2  1 * *      root  /usr/bin/rsnapshot monthly  

This will run the snapshot every 4 hours then also do a daily (3:30AM), weekly (3:00AM), and monthly (2:30AM) run. adjust as you see fit. since they are on the same file system they use little space as rsnapshot use hardlinks and renames folders from the current hourly to the oldest monthly snapshot.

Next is to install haveged. This helps a VM to generate entropy for randomization such as SSH key generation and what not. It keeps /dev/random and /dev/urandom full.

 sudo apt-get install haveged  

Edit /etc/default/haveged to contain

 DAEMON_ARGS="-w 1024"  

and set it to start at boot

 update-rc.d haveged defaults  

Set the timezone so shell shows correctly using the wizard.

 dpkg-reconfigure tzdata  

Finally clean up old packages

 sudo apt-get autoremove  

Now the OS is all ready to go. From here, follow Glenn's directions in his forum post which is pretty straight forward. I will not cover that here. I used 5.9 to replace the 5.6 I had on the Windows 7 VM. I have not needed to upgrade the controller yet but Glenn has a script for that also at the same link.

Once the controller was installed I modify my DNS entry for unifi.mydomain.com to point to this system as the AP will look that up from time to time. Create one if you dont have one. Then open the web interface to https://unifi.mydomain.com:8443 and imported my config and after a few minutes it found the AP and adopted it.

As I use the free Veeam scripts and rsnapshot I do not backup the controller config to another host, but you can adapt my FreeNAS and ESX posting with keygen.  Be sure to set this VM autostartup setting in ESX.

After running for a few months it barely is noticed on the host

Something else to consider is a DHCP option on your router to tell your Unifi equipment where the controller is.  Option 43 specifically. More info here as it is not straight forward.

ConfigMgr upgrade error "Failed to apply update changes 0x87d20b15"

We attempted to upgrade one of our ConfigMgr environments from 1802 to 1810 and it failed out in the wizard on the 'Upgrade ConfigMgr Database' step. Looking at cmupdate.log we found this error.

 Failed to apply update changes 0x87d20b15  
 Error information persisted in the database.  

Going further back we then find this nice error.

 ERROR: Failed to execute SQL Server command: ~ DECLARE @viewname  nvarchar(255) ~ DECLARE newviews INSENSITIVE CURSOR FOR ~  SELECT name FROM sysobjects WHERE type='V' AND name like 'v[_]%' ~ OPEN newviews ~ FETCH NEXT FROM newviews INTO @viewname ~ WHILE (@@FETCH_STATUS = 0) ~ BEGIN ~  EXEC('GRANT SELECT ON ' + @viewname + ' to smsschm_users') ~  FETCH NEXT FROM newviews INTO @viewname ~ END ~ CLOSE newviews ~ DEALLOCATE newviews  

After much investigation, it turns out someone created a custom view for the ConfigMgr database. We backed up these views and deleted them and retried the upgrade. That fixed this issue.

For the rest of the upgrade story it was a loooong weekend and not having a good time. It progressed further but still on the Upgrade ConfigMgr Database step failed out. cmupdate.log showed that we encountered some deadlocks on the database and the upgrade panicked out and retry stayed greyed out after other tasks were performed to reset it. We then chose to do a site recovery to get us back to pre upgrade state (you do manually fire those off before upgrade right?) We then tried again and it froze at another spot past the ConfigMgr database step. sWe also had a "ghost" secondary. It was torn down years ago and now that we are without Secondarys we actually had a site link present still. This prevented us from using the upgrade reset tool that recently came out. We ended up involving Microsoft who got us going with 1810 and fixed these other quirky things also.


Friday, January 11, 2019

PxeInstalled (PXE Provider is not installed) error in smsdpprov.log

As my firm got purchased by another I am now starting to collapse the ConfigMgr environment. As it was designed to service 50K endpoints before breaking a sweat, it now manages a couple thousand systems that have not been migrated yet so is way overkill now. As we also use 1E Nomad all we have are the back office roles to contend with. First was to downsize the primary site to remove the multiple MP/DP/SUP and then SUPs on the secondaries. After that, I was to start collapsing the secondaries and converting those hosts to be only DPs. Eventually, this instance will drop to a couple of servers to be kept around for historical use.

So after uninstalling the secondary and cleaning up SQL etc I installed a DP role on it. Once done I started injecting the prestaged content, however I started seeing the following errors in the DP's smsdpprov.log.

 [D28][Thu 01/10/2019 04:06:09]:RegQueryValueExW failed for Software\Microsoft\SMS\DP, PxeInstalled  
 [D28][Thu 01/10/2019 04:06:09]:RegReadDWord failed; 0x80070002  
 [D28][Thu 01/10/2019 04:06:09]:PXE provider is not installed.  

PXE was not flagged during install so after double-checking settings I ran off to Google to see what others have around this. All I could find was people saying to live with it. Seemed strange as this was not on other DPs I checked, so I thought I'd try a quick PXE enable then disable on the DP. By quick I mean enable it, come back later and disable once I validated it was installed via smsdpprov.log and distmgr.log. Sure enough, smsdpprov.log only shows 'PXE provider is not installed' occasionally so the problem is fixed.

Now if it will finish hashing the prestaged content so I can update its boundry group so it will serve its purpose and move onto the next one!