Tuesday, May 26, 2020

Finding Mis-configured DNS with Pi-Hole

Like many, I use Pi-Hole to help block ads and telemetry. Both of mine run on tiny VMs, setup similar to my Unifi controller. Ubuntu with 2 core/512M memory/5GB storage. I happened to look at my parents instance and had a that's not good moment.




98%??? What is going on here? Normally it's in the 20-30% block range. So I go scrolling down the admin page a little and there's this spike around 6AM. A big spike. A really big spike.




This has a feel of some nefarious malware going on. Did my parents join a botnet?





Looking further in the GUI, there were two hosts that caused the jump in lookups.




Both of the top ones are Domain Controllers. At this site, DHCP hands out only the Pi-Hole server and the Pi-Hole is configured to query the DCs for local DNS and Active Directory. They in turn are configured to use the Pi-Hole as their upstream instead of the roots as is the default.

Trying to dig into the logs via the GUI was not going anywhere. There was just too much data and it kept timing out. I did not do any command line parsing of the logs here; thought I'd start with the basics since this was a recent occurrence. The first thing I did was to check all the Windows systems with things like MBAM and Crowdstrike tools, and I looked at the DCs way more closely. Nothing was found. Next was to make sure everything was using the Pi-Hole for DNS. All the Windows devices were, except the DCs; they are set to best practice which is to point to the other than themselves. Note this was the IP of itself, not loopback (127.0.0.1). When I reverted this change, I used loopback.


Back on the Pi-Hole, I found several names that concerned me, places their environment should not be going to, so I started adding those to the blacklist in Pi-Hole. Another day, new hosts in CN TLD.







The two SH.CN hosts became #1 and #2 after I blocked the two highlighted ones above.

The light above my head went off. I only looked at the Windows hosts, not the rest. So I go see what DNS they point to. ESX was using the DCs so I changed it to the Pi-Hole. My backup Linux server was using the Pi-Hole, and both DCs.

Linux does not have the concept of primary DNS and Secondary DNS, just DNS Server 1, 2, etc. At this point I performed a cardinal sin in tracking problems down – I changed multiple things. Ugh. I changed this Linux box to use Pi-Hole only. Secondly, I changed the DCs to point to Pi-Hole as primary and the other DC as secondary. I also turned on DNS logging on the DCs. The next day lookups dropped exponentially but were still high around 6AM, with around 6000 lookups instead of the nearly billion. All PTR records. That's strange. The DNS logs in the DCs only showed normal traffic for the Active Directory domain with a few Microsoft domains being forwarded to the Pi-Hole.

Exactly what I expected to see from them, however this pointed me to the troubled host – my Linux backup server. The problem is my Linux box. Again, I thought nefarious malware. Ugh, how? It’s a stripped server install and only I use it. Going back to the Pi-Hole logging is usable again and it shows the spike from my Linux box after 6am. The spike starts at 6:25am and stops at 6:32am. I concurrently was in my email and notice my daily logwatch had a timestamp of 6:32am from that server. Logwatch parses all your system logs and sends you a nicely formatted email of what happened that day. So I get on it and see when logwatch would run. Its script is located in /etc/cron.daily and that is scheduled to run at 6:25am according to crontab.


 kevin@backupserver:~$ cat /etc/crontab  
 # /etc/crontab: system-wide crontab  
 # Unlike any other crontab you don't have to run the `crontab'  
 # command to install the new version when you edit this file  
 # and files in /etc/cron.d. These files also have username fields,  
 # that none of the other crontabs do.  
 SHELL=/bin/sh  
 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin  
 # m h dom mon dow user command  
 17 *  * * *  root  cd / && run-parts --report /etc/cron.hourly  
 25 6  * * *  root  test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )  
 47 6  * * 7  root  test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )  
 52 6  1 * *  root  test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )  
 #  

I remove logwatch script from cron.daily and the next day, no spike in lookups. I have a sigh of relief. It’s not compromised, just misconfigured. Looking at the logwatch email, the fail2ban section was most of it. For some reason my parents’ ISP gets scanned a lot more than mine. Partly due to me using GeoIP via PFBlockerNG in PFSense and them using a commodity router. This backup server does have SSH enabled, certs-only, no logins. I use fail2ban to block brute force SSH attempts. It adds the IP to the firewall blocking them after X attempts by watching various log files, and it has a module for logwatch.

I'm not finding anything in its files, so parsing fail2ban wiki I find this article around DNS lookupsSure enough, /etc/fail2ban/jail.local, use_dns is set to Yes. I change it to No and the daily lookups drop to around 1000. I checked another system and it was set to warn. I changed it to No as well. I don’t care what DNS reports back from these IPs doing the scanning.

All happy now and back where expected.





There is still a small spike but much more tolerable.





So what happened? While Logwatch was the root cause IMO, DNS misconfiguration was what happened. Fail2ban shows thousands upon thousands of SSH login attempts. Someone smarter in DNS than me would have to confirm or deny, but my theory is all the DNS servers did multiple lookups per record and none liked the NXDOMAIN response. My server looked it up against Pi-Hole. It didn’t like the response, so looked up against DC A. DC A in turn looked up against DC B, which went upstream to the Pi-Hole. It didn’t like the response so it went to its second server, itself. DC A in turn went upstream to the Pi-Hole. My server didn’t like the response so I went to DC B, which in turn went to its primary, DC A, which went upstream. Then DC B went upstream. Finally, my server stopped once it exhausted all its DNS servers. I need to draw this out!

Regarding my concern around the CN TLD, many of the IPs I researched, those hosts were the authoritative DNS for them. This partially explains why they were so high in the query lists.

-Kevin








Monday, May 18, 2020

Perform Monthly ConfigMgr Patch Cycle Work

So I recently updated a document for someone on how to perform their monthly Microsoft patch cycle within ConfigMgr and thought I would share it. Some info has been changed to protect the innocent. 

How does yours compare?





Sunday, January 19, 2020

Create Final Windows 7 AIO ISO

Now that Windows 7 is EOL after receiving its final patches on the 14th, I went on to update my AIO (All in One) install with one that is 99% patched.  If you are unaware, it's just an Install.wim with several indexes. There are many reasons to still use Windows 7 so having it start up initially being patched already is a time saver and smart security-wise.

My Current AIO has all versions from Starter up to Enterprise however it excludes K and KN versions. I never expected to use those under any circumstances. I only installed Starter to see what it was about.


First is to update it, which Jason Sandys covers in great detail and all credit to him. He covers how to create a slipstreamed WIM that has the majority of updates injected to save a ton of time and be safer. I started down the road of a simple bat file to inject updates via DISM but Jason's script is way more intuitive and powerful.

While he wrote it in November of 2019 there are a few newer patches to apply. I just modified his script to change the KB numbers. You could also append the Jan 2020 final cumulative after applying the November 2019 one Jason covers.
  • KB4525235 -> KB4534310 (Windows 7 Cumulative)
  • KB4525106 -> KB4534251 (IE11 cumulative)
  • KB4523206 -> KB4536952 (November 12 2019 Servicing Stack update)
Or for the relevent lines with original ones commented out.

 ::SET WINDOWS7=4474419 3020369 3125574 4490628 4523206 4525235  
 SET WINDOWS7=4474419 3020369 3125574 4490628 4536952 4534310  
 ::SET IECU=4525106  
 SET IECU=4534251  

Once you have the updated WIM ready to go you can create the AIO. There are many ways to create one. I just use dism to export a single WIM into the AIO WIM but this link shows a few ways to do it. GIMAGEX is another option I like to use. For dism its pretty simple. I adapted Jasons folder structure.

 dism /Export-Image /SourceImageFile:image\Prox64.WIM /SourceIndex:1 /DestinationImageFile:image\install.wim /DestinationName:"Windows 7 Professional Final x64" /Compress:max  
 dism /Export-Image /SourceImageFile:image\Enterprisex64.WIM /SourceIndex:1 /DestinationImageFile:image\install.wim /DestinationName:"Windows 7 Enterprise Final x64" /Compress:max  
dism /Export-Image /SourceImageFile:image\HomePremiumx64.WIM /SourceIndex:1 /DestinationImageFile:image\install.wim /DestinationName:"Windows 7 Home Premium Final x64" /Compress:max

Some suggestions to share:

  • If you are doing both 32-Bit and 64-Bit use the 32-Bit media as the ISO source as it can install a 64-Bit OS, however, the opposite is not true. Due to this I also have a 7ZIP file with the 64-Bit ISO, minus the two WIM files, to perform an edition change and some types of repair you boot from the install media for. This is few and far between though.
  • Use something to differentiate 32-Bit and 64-Bit in the WIM using the Name attributes. It will detect which Edition and architecture it is but won't let you use the same name in the WIM. I just use 'x64' for all 64-Bit ones.
  • delete the sources\ei.cfg file. Otherwise, it will install the OS Edition stated in ei.cfg. If this file is missing then you are prompted as shown above for the edition and architecture you want.
  • My first AIO has Starter as index 1 so I have to scroll down for Professional or Ultimate when I use the AIO personally and Enterprise professionally. Rarely do I use the other Editions. I created my final AIO with the fully patched Pro and Enterprise that I use the most as index 1 and 2. Then went from unpatched (SP1 only) Enterprise down to Starter.
  • All 64-bit was first then 32-Bit. As my first AIO was from 2010 there was a lot of 32-Bit work. I have not touched 32-Bit in many years but it's still there just in case I get an old system CPU without the X86-64 instruction set.
  • To test, just take your patched one and replace USB\sources\install.wim. I did this on all modified WIMs before going through the AIO process.
Once your Final AIO WIM is ready you can create the ISO for VM use or the rare case of burning optical media. With the injected updates the WIM is over 4GB in size so it will not fit on a USB stick formatted with FAT32. I talk about this in relation to Windows 10 here and my need to use optical media for it. For USB, Windows 7 does not support a second NTFS volume however you can just split the WIM using dism. In my case this WIM is 6.7GB so I have to replace install.wim with install.swm and install2.swm. This also means I have to burn to a DVD9 (Dual layer) instead of DVD5 (single layer). You could make a 64-Bit only if you never anticipate using really old hardware.

 Dism /Split-Image /ImageFile:C:\sources\install.wim /SWMFile:C:\sources\install.swm /FileSize:4700  


I did all this on a Windows 10 Pro workstation. For the ISO file creation, you will need to install the ADK relevant to your OS. In my case the 1903 version since I am on 1909. This gives you the oscdimg.exe executable needed to generate the ISO file for your new AIO. On my 1903 ADK its located in Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment Tools\amd64\Oscdimg\. An example command to generate an ISO file. 

 oscdimg.exe -lWindows_7_SP1_AIO_Final -m -u2 -bE:\W7AIOFinal\Boot\etfsboot.com E:\W7AIOFinal E:\Windows7_SP1_AIO_Final.ISO  



My flow is
  1. Copy Windows7_SP1_AIO.ISO from FreeNAS
  2. Extract to folder Win7AIOFinal
  3. extract relevant index to be patched
  4. run through Jasons script to patch
  5. import modified WIM into new WIM file
  6. import rest of AIO indexes into new WIM File
  7. Split install.wim to install.swm
  8. replace WinAIO7final\sources\install.wim with install.swm files
  9. Create ISO file
  10. test in ESX
  11. update Windows 7 AIO USB stick
  12. Put ISO onto FreeNAS and backup

Extras

As long as Windows 7 has been around you may also want to inject newer storage drivers into the Boot.wim so it can install Windows 7 on newer storage controllers. I touch on this while getting Windows 7 on an unsupported AM4 Ryzen 3 system personally. Several motherboard manufacturers have a tool to inject drivers needed for newer hardware into the boot.wim so you don't have to track them down. ASUS, MSI and ASROCK to name a few. You can do this yourself with dism and the add-driver switch. This is also useful for the NVMe and/or TPM 2.0 patch as Microsoft pulled it so you can install Windows 7 to NVMe storage.If you have them, you can inject the NVMe and TPM 2.0 updates into the install WIM as well. I talk about NVME and Windows 7 here.

With Microsoft charging for support past its original end, this Final may not be really the final. See what that entails!

This script is provided as-is; no warranty is provided or implied. The author is NOT responsible for any damages or data loss that may occur through the use of this script.  Always test, test, test before rolling anything into a production environment.

I created a script to do all the work you can use. Not very pretty but it will get the job done and can be easily modified. It will perform the following:

  • export all WIMs from the AIO
  • run Jasons script against a few
  • Create New WIM
  • Split the WIM into SWM files
  • Create ISO