OPNSense ZFS scrub cron task

I recently moved my OPNSense from a Dell R220 to a GoWin R86S which reminded me that I really should get a scheduled ZFS scrub going until the developers include something natively. After trying a few different methods I settled on this method to perform a scrub on a monthly basis.

First step is to create the task. SSH to the server and cd to

/usr/local/opnsense/service/conf/actions.d 

and create a new file called actions_zfs.conf with the following content:

[scrub_zroot]
command:/sbin/zpool scrub zroot
parameters:
type:script
description:ZFS Scrub zroot Pool
message:Performing a ZFS scrub of the zroot Pool

[trim_zroot]
command:/sbin/zpool trim zroot
parameters:
type:script
description:ZFS trim zroot Pool
message:Performing a ZFS trim of the zroot Pool

After saving the file you will need to restart a service so it picks up the new configuration file by running

service configd restart

Head over to the OPNSense web GUI and navigate to SYSTEM: SETTINGS: CRON and select the plus sign to add a cron task and fill out as you see fit. In my example below I am choosing 02:00 AM on the first of the month to run it. For the Command just type 'zfs' and you can select the scrub command we just created. Finally, give it a description and click Save.

After selecting save, be sure to select Apply on the Cron page and you're done. 

If you are using SSD storage that supports Trim (most do), you can create another cron entry to run that task. Performing weekly is good for this IMO since it's not a file server such as TrueNAS. With that said, a better option would be to just enable autotrim for the zroot pool and it will perform trim on blocks when it is convenient. Simply SSH to the OPNSense system and run: 

zpool set autotrim=on zroot

Note that trim has a cost depending if it is an automatic or a manual process. More detail at the man page:

When set to on space which has been recently freed, and is no longer allocated by the pool, will be periodically trimmed ... Automatic TRIM does not immediately reclaim blocks after a free. Instead, it will optimistically delay allowing smaller ranges to be aggregated in to a few larger ones. These can then be issued more efficiently to the storage.

As a bonus you can follow this OPNSense forum post to setup a Monit task so you get alerted when important things happen to the ZFS pool such as it getting full or mirrored drive dies, etc.

Update 

Per the 23.1 release notes, there are now two built-in tasks to perform these ZFs functions built in. They are very similar to the ones I created above. 

If you use this howto and then upgrade you will need to delete the two tasks and recreate using the included ones as they used the same actions_zfs.conf file. Be sure to put zroot in the Parameters field or whatever your pool is called if you did not use the default of zroot.

-Kevin

Create Bi-monthly Pay Calendar Events

Someone I know is paid on a Bi-monthly schedule, aka Semi-Monthly, and wanted some help on having reminders in their calendar. A Bi-monthly schedule is generally where a person is paid on the 15th and the last business day of the month. Its more about the bank ETF processing business days than the firms days of operation. The issue is when these fall on a weekend the payday processing is shifted forward to Friday. I used to be on a pay schedule like this myself way back and was slightly annoyed by not being able to have a calendar item automatically address it. July, August, and February (Leap years!)  made end of month recurrances difficult.My friend uses Google Workspace so my focus was on that but this works with Outlook and others.

So how can we automate this? Google Workspace (also Gmail) nor Outlook natively allows you to shift events around based on conditions such as weekends. You pick the 15th of the month or the third Tuesday, that type of repeating. In this case, we not only have weekends but also for thethe  last day of month with it ending on either 30 or 31 and of course, there is February ending on the 28th with the occasional leap year along with maybe a phase of the moon in there somewhere.

Luckily there is a standard for it that can facilitate way more advanced functionality than even what I'm doing here. Its called iCalendar and its covered by RFC5545 and supplemented by RFC7986. When you export calendar events from Google Workspace/Gmail, Outlook, and most anything else it uses this standard via the .ics file format. This replaced the former VCalendar (.vcs) format.

With all of its power, I was unable to get it into one file so there is one for each pay cycle during the month. Therefore one for the 15th and one for the end of the month. I parsed all sorts of developer documents and such around the fields within an iCal file as well as exporting several of my calendar entries to understand the syntax. I know way more than I ever need to know about this. So lets start with what should be the easier one, the first one on the 15th:

 BEGIN:VCALENDAR  
 VERSION:2.0  
 BEGIN:VEVENT  
 RRULE:FREQ=MONTHLY;INTERVAL=1;BYSETPOS=-1;BYMONTHDAY=13,14,15;BYDAY=MO,TU,WE,TH,FR  
 SUMMARY:My Payday  
 DTSTART;VALUE=DATE:20180430  
 SEQUENCE:0  
 DESCRIPTION:  
 END:VEVENT  
 END:VCALENDAR  

The key is the RRULE (Repeat Rule) line. Let's break it down, however, they can pivot off each other so definitions can change based on what attributes are present and there are many other attributes that can be a part of the RRULE. Confusing right? So I have it listed more from a flow perspective since they build on each other which makes more sense to me.

• FREQ - This is telling it to do a monthly repeat
• INTERVAL - This says how many times a month, in this case we want once
• BYDAY - What days will this apply? Business days of Monday to Friday and exclude Saturday and Sunday.
• BYMONTHDAY - What calendar date can it apply? If Monday to Friday is the 15th then create on the 15th. If the 15th is Saturday then Friday is the 14th so create on that as it matches BYDAY of Friday. If the 15th is Sunday then Friday is the 13th so use that as it matches BYDAY of Friday.  INTERVAL controls how many events can be created.
• BYSETPOS - This is the nth instance once you calculated the other attributes. BYSETPOS is what tells it to back up to the earlier match of Friday due to BYMONTHDAY also matching vs just skipping this instance that FREQ dictates. Officially this is defined as 'Each BYSETPOS value can include a positive (+n) or negative (-n) integer. If present, this indicates the nth occurrence of the specific occurrence within the set of occurrences specified by the rule.'

The other lines of interest would be the SUMMARY which is the title of the calendar entry and the DTSTART which is when to start the calendar entry. I did this starting on my friend's hire date.

If you want to learn way more about all these fields you can reference this article. 

I thought the end of the month would be harder but it's actually simpler as we do not need the BYMONTHDAY field. iCalendar defaults to the first day of the month however since the BYSETPOS is negative, that goes back a day which is the last day of the previous month, and then BYDAY dictates the other half of when it can apply.

 BEGIN:VCALENDAR  
 VERSION:2.0  
 BEGIN:VEVENT  
 RRULE:FREQ=MONTHLY;INTERVAL=1;BYSETPOS=-1;BYDAY=MO,TU,WE,TH,FR  
 SUMMARY:My Payday  
 DTSTART;VALUE=DATE:20180430  
 SEQUENCE:0  
 DESCRIPTION:  
 END:VEVENT  
 END:VCALENDAR  

This does not account for any banking holidays nor if the payday policy states if it falls on a Sunday the paydate is moved forward to Monday but it can be modified for this scenario. New years eve is the only one that might come into play.

If you want to make use of this then copy each code block above to a separate text file ending in .ics. Edit as you see fit such as SUMMARY and DTSTART attributes. To import into Google Workspace (or Gmail) you can follow step 2 of this KB. For Outlook you can follow this KB. For other less used calendar programs you can look up the process yourself. For Outlook and Workspace you can only edit the reminders and edit the events except you cannot edit the recurrence part. Note it will use your default reminder settings so you may want to adjust them after you import.

Maybe I will get motivated to see if I can address weekly/bi-weekly pay cycles and holidays such as US Thanksgiving and Christmas.

-Kevin

Yes, you can still use USB to boot Homelab Servers

Much software used within homelab as well as SME would get installed on flash media. USB and/or SD cards to be specific. TrueNAS and ESX in particular destroy these types of media, more so in recent versions. VMWare has a KB on it for ESX. It formally stated it was unsupported but they backed down and are stating it will be removed in a future release. William Lam has a good article on it as well. For TrueNAS, IXSystems states its not recommended. Even a Pi-hole on an RPi will wear out a commodity SD Card so people use something like log2ram for it to help with wear.

For my homelab I am going through a hardware refresh. My old servers had a dual SD Card internally that ran the host OS. While I used high-end write-focused SD/USB flash media, I did have one SD card die after several years of use so I do want to be mindful of this in my next refresh. What they are saying is don't use the cheap flash media, NOT the USB interface itself, at least currently.

  

My new servers support USB3 internally however USB2 would be fine IMO. So for ESX I found a 128GB M.2 for $15 and got a USB to M.2 adapter for another $15. Since this card was a 2230 form-factor I also obtained this adapter as I didn't like just using the rubberband the adapter came with.

For TrueNAS since it supports multiple OS drives, I made use of two 128Gb mSATA drives I found and USB to mSATA adapters that were $9 each as well. Due to mSATA width, I used a 6" USB3 cable for each. I just used double-sided foam tape to hold them down.

Been using this configuration for a few months now and works just fine. Startups and shutdowns are nice and quick. TrueNAS shows them as 400MB/s vs 600MB/s for SATAIII. My new servers also have internal SATA headers and power so I could possibly transition to that, at least for ESX as I like the ZFS mirror TrueNAS uses for its system OS. Perhaps a SATADOM would work otherwise just have to figure out the vendor's cable part number for the optical header or build my own.

-Kevin