I found there were two parts to this, first I needed to identify the current array that was setting the 'default' gear icon, then I needed to set a new one.
This page describes the basics of the tabset hook - so I started with that as a base: https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_tabset/ I built the basic hook
Then I had to figure out which nested array held the information for my custom data and I was unable to perform the code that follows without the array number (not just the custom data number). In my case, it is an 'alerts' tab. This was tricky because the number can change depending on the contact, so I couldn't just ask it to look for the 15th array, because it could be a different number per contact. I used this post's logic to search the array for the id containing my custom data.
With the array number, I was able to reset the icon part of the array
function mymodule_civicrm_tabset($tabsetName, &$tabs, $context) {
//check if the tabset is Contact Summary Page
if ($tabsetName == 'civicrm/contact/view') {
// get the number of the tabset, change custom_00 to the number of your custom data fieldset
$alertstabnum = array_search('custom_00', array_column($tabs, 'id'));
// change the icon from the default gear icon
$tabs[$alertstabnum]['icon'] = 'crm-i fa-flag';
}
}
If there's an 'easier' way or something already set up in the tabset hook, I'm all ears this is just what I found to work.