I have filtered the data being shown in a custom group tab on contact page but I'm struggling on how to change the count being shown next to the tab name. Could anyone help me how? thanks
I assume you used a tabset hook to adapt the tab on the contact page. That is where you can influence the count as you can see here:
$tabs = [ 'id' => "mediweattestbeheer", 'url' => CRM_Utils_System::url("civicrm/mediweattestbeheer/page/mediweattest", "reset=1&mid=" . $contactId, TRUE), 'title' => "Attesten", 'weight' => 16, 'count' => CRM_Mediweattestbeheer_BAO_MediweAttest::telAttestenMedewerker($contactId), 'class' => "livePage", ];
For the count parameter I have used a function that counts the number of records in the table for the specific contact.