Can I override core behavior of CRM/SMS/Providers.php by creating the file with the changes in my civicrm_custom_php directory?
The answer is "yes" but...
Overriding an entire file means upgrades will become difficult, as you'll have to manually diff the contents of that file with the one in the new version, and reapply your changes.
If possible, try accomplishing your goal using a hook.
The answer is yes, you can do that. I tested it and it worked.