I created a custom field on groups, which contains a string. When this string is not empty, I want it to be unique. What is the preferred way to do this? Should I implement a hook?
I think a hook's the way to go. For non-custom fields you can specify this a little more easily in the XML definition of the field, but for a custom field, I think the hook is your best best.
I think I'd use
hook_pre. The first is more user-friendly, but since it's at the form level you have to identify everywhere the field can be edited. It's a fairly easy hook to write though - just make an API "get" call to CustomValue with that value and see if you get a count > 0.