We've faced duplicate records batch merge issue while moving big data set (around 140K Individuals as Organisation members and cca. 400 Organisations accordingly) from proprietary CRM to CiviCRM and would appreciate any advice of how to move forward. We've successfully started from importing small records amount both via Import GUI and API calls; however after bigger import we've found that a large amount of nested Organisations have been created via Relationships, so, we have one "main" Organisation with properly filled address, etc, and several members and, say, 400 "phantom" organisations with just name and one member visible at Relationships tab.
We have tried "Find and Merge Duplicate Contacts" Individual Rules via GUI, however, after leaving it works for a night we ended up with same total Individuals number and an url like this: civicrm/contact/dedupefind?reset=1&action=map&rgid=7&skipped=XYZ&merged=0 (where XYZ was 8175 just before it was stopped); since no status report has been received, guess "merged=0" part of an url shows number of successfully merges.
api/explorer (Contacts -> Merge) has almost no parameters to work with, just "$result = civicrm_api3('Contact', 'merge');" and "Other" as choose parameter option, so it's not quite clear how to use it.
What really works for us is a multi-step GUI workaround, like:
- Find all instances of Organisation with same name
- Assign search results set to a new group
- Apply one of the Find and Merge Organization Rules
- Wait (cca. half of an hour, thanks to fine tuned MySQL with 13G innodb_buffer_pool_size and tons of other improvements)
- Manually merge several remained records
It works almost great, however it is quite time-consuming since we will have to repeat a process around 400 times. Hope there is alternative/faster way(s), we've no clue where to start from. Please help!
UPD: Thank you all for a valuable responses, it gives us some inspiration of how to attack an issue from an another angle(s). I'd accept both replies, however Xavier's approach sounds more easy for us at this stage: we've got some positive experience using api/scripting before, even if this method did not work 100% for us previously, we're going to try it again.