CiviCRM Stack Exchange is a question and answer site for administrators and users of the CiviCRM Constituent Relationship Management software. It only takes a minute to sign up.
Sign up to join this community
I'm helping out a nonprofit with their CiviCRM. They have thousands upon thousands of duplicate contacts. These contacts are being pulled from an email service, that is sending the same email, sometimes 12+ times, to the same address.
Is there any way to merge all duplicate contacts automatically? Or a way to export all contacts, sort them in, say, Excel, and re-import?
Merging each contact one-by-one using the rules would take weeks.
Thanks!
One Option: The easiest way I have found (in 4.6.4) is to export like you said, with contact ID, do your manipulation in excel to pull out the records you would like to keep, and re-import the records into a group, choosing 'update' when you do so.
If you need to merge information, you could put it on the line with the contact ID you will be keeping.
Then you can grab all of the contacts in your group (or all contacts not in your group, depending on which way you selected) and delete them.
To export these contacts, search for all, select all, and choose 'action>export'.
The import tool can be found under 'contacts>import contacts'.
One thing I have found on our setup is if I import a large dataset (say, more than 10k records at a time) I will get a timeout and the import will stop with 'db error', which is frustrating, so I usually break these up into smaller sheets to import.
Another Option: I believe you can do this programmatically with the API if you are more comfortable doing that. I haven't used this approach though so I don't know how well it works.
Possibly related: Batch merge automation of big data set
Dedupe is an area that's getting a lot of love these days; however, without knowing what version you're using or what you've tried, it's hard to make a recommendation.
Finally - if you're getting so many duplicates, I'm guessing the Supervised or Unsupervised dedupe rule needs some tweaking. You should be able to cut down on the number of duplicates you're facing to begin with.
I would be very careful about using batch deduping on anything other than the latest version. I have fixed multiple issues in conjunction with John Kingsnorth over the past few months.
It is possible to do batch deduping through the UI as Jon describes or via the api - we are almost ready to start running it as a batch job on an ongoing basis using a small drush wrapper script - the main goal of which is to do small chunks at a time, which may be less relevant on a smaller database
