3

I'm helping out a nonprofit with their CiviCRM. They have thousands upon thousands of duplicate contacts. These contacts are being pulled from an email service, that is sending the same email, sometimes 12+ times, to the same address.

Is there any way to merge all duplicate contacts automatically? Or a way to export all contacts, sort them in, say, Excel, and re-import?

Merging each contact one-by-one using the rules would take weeks.

Thanks!

1
  • 2
    what version CiviCRM you are using? It is always helpful if you mention the version when asking questions here :)
    – Ghost
    Jul 25, 2016 at 15:37

3 Answers 3

3

One Option: The easiest way I have found (in 4.6.4) is to export like you said, with contact ID, do your manipulation in excel to pull out the records you would like to keep, and re-import the records into a group, choosing 'update' when you do so.

If you need to merge information, you could put it on the line with the contact ID you will be keeping.

Then you can grab all of the contacts in your group (or all contacts not in your group, depending on which way you selected) and delete them.

To export these contacts, search for all, select all, and choose 'action>export'.

The import tool can be found under 'contacts>import contacts'.

One thing I have found on our setup is if I import a large dataset (say, more than 10k records at a time) I will get a timeout and the import will stop with 'db error', which is frustrating, so I usually break these up into smaller sheets to import.

Another Option: I believe you can do this programmatically with the API if you are more comfortable doing that. I haven't used this approach though so I don't know how well it works.

Possibly related: Batch merge automation of big data set

2
  • I was able to download a CSV of all 38k+ contacts and will need to sort through these in Excel. I'm hoping that since I exported ALL contacts, that all of their relevant data was moved with them. I'll provide an update once I'm able to sort through all this data. How would I determine which Internal Contact ID I should be keeping? Or is that something that would be up to me?
    – user3540
    Jul 26, 2016 at 16:47
  • Yeah deciding which Internal Contact ID to keep is something that would be up to you. As long as you consolidate all the info you need per person per ID it shouldn't matter. I can't speak to whether all relevant data is there, as I don't know your setup, any custom fields that might exist, etc. To be clear, this method is way easier if most of these contacts are empty due to website signups with slightly different organization names or something, which often happens to us. I would try the UI or API suggestions below first if most of these contacts have actual data that needs to be transferred. Jul 26, 2016 at 17:32
2

Dedupe is an area that's getting a lot of love these days; however, without knowing what version you're using or what you've tried, it's hard to make a recommendation.

  • If you haven't tried it yet, try the "Batch Merge Duplicates" button (see screenshot below). If you're not using the very latest versions of CiviCRM, you'll probably have one and not two of those buttons.
  • If you're using version 4.7+, you can use "Force Batch Merge" to get past any data conflicts.

Finally - if you're getting so many duplicates, I'm guessing the Supervised or Unsupervised dedupe rule needs some tweaking. You should be able to cut down on the number of duplicates you're facing to begin with.

enter image description here

3
  • I'm running version 3.4.6. I went into the Find and Merge Duplicate Contacts' section and do not appear to have any Supervised or Unsupervised Dedupe rules, nor options for Batch Merge. My screen looks like this: imgur.com/a/uYlME
    – user3540
    Jul 26, 2016 at 15:33
  • @jbuenooooo That's a VERY old and insecure version - please update immediately! But also, you won't see those buttons until you press "Use Rule" to actually find duplicates. Please read docs.civicrm.org/user/en/stable/common-workflows/… and let us know what questions you have after that! Jul 26, 2016 at 17:32
  • I don't think the organisation i'm helping out is looking to upgrade right now. I looked through your link and it all makes sense - however when I run a rule for individuals, based on their email address I do not have the option to Batch Merge. I'm guessing it's a feature introduced after version we're using. As you can see, there's no checkbox to the left imgur.com/a/38FOi
    – user3540
    Jul 26, 2016 at 19:43
1

I would be very careful about using batch deduping on anything other than the latest version. I have fixed multiple issues in conjunction with John Kingsnorth over the past few months.

It is possible to do batch deduping through the UI as Jon describes or via the api - we are almost ready to start running it as a batch job on an ongoing basis using a small drush wrapper script - the main goal of which is to do small chunks at a time, which may be less relevant on a smaller database

https://gerrit.wikimedia.org/r/#/c/290828/5/sites/all/modules/wmf_civicrm/scripts/civicrm_merge.drush.inc

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.