On sites with fairly small numbers of contacts, running a "Find duplicate contacts" operation on all contacts is not an issue.

But as the number of contacts grows, this becomes increasingly impossible. (And I acknowledge that a lot of good work has been done in recent years to address this, which is great!) So we run the duplicate search against smaller groups of contacts. And we tweak the server parameters, and maybe database indexes, to facilitate longer and/or more efficient searches. We also try to avoid rules that might skip indexes (e.g. rules using a Length specifier).

That last one is reasonable enough, but the first two remedies have significant limitations:

  • By searching on increasingly smaller groups, we increase the tedium of the search operation, which can lead to a snowballing tendency to avoid running duplicate scans (it becomes more tedious as our database grows and thus more likely to be avoided by busy staff).
  • Tweaking server configurations requires specialized knowledge or access which is beyond the reach of most organizational staff.

For sites with a "larger" number of contacts (where "larger" means "too large to run a duplicate search on all contacts given my current server configuration"), what's the most common approach for finding potential duplicates, other than the above-mentioned options?

  • Is the issue that running it is too slow, or gives too many results?
    – petednz - fuzion
    Commented Feb 2, 2023 at 18:26
  • Petednz, non-technical site admins will answer with "The problem is that the browser times out, and meanwhile my site is frozen." My impression is that in most cases it's an N-squared problem caused by searching on a large group (or full db) against the full db. More complex dedupe rules make it worse, but even simple rules will hit this problem at larger values of N.
    – TwoMice
    Commented Feb 3, 2023 at 13:52
  • we did some work so that a Sch Job runs once a day, builds a Group based on all the matching rules (3 of them) which we wanted to use for a Dedupe, and added a feature so that you can specify a Group to dedupe from (as well as against) so it becomes much more specific, runs more quickly. I can ask Jitendra to explain share more if interested
    – petednz - fuzion
    Commented Feb 3, 2023 at 19:20
  • Yes petednz very interested. Sounds like some potential for a generalizable solution (insert caveat here etc.)
    – TwoMice
    Commented Feb 3, 2023 at 20:26

2 Answers 2


You may want to have a look at the Extension X-Dedupe which was built for a number of use cases built in dedupe features cannot handle.

  • Yep, saw that. Haven't made time to play with it yet. The docs say "Larger datasets can quickly bring the built-in deduplication to its limits .... XDedupe follows another general approach ...". This sounds like addressing the issue in my question. Anywhere I can read more about this "another general approach"?
    – TwoMice
    Commented Feb 3, 2023 at 13:56

Reserved Rules

I've found that the reserved rules are much more performant that custom dedupe rules created.

  • Name and Address (reserved)
  • Name and Email (reserved)

In one use case, a database of all ~1 million contacts would complete the scan in 20-30 seconds. For custom rules, it would take a multitude times longer or just fail. It would be interesting to investigate why these are so different.

External Processing

With some more technical help, you can scan your MySQL for dupes and then re-import the known dupe contact ID's to a group to target just those.

Related Tips for Site Admins

Other tips worth mentioning, even though they fall into the limitations category you mentioned.

It does avoid user frustration by setting "Default limit for dedupe screen" at Administer > Misc (Undelete, PDFs, Limits, Logging, etc.) or via settings file

$civicrm_setting['CiviCRM Preferences']['dedupe_default_limit'] = '20000';

From a practical standpoint as a site admin, I encourage them to put their most valuable contacts (contributors, members, event attendees) into a group and work thru those dupes first.

Finally, if you setup a dedupe rule that is absolutely bullet proof I've run process_batch_merge scheduled job in aggressive mode to handle dupes straight away that should have never occurred in the first place and are simple merges.

  • Interesting comment about the speed of Reserved. Have you made an equivalent rule that is not reserved, and then run it as a direct comparison?
    – petednz - fuzion
    Commented Feb 6, 2023 at 19:02
  • I don't have access to the install to test this anymore but that would definitely be the next test to confirm. It just adds to civicrm_dedupe_rule and civicrm_dedupe_role_group just like reserved rules do, not a custom option group etc. Pretty unrelated but similar slowness is experienced when importing to custom fields and not only core fields.
    – Andy Burns
    Commented Feb 6, 2023 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.