4

Recently I have encountered a problem related to the import of contacts and has to do with the dedupe rules.

When you import a list of contacts through a CSV file and select a dedupe rule, the CRM seems to ignore that rule and go directly to use the 'Unsupervised' rule.

For example, if there is a user with mail xxxxx@eeee.com and you use a 'General' rule with the Weight Threshold and the Weight of 10 in email but in the 'Unsupervised' rule the weight is 10 and the Weight Threshold is 15, the mail would be introduced in the CRM even if the rule that was chosen was the 'General'.

This behavior also happens in dmaster.

Is there any reason to prioritize the 'Unsupervised' rule to the rule you have chosen to perform the import?

EDIT

After more tests I have been able to conclude the following:

Importing data: Name, Last name and email and selecting the skip option.

Unsupervised rules:

A) Name (10) Last name (10) Treshold (30)
B) Email (10) Treshold (10)

General rules:

C) Name (10) Last name (10) Treshold (20)
D) Name (10) Last name (10) Mail (5) Treshold (25)

Then:

A + C = duplicate entry. If the general rule would have been applied correctly should not been duplicated, just skipped.

A + D = duplicate entry.

B + C = skip the entry (works fine)

B + D = skip the entry (works fine)

So as Jon G - Megaphone Tech said, the unsupervised dedupe rule always run, and from there, run your dedupe as well.

So the import first check the unsupervised rule and if the Treshold is greater than the weight of the fields always will import the contact independently the general rule you select.

  • 2
    Can you confirm how you are sure which rule is ends up using? Note that if you have a Contact with more than one of the same email, then each instance of that email is counted towards the Threshold. Hence we recommend something like First 10, Last 10, Email 1, Threshold 21 and not F 10, L10, E10, with T of 30, as the latter will count as a match for a contact that has the same email 3 times (it happens!) even if F and L do not match. – petednz - fuzion May 9 '18 at 20:04
  • 1
    I will do more tests to confirm the problem and being carefull with the weight of each field. When I do this tests i'll edit the main posts and show the results. – Albert V May 10 '18 at 10:00
4

The import code's use of dedupe rules as of this writing (Civi 5.1) is...interesting.

The rules, which are defined in CRM_Contact_Import_Parser_Contact::getPossibleContactMatches involve short-circuiting the dedupe matching if the external identifier is a match. After that, it will ALSO run the unsupervised dedupe - and from there, run your dedupe as well.

While it'd be great to fix this (it's happening here) realistically it will take someone putting in resources (coding time or hiring a coder) to resolve this.

For now, the workaround is to modify your unsupervised rule.

  • Should I open an issue at jira to report this behavior? – Albert V May 15 '18 at 9:47
  • Still true as of Civi 5.12.1. Note the justification for checking the external ID: it's a unique key for contacts, so if the new record matches on external ID, the only two options are to merge/overwrite the new record onto the matching existing record, or to reject the new record altogether. The code chooses to reject the new record if other dedupe criteria would lead to two records having the same external ID. – Noah Miller Apr 18 at 5:00
0

Possibly related is that when Skip is selected (the default option) Contact ID is not shown as a match possibility. Contact ID does show when Fill or Update or No Dedupe is selected. This seems like a bug. (When was Fill added to the options? did that cause regression?)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.