Import Contacts with dedupe rules

Question

Recently I have encountered a problem related to the import of contacts and has to do with the dedupe rules.

When you import a list of contacts through a CSV file and select a dedupe rule, the CRM seems to ignore that rule and go directly to use the 'Unsupervised' rule.

For example, if there is a user with mail [email protected] and you use a 'General' rule with the Weight Threshold and the Weight of 10 in email but in the 'Unsupervised' rule the weight is 10 and the Weight Threshold is 15, the mail would be introduced in the CRM even if the rule that was chosen was the 'General'.

This behavior also happens in dmaster.

Is there any reason to prioritize the 'Unsupervised' rule to the rule you have chosen to perform the import?

EDIT

After more tests I have been able to conclude the following:

Importing data: Name, Last name and email and selecting the skip option.

Unsupervised rules:

A) Name (10) Last name (10) Treshold (30)
B) Email (10) Treshold (10)

General rules:

C) Name (10) Last name (10) Treshold (20)
D) Name (10) Last name (10) Mail (5) Treshold (25)

Then:

A + C = duplicate entry. If the general rule would have been applied correctly should not been duplicated, just skipped.

A + D = duplicate entry.

B + C = skip the entry (works fine)

B + D = skip the entry (works fine)

So as Jon G - Megaphone Tech said, the unsupervised dedupe rule always run, and from there, run your dedupe as well.

So the import first check the unsupervised rule and if the Treshold is greater than the weight of the fields always will import the contact independently the general rule you select.

Answer 1

The import code's use of dedupe rules as of this writing (Civi 5.1) is...interesting.

The rules, which are defined in CRM_Contact_Import_Parser_Contact::getPossibleContactMatches involve short-circuiting the dedupe matching if the external identifier is a match. After that, it will ALSO run the unsupervised dedupe - and from there, run your dedupe as well.

While it'd be great to fix this (it's happening here) realistically it will take someone putting in resources (coding time or hiring a coder) to resolve this.

For now, the workaround is to modify your unsupervised rule.

Answer 2

Still having this issue in January 2021, with version 5.32.1 But the suggested workaround works fine.

To summarize the long-winded explanation below: When trying to import contacts using a dedupe rule of External ID Only, it will not work as expected. An effective workaround is to temporarily replace the unsupervised rule (Email only) with the desired rule (External Id Only). Things work as expected, and afterward, unsupervised rule can be changed back to (Email only).

Our problem - an organization has been keeping a flat file in a spreadsheet for years, with columns for various meetings, volunteer dates, etc. So trying to import 2400 rows of data with about 50 columns of sparse data. Individual constituents can appear multiple times in the rows, and rows are in order by when the data was collected, making information in a particular row more recent (accurate) than in prior rows. The desired total information import for a contact would then be the latest information in each column, and where the latest row is empty for a given column, then the data in the next most recent row is used, and so on.

So we have made up a unique External ID for each row that effectively uniquely identifies each individual. Since we already have contacts being generated by program registrations through CiviEvent, we want to have the info in the system (e.g. email address - this latest piece of data is presumed to be more accurate than earlier transactions with this person) be kept, but have any blank fields filled in from the most recent row for that contact, (i.e. the "fill" option in contact import), etc. until there are no more rows for that contact.

Our import spreadsheet is in ascending order of External ID, and then secondarily in descending order of the order that was in the original spreadsheet. Thus, for e.g. External ID 24, there might be three rows (e.g. 31-33) of data for that individual, and so the import spreadsheet has ID 24 in three rows, which in the import spreadsheet are in the order of 33, 32, 31. That is, for a given External ID, the most recent data is encountered in the import spreadsheet first, then the second most recent second, etc.

So we set up a dedupe rule "External ID only" and use it with the "fill" option. We presumed that the data in the system (where we had manually added the correct external id) would prevail, augmented by any columns in 33, then 32, then 31, where only the first (latest) information would be retained.

We were then surprised that the import didn't work, and only a few of the contacts were being imported with a test file of the first 100 rows. Finding this issue here on Stack Exchange, we realized we were being affected by something similar (or the same - we haven't looked at the code as suggested).

We used the workaround suggested; changed the unsupervised rule (email only) to general, created a new rule (External ID only) and made it the unsupervised rule, then did our import. It works exactly as expected. After the big import is completed, we will change the unsupervised rule back to email only (so that people signing up for events won't all result in duplicate contacts, etc.)

Answer 3

Possibly related is that when Skip is selected (the default option) Contact ID is not shown as a match possibility. Contact ID does show when Fill or Update or No Dedupe is selected. This seems like a bug. (When was Fill added to the options? did that cause regression?)