4

On a reasonably big data set (e.g. one with 70,000 individual contacts), it's not hard to kill your server with a naive manual de-duplication attempt. For example, if you don't specify a group, that number of contacts generates about 5 billion comparisons.

Has anyone thought of a way to prevent unreasonable deduplication attempts?

2

I think Parvez's work on deduplication performance is a big piece of this puzzle: http://vedaconsulting.co.uk/civicrm-dedupe-workflow-ui (see step 1). Some of the answers here are also relevant to improving performance.

However, neither of these solves what's probably your most common use case - someone running a massive deduplication script. One solution might be to feed deduplication into a queueing system. Another might be to permission the dedupe screen - or even permission specific dedupe jobs, so that someone can run Parvez's smart dedupe but not a standard dedupe rule.

  • Yes, this is a good answer to a complementary question. What I really want is a way to intercept the request, do a calculation on how big a request it is, and stop with an error message if it's too big. I'd guess the right place to do that is the form submit hook. It might be nice to make the maximum number of comparisons configurable. The other thing to do is ensure that the criteria is going to be reasonable - e.g. a non-zero match total (which should probably be in core, anyway). – Alan Dixon Jun 19 '15 at 18:41
0

It occurs to me that when you can't find the appropriate screwdriver, there is of course the hammer.

Here's one version of a hammer: https://stackoverflow.com/questions/1189039/how-do-i-routinely-kill-mysql-queries-that-have-been-alive-for-too-long

It'd be nicer to have an extension that analyses the sql before executing it and catching/analysing it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.