2

I love civi! But mainly i use it for small datasets of sub 10,000

I have the opportunity to suggest its use in an organisation with a voter pool of 1 million+ I know this is an open ended question but does anyone out there have experience of using civi with large datasets. Is there anything BAD I shoudl know!Hopefully I won't get told off for asking the question!

Any thoughts welcome

Thanks

  • Hi Caroline. Jon G, Peter Petrik and a couple other folks did a great workshop at CiviCon in Denver last year about this - see youtube.com/… . Peter is here at civicrm.stackexchange.com/users/32/peter-petrik and Jon is here at civicrm.stackexchange.com/users/12/jon-g – Joe McLaughlin Sep 21 '16 at 13:42
  • Brilliant thanks I will watch the YouTube first 😀 – Caroline B Sep 21 '16 at 14:20
  • @CarolineB Consider talking to the folks who use it for very large installs - Seamus Lee of the Australian Greens, Eileen for Wikimedia Foundation, Brian Shaughnessy for the New York State Senate. 1M+ instances exist but I wouldn't expect it to just work out of the box! – Jon G - Megaphone Tech Sep 21 '16 at 14:32
  • @CarolineB feel free to get in touch my work email is seamus@nsw.greens.org.au. – Seamus Lee Sep 21 '16 at 20:35
2

I've never used Civi with a dataset that large, but there are some general scaling principles that apply to Civi as with any other system.

The most important one I learned is that work = size * complexity. Okay, that might be a bit of an exaggeration, and in some ways, size doesn't matter very much at all (e.g. you can simply scale your servers to some extent to handle the extra records), but in any real project, there will be exceptions and workarounds and custom bits, and those little bits will be magnified, sometimes exponentially, with the size of your dataset.

For example - suppose you're importing some data that isn't perfect. On a small contact dataset, you might load it into excel and do a search and replace so that USA is replaced by United States. And after doing an import you might have to clean up a few extra records. But that kind of process isn't feasible if you multiply your dataset by 1000.

In other words, scaling is not simply linear. There are some aspects that will scale almost magically, and overall Civi is well-designed to scale and there are certainly Civi installs out there with that kind of size. But with any specific install, especially one that has some kind of migration that includes custom data and idiosyncratic requirements, you're going to get various kinds of unexpected points in your scaling that will require new things to learn, new processes, and quite likely, custom code.

2

CiviCRM handles large datasets fine. There are 100+ sites with more than 250k contacts (https://stats.civicrm.org/?tab=sites).

You will need to ensure that the server used is powerful enough. To save some money, you can improve performance on a smaller server by turning off some things, in particular Administer > Customize Data and Screens > Search Preferences, Include Order By Clause.

  • what do you recommend as specs for a server running Civi with 250k + contacts? – Joe McLaughlin Sep 21 '16 at 15:57
  • In the Australian greens situation, I can;t remember all the deets off the top of my head, but primary we have 2 separate servers on different hardware one being the database and one being the web. Our database server runs off SSDs – Seamus Lee Sep 21 '16 at 20:35
  • The main variables for CiviCRM besides db size are number of concurrent site visitors and number of concurrent staff users who might be doing things like searching, running reports, and especially doing dupe elimination. CiviCRM is single threaded so a request can't really take advantage of multiple CPUs very effectively. MySQL now takes advantage of available multiple cores by default. I think we're running 650k contacts without that many activity records on a 24GB RAM VPS on a server using SSDs that doesn't restrict this VPS's use of 12 core/24 threads running at a relatively slow 2.4GHz. – Joe Murray Sep 27 '16 at 16:54
2

Skvare has worked with a number of CiviCRM implementations with data sets of significant size. As has been mentioned, do expect to have a bit of a "learning" curve - especially with the LAMP stack setup. It will require a well-tuned VPS.

Data handling will be another major challenge. First, small variations in addresses, prefix or suffix fields, etc. Standardization will be very important and will need to have pre-defined rules akin to a style guide for designers. That may include corrections of character sets, capitalization, and more.

Next, the built-in GUI import tools are not suitable for large and complex imports. We now have an internal library of methods that use Drupal's batch API with the ability to source the data, manipulate it according to established standards, and then insert it into appropriate fields within CiviCRM including the necessary conditions/dependencies/relationships.

Lastly, the number of contacts in a database is not a good indicator of the above requirements. 250K contact records with sparse related info will be a very different setup (both in ongoing server requirements and during data import) from 250K contact records with 30 years worth of contributions, memberships, event registrations, activities, relationships, groups/tags, etc.

As an option, we've also set up CiviCRM to interface with a 3rd party CRM from which the client wished to migrate off. Using the API we were able to dynamically pull data into CiviCRM upon specific user action (ie login, membership renewal, event registration), have the user validate/correct it, and then store it and mark it in the "old" CRM system as migrated. Depending on the level of user engagement that may be a longer timeline but the approach successfully crowd sources the "elbow grease" responsibility for cleaning up data.

Bottom line - CiviCRM is up to the task; the variable aspects for scaling it to accommodate large record sets is mostly a factor of organizational approach and methodology for data management.

  • Thanks Peter. It would be very interesting to learn more about the interface to the third party CRM using the API. What is the third party CRM? You used the Civi API or the third-party CRM API? – Joe McLaughlin Sep 21 '16 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.