2

I’m hoping to use CiviCRM to manage campaign contacts for voters across many counties and need to upload BOE data to do it. It’s about 210 MB in total, but I have it broken up into CSVs under 30 MB each. Unfortunately, I keep running into a 503 error when I try to use the import wizard with my smallest file (around 3 MB).

I’m using CiviCRM 5.17.4 on Joomla 3.9.12. I’ve increased Joomla’s session timeout and added a bit of code to civicrm_settings.php, following a tip on this guide: https://wiki.civicrm.org/confluence/display/CRMDOC/Importing+Data

I know uploads of this size are a tall order; is there anything that I’m missing or anything else that can be done? There are about a million registered voters here and I don’t think uploading them 4,000 at a time is feasible.

EDIT: If this goes unanswered and someone else runs into the same problem, you can whip up a BAT file to cut up CSVs for you.

https://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/

Improve this question

1 Answer 1

Reset to default
2

That's almost certainly a timeout issue - you can increase your timeout in php.ini, but I wouldn't use the UI at all for a million contacts. Consider using a command-line import, where there should be no timeouts.

There are several approaches you can take.

  • Use the command-line API CSV import. This is pretty solid but you need to import each file multiple times - once for contact data, once for postal addresses, once for phone, once for email. If you script it, it's not bad. I often use Hitachi Kettle to do this.

  • Use the Data Processor plugin and the REST API. There's a recent blog post detailing this approach (using Hitachi Kettle actually). This lets you import each file once. However, the REST API is going to do something like 10 records/second, command-line import is far faster but requires command-line access to the server.

  • Write your own PHP script so you don't need to break the data up.

Improve this answer
3
  • There's an option that I believe has a lower technical barrier to entry, but I can't find it. I believe there's an extension that will let you save an import you create in the UI, and run it on the command line. I can't find it but I'll ask the person I thought wrote it.
    – Jon G - Megaphone Tech
    Commented Sep 25, 2019 at 2:40
  • jon - any thoughts on that last comment by you?
    – petednz - fuzion ♦
    Commented May 19, 2021 at 21:51
  • Unfortunately not. Ideally we would modify Mathieu's advimport extension to use standard Civi import classes as one of its options, because it handles queuing. But that's several hours of work, and there's more work because advimport has its own limitation on size (related to how it stores the data). However, advimport's problems can all be solved in the extension itself; fixing standard import requires access to the server's configuration for Apache and PHP.
    – Jon G - Megaphone Tech
    Commented May 19, 2021 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.