2

I’m hoping to use CiviCRM to manage campaign contacts for voters across many counties and need to upload BOE data to do it. It’s about 210 MB in total, but I have it broken up into CSVs under 30 MB each. Unfortunately, I keep running into a 503 error when I try to use the import wizard with my smallest file (around 3 MB).

I’m using CiviCRM 5.17.4 on Joomla 3.9.12. I’ve increased Joomla’s session timeout and added a bit of code to civicrm_settings.php, following a tip on this guide: https://wiki.civicrm.org/confluence/display/CRMDOC/Importing+Data

I know uploads of this size are a tall order; is there anything that I’m missing or anything else that can be done? There are about a million registered voters here and I don’t think uploading them 4,000 at a time is feasible.

EDIT: If this goes unanswered and someone else runs into the same problem, you can whip up a BAT file to cut up CSVs for you.

https://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/

2

That's almost certainly a timeout issue - you can increase your timeout in php.ini, but I wouldn't use the UI at all for a million contacts. Consider using a command-line import, where there should be no timeouts.

There are several approaches you can take.

  • Use the command-line API CSV import. This is pretty solid but you need to import each file multiple times - once for contact data, once for postal addresses, once for phone, once for email. If you script it, it's not bad. I often use Hitachi Kettle to do this.

  • Use the Data Processor plugin and the REST API. There's a recent blog post detailing this approach (using Hitachi Kettle actually). This lets you import each file once. However, the REST API is going to do something like 10 records/second, command-line import is far faster but requires command-line access to the server.

  • Write your own PHP script so you don't need to break the data up.

  • There's an option that I believe has a lower technical barrier to entry, but I can't find it. I believe there's an extension that will let you save an import you create in the UI, and run it on the command line. I can't find it but I'll ask the person I thought wrote it. – Jon G - Megaphone Tech Sep 25 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.