3

I'm trying to understand what happens with an import.

A file gets uploaded in a POST operation, then there's GET requests polling(?) of civicrm/ajax/status?id=somelongid

Which request does the work? Is is the POST?

Is it that the POST is a really long running script (server timeouts permitting) but that it's triggered in the background? (EDIT: yes, that's true.) Where does the somelongid come from for the GET polling? Or do the GET requests trigger processing the next batch? Does the process continue without the status requests (e.g. if you close the browser).

The reason for wanting to understand is primarily that I have a timeout issue and I don't want to increase the global timeout for all requests, so I change the timeout for certain paths in nginx config. But also I'd just like to know and couldn't find it in the docs.

  • Would be very useful to understand this. I get time out messages from imports but they seem to have processed fine. – Mick Kahn Jul 23 at 10:53
1

It depends on the job.

Import jobs (Contacts, Contributions...) work by having one long-running POST request, and then your browser polls for updates on progress.

This means your browser/server/interenet connection need to be able to cope with the time that POST request requires. Whether a job completes after apparently timing out depends on how PHP is set up on your site.

For example:

  • Using Apache + mod_php: I believe that if the script times out on the POST request, processing stops at the point it got to.

  • Using nginx + PHP FPM: nginx will give up and issue a Gateway Timeout according to its configuration, but the PHP process may still carry on regardless, until completion.

So for import jobs, it's best to configure your server's timeouts to exceed the time taken for the biggest import you're likely to need to do, and/or break your imports down into smaller chunks that can be handled within those timeouts

Other jobs (dedupe merge, and applying database upgrades are examples) may use CiviCRM's queue system. This has two ways to get processed.

  1. A server-side CLI script may chug through the entire queue without any timeouts.

  2. A browser-side process with a progress bar works differently and requires the browser to process each step of the queue. If a single step ("task") takes longer than the configured timeouts, you're in trouble. If you close your browser, the next step won't run and the last step may (or may not) fail. If you close your browser and re-open it, you may start running the tasks more than once, in parallel (so don't do this!).

See my newly added Queue Reference in the developer docs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.