Override CiviCRM contact import timeout using PHP-fpm and Nginx

Question

I'm attempting to override the fpm time out from nginx just for the import. We are importing very large files, and it might take up to a couple hours to complete, so I need to set:

fastcgi_send_timeout 0;
fastcgi_read_timeout 0;

I'm having difficulties understanding what exactly happens during the import in which location I should add this override to.

There seems to be a lot happening during the import, ajax, POST requests etc, and I have tried adding this override to just certain paths, all of which didn't fix the problem.

Our installation is at /vendor/civicrm/civicrm-core, and adding the following location block didn't work.

location ^~ /vendor/civicrm/civicrm-core/CRM/Contact/Import {
        try_files $uri /index.php?$query_string;
        location ~ \.php$ {
            include snippets/fastcgi-common.conf;
            fastcgi_send_timeout 0;
            fastcgi_read_timeout 0;
            fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        }
}

I have tried /civicrm/import/contact, but it still times out with:

upstream timed out (110: Connection timed out) while reading response header from upstream, client: ::ffff:REDACTED, server: REDACTED.com, request: "POST /civicrm/import/contact HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.2-fpm.sock", host: "REDACTED.com", referrer: "https://REDACTED.com/civicrm/import/contact"

---

Trying to use an invalid fastcgi socket seems to imply it's not being matched since it doesn't error out:

location ^~ /civicrm/import/contact {
    ## Standard site protection
    try_files $uri /index.php?$query_string;
    location ~ \.php$ {
        include snippets/fastcgi-common.conf;
        fastcgi_send_timeout 0;
        fastcgi_read_timeout 0;
        fastcgi_pass unix:/tmp/for_sure_will_500_and_doesnt_exist.sock;
    }
}

location ~ \.php$ {
    include snippets/fastcgi-common.conf;
    fastcgi_pass unix:/run/php/php7.2-fpm.sock;
}

The debug log also shows its matching ~ \.php$, why would it not match the import path first?

Answer 1

The import do generate a lot of requests indeed, the one you should care about is the POST one (that does the import). The others are mostly cosmetic to update the counter of how many contacts have been processed so far.

What matters for your rewrite isn't where the code is, but what's the url of that post. It depends of your install (check your log or the console on your browser), but it's likely not including the full path (vendor/civicrm/civicrm-core), eg something like:

location ^~ /civicrm/import/contact { ...

one trick I use to see if the location nginx rule is the correct one is to put an invalid fastcgi_pass, 50x errors are easier to see than waiting and trying to guess if the timeout worked or not ;)

This being said, if you have a lot of contacts to import, it's probably safer/easier to write a php script that you run from the CLI and that uses civicrm api to create the contacts (you don't have to deal with timeout, you have more memory, easier to do exactly the tests you want and create the data you need the way you want it.... and that's way easier to re-run if something breaks because some input is invalid)

X+