Upgrade froze at 4.7.8, Can't find valid value for qrid

Question

My upgrade has been running for about 8 hours, but it stalled after completing 4.7.7, from 4.3.8 at. I checked top to see if Drupal or CiviCRM were running, and there was no indication of the process. I decided to refresh my browser and my cookie must have expired, because Drupal asked me to log in again. Then I pulled up the queue runner in the browser (https://dev.example.com/civicrm/upgrade/queue/runner) it spit out the error:

Sorry but we are not able to provide this at the moment. Could not find valid value for qrid

The log shows that it could not find valid value for qrid. Below is the log from /sites/files/civicrm/ConfigAndLog, starting after completing upgrade to 4.7.6:

Dec 21 12:18:17  [info] Running task: Begin Upgrade to 4.7.7
Dec 21 12:18:19  [info] Running task: Upgrade DB to 4.7.7
Dec 21 12:18:21  [info] Running task: Upgrade DB to 4.7.7: SQL
Dec 21 12:18:29  [info] Running task: Finish Upgrade DB to 4.7.7
Dec 21 12:20:15  [info] Running task: Begin Upgrade to 4.7.8
Dec 21 12:20:17  [info] Running task: Upgrade DB to 4.7.8
Dec 21 12:20:23  [info] Running task: Upgrade DB to 4.7.8: SQL
Dec 21 12:20:56  [info] Running task: Upgrade mailing foreign key constraints
Dec 21 13:26:27  [info] $Fatal Error Details = Array

(
    [message] => Could not find valid value for qrid
    [code] => 
)

Dec 21 13:26:27  [info] $backTrace = #0 /var/www/html/drupal/docroot/sites/all/modules/civicrm/CRM/Core/Error.php(373): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /var/www/html/drupal/docroot/sites/all/modules/civicrm/CRM/Utils/Request.php(120): CRM_Core_Error::fatal("Could not find valid value for qrid")
#2 /var/www/html/drupal/docroot/sites/all/modules/civicrm/CRM/Queue/Page/Runner.php(52): CRM_Utils_Request::retrieve("qrid", "String", Object(CRM_Queue_Page_Runner), TRUE)
#3 /var/www/html/drupal/docroot/sites/all/modules/civicrm/CRM/Core/Invoke.php(312): CRM_Queue_Page_Runner->run((Array:4), NULL)
#4 /var/www/html/drupal/docroot/sites/all/modules/civicrm/CRM/Core/Invoke.php(86): CRM_Core_Invoke::runItem((Array:13))
#5 /var/www/html/drupal/docroot/sites/all/modules/civicrm/CRM/Core/Invoke.php(54): CRM_Core_Invoke::_invoke((Array:4))
#6 /var/www/html/drupal/docroot/sites/all/modules/civicrm/drupal/civicrm.module(448): CRM_Core_Invoke::invoke((Array:4))
#7 [internal function](): civicrm_invoke("upgrade", "queue", "runner")
#8 /var/www/html/drupal/docroot/includes/menu.inc(527): call_user_func_array("civicrm_invoke", (Array:3))
#9 /var/www/html/drupal/docroot/index.php(21): menu_execute_active_handler()
#10 {main}

I suspect that the qrid is the queue runner ID and that the queue was lost when Drupal asked me to log in again. The log doesn't show that there was any other issues before the qrid problem. I'm not certain if something was wrong before I was forced to log in again or if that is what did it it.

Furthermore, the file permissions for the civicrm directory in sites/all/modules was somehow changed to an owner I am not familiar with on our server (test2?):

    drwxr-xr-x 19 test2 csf      4096 Nov  2 19:32 civicrm
drwxr-sr-- 80 fclub www-data 4096 Oct 16  2015 contrib
drwxr-sr-- 15 fclub www-data 4096 Oct 16  2015 custom
-rw-r--r--  1 fclub www-data  952 Aug  5  2015 README.txt
drwxr-sr-x  2 fclub www-data 4096 Nov  3 22:40 timeago

I tried loading this URL to see if that would help:

civicrm/upgrade/queue/runner?reset=1&qrid=‌​CRM_Upgrade

but then received the error:

Queue runner must be configured before execution.

I should have used Drush to run the upgrade so I wouldn't have to deal with Browser issues.

Question 1: Is there anyway to salvage the progress and continue from 4.7.7, repair the database or reinstate the queue runner?

Question 2: How can I figure out what went wrong so that I can try running the upgrade again without this happening?

Answer 1

The error message comes from refreshing the browser rather than the upgrade process. The last entry before the error is:

Dec 21 12:20:56  [info] Running task: Upgrade mailing foreign key constraints

You can find the detail in CRM/Upgrade/Incremental/php/FourSeven.php Adding foreign key constraints creates an index which can take a while on a big table. How many rows (count(*)) do your civicrm_mailing_event_queue and civicrm_mailing_recipients tables have?

Once the upgrade has started, I don't know how dependent it is on the browser, so it would be worth checking the log later to see if you have any further messages from the upgrade.

If you need to start again, then I'd try this:

1. Restore your database and files back to your starting point at 4.3.8 (you did take full backups didn't you??)
2. Upgrade to 4.7.7
3. Check all is working as expected
4. Take new backups
5. Try upgrading to 4.7.8

If your mailing tables are large you'll need to be patient. If it fails, you can easily go back to 4.7.7 and then work out what to do next.