7

The CiviMail documentation and settings pages indicate that it is possible to have multiple mailer delivery jobs executing simultaneously.

See Settings - CiviMail (/civicrm/admin/mail?reset=1) :

  • Mailer Job Size: If you want to utilize multi-threading enter the size you want your sub jobs to be split into. Recommended values are between 1,000 and 10,000. Use a lower value if your server has multiple cron jobs running simultaneously, but do not use values smaller than 1,000. Enter "0" to disable multi-threading and process mail as one single job - batch limits still apply.

  • Mailer CRON job limit: The maximum number of mailer delivery jobs executing simultaneously (0 = allow as many processes to execute as started by cron)

I have not been able to get this to work in practice in CiviCRM 4.4.14 . I have tried various settings, e.g.

batch limit = 2500
throttle time = 5
job size = 2500
cron job limit = 2
civimail_server_wide_lock = off

Civi cron is running every minute. Mailings scheduler is the only scheduled job that runs on every cron run (others are hourly or daily). The mailing is correctly split into sub-jobs of 2500 mails, however these get processed consecutively. Each batch takes about 6 minutes to run. Every time a new instance of the scheduled job tries to start while an existing instance is running, we get an error like this:

Finished execution of Mailings scheduler with result:
Failure, Error message: lock aquisition for mydb.1.civimail.job.12415
attempted when mydb.1.civimail.job.12414is not released

Hence only one instance runs at a time.

The error message above is generated by CRM_Core_Lock::hackyHandleBrokenCode(), which was added for CRM-12856 and is called from CRM_Core_Lock::__construct(). It prevents multiple locks being created with names containing "civimail.job.". If this is attempted, hackyHandleBrokenCode() throws an exception and so the scheduled job exits. This appears to be what is preventing multiple mailer delivery jobs executing simultaneously.

Looking at CRM_Mailing_BAO_Mailing::processQueue(), the code for $config->mailerJobsMax > 1 seems to work on the assumption that a CRM_Core_Lock can be created independently for each of the mailerJobsMax cron jobs. But since CRM-12856, this is disallowed. I believe this may not have worked correctly in any case, given that each MySQL connection can only acquire one lock: acquiring a second, even if it has a different name, releases the first: https://dev.mysql.com/doc/refman/5.5/en/miscellaneous-functions.html#function_get-lock.

However some users have reported this feature to be working correctly - can anyone confirm that mail jobs are indeed sending in parallel in their post-CRM-12856 Civi?

2 Answers 2

2

Just to follow up on this, this was a bug that will be fixed for the 4.6.4 release. More details can be found at issues.civicrm.org/jira/browse/CRM-16387

1
  • Yes, I noted this in my comment of 27 Apr on Joe's answer. But useful to have this fact in an answer rather than a comment, so I'll accept this answer. I was involved in testing the changes for CRM-16387 and they allowed concurrent sending to occur. The fix is going into 4.6.5, not 4.6.4 .
    – davejenx
    Jul 3, 2015 at 9:32
1

It seems you have found a bug.

IIRC the design is intended to queue up all the necessary jobs initially, then have runner task threads dequeue a job and execute it.

The function name you cite looks like someone knew they were only doing a poor job, likely intended to address a bigger problem. Try using blame in github to find who added that code and ask them for advice on how to go about fixing your problem without causing other error(s) to recur. Based on the function name, I bet it was Eileen.

In response to your comment, please create an issue including forum link and link to blame along with description.

4
  • Yes, it was Eileen: issues.civicrm.org/jira/browse/… Comments show that she knew it was a temporary fix: "this is all a big hack... should not be seen as a fix. Not sure correct fix... maybe locks should be used more selectively? Or else we need to handle is some cool way that Tim is yet to write :-)" But others report concurrent mail jobs working, e.g. forum.civicrm.org/index.php/… so trying to confirm whether it's a genuine bug.
    – davejenx
    Apr 10, 2015 at 10:19
  • did you find out if it's a genuine bug?
    – Xavier
    Apr 20, 2015 at 10:32
  • I've discussed with the user who reported it to be working correctly at forum.civicrm.org/index.php/… - on examining job logs, he's getting the problem too. He is filing an issue. I'll post details when I have them.
    – davejenx
    Apr 21, 2015 at 10:25
  • An issue has now been filed in the Civi issue tracker: issues.civicrm.org/jira/browse/CRM-16387
    – davejenx
    Apr 27, 2015 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.