What could be causing slow mailing from CiviCRM (via SMTP)

Question

We're currently investigating why our mailings are sending slowly from CiviCRM. We use an external SMTP server.

Our mailing settings:

• Mailer batch limit: 30000
• Mailer throttle time: 2100
• Mailer job size: 0
• Cron job limit: 1

However, when we look at the delivery table, the times individual emails were delivered are a bit strange:

• 10 emails delivered all on the same second
• A delay of 2-3 seconds
• Another 10 emails delivered on the same second
• A delay of 2-3 seconds
• Rinse, repeat

Has anyone run into a similar issue?

Update: After some testing, this only happens for SMTP (ie, not mail()), and it is happening with two separate SMTP providers. Investigations continue...

Update: Emails also send slowly from a standalone script which uses the PHPMailer library on the same server. That actually sends slower, at 2/3 emails per second but without a delay inbetween. The next step would be for us to try concurrent sending jobs (once this is fixed in 4.6.10).

Answer 1

There are a couple of points to this.

1 - SMTP is chatty, it is a fairly slow protocol. To combat this, run several mailing jobs in parallel. This can be achieved by calling the process_mailing cron job multiple more frequently. Ensure that the 'cron job limit' is equal to the maximum number mailing jobs you would want to be active at any one time. This is useful if you have a sending limit on your SMTP server of x per second.

2 - The issue with 10 emails sent 'on the same second' and then a delay: MailingJob.php processes 10 emails at once, and then writes the transactions to the database all at once. So it looks like the emails were all sent on the same second. The '10' is defined in BULK_MAIL_INSERT_COUNT in CRM_Mailing_Config. Ideally the actual time of sending would be recorded in the database, rather than the time of writing.

To speed up mailings we now call the process_mailing job independently every 5 minutes, and limit the number of emails that are sent on a cron run. This means we have several cron jobs running sending emails at once for large mailings, which greatly improves our send rate.

Answer 2

when civimail uses smtp to deliver, it doesn't create a new connection to the smtp server for every contact, but pipeline them. this might explain the bursts pattern?