0

A related question is Has anyone used Emojis in CiviMail subject/body? and the answer is to change the database to use utf8mb4 which is going to be standard in CiviCRM in upcoming versions, but using utf8 currently I'm seeing that depending on the database (e.g. mysql vs. mariadb) and version (e.g. mysql 5.5 vs 5.7) it can fail noisily or can fail silently and just truncates anything after the emoji. So this is partly a PSA in case others haven't noticed silent fails.

I was able to run this grep on the sites/default/files/civicrm/custom/CiviMail.processed folder (drupal - adjust as needed) to locate at least some candidates that have been truncated, but I doubt there's a way to then automate updating the activities in civi. If I find one will post back here.

grep -r "=F[0-7]=[0-9A-F][0-9A-F]=[0-9A-F][0-9A-F]=[0-9A-F][0-9A-F]" sites/default/files/civicrm/custom/CiviMail.processed

Improve this question

1 Answer 1

Reset to default
2

For those looking for a workaround for this issue, we created a tiny extension called Multibite which replaces unsupported multi-byte characters with the Unicode replacement character (or any other character, depending on configuration). We did not want to pollute core with this code as the goal should be to support utf8mb4, after which the extension becomes obsolete.

I should note that we aren't running this in production yet, so it hasn't seen much real-world testing.

Improve this answer
2
  • 2
    Just a note that you can change your database tables now to utf8mb4 and it doesn't seem to break anything in core and it works for new inbound emails going forward. But I never found a good way to retrieve the lost data and get it into civi.
    – Demerit
    Sep 13, 2019 at 13:53
  • @Demerit that's definitely a workable approach that could be the better option for some or perhaps even most sites! My consideration was to keep our installations easily reproducible (we'd have to ship the charset change as an extension) and to avoid potential bugs with Civi's use of temporary tables and similar (Civi tends to create them ad-hoc for some operations. They mostly seem to store IDs, but if there are any with varchar columns, and the charset is explicitly set by core, I'd expect to see issues when unsupported multi-byte characters are involved. (Applying PR #13633 would work too.)
    – pfigel
    Sep 14, 2019 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.