Custom javascript file is not refreshed

Question

I embedded a JavaScript file inside a custom extension via the buildForm hook. The file gets loaded and does its job, however it is not reloaded reliably after an update. When I upload a new version of the file it takes quite some time until it is picked up by CiviCRM, sometimes hours. I have cleared caches and disabled asset caching, didn't help. I am on Drupal 9 and CiviCRM 5.67.0. Any help is appreciated.

Answer 1

... a JavaScript file ... When I upload a new version of the file it takes quite some time until it is picked up by CiviCRM, sometimes hours. I have cleared caches and disabled asset caching, didn't help. ... composer civicrm:publish doesn't seem to help with this... I am a total newbie in CiviCRM development and don't understand the caching going on here.

Yeah, there are many layers of caching in real-world systems. Even experienced folks will lose track of them.

Short thought: Check the references to your JS file. There should be a "resource caching code" (random suffix).

Long explanation: These symptoms sound like an issue in the browser-based caching and URL-construction. I don't have a reference to your codebase, so I'll make up an example.

Many HTTP servers enable caching for static files, such as jquery.min.js. It works like so:

<!-- The HTML page requires `jquery.min.js` -->
<script
  type="text/javascript" 
  src="/sites/all/modules/civicrm/bower_components/jquery/dist/jquery.min.js">
</script>

<!-- So the browser requests the file -->
GET /sites/all/modules/civicrm/bower_components/jquery/dist/jquery.min.js

<!-- Then the server sends the file along with caching instructions -->
Content-Type: application/javascript
Content-Length: 96830
Cache-Control: max-age=7200

!function(a,b){"object"==typeof...

The server tells the browser to cache jquery.min.js for 7200 seconds (2 hours) -- which is great for performance but frustrating for deployment. For a clean deployment, one needs a work-around.

In CiviCRM, the conventional work-around is to append a random suffix to the URL. For example, I went to this demo page and used "View Source" to find jquery.min.js. The actual URL has a suffix ?s3tkpz.

<!-- Here's how the HTML actually references the file -->
<script
  type="text/javascript" 
  src="https://dmaster.demo.civicrm.org/sites/all/modules/civicrm
                 /bower_components/jquery/dist/jquery.min.js?s3tkpz">
</script>

<!-- So the browser requests the file -->
GET /sites/all/modules/civicrm/bower_components/jquery/dist/jquery.min.js?s3tkpz

When CiviCRM has a major change on the server-side, it will switch out the suffix.

jquery.min.js?s3tkpz => jquery.min.js?neWc0d3

This doesn't actually clear the browser caches -- but as far as the browser is concerned, it looks like a different URL for a different file. The old caches become irrelevant. The browser will re-read the file.

This convention is extremely common for CiviCRM code. Several methods - like addScriptFile() and addStyleFile() from Resources API - will add the code by default. (It's so common that we take for granted...) But it's not absolute or universal. You might need to do one of these:

• Switch to Civi::resources()->addScriptFile() or Civi::resources()->addStyleFile()
• Use Civi::resources()->getUrl() and set the third option ($addCacheCode=TRUE).
• Lookup the cache-code (Civi::settings()->get('resCacheCode')) and manually append it to a URL.
• Tell your browser to force-reload (usu Ctrl-Shift-R or Cmd-Shift-R).

(I don't know if the above is actually your problem - but it does fit the symptoms: you're working with static JS files and clearing server-side caches and not seeing the expected content. Also, the behavior of HTTP caches can vary between different web-servers - e.g. your local dev server and production web server might have different caching policies.)

(Also note that the random suffix may look a bit different depending on the environment and code-flow. It might look like ?xxxxxx or ?r=xxxxxx, but it's the same concept either way.)

Answer 2

Following-up on Coleman's comment about composer civicrm:publish.... I don't know if that was confirmed to be part of the issue here. But here's a little braindump about when that should or shouldn't be necessary.

Assumptions: You have a typical Civi-D9/D10 layout (as per civicrm/civicrm-asset-plugin). You are deploying custom/contrib extensions.

Considerations: The location and mechanism of the download can vary, depending on your local-configuration/preferences/etc.

• Location: Where is the extension code stored? Typically, it's either the public folder (sites/default/files/civicrm/ext/) or the private folder (vendor/).
• Mechanism: Which mechanism performs the download? Typically it's either the built-in downloader (CRM_Extension_Downloader) or a manual downloader (git clone or wget && unzip) or a composer-based downloader

Expectations: There are two typical approaches.

1. For extensions in the public folder sites/default/files/civicrm/ext/...
   • It makes sense to me to use the built-in downloader and/or manual downloader.
   • The assets (JS/CSS/PNG/etc) are already public. These downloaders do not integrate with civicrm-asset-plugin. (It would be complex+unnecessary.)
   • You shouldn't have to call composer civicrm:publish... because it's fundamentally not needed.
2. For extensions in the private folder (vendor/XXX/YYY/)...
   • It makes sense to me to use composer as the downloader.
   • The assets (JS/CSS/PNG/etc) are private and need to be copied to a public location. This is why civicrm-asset-plugin integrates into composer - so that it can automatically publish these assets.
   • You shouldn't have to call composer civicrm:publish in typical usage... because civicrm-asset-plugin hooks into other composer commands. It should run automatically (during composer install or composer require or composer update).
3. For other arrangements... that's getting into oddball territory. I might see cases where you need to manually call composer civicrm:publish. But we really need to clarify the scenario. Here are some examples of oddball scenarios:
   • Manually editing files in vendor/
   • Configuring extensionsDir to point at a private folder
   • Using composer to download extensions into a public folder

(There may be mitigations to make the oddball scenarios better. For example, if you frequently do direct edits in the vendor/ tree, then you should configure CIVICRM_COMPOSER_ASSET=symdir or file-mode: symdir.)

Answer 3

The solution is to run the command composer civicrm:publish. See note in the upgrade guide (which really ought to be more prominent)!