The Unsubscribe link (in mailings) works fine if I am logged in as administrator, but I have "Access Denied" to the opt-out form if not.
(I have defined an Unsubscribe Group, with public visibility)
How can I fix that ?
There is a permission for it. Go to your Drupal -> People -> Permissions and give anonymous (and authenticated) permission to Unsubscribe:
Access CiviMail subscribe/unsubscribe pages