As referenced here and other places in the forum:
IPN's from paypal are apparently not passing muster with CiviCRM. The errors in the log are somewhere along the lines of "Could not find an entry for contactID in GET". This log is generated by testing using curl, like so:
curl -k -X POST -d "[paste in IPN params from Paypal IPN log]" https://oursite.org/sites/all/modules/civicrm/extern/ipn.php?reset=1&contactID=[contact_id]&contributionID=[contribution_id]&module=contribute
The words in brackets are my substitutions; clearly there is an integer contactID in the GET request; what's going on here?
UPDATE: More perplexing is that the civicrm_system_log table varies depnding on how the IPN is sent. From Paypal, it shows all the info from the IPN as it's logged in Paypal - with the GET parameters first, followed by the POST parameters, all within the same JSON-looking structure, in the log table's 'context' field. If I simulate a re-send of the same IPN using curl as described above, the parameters from the GET part of the request are missing from the context field. I'm not sure if they're supposed to show up in the log or not, as it seems the POST variables are what show up in the log, so it's a little tricky to know how to debug this.
In any event, the actual IPN's that came through from Paypal, and which Paypal says were met with 500 errors, are logged in the database, and there are no corresponding entries in Civicrm.[hash].log from those times indicating any errors, so presumably the IPN should have processed correctly. This is rather complicated and confusing.. could someone please help?
Update: Including the GET data in the curl request with -X GET -d "reset=1&contactID=[contact_id]&contributionID=[contribution_id]&module=contribute" results in a "302 found" response, but no more information than that, and I cannot find any evidence in the database log of the request, and there is no mention at all in Civicrm.[..].log.
Addendum: The PayPal IPN simulator reports that it cannot successfully handshake with either the dev or live sites. That is strange considering that IPN logs are appearing in the live site's civicrm_system_log table..
UPDATE: Apparently here's what was happening. I have a Civi extension where I call a function defined in a Drupal module, and the 500 error was about calling an undefined function. I fixed that in the extension code, but the function in question called the Drupal watchdog(..) function, which is also undefined in the context of being called from ipn.php. So here's a better question: What's the solution to this problem? Do I edit ipn.php and bootstrap Drupal before doing anything else? It seems odd that, just for IPN's, something would break because it is not running under the same assumptions that the rest of Civi runs under.. of course, I could also switch from Drupal watchdog in my extension to whatever the Civi equivalent is.. any suggestions?