Some thoughts
Anonymizing data is hard work.
Eg: Would it be OK if your developer could de-anonymize your contacts by identifying from geolocated data? What about identifying by "metadata", if suburb data is left in? Does your developer need a DB with activities from inbound email left in? Has the situation improved in the decade since WIRED Why 'Anonymous' data sometimes isn't, or gotten worse? (It got worse!)
As you anticipated, generating real-looking fake data is much slower than garbage data, and that's one tradeoff. The other is that a CRM has data in places you might not think to look, so you are safer treating the data as "mostly anonymized" even when you're done. (See that Wired article!)
One approach, which may also speed up your testing, is to remove data unless you know it's needed (and only anonymize what you didn't already remove).
- Not testing mailing reports? Truncate those huge mailing tables.
- Not testing custom data? Great, that stuff is hard to anonymize! Drop it!
- Not testing performance? Cull your test DB down to 500 contacts before anonymizing.
What you choose depends on how you need to use it, and what's OK for your testing. As you say, for debugging developers need a "near exact" copy of the live data ... but the more a test data set matches your site's profile, the less anonymous it will be.
Some starting points
- anonymize extension (anonymize) aims to anonymize a DB. Offers two strategies: Faker (slow, generates real-looking names like "King, Lesley") and SQL (fast, generates names like "Tdfv, Fnedftd"). With recent updates, this is probably your best bet - but you'll be an early user, so please direct feedback to the issue queue in CiviCRM's Gitlab!
- gibberish extension (gibberish) creates SQL to replace the contents of your database with gibberish
- Michael McAndrew's POP library is the most featureful IMO, its goal seems to be to populate a site with fake data rather than anonymize an existing DB. See the blog post Populate your CiviCRM site with fake data. A similar example using nodejs is civi-charlatan.
- anonymiser (systopia/de.systopia.anonymiser) is another CiviCRM anonymiser extension under recent development as of June 2018. I haven't checked this out (and Sean and I didn't find it when we started work on anonymizer).
- redaction_tool (johnff/redaction_tool) is another tool to redact contact data (WIP)
- civicrm-fake-data (on github) is a ruby script to replace existing data with fake data.
- civicrmanonymisation is an SQL approach to anonymising a DB.
- encryptDB.php was in CiviCRM core, but it was a poor starting point; MD5 is not encryption, so what changes it does make are reversible. It also targeted a limited number of fields, so your DB would still contain identifying data even if not deanonymised ...
- civicrm_obfuscate_db.php (based on
encryptDB.php above) is another script I wrote several years back. It generates a set of SQL commands to obfuscate your DB. It's SQL-based so relatively fast, generates incomprehensible names ("bggerns srrssbb"), and like all the above options is not a complete solution.
- Another option would be to construct a test environment with test only data that simulates all situations you need to test, and keep expanding it as you identify new bugs. Let's be realistic, few CiviCRM sites have budget for that 😜
To recap: none of the above are complete working solutions for what you want, and the solution you need depends on your context. But please do contribute your experiences to the community - most of the above have a Github issue queue you could submit improvements / feedback to!
One particular sticking point for several of the above seems to be CiviCRM's "Database Logging" tables, which by default use a format which complicates sanitisation. If you're testing or using any of the above, you should definitely review whether those tables are being de-identified completely in the process.
