How to migrate Raisers edge data to CiviCRM?

Question

We have large data in Raisers edge.We need to migrate Raisers edge data to CiviCRM.

How to migrate Raisers edge data to CiviCRM?

Answer 1

It is INCREDIBLY difficult to migrate data from Raiser's Edge. You will definitely want to use an ETL tool, as suggested in another answer.

I have made my documentation available here. I also have my Raiser's Edge to CiviCRM transforms available here. There's also a video available here.

Answer 2

I've recently done this myself with around 6800 contacts and 860 activities, when a new member organisation (who were previously using their own Raiser's Edge installation) joined and wanted to move their data to our platform. They provided various CSV files exported from Raiser's Edge.

I wrote some scripts, to load the CSV files and translate each line into the parameters to a chained API request. There was extra work in differentiating e-mail addresses from phone numbers, checking for invalid input, the different parts of postal addresses, date formatting, and things that were given as text in the data provided but are stored as IDs in Civi. We also needed to add several new titles/prefixes.

Answer 3

To migrate you will need some kind of ETL process (Extraction, Transformation, and Loading).

If you have experience with Pentaho, it has an adapter for civicrm and might be faster.

If you don't, it's probably faster to write a php-cli script that goes through the data exported and re-create it using CiviCRM API.

This being said, the harder in that kind of job it to find for each row for each entity in the source system where to put it in the destination one. And how to deal with "impedence mismatch". As an example, CiviCRM expects to have the first and last name separated, but some other system have the name in a single field. Is the first "word" mostly the first name or last name? What to do when you have 3 words, is this two first names and one last name or the other way around?

Erik's suggestion of reaching to someone that has already done that work and done the mistakes you always do the first time is a wise answer IMO, but nothing is impossible, just time consuming ;)

Answer 4

Just to add to the mix, we did a migration using the Migrate approach that Eileen has favoured. Agree about the difficulty of making RE data 'fit' in to a civicrm structure.

Answer 5

There are several not-so straight forward table mappings between Raiser's Edge (RE) and CiviCRM that require a lot of hand-holding and tweaking. Most of the basic contact information is easily transferred, but the constituent codes, prefixes, RE appeals, etc. don't map cleanly to a standard CiviCRM install.

For large Raiser's Edge databases that contain more than 200k records these types of migrations using the API v3 can take days to complete. We use a RAMdisk mapped MySQL instance to speed up the import and have migrated 500k contacts in under less than 6 hours (compared to 96 hours per migration iteration if using disk based MySQL.)

For these types of migrations from Raiser's Edge to CiviCRM it's important to be able to add new data mappings incrementally to the ETL and a flexible tool like Pentaho Data Integration is key.

Emphanos has an existing ETL process that can take a Raiser's Edge SQL Server .BAK file backup and creates a CiviCRM migrated instance that maps about 85% of all fields and performs a full migration with sub-sampling of about 2000 contact records in less than 1 hour.