1

I'm organizing my data to prep for CiviCrm. One of the data banks we have is an online directory listing of hundreds of names, contact information and bios. NONE of this is organized into a spreadsheet.

The information is 1. Hand coded into HTML 2. Landing page has an alphabetical listing of hundreds of names that link to 3. A page in a separate digital directory with a html table with the contact information on one side and the bio on the left.

QUESTION: How do I automate the scrapping of this information and pushing into a .CSV without having to do this one by one?

Here is an example of the full directory.

Improve this question
2
  • 1
    This isn't really a CiviCRM-specific question, as the same problem would apply regardless of what system you were importing into. You might get better responses by asking on a more general Stack.
    – choster
    Aug 5, 2016 at 21:48
  • These are all very helpful. I also looked at import.io which has a pretty simple interface. I do think, however, this is going to end up in a manual mode since the original data was not organized with any streamlined system. Thanks for all your answers.
    – Danielle Siembieda
    Aug 15, 2016 at 22:00

3 Answers 3

Reset to default
2

I would recommend to use "Open Refine" for preparing your data!

Open Refine (formerly: Google Refine) is an open source tool, running on your own computer, that allows to perform data manipulation as you need for any CiviCRM migration project.

To get a quick idea how it works, you could take a look on these tutorial videos:

https://www.youtube.com/watch?v=yNccGtn3Wb0 (Introduction 1 of 3)

https://www.youtube.com/watch?v=45EnWK-fE9k (Introduction 2 of 3)

https://www.youtube.com/watch?v=5tsyz3ibYzk (Introduction 3 of 3)

The third video will show how to "add column by fetching URL's" - this should deliver what you need!

I have save hundreds of hours using OpenRefine during CiviCRM data migration projects. Of course, you will have a learning curve in using it. But this is one of the rare tools where you will save time already the first time you are using it.

Improve this answer
1

I've actually had good luck ("luck" being the operative word here) copying and pasting HTML tables in to a spreadsheet. You might need to paste it into a word processor document so it's in a table there first. I've done this with OpenOffice. YMMV with Microsoft Office.

Improve this answer
1
  • 1
    Ugh. I just looked at the directory you're trying to work with. You might be able to automate that to some extent, but I'm not sure it's going to be less work than copying and pasting the data.
    – Dan O'Brien
    Aug 4, 2016 at 15:48
1

Maybe a PHP script that uses DOM references to build an array of all the <li> items' names and link URLs, then cURLs each page and again uses DOM references to grab the contact info and add it to the relevant item in the array. There's probably a less procedural/more OOPy way to do it if that's your flavour.

In the linked pages, you may be able to distinguish between the email address, web address, and bio with an if/elseif type of thing where if the contents of the A HREF contain "mailto:", that's the email; if it contains "http://" it's the web URL; failing both those conditions it's the bio.

CSV is literally just comma-separated plain text where commas delineate columns and line-returns indicate rows, so if you can get your script to output "Name, email, web URL, bio (line return)" for each person in the database, you should be able to copy that text, paste it into a plain text document, and save it as CSV.

Good luck!

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.