The public facing membership signup (contribution) form (wordpress) will only create individual contacts. We need to control the type of contact created based on the membership type the user selects. If they select a 'Student' membership then we need an individual contact. If they choose a 'household' membership then we want a household contact to be created. Shouldn't be too hard to write, but I don't know where the code is or how the internal api works. What would be the most relevant php files to work with and what api commands are involved in contact creation?
Have you experimented with the api explorer?
As far as the API is concerned on the wp demo site you will find a great explorer at http://wpmaster.demo.civicrm.org/wp-admin/admin.php?page=CiviCRM&q=civicrm%2Fapi%2Fexplorer#explorer. The URL will be similar for your own site.
Remember, the eplorer is "live" so you should experiment on the demo site or on your test site, not your live site.
As far as the rest is concerned, I am not a delevoper so I can't comment on whether what you want to do might be achievable or not, but it sounds rather like you intend to hack core and that is not recommended. An extension is the preferred approach.
You're mentioning using file overrides, but that's not maintainable in the long run - and is more complicated than it needs to be IMO.
I think a preferable approach is to create an extension that calls hook_civicrm_post on the Membership entity. Combine that with the API to edit the contact after the membership is created.
This will allow you to set a contact subtype. Changing the contact type is a little trickier because the fields that will be available aren't necessarily the fields you'd collect for a household. But that's OK - because for a household membership, you probably want to keep an individual, then create a household separately, create a relationship between them, then move the membership on to the household. You can do all this via the API.
If you DO want to go the override route, this page shows how to override a single contribution page.