Is there a safe way to remove/delete/deactivate CiviCRM contacts who have attended events, without affecting the events attendance data?
First of all: there is NO way in which you can delete contacts without affecting events attendance data.
I would try in a test environment to trash CiviCRM contacts that have attended events and check if you still see what you want to see. (And also it will help if you explain what you mean by attendance data)
It might depend on how you need your 'attendance data' stored. Eg after each event you could enter a 'count' of your participants in to one of more custom fields on the Event, eg one for 'attended', one for no show, etc.
That way the 'data' is stored on the event, you can always see how many folk attended, but you can delete the Contacts. Downside of course is that you would then also need some home made reporting on this but that doesn't seem tricky.