We have about 1 million contacts now (grown from 50k to 1 million over 3 years) and have millions of activities. We are now using the full CiviMail to replace external email marketing systems which will greatly increase activities. We use complete contribution management and event management, we are now doing much more volunteer management which will also increase activities.
We moved from shared server, to virtual private server, to outsourcing hosting to a Civi service provider because of performance issues and lack of server expertise. If you have the server (or cloud) knowledge you could do it yourself.
We are just beginning to incorporate some complex smart groups and will see if they impact performance.
The organizational structure and ACLs are more complicated and I would recommend getting experienced advice.