If you don't want or need a CMS, Drupal works a little better than WordPress for this. WordPress wants to make it as easy as possible to run a blog, so it makes decisions for you that can work against having a minimalist framework.
On the other hand, you can easily disable a ton of Drupal's functionality and even make CiviCRM the homepage. Direct URLs to CiviCRM contacts aren't quite as pretty as you like, but they're much more concise than WordPress or Joomla: foo.org/civicrm/contact/view?cid=1234&reset=1
brings up contact 1234.
Working with Drupal also gives you a lot more flexibility in your page layout: you can use any theme, and there are a number of blocks (widgets in WP parlance), unavailable in WordPress, for CiviCRM admins that let you view the most recent contacts, quickly add new activities/contacts/etc., and more.
A friend mocks Drupal by saying that it's like going to the car dealer and being asked, "Would you like a steering wheel with that? How about a brake pedal?" WordPress makes it easy to build a basic website because it makes a lot of assumptions. However, if you just want a stripped-down framework to render CiviCRM and let people log in, Drupal's lack of assumptions will make it a lot easier.