In CRM_SMS_Provider::processInbound, there are the lines
'source_contact_id' => $toContactID,
'target_contact_id' => $fromContactID,
which suggests that the source contact of the activity will be 'us', and the target will be the remote contact, who sent the message.
This is not the same as Inbound Email, which in _civicrm_api3_deprecated_activity_buildmailparams sets the source contact to the sender, and the targets as an array of recipients.
This is not going to be an issue for most users, as neither Clickatell nor Twilio extensions pass the 'To' number through to processInbound, and in this case the target will be set to the same as the source, attaching the activity to only the remote contact. It can arise if one uses a provider extension that does pass the 'To' number to processInbound, or Clickatell or Twilio have been appropriately modified, for example so messages received on multiple inbound lines can be told apart.
Should the source contact be the sender?