I am trying to create a smart group for contacts that have activity added by specific person.

Advanced search allows me to search either (1) for contacts based on some activity; (2) for activities based on who has added the activity.

But unfortunately not for contacts based on who has added the activity.