I have an experimental search extension that provides many missing features, and includes your needs, I believe.
You can try it if you're up to that https://lab.civicrm.org/artfulrobot/keenies but it should be considered alpha, beta at a push! Please see the README for the searches it can do.
It includes a search on the manual added/removed from a group (smart or not), so using this on a smart group basically ignores the criteria.
I built it because if you're using a smart group for a mailing group, it's helpful to be able to search for people who have unsubscribed, but there could be many uses.
It's in early stages, and may be abandoned if/when the core CiviCRM new search project comes to fruition and provides this functionality (but I built this now because that could be a while).
EDIT: for convenience from the README as of 9 July 2020:
Implemented searches
Has Activity: choose activity types; relationship (with/target, assignee, created); date range; subject (exact or pattern);
location.
Contact details: contact type(s); created date range; source field.
Manually added/removed from group: select group(s) and a status: Added/Removed. e.g. find exceptions for smart groups.
In group: select group(s). This is like a normal group search, so works for smart and non-smart groups. You can put this criteria
in a 'not' wrapper to find people not in a group.
Has clicked a link in a mailing: date range for the click; select mailing(s) they must have clicked in.
Has opened a mailing: date range for the open; select mailing(s) they must have clicked in.
Was sent a mailing: date range and mailing(s) selector.
Has tag(s): select one or more tags.