What if I have one work item I want private for a public contact or an internal job?

In this case, you can create a second contact (without an email address) and set the appropriate level of privacy for that contact. 

Any work items associated with this private or hidden contact will be private to you and anyone on the client team.

You can grant access to others by adding them to the work team.

Further reading

Granting access to a private or hidden contact
Granting access to a private or hidden work item
Removing access to contacts and work items