The only emails that your colleagues will see on your to-do list will be those that are already visible on timelines—i.e. emails that originated from work timelines or emails attached to public contacts.

If you add an email from a private contact, the email remains private and only visible to you.