The user will still show up in the list of Colleagues Week, however, they will now have a padlock next to their name and you will not have access to the number of items they have in each bucket.
This privacy setting can only be turned on by each individual. If you want all of your team to share their My Week, you can ask them all to turn this setting on for themselves.
There are certain items in my Colleague's Week I can't see
The numbers in Colleague's Week represent the number of items that are in their My Week. If an item is not on a public or shared timeline, only that person will have access to the item.