If the permission "Team Member Saved Time" is disabled, team members will not be able to see other team member's time. However, they will see that time in detail and even drill down that data if they have access to Insights. Please have permission sets that hide other team member data cross into Insights to keep that data hidden.