No, it will not affect your response time. The response time is only calculated based on actual replies to emails. Here's how it works:
If you never reply to an email and just clear it: This action is not counted towards your response time.
If you leave an email in your triage for 10 days and then clear it without replying: This action is also not counted towards your response time.
If you clear an email and then reply to it 2 months later: The response time is considered to be 2 months.
Your response time metric is purely based on the time it takes for you to reply to an email. Non-reply actions such as clearing an email from your triage do not influence this metric.