What kind of a question is that? You've been here at her temple. You know I don't even live here.
[He's assuming an underlying question about the nature of their relationship and he's preemptively getting defensive about it.]
Other options:
no subject
What kind of a question is that? You've been here at her temple. You know I don't even live here.
[He's assuming an underlying question about the nature of their relationship and he's preemptively getting defensive about it.]