[Shun gives a faint hum of assent in answer to Goku's belated recognition of him. Honestly, he's not exactly surprised it took him a moment, since they've only met once since Goku came back to this world, and Goku seems to be a somewhat absent-minded type.

He raises a brow slightly at just saying it's "kinda" his temple, though.] These places disappear when people leave this world, so you're not taking it away from anyone else. It's yours or it's not.

[Granted, Shun doesn't sound terrifically impressed with the concept either, but that seems to be how it is. The temples don't remain for any kind of original owner once a newcomer god disappears.]