Sherlock is surprised by the location they are approaching. He'd never considered the tree as a possible landmark for a secret passage.

"I don't do one-sided love affairs. I won't be the sort of idiot who holds a torch to someone without expectation."