"I wondered if I was letting people too far in," the "Thor" actor explained.