I manage and work on a small team of 4 people. Myself and one other are remote, I am at a 13 hour time difference from all of them(!) The hardest part for me is knowing what's going on during their normal workday. Something can come up that de-rails their whole day and I'm not there to be their buffer or to help prioritize. So I guess the hardest part for me, isn't my employees, it's the other employees in the company that distract them or take up their time.
We are keeping person in the office at least half a year (6 payrolls). During this time they are trained and you can understand if she/he/it fit your company. After that period we can give remote login only.
Synching up is a challenge. I am remotely managing a team of mostly young, inexperienced but highly motivated individuals at a very early stage startup. I have to spend a lot of time on calls and in chats to understand their individual contexts, coordinate work, provide guidance where necessary. I also sometimes am late responding to their queries over chat when busy with technical work, which adds to their problems at times. I also find that i have to put in a greater effort to win and keep their trust and that even small mistakes at my end create a greater trust deficit which needs to be breached somehow. There are times when i feel that i am not doing justice to the team and that it would be easier for them to be managed by someone who was colocated with them.
I found out that developing trust with other people is much harder remotely