It's the "sein" at the end of the sentence that enforces the "bei".

You could write "Um 11:00 muss ich dann zu einem Kollegen gehen." or "Um 11:00 muss ich dann mit einem Kollegen arbeiten."

"bei jemandem sein" means to be at a person's place. "Ich werde bei meinem Freund sein" means that I will be at my friend's place (not just be with him).

So the sentence answers the question: "Where do you have to be at 11:00?" - "I'll have to be at my collegue's place."

It's the "sein" at the end of the sentence that enforces the "bei".

You could write "Um 11:00 muss ich dann zu einem Kollegen gehen." or "Um 11:00 muss ich dann mit einem Kollegen arbeiten."

"bei jemandem sein" means to be at someone's place. "Ich werde bei meinem Freund sein" means that I will be at my friend's place (not just be with him).

So the sentence answers the question: "Where do you have to be at 11:00?" - "I'll have to be at my collegue's place."