I agree with c.p.'s solution ("Karl arrives at 22:30 and he is the fourth"), but the original sentence - without context - is too short for being unambiguous. Placed in different contexts, it will shift meaning.
The sentence lacks modifiers to make it unambiguous, and so the recipient will automatically add such modifiers depending on what information ever he has (e.g. situational context).
Sou you can resolve the sentence like:
Wir sind erst zu dritt, [und] ab ca 22.30 [dann] mit Karl.
which is c.p.'s solution. Based on everyday living experience it is the most probable situation. Also it is relatively simple.
But you could also resolve it like:
Wir sind erst zu dritt, [nämlich Paul und ich, und] ab ca 22.30 [dann auch] mit Karl. Und ganz am Ende, gegen Mitternacht, kommen noch Lisa und Lena dazu.
meaning that first they are two, then three, then five. Admittedly it is a rather complicated thought (especially as the narrator first says they are three of them, then makes a step back in time and mentions that they start with being two of them until the third person joins); but it is a legitimate interpretation of the bare-bone sentence.
The idiots are those who call you an idiot. (The only question is: how many are they, and at what point of time.)