The main problem most likely is that your brain cannot easily create the framework for new grammars, so when you pick up a new language you first understand it via "software-decoding", i.e. thinking about structure and making sense of it as opposed to "hardware decoding", which would be the intuitive understanding of how the words relate to each other that has been formed when first learning the language as a mother tongue. The latter is obviously faster and lets you keep up with the speed of spoken language.
There are additional problems that also slow things down. One is the fact that one can create really monstrous sentences in German. Another is how you encode and decode meaning.
IMO it is always more difficult to understand a language in spoken form compared to reading in it. It just takes a bit of time to get used to the sound and the flow of it.
If you can confirm this passage from embert's answer you most likely approach learning the language visually and textually. This is problematic as it adds unnecessary steps to all transmissions
Meaning --> Sound --Air--> Sound --> [Writing] --> Meaning
Meaning --> [Writing] --> Sound --Air--> Sound --> Meaning
So listen more, a lot more. This helps in learning to decode words faster and it trains your feeling for the natural structure of sentences.
There are quite a few Japanese words which i never read or wrote but know the meaning of because i learned them from audio only; picking out those words in conversations is easier than those which i learned "from paper".