First a little tip, as you can see from Jankas answer she is looking for explanations by using the adjective or adverbial form. It helps to get finer nuances for sure. In general these kind of words have areas for which they might be used interchanged but sometimes the differences are really subtle. My answer is also not scientifically but simply spoken from experience as a native. I try to line out some of the differences as I know it personally:
Ungeheuer is a rather dangerous beast or bad person, it is also often used for people who behave very bad, being rude and not following any rules. In contradiction to a monster it does bad things by intention with a really bad mind. It could behave better if wanted. You can get a Ungeheuer from bad experience in life. If I speak from a Ungeheuer it make me or others a scary feeling. An Ungeheuer must not be big but it can.
Monster is rather the visual ugly or bad. A monster you can see in a movie. A monster implies it is bad shaped, looking extraordinarily, smells bad. A monster can also be very big destroying things. A monster is usually born like this. About a monster you can laugh sometimes because it is funny to see, this you can not say from an Ungeheuer. Also a person can be called a monster when it is fat or extremely bad behaving.
Conclusion: I think the visual aspect is very important. Also a Ungeheur is sometimes bad by intelligence/intention.
Lookup "ungeheuerlich" and "monströs" to get further differences. Also mind the word "Geist", it also belongs to this group.