In my understanding (as a German native speaker) there is a just so slight difference between Fehlübersetzung and Übersetzungsfehler:
Übersetzungsfehler tends to mean a concrete, specific mistake in translation. Also, it refers more to clear mistakes (as opposed to disputable translations).
Fehlübersetzung takes a broader approach and looks rather at an entire phrase or paragraph, or in other words rather at an entire thought than just a word. Also, it would cover problems in translations that perhaps are not 100 per cent false, but are disputable in how they interpret the original.
This includes that you cannot sharply separate Fehlübersetzung from Übersetzungsfehler. Often it is rather the way you - as the speaker - look at things or present things by calling them (broadly and midly) Fehlübersetzung or (narrowly and sharply) Übersetzungsfehler.
It also includes that Fehlübersetzung sounds more polite, whereas Übersetzungsfehler sounds very direct and harsh.
This is related to how the words are built: Übersetzungsfehler is first of all a Fehler; Fehlübersetzung is first of all an Übersetzung; the added nouns are modifiers.
Additionally you may pay attention to emphasis: In both words, the first component (the modifier) is emphasised by pronuncation. That means you could use the words for further differentiation:
Das ist Übersetzungsfehler, kein Tippfehler!
Das ist eine Fehlübersetzung, keine zulässige Interpretation.
Note that these are intricacies altogether. In everyday communication, you can use the one word or the other without getting noticably misunderstood. (With Fehlübersetzung used in oral communication you may be sounding a bit posh, though.)