Note that "being offensive" is a cultural and not a language thing. While in the English-speaking world you typically rotate around and around if you want to say something, in the German-speaking world you typically just say it. It is polite vs. honest, or dishonest vs. impolite, and neither is right or wrong (or better than the other).
Why not just say "Das hast du falsch gemacht, das liegt dir einfach nicht"? If you want to say something offensive (and you do want that, otherwise you wouldn't blame it on the inability to do something, but rather on the mistakes done), then say it.
I don't see how blaming someone's inability is less offensive than blaming it on his Unfähigkeit. It will be slightly more polite if you specify what the Unfähigkeit is about, like Unfähigkeit, [x] zu machen.
But since the question is about alternative phrases:
- Du hast hier einige Fehler gemacht.
- Du hast in diesem Bereich zu wenig Erfahrung.
- Diese [Dinge] sind einfach nicht deine Stärke.
- Du hast in diesem Bereich einige Defizite.
- Wir sollten dieses Problem lieber gemeinsam lösen.
Note that you'll be kind of offensive anyway, because you have to tell them that they screwed up, somehow. After all, they did make a mistake, and unless you want to help him fix the situation, in which case you can talk about the mistake, rather than his general inability, there is no way around. You can, however, avoid being insulting by using phrases like the ones I suggested.