X sollte dich nicht betrüben, Y umso mehr.
= "X should not pose you any problem – Y (on the other hand) all the more."
I have no problem understanding "umso mehr" when it is used in the "je mehr ... , desto/umso mehr" construction. I'm not sure, though, what exactly "umso mehr" means in this specific sentence.
The rather literal translation "all the more" does not seem to make much sense.