Und ginge es nur ums Aussehen, wäre das ja auch nicht weiter schlimm. Aber ...
For the sake of breaking down the meaning of the sentence, I go with a rather literal translation here:
= "If it were just a matter of appearance, that would indeed really not so bad. But ..."
At first, I thought this auch means "too/as well" in relation to what has been said in a previous context, but I realised that it might be more like "wirklich/really" to talk about a real situation.
One by one:
ja = indeed {in relation to the subsequent aber}
auch = wirklich/really {to talk about a real situation}
nicht weiter = not further / not really / not so
I wonder if I'm on the right track in my interpretation?