"Ich verstehe mich wirklich gut mit meiner Mutter Sylvia, der sehr intelligent ist, weil sie nett ist."
Is the usage of "der" in the relative clause correct here, since 'mit' takes the dative case?
No, it's not correct. The case of the relative pronoun (here:
So, the correct form is:
Indeed dative for feminine determined article is der. But the relative clause is in nominative, because you have the verb sein there.
Maybe you thought you should use dative, because you use it inside an apposition, which indeed inherits the case of the object it is describing:
Notice there is no verb. But I don't really know if in German one speaks like that about one's mother. That is I guess I just find strange meine Mutter Sylvia, since you don't have two mothers. And if you want to mention her name, maybe it's a better idea to split the sentence in