You are absolutely correct in your first assumption: "dem" refers to "Sinn" and is Dativ. The "dem" is here not an article, though, but short for "d[ies]em" and a Demonstrativpronomen - a pronoun which points to another part of the sentence. A comparable role in English would play the "who" in i.e. "He, who is ...". One could replace "dem" with "diesem", "jenem", "welchem", etc..
The sentence parses out like this:
Ich nehme den Begriff der Lebensweisheit hier gänzlich im immanenten Sinne
He wants to talk about "Lebensweisheit" and explains what he means by that term.
nämlich in dem
The "dem" here represents the "im immanenten Sinne" from the main sentence. Translated that would be "namely the one which ...".
This indeed is Genitiv but inside a Dativobjekt. If you replace the Demonstrativpronomen with the part it refers to, like:
nämlich im gänzlich immanenten Sinne der Kunst
it becomes clearer how these parts hang together.
The remainder of the sentence is just a clarification, which art ("Kunst") exactly he is talking about.
A possible translation of the whole sentence could be (although my English wouldn't translate the "slightly antiquated" feeling of Schopenhauers phrasing):
I use the term "Lebensweisheit" here in a completely immanent sense: the art of living ones life as comfortable and happy as possible.