Which is correct?
Er setzt sich auf einen Stuhl an dem Tisch.
Er setzt sich auf einen Stuhl an den Tisch.
In my book both are written; my guess is the first one because the table is not involved in the movement.
The first one gives a description about the location of the chair. That is why we need the dative (position).
"Der Stuhl steht am (an dem) Tisch".
"Er setzt sich auf einen Stuhl am Tisch"
This means he's sitting down on a chair that is located at a table.
The second sentence tells us where he's sitting with 2 descriptive objects. He's sitting down at the table and also on a chair. Since sitting down is an action that indicates a direction, we need the accusative, twice.
"Er setzt sich an den Tisch" He's taking a seat at a table
"Er setzt sich auf einen Stuhl" He's taking a seat on a chair
"Er setzt sich auf einen Stuhl an den Tisch" both together
Neither is optimal for what I think is obviously what you want to say.
Er nimmt auf einem Stuhl am Tisch platz.
This has both auf einem Stuhl am Tisch and am Tisch platz nehmen in the suitable dative case. In this case, the case commands the noun.
Neither Tisch nor Stuhl are direct object or focus of the phrase. sich is. It could likewise say, Er wurde auf einem Stuhl am Tisch plaziert, which in active voice should have him as the direct object, Ihn.
Your first example is not idiomatic (not expected to be found in a text corpus). I can't quite say way. It's just not a statement that needs saying, I guess, because when you talk about sitting and tables, it is understood that a chair will be involved. So we would most often simply say Er setzte sich an den Tisch. This shows that the direct object in accusative case is the focus of the phrase. If the chair needed focus, it would most likely come last, ordered from general to specific. Meanwhile, if different chairs are in focus, and Tisch specifies the chair of choice I would either say einen der Stühle--or ... or ... nevermind. It's indecicive whether this should be deemed grammatical (whatever rules of thumb your textbooknassumes, I guess).
Those constraints of focus and order are not maintained in your second example, either, because the agreement in case implies equivalence, thus an enumeration and chronological order. Basicly what @Holger commented
Er setzt sich auf einen Stuhl an den Tisch“ could even imply that he moved the chair to the table in the action whereas „an dem Tisch“ invariably implies that the chair was already at the table. – Holger 6 hours ago
Alas, if you first sit down and then move the chair to the table (for a noteworthy distance), this implies a rather funny picture. You cannot sit twice in a row. I suppose the actual difficulty here lies with translations of on and which cases those prepositions rule.
However, if both objects were indefinite, the first example would be acceptable without further context.
Er setzte sich auf einen Stuhl am Tisch.
Otherwise I have to imagine an ekaborate piece of literature tgat details the environment to determines the setting with a surprising conclusion. A real page turner, why the table not the bar, and why a chair, if he could sit upon the table or downon the floor? I