Both your sentences are missing a verb ;)
In general, constructions like "am (Verb) sein" are a progressive form mostly used in colloquial speech. It was known as the "rheinische Verlaufform" (roughly "Rhinelandian progressive form"), but recently the less localized term am-Progressiv is used more often.
The idea of the construction is roughly similar to something like "to be currently at (verb)-ing":
Sie ist am Lesen.
She's at reading.
or, in more natural English
She's reading at the moment.
She's busy reading.
Other examples would be
Was möchtest Du essen? - Ich bin noch am Überlegen.
What would you like to eat? - I'm still thinking.
Die Stimmung war am Kochen.
The mood was boiling / at the boiling point.
So, one of your sentences in question needs a form of "sein":
Er ist am Denken.
And in Konjunktiv it would be
Er blieb still, als wenn er am Denken wäre.
Er blieb still, als wenn er am Denken sei.
The difference between "wäre" and "sei" in this case would probably warrant a question of its own.
On the other hand, "in etwas sein" just means "to be in something". So, "Er ist im Denken" would be something like "he's in thinking", which could be understood as similar to "he's in thought". But it wouldn't be phrased like that in German. The example you found at DWDS uses a construction similar to "free in thinking" or "free at thought".
So, of the sentences you suggested, only
Er blieb still, als wenn er am Denken wäre.
would be correct.
But as mentioned before, the am-Progressiv is a quite colloquial construction. So there's a bit of a styles clash between the two parts of your sentence, as they are from different registers. A better way of phrasing, as Sixpence already suggested, could be
Er blieb still, als wenn er in Gedanken wäre.
Er blieb still, als wenn er in Gedanken versunken wäre.
"In Gedanken versunken" ("to be sunk into thought") is a fixed phrase that means something like "to be absorbed in thought".