I think I recognize the word "Alp" as a medieval word for some kind of
demon or ghoul. That may be the origin of the word "Alptraum"
Correct, but according to the German Wikipedia page, which refers to "Duden Herkunftswörterbuch", both variants exist since medieval and both are equal.
It's actually not possible to recommend the one or the other.
"Alp" does also mean "mountain pasture" and, of course, words referring to that mountains begin with "Alp...".
So maybe "Albtraum" is the better choice?
However, there are geographic regions like "Fränkische Alb" or "Schwäbische Alb", which have nothing to do with the meaning of "Alb" in "Albtraum".
For these names, Wikipedia says: "Alb" was formerly supposed to come from Latin "montes albi" (white mountains), but more likely it was a celtic word for "mountain pasture".
After all, it's really up to you which variant to use.
Coming from Frankonia, where "p" is anyway spoken as "b", I would prefer "Albtraum".
But I would never argue with someone stronger who insists on "Alptraum" being the one any only correct word - this could end up in a nightmare... ;-)