As far as only word order is concerned, (b) would be correct. However, there are two problems with (b). One is that this is probably not what you want to say as it's hard to come up with a context in which it would make much sense. We can rectify this by replacing erwartet by gesagt to get better examples:
(a) Ich habe nicht gesagt, dass er dich besuchen wollen hat.
(b) Ich habe nicht gesagt, dass er dich hat besuchen wollen.
(c) Ich habe nicht gesagt, dass er dich besuchen hat wollen.
As you can see, I have added a third word order. Only (b) is officially correct, but it sounds very stilted and awkward. (Which is the second problem.) (a) is correct in some dialects, so speakers of those dialects tend to use it in standard German even though it's not correct there. (c) is the word order used in Dutch and probably also in some German dialects, but not correct in standard German.
Basically you have found a defect in the German language. In a century or so this will probably have been resolved one way or another, but today most people avoid the issue as follows:
(d) Ich habe nicht gesagt, dass er dich besuchen wollte.
I think the reason for the awkwardness is that it only works like that when the infinitive (here: wollen) is used in place of the past participle (here: gewollt), as in this case. In colloquial German most people would typically use gewollt, in which only the word order of (a) would be correct:
(a') Ich habe nicht gesagt, dass er dich besuchen gewollt hat.
So it's a really weird corner case of the language.