As far as I remember it is pronounced like ['vaɪəʁˌʃtræs] but not quite sure about it (specially whether it is a "ʁ", "ʀ" or "x" in the middle).
Usually, the "r" in the middle is vocalic, that is, ['vaɪəɐ̯ˌʃtʁas] or ['vaɪɐ̯ˌʃtʁas]. Using [ʁ] instead of [ɐ̯] is also fine (some might claim that this is the "official" pronunciation). I would not use a trill here.
The second "r" may be a fricative [ʁ] or a trill [ʀ], [r], depending on your dialect. The last vowel is usually pronounced short by mathematicians, but a long vowel would also be possible. It's certainly not [æ].
It is pronounced ['vaɪ̯əʁˌʃtʀas], ['vaɪ̯ɐˌʃtʀas] or ['vaɪ̯əʀˌʃtʀas], depending on the dialect, under the assumption that the a is a short vowel. According to a lot of old orthographies, ß could be preceded by both, a short and a long vowel. This does not exclude ['vaɪ̯əʁˌʃtʀaːs] and similar, however, having studied Mathematics in Germany for some years, I have never heard anyone claiming that he would be pronounced like this.
I do not see any reason to assume the pronunciation ['vaɪ̯əʁˌʃtʀæs] or ['vaɪ̯əxˌʃtʀas].
Not being a mathematician, my inclination was to pronounce it
with the sound of the third syllable equivalent to the
ʃtʁas in Strass (rhinestones) or like the
as in Trasse. (Since the recent orthography reforms, vowel before 'ß' gets a long or half-long sound, but the spelling of proper names remains unaffected, thus names like Haß or Heß sound exactly like Hass or Hess.)
A video recently added to Youtube confirms my intuition by showing the phonetic rendering –
'vaɪɐʃtʁas – I expected. Confusingly, though, the speaker pronounces the vowel in the third syllable half-long, some might even say long (as in Straße):
Cross-checking with a teaching video also on Youtube, I find my confusion resolved in favor of my original intuition. The author, presumably a mathematician, pronounces the name exactly as I would expect: