As far as I know, "unendlich" is an adverb/adjective. And after bis should come a noun. How is the first citation ok? Can someone explain? Thanks.
Actually I see no reason for a substantive there. unendlich is the appropriate number word on the same level as zero, one, pi and four (which are no substantives either). In German Unendlichkeit is the term for philosophy and religion and I find it quite appropriate, to have that separated.
Finally: yes, integrieren von 0 bis unendlich is fine (no reason for uppercase).
And after bis should come a noun
... no, not at all:
Bis morgen, bis hierher, bis dann, bis fünf, bis unendlich
where do you see nouns used here?
Nouns are used with bis + prep., e.g.
bis zum Ende, bis an die Kante, ...
Correct German would be "bis unendlich" and "bis zur/an die Unendlichkeit". Only the former is used in mathematics, consistent with other numeral words like "bis fünf" (and not "bis zur Fünf")