Both are correct, but "Es sei ..." is the old-fashioned way of saying it and "Sei ..." is the modern one. Fortunately, the Google Books corpus includes mathematics books, and fortunately it treats the simplest formulas as words. Since the n-gram viewer is capitalisation-sensitive, we can get a diagram comparing the relative frequencies of "Es sei ..." and "Sei ..." for the typical variable or function names x, y, f, g.
See here.
You can see that Sei ... has always been around, but that ever since the 1970s it has been more common than Es sei ... and that more recently it is in fact a lot more common.
I think that if one were to look closer one would also see that Sei ... and Es sei ... are used differently. Consider the following four equivalent ways of postulating continuity of a function f:
- Sei f stetig.
- Sei f eine stetige Funktion.
- Es sei f eine stetige Funktion.
- Es sei f stetig.
1 and 2 are standard nowadays. 3 sounds old-fashioned. 4 just sounds wrong. I think this is primarily because it combines modern colloquial brevity with the antiquated use of an unnecessary extra word. I guess that this combination, even if it should have occurred occasionally, can never have been very popular.