I am not sure that we can call a text that has not yet been typeset a "book". In any case, do you know any such format that does not completely botch equations? The closest I can imagine is html with mathjax.
If you want a standalone HTML solution I reckon you could embed the formula as SVG. Or maybe you can somehow get MathML to look nice, but so far that seems tricky without using JavaScript to help (which might not be supported/desirable in a document format) and I'm not sure about the state of MathML support in e-readers.
One of the problems with including mathematical formulae in a reflowable document format is that the concept of reflowable mathematical notation simply doesn't seem to exist, so In practice you'll end up with something equivalent to a picture.
Wait, what format do you expext a book to be? I mostly skip any book that is not on pdf