The output of iBooks Author is, as far as I can tell, HTML5 — pretty much ePub 3 with whatever nonstandard liberties Apple saw fit to take in order to achieve the results they wanted. It’s not a standard format in the sense of following a spec from a standards body like the W3C, but it’s just HTML5 rendered by WebKit — not a binary blob tied to iOS or Cocoa. It may not be easy, but I don’t think it would be that much work for anyone else with an ePub reader that’s based on WebKit to add support for these iBooks textbooks.