Long-term Goals for Herbie

Since I maintain Herbie in my free time (I have other research projects), making Herbie easier to maintain is a central goal. Picking an input format, a math library, and a sampling mechanism settled questions that otherwise had to be decided again and again.

The FPCore input format gave Herbie a standard, well-documented input format. It has a proper parser, with syntax and type checking, instead of eval-ing each input file, which provides better warnings and errors.

Supporting math.h gives Herbie a clear answer to what functions Herbie supports and how they are implemented. We sacrificed some convenience functions like sqr and cube, but we gained clarity.

Removing samplers, so Herbie always samples input points uniformly over bit patterns, removed a confusing and poorly-implemented feature. The main use for custom distributions was to efficiently sample from small ranges like 12 ≤ x ≤ 14. Now, Herbie uses a static analysis to do the same.

Often a good idea starts with a pile of supporting hacks and heuristics. Removing those bit by bit makes Herbie more reliable and eliminates errors and corner cases.

Herbie now soundly finds exact outputs for programs with interval-analysis, to correctly measure how much a program's accuracy. That used to involve heuristics. Heuristics are useful, but not in the part of Herbie that measures how well Herbie works.

Herbie's regime inference is better thanks to binary search, which generates accurate if statements that don't change by factors of ten with every run. Plus, regime inference now supports more types of conditions.

Making the web demo and HTML reports the standard interface to Herbie gives users more information about what Herbie did and made Herbie easier to use. I don't know if anyone even uses the command-line version any more.

Herbie now collects a lot of metrics about its internals and presents it in a pretty GUI. That makes it a lot easier to track down problems with accuracy and speed, and has helped make Herbie much faster.

I have high hopes Herbie will continue getting better—it's a fun project to work on, with real users, and leverage to improve the state of scientific computing.

Refactoring Herbie with multiple formats will separate the notion of a type and its representation. That will support mixed-precision programs and new formats like posits. It could also help make Herbie a platform for research into new FP data types.

Support for pluggable libraries means Herbie could be extended to support complex numbers, matrices, vectors, and all sorts of other libraries, potentially even by users. This could make Herbie much more useful to scientists.

Faster simplification is essential for speeding Herbie up, since simplification is now the majority of Herbie's runtime. Simplification is a bottleneck on the search process: if it were faster, Herbie could try more things and get better results.

The web interface should give users access to preconditions, precisions, or command-line flags. The web should be a first-class interface to Herbie. Developing UIs takes time, but it is worth it.

Herbie's core search algorithm is too focused, and often discards promising candidates too early. I think I know how to fix it. Even if it doesn't improve Herbie's results, it should remove a heuristic and make those results more reliable.