Herbie - the tool which exists today - is excellent, but the perhaps more significant insight is that this could and should in principle be baked into compilers.
That is, a future language should let you say here's a real expression, and I want you to give me an approximation in my 64-bit floating point representation, and here's the parameters for accuracy and performance. What do those parameters look like? Herbie's existing UI might help us somewhat but it's definitely up for experimentation.
Compare regular expression handling - you are not expected in any popular language today to manually translate the expression into an FSM and then run the FSM, that's all done for you, you just write down the expression, real arithmetic could be the same.
Definitely an interesting idea. It has been discussed here a bunch of times before. Usually someone points out cases where it doesn't do so well. Has it been updated lately?
This tool is amazing. I've used it many times when I want to run simulations with symplectic integrators over large time horizons in an effort to reduce the total accumulated error !
Close floating-point subtraction loses precision: when x >> 1, sqrt(x+1) ≈ sqrt(x), so their difference suffers cancellation and end up rounding to zero. In contrast, sqrt(x+1) + sqrt(x) approaches 2*sqrt(x) smoothly.
I wonder whether this could help optimize AI inference and training engines or are they already squeezed to the max. llama.cpp could potentially be a great target.
Herbie - the tool which exists today - is excellent, but the perhaps more significant insight is that this could and should in principle be baked into compilers.
That is, a future language should let you say here's a real expression, and I want you to give me an approximation in my 64-bit floating point representation, and here's the parameters for accuracy and performance. What do those parameters look like? Herbie's existing UI might help us somewhat but it's definitely up for experimentation.
Compare regular expression handling - you are not expected in any popular language today to manually translate the expression into an FSM and then run the FSM, that's all done for you, you just write down the expression, real arithmetic could be the same.
Definitely an interesting idea. It has been discussed here a bunch of times before. Usually someone points out cases where it doesn't do so well. Has it been updated lately?
https://news.ycombinator.com/item?id=38451680 (159 points)
https://news.ycombinator.com/item?id=24069465 (152 points)
https://news.ycombinator.com/item?id=10964617 (117 points)
This tool is amazing. I've used it many times when I want to run simulations with symplectic integrators over large time horizons in an effort to reduce the total accumulated error !
Can someone explain why sqrt(x+1) - sqrt(x) is “inaccurate” for all x>1? Inaccurate how, and why, and why is 1/(…) any better? Thanks.
Close floating-point subtraction loses precision: when x >> 1, sqrt(x+1) ≈ sqrt(x), so their difference suffers cancellation and end up rounding to zero. In contrast, sqrt(x+1) + sqrt(x) approaches 2*sqrt(x) smoothly.
I wonder whether this could help optimize AI inference and training engines or are they already squeezed to the max. llama.cpp could potentially be a great target.
Just hope that Herbie doesn't go bananas again
This looks like really interesting and valuable work.