Can someone who undertands verification systems better explain how the spec/intents are created for such usecases along LLMs? Does a human form the spec or the system generates one based on human's natural language intent?
I haven't followed this area but it seems like LLMs are used to bridge and form a hybrid spec?
Tudor here, I work at Harmonic. Aristotle auto-formalizes your natural language specifications into theorems about the embedding of your program based on the formal semantics of the Rust mid-level IR. You can see the full docs here: https://aristotle.harmonic.fun/docs/software/rust
For Verilog which is another language we support, for hardware, we compile the SVA assertions deterministically into Lean theorem statements about the compiled Lean embedding of the Verilog.
Lean is great. But these days with Aristotle you can check memory safety for other languages like C so it's not totally obvious it's worth rewriting everything in Rust.
> Or check SIGPLAN papers on machine learning based compilers.
Mind providing some examples? My (admittedly probably dated) understanding of how ML is used in compilers is for picking heuristics/thresholds/other things for which there's no known optimal "right" answer (e.g., pass order, inlining thresholds). Everything else was done in the "traditional" manner, which meant that the machine learning bits were, strictly speaking, completely optional. I haven't heard of compilers where ML techniques are integral to the process to the point where removing any ML bits would result in a nonfunctional compiler.
Rust is a terrible language for many of the tasks people try to shoehorn it into. As processors and memory get faster, garbage collection only becomes more attractive, while the borrow checker remains a PITA that gets in the way of plenty of sensible architectures. In a sane world, Rust will never take over frontend development, for example - it’s a terrible fit.
(I qualify with “in a sane world” because obviously it could happen. In a truly sane world, JavaScript would never have happened, either!)
> As processors and memory get faster, garbage collection only becomes more attractive, while the borrow checker remains a PITA that gets in the way of plenty of sensible architectures.
I’d counter with: as LLM driven coding becomes more and more common the PITA borrow checker is something individual developers interact with less than less. Garbage collection looks more and more like a waste when an LLM can examine memory allocations before the app is even built.
Maybe, but whatever workarounds or compromises the LLM has to settle on to satisfy the borrow checker will live in the code forever, while garbage collection can continue to get faster in the same executable.
Eh. In my experience any long-running Rust project will go through a lot of refactors, often driven by borrow checker issues. The only bright spot is that the compiler makes them relatively easy and safe. But it would be better not to have to do them in the first place if your architecture is already solid, just not in a way that the borrow checker understands.
> As processors and memory get faster, garbage collection only becomes more attractive
Just because they did in the past doesn't mean they will continue to do so in the future. If we are honest then Moore's Law ended a decade ago (the latest) and we've been picking the low hanging fruit we left alone during Moore's era since. This too will come to an end for sure, just not a pretty and well defined one.
So, my conclusion is the opposite. Garbage collected and DX optimized languages made sense while processors and memory got faster, the future will be elsewhere.
Did Moore’s Law really end? My ARM Mac from five years ago is a literal order of magnitude faster than a gigantic, beefy desktop tower from five years before that, and my newest ARM Mac from this year is another 5x or so faster than my first. Computers continue to get much, much faster, at least in terms of time to completion of normal workloads.
As processors and memory get faster, people will expect more from them. And the fact that languages like Rust give you fine-grained control over memory allocation is part of what allows it to be so performant on modern hardware where memory latency is still relatively high and bandwidth still relatively low. My intuition is that any data-intensive operation (such as blurring an image) will almost always be far faster in rust than javascript.
I think most of those tradeoffs flip as more of the engineering is done by AI systems. I think that's true on its own, but it's especially true if those systems are increasingly capable of autonomously verifying their own correctness.
I'm not advocating for Rust (or against it), I'm just stating that currently it's hard to open the news without seeing Rust expanding somewhere, and it feels like "taking over the world". Whether that is bad or good is another question.
And yeah, it feels a bit like JavaScript a while ago, that's a good comparison.
That's true and there are some truly high throughput systems that absolutely benefit from garbage collection (I've written one of them, and can't sing enough praises of not one but several java garbage collection types), but there are two problems here:
1) most garbage collectors are pretty lame and can't even come close to the throughput/efficiency of manually managed memory, and
2) the vast majority of software does not even come close to the frontier of what is possible with software, and therefore the thing that matters most is correctness.
Rust does a better job at correctness than most languages because the type system is so robust, and it does a better job at memory management than most languages because most languages do garbage collection so poorly.
I don’t think you can do a “you’re holding it wrong” with UI code when it seems like every UI framework eventually evolves to let people hold it that way. People find it difficult to build fully-featured UI frameworks in Rust, and people writing UIs don’t tend to pick Rust frameworks. They aren’t wrong for that!
No language that perfectly fills that niche exists, because none was ever needed. JavaScript eventually became Good Enough. Python is OK, but it’s an old language at this point and it suffers a lot of technical debt.
JavaScript’s biggest weakness IMO is its absolutely abysmal standard library, particularly in terms of data structures. (At least we have Map and Set now, but still no way to define a hash/equality function for a type…) The situation is so bad that essentially any other modern programming language would be an improvement.
Can someone who undertands verification systems better explain how the spec/intents are created for such usecases along LLMs? Does a human form the spec or the system generates one based on human's natural language intent? I haven't followed this area but it seems like LLMs are used to bridge and form a hybrid spec?
Tudor here, I work at Harmonic. Aristotle auto-formalizes your natural language specifications into theorems about the embedding of your program based on the formal semantics of the Rust mid-level IR. You can see the full docs here: https://aristotle.harmonic.fun/docs/software/rust
For Verilog which is another language we support, for hardware, we compile the SVA assertions deterministically into Lean theorem statements about the compiled Lean embedding of the Verilog.
For those wondering what “embedding” is:
> Aristotle compiles the provided Rust code deterministically into our proprietary Lean formal model of Rust execution (onward: the embedding).
I was also wondering how readable the auto-formalized spec is, but “proprietary” suggests the formal spec is hidden?
As a user I’d much prefer looking at the formalization myself instead of trusting the auto-formalization from natural language.
The proofs are hidden, but a formal Lean spec will in most cases be returned along a natural language description of the proven property.
Is there documentation on a set of “user-facing” Lean definitions to help me understand the formal spec returned?
Here's an example result, on a specific function of the Symphonia FLAC decoder: https://aristotle.harmonic.fun/projects/55dc5e6e-4e12-47ab-b...
More documentation on Rust specifically can be found here: https://aristotle.harmonic.fun/docs/software/rust
Is there an open source alternative to this?
> Aristotle can prove that your code has no undefined behavior ... even inside unsafe blocks.
That's crazy, I haven't seen any other system that really tries to do that.
Some others in the Rust ecosystem, not a complete list:
- cargo-anneal - VeriFast - RefinedRust - safety-tags - Verus
These 5 use very different mechanisms. This is still a very active research area.
Most people using rust unsafe should likely be using
https://github.com/google/rust-skills/blob/main/unsafe_rust_...
Better to verify imo.
Wow, this is a pretty advanced use of syllogistic logic!
There are a lot of verification tools for rust popping up recently it's getting hard to navigate the space.
Formal verification of Rust is the final frontier for these kinds of tools
"Feels like in the future the only languages would be Rust and Lean"
Lean is great. But these days with Aristotle you can check memory safety for other languages like C so it's not totally obvious it's worth rewriting everything in Rust.
Right, but Aristotle easily burned my initial $500 free credits doing so. Rust's memory safety is pretty cheap in comparison.
Doesn't Aristole work by compiling to Lean?
I would go with English, generating directly machine code, as dreamed by 4 GL and Japan's fifth generation project.
You can already see this taking of on low code, no code tooling, reinvented as SaaS agentic runtimes.
Or check SIGPLAN papers on machine learning based compilers.
> Or check SIGPLAN papers on machine learning based compilers.
Mind providing some examples? My (admittedly probably dated) understanding of how ML is used in compilers is for picking heuristics/thresholds/other things for which there's no known optimal "right" answer (e.g., pass order, inlining thresholds). Everything else was done in the "traditional" manner, which meant that the machine learning bits were, strictly speaking, completely optional. I haven't heard of compilers where ML techniques are integral to the process to the point where removing any ML bits would result in a nonfunctional compiler.
Rust is a terrible language for many of the tasks people try to shoehorn it into. As processors and memory get faster, garbage collection only becomes more attractive, while the borrow checker remains a PITA that gets in the way of plenty of sensible architectures. In a sane world, Rust will never take over frontend development, for example - it’s a terrible fit.
(I qualify with “in a sane world” because obviously it could happen. In a truly sane world, JavaScript would never have happened, either!)
> As processors and memory get faster, garbage collection only becomes more attractive, while the borrow checker remains a PITA that gets in the way of plenty of sensible architectures.
I’d counter with: as LLM driven coding becomes more and more common the PITA borrow checker is something individual developers interact with less than less. Garbage collection looks more and more like a waste when an LLM can examine memory allocations before the app is even built.
Maybe, but whatever workarounds or compromises the LLM has to settle on to satisfy the borrow checker will live in the code forever, while garbage collection can continue to get faster in the same executable.
Rust code tend to be "better" because those concerns forces the architecture upfront and guaranteed that nothing breaks later on.
Eh. In my experience any long-running Rust project will go through a lot of refactors, often driven by borrow checker issues. The only bright spot is that the compiler makes them relatively easy and safe. But it would be better not to have to do them in the first place if your architecture is already solid, just not in a way that the borrow checker understands.
The same arguments could often apply to Zig with 100% up-front static allocation (like TigerBeetle uses).
> As processors and memory get faster, garbage collection only becomes more attractive
Just because they did in the past doesn't mean they will continue to do so in the future. If we are honest then Moore's Law ended a decade ago (the latest) and we've been picking the low hanging fruit we left alone during Moore's era since. This too will come to an end for sure, just not a pretty and well defined one.
So, my conclusion is the opposite. Garbage collected and DX optimized languages made sense while processors and memory got faster, the future will be elsewhere.
Did Moore’s Law really end? My ARM Mac from five years ago is a literal order of magnitude faster than a gigantic, beefy desktop tower from five years before that, and my newest ARM Mac from this year is another 5x or so faster than my first. Computers continue to get much, much faster, at least in terms of time to completion of normal workloads.
As processors and memory get faster, people will expect more from them. And the fact that languages like Rust give you fine-grained control over memory allocation is part of what allows it to be so performant on modern hardware where memory latency is still relatively high and bandwidth still relatively low. My intuition is that any data-intensive operation (such as blurring an image) will almost always be far faster in rust than javascript.
I think most of those tradeoffs flip as more of the engineering is done by AI systems. I think that's true on its own, but it's especially true if those systems are increasingly capable of autonomously verifying their own correctness.
I'm not advocating for Rust (or against it), I'm just stating that currently it's hard to open the news without seeing Rust expanding somewhere, and it feels like "taking over the world". Whether that is bad or good is another question.
And yeah, it feels a bit like JavaScript a while ago, that's a good comparison.
That's true and there are some truly high throughput systems that absolutely benefit from garbage collection (I've written one of them, and can't sing enough praises of not one but several java garbage collection types), but there are two problems here:
1) most garbage collectors are pretty lame and can't even come close to the throughput/efficiency of manually managed memory, and
2) the vast majority of software does not even come close to the frontier of what is possible with software, and therefore the thing that matters most is correctness.
Rust does a better job at correctness than most languages because the type system is so robust, and it does a better job at memory management than most languages because most languages do garbage collection so poorly.
I wonder if Rust is really bad for UI or UI code is just a mess of terrible practices like having event listeners for everything.
I don’t think you can do a “you’re holding it wrong” with UI code when it seems like every UI framework eventually evolves to let people hold it that way. People find it difficult to build fully-featured UI frameworks in Rust, and people writing UIs don’t tend to pick Rust frameworks. They aren’t wrong for that!
> In a truly sane world, JavaScript would never have happened, either!
Out of curiosity, what do you envision would have taken its place in frontend, in that sane world? (Honest question, no bad faith agenda.)
No language that perfectly fills that niche exists, because none was ever needed. JavaScript eventually became Good Enough. Python is OK, but it’s an old language at this point and it suffers a lot of technical debt.
JavaScript’s biggest weakness IMO is its absolutely abysmal standard library, particularly in terms of data structures. (At least we have Map and Set now, but still no way to define a hash/equality function for a type…) The situation is so bad that essentially any other modern programming language would be an improvement.
I for one wish python had taken the spot JavaScript holds.
Maybe a hot take, but I don't think Aristotle is a good name for an AI.
An excellent ending.