Very cool and I like the idea of a "meta-assembler." The most-recent version of flatassembler (fasm 2) is built with fasmg which is also a "meta-assembler" of sorts, in that it also doesn't directly support a specific instruction set and instead is a very powerful macro assembler. I'm keen to check out functionality overlaps between the two implementations.
Oh neat! Thanks for the link, I hadn't heard of fasmg before.
It looks like fasmg builds up from the byte level, so it would only work for architectures that use 8-bit words. Torque builds up from the bit level, so it can assemble code for architectures like in PIC microcontrollers, using word sizes of 12 or 14 bits.
However, fasmg does allow a lot more control over the syntax of the language. The documentation shows some pretty powerful string manipulation that's used to parse real x86 assembler code, which makes sense given the purpose of it. Torque doesn't allow overriding the syntax to that degree, the macro invocation syntax is baked into the assembler.
One thing that intrigues me about fasmg is how it handles circular dependencies in expressions [0] (search for 'circular'). Currently in Torque it isn't possible to use a label reference inside a predicate, because the predicate evaluating one way could insert additional code, moving the label and causing the predicate to evaluate the other way [1]. But in fasmg it's possible to use the result of an expression as part of its own calculation.
Tomasz (of fasmg) added "one more quick thought" re: 8-bit word not being a limitation, he said that since fasm1 had macros for LZW compression [0] (from the assembler alone) and that this is even easier with fasmg the word length isn't a restricting factor with his design either. Cheers and thanks for sharing.
I reached out to the author of fasmg WRT your post and circular dependency interest and he pointed me toward two posts that he wrote very specifically to explain what he believes is unique to fasm/fasmg and allows to handle circular dependencies of many kinds. [0] Types of multi-pass assembly, and [1] related pitfalls.
Thank you so much! It looks like the issue I described is what he calls the 'oscillator problem' [0]. This is an absolute goldmine, I'm going to be reading for days.
Thank you! My main inspiration was the Uxn assembly language [0], which is itself heavily inspired by Forth. I loved how easy it was to build something that looks like a high-level language by just stacking up macros, and I wanted to have that with embedded development too.
Rust isn't involved past implementing the Torque executable; you write your program with the Torque language and then run the assembler on it to convert it to machine code. You can see the whole process of running code on the TRS-80 from start to finish here [1].
For laying out structs, I'd build a macro that expands to the memory representation of the struct. If I wanted a struct representing, say, a 2D point with signed 16-bit little-endian integers for the x and y coords, I would build it from scratch like this (this is a valid program, you can assemble it with Torque):
Creating a DSL for an existing language wasn't something I'd ever considered. By being a standalone executable it's really easy to use and share, people don't have to install a whole language toolchain in order to use it.
Regarding constraints solving and jumping, Torque already throws an error if you try to pack too large a value into too small a field. This works really well for things like relative jumps, because jumping too far will create a value that can't fit in the instruction. I'm planning on adding an error-throwing token to the language that could be used alongside expressions and conditions to further constrain the values accepted by a macro, but I'm really happy with the simplicity of the language so far.
The actual internal representation isn't what I'd call an 'IR' per se, nothing like with a C compiler. It's all very pedestrian; the syntax tree is baked down across multiple passes, with macros acting as a glorified copy-paste system.
Thanks for the interest and the links, every one of those linked projects is new to me.
At first I thought it was useless: "but each ISA will still end up having different effective syntax because the underlying macro systems will not be designed the same".
But then I reread it and realised I was not paying attention to the usecase. It's about making it easy to write assemblers. So this isn't for your Arms and RISC-Vs it's for your random niche microcontrollers where the vendor-provided toolchain kinda sucks.
Seems cool!
I've experienced a couple of under-documented assemblers in my time. In neither case did this turn out to be that much of a problem in practice, but I guess I just don't write that much assembly.
I'm reminded of a 2016 [talk][1] where Rob Pike describes the common-denominator assembly language that the Go compiler generates. Then that assembly is translated into machine-specific code via table lookups. See the 11 minute mark.
I remember there were a few other meta-assemblers I came across in the 80s-90s, so this is definitely not "unchartered territory", but it's good to see another one show up.
Of course, in the other direction there are meta-disassemblers used for analysis in tools like Ghidra.
With character sets, I was initially going to support non-Unicode text by adding a --char-set flag to the assembler, but I decided that the character set should be defined somehow inside each program. My thought was that they could be defined as large table-like macros, something like the following:
This is, admittedly, quite unweildy for character sets exceeding a few hundred characters, but it would work passably for small character sets like those used for HD44780-style LCD screens. What character sets did you have in mind?
Octal was another feature I couldn't make up my mind about, just because I wasn't familiar with any architectures that require it. It'll be trivial to tack on though. For the Z80 instruction set, since the instruction encoding tends to cleave along octal lines, I used the following macro to pack octal digits into bytes, which has the advantage of allowing variables to be passed into each digit (the ADDr macro shows how it's used):
%XYZ:x:y:z #xxyyyzzz ;
%ADDr:r XYZ:2:0:r ;
Thanks for the heads-up about the table of contents, the links should all work now.
I think the "0" prefix for octal (as used in the C programming language) is not so good and that "0o" is better; so, if octal literals are implemented then "0o" is better.
For non-Unicode text, probably the simplest thing would be to treat the input as a sequence of bytes instead of Unicode characters; or equivalently to treat it as ISO-8859-1 (although programming it to use ISO-8859-1 may be less efficient then just using bytes, possibly; I don't know much about the working of Rust programming, so I don't actually know if it is or not).
By "non-Unicode text", I did not mean character mapping, although character mapping is another feature that would be useful to implement, similar to what you mentioned although it could be made more efficient (like you mention). Some way to map a input sequence of bytes (whether or not it is valid UTF-8) to a output character code, would work, probably.
I wholeheartedly agree with using "0o" as the octal prefix, I've never been a fan of "0". I've jut implemented this feature and released it in v2.2.0, you can grab it from the project page. Thanks for the suggestion!
I'm not too sure I understand what you're describing with non-Unicode text. Torque doesn't have a built-in concept of bytes, instead each character is treated as an integer with the value being the Unicode code point of that character (using decimal we have 65 for 'A', 955 for 'λ', 129302 for 'robot emoji', etc). It's up to the programmer to choose how to pack the character (integer) into a sequence of bits. Code points are different to encodings like UTF-8 or UTF-16, which define how a code point (integer) is packed down into one or more bytes.
If you want to assemble 7-bit ASCII text, one byte per character, you define a macro that packs each character value into the lower 7 bits of an 8-bit byte. If the string contains a non-ASCII character, the character value will be too large to fit into the field and an error will be displayed.
%ASCII:c #0ccc_cccc ;
ASCII:"This is a string."
Assembling ISO-8859-1 text would be similar, but would involve remapping the characters above 0x7F like this:
UTF-8 being a variable-width encoding requires a more complicated macro arrangement (you can see an example here [0]). But the key point is that strings aren't treated as byte sequences, they're just character/integer sequences until they get baked down into a byte encoding.
Please let me know if that doesn't answer your suggestion, I'm keen to understand your use-case.
I meant that the input should not have to be Unicode. The input could be any extended ASCII encoding (supporting ASCII is necessary, in order that the syntax will work; therefore some encodings such as Shift-JIS and UTF-16 are not suitable) which does not have to be mapped to Unicode. (Even if the input actually is UTF-8, you may have a character that is made up of multiple code points. If it is treated as a sequence of bytes then it can be any sequence of bytes, including a combination of multiple UTF-8 code points.)
It is another issue than mapping the character codes for output, which as you say can be unwieldly for large character sets, so another kind of table specification (which might also be useful for stuff other than text), would probably help and be more efficient than using a sequence of conditions in a macro.
About octal, someone said against using "0o"; it is usually not as difficult to distinguish if the "o" is always lowercase. Another alternative would be how it works in PostScript, which uses "8#" prefix for octal and "16#" prefix for hexadecimal. (My opinion is that "0o" is good enough though)
Oh I think I see what you mean now, you're talking about the encoding of the Torque source code that gets fed into the assembler. To be honest I'd never really considered anything other than UTF-8, the parsing is all implemented in Rust which requires strings to be valid UTF-8 anyway. Are you wanting to write Torque code with a non-Unicode text editor, or are you thinking about how the document encoding affects the handling of string literals inside the assembler?
Some kind of table syntax would be useful for character mappings, but I'm not sure what it'd look like or if it'd be applicable outside of dealing with characters. I'll think more on that.
> I wholeheartedly agree with using "0o" as the octal prefix
Can I give a vote against this? Distinguishing "o" and "0" can be a huge pain when using the wrong font.
Also, why not 0c? 0x already uses the "x" in "hexadecimal", so why not the "c" in "octal". That it also reads a bit like a 13375p34k abbreviation of "octal" is a nice bonus.
Thanks for the interest! Torque is public domain, feel free to use it or change it or share it however you want. I'd love to hear how people use it, but no obligation.
If you shared it someplace public I'd love to pop by, if you have a link.
I shared it on the catlang discord, Devine immediately replied "wow, that looks so much like uxn 0_o" hahaha. It's not the most active server but the response has been very positive! A few people thought it would probably be fun to use it to build a custom IR as part of a compiler pipeline, that was the context of the question.
I personally really love the utf8 and the "z80 for loop" examples, those are beautiful and make me nostalgic for the days I was writing stuff for the TI-83+ in Z80 assembly.
I've since been informed that public domain isn't really a proper term, legally speaking, so to clarify things I've released Torque under the MIT license.
Very cool and I like the idea of a "meta-assembler." The most-recent version of flatassembler (fasm 2) is built with fasmg which is also a "meta-assembler" of sorts, in that it also doesn't directly support a specific instruction set and instead is a very powerful macro assembler. I'm keen to check out functionality overlaps between the two implementations.
https://board.flatassembler.net/topic.php?t=19389
https://flatassembler.net/download.php
Oh neat! Thanks for the link, I hadn't heard of fasmg before.
It looks like fasmg builds up from the byte level, so it would only work for architectures that use 8-bit words. Torque builds up from the bit level, so it can assemble code for architectures like in PIC microcontrollers, using word sizes of 12 or 14 bits.
However, fasmg does allow a lot more control over the syntax of the language. The documentation shows some pretty powerful string manipulation that's used to parse real x86 assembler code, which makes sense given the purpose of it. Torque doesn't allow overriding the syntax to that degree, the macro invocation syntax is baked into the assembler.
One thing that intrigues me about fasmg is how it handles circular dependencies in expressions [0] (search for 'circular'). Currently in Torque it isn't possible to use a label reference inside a predicate, because the predicate evaluating one way could insert additional code, moving the label and causing the predicate to evaluate the other way [1]. But in fasmg it's possible to use the result of an expression as part of its own calculation.
[0] https://flatassembler.net/docs.php?article=fasmg
[1] https://benbridle.com/projects/torque/roadmap.html
Tomasz (of fasmg) added "one more quick thought" re: 8-bit word not being a limitation, he said that since fasm1 had macros for LZW compression [0] (from the assembler alone) and that this is even easier with fasmg the word length isn't a restricting factor with his design either. Cheers and thanks for sharing.
[0] https://board.flatassembler.net/topic.php?t=14612
I reached out to the author of fasmg WRT your post and circular dependency interest and he pointed me toward two posts that he wrote very specifically to explain what he believes is unique to fasm/fasmg and allows to handle circular dependencies of many kinds. [0] Types of multi-pass assembly, and [1] related pitfalls.
[0] https://board.flatassembler.net/topic.php?t=20249
[1] https://board.flatassembler.net/topic.php?t=21060
Thank you so much! It looks like the issue I described is what he calls the 'oscillator problem' [0]. This is an absolute goldmine, I'm going to be reading for days.
[0] https://board.flatassembler.net/topic.php?p=178828#178828
Super fun site!
Did you get inspiration from other assemblers or macro processors?
You have it running on a TRS-80, how does that work? I had no idea Rust could target a TRS-80.
I am getting hints of Forth, Lisp and TCL.
How would you go about laying out structs in memory?
I am sure you considered an internal DSL, what caused you go with something stand alone?
Any thoughts on adding a constraint solver, like Z3 and allowing end users to set constraints on things like the size of a jump.
I could see taking this an growing it into a compiler by making macro(macro(macros txt)))
Is there an internal IR?
Projects for inspiration
https://github.com/mattbierner/Template-Assembly
Specifying representations of machine instructions https://dl.acm.org/doi/pdf/10.1145/256167.256225
https://www.semanticscholar.org/paper/Specifying-representat...
Typed Assembly Language (TAL) https://www.cs.cornell.edu/talc/
And you haven't come across it, you are in for a treat https://en.wikipedia.org/wiki/META_II has spawned a whole trove of clones
https://en.wikipedia.org/wiki/OMeta
https://github.com/DalekBaldwin/clometa
Thank you! My main inspiration was the Uxn assembly language [0], which is itself heavily inspired by Forth. I loved how easy it was to build something that looks like a high-level language by just stacking up macros, and I wanted to have that with embedded development too.
Rust isn't involved past implementing the Torque executable; you write your program with the Torque language and then run the assembler on it to convert it to machine code. You can see the whole process of running code on the TRS-80 from start to finish here [1].
For laying out structs, I'd build a macro that expands to the memory representation of the struct. If I wanted a struct representing, say, a 2D point with signed 16-bit little-endian integers for the x and y coords, I would build it from scratch like this (this is a valid program, you can assemble it with Torque):
If I want the address of a field, I can add an offset to the struct address, using macros to name the offset values: Creating a DSL for an existing language wasn't something I'd ever considered. By being a standalone executable it's really easy to use and share, people don't have to install a whole language toolchain in order to use it.Regarding constraints solving and jumping, Torque already throws an error if you try to pack too large a value into too small a field. This works really well for things like relative jumps, because jumping too far will create a value that can't fit in the instruction. I'm planning on adding an error-throwing token to the language that could be used alongside expressions and conditions to further constrain the values accepted by a macro, but I'm really happy with the simplicity of the language so far.
The actual internal representation isn't what I'd call an 'IR' per se, nothing like with a C compiler. It's all very pedestrian; the syntax tree is baked down across multiple passes, with macros acting as a glorified copy-paste system.
Thanks for the interest and the links, every one of those linked projects is new to me.
[0] https://wiki.xxiivv.com/site/uxn.html
[1] https://benbridle.com/articles/torque-programming-the-trs-80...
At first I thought it was useless: "but each ISA will still end up having different effective syntax because the underlying macro systems will not be designed the same".
But then I reread it and realised I was not paying attention to the usecase. It's about making it easy to write assemblers. So this isn't for your Arms and RISC-Vs it's for your random niche microcontrollers where the vendor-provided toolchain kinda sucks.
Seems cool!
I've experienced a couple of under-documented assemblers in my time. In neither case did this turn out to be that much of a problem in practice, but I guess I just don't write that much assembly.
I'm reminded of a 2016 [talk][1] where Rob Pike describes the common-denominator assembly language that the Go compiler generates. Then that assembly is translated into machine-specific code via table lookups. See the 11 minute mark.
[1]: https://www.youtube.com/watch?v=KINIAgRpkDA
Reminds me of TDASM from the turn of the century: https://web.archive.org/web/20230906054935/http://www.pengui...
I remember there were a few other meta-assemblers I came across in the 80s-90s, so this is definitely not "unchartered territory", but it's good to see another one show up.
Of course, in the other direction there are meta-disassemblers used for analysis in tools like Ghidra.
Any time I read about Z80, my mind wants to try to run something on a GameBoy emulator.
Pretty cool project!
Is there any support for non-Unicode text? Is there any support for octal numbers? These things should be corrected, hopefully.
Also, some of the links in the table of contents of the documentation does not seems to work.
With character sets, I was initially going to support non-Unicode text by adding a --char-set flag to the assembler, but I decided that the character set should be defined somehow inside each program. My thought was that they could be defined as large table-like macros, something like the following:
This is, admittedly, quite unweildy for character sets exceeding a few hundred characters, but it would work passably for small character sets like those used for HD44780-style LCD screens. What character sets did you have in mind?Octal was another feature I couldn't make up my mind about, just because I wasn't familiar with any architectures that require it. It'll be trivial to tack on though. For the Z80 instruction set, since the instruction encoding tends to cleave along octal lines, I used the following macro to pack octal digits into bytes, which has the advantage of allowing variables to be passed into each digit (the ADDr macro shows how it's used):
Thanks for the heads-up about the table of contents, the links should all work now.I think the "0" prefix for octal (as used in the C programming language) is not so good and that "0o" is better; so, if octal literals are implemented then "0o" is better.
For non-Unicode text, probably the simplest thing would be to treat the input as a sequence of bytes instead of Unicode characters; or equivalently to treat it as ISO-8859-1 (although programming it to use ISO-8859-1 may be less efficient then just using bytes, possibly; I don't know much about the working of Rust programming, so I don't actually know if it is or not).
By "non-Unicode text", I did not mean character mapping, although character mapping is another feature that would be useful to implement, similar to what you mentioned although it could be made more efficient (like you mention). Some way to map a input sequence of bytes (whether or not it is valid UTF-8) to a output character code, would work, probably.
I wholeheartedly agree with using "0o" as the octal prefix, I've never been a fan of "0". I've jut implemented this feature and released it in v2.2.0, you can grab it from the project page. Thanks for the suggestion!
I'm not too sure I understand what you're describing with non-Unicode text. Torque doesn't have a built-in concept of bytes, instead each character is treated as an integer with the value being the Unicode code point of that character (using decimal we have 65 for 'A', 955 for 'λ', 129302 for 'robot emoji', etc). It's up to the programmer to choose how to pack the character (integer) into a sequence of bits. Code points are different to encodings like UTF-8 or UTF-16, which define how a code point (integer) is packed down into one or more bytes.
If you want to assemble 7-bit ASCII text, one byte per character, you define a macro that packs each character value into the lower 7 bits of an 8-bit byte. If the string contains a non-ASCII character, the character value will be too large to fit into the field and an error will be displayed.
Assembling ISO-8859-1 text would be similar, but would involve remapping the characters above 0x7F like this: UTF-8 being a variable-width encoding requires a more complicated macro arrangement (you can see an example here [0]). But the key point is that strings aren't treated as byte sequences, they're just character/integer sequences until they get baked down into a byte encoding.Please let me know if that doesn't answer your suggestion, I'm keen to understand your use-case.
[0] https://benbridle.com/projects/torque/manual-v2.2.0.html#usi...
I meant that the input should not have to be Unicode. The input could be any extended ASCII encoding (supporting ASCII is necessary, in order that the syntax will work; therefore some encodings such as Shift-JIS and UTF-16 are not suitable) which does not have to be mapped to Unicode. (Even if the input actually is UTF-8, you may have a character that is made up of multiple code points. If it is treated as a sequence of bytes then it can be any sequence of bytes, including a combination of multiple UTF-8 code points.)
It is another issue than mapping the character codes for output, which as you say can be unwieldly for large character sets, so another kind of table specification (which might also be useful for stuff other than text), would probably help and be more efficient than using a sequence of conditions in a macro.
About octal, someone said against using "0o"; it is usually not as difficult to distinguish if the "o" is always lowercase. Another alternative would be how it works in PostScript, which uses "8#" prefix for octal and "16#" prefix for hexadecimal. (My opinion is that "0o" is good enough though)
Oh I think I see what you mean now, you're talking about the encoding of the Torque source code that gets fed into the assembler. To be honest I'd never really considered anything other than UTF-8, the parsing is all implemented in Rust which requires strings to be valid UTF-8 anyway. Are you wanting to write Torque code with a non-Unicode text editor, or are you thinking about how the document encoding affects the handling of string literals inside the assembler?
Some kind of table syntax would be useful for character mappings, but I'm not sure what it'd look like or if it'd be applicable outside of dealing with characters. I'll think more on that.
> I wholeheartedly agree with using "0o" as the octal prefix
Can I give a vote against this? Distinguishing "o" and "0" can be a huge pain when using the wrong font.
Also, why not 0c? 0x already uses the "x" in "hexadecimal", so why not the "c" in "octal". That it also reads a bit like a 13375p34k abbreviation of "octal" is a nice bonus.
Excellent work man! I know some low level guys who would really appreciate this.
Thanks, that's great to hear! What platforms do they develop for?
Well done! I write quite a bit of z80 for msx and this seems a nice addition.
This looks very cool, maybe I'll try this for risc-v if I have some time. I guess you could also use this to write RAW images, or other binary data.
Nice!
Would be interesting to target the RISC CPU of https://www.projectoberon.net with it.
Need to address this point:
"Assemblers tend to be poorly documented"
I wish everything in programming was as good documented as assemblers and ISAs.
I should clarify, I was thinking of microcontroller toolchains when I wrote this. The documentation I've seen for Intel processors is solid.
Not to mention ARM64...
And 6502 ;-) (and probably most of the ancient ones)
I shared Torque elsewhere and people asked what license it has
Thanks for the interest! Torque is public domain, feel free to use it or change it or share it however you want. I'd love to hear how people use it, but no obligation.
If you shared it someplace public I'd love to pop by, if you have a link.
I shared it on the catlang discord, Devine immediately replied "wow, that looks so much like uxn 0_o" hahaha. It's not the most active server but the response has been very positive! A few people thought it would probably be fun to use it to build a custom IR as part of a compiler pipeline, that was the context of the question.
I personally really love the utf8 and the "z80 for loop" examples, those are beautiful and make me nostalgic for the days I was writing stuff for the TI-83+ in Z80 assembly.
I've since been informed that public domain isn't really a proper term, legally speaking, so to clarify things I've released Torque under the MIT license.