As a gamedev i love it, if you bet on JS for games and need "native" packaging then the platform runtimes has been quite a bit hit-or-miss but with V8 as default (or JSC) then it's sane runtimes with a good WebGPU backend (same as in the browsers).
Only thing "missing" is perhaps a companion builder to emulate the C++ API with a WKWebView bundling setup for iOS.
For those reading, if Apple still disallows JIT:ed code, then a WKWebView might still be the best option in terms of pure JS simulation performance even if the WebView might steal some GPU perf.
What's the story/idea on controls (thin DOM emulation for pointerevents,keyboard,etc?), accelerometers, input-fields and fonts.
As much as I like controlling what I render, having good support for font/input-handling and system inputs is a clear plus of using web tech.
Hi, thanks! Yeah for controls I'm emulating pointerevents and keydown, keyup from SDL3 inputs & events. The goal is that the same JS that you write for a browser should "just work". It's still very alpha, but I was able to get my own WebGPU game engine running in it & have a sponza example that uses the exact key and pointer events to handle WASD / mouse controls: https://mystraldev.itch.io/sponza-in-webgpu-mystral-engine (the web build there is older, but the downloads for Windows, Mac, and Linux are using Mystral Native - you can clearly tell that it's not Electron by size (even Tauri for Mac didn't support webp inside of the WebGPU context so I couldn't use draco compressed assets w/ webp textures).
I put up a roadmap to get Three.js and Pixi 8 (webgpu renderer) fully working as part of a 1.0.0 release, but there's nothing that my JS engine is doing that is that different than Three.js or Pixi. https://github.com/mystralengine/mystralnative/issues/7
I did have to get Skia for Canvas2d support because I was using it for UI elements inside of the canvas, so right now it's a WebGPU + Canvas2d runtime. Debating if I should also add ANGLE and WebGL bindings as well in v2.0.0 to support a lot of other use cases too. Fonts support is built in as part of the Skia support as well, so that is also covered. WebAudio is another thing that is currently supported, but may need more testing to be fully compatible.
Followup comment about Apple disallowing JIT - will need to confirm if JSC is allowed to JIT or only inside of a webview. I was able to get JSC + wgpu-native rendering in an iOS build, but would need to confirm if it can pass app review.
There's 2 other performance things that you can do by controlling the runtime though - add special perf methods (which I did for draco decoding - there is currently one __mystralNativeDecodeDracoAsync API that is non standard), but the docs clearly lay out that you should feature gate it if you're going to use it so you don't break web builds: https://mystralengine.github.io/mystralnative/docs/api/nativ...
The other thing is more experimental - writing an AOT compiler for a subset of Typescript to convert it into C++ then just compile your code ("MystralScript") - this would be similar to Unity's C# AOT compiler and kinda be it's own separate project, but there is some prior work with porffor, AssemblyScript, and Static Hermes here, so it's not completely just a research project.
Is AssemblyScript good for games though? last I checked it lacks too much features for game-code coming directly from TS but might be better now? No idea how well static hermes behaves today (but probably far better due to RN heritage).
I've been down the TS->C++ road a few times myself and the big issue often comes up with how "strict" you can keep your TS code for real-life games as well as how slow/messy the official TS compiler has been (and real-life taking time from efforts).
It's better now, but I think one should probably directly target the GO port of the TS compiler (both for performance and go being a slightly stricter language probably better suited for compilers).
I guess, the point is that the TS->C++ compilation thing is potentially a rabbit-hole, theoretically not too bad, but TS has moved quickly and been hard to keep up with without using the official compiler, and even then a "game-oriented" typescript mode wants to have a slightly different semantic model from the official one so you need either a mapping over the regular type-inference engine, a separate on or a parallell one.
Mapping regular TS to "game-variants", the biggest issue is how to handle numbers efficiently, even if you go full-double there is a need to have conversion-point checking everywhere doubles go into unions with any other type (meaning you need boxing or a "fatter" union struct). And that's not even accounting for any vector-type accelerations.
AssemblyScript was just mentioned as some prior work, I don't think that AssemblyScript would work as is for games.
I realize the major issues with TS->C++ though (or any language to C++, Facebook has prior work converting php to C++ https://en.wikipedia.org/wiki/HipHop_for_PHP that was eventually deprecated in favor of HHVM). I think that iteratively improving the JS engine (Mystral.js the one that is not open source yet but is why MystralNative exists) to work with the compiler would be the first step and ensuring that games and examples built on top with a subset of TS is a starting point here. I don't think that the goal for MystralScript should be to support Three.js or any other engine to begin with as that would end up going down the same compatibility pits that hiphop did.
Being able to update the entire stack here is actually very useful - in theory parts of mystral.js could just be embedded into mystralnative (separate build flags, probably not a standard build) avoiding any TS->C++ compilation for core engine work & then ensuring that games built on top are using the strict subset of TS that does work well with the AOT compilation system. One option for numbers is actually using comment annotations (similar to how JSDoc types work for typescript compiler, specifically using annotations in comments to make sure that the web builds don't change).
Re: TS compiler - I do have some basics started here and I am already seeing that tests are pretty slow. I don't think that the tsgo compiler has a similar API though for parsing & emitters right now, so as much as I would like to switch to it (I have for my web projects & the speed is awesome), I don't think I can yet until the API work is clarified: https://github.com/microsoft/typescript-go/discussions/455
This is so interesting!! I had a very similar idea!
I would love to see if you could port games made with phaser or three js to this engine (though I think anything that involves the DOM is a no go sadly)
So I am stubbing parts of the DOM api (input handling like keydown, pointer events, etc.), so you shouldn't need to rewrite any of that.
Three.js and Pixi 8 with the WebGPU renderer are part of the v1.0.0 roadmap (verifying that they can work correctly on all platforms), right now most of the testing was done against my own engine (tentatively called mystral.js which will also be open sourced as part of v1.0.0, it's already used for some of the examples, just as a minified bundle): https://github.com/mystralengine/mystralnative/issues/7
Love it. Double bonus points if we could somehow add hardware ray tracing, which is missing from the browser WebGPU implementations. I'll definitely give this a shot with my 3D interactive WebGPU chessboard. A desktop implementation opens up the possibility of publishing on steam and other services which all expect a desktop app.
So in theory it should be possible, but it might require customizing the Dawn or wgpu-native builds if they don't support it (this is providing the JS bindings / wrapper around those two implementations of wgpu.h). But I've already added a special C++ method to handle draco compression natively, adding some mystral native only methods is not out of the question (however, I would want to ensure that usage of those via JS is always feature flagged so that it doesn't break when run on web).
Did you write your WebGPU chessboard using the raw JS APIs? Ideally it should work, but I just fixed up some missing APIs to get Three.js working in v0.1.0, so if there are issues, then please open up an issue on github - will try to get it working so we close any gaps.
Here's a dawn implementation with support for ray tracing that was implemented a number of years ago but never integrated into browsers. Perhaps it will help?
Yes, chessboard3d.app is written with raw JS APIs and raw WebGPU. It does use the rapier physics library, which uses WASM, which might be an issue? It implements its own ray tracing but would probably run 10x faster with hardware ray tracing support.
I think you'd get a lot of attention if you had hardware ray tracing, since that's only currently available in DirectX 12 and Vulkan, requiring implementation in native desktop platforms. FWIW, if the path looks feasible, I would be interested in contributing.
WASM shouldn't be an issue since the draco decoder uses it - but it may only work with V8 (for quickjs builds it wouldn't work, but the default builds use V8+dawn). Obviously with an alpha runtime, there may be bugs.
I think it would be super cool to have some sort of extension before WebGPU (web) has it. I was taking a look at the prior example & it seems like there's good ongoing discussion linked here about it: https://github.com/gpuweb/gpuweb/issues/535. Also I believe that Metal has hardware ray tracing support now too?
Re: Implementation, a few options exist - a separate Dawn fork with RT is one path (though Dawn builds are slow, 1-2 hours on CI). Another approach would be exposing custom native bindings directly from MystralNative alongside the WebGPU APIs - that might make iteration much faster for testing feasibility. The JS API would need to be feature-flagged so the same code gracefully falls back when running on web (did this for a native draco impl too that avoids having to load wasm: https://mystralengine.github.io/mystralnative/docs/api/nativ...).
Very interesting. It would be great if this works with a "higher" level library like Pixi.js or Phaser.js, that way one could build with ease in JS/TS, using a rich library/ecosystem and still distribute a "native" app (it probably already works with these libs but not sure).
Phaser is not supported right now because phaser is still using a WebGL renderer from my understanding (maybe in a v2.0.0 adding ANGLE + WebGL support is an option, but debating if that's a good idea or not).
I remember reading about Ejecta a long time ago! I had completely forgotten about it, but it is similar! The funny thing is to support UI elements, I had to also support canvas2d through Skia (although not 100% yet), so maybe impact could even work at some point (would require extensive testing obviously).
As a gamedev i love it, if you bet on JS for games and need "native" packaging then the platform runtimes has been quite a bit hit-or-miss but with V8 as default (or JSC) then it's sane runtimes with a good WebGPU backend (same as in the browsers).
Only thing "missing" is perhaps a companion builder to emulate the C++ API with a WKWebView bundling setup for iOS.
For those reading, if Apple still disallows JIT:ed code, then a WKWebView might still be the best option in terms of pure JS simulation performance even if the WebView might steal some GPU perf.
What's the story/idea on controls (thin DOM emulation for pointerevents,keyboard,etc?), accelerometers, input-fields and fonts.
As much as I like controlling what I render, having good support for font/input-handling and system inputs is a clear plus of using web tech.
Hi, thanks! Yeah for controls I'm emulating pointerevents and keydown, keyup from SDL3 inputs & events. The goal is that the same JS that you write for a browser should "just work". It's still very alpha, but I was able to get my own WebGPU game engine running in it & have a sponza example that uses the exact key and pointer events to handle WASD / mouse controls: https://mystraldev.itch.io/sponza-in-webgpu-mystral-engine (the web build there is older, but the downloads for Windows, Mac, and Linux are using Mystral Native - you can clearly tell that it's not Electron by size (even Tauri for Mac didn't support webp inside of the WebGPU context so I couldn't use draco compressed assets w/ webp textures).
I put up a roadmap to get Three.js and Pixi 8 (webgpu renderer) fully working as part of a 1.0.0 release, but there's nothing that my JS engine is doing that is that different than Three.js or Pixi. https://github.com/mystralengine/mystralnative/issues/7
I did have to get Skia for Canvas2d support because I was using it for UI elements inside of the canvas, so right now it's a WebGPU + Canvas2d runtime. Debating if I should also add ANGLE and WebGL bindings as well in v2.0.0 to support a lot of other use cases too. Fonts support is built in as part of the Skia support as well, so that is also covered. WebAudio is another thing that is currently supported, but may need more testing to be fully compatible.
Followup comment about Apple disallowing JIT - will need to confirm if JSC is allowed to JIT or only inside of a webview. I was able to get JSC + wgpu-native rendering in an iOS build, but would need to confirm if it can pass app review.
There's 2 other performance things that you can do by controlling the runtime though - add special perf methods (which I did for draco decoding - there is currently one __mystralNativeDecodeDracoAsync API that is non standard), but the docs clearly lay out that you should feature gate it if you're going to use it so you don't break web builds: https://mystralengine.github.io/mystralnative/docs/api/nativ...
The other thing is more experimental - writing an AOT compiler for a subset of Typescript to convert it into C++ then just compile your code ("MystralScript") - this would be similar to Unity's C# AOT compiler and kinda be it's own separate project, but there is some prior work with porffor, AssemblyScript, and Static Hermes here, so it's not completely just a research project.
Is AssemblyScript good for games though? last I checked it lacks too much features for game-code coming directly from TS but might be better now? No idea how well static hermes behaves today (but probably far better due to RN heritage).
I've been down the TS->C++ road a few times myself and the big issue often comes up with how "strict" you can keep your TS code for real-life games as well as how slow/messy the official TS compiler has been (and real-life taking time from efforts).
It's better now, but I think one should probably directly target the GO port of the TS compiler (both for performance and go being a slightly stricter language probably better suited for compilers).
I guess, the point is that the TS->C++ compilation thing is potentially a rabbit-hole, theoretically not too bad, but TS has moved quickly and been hard to keep up with without using the official compiler, and even then a "game-oriented" typescript mode wants to have a slightly different semantic model from the official one so you need either a mapping over the regular type-inference engine, a separate on or a parallell one.
Mapping regular TS to "game-variants", the biggest issue is how to handle numbers efficiently, even if you go full-double there is a need to have conversion-point checking everywhere doubles go into unions with any other type (meaning you need boxing or a "fatter" union struct). And that's not even accounting for any vector-type accelerations.
AssemblyScript was just mentioned as some prior work, I don't think that AssemblyScript would work as is for games.
I realize the major issues with TS->C++ though (or any language to C++, Facebook has prior work converting php to C++ https://en.wikipedia.org/wiki/HipHop_for_PHP that was eventually deprecated in favor of HHVM). I think that iteratively improving the JS engine (Mystral.js the one that is not open source yet but is why MystralNative exists) to work with the compiler would be the first step and ensuring that games and examples built on top with a subset of TS is a starting point here. I don't think that the goal for MystralScript should be to support Three.js or any other engine to begin with as that would end up going down the same compatibility pits that hiphop did.
Being able to update the entire stack here is actually very useful - in theory parts of mystral.js could just be embedded into mystralnative (separate build flags, probably not a standard build) avoiding any TS->C++ compilation for core engine work & then ensuring that games built on top are using the strict subset of TS that does work well with the AOT compilation system. One option for numbers is actually using comment annotations (similar to how JSDoc types work for typescript compiler, specifically using annotations in comments to make sure that the web builds don't change).
Re: TS compiler - I do have some basics started here and I am already seeing that tests are pretty slow. I don't think that the tsgo compiler has a similar API though for parsing & emitters right now, so as much as I would like to switch to it (I have for my web projects & the speed is awesome), I don't think I can yet until the API work is clarified: https://github.com/microsoft/typescript-go/discussions/455
This is so interesting!! I had a very similar idea! I would love to see if you could port games made with phaser or three js to this engine (though I think anything that involves the DOM is a no go sadly)
So I am stubbing parts of the DOM api (input handling like keydown, pointer events, etc.), so you shouldn't need to rewrite any of that.
Three.js and Pixi 8 with the WebGPU renderer are part of the v1.0.0 roadmap (verifying that they can work correctly on all platforms), right now most of the testing was done against my own engine (tentatively called mystral.js which will also be open sourced as part of v1.0.0, it's already used for some of the examples, just as a minified bundle): https://github.com/mystralengine/mystralnative/issues/7
For any thin parts that needs the DOM one could make JS stubs that mimic behaviour.
Love it. Double bonus points if we could somehow add hardware ray tracing, which is missing from the browser WebGPU implementations. I'll definitely give this a shot with my 3D interactive WebGPU chessboard. A desktop implementation opens up the possibility of publishing on steam and other services which all expect a desktop app.
So in theory it should be possible, but it might require customizing the Dawn or wgpu-native builds if they don't support it (this is providing the JS bindings / wrapper around those two implementations of wgpu.h). But I've already added a special C++ method to handle draco compression natively, adding some mystral native only methods is not out of the question (however, I would want to ensure that usage of those via JS is always feature flagged so that it doesn't break when run on web).
Did you write your WebGPU chessboard using the raw JS APIs? Ideally it should work, but I just fixed up some missing APIs to get Three.js working in v0.1.0, so if there are issues, then please open up an issue on github - will try to get it working so we close any gaps.
Here's a dawn implementation with support for ray tracing that was implemented a number of years ago but never integrated into browsers. Perhaps it will help?
https://github.com/maierfelix/dawn-ray-tracing
Yes, chessboard3d.app is written with raw JS APIs and raw WebGPU. It does use the rapier physics library, which uses WASM, which might be an issue? It implements its own ray tracing but would probably run 10x faster with hardware ray tracing support.
I think you'd get a lot of attention if you had hardware ray tracing, since that's only currently available in DirectX 12 and Vulkan, requiring implementation in native desktop platforms. FWIW, if the path looks feasible, I would be interested in contributing.
WASM shouldn't be an issue since the draco decoder uses it - but it may only work with V8 (for quickjs builds it wouldn't work, but the default builds use V8+dawn). Obviously with an alpha runtime, there may be bugs.
I think it would be super cool to have some sort of extension before WebGPU (web) has it. I was taking a look at the prior example & it seems like there's good ongoing discussion linked here about it: https://github.com/gpuweb/gpuweb/issues/535. Also I believe that Metal has hardware ray tracing support now too?
Re: Implementation, a few options exist - a separate Dawn fork with RT is one path (though Dawn builds are slow, 1-2 hours on CI). Another approach would be exposing custom native bindings directly from MystralNative alongside the WebGPU APIs - that might make iteration much faster for testing feasibility. The JS API would need to be feature-flagged so the same code gracefully falls back when running on web (did this for a native draco impl too that avoids having to load wasm: https://mystralengine.github.io/mystralnative/docs/api/nativ...).
Great project! I also had similar thoughts when I saw ability to make WebGPU calls in deno. I wonder how performant could games get on this runtime
Very interesting. It would be great if this works with a "higher" level library like Pixi.js or Phaser.js, that way one could build with ease in JS/TS, using a rich library/ecosystem and still distribute a "native" app (it probably already works with these libs but not sure).
Phaser is not supported right now because phaser is still using a WebGL renderer from my understanding (maybe in a v2.0.0 adding ANGLE + WebGL support is an option, but debating if that's a good idea or not).
Pixi 8 has a WebGPU renderer so that should be supported as part of a v1.0.0 release - it's on the roadmap to verify that three and pixi 8 work correctly: https://github.com/mystralengine/mystralnative/issues/7
Very cool! This reminds me of Ejecta, which was something like this for 2D games on iOS, a very long time ago: https://impactjs.com/ejecta
I remember reading about Ejecta a long time ago! I had completely forgotten about it, but it is similar! The funny thing is to support UI elements, I had to also support canvas2d through Skia (although not 100% yet), so maybe impact could even work at some point (would require extensive testing obviously).
Cool project and very clear explanation for the motivation kudos!