The CLI is written in Go. Interpreted/VM approaches (specifically Python/Java) felt more involved (particularly at install time) for what was intended to be a lightweight, easy to install utility. Being able to output executable binaries was really important, and the high level language looked nice (I'm new to Go).
There's no library (yet) - it's a CLI tool. My thinking is that a lot of the file management functionality has to talk to the file system (gigabyte files are technically doable, currently capped at 100MB while we work out the issues). The more I thought about adding multiple programming languages and flexible (stream to file, to string, to byte array) interfaces, the harder to use it looked; offloading concerns like memory usage and hash verification to the developer. With that said, there will be an API (I'm working on it now), and it'll have all the capabilities/interfaces of the CLI, but obviously be a roll-your-own logic situation. On the CLI, I really like how minimal the integration is - you write one file poller (examples in the docs) and you're basically done with the "get the data" problem and onto the more interesting "use the data" problem. It is a bit of an unusual approach though.
I'm considering adding a callback command to the CLI that triggers after each payload is written, that would make integration even simpler and help scripts trigger appropriately.
API
The clients (just the CLI at the moment) subscribe to a websocket to listen to files being published (one for each subscription). This is the one technical decision that took me the most time. Long poll / regular poll were both considered and discarded due to the fairly heavy impact it has on LB billing when scaled out, particularly for heavy clients with high frequency polls. Directly exposing a Kafka queue had its own issues, and would have pushed the architecture to be a shape I wasn't a fan of for a public service (less auth control, looser control of server balancing, client version tie-in, etc).
Downloads are via a signed link - this allows us to control access to the CDN and prevent us being flooded with requests from a given IP or account. I'm solo and self-funded so controlling spend is critical for us at the moment.
Validation
We do some basic checks for XML and JSON (e.g. XXE and other attacks) in all published files and we'll build up more checks and more data types over time. No file is visible to subscribers until it has passed validation.
What's planned
Priority 1 is on-boarding publishers so we have useful data on the site.
General polish is high on the list (anything UX or DX - CLI, API, site usability)
I have a few plans for different types of stream - high speed feeds being the main one.
Supporting emerging (schema) standards is a consideration too (e.g. a community authored schema repository, being able to search by schema for similar data types)
Some more technical details.
CLI
The CLI is written in Go. Interpreted/VM approaches (specifically Python/Java) felt more involved (particularly at install time) for what was intended to be a lightweight, easy to install utility. Being able to output executable binaries was really important, and the high level language looked nice (I'm new to Go).
There's no library (yet) - it's a CLI tool. My thinking is that a lot of the file management functionality has to talk to the file system (gigabyte files are technically doable, currently capped at 100MB while we work out the issues). The more I thought about adding multiple programming languages and flexible (stream to file, to string, to byte array) interfaces, the harder to use it looked; offloading concerns like memory usage and hash verification to the developer. With that said, there will be an API (I'm working on it now), and it'll have all the capabilities/interfaces of the CLI, but obviously be a roll-your-own logic situation. On the CLI, I really like how minimal the integration is - you write one file poller (examples in the docs) and you're basically done with the "get the data" problem and onto the more interesting "use the data" problem. It is a bit of an unusual approach though.
I'm considering adding a callback command to the CLI that triggers after each payload is written, that would make integration even simpler and help scripts trigger appropriately.
API
The clients (just the CLI at the moment) subscribe to a websocket to listen to files being published (one for each subscription). This is the one technical decision that took me the most time. Long poll / regular poll were both considered and discarded due to the fairly heavy impact it has on LB billing when scaled out, particularly for heavy clients with high frequency polls. Directly exposing a Kafka queue had its own issues, and would have pushed the architecture to be a shape I wasn't a fan of for a public service (less auth control, looser control of server balancing, client version tie-in, etc).
Downloads are via a signed link - this allows us to control access to the CDN and prevent us being flooded with requests from a given IP or account. I'm solo and self-funded so controlling spend is critical for us at the moment.
Validation
We do some basic checks for XML and JSON (e.g. XXE and other attacks) in all published files and we'll build up more checks and more data types over time. No file is visible to subscribers until it has passed validation.
What's planned
Priority 1 is on-boarding publishers so we have useful data on the site.
General polish is high on the list (anything UX or DX - CLI, API, site usability)
I have a few plans for different types of stream - high speed feeds being the main one.
Supporting emerging (schema) standards is a consideration too (e.g. a community authored schema repository, being able to search by schema for similar data types)