It's time for me to re-read the man page for bash. I was not aware of BASH_REMATCH, wow. It's in the first snippet on the linked page, and would save the hassle of using multiple var expansions of the %% and ## et al sort.
Bash is slower than other POSIX compatible shells but once you start running external commands for any substring or replace operation you loose much of this performance edge since forking is comparable slow.
One reason why I personally prefer to use Bashisms like ${x//str/replace} or [[ $value =~ $pattern ]] instead of doing the common x=$(echo $x | sed s/str/replace/) which has to launch two processes just for the sake of avoiding Bashism. (or grep -oP ... which is nice but a BASH_REMATCH is often simpler)
If the script does a lot of unstreamable replacements, you're right. But there are still ways out of bash.
I prefer no fork, no bashism, reusable functions:
set -euf
replace () {
local t=
REPLY=$1
t="${REPLY#*"$2"}"
test "$t" != "$REPLY" || return
REPLY="${REPLY%"$2$t"}$3$t"
}
replace_all () {
REPLY=$1
shift
while replace "$REPLY" "$@"; do :; done
}
input="foo bar foo baz"
replace_all "$input" "foo" "HELLO"
echo $REPLY
Not exactly easy to write, but now that they're functions, it doesn't matter since I can reuse them.
Regarding performance, it is slower than bash, but not significantly.
Times for 1000 calls:
9.908s -- sed (one fork per replacement)
0.015s -- bash x//str/replace
0.088s -- sh replace_all
Times for 50.000 calls:
0.351s -- bash x//str/replace
0.631s -- sh replace_all
Also, you can get further performance by inlining replace inside replace_all instead of making one call another.
Note that I could have done several replacements inside a single sed pipe, but I decided to count the performance for doing it like you suggested x=$(echo $x | sed s/str/replace/). The same goes for my functions, one invokation per replacement (in fact, they are tuned for that scenario).
sed can absoltely beat the shell in scenarios where you can make one fork do lots and lots of replacements. It depends on the scenario, and how proficient you are in writing sed (which can do branching, keep state, all sorts of things portably).
--
From an architectural point of view, it makes sense to have a simpler `sh` and keep a sort of standard library of functions, instead of feature creeping the interpreter with weird arcana. It makes shells like dash easier to maintain, easier to debug, easier to port and safer.
The reason Bash has so many features is that doing these things natively in the shell is faster and more convenient. After all, these features weren't just added randomly.
These features were added slowly, randomly, as time passed. The weird syntaxes for all of these are a clear sign of this.
Practically all shell interpreters suffer from decades of feature creep since the original bourne shell. They're full of weird arcana, hard to maintain and debug.
Many people tried to replace bash and died on the hill because of these weird features, or ended up creating a replacement that is even slower, or ended up rediscovering what perl is.
Oh yeah! I was unaware too! Nowadays I quickly jump to python instead of using Bash even for the simplest of scripts , but this could help creating tiny and easy to understand scripts for some integrations...
> I quickly jump to python instead of using Bash even for the simplest of scripts
You don't seem to respect the old, venerable, well-tested adage: "once your shell script becomes too complex, switch to a real programming language like python".
Or, the zen version (formally equivalent, but with quite a different tone): "once your program becomes sufficiently simple, turn it into a beautiful shell script".
The true power of shell script is to coordinate programs. Once you find yourself altering data with the shell constructs, that's the sign to use $LISP instead.
> The true power of shell script is to coordinate programs. Once you find yourself altering data with the shell constructs, that's the sign to use $LISP instead.
one might awk how much logic one can bash into a script before leaving the beloved shell
The hate for the HomeAssistant OS is misplaced. It's just an appliance that manages its own software packages. Like for example the author could have set up Zigbee2MQTT entirely through the web GUI.
Just to be clear, the one paragraph that indirectly references HA was not intended as "hate". It was intended solely as gentle mockery and a bit of humour. I think it should be very clear from the article that my Mosquitto + Z2M + mqttr + bash + jq solution, while feature-complete for my purposes, is not a competitor to HA, which I have never used or tried.
Why have I never tried HA? To run it, I would need to either create a VM or buy dedicated hardware. First, I don't want any more dedicated hardware in my house. Second, I like to keep the number of hosts I need to secure, manage, and keep running to a minimum. Third, from what I've heard (from people insisting it can't run on cheaper SBCs), it's not light; I was genuinely concerned about the server load on my single, low-power home server which already handles other things.
I imagine that for many people, Home Assistant works great: the UI is what they like, it never breaks, and they never have to troubleshoot it. Or, if it does break, they're satisfied with just backing up their configuration and reinstalling it.
But for me, when it comes to things I set up in my own time for my own use, I like to understand everything I am managing so I can quickly understand any failures. If I am going to use a ZBMINIR2 in detached mode to have my smart lights effectively on 100% of the time with the only method of control being Zigbee + glue, the glue had better be: a. rock-solid, b. trivial to troubleshoot, and c. easy to fix. I am not confident that I could learn, and retain, enough knowledge about HA to be able to achieve those three goals.
Lastly, I have extensive experience of DIYing my own solutions to problems for which working off-the-shelf solutions exist. This is not because I am a contrarian who hates everything that works, although maybe there's a tiny bit of that. The reason I didn't use HA wasn't because I hate it; I didn't use it because I wanted the pleasure of solving that problem myself.
First, thank you for the article, this is exactly what I need.
As for Home Assistant, I share your sentiment. I run my own stuff because I want to understand my infrastructure (vs. a black box from Philips) and to minimize the amount of code running around.
I'm sure HA is a better opaque box than the commercial ones for many people, but running it would still leave me with the same feeling of having outsourced critical knowledge - plus the compute costs.
HomeAssistant's dependency requirements are so complex as to make a deployment of it essentially read-only. Trying to mod it with HACS is so hard that I gave up even though I write python for a living. I can't recommend it to anyone that doesn't know how to use linux at an expert level. This detracts from it's mission IMO; I would like to tell my family to use it instead of Google Home or Amazon whatever but it is so fragile that I can't do that in good conscience.
My install has been essentially maintenance free for years. I don't imagine many casual users that just want an app/gui for their smart outlets will be reaching for HACS. HA even warns you when you install HACS that you're voiding all the warranties.
HACS is very easy _if_ you’re using the HA OS installation method, in which Home Assistant manages its own containers (requires bare metal or a dedicated VM). I’d have no hesitation recommending it to someone who is just tech savvy enough to operate Home Assistant itself (e.g. setting up automations).
I’ll also echo the sibling commenter re: running it for years with no maintenance besides occasionally installing the updates.
It's really not though; I went with it for a while then it did this weird diverging thing with updates and full versions that I still don't fully understand because I just got fed up with it.
That aside, I'm currently contemplating whether running TrueNAS Scale for web apps is really necessary. I like the fact that I only need to click a few buttons or fill in a few fields, but on the otherhand my setup can be kept inside a docker-compose.yml
We'll never know, since I like the safety of running my apps through something like TrueNAS Scale. It feels professional
Interesting... I didn't know about this, although I was pretty satisfied with the simplicity of MqttDroid (after I fixed a bug: https://github.com/LightJockey/MqttDroid/pull/3). Will investigate to compare.
You don't need to tell me, I've read enough "bash" written by people who "know bash" in the wild to know that my house lights running on bash is (worryingly) probably some of the most sensible and least fragile use of bash on the planet.
It's time for me to re-read the man page for bash. I was not aware of BASH_REMATCH, wow. It's in the first snippet on the linked page, and would save the hassle of using multiple var expansions of the %% and ## et al sort.
Yes, but also:
https://wiki.ubuntu.com/DashAsBinSh#Why_was_this_change_made...
> [bash] is rather large and slow to start up and operate by comparison with dash
For more complex regular expressions, you can use `sed`.
--
It's all a matter of context. Sometimes simple ## and %% param substitutions are the best tool for the job.
I think bash is a fantastic interactive shell and a lousy script runner.
Bash is slower than other POSIX compatible shells but once you start running external commands for any substring or replace operation you loose much of this performance edge since forking is comparable slow.
One reason why I personally prefer to use Bashisms like ${x//str/replace} or [[ $value =~ $pattern ]] instead of doing the common x=$(echo $x | sed s/str/replace/) which has to launch two processes just for the sake of avoiding Bashism. (or grep -oP ... which is nice but a BASH_REMATCH is often simpler)
If the script does a lot of unstreamable replacements, you're right. But there are still ways out of bash.
I prefer no fork, no bashism, reusable functions:
Not exactly easy to write, but now that they're functions, it doesn't matter since I can reuse them.Regarding performance, it is slower than bash, but not significantly.
Times for 1000 calls:
Times for 50.000 calls: Also, you can get further performance by inlining replace inside replace_all instead of making one call another.Note that I could have done several replacements inside a single sed pipe, but I decided to count the performance for doing it like you suggested x=$(echo $x | sed s/str/replace/). The same goes for my functions, one invokation per replacement (in fact, they are tuned for that scenario).
sed can absoltely beat the shell in scenarios where you can make one fork do lots and lots of replacements. It depends on the scenario, and how proficient you are in writing sed (which can do branching, keep state, all sorts of things portably).
--
From an architectural point of view, it makes sense to have a simpler `sh` and keep a sort of standard library of functions, instead of feature creeping the interpreter with weird arcana. It makes shells like dash easier to maintain, easier to debug, easier to port and safer.
The reason Bash has so many features is that doing these things natively in the shell is faster and more convenient. After all, these features weren't just added randomly.
These features were added slowly, randomly, as time passed. The weird syntaxes for all of these are a clear sign of this.
Practically all shell interpreters suffer from decades of feature creep since the original bourne shell. They're full of weird arcana, hard to maintain and debug.
Many people tried to replace bash and died on the hill because of these weird features, or ended up creating a replacement that is even slower, or ended up rediscovering what perl is.
Oh yeah! I was unaware too! Nowadays I quickly jump to python instead of using Bash even for the simplest of scripts , but this could help creating tiny and easy to understand scripts for some integrations...
> I quickly jump to python instead of using Bash even for the simplest of scripts
You don't seem to respect the old, venerable, well-tested adage: "once your shell script becomes too complex, switch to a real programming language like python".
Or, the zen version (formally equivalent, but with quite a different tone): "once your program becomes sufficiently simple, turn it into a beautiful shell script".
The true power of shell script is to coordinate programs. Once you find yourself altering data with the shell constructs, that's the sign to use $LISP instead.
> The true power of shell script is to coordinate programs. Once you find yourself altering data with the shell constructs, that's the sign to use $LISP instead.
one might awk how much logic one can bash into a script before leaving the beloved shell
if it’s structured text, especially line based format. You can get in trouble when you want maps,2+D arrays, and variable length lists
[dead]
How do you actually read man pages? Do you skim them, to actually read and focus on interesting parts? Do you try to memorize features?
The hate for the HomeAssistant OS is misplaced. It's just an appliance that manages its own software packages. Like for example the author could have set up Zigbee2MQTT entirely through the web GUI.
Author here.
Just to be clear, the one paragraph that indirectly references HA was not intended as "hate". It was intended solely as gentle mockery and a bit of humour. I think it should be very clear from the article that my Mosquitto + Z2M + mqttr + bash + jq solution, while feature-complete for my purposes, is not a competitor to HA, which I have never used or tried.
Why have I never tried HA? To run it, I would need to either create a VM or buy dedicated hardware. First, I don't want any more dedicated hardware in my house. Second, I like to keep the number of hosts I need to secure, manage, and keep running to a minimum. Third, from what I've heard (from people insisting it can't run on cheaper SBCs), it's not light; I was genuinely concerned about the server load on my single, low-power home server which already handles other things.
I imagine that for many people, Home Assistant works great: the UI is what they like, it never breaks, and they never have to troubleshoot it. Or, if it does break, they're satisfied with just backing up their configuration and reinstalling it.
But for me, when it comes to things I set up in my own time for my own use, I like to understand everything I am managing so I can quickly understand any failures. If I am going to use a ZBMINIR2 in detached mode to have my smart lights effectively on 100% of the time with the only method of control being Zigbee + glue, the glue had better be: a. rock-solid, b. trivial to troubleshoot, and c. easy to fix. I am not confident that I could learn, and retain, enough knowledge about HA to be able to achieve those three goals.
Lastly, I have extensive experience of DIYing my own solutions to problems for which working off-the-shelf solutions exist. This is not because I am a contrarian who hates everything that works, although maybe there's a tiny bit of that. The reason I didn't use HA wasn't because I hate it; I didn't use it because I wanted the pleasure of solving that problem myself.
First, thank you for the article, this is exactly what I need.
As for Home Assistant, I share your sentiment. I run my own stuff because I want to understand my infrastructure (vs. a black box from Philips) and to minimize the amount of code running around.
I'm sure HA is a better opaque box than the commercial ones for many people, but running it would still leave me with the same feeling of having outsourced critical knowledge - plus the compute costs.
HomeAssistant's dependency requirements are so complex as to make a deployment of it essentially read-only. Trying to mod it with HACS is so hard that I gave up even though I write python for a living. I can't recommend it to anyone that doesn't know how to use linux at an expert level. This detracts from it's mission IMO; I would like to tell my family to use it instead of Google Home or Amazon whatever but it is so fragile that I can't do that in good conscience.
My install has been essentially maintenance free for years. I don't imagine many casual users that just want an app/gui for their smart outlets will be reaching for HACS. HA even warns you when you install HACS that you're voiding all the warranties.
HACS is very easy _if_ you’re using the HA OS installation method, in which Home Assistant manages its own containers (requires bare metal or a dedicated VM). I’d have no hesitation recommending it to someone who is just tech savvy enough to operate Home Assistant itself (e.g. setting up automations).
I’ll also echo the sibling commenter re: running it for years with no maintenance besides occasionally installing the updates.
It's really not though; I went with it for a while then it did this weird diverging thing with updates and full versions that I still don't fully understand because I just got fed up with it.
It really is the design and intent, though. What year was this? Things are much more stable than in the early days.
MQTT is a fantastic protocol.
I basically do the same as the author but use nutshell and its powerful pattern matching [0] as the router
-[0] https://www.nushell.sh/cookbook/pattern_matching.html#patter...
I respect both the DIY's and the HA fans.
That aside, I'm currently contemplating whether running TrueNAS Scale for web apps is really necessary. I like the fact that I only need to click a few buttons or fill in a few fields, but on the otherhand my setup can be kept inside a docker-compose.yml
We'll never know, since I like the safety of running my apps through something like TrueNAS Scale. It feels professional
For those using termux, termux widgets are exposed to Android device control so you don't need the MqttDroid app
Interesting... I didn't know about this, although I was pretty satisfied with the simplicity of MqttDroid (after I fixed a bug: https://github.com/LightJockey/MqttDroid/pull/3). Will investigate to compare.
> Bash as critical infrastructure? Am I alright? Do I need to see a psychiatrist?
Who’s gonna tell him
You don't need to tell me, I've read enough "bash" written by people who "know bash" in the wild to know that my house lights running on bash is (worryingly) probably some of the most sensible and least fragile use of bash on the planet.
people and their mazochistic ways