For everyone who's saying "if you need unit tests then you should rewrite it in a different language", you're not wrong but you're not right either.
Bash is a lowest-common-denominator language, available on almost all platforms, with a very stable (old?) API, which makes it ideal for broad distribution & bootstrapping of systems.
When you use another other languages, like Python, suddenly you're not only worrying about whether Python is installed, but also which version is available, and you still can't use anything that's not part of the standard library (because then you enter package management/dependency hell).
I agree that even Python with stdlib-only is still better than Bash, but I guarantee that you'll find Bash >=3.0 on every host.
> I agree that even Python with stdlib-only is still better than Bash, but I guarantee that you'll find Bash >=3.0 on every host.
Shouldn't you go for POSIX at that point? I'm not an expert, but I still have some idea how to test that my script is (somewhat) POSIX compliant (e.g. use shellcheck, run the script with dash or a shell in POSIX-compatible mode). I have Bash 5 on my Debian, how do I test that a script I tested with this version works on Bash 3 as well? Or have the changes between Bash 3 and Bash 5 been so minimal that I would have to do it on purpose to find an incompatibility?
I'm also curious about the number of machines where Bash scripts are run but don't have, say, Python 3.4 installed, and projects where it's a better trade-off to spend time developing and testing in Bash rather than have a Python 3 dependency. I'd say it's relatively rare.
Bash also has a -o posix option, which changes a number of things which may be significant, but I rarely see it used.
But none of those 3 solutions really check for POSIX compliance (including -o posix, since of course you can use many non-POSIX features when it's on). I am not aware that ShellCheck actually checks POSIX compliance; I am pretty sure it just checks for common errors in your bash/sh scripts (which is what it says on the home page).
I think what you may mean is "portable shell", i.e. portable between shell X and Y and system X and Y. That is a decent goal but many people use bash just because it alone is portable enough! The system tools are often unportable though. Limiting yourself to POSIX in that case is pretty painful and also virtually untestable. It's better to say "portable between X and Y" because I think that's what you mean.
In other words, there is plenty of stuff that's portable but not POSIX that you probably want to use. local variables are probably the biggest example. Every shell I know of supports those, including dash, but it's not POSIX (though maybe they're thinking about adding it; the spec is pretty behind)
> I am not aware that ShellCheck actually checks POSIX compliance; I am pretty sure it just checks for common errors in your bash/sh scripts
Bashisms are errors if you specify #!/bin/sh. If you want to try it out, you can load random examples on https://www.shellcheck.net/ until you get a #!/bin/sh. Then you'll get warnings such as "SC3010: In POSIX sh, [[ ]] is undefined."
To go back to the discussion: if you want to use Bash features, you're better off switching to Python.
OK interesting, yeah with #!/bin/sh it points out that 'local' is not POSIX.
But every shell I know of supports 'local'. It's not a "bash-ism' because it's universally supported.
I consider it essential; otherwise you might as well not use functions in shell.
This is a long argument (and is addressed in the same FAQ), but as someone who's written hundreds of thousands of lines of Python, I think shell is still better for a large set of tasks. A typical small project I write might have 300 lines of shell and 1000 lines of Python, rather than 5000 lines of Python.
But of course there are problems with using shell; if there weren't then Oil wouldn't exist :)
I had something like this in my last job: I had to create an installer for a VCS aimed at chip designer.
The installer would be ignored as much as my coworkers could manage, and as many customers as possible had to be able to just use it unmodified. Since our supported OSes were RHEL6, RHEL7, SLES12, and SLES15, the best option was bash. Also had to build it on the off chance one of our more restrictive customers was still on RHEL5.11
>Bash is a lowest-common-denominator language, available on almost all platforms, with a very stable (old?) API, which makes it ideal for broad distribution & bootstrapping of systems.
/bin/sh is, bash isn't. Portability is a good excuse for using /bin/sh, bash is just a crappy middle ground, it's not as portable as the POSIX shell and it's still not a decent substitution for a proper scripting language.
Portability is not important in most cases. Most shell scripts are written for and run on a single platform usually as setup or teardown for something else.
I used bash to write a testing framework for a project I have that needs minimal dependencies. E.g. I don't want to depend on having Python (and the transitive closure of its batteries-included experience) installed. For that purpose, bash works pretty well. I also want to run on some older machines, like 32-bit macs.
Turns out that bash does evolve...slowly. E.g. I found out that the default bash on MacOS 10.4 doesn't support the "=~" operator...oi...
What code are you writing that is pure Bash? Every Bash script I've written or find in the wild is always used to control subprocesses, and then you have to worry about compatibility of those programs, which are all over the place.
Python stdlib does way more than Bash, it's generally easier to just install Python and run a script that sticks with stdlib, than to install Bash and then figure out what subprocesses it invokes, what packages those map to in the OS package manager, what versions they are, if they're compatible with what you wrote, etc. And if you need more libs, Python also has a cross-platform package manager with a consistent interface. As an example, consider coreutils between macOS and Linux--macOS ships with dramatically different coreutils programs (such as "ls"). The only feasible way to get them consistent across OSes is to stuff everything in a Docker image, but that has its own problems and limitations. Or ask the user to figure it out for themselves, e.g. "use Homebrew".
> Yes, when working with sub-processes you need to account for platform differences...which is also true when Python.
The cases in which you even need to use subprocesses is much smaller with Python, because the stdlib replaces a huge amount of external programs typically used with Bash (cat, grep, awk, sed, cut, paste, wc, ls, find, etc.). And so if you stick with stdlib without subprocesses, you don't need to account for platform differences. Python has already done that work for you.
In simple cases, or cases where portability doesn't really matter, then I agree with you. But once you start caring about portability, you have to be careful to consider the compatibility differences of external programs across distributions and OSs on which Bash runs, which vary dramatically among even the simplest or most common of programs such as "ls" or "grep".
Bash scripts being portable has little to do with Bash itself, it's either deliberate by the programmer, or a happy coincidence. IME it's almost always the latter, if it's portable at all (which isn't uncommon in my work).
(Also wanted to add that this isn't a unique property to Python either. Any "real" language typically has these same properties.)
Totally agree. You can avoid lots of this by using Bash built-ins, but that's more advanced/esoteric Bash development that most people aren't familiar with.
Thankfully, with the official deprecation of Python 2.7 and the slow march of progress I hope to be able to better standardize on Python 3 + stdlib (or similar).
That is true, but most languages do work on all of those things, e.g. For parts of the D ecosystem we have moved over to build scripts actually written in D, and they almost always work even though the makefiles and bash scripts are sometimes completely fucked whenever I fiddle with my computer. The best part is with the scripts in a real programming language (Python is good enough too), you can actually read them and show how they work to people coming from visual studio who can't read bash or make
There are platforms where bash is not available. I wonder why people who want to run it on various platforms use bash.
ShellSpec (https://shellspec.info/) is a POSIX compliant testing framework supports all POSIX shells (Bash >= 2.0, dash, ksh88, zsh >=3.1, etc). I'm the author of ShellSpec.
Bash isn't installed default on OpenBSD or FreeBSD, and the bash on MacOS is a very old version. Also not installed by default on Alpine Linux, which would be fairly popular in the container world.
There’s clearly some good thought gone into this - i especially like the ergonomics of the “fake” command and arguments.
However, having been down this path myself, i don’t think it’s a great idea.
Beyond the hello world examples you start to run into the need to defang commands being tested or getting better visibility of what they’re doing just so you can make a useful assert, you end up LD_PRELOAD’ing shims to intercept calls and it gets a bit horrifying, very quickly.
I agree: this is great as an intellectual exercise but I’m reminded about how many times I’ve been glad that I set a policy in the 2000s that any bash script too large for a normal size screen should be rewritten in Python.
GNU make? BSD make? (Trick question - which BSD make?) nmake? (Another trick question...) - and that’s just Linux, MacOS and Windows today. Admittedly it was even worse previously.
There’s so many flavours of make with annoyingly incompatible syntax and that’s before you get into the presence of GMSL or equiv on each platform.
There’s only one fitting answer to this and it’s the output of the following command:
make love
Except depending on your make, this joke doesn’t work :-(
Shellspec is delightful to use. Every time I go to test with it I end up finding bugs. But also every time I go to test with it I end up convincing myself to not use bash. I imagine the same would be true of bash_unit.
I used bats a while ago to test a homemade CLI to open github pull-requests, it worked well enough for me (https://github.com/williamdclt/git-pretty-pull-request if you're looking for an example, although I did not maintain the tests out of laziness so they are very red).
Cannot compare it to bash_unit, but I'm happy there's alternatives!
Yeah i think most people looking for this don't necessarily care about the distinction between unit/integration (i myself don't really know anymore because so many of my elixir "exunit" tests hit the db -- it's fine) they just want something TAP compliant and maybe also offers green dots.
On the 3rd hand, a major aspect of the 'engineering' part of being a 'software engineer' is learning how to choose which side of various tradeoffs is appropriate for a project. Should we perform the expensive computation on demand at runtime, or is trading time for space with a lookup table more appropriate? Is the task simple enough for a basic 'glue language' like bash, or does it require a full-featured language like ruby or python?
Regardless, as you pointed out, regardless of the language some sort of testing is still a good idea. Even small, supposedly 'trivial' scripts deserve some testing!
When I was writing my IRC Bot In Bash, I was looking for something akin to this. for the most part I got away with simple mocking for the IRC bits, but all the functionality (plugins & commands) is still untested.
Around the time I created it bats was a thing, but it didn't seem that unobtrusive. this seems more in line with how I'd probably do testing for bash, though at this point I'm not so sure what I'd use it for. Maybe It's worth going over some personal or work projects and trying this out.
Hey, any ideas how to leverage your current testing framework to add coverage or if there's any pretty way to print coverage results that are somehow as standard as TAP?
on embedded boards which are deployed 100x more than desktops and are typically resource restricted you have to use posix shell instead of the mega-bytes bash there.
I appreciate the effort but I think that if you are unit testing shell scripts, I think it’s probably better to use a language like Python or something similar.
I am serious.
I have decided for my own efforts to use python for any kind of command line script and I’m always glad I did.
I still find it a bit awkward to use python just to run some commands in sequence with some massaging of input/output data and parameters based on simple logic. bash scripts are great for that. And in that case it’s still a good idea to have automated testing instead of relying on running the script a few times to ensure it behaves as it should. I’ll definitely give this a try :)
I do agree though that if you need extensive massaging of output or arguments, python can help and make the whole thing easier.
The trouble is that bash is necessary to setup python. For instance, the lingua franca of Docker is bash (well sh, but you should obviously change that).
I've been resisting learning (any more) bash for a while now, but I think I'm going to have to. This looks like it could reduce some of my terror at the insanities of shell scripts (why does one even need two quote characters that do different things?)
A couple recommendations that make bash a much saner and avoid entire classes of problems:
First, whenever you are expanding a list of args, use "$@"! That exact four character sequence expands to the properly quoted positional params ("$1" "$2" ...) with any necessary escaping included so each param expands as a single word. Almost all of the problems you've probably heard about bash "not handling spaces properly" or otherwise having problems with whitespace or strange characters in filenames are fixed by using "$@". If you're using arrays/hashes, you can get the same effect using "${somearrayorhash[@]}" (quotes included, just like "$@"). Removing the quotes or using the tradition $* is almost always a bug.
Second, always use explicit quotes/brackets! Forget that they were ever optional. Using "$@" fixes most of the whitespace-in-filename problems; expanding your variables with explicit quotes fixes the rest. Assuming these:
showargs() {
echo "$# args"
for i in "$@" ; do
echo "arg[${i}]"
done
}
declare -- name="filename with spaces\\!.txt"
declare -A h='([a]="b c" [foo]="'\''bar'\'' \"baz\" qu*x" )'
Instead of using the traditional shortcuts (which cause problems):
Always using quotes/brackets simply does the right thing:
showargs "${name}"
# 1 args
# arg[filename with spaces\!.txt]
showargs "${h[@]}"
# 2 args
# arg[b c d e]
# arg['bar' "baz" qu*x]
Bash still has it quirks and strange historical baggage, but in my experi4nce, using these two rules (and actually taking the time to read the bash(1) manpage...) changed writings shell scripts from an annoying mess of buggy arcane incantations into an actually sane(-ish) programming language.
You're right, but it's neither of those for command interpolation. Double quotes enable variable expansion, single quotes do not. Command interpolation is back quotes, which is a carryover from sh. The more modern bash way is as follows:
$ echo $(pwd)
Incidentally: how does one type back quotes on an iPhone?!
Edited to add: shellcheck [0] will flag the back quote usage if you're writing a bash script instead of a sh script.
It was much better on the 3D Touch equipped models, press “through” the keyboard to get cursor placement, press “through” again while on a word to select the whole word.
If it’s just a few lines or some one-off thing, I understand the use of bash.
I start to get the feeling that if you feel the need for shell scripts, it might be wise to pause and wonder of this is really the right approach long-term. Especially if you feel the need to put it in git or something.
My experience is that there are often I need to put in some checks for safety and before I know it, you create a mess of grep awk cut sed and you wish you started out with python.
Are you really that much in a hurry or do you have the time to calmly spend a little bit more time to ‘do it right?’
Or you literally just need to run a sequence of commands to create some stuff in the filesystem without doing any text processing whatsoever, which is my use case for the three or four large-ish BASH scripts I've written professionally. Make was totally inappropriate in that case because I would have had to enumerate a lot of intermediate files and would have wound up with a parallelizable series of mini-scripts that would need to run serially to work correctly. And Python would ultimately turn into a DSL that looks almost exactly like BASH because the problem domain is "run a bunch of commands in sequence" which is what BASH is designed to do.
- use shell=True and subprocess module i.e., use sh for what it is good for (one-liners that run external commands) and use python to glue it together in a sane manner
(consider shell as a DSL in this case).
- you could use the plumbum module to embed commands in Python itself
https://plumbum.readthedocs.io/en/latest/#piping
- for Makefile-like functionality invoke/fab could be used (flexible configuration, composition, debugging of the commands)
https://docs.fabfile.org/en/2.6/getting-started.html#addendu...
One of the biggest downside is that the startup time for python is significantly slower than the shell. With just one script this isn't really noticeable, but if you're composing a lot of small scripts together, python becomes noticeably less responsive for interactive commands than bash.
Along similar lines bash's syntax is incredibly streamlined for composing standalone scripts and programs through pipes. A simple bash one-liner like the below would be much more awkward to write in python:
> diff <(netcat $server | grep town | sed 's/street/St' | cut -f 3 | head -n 5) <(cat ./$(psql $query).dat)
I wonder if that ‘lots of small scripts together’ is a desireble situation. Can’t it be just one app that performs all the steps?
It is all about context to me. A shell oneliner is not something I would replace with python but as soon as you start up an editor, think again I would say
Agreed entirely. I inherited a system with 50k lines of unreliable shell scripts. I figured out ways to unit test them, do mocking, and pushed developers to unit test their scripts. I even wrote long articles of best practices for writing reliable scripts, including massive lists of gotchas.
Eventually I realised it was a lost cause and really you just shouldn't use shell scripts for anything that you want to be reliable above a trivial level of complexity.
That was with PowerShell, but I wouldn't be surprised if the same applies to bash as well.
I agree that you should go to a more robust language when shell scripts get bigger, esp if you are needing or wanting unit tests. I would avoid Python if there are lots of dropping into the subprocess module; it suffers the same problem as Go - lots of ceremony and boiler plate to run shell commands. I recommend Ruby or Perl because they can drop into a shell-like mode and you can execute shell commands more “naturally.”
We have tones of PowerShell and it works like a charm, although we have some experienced posh devs
We also test REST backend in PowerShell using Pester and home made Posh rest client.
PowerShell is preferred in this house because
a) you can run it on any Windows OS on the spot and modify it in ad hoc manner, you can even debug it with breakpoints etc easily. Also our Linux machines have it so it is unifying admin interface.
b) its powerful, you can do anything in it with few lines of code (one case: we did 10 million SOAP requests using certificate per day for entire country)
c) many Windows tools use it like SqlServer, IIS etc. which makes management way easier - for example we use [1] to install sql server on all dev/prod machines or use [2] to monitor all our servers or use [3] to send CI metrics to influx (all those are just minor samples, we have bunch of stuff like that)
d) we find it way easier to keep CI/CD vars in PowerShell hashtables then in yaml, so our yaml fiels are one liners and everything works locally.
e) Python, ruby and friends are NOT designed for shell work. Its akward, unfriendly and most of all not there on Windows OTB.
Hey, since you appears to use Chocolatey and there's few people using it, how do you feel it? Do you think it will keep existing? I am not that into the way MS is doing the winget thing, but I also noticed it appears to impact how people adopt or not chocolatey. I see a lot of manual scripts for installing things on Windows CI systems (some using now defunct fciv and soon to be defunct bitsadmin)... I really like chocolatey but I am worried it will disappear soon.
Chocolatey is the only thing I use to install stuff both on CI and on each dev machine and I regularly create packages for it [1]. I worked hard to make what I need stable and not depend on their existence - [2] and all the repos using the same methodology release packages on GH [4] and there is a handy script to install from there. I also created AU for it [3] and managed to convince people to embed software in packages [5] so packages always work (you can cache them on your own via file system, artifactory, nexus etc). You can also host your own gallery in number of different ways. So, in short, there is escape plan. TBH, it looks like choco is going better then ever. And you can't simply find any better repository for sw, its better and more up to date then most linux package repos (on par with Arch).
> I am not that into the way MS is doing the winget thing,
That is years away IMO, no scripting there too, and it moves like a snail. I would really be embarrassed if I were leading that team.
> I see a lot of manual scripts for installing things on Windows CI systems
Yeah, most people suck, like their scripts :-) There is literary 0 chance for you to make reliable installation script in general that works in any context.
> I really like chocolatey but I am worried it will disappear soon.
Just use it. I don't work for them. I maintain core team repo [2]. Its great tool now. What will happen tomorrow nobody knows but like I said, you have escape plan and even if they go down your CI will still work for many years if you set it up properly.
bats is used for running tests against command line interfaces of your program. This is useful for making sure weird strings and argument patterns are handled correctly.
Bash is a lowest-common-denominator language, available on almost all platforms, with a very stable (old?) API, which makes it ideal for broad distribution & bootstrapping of systems.
When you use another other languages, like Python, suddenly you're not only worrying about whether Python is installed, but also which version is available, and you still can't use anything that's not part of the standard library (because then you enter package management/dependency hell).
I agree that even Python with stdlib-only is still better than Bash, but I guarantee that you'll find Bash >=3.0 on every host.