I like this pattern a lot, but it's important that the code in the dry path is representative. I've been bitten a few too many times by dry code that just runs `print("would have updated ID: 123")`, but not actually running most of the code in the hot path. Then when I run it for real, some of the prep for the write operation has a bug / error, so my dry run didn't actually reveal much to me.
Put another way: your dry code should do everything up until the point that database writes / API calls / etc actually happen. Don't bail too early
This is why I end up creating an interface for the calls that perform modifications. Then I have one implementation that logs and one implementation that does the actual work. I end up with the output being as representative as possible as to what would happen. I also feel a lot more comfortable that a dry run truly won't write anything when the only class that could ever actually write anything is not even instantiated in dry run mode. I don't get that same comfort when there's a ton of branches sprinkled throughout checking for a dry run flag.
Doesn’t this conflate dry-running with integration testing? ASAIK the purpose of a dry-run is to understand what will happen, not to test what will happen. For the latter we have testing.
> ASAIK the purpose of a dry-run is to understand what will happen, not to test what will happen. For the latter we have testing.
Not really. Testing is a way to increase confidence that code does what it is specified to do, because it is cheaper than full-blown formal analysis :)
The problem raised by OP here is granularity. Operation like `update(record, field, value)` is itself a tree of smaller sub-operations that may do some permissions checking, locking, network calls, even checking for presence of record if it has upsert semantics, all of which could fail. A dry run with a plan that is too coarse can succeed while the actual operation fails over things left unchecked.
For little scripts, I'm not writing unit tests- running it is the test. But I want to be able to iterate without side effects, so it's important that the dry mode be as representative as possible for what'll happen when something is run for real.
You understand how subjective that is right? Someone might expect that the database doesn't do the last commit step while other people is perfectly happy that the database engine checks that it has enough writing permissions and is running as a user that can start the process without problems.
Sure, where you draw the line will vary between projects. As long as its exact placement doesn't matter too much.
For me personally, I tend to draw the line at write operations. So in your example, I'd want a dry run to verify the permissions that it can (if I expect those to be a problem). But if that can't easily be done without a write, then maybe it's not worth it. There are also situations where you want a dry run to be really fast, so you forego some checks (allowing for more surprises later). Really just depends.
We recently welcomed a new baby and have been discussing whether she'll ever learn to drive. I love bets with long time horizons, so we put together a survey so friends and family could pick a side. We kept thinking of fun questions to ask, so it grew into the survey I've linked above, which we're opening to the public.
It's short, fun, and totally informal. Thanks for looking!
This was originated (or at least popularized) in an episode of Archer from season 1. It remains maybe my favorite 5 minutes of comedy ever out to screen. It's just so tight!
If you're interested in some (light) science behind rivalry and/or the Michigan/OSU rivalry, I highly recommend the documentary "Rivals: Ohio State vs. Michigan". It's a fun look into why a rivalry drives better performance.
I found this to be an interesting new direction from Airtable, a product I've really enjoyed. I hope it doesn't muck up the core functionality too much.
My rule of thumb is that as soon as I write a conditional, it's time to upgrade bash to Python/Node/etc. I shouldn't have to search for the nuances of `if` statements every time I need to write them.
An if statement in, for instance bash, just runs any command and then runs one of two blocks of code based on the exit status of that command. If the exit status is truthy, it runs what follows the `then`. If it's falsey, it rhns what follows the `else`. (`elsif` is admittedly gratuitous syntax— it would be better if it were just implemented as an if inside an else statement.) This seems quite similar to other programming languages and like not very much to remember.
I'll admit that one thing I do in my shell scripts is avoid "fake syntax"— I never use `[` or `[[` because these obscure the real structure of the statements for the sake of cuteness. I just write `test`, which makes clear that it's just an ordinary command, ans also signals to someone who isn't sure what it's doing that they can find out just by running `man test`, `help test`, `info test`, etc., from the same shell.
I also agree that if statements and if expressions should be kept few and simple. But in some ways it's actually easier to do this in shell languages than in many others! Chaining && and/or || can often get you through a substantial script without any if statements at all, let alone nested ones.
The difference being, as far as I know, that `[[` is the real syntax. This from what I remember helps in avoiding certain class of issues, gives better error messages and is more certain to be a bash built-in.
What I would worry about more is that it breaks `sh` compatibility.
`test` and `[` are Bash builtins just like `[[` is built into bash. But `[[`'s implementation does some things that actual commands can't do because it gets parsed differently than a normal command.
When Bash sees it, it treats it as something that needs to be matched, like a string, and won't stop taking user input in an interactive session until it sees the `]]` or something else that causes a syntax error. If I write `[` and just hit enter, I get an error from the `[` command, same as if I ran an external one. But if I use a `[[`, I get an error message back from Bash itself about a malformed conditional (and/or it will wait for input before trying to execute the command):
2 bash which -a [
/Users/pxc/.nix-profile/bin/[
/bin/[
2 ~
bash [
bash: [: missing `]'
2 ~
2 bash /bin/[
[: missing ]
2 ~
2 bash [[
∙ no prompt
bash: conditional binary operator expected
bash: syntax error near `prompt'
2 ~
2 bash [[
∙ a =
bash: unexpected argument `newline' to conditional binary operator
bash: syntax error near `='
The other thing `[[` does is it has you write `&&` and `||` instead of `-a` and `-o`. A normal command can't do this, because it can't influence the parser of the shell running it-- `&&` will get interpreted by the shell rather than the command unless it is escaped.
This same kind of special handling by the parser probably allows for other differences in error messages, but I don't write Bash that produces such errors, so I couldn't tell you. ;)
> more certain to be a bash built-in
If you want to be sure that you're using a built-in rather than a command, you can use the `builtin` command. But because `[[` is actually special syntax, it's technically not a builtin, so you can't use it this way! Check it out:
~ took 34m8s
2 bash
2 ~
bash builtin [[
bash: builtin: [[: not a shell builtin
2 ~
1 bash builtin [
bash: [: missing `]'
The thing that lets `[[` yield more sophisticated error messages in some ways is actually the very reason I prefer to stay away from it: it's special syntax when an ordinary command works just fine. I think the any-command-goes-here-and-all-commands-are-equal structure of if statements in Unix shells is elegant, and it's already expressive enough for everything we want to do. Stuff like `[[` complicates things and obscures that elegance without really buying us additional power, or even additional concision.
Imo, that's the real reason to avoid it. I'm all for embracing Bashisms when it makes code more legible. For instance, I think it's great to lean on associative arrays, `shopt -s lastpipe`, and `mapfile`. They're innovations (or deviations, depending on how you look at it ;), like `[[`, but I feel like they make the language clearer and more elegant while `[[` actually obfuscates the beauty of `if` statements in shell languages, including Bash.
I mean, there are 3 equally valid ways to write an if statement: `test`, `[`, and `[[`. In the case of the latter two, there are a mess of single-letter flags to test things about a file or condition[0]. I'm not sure what makes them "fake syntax", but I also don't know that much about bash.
It's all reasonable enough if you go and look it up, but the script immediately becomes harder to reason about. Conditionals shouldn't be this hard.
You don't need any of those to write an if statement. I frequently write if statements like this one
if ! grep -qF something /etc/some/config/file 2>/dev/null; then
do_something
fi
The `test` command is there if you want to use it, but it's just another command.
In the case of Bash, `test` is a built-in command rather than an external program, and it also has two other names, `[` and `[[`. I don't like the latter two because they look, to a naive reader, like special syntax built into the shell— like something the parser sees as unique and different and bear a special relationship to if-statements— but they aren't and they don't. And in fact you can use them in other shells that don't have them as built-ins, if you implement them as external commands. (You can probably find a binary called `[` on your system right now.)
(Actually, it looks like `[[` is even worse than "fake syntax"... it's real special syntax. It changes how Bash interprets `&&` and `||`. Yikes.)
But if you don't like `test`, you don't have to use it; you can use any command you like!
For instance, you might use `expr`:
if expr "1 > 0"; then
echo this will always run
else
echo this will never run
fi
Fish has some built-ins that fall into a similar niche that are handy for simple comparisons like this, namely `math` and `string`, but there are probably others.
If you really don't like `test`, don't even need to use it for checking the existence or type (dir, symlink, socket, etc.) of files! You can use GNU `find` for that, or even sharkdp's `fd` if you ache for something new and shiny.
Fish actually has something really nice here in the `path` built-in, which includes long options like you and I both wish `test` had. You can write:
if path -q --type=dir a/b/c
touch a/b/c/some-file
end
You don't need `test` for asking about or asserting equality of variables, either;
grep -qxF "$A" <<< "$B"
is equivalent to
test "A" = "$B"
or with the Fish `string` built-in
string match --entire $A $B
The key is that in a shell, all commands are truthy in terms of their exit status. `&&` and `||` let you combine those exit statuses in exactly the way you'd expect, as do the (imo much more elegant) `and` and `or` combiner commands in Fish.
Finally, there's no need to use the likes of `test` for combining conditions. I certainly never do. You can just write
test "$A" = "$B" && test "$C" = "$D"
instead of something like
[ "$A" = "$B" -a "$C" = "$D" ]
If-statements in shell languages are so simple that there's practically nothing to them. They just take a single command (any!) and branch based on its exit status! That's it.
As for readability: any program in any language is difficult to understand if you don't know the interfaces or behaviors of the functions it invokes. `[`/`test` is no different from any such function, although it appears that `[[` is something weirder and, imo, worse.
This is a decent heuristic, although (IMO) you can usually get away with ~100 lines of shell without too much headache.
Last year I wrote (really, grew like a tumor) a 2000 line Fish script to do some Podman magic. The first few hundred lines were great, since it was "just" piping data around - shell is great at that!
It then proceeded to go completely off the rails when I went full sunk cost fallacy and started abusing /dev/shm to emulate hash tables.
E: just looked at the source code. My "build system" was another Fish script that concatenated several script files together. Jeez. Never again.
Historically; my rule of thumb is as soon as I can't see the ~entire script without scrolling - time to rewrite in Python/ansible. I Think about the rewrite, but it usually takes awhile to do it (if ever)
Put another way: your dry code should do everything up until the point that database writes / API calls / etc actually happen. Don't bail too early
reply