Run It Twice, Trust It Once: Engineering Scripts That Survive Repeated Execution
Photo by Photo by Tyler on Unsplash on Unsplash
Every automation engineer eventually learns this lesson the hard way: the script that worked flawlessly in testing will, at some point, run twice. A cron job overlaps with a manual run. A CI pipeline retries a flaky step. An on-call engineer, staring at a half-finished deployment at 2 a.m., hits Enter again because they cannot tell whether the first attempt succeeded.
What happens next separates two very different tiers of script author.
The Cost of Assuming a Clean Slate
Consider a provisioning script that appends a line to /etc/hosts, creates a service account, and starts a daemon. Run once, it works. Run twice, and you now have a duplicate hosts entry, a failed useradd (or worse, a silently recreated account with a fresh UID), and a daemon that refuses to bind a port already in use.
None of these failures are exotic. They are the ordinary consequence of writing scripts that mutate state without first inspecting state. In a production environment — where retries, concurrency, and human intervention are facts of life — a non-idempotent script is not a script. It is a liability with a shebang.
The principle is simple to state and demanding to practice: running the script again after a previous run should produce no additional changes.
Patterns That Make Bash Re-Runnable
Idempotency in shell scripting mostly comes down to replacing blind actions with checked ones.
Check before you create. Rather than:
useradd -m svc-deployer
write:
id -u svc-deployer &>/dev/null || useradd -m svc-deployer
Prefer declarative writes over appends. Appending configuration lines (echo 'export EDITOR=vim' >> ~/.bashrc) is the classic idempotency killer. Instead, match first:
grep -q 'EDITOR=' ~/.bashrc || echo 'export EDITOR=vim' >> ~/.bashrc
Use tools that are idempotent by design. mkdir -p, chmod (setting, not adding), systemctl enable, and package managers' install commands are safe to re-run. Learn which of your everyday commands share that property and which do not.
PowerShell Makes It Easier — If You Let It
PowerShell's cmdlets lean heavily toward idempotent semantics. Set-Item overwrites rather than duplicates. New-LocalUser throws if the account exists, which forces the check:
if (-not (Get-LocalUser -Name 'svc-deployer' -ErrorAction SilentlyContinue)) {
New-LocalUser -Name 'svc-deployer' -NoPassword
}
The broader pattern generalizes: Get-* to inspect, compare against desired state, then Set-* or New-* only when the diff demands it. This is, not coincidentally, exactly the model that made DSC and Ansible successful. You do not need a configuration management platform to borrow its philosophy.
Testing for Idempotency Before the Scheduler Does
You can verify idempotency cheaply, and you should:
- Snapshot state before the first run — file hashes, user lists, installed packages, whatever the script touches.
- Run the script twice back to back.
- Diff the snapshots. Any difference after the second run is a bug.
Automate this in CI with a container or a disposable VM, and it becomes a five-second gate. Pester tests in PowerShell or Bats in Bash can encode the assertions permanently: run the setup, run it again, assert that the second run changed nothing and exited zero.
The teams that treat re-runnability as a testable property — rather than a hope — are the ones whose 2 a.m. retries end quietly instead of in a postmortem. The scheduler will eventually run your script twice. The only question is whether you find out first.