Skip to content

Lesson 13 ยท sed Scripting

sed Script Files and Classic Recipes

Organize sed programs in script files with comments and shebangs, pass shell variables safely, and learn a dozen classic recipes: trim, number, strip tags, CRLF and more.

A sed program starts as a one-liner and, if it is useful, grows. At some point you want comments, a name, a place in version control. This lesson shows how to put sed programs in script files, how to pass shell data into them safely, and then collects a set of classic recipes you will use again and again. Each recipe is short, tested, and explained.

Script files

Put commands in a file, one per line, and run it with -f:

TEXT
sed -f cleanup.sed input.txt

Rules for script files:

  • A line starting with # is a comment. If the first two characters of the file are #n followed by a newline, it acts like -n.
  • Commands are separated by newlines (or ;). Labels and a/i/c text end at the newline.
  • Blank lines and leading spaces are ignored.
  • A script can start with #!/bin/sed -f (or #!/usr/bin/sed -f) and be made executable: ./cleanup.sed input.txt. Many systems also need -nf in the shebang for -n.
Bash A commented script file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Each section is a recipe in itself, and the comments tell the next person (or you in six months) why it exists. For a few commands you can still use -e; for anything that people will read, use a file.

Mixing -f and -e, and several files

You may repeat -f and -e. The fragments are joined in order. That lets you keep a reusable library of scripts and add a project-specific tweak:

TEXT
sed -f common.sed -e 's/staging/production/' app.conf

Passing shell variables safely

A script file cannot see the shell's variables. For one-liners you may switch to double quotes, but then the shell processes $, backticks and backslashes, and the value is pasted into the regex or replacement without escaping. A value containing /, & or \ can break the command or change its meaning.

Bash Variables in sed commands

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last example escapes \, & and | (the delimiter used) so the value is inserted literally. For anything more complex or for untrusted values, a better approach is to give the value to AWK via -v, which does no regex interpretation of the data (see AWK variables).

Recipes

Every recipe below was tested. Try them one at a time.

Bash Recipe set 1: whitespace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last one works like cat -s: on an empty line, N appends the next line; if that is empty too (\n$), D deletes the first of the two and restarts.

Bash Recipe set 2: numbering, ranges, reversing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Bash Recipe set 3: files, paths, markup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last recipe uses the empty regex: inside /BEGIN/,/END/, // means "the last regex tried", which is BEGIN on its first line and END on the last, so //!p skips both marker lines.

Bash Recipe set 4: editing configuration safely

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

#& puts a hash before the whole match: a common way to comment out lines without retyping them.

Writing maintainable scripts

  1. Name your scripts for what they do (normalise-config.sed), not how.
  2. Comment each section, including why a regex looks the way it does.
  3. Test on a small sample file and compare with diff before using -i.
  4. Keep sample input and expected output next to the script so changes can be checked.
  5. Know when to stop: if the script has several nested loops or hold-space juggling, rewrite it in AWK or another language.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a first line #n in a sed script file do?

    #n on the very first line of a script (alone on that line) works like -n.
  2. 2. How do you run the script file fix.sed on data.txt?

    -f reads the script from a file; -e would treat the text fix.sed as a command.
  3. 3. Which command deletes the lines between markers while keeping the markers?

    Inside the range, // is the last regex used, so //!d deletes every line except the two marker lines.
  4. 4. Why is putting $value into a double-quoted sed command risky?

    The value is pasted into the script as text, so special characters change its meaning unless you escape them.

Summary

  • Put real programs in .sed files with comments; run them with -f; #n on line 1 means -n.
  • Combine -f and -e; avoid pasting untrusted shell values into scripts.
  • Memorize the recipes: trim, squeeze blank lines, number lines, strip tags, CRLF conversion, section extraction, comment out lines.
  • Test with diff before -i, keep samples, and switch to AWK when the script gets hard to read.

Next: Your First AWK Program