Input/Output redirection

This module goes over how input and output can be manipulated on the command line.

Introduction

There are 3 important components of how text is handled: standard out, standard in, and standard error. This is how information is handled for programs and users. Additionally, using redirection operations to streamline output from one command to the input of another command or to a file.

NB What makes these powerful is that you can keep inserting redirection operators to string commands together to accomplish some task

Type	ID	Description
`stdin`	0	standard input
`stdout`	1	standard output
`stderr`	2	standard error

stdout - standard output

In the terminal we look at the text output from programs and scripts, this text output is written to standard output (stdout). For example,

$ echo "Hello World!"
Hello World!

Prints “Hello World!” to stdout so the user can read it.

stdin - standard input

Standard Input (stdin) is typically where a user can input text from a keyboard into a program. Much like how we type arguments to a command.

Redirection operations

What is really useful about stdin, stdout and stderr is that you can use redirection operations from files or programs to read/manipulate/filter text streams to be either more readable or more useful for other programs.

Operation	Name	Descsription
`>`	stdout to file	creates/overwrite file and dumps stdout to it
`>>`	append stdout to file	similiar to `>` but appends to the end of the file
`<`	file as stdin	use contents of file as stdin
`\|`	pipe	use stdout as stdin for next program

Write and append to file - `>` and `>>`

A simple example using echo and cat to create a new file

$ echo "Hello World!" > file.txt
$ cat file.txt
Hello World!

We can overwrite the file

$ echo "Hello!" > file.txt
$ cat file.txt
Hello!

We can also append to the file

$ echo "Bye!" >> file.txt
$ cat file.txt
Hello!
Bye!

These operators are useful for creating data files from programs that just print to stdout

Pipe - use stdout as stdin for program

The | operator is used to convert stdout of one program to the stdin of another program. This is useful as it allows filtering to be done on the input before running it through another command.

If we assume we have a file containing the tenents of the unix philosophy

$ cat unixphilosopy.txt
Small is beautiful.
Make each program do one thing well.
Build a prototype as soon as possible.
Choose portability over efficiency.
Store numerical data in flat ASCII files.
Use software leverage to your advantage.
Use shell scripts to increase leverage and portability.
Avoid captive user interfaces.
Make every program a filter.

Mike Gancarz, The Unix Philosophy, ISBN 1-55558-123-4, 1994

We can use the pipe command to count many lines, words and characters using wc are within the first 3 lines using head of the unixphilosophy.txt

$ head -n 3 unixphilosophy.txt | wc
    3      17      96

Compared that to just running wc

$ wc unixphilosophy
    9      51     331 unixphilosopy.txt

This is extremely powerful when using commands like grep, sed and awk. See module Regular Expressions (regex)

Use a file as stdin `<`

Not common but is used in places, the < operator can be used to take the contents of a file and use it as stdin. As with the wc example we could easily type

$ wc < unixphilosophy
    9      51     331

Notice that the filename is no longer present on the line. This can also be done by using cat and |

$ cat unixphilosophy.txt | wc
    3      17      96

Depending on if you need to make adjustments to the file on the fly using regex, the | method might be more useful than <.

NB this is an example. Typically using cat and | into a program is unecessary as the program will handle input from files or stdin directly.

stderr - standard error

It may seem strange to talk about standard Error after the I/O redirection section but it is more useful as (stderr) is similar to stdout and prints to the screen, however it is not the same as stdout. stderr will not save to a file if you try to, one must use the redirection operators to do this.

Assuming we have a file hello.txt

$ cat hello.txt
Hello World!

We can try to see the contents of a non existent file bye.txt

$ ls
hello.txt
$ cat bye.txt
cat: bye.txt: No such file or directory
$ cat bye.txt > log.txt
cat: bye.txt: No such file or directory
$ cat log.txt
$

One can redirect the stderr output to a file by using the ID from above

$ cat bye.txt 2> log.txt
$ cat log.txt
cat: bye.txt: No such file or directory

You can also redirect stderr to stdout by the same process using the file description operator >& where 1 is stdout as above

$ cat bye.txt 2>&1
cat: bye.txt: No such file or directory

If you want to record both stdout and stderr to a file then

$ cat hello.txt bye.txt > log.txt 2>&1
$ cat log.txt
Hello World!
cat: bye.txt: No such file or directory