Pipes and filters

ls shows all files and directories at this point.

!ls

create-delete.ipynb   grep-find.ipynb       shell-variables.ipynb
dvc.list              index.rst
file-system.ipynb     pipes-filters.ipynb

With *.rst we restrict the results to all files with the suffix .rst:

!ls *.rst

index.rst

We can also output only the number of lines, words and characters in these documents:

!wc *.rst

      18      48     450 index.rst

Now we write the number of characters in the file length.txt and then output the text with cat:

!wc -m *.rst > length.txt

!cat length.txt

     450 index.rst

We can also have the files sorted by the number of characters:

!sort -n length.txt

     450 index.rst

!sort -n length.txt > sorted-length.txt

We can also overwrite the existing file:

!sort -n length.txt > length.txt

If we only want to know the total number of characters, i.e. only output the last line, we can do this with tail:

!tail -n 1 length.txt

> is used to overwrite a file while >> is used to append to a file.

!echo amount of characters >> length.txt

!cat length.txt

amount of characters

Pipe |

You can connect commands with a pipe (|). In the following one-liner, we want to display the number of characters for the shortest file:

!wc -l *.rst | sort -n | head

      18 index.rst

If we want to display the first lines of the main text (without the first three lines for the title):

!cat index.rst | head -n 5 | tail -n 2

Any command on the command line will also work in Jupyter Notebooks if prefixed
with ``!``. The results can then interact with the Jupyter namespace, see