|
| 1 | +<a href="https://github.com/ipython-books/cookbook-2nd"><img src="../cover-cookbook-2nd.png" align="left" alt="IPython Cookbook, Second Edition" height="140" /></a> *This is one of the 100+ free recipes of the [IPython Cookbook, Second Edition](https://github.com/ipython-books/cookbook-2nd), by [Cyrille Rossant](http://cyrille.rossant.net), a guide to numerical computing and data science in the Jupyter Notebook. The ebook and printed book are available for purchase at [Packt Publishing](https://www.packtpub.com/big-data-and-business-intelligence/ipython-interactive-computing-and-visualization-cookbook-second-e).* |
| 2 | + |
| 3 | +▶ *[Text on GitHub](https://github.com/ipython-books/cookbook-2nd) with a [CC-BY-NC-ND license](https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode)* |
| 4 | +▶ *[Code on GitHub](https://github.com/ipython-books/cookbook-2nd-code) with a [MIT license](https://opensource.org/licenses/MIT)* |
| 5 | + |
| 6 | +[*Chapter 2 : Best practices in Interactive Computing*](./) |
| 7 | + |
| 8 | +# 2.1. Learning the basics of the Unix shell |
| 9 | + |
| 10 | +Learning how to interact with the operating system using a command-line interface (or terminal) is a required skill in interactive computing and data analysis. We will use a command-line interface in most recipes of this book. IPython and the Jupyter Notebook are typically launched from a terminal. Installing Python packages is typically done from a terminal. |
| 11 | + |
| 12 | +In this recipe, we will show the very basics of the Unix shell, which is natively available in Linux distributions (such as Debian, Ubuntu, and so on) and macOS. On Windows 10, one can install the **Windows Subsystem for Linux**, a command-line interface to a Unix subsystem integrated with the Windows operating system (see https://docs.microsoft.com/windows/wsl/about). |
| 13 | + |
| 14 | +## Getting ready |
| 15 | + |
| 16 | +Here are the instructions to open a Unix shell on macOS, Linux, and Windows. **bash** is the most common Unix shell and this is what we will use in this recipe. |
| 17 | + |
| 18 | +On macOS, bring up the Spotlight Search, type `terminal`, and press Enter. |
| 19 | + |
| 20 | +On Windows, follow the instructions at https://docs.microsoft.com/en-us/windows/wsl/install-win10. Then, open the Windows menu, type `bash`, and press Enter. |
| 21 | + |
| 22 | +On Linux, open the Dash by clicking on the top-left icon on the desktop, type `terminal`, and open the `Terminal` application. |
| 23 | + |
| 24 | +If you want to run this notebook in Jupyter, you need to install `bash_kernel`, available at https://github.com/takluyver/bash_kernel. Open a terminal and type `pip install bash_kernel && python -m bash_kernel.install`. |
| 25 | + |
| 26 | +This will install a bash kernel in Jupyter, and it will allow you to run this recipe's code directly in the Notebook. |
| 27 | + |
| 28 | +## How to do it... |
| 29 | + |
| 30 | +The Unix shell comes with hundreds of commands. We will see the most common ones in this recipe. |
| 31 | + |
| 32 | +1. The terminal lets us write text commands with the keyboard. We execute them by pressing Enter, and the output is displayed below the command. The **working directory** is the directory of our file system that is currently "active" in the terminal. We can get the absolute path of the working directory as follows: |
| 33 | + |
| 34 | +```bash |
| 35 | +pwd |
| 36 | +``` |
| 37 | + |
| 38 | +```{output:stdout} |
| 39 | +~/git/cookbook-2nd/chapter02_best_practices |
| 40 | +``` |
| 41 | + |
| 42 | +> the dollar `$` sign must not be typed: it is typically used by the shell to indicate where the user can start typing. The information written before it may show the user name, the computer name, and part of the working directory. Here, only the three characters `pwd` should be typed before pressing Enter. |
| 43 | +
|
| 44 | +2. We can list all files and subdirectories in the working directory as follows: |
| 45 | + |
| 46 | +```bash |
| 47 | +ls |
| 48 | +``` |
| 49 | + |
| 50 | +```{output:stdout} |
| 51 | +00_intro.md 03_git.md 07_high_quality.md |
| 52 | +01_shell.md 04_git_advanced.md 08_test.md |
| 53 | +02_py3 05_workflows.md 09_debugging.md |
| 54 | +02_py3.md 06_tips.md images |
| 55 | +``` |
| 56 | + |
| 57 | +```bash |
| 58 | +ls -l |
| 59 | +``` |
| 60 | + |
| 61 | +```{output:stdout} |
| 62 | +total 100 |
| 63 | +-rw-rw-r-- 1 owner 769 Dec 12 10:23 00_intro.md |
| 64 | +-rw-rw-r-- 1 owner 2473 Dec 12 14:21 01_shell.md |
| 65 | +... |
| 66 | +-rw-rw-r-- 1 owner 9390 Dec 12 11:46 08_test.md |
| 67 | +-rw-rw-r-- 1 owner 5032 Dec 12 10:23 09_debugging.md |
| 68 | +drwxrwxr-x 2 owner 4096 Aug 1 16:49 images |
| 69 | +``` |
| 70 | + |
| 71 | +The `-l` option displays the directory contents as a detailed list, showing the permissions and owner of the files, the file sizes, and the last modified dates. Most shell commands come with many options that alter their behavior and that can be arbitrarily combined. |
| 72 | + |
| 73 | +3. We use the `cd` command to navigate between subdirectories. The current directory is named `.` (single dot), and the parent directory is named `..` (double dot): |
| 74 | + |
| 75 | +```bash |
| 76 | +cd images |
| 77 | +``` |
| 78 | + |
| 79 | +```bash |
| 80 | +pwd |
| 81 | +``` |
| 82 | + |
| 83 | +```{output:result} |
| 84 | +~/git/cookbook-2nd/chapter02_best_practices/images |
| 85 | +``` |
| 86 | + |
| 87 | +```bash |
| 88 | +ls |
| 89 | +``` |
| 90 | + |
| 91 | +```{output:stdout} |
| 92 | +folder.png github_new.png |
| 93 | +``` |
| 94 | + |
| 95 | +```bash |
| 96 | +cd .. |
| 97 | +``` |
| 98 | + |
| 99 | +```bash |
| 100 | +pwd |
| 101 | +``` |
| 102 | + |
| 103 | +```{output:result} |
| 104 | +~/git/cookbook-2nd/chapter02_best_practices |
| 105 | +``` |
| 106 | + |
| 107 | +4. Paths can be specified as relative (depending on a reference directory, generally the working directory) or absolute. The **home directory**, specified as `~`, contains the user's personal files. Configuration files are often stored in a directory like `~/.program_name`. For example, `~/.ipython` contains configuration files of IPython: |
| 108 | + |
| 109 | +```bash |
| 110 | +ls -la ~/.ipython |
| 111 | +``` |
| 112 | + |
| 113 | +```{output:stdout} |
| 114 | +total 20 |
| 115 | +drwxr-xr-x 5 cyrille 4096 Nov 14 16:16 . |
| 116 | +drwxr-xr-x 93 cyrille 4096 Dec 12 10:50 .. |
| 117 | +drwxr-xr-x 2 cyrille 4096 Nov 14 16:16 extensions |
| 118 | +drwxr-xr-x 2 cyrille 4096 Nov 14 16:16 nbextensions |
| 119 | +drwxr-xr-x 7 cyrille 4096 Dec 12 14:18 profile_default |
| 120 | +``` |
| 121 | + |
| 122 | +> in most terminals, we can use the arrow keys on the keyboard to navigate in the history of past commands. Also, the Tab key enables tab completion, which automatically completes the first characters of a command or a file. For example, typing `ls -la ~/.ipy` and pressing Tab would automatically complete to `ls -la ~/.ipython`, or it would present the list of possible options if there are several files or directories that begin with `~/.ipy`. |
| 123 | +
|
| 124 | +5. We can create, move, rename, copy, delete files and directories from the terminal: |
| 125 | + |
| 126 | +```bash |
| 127 | +# We create an empty directory: |
| 128 | +mkdir md_files |
| 129 | +# We copy all Markdown files into the new directory: |
| 130 | +cp *.md md_files |
| 131 | +# We rename the directory: |
| 132 | +mv md_files markdown_files |
| 133 | +ls markdown_files |
| 134 | +``` |
| 135 | + |
| 136 | +```{output:stdout} |
| 137 | +00_intro.md 05_workflows.md |
| 138 | +01_shell.md 06_tips.md |
| 139 | +02_py3.md 07_high_quality.md |
| 140 | +03_git.md 08_test.md |
| 141 | +04_git_advanced.md 09_debugging.md |
| 142 | +``` |
| 143 | + |
| 144 | +```bash |
| 145 | +rmdir markdown_files |
| 146 | +``` |
| 147 | + |
| 148 | +```{output:stdout} |
| 149 | +rmdir: failed to remove 'markdown_files': |
| 150 | + Directory not empty |
| 151 | +``` |
| 152 | + |
| 153 | +```bash |
| 154 | +rm markdown_files/* |
| 155 | +``` |
| 156 | + |
| 157 | +```bash |
| 158 | +rmdir markdown_files |
| 159 | +``` |
| 160 | + |
| 161 | +> The `rm` command lets us delete files and directories. The `rm -rf path` deletes the given path recursively, even if subdirectories are not empty. It is an extremely dangerous command as it cannot be undone: the files are immediately and permanently deleted, they do not go into a trash directory first. See https://github.com/sindresorhus/guides/blob/master/how-not-to-rm-yourself.md for more details. |
| 162 | +
|
| 163 | +6. There are several useful commands to deal with text files: |
| 164 | + |
| 165 | +```bash |
| 166 | +# Show the first three lines of a text file: |
| 167 | +head -n 3 01_shell.md |
| 168 | +``` |
| 169 | + |
| 170 | +```{output:stdout} |
| 171 | +# Learning the basics of the Unix shell |
| 172 | +
|
| 173 | +Learning how to interact with the operating system (...) |
| 174 | +``` |
| 175 | + |
| 176 | +```bash |
| 177 | +# Show the last line of a text file: |
| 178 | +tail -n 1 00_intro.md |
| 179 | +``` |
| 180 | + |
| 181 | +```{output:stdout} |
| 182 | +We will also cover more general topics (...) |
| 183 | +``` |
| 184 | + |
| 185 | +```bash |
| 186 | +# We display some text: |
| 187 | +echo "Hello world!" |
| 188 | +``` |
| 189 | + |
| 190 | +```{output:stdout} |
| 191 | +Hello world! |
| 192 | +``` |
| 193 | + |
| 194 | +```bash |
| 195 | +# We redirect the output of a command to |
| 196 | +# a text file with `>`: |
| 197 | +echo "Hello world!" > myfile.txt |
| 198 | +``` |
| 199 | + |
| 200 | +```bash |
| 201 | +# We display the entire contents of the file: |
| 202 | +cat myfile.txt |
| 203 | +``` |
| 204 | + |
| 205 | +```{output:stdout} |
| 206 | +Hello world! |
| 207 | +``` |
| 208 | + |
| 209 | +Several command-line text editors are available, such as `pico`, `nano`, or `vi`. Learning these text editors requires time and effort, especially vi. |
| 210 | + |
| 211 | +7. The `grep` command lets us search substrings in text. In the following example, we find all instances of `Unix` followed by a word (using regular expressions): |
| 212 | + |
| 213 | +```bash |
| 214 | +grep -Eo "Unix \w+" 01_shell.md |
| 215 | +``` |
| 216 | + |
| 217 | +```{output:stdout} |
| 218 | +Unix shell |
| 219 | +Unix shell |
| 220 | +Unix subsystem |
| 221 | +Unix shell |
| 222 | +(...) |
| 223 | +Unix shell |
| 224 | +Unix shell |
| 225 | +``` |
| 226 | + |
| 227 | +8. A major strength of the Unix shell is that commands can be combined with **pipes**: the output of one command can be directly transferred to the input of another command: |
| 228 | + |
| 229 | +```bash |
| 230 | +echo "This is a Unix shell" | grep -Eo "Unix \w+" |
| 231 | +``` |
| 232 | + |
| 233 | +```{output:stdout} |
| 234 | +Unix shell |
| 235 | +``` |
| 236 | + |
| 237 | +## There's more... |
| 238 | + |
| 239 | +We only scratched the surface of the Unix shell in this recipe. There are many other commands that can be combined in an infinite number of ways. Many repetitive tasks that would take hours of manual work can be done in a few minutes by writing the appropriate commands. Mastering the Unix shell may take a lot of effort, but it leads to dramatic time gains in the long term. |
| 240 | + |
| 241 | +Here are a few references: |
| 242 | + |
| 243 | +* Linux tutorial at https://ryanstutorials.net/linuxtutorial/ |
| 244 | +* Bash commands at https://ss64.com/bash/ |
| 245 | +* Learn Bash in Y minutes, at https://learnxinyminutes.com/docs/bash/ |
| 246 | +* Learn the shell interactively, at http://www.learnshell.org/ |
| 247 | +* The fish shell, at https://fishshell.com/ |
| 248 | +* xonsh, a Python-powered shell, at http://xon.sh/ |
| 249 | +* Windows Subsystem for Linux, at https://docs.microsoft.com/windows/wsl/about |
| 250 | + |
| 251 | +## See also |
| 252 | + |
| 253 | +* Ten tips for conducting reproducible interactive computing experiments |
0 commit comments