Introduction To Linux - UAntwerpen

Managing processes Terminate processes Stop and resume processes Run processes in backgroundExample: run xclock with xclock -update 1The process is started, you have no prompt.To terminate the foreground process, press Ctrl cxclock disappears, the prompt returns.To stop (pause) the foreground process, press Ctrl zThe process is stopped in the background, the prompt returns. fg process resumes in the foreground. bg process continues in the background.VLAAMSSUPERCOMPUTER CENTRUM

To start a process in the background, terminate the command by & xclock -update 1 & bash prints the job number and PID, e.g. [1] 9582Multiple background jobs: use jobs to see a list: xclock -update 1 &[1] 9582 xclock -update 1 &[2] 9588 jobs[1]- Running[2] Runningxclock -update 1 &xclock -update 1 &Use the job number to control different processes, e.g. fg %2 run job 2 in the foregroundVLAAMSSUPERCOMPUTER CENTRUM

Terminate a processReminder: Ctrl c terminates the foreground process.Use the command kill PID to terminate any process (owned by you) kill 12345 Terminate process with id 12345. The process maybelong to another shell.kill % jobnum terminates a background process: kill %2 Terminate job 2, with time for cleanup. kill -KILL %2 Terminate job 2 immediately.Use kill -STOP and kill -CONT to pause/resume processes.VLAAMSSUPERCOMPUTER CENTRUM

ThreadsThe example program omp pi can run with multiple threads: OMP NUM THREADS 4 ./omp pi ps -T displays each process’ threads: ps -TPID SPID17058 1705817058 1705917058 1706017058 0:00:3200:00:33CMDomp piomp piomp piomp pi top -H displays CPU usage for each thread in real time.When running top, hit f to display other info (e.g. CPU number).VLAAMSSUPERCOMPUTER CENTRUM

Streams, redirection, pipelinesOutput of commands is shown in the terminal; some commands readinput from the keyboard.This is managed using fle descriptors: Normal output is written to standard output, fd 1. Warnings and errors are written to standard error, fd 2. Commands can read from standard input, fd 0.By default, “stdout” and “stderr” fle descriptors are attached toterminal, “stdin” is read from the keyboard.We can redirect output and input: write output to a fle send output from one command to input of another command read stdin from a fleVLAAMSSUPERCOMPUTER CENTRUM

Output redirectionThe operator i redirects fle descriptor i to a fle.Example: ls 1 ls-output.txt File ls-output.txt is created, contains the command’s output. stderr still shown in terminal Inspect the fle with less ls-output.txtRedirect stderr: ls wrong-filename 2 ls-error.txtMultiple redirections for one command: ls *.txt *.jpg 1 ls-output.txt 2 ls-errors.txt without fd number redirects stdout: ls ls-output.txtVLAAMSSUPERCOMPUTER CENTRUM

/dev/null is a special “fle” that discards everything written to it. hide a program’s output: ./omp pi /dev/nullNote: creates a new fle Existing fle with same name is replaced (!) If command produces no output: empty fle.1 and 2 append stdout or stderr to the end of a fle, withouterasing previous content. date diary.txt echo "Dear diary, today ." diary.txti &j: attach fle descriptor i to the same fle as descriptor j: ls *.txt *.jpg ls-all.txt 2 &1 write stdout and stderr to the same fle.VLAAMSSUPERCOMPUTER CENTRUM

Input redirectionStandard input (fd 0) is read from the keyboard. Example: try bc.The input redirection operator filename opens a fle, from which theprogram now reads standard input: echo "2 * 17" homework.txt bc homework.txt34Most commands also accept a fle name as an argument. e.g., thesecommands have the same result: less homework.txt less homework.txtRedirecting input and output: bc homework.txt answers.txtVLAAMSSUPERCOMPUTER CENTRUM

PipelinesWe can chain 2 or more commands with the (“pipe”) operator: command1 command2 command3 [ .] stdout from command1 is directly sent to stdin of command2, etc.Commands run in parallel, each command processes input as itbecomes available.Example: scrolling through the list of all processes with less. ps aux lessCreate complex commands from simple building blocks.Note: to pipe stderr from a command, redirect it to stdout: command1 2 &1 command2VLAAMSSUPERCOMPUTER CENTRUM

Typical commands for pipelines: wc print the number of lines, words and bytes of input grep flter lines which match a given search pattern head / tail print frst/last lines of input sort sort input alphabetically uniq report or leave out repeated lines sed transform input (pattern replacement and more)VLAAMSSUPERCOMPUTER CENTRUM

Hands-on: pipelinesBuild pipelines with ps, head / tail and grep to fnd out What is the name of the frst process (PID 1)? How many processes are not owned by user root?Using the fle chemistry.txt in input.zip, and the commands wc,grep, sort, tail and uniq, answer the following: How many courses are there? Which courses are taught by Wouter Herrebout in the frst semester? Which are, in alphabetical order, the last 5 course codes starting with1001WET?alphabetically sorted by course code, oralphabetically sorted by course title Which course is listed twice?VLAAMSSUPERCOMPUTER CENTRUM

Introduction to LinuxStefan BecuweThomas DanckaertHPC core facility CalcUAVlaams Supercomputer Centrum

Inside the shell

Shell“Each time you type a command line and press the enter key, bashperforms several processes upon the text before it carries out yourcommand. The process that makes this happen is called expansion.”1.2.3.4.5.6.7.8.Brace expansion (e.g. {my,your}file - myfile yourfile)” ” expansion (e.g. cd cd /home/username)Variable expansion (e.g. HOME /home/username)Arithmetic expansion (e.g. ((2 2)) 4)Command substitution (e.g. (pwd) /path/to/this/dir)Word splitting (e.g. grep 1e semester vs grep '1e semester')Filename expansion (e.g. ls file.* ls file.txt file.jpg)Quote removal

ShellTilde ( ) expansion: echo § your own home directory echo user2§ user2’s home directoryFilename expansion: wildcards are expanded into matching file names echo * § * is expanded (non-hidden files in current directory) before echo isexecuted. echo /.[a-z]*

ShellArithmetic expansion: ((expression)) result of expression echo ((10 5 3)) § arithmetic expression (only integers!)§ operators: additionsubtraction*multiplication/integer division%remainder**exponentiation§ single parentheses may be used to group multiple subexpressions: echo (( (5**2) * (3*4) ))

Shell {} brace expansion§ List: echo Front-{A,B,C}-Back§ Sequences: echo {Z.A} mkdir {07.09}-0{1.9} {07.09}-{10.12}§ Nested: echo a{A{1,2},B{3,4}}b

Shell Variable expansion: variable name variable’s current valueOptional {}: {variable name} echo USER set# display all variables echo SUER# what if variable doesn’t exist? echo {USER} home echo USER home # doesn’t work without {}! myvar 'Hello, world!’# set a variable echo myvarCommand substitution: expand (command) to output of command echo We are now (date) echo I see (ls –A wc -l) files and subdirsequivalent – but old-fashioned: echo I see ls -A wc -l files and subdirs

Shell: escapes & quotes echo The total is 100.00# ?! Use “escape” character \ for literal use of special characters echo The total is \ 100.00 Text inside double quotes "": special characters lose their meaning,EXCEPT , \ and touch "two words.txt" ls –l two words.txt ls –l "two words.txt" ls –l two\ words.txt echo " USER ((2 2)) (cal)" echo "The total is \ 100.00"

Shell: escapes & quotesNo expansion at all inside single quotes, compare: echo text /*.txt {a,b} (echo foo) ((2 2)) USER echo "text /*.txt {a,b} (echo foo) ((2 2)) USER" echo 'text /*.txt {a,b} (echo foo) ((2 2)) USER' Word splitting: words separated by space become separate arguments ls my directory ls 'my directory' Quote removal: after all expansions, but before executing the command,quotes are removed. echo "hello world"§ unless you escape or quote the quotes echo \"hello\" '"world"'

Hands on: find The command find can search files on your file system based onvarious criteria (see reference, or man find ). Build pipeline with find to count the total number of files anddirectories in your home directory (and its subdirectories). Show it the result using echo. i.e. print a message like/home/user contains x files and directories. How to count regular files and directories separately?

The environment

Environment Recall: shell has variables§ set value for variable myvar: myvar some value# no spaces around ‘ ’§ get myvar’s value (“variable expansion”): echo myvar “Plain” variables: only exist in the shell itself set# display all variablesEnvironment variables are special: passed on to processes startedfrom the shell. export myvar # make myvar an environment variable# display environment variables printenv Environment variables are another way to influence the behaviour ofprograms (e.g. OMP NUM THREADS).

EnvironmentEDITORThe name of the program to be used for text editing.SHELLThe name of your shell program.HOMEThe pathname of your home directory.LANGDefines the character set and collation order of your language.PWDThe current working directory.OLDPWDThe previous working directory.PATHA colon-separated list of directories that are searched when you enter thename of a executable program.PS1Prompt String 1. This defines the contents of your shell prompt.USERYour user name.TMPDIRDirectory for temporary files

Environment Example: access environment variables from a Python script: python -c 'import os print "hi there,", os.getenv("USER"), "!" '

Environment Persistent settings for your environment:§ Applied once at login: /etc/profile(system wide, for all users) /.bash profile /.bash login /.profile§ Applied every time you start a shell: /.bashrc see also bash manual page under “invocation” You can also define your own aliases and functions here.

Writing shell scripts

Shell scripts shell script text file containing a series of commands Example file “myscript.sh”my analysis input.data my results/science.txttar -cvzf my results.tar.gz my resultsrm input.data bash myscript.sh Commands are executed one after the other, just as if you entered themmanually Commands are separated by new lines, or by ‘;’

Editing text Editors available on (almost) any Linux system, can run inside terminal:§ nano – simple editor open (“read”) Ctrl-r save (“write out”) Ctrl-o exit Ctrl-x § vi – the default Unix editor. Takes some practice. vim: “vi improved”, run vimtutor for a quick tutorial§ emacs – more advanced editorOthers§ gedit – GNOME text editor§ notepad – available for Windows (https://notepad-plus-plus.org)§ TextEdit – comes with macOS (use “plain text” format for scripts)§

Editing text Remark: line endings are encoded differently under Windows and Unix.This might introduce some problems for text files (especially job scripts).If you created your script file in a Windows environment, we advise toconvert your ”Windows style” (“carriage return line feed”) file into a“Unix style” (”newline”) file in the following way: dos2unix filename Beware: filename will be overwritten! A suitable text editor can do this as well

Shell scripts cat scripts/script01.sh“shebang”#!/bin/bash# This is our first script.echo 'Hello World!'# comment bash script01.sh# call the interpreter (bash) ourselves chmod x script01.sh script01.sh ./script01.sh# doesn’t work because work dir is not in PATH!# the interpreter from the ‘shebang’ is used

Shell scripts #! is called “shebang”. It tells the system which interpreter shouldexecute the script. For a bash script:#!/bin/bash This works for any scripting language, not just bash. Example forpython:#!/usr/bin/python§ or when using Python from a module:#!/usr/bin/env python uses the first python found in PATH

Shell scripts#!/bin/bash# script02currenttime (date "%x %r %Z")myname USERecho "id: myname, current time: currenttime" Remarks:§ user variables can not start with a digit: 1, 2, are specialvariables (command line arguments, see later)§ setting a variable: without , e.g., myname value § Variable expansion: with , e.g., echo myname

Checking commands How to detect and handle errors in a script?A finished command has an exit status. Convention:§ succes exit status 0§ error exit status non-zero (status values differ for each command)The special variable “?” holds the last process’ exit status: ls existing fileexisting file echo ?0 ls missingls: cannot access missing: No such file or directory echo ?2 echo ?0

ifif ls file.txtthen echo "That file exists."else echo "That file doesn't exist."fiif test 1; then commands 1elif test 2; then commands 2elif else commands nfi

if Most frequently used command with if isif test expressionor its equivalent formif [ expression ] bash has an extended replacementif [[ expression ]]which is easier to use, e.g. in combination with variables

if – test expressions#!/bin/bashx 5if [ x -eq 5 ]; thenecho "x equals 5."elseecho "x does not equal 5."fi(equivalent to:)if test x -eq 5; then if [[ x -eq 5 ]]; then # script04

test expressions: filesfile1 –nt file2file1 –ot file2-d file-f file-s file-L file-r file-w file-x file file1 is newer than file2file1 is older than file2file exists and is a directoryfile exists and is a reg

VLAAMS SUPERCOMPUTER CENTRUM Command line arguments Interpreted by the command itself usage depends on the command Order of arguments often doesn't matter. Convention: options frst, non-option arguments last. Short options can be combined, i.e. date -R -u date -Ru For some commands, strict rules apply, e.g. find Meaningof arguments Non-option argument: often a fle name