The Linux Command Line & Shell Scripting - UT Southwestern

1The Linux Command Line & Shell -help@utsouthwestern.eduUpdated for 2017-11-18

Study Resources : A Free Book500 pages* Some of the materials covered in today’s training is from this book2

Study Resources : Tutorial Website3

Study Resources : Cheat SheetsOn the portalTraining - Slides & Handouts4

Study Resources: Man pagesGet a command’s help page: man command Press q or Ctrl-C to exit the man page5

Study Resources : Follow Along You can follow along using:1. the Nucleus Web terminal on the BioHPC h/2. Putty or SSH from your PC3. Terminal from your MacBookssh username @nucleus.biohpc.swmed.edu6

The Shell and Command Line InterfacesThe interaction between user and the operating system is provide by a shellGraphic User Interfaces & Command Line InterfacesBy far the most common shell on Linux is bash (Bourne Again SHell)It has a lot of built-in commandsuser nameargumentsmachine namecurrent directoryoptions/switchesname of the command7

Linux Basics: The File SystemFiles on a Linux system are arranged in a hierarchical directory structure./appsbintmphome2{uid} projectwork{department} shared shared {department}shared {PI lab}{uid}shared {uid or project name}8shared

Linux Command Line: Files and DirectoriesRelative path v.s. Absolute path (full path)/home2ydu (I am here)Training Relative path— path related to the presentworking directory (pwd)If my current directory is /home2/ydu, relative path is:Trainingpwd will return : /home2/yduIf change directory to home2, relative path becomesydu/Trainingpwd will return: /home2 Absolute path—specify the location of afile or directory from the root directory (/)/home2/ydu/Training9

Linux Command Line: Files and DirectoriesEditorsvi / vimCryptic commands! Cheat sheet on the portal.Quick tutorial: acsAn extensible, customizable text editorQuick tutorial: http://www.gnu.org/software/emacs/tour/nanoEasier to use.Quick tutorial: http://mintaka.sdsu.edu/reu/nano.htmlAny text editor from your PC or MacMount your directories as network es/biohpc-cloud-storage/10

DemoExample 1: Manage Files/FoldersExample 2: WildcardsExample 3: Disk UsageExample 4: Create Shared Folder/FilesExample 5: Text File Manipulation with Linux Tools(view/find patterns/sort/replace)as important as knowing how to use Linux itselfpopular and convenient in BioData processing and Analysis11

Example 1: Manage Files/FoldersCommandWhat does it do?pwdPrint Working Directory – where am I?lsList files in current directoryls -lahList files with detail (l long format, a all files including hidden, h human readable file sizes)cd TrainingChange directory to today’s training foldermkdir example1Make a directory called example1touch example1/newfile.txtCreate an empty new file inside example1 folderls –lah example1List files inside the example1 directorymkdir new exampleMake the directory new examplecp example1/newfile.txt new example/Copy newfile.txt from example1 into new examplecp –r example1 newer exampleCopy example1 recursively (directory and everything inside) to adirectory called newer example)12

Example 1: Manage Files/FoldersCommandWhat does it do?mv new example old exampleMove/rename new example directory to old examplermdir old exampleTry to remove old examplecd old exampleChange directory into old examplels -alhList files with detailcd .Change directory to parent folderrmdir old exampleDelete an empty folderrm –r newer exampleRecursively delete all files and folders insidenewer example13

Example 2: Wildcards*Match any number of characterslslslsls?*notes**.txt*2015*Match a single characterls data 00?.txt[]Any fileAny file beginning with notesAny file ending in .txtAny file with 2015 somewhere in its nameMatches data 001, data002, data 00A etc.Match a set of characters (bracket expression)ls data 00[0123456789].txtls data 00[0-9].txtMatches data 001 – data 009, not data 00A14

Example 3: Disk Usagequota: display disk usage and limits (How much space do I have)15

Example 3: Disk Usagedu: estimate file space usage (How much space am I taking up)Total folder size:Detailed usage for each sub-folders:16

Example 3: Disk UsageArchive files and folders will reduce both space usage and number of filesWe plan to limit the number of files for each user in the future.mysandybox mysandybox.tar.gzSpace usage : 0.07T 0.02TNumber of Files : 306 1317

Example 3: Disk UsageCommandWhat does it do?gunzip mydata.txt.gzExtract a .gz gzipped filetar xvf archive.tarExtract (x) everything from a tar archive file (f). Show verbose output (v).cd .Change directory into parent directory (CLI)du –sh example3Estimate space usage of folder example3cd example3Change directory into parent directory example3tar cvzf new.tar.gz file1 folder1Create (c) a compressed tar.gz archive file (f) adding files and folders to it.gzip mydata.txtCompress a file using gzip – becomes a .gz filecd .Change directory into parent directory (CLI)du –sh example3Estimate space usage of folder example3tar zxvf example3/archive.tar.gzExtract a compressed gzipped tar archive in one step18

Permissionsls -ldrwxr-xr-x 4 dtrudgian biohpc admindrwxr-xr-x 7 dtrudgian biohpc admindrwxr-xr-x 2 dtrudgian biohpc admindrwxr-xr-x 8 dtrudgian biohpc admindrwxr-xr-x 3 dtrudgian biohpc admindrwxr-xr-x 2 dtrudgian biohpc admindrwxr-xr-x 9 dtrudgian biohpc admin-rw-r--r-- 1 dtrudgian biohpc admindrwxr-xr-x 10 dtrudgian biohpc admindrwxr-xr-x 17 dtrudgian biohpc admindrwxr-xr-x 2 dtrudgian biohpc admindrwxr-xr-x 5 dtrudgian biohpc 1014:3215:1612:4515:1714:1811:19all trainingAppscli trainingCluster oudPicturesportal jobs

File Permissions20

Octal Permissionsrwx 421-rw-r--r--62144Add up the permissions you need for each class, e.g.rx 5rw 6rwx 71 dtrudgian biohpc adminOwner can read writeGroup can readOthers can read336 Feb 16 15:16 error.txt

PermissionschmodClass:u user (owner)g groupa allu/g/a -chmod g rw test.txtchmod a x script.shchmod g-x script.shchmod 700 script.sh -rwx-----chmod 640 script.sh –rw-r----22 /-r/w/xAdd permissionRemove permissionfilenamerwxreadwriteexecuteAdd read/write permissions for the groupAdd execute permission for everyoneRemove execute permission for the group

Example 4: Create Shared FolderCommandWhat does it do?cd /project/biohpcadmin/shared/Change directory to your lab’s shared folderls -alList files with detail to check permissionmkdir shared dataCreate shared foldercd shared dataChange directory to newly created shared data foldercp /project/biohpcadmin/ydu/ideas.txt .Copy file to shared folder, you group members can read and copy itchmod g w ideas.txtGrant write permission to allow group members edit the ideas.txtcd .Change directory to parent folderchmod g w shared dataGrant write permission to allow group members create new filesinside shared data folder 23You can only apply chmod command to the files/folders owned by you.Send email to biohpc-help@utsouthwestern.edu if you need help to set up a shared folder

Example 5: Text File ManipulationCommandWhat does it do?cat story.txtShow the content of story.txt in the terminalless story.txtShow the content of story.txt in a pager that allows scrolling & searchinghead –n 15 story.txtShow the first 15 lines of story.txttail –n 5 story.txtShow the last 5 lines of story.txthead –n 15 story.txt tail –n 5Shows lines 11-15 of story.txt. We take the first 15 lines from story.txt usinghead, and extract the bottom of that selection by piping it through tail.grep "elephant" story.txtFind all lines in story.txt that contain “elephant” case-sensitive.grep –i "mouse" story.txtFind all lines in story.txt that contain “mouse” non-case sensitive.grep –c "at" story.txtCount the number of lines in story.txt that contain “at”.grep “ The” story.txtFind lines beginning with “The”grep “water\. ” story.txtFind lines ending with “water”. Note that ‘.’ is a special character so we haveto escape it.grep searches for patterns using regular expressions - http://www.robelle.com/smugbook/regexpr.html24

Example 5: Text File ManipulationCommandWhat does it do?sort fruits.txtSort the lines in a file alphabeticallysort -r fruits.txtSort the lines in a file in reverse alphabetical ordersort –nr numbers.txtSort the lines in a file in reverse numerical ordersort fruits.txt sorted fruits.txtSend the output of the sort command into sorted fruits.txt overwritingexisting content.sort fruits.txt sorted fruits.txtSend the output of the sort command into sorted fruits.txt, appending tothe end of the file.sort fruits.txt 2 error.txtSend the error output of the sort command into error.txtsort fruits.txt 2 /dev/nullDiscard the error output of the command ‘which nonsense’sort fruits.txt 2 &1 output.txtCombine the error output into the standard output, and direct into the fileoutput.txtsed "s/dog/cat/g" story.txtChange all instances of cat to dog, printing the results(substitute)/old/new/g(lobal)sed can do a lot - http://www.grymoire.com/Unix/sed.htmlawk can do even more - http://www.grymoire.com/Unix/Awk.html25

Bash Shell ScriptingIn addition to the interactive mode, bash shell script is a programing language in itselfone command at a timerun an entire script of commandsA little knowledge can make difficult things easy, and time-consuming things quick. 26

Bash Shell Scripting : Variablesscript varables.shA variable holds information to be used laterSet them using:Get their value using:name value name#!/bin/bashSpecify we are using bash shellMY NAME daveecho Hello MY NAMESet the variable MY NAMERun echo using the value in MY NAME27

Bash Shell Scripting : How to runHow to run a bash script Method 1: convert scripts into an executable filechmod x script variables.sh./scripts variable.sh Method 2: use bash or shbash script variables.sh-orsh script variables.sh28

Bash Shell Scripting : Input/Outputscript IO.shWe can assign the output of programs/commands into variables:#!/bin/bashNOW (date %Y-%m-%d)Put the date (YY-MM-DD) into NOWDATA DIR "data NOW"mkdir DATA DIRCreate a directory with a nameincorporating the datemodule load matlabmatlab –nodisplay -nosplash hello.m DATA DIR/output.txtRun matlab with input hello.m, and send the output into our data directory29

Bash Shell Scripting : If-else Statementscript if.sh#!/bin/bashecho "This scripts checks for the dummy file."echo "Checking."file exists test is -fif [ -f "dummy" ]; thenecho "dummy exists."elseecho "Could not find it! "fiRead about the available tests sect 07 01.html30

Bash Shell Scripting : Loopsscript loops.shThis script lists all of the files in the current directory using a for loop:#!/bin/bashSet variable i to each value in the output of the ls commandfor i in ( ls ); doecho "item: i"Echo the current value of idoneRead about the other loops html31

Bash Shell Scripting : Pass Command Line Argumentsscript arguments.sh#!/bin/bashecho "Script Name: 0 "echo "Total Number of Argument s: #"echo "1st Argument: 1"echo "2nd Argument: 2 "echo "All Arguments are: *" * – Store all command line arguments # – Store count of command line arguments 0 – Store name of script itself 1 – Store first command line argument 2 – Store second command line argument 3 – Store third command line argument 32

Bash Shell Scripting : Reorganize data to fit the software requirementImages generated byTotal Internal Refraction Fluorescence MicroscopeInput data structure required by DeBiasTEST INPUT1 101 11TEST OUTPUT1 12img 000000000 TIRF 488 000.tifimg 000000000 TIRF 561 000.tif33folder reorganize.sh1488.tif23561.tif