Differences

This shows you the differences between two versions of the page.

--- linux-survival [2024/01/16 15:00] – [Linux basics] jonas
+++ linux-survival [2025/10/09 15:11] (current) – [Logging in to a remote Linux server] jonas
@@ Line 3: / Line 3: @@
 ===== Why Linux? =====
-Most of the software used for processing and analysing neuroimaging data runs under the Linux operating system (OS). There are good reasons for this - Linux is fast and powerful (all of the 500 fastest supercomputers in the world run Linux and most CGI effects in films are made on Linux computers), widely used (major online providers like Amazon and Google run Linux and 96% of the top 1 million most visited websites are run by Linux, as are 85% of smartphones - Android being a derivative of Linux) and highly customisable. It's also free and open source, making it eminently compatible with Open Science.
+Linux is the //de facto// operating system for scientific computing thanks to its high performance and extensive customisability, and most of the software used for processing and analysing neuroimaging data runs under Linux. There are good reasons for this - Linux is fast and powerful (all of the 500 fastest supercomputers in the world run Linux and most CGI effects in films are made on Linux computers), widely used (major online providers like Amazon and Google run Linux and 96% of the top 1 million most visited websites are run by Linux, as are 85% of smartphones - Android being a derivative of Linux) and highly customisable - for example, NASA uses it to fly helicopters on Mars!
+Equally importantly, as the premier open source project in the world, Linux embodies the spirit of open science - free and accessible for anyone, without being reliant on licensing and subscription fees or forcing users to be tied into a specific software or hardware ecosystem. Even if you use open source software on Windows or MacOS, you are still reliant on closed source, commercial software to underpin anything you do. Linux frees you from such constraints, and is also much more secure than Windows and MacOS, so you don't in general need to worry about viruses or malware.
 However, Linux can seem an unfamiliar environment for individuals used to working on Windows or Mac. Below are some tips to quickly master essential Linux skills necessary for running most neuroimaging software.
@@ Line 15: / Line 17: @@
 ===== Logging in to a remote Linux server =====
-  ***//SSH//**
+UNDER CONSTRUCTION
+***//SSH//**
-SSH stands for Secure Shell. It is a command-line protocol which provides strong password authentication and public key authentication, as well as encrypted data communications between two computers connecting over an open network, such as the internet. To be able to connect to the server //psycomp.rhul.ac.uk// over SSH, open the command-line terminal and use the following command:
+SSH stands for Secure Shell. It is a command-line protocol which provides strong password authentication and public key authentication, as well as encrypted data communications between two computers connecting over an open network, such as the internet. It is run from the command line as follows:
+<del>
+To be able to connect to the server //psycomp.rhul.ac.uk// over SSH, open the command-line terminal and use the following command:
@@ Line 30: / Line 36: @@
   ***//VNCViewer//**
-Open the VNCViewer app and enter //psycomp.rhul.ac.uk//:[VNC_session_number] in the VNC server dropdown box. The [VNC_session_number] is an unique session code that you should have received from the IT team (If you haven't been provided one, please email the psychology IT Team [[Can.Keles@rhul.ac.uk]]). After clicking Connect enter your VNC password and log-on to the cluster.
+Open the VNCViewer app and enter //psycomp.rhul.ac.uk//:[VNC_session_number] in the VNC server dropdown box. The [VNC_session_number] is an unique session code that you should have received from the IT team (If you haven't been provided one, please email the psychology IT Team [[Can.Keles@rhul.ac.uk]]). After clicking Connect enter your VNC password and log-on to the cluster.</del>
 ===== The Linux desktop =====
 There are a variety of desktops available on Linux distributions but most common distributions (such as Ubuntu or Debian) have a desktop that looks fairly familiar to a Windows or Mac user - there will be a menu bar at the top or the bottom of the screen which you can use to launch different programs, and a desktop with some folders in it that you can click on to open and navigate through the file system. For our purposes we will focus upon the Plasma KDE desktop.
 ===== The Linux terminal =====
+Although modern Linux versions have a look and feel that is similar to Windows or Mac, a lot of the power of Linux resides in the use of the //terminal// - the software tool used for running programmes from the command line. Almost everything in Linux can be done from the command line, and although it can seem daunting and cumbersome at first to have to enter commands by typing rather than pointing and clicking you will soon realise that it is both much faster and allows automating (scripting) tasks that would take a very long time to do using the mouse. Once you get the hang of it it is straightforward to write scripts and even programmes without having to be a seasoned software engineer.
+Below are some useful tips and tricks for using the terminal that will help you use its full potential.
+==== The shell ====
+The terminal window runs a program called the //shell// to interpret the commands you write. There are different types of shell programs but the most common ones that you'll encounter are //sh// or //bash//, or less commonly, //csh//. The information below refers to bash, which is the standard in most modern Linux distributions.
+=== Special characters ===
+The command prompt:
+The terminal prints a "prompt" at the beginning of a line when it expects command input. This is typically of the format "username@computer:current folder$", so if you are logged into the computer psych01 and your user name is jonas and you're in the folder /MRIWork04, the prompt would look like this:
+<code sh>
+jonas@psych01:/MRIWork04$
+# Change to home folder - note you only input "cd" below
+jonas@psych01:/MRIWork04$ cd
+jonas@psych01:~$
+</code>
+In this guide, the command prompt is not shown in any of the examples below to make it easy for you to copy and paste commands into the terminal.
+#: In the terminal, any line that starts with a hash character # is treated as a comment and ignored.
+<code sh>
+# You can write anything after a hash character, it is ignored by the shell
+</code>
+&: The ampersand character & //after// a command means "run the program in the background".
+Normally, when you run a program that doesn't immediately return a result (for example if you open a browser by typing
+<code sh>
+firefox
+</code>
+the terminal is locked (you can't enter any new commands) until that command has finished (in this case, when you close the browser window).
+If instead you type
+<code sh>
+firefox &
+</code>
+the terminal will keep the browser running in the "background" so you can keep using the terminal window without having to wait for the browser window to close (it will only stop running when you close it yourself, or you logout.
+If you've started a program without the & character you can put it in the background by pressing **CTRL+Z** - this will halt (suspend) the program. You can then start it again as a background process by typing
+<code sh>bg</code>
+which means "run in background". If for some reason you want to return the program to run in the "foreground", so your terminal is blocked, just type
+<code sh>fg</code>
+CTRL+C: if you are running a program in foreground mode, you can stop it by pressing CTRL+C - it will stop executing more or less immediately and not gracefully (i.e. it wont save any files or make a controlled exit; it will just stop).
+/ The forward slash is used to describe the location of files and folders and is used to delimit folders. In Linux the top-level directory is / and directories below are located by relation to this top level directory. For example, programs commonly used by users (rather than the system itseld) will be in a folder called //bin// which is located within a subfolder of the top-level directory called //usr//. The full path to this folder is therefore written
+/usr/bin
+. A single period (.) means "here" - the current directory. So if you're say copying a file from /usr/local/games/mygame to where you are located, you would write
+<code sh>cp /usr/local/games/mygame .</code>
+.. Two periods mean "the folder above your current location". So if your current location is /home/jonas, ..  would mean /home. This is especially useful when changing directories, for example:
+<code sh>
+# Change directory to the folder above
+cd ..
+</code>
+~ The tilde means your home directory. If your user name is jonas, your home directory will be /home/jonas, but you could use ~ instead.
+<code sh>
+# List contents of home directory
+ls /home/jonas
+# List contents of home directory
+ls ~
+</code>
+=== Wildcards ===
+You often want to be able to find or list files of a particular type or name but don't want to have to type out each file individually. What you can use then is what is known as //wildcards// - special characters that can stand for any text. To make this clearer, suppose you want to list all files that end with the extension IMA in your current folder (this is how the raw data from the scanner is stored). If you go into the directory /MRIWork/MRIWork04/jl/msc2425/functional/20250218/20030215AAPU/scan01 and type ls:
+<code sh>
+cd /MRIWork/MRIWork04/jl/msc2425/functional/20250218/20030215AAPU/scan01
+ls
+</code>
+you will see the following:
+<code sh>
+20030215AAPU.MR.HOLLOWAY_JONAS.0001.0001.2025.02.18.12.47.03.0.136922180.IMA  20250218_120320s001a1001A.hdr  20250218_120320s001a1001.hdr
+20030215AAPU.MR.HOLLOWAY_JONAS.0001.0003.2025.02.18.12.47.03.0.136922202.IMA  20250218_120320s001a1001A.img  20250218_120320s001a1001.img
+</code>
+Suppose you just want to list the IMA files. You can then type
+<code sh>
+ls *IMA
+</code>
+and the output will now be
+<code sh>
+20030215AAPU.MR.HOLLOWAY_JONAS.0001.0001.2025.02.18.12.47.03.0.136922180.IMA  20030215AAPU.MR.HOLLOWAY_JONAS.0001.0003.2025.02.18.12.47.03.0.136922202.IMA
+</code>
+The asterisk * is a wildcard - it means "match any character" so *IMA means "any file that ends with IMA". You can use multiple wildcards, for example if you type
+<code sh>
+ls *s001a*
+</code>
+you will see
+<code sh>
+20250218_120320s001a1001A.hdr  20250218_120320s001a1001A.img  20250218_120320s001a1001.hdr  20250218_120320s001a1001.img
+</code>
+- that is, every file whose name contains the string "s001a". Wildcards can be very handy when selecting files in scripts, for instance, or if you want to move large numbers of files of a particular type or name from one location to another much more quickly and efficiently than clicking and dragging files on a desktop, for example.
+Most of the time you only need to use the * wildcard, which means "any numbers of characters" (so in the above example, *IMA would match both m.IMA and mmmmmm.IMA). Sometimes though you might want to use a wildcard for just a single character - then you can use a ? (question mark) which means "any single character". For instance, assume there are two sets of files in your folder, one set named m1.IMA, m2.IMA, m3.IMA, .. m9.IMA, and another named mm1.IMA, mm2.IMA,...mm9.IMA. If you want to list only the files with a single m, you couldn't use the * (because m*IMA would list both m1.IMA... and mm1.IMA..). Instead you could write
+<code sh>
+ls m?.IMA
+</code>
+which means "list the files whose names begins with m, followed by an arbitrary single character, followed by .IMA" and this would only list m1.IMA, m2.IMA.. etc but not mm1.IMA etc.
+==== Command and file name completion ====
+If you don't remember the exact name of a command, start typing the first letter(s) of the command and press the Tab key - this will list all the commands that begin with those letters - if nothing appears then there is no command that matches.
+If you start typing a command followed by a space, you will instead be shown the files in the current directory - if there are many files a prompt will appear saying "Display all XX possibilities? (y or n)" and if you press y the files are shown one page at a time (there will be a prompt that says "--More--" at the bottom) and by pressing the space bar you can step through them. You can start typing the first letters of a filename, and it will complete that too, so lets say you have two files called myfile1 and myfile2 in your folder, if you type
+<code sh>
+ls myf
+</code>
+and press the Tab key rather than Enter, you will hear a beep and a list will be shown of the two files, like so:
+<code sh>
+ls myf # pressing Tab
+myfile1
+myfile2
+ls myf
+</code>
+==== Command history ====
+Use the up and down arrows on the keyboard to step through past commands, and hit enter to re-run them (or Ctrl-C to quit).
+To list recent commands, type
+<code sh>
+history
+</code>
+which will produce a list of the last (normally around 1000) commands preceded by a number. To re-run any of these command, just type an exclamation mark followed by the number, e.g. if the command numbered 1500 was ls -ltRr (a recursive full listing of all directoris and subdirectories in reverse time order), typing
+<code sh>
+!1500
+</code>
+will have the same effect as re-typing
+<code sh>
+ls -ltRr
+</code>
+Often it is handy to use the history function to search for a particular command, e.g. you might not remember how you connected to another computer by ssh, so you want to search all commands including ssh. For this you can use the grep command, which searches for a string:
+<code sh>
+history | grep ssh
+</code>
+The vertical bar | is a //pipe// - this means that the output from the first command (history) is sent as input to the following command (grep) rather than  to the terminal output. Pipes are very useful constructs in Linux that allow you to combine multiple commands into one, whcih can be very handy. For example, the following command will find all instances of ssh in the command history, and then print them out sorted according to the name of the server (the third field of the output of history) in reverse order - here, the output of history is piped to grep, which in turn is piped to sort:
+<code sh>
+history |grep ssh|sort -k 3,3 -r
+</code>
+==== Text editors ====
+Because so much in Linux is done by manipulating text files (as well as writing code), you will often need to use a program to edit text - a text editor. Standard word processing programs are unsuitable for this purpose as they save their output in a non-text format and are not intended for writing simple text commands (or computer code). Much better is to use text editors that are designed for this specific purpose. Two of the most popular editors in Linux are emacs and vi (or vim). Emacs is a lot easier to use and has a more intuitive interface - vi seems to be designed with the express purpose of frustrating novice users and tends to lead to a lot of beeping and error messages. Which one you prefer is a personal decision; they both accomplish the same task.
-  ***//The shell//**
+To run them, just type <code sh>  emacs</code> or <code sh>vi</code> or <code sh>vim</code>
-The shell is a program that takes commands from the keyboard and gives them to the operating system to perform. Earlier it was the only user interface available on a Unix-like system such as Linux. Nowadays, we have graphical user interfaces (GUIs) in addition to command line interfaces (CLIs) such as the shell.
+Two more modern and user-friendly editors are pluma and gedit - these may or may not be installed on your system, but try:
-On most Linux systems a program called **bash** (which stands for Bourne Again SHell, an enhanced version of the original Unix shell program, **sh**, written by Steve Bourne) acts as the shell program. Besides bash, there are other shell programs available for Linux systems. These include: ksh, tcsh and zsh.
+<code sh>pluma</code> or <code sh>gedit</code>
-On the psycomp server, we will use **bash** commands.
+==== Useful commands & programs ====
+== cd - change directory ==
-  ***//Terminal Emulator//**
+Syntax: cd <path to directory>
-In order to run **bash** commands, a particular tool called //Terminal Emulator// is used. On a Linux and Mac-OS systems a widely used Terminal Emulator program is called **Terminal**. Psycomp server also has **Terminal** program that can be accessed under the Applications tab.
+<code sh>
+cd /home/jonas
+# Move up one directory
+cd ..
+# Move to home directory
+cd
+</code>
+== pwd - show current directory ==
-  ***//Command-line completion//**
+Syntax: pwd
+<code sh>
+cd /home/jonas
+pwd
+/home/jonas
+</code>
-    *Writing a shell script
+== ls - list files ==
-    *Navigating the file system
+Syntax: ls [<directory or file names>]
+<code sh>
+# List files in current directory
+ls
+# List files in /home/jonas
+ls /home/jonas
+# List files in current folder in long format showing size and ownership
+ls -l
+# List files in reverse order by time (last first)
+ls -tr
+# List files in long format and reverse order by time (last first)
+ls -ltr
+# List files recursively in current folder and subfolders
+ls -R
+# List files beginning with myf
+ls myf*
+# List files ending with ile
+ls *ile
+# List files containing the string yfi
+ls *yfi*
+</code>
+== mv - move or rename files ==
-===== Essential commands =====
+Syntax: mv <input file> <output file name or directory>
+<code sh>
+# Rename a file in current directory
+mv myfile1 myfile2
+# Move a file to the directory above
+mv myfile1 ..
+# Move a file to /home/jonas
+mv myfile1 /home/jonas
+</code>
+== cp - copy files ==
-**//Finding where you are //** pwd
+Syntax: cp <input file> <output file name or directory>
+<code sh>
+</code>
+== rm - remove files ==
-After logging in the psycomp server either through the command line interface or through GUI, it will be important to know which location you areon the computer.
+Syntax: rm <file/s>
+<code sh>
+# Remove a single file
+rm myfile1
+# Remove multiple files matching pattern myf*
+rm myf*
+# Remove a folder and all of its contents (beware!)
+rm -rf myfolder
+</code>
+== mkdir - make a directory (folder) ==
+Syntax: mkdir <new folder name>
+<code sh>
+# Make a new folder in current directory
+mkdir mynewfolder
+# Make a new folder in another directory
+mkdir /home/jonas/mynewfolder
+# Make a new folder containing a subfolder
+mkdir -p /home/jonas/mynewfolder/mynewsubfolder
+</code>
+== rmdir - remove a directory ==
+Syntax: rmdir <directory name>
+<code sh>
+# Remove an empty folder (folders with contents need to be removed with rm -rf, see above)
+rmdir myfolder
+</code>
+== more and cat - show the content of text files ==
+Syntax: more <filename>
+cat <filename>
+<code sh>
+# List contents of a text file, one page at a time (step through using space, press q to quit)
+more myfile1
+# List full contents of a text file
+cat myfile1
+</code>
+== find - find files ==
-**//List files://** ls
+Syntax:
+<code sh>
+</code>
+== grep - find text in a file ==
-**//Change directory://** cd
+Syntax:
+<code sh>
+</code>
+== sort - sort text ==
-**//Rename or move files: //** mv
+Syntax:
+<code sh>
+</code>
+== man - show help for a program ==
-**//Create files and directories: //** touch, mkdir
+Syntax:
+<code sh>
+</code>
+== ps, top - show CPU usage and running processes ==
-**//Delete files and directories: //**rm, rmdir
+Syntax:
+<code sh>
+</code>
+== kill, pkill, xkill - stop a program or process ==
-**//Find files and directories: //**find
+If you come from Windows, you will be familiar with the task manager which can be handy when wanting to stop a program that has crashed or is non-responsive. Linux programs tend to be more stable than Windows ones, but you might still from time to time need to stop an out of control program. Unlike the Windows task manager, which seems to meekly ask programs whether they might be so kind as to stop and can take a while to have any effect, the Linux equivalents show no mercy and are instanteneous and uncompromising - the program or process will stop instantly without saving files etc. Beware that if you kill a process used by the system, you could cause unintended consequences for yourself (e.g. if you stop the window system X). However, the inherently secure nature of Linux means you cannot affect the system as a whole or other users, and logging out and logging in will bring you back to a normal state. The bottom line is, in general don't kill processes you don't know what they are or do.
-**//Check CPU usage: //**top, ps
+<code bash>
+# To kill a program if you know it's process number (see above), use kill -9:
+kill -9 <process number>
-**//Text editing: //**pluma, kate, nano
+#To kill a program whose name you know, use pkill -9:
+pkill -9 firefox
-**//PDF viewer: //**atril
+#To kill a window program you can use xkill; you will then need to click the window of the program you want to stop:
+xkill
+</code>
-**//Desktop applications: //**LibreOffice, Firefox
+== ssh - remote login to another computer ==
-**//Common neuroimaging applications://** FSL, Matlab, AFNI
+Syntax: pwd
+<code sh>
+</code>
+== scp - copy files to another computer ==
-**//copying files and directories: //** cp, scp
+Syntax:
+<code sh>
+</code>
+== rsync - copy large numbers of files fast and efficiently ==
-**// Getting help: //** man
+Syntax:
+<code sh>
+</code>