This is an old revision of the document!
Table of Contents
Linux survival guide
Why Linux?
Linux is the de facto operating system for scientific computing thanks to its high performance and extensive customisability, and most of the software used for processing and analysing neuroimaging data runs under Linux. There are good reasons for this - Linux is fast and powerful (all of the 500 fastest supercomputers in the world run Linux and most CGI effects in films are made on Linux computers), widely used (major online providers like Amazon and Google run Linux and 96% of the top 1 million most visited websites are run by Linux, as are 85% of smartphones - Android being a derivative of Linux) and highly customisable - for example, NASA uses it to fly helicopters on Mars!
Equally importantly, as the premier open source project in the world, Linux embodies the spirit of open science - free and accessible for anyone, without being reliant on licensing and subscription fees or forcing users to be tied into a specific software or hardware ecosystem. Even if you use open source software on Windows or MacOS, you are still reliant on closed source, commercial software to underpin anything you do. Linux frees you from such constraints, and is also much more secure than Windows and MacOS, so you don't in general need to worry about viruses or malware.
However, Linux can seem an unfamiliar environment for individuals used to working on Windows or Mac. Below are some tips to quickly master essential Linux skills necessary for running most neuroimaging software.
Linux basics
Linux was initially created as a hobby project by the Finnish software engineer Linus Torvalds in 1991. The name Linux is a portmanteau of his first name and Unix, the OS from which Linux is derived (in fact, Linux is a version of Unix). Very important note: Linux is pronounced as in Finnish Swedish (Linus Torvalds being part of the Swedish-speaking minority in Finland), with a long i as in “bee” and long u as in “book”: lee-nooks, not line-axe: https://upload.wikimedia.org/wikipedia/commons/0/03/Linus-linux.ogg
Unlike Windows or MacOS, there are many varieties of Linux, so-called distributions, which all share the same fundamental architecture and “kernel” (the program that actually runs the OS), but differ in their look and feel, the software included, and ease of use. The reason Linux is so widely used is in part due to the ease of modifying Linux to create a distribution suitable for a particular application (e.g., small and lightweight for low-power computing devices, fast and working in real time for time critical applications such as rocket launches or flying helicopters on Mars, or with lots of bells and whistles to make them user friendly for desktop use). The most common distributions, such as Ubuntu and Debian, are geared towards user-friendliness and will look fairly familiar to users of Windows and Macs used to a “point-and-click” interface, while the more specialised ones often make do without any graphical user interface at all, instead using a terminal window only (more on that below).
Logging in to a remote Linux server
UNDER CONSTRUCTION
*SSH
SSH stands for Secure Shell. It is a command-line protocol which provides strong password authentication and public key authentication, as well as encrypted data communications between two computers connecting over an open network, such as the internet. To be able to connect to the server psycomp.rhul.ac.uk over SSH, open the command-line terminal and use the following command:
ssh[user_ID]@psycomp.rhul.ac.uk
*RDP
RDP stands for Remote Desktop Protocol. The psycomp.rhul.ac.uk can be accessed for GUI (Graphical User Interface) tools on the cluster. This node is provided with high-performance graphics drivers for this. This psycomp.rhul.ac.uk can be accessed directly while operating from a local PC on campus and for off-campus access is granted only through VPN (details of VPN access is provided here). After connecting to the VPN use the Remote Desktop Connection software (Windows RDP / Ubuntu Remmina) and connect to the psycomp Server( server address: psycomp.rhul.ac.uk ). Use the [user_ID] and password and domain as cc to log on to the server.
*VNCViewer
Open the VNCViewer app and enter psycomp.rhul.ac.uk:[VNC_session_number] in the VNC server dropdown box. The [VNC_session_number] is an unique session code that you should have received from the IT team (If you haven't been provided one, please email the psychology IT Team Can.Keles@rhul.ac.uk). After clicking Connect enter your VNC password and log-on to the cluster.
The Linux desktop
There are a variety of desktops available on Linux distributions but most common distributions (such as Ubuntu or Debian) have a desktop that looks fairly familiar to a Windows or Mac user - there will be a menu bar at the top or the bottom of the screen which you can use to launch different programs, and a desktop with some folders in it that you can click on to open and navigate through the file system. For our purposes we will focus upon the Plasma KDE desktop.
The Linux terminal
Although modern Linux versions have a look and feel that is similar to Windows or Mac, a lot of the power of Linux resides in the use of the terminal - the software tool used for running programmes from the command line. Almost everything in Linux can be done from the command line, and although it can seem daunting and cumbersome at first to have to enter commands by typing rather than pointing and clicking you will soon realise that it is both much faster and allows automating (scripting) tasks that would take a very long time to do using the mouse. Once you get the hang of it it is straightforward to write scripts and even programmes without having to be a seasoned software engineer.
Below are some useful tips and tricks for using the terminal that will help you use its full potential.
The shell
The terminal window runs a program called the shell to interpret the commands you write. There are different types of shell programs but the most common ones that you'll encounter are sh or bash, or less commonly, csh. The information below refers to bash, which is the standard in most modern Linux distributions.
Special characters
The command prompt:
The terminal prints a “prompt” at the beginning of a line when it expects command input. This is typically of the format “username@computer:current folder$”, so if you are logged into the computer psych01 and your user name is jonas and you're in the folder /MRIWork04, the prompt would look like this:
jonas@psych01:/MRIWork04$ # Change to home folder - note you only input "cd" below jonas@psych01:/MRIWork04$ cd jonas@psych01:~$
In this guide, the command prompt is not shown in any of the examples below to make it easy for you to copy and paste commands into the terminal.
#: In the terminal, any line that starts with a hash character # is treated as a comment and ignored.
# You can write anything after a hash character, it is ignored by the shell
&: The ampersand character & after a command means “run the program in the background”. Normally, when you run a program that doesn't immediately return a result (for example if you open a browser by typing
firefox
the terminal is locked (you can't enter any new commands) until that command has finished (in this case, when you close the browser window). If instead you type
firefox &
the terminal will keep the browser running in the “background” so you can keep using the terminal window without having to wait for the browser window to close (it will only stop running when you close it yourself, or you logout.
If you've started a program without the & character you can put it in the background by pressing CTRL+Z - this will halt (suspend) the program. You can then start it again as a background process by typing
bg
which means “run in background”. If for some reason you want to return the program to run in the “foreground”, so your terminal is blocked, just type
fg
CTRL+C: if you are running a program in foreground mode, you can stop it by pressing CTRL+C - it will stop executing more or less immediately and not gracefully (i.e. it wont save any files or make a controlled exit; it will just stop).
/ The forward slash is used to describe the location of files and folders and is used to delimit folders. In Linux the top-level directory is / and directories below are located by relation to this top level directory. For example, programs commonly used by users (rather than the system itseld) will be in a folder called bin which is located within a subfolder of the top-level directory called usr. The full path to this folder is therefore written
/usr/bin
. A single period (.) means “here” - the current directory. So if you're say copying a file from /usr/local/games/mygame to where you are located, you would write
cp /usr/local/games/mygame .
.. Two periods mean “the folder above your current location”. So if your current location is /home/jonas, .. would mean /home. This is especially useful when changing directories, for example:
# Change directory to the folder above cd ..
~ The tilde means your home directory. If your user name is jonas, your home directory will be /home/jonas, but you could use ~ instead.
# List contents of home directory ls /home/jonas # List contents of home directory ls ~
Wildcards
You often want to be able to find or list files of a particular type or name but don't want to have to type out each file individually. What you can use then is what is known as wildcards - special characters that can stand for any text. To make this clearer, suppose you want to list all files that end with the extension IMA in your current folder (this is how the raw data from the scanner is stored). If you go into the directory /MRIWork/MRIWork04/jl/msc2425/functional/20250218/20030215AAPU/scan01 and type ls:
cd /MRIWork/MRIWork04/jl/msc2425/functional/20250218/20030215AAPU/scan01 ls
you will see the following:
20030215AAPU.MR.HOLLOWAY_JONAS.0001.0001.2025.02.18.12.47.03.0.136922180.IMA 20250218_120320s001a1001A.hdr 20250218_120320s001a1001.hdr 20030215AAPU.MR.HOLLOWAY_JONAS.0001.0003.2025.02.18.12.47.03.0.136922202.IMA 20250218_120320s001a1001A.img 20250218_120320s001a1001.img
Suppose you just want to list the IMA files. You can then type
ls *IMA
and the output will now be
20030215AAPU.MR.HOLLOWAY_JONAS.0001.0001.2025.02.18.12.47.03.0.136922180.IMA 20030215AAPU.MR.HOLLOWAY_JONAS.0001.0003.2025.02.18.12.47.03.0.136922202.IMA
The asterisk * is a wildcard - it means “match any character” so *IMA means “any file that ends with IMA”. You can use multiple wildcards, for example if you type
ls *s001a*
you will see
20250218_120320s001a1001A.hdr 20250218_120320s001a1001A.img 20250218_120320s001a1001.hdr 20250218_120320s001a1001.img
- that is, every file whose name contains the string “s001a”. Wildcards can be very handy when selecting files in scripts, for instance, or if you want to move large numbers of files of a particular type or name from one location to another much more quickly and efficiently than clicking and dragging files on a desktop, for example.
Most of the time you only need to use the * wildcard, which means “any numbers of characters” (so in the above example, *IMA would match both m.IMA and mmmmmm.IMA). Sometimes though you might want to use a wildcard for just a single character - then you can use a ? (question mark) which means “any single character”. For instance, assume there are two sets of files in your folder, one set named m1.IMA, m2.IMA, m3.IMA, .. m9.IMA, and another named mm1.IMA, mm2.IMA,…mm9.IMA. If you want to list only the files with a single m, you couldn't use the * (because m*IMA would list both m1.IMA… and mm1.IMA..). Instead you could write
ls m?.IMA
which means “list the files whose names begins with m, followed by an arbitrary single character, followed by .IMA” and this would only list m1.IMA, m2.IMA.. etc but not mm1.IMA etc.
Command and file name completion
If you don't remember the exact name of a command, start typing the first letter(s) of the command and press the Tab key - this will list all the commands that begin with those letters - if nothing appears then there is no command that matches. If you start typing a command followed by a space, you will instead be shown the files in the current directory - if there are many files a prompt will appear saying “Display all XX possibilities? (y or n)” and if you press y the files are shown one page at a time (there will be a prompt that says “–More–” at the bottom) and by pressing the space bar you can step through them. You can start typing the first letters of a filename, and it will complete that too, so lets say you have two files called myfile1 and myfile2 in your folder, if you type
ls myf
and press the Tab key rather than Enter, you will hear a beep and a list will be shown of the two files, like so:
ls myf # pressing Tab myfile1 myfile2 ls myf
Command history
Use the up and down arrows on the keyboard to step through past commands, and hit enter to re-run them (or Ctrl-C to quit).
To list recent commands, type
history
which will produce a list of the last (normally around 1000) commands preceded by a number. To re-run any of these command, just type an exclamation mark followed by the number, e.g. if the command numbered 1500 was ls -ltRr (a recursive full listing of all directoris and subdirectories in reverse time order), typing
!1500
will have the same effect as re-typing
ls -ltRr
Often it is handy to use the history function to search for a particular command, e.g. you might not remember how you connected to another computer by ssh, so you want to search all commands including ssh. For this you can use the grep command, which searches for a string:
history | grep ssh
The vertical bar | is a pipe - this means that the output from the first command (history) is sent as input to the following command (grep) rather than to the terminal output. Pipes are very useful constructs in Linux that allow you to combine multiple commands into one, whcih can be very handy. For example, the following command will find all instances of ssh in the command history, and then print them out sorted according to the name of the server (the third field of the output of history) in reverse order - here, the output of history is piped to grep, which in turn is piped to sort:
history |grep ssh|sort -k 3,3 -r
Text editors
Because so much in Linux is done by manipulating text files (as well as writing code), you will often need to use a program to edit text - a text editor. Standard word processing programs are unsuitable for this purpose as they save their output in a non-text format and are not intended for writing simple text commands (or computer code). Much better is to use text editors that are designed for this specific purpose. Two of the most popular editors in Linux are emacs and vi (or vim). Emacs is a lot easier to use and has a more intuitive interface - vi seems to be designed with the express purpose of frustrating novice users and tends to lead to a lot of beeping and error messages. Which one you prefer is a personal decision; they both accomplish the same task.
To run them, just type
emacs
or
vi
or
vim
Two more modern and user-friendly editors are pluma and gedit - these may or may not be installed on your system, but try:
pluma
or
gedit
Useful commands & programs
cd - change directory
Syntax: cd <path to directory>
cd /home/jonas # Move up one directory cd .. # Move to home directory cd
pwd - show current directory
Syntax: pwd
cd /home/jonas pwd /home/jonas
ls - list files
Syntax: ls [<directory or file names>]
# List files in current directory ls # List files in /home/jonas ls /home/jonas # List files in current folder in long format showing size and ownership ls -l # List files in reverse order by time (last first) ls -tr # List files in long format and reverse order by time (last first) ls -ltr # List files recursively in current folder and subfolders ls -R # List files beginning with myf ls myf* # List files ending with ile ls *ile # List files containing the string yfi ls *yfi*
mv - move or rename files
Syntax: mv <input file> <output file name or directory>
# Rename a file in current directory mv myfile1 myfile2 # Move a file to the directory above mv myfile1 .. # Move a file to /home/jonas mv myfile1 /home/jonas
cp - copy files
Syntax: cp <input file> <output file name or directory>
rm - remove files
Syntax: rm <file/s>
# Remove a single file rm myfile1 # Remove multiple files matching pattern myf* rm myf* # Remove a folder and all of its contents (beware!) rm -rf myfolder
mkdir - make a directory (folder)
Syntax: mkdir <new folder name>
# Make a new folder in current directory mkdir mynewfolder # Make a new folder in another directory mkdir /home/jonas/mynewfolder # Make a new folder containing a subfolder mkdir -p /home/jonas/mynewfolder/mynewsubfolder
rmdir - remove a directory
Syntax: rmdir <directory name>
# Remove an empty folder (folders with contents need to be removed with rm -rf, see above) rmdir myfolder
more and cat - show the content of text files
Syntax: more <filename> cat <filename>
# List contents of a text file, one page at a time (step through using space, press q to quit) more myfile1 # List full contents of a text file cat myfile1
find - find files
Syntax:
grep - find text in a file
Syntax:
sort - sort text
Syntax:
man - show help for a program
Syntax:
ps, top - show CPU usage and running processes
Syntax:
kill, pkill, xkill - stop a program or process
If you come from Windows, you will be familiar with the task manager which can be handy when wanting to stop a program that has crashed or is non-responsive. Linux programs tend to be more stable than Windows ones, but you might still from time to time need to stop an out of control program. Unlike the Windows task manager, which seems to meekly ask programs whether they might be so kind as to stop and can take a while to have any effect, the Linux equivalents show no mercy and are instanteneous and uncompromising - the program or process will stop instantly without saving files etc. Beware that if you kill a process used by the system, you could cause unintended consequences for yourself (e.g. if you stop the window system X). However, the inherently secure nature of Linux means you cannot affect the system as a whole or other users, and logging out and logging in will bring you back to a normal state. The bottom line is, in general don't kill processes you don't know what they are or do.
# To kill a program if you know it's process number (see above), use kill -9: kill -9 <process number> #To kill a program whose name you know, use pkill -9: pkill -9 firefox #To kill a window program you can use xkill; you will then need to click the window of the program you want to stop: xkill
ssh - remote login to another computer
Syntax: pwd
scp - copy files to another computer
Syntax:
rsync - copy large numbers of files fast and efficiently
Syntax:
The Linux file system
The home directory: /home/myusername
The root directory: /
System directories: /usr, /bin, /etc, /lib, /media
Mounting external file systems
File permissions
Recursive commands