Differences

This shows you the differences between two versions of the page.

--- linux-survival [2023/03/27 14:08] – ayan
+++ linux-survival [2025/10/09 15:11] (current) – [Logging in to a remote Linux server] jonas
@@ Line 3: / Line 3: @@
 ===== Why Linux? =====
-Most of the software used for processing and analysing neuroimaging data runs under the Linux operating system (OS). There are good reasons for this - Linux is fast and powerful (all of the 500 fastest supercomputers in the world run Linux and most CGI effects in films are made on Linux computers), widely used (major online providers like Amazon and Google run Linux and 96% of the top 1 million most visited websites are run by Linux, as are 85% of smartphones - Android being a derivative of Linux) and highly customisable. It's also free and open source, making it eminently compatible with Open Science.
+Linux is the //de facto// operating system for scientific computing thanks to its high performance and extensive customisability, and most of the software used for processing and analysing neuroimaging data runs under Linux. There are good reasons for this - Linux is fast and powerful (all of the 500 fastest supercomputers in the world run Linux and most CGI effects in films are made on Linux computers), widely used (major online providers like Amazon and Google run Linux and 96% of the top 1 million most visited websites are run by Linux, as are 85% of smartphones - Android being a derivative of Linux) and highly customisable - for example, NASA uses it to fly helicopters on Mars!
+Equally importantly, as the premier open source project in the world, Linux embodies the spirit of open science - free and accessible for anyone, without being reliant on licensing and subscription fees or forcing users to be tied into a specific software or hardware ecosystem. Even if you use open source software on Windows or MacOS, you are still reliant on closed source, commercial software to underpin anything you do. Linux frees you from such constraints, and is also much more secure than Windows and MacOS, so you don't in general need to worry about viruses or malware.
 However, Linux can seem an unfamiliar environment for individuals used to working on Windows or Mac. Below are some tips to quickly master essential Linux skills necessary for running most neuroimaging software.
@@ Line 11: / Line 13: @@
 Linux was initially created as a hobby project by the Finnish software engineer Linus Torvalds in 1991. The name Linux is a portmanteau of his first name and Unix, the OS from which Linux is derived (in fact, Linux is a version of Unix). **Very important note:** Linux is pronounced as in Finnish Swedish (Linus Torvalds being part of the Swedish-speaking minority in Finland), with a long i as in "bee" and long u as in "book": lee-nooks, not line-axe: https://upload.wikimedia.org/wikipedia/commons/0/03/Linus-linux.ogg
-Unlike Windows or MacOS, there are many varieties of Linux, so-called distributions, which all share the same fundamental architecture and "kernel" (the program that actually runs the OS), but differ in their look and feel, the software included, and ease of use. The reason Linux is so widely used is in part due to the ease of modifying Linux to create a distribution suitable for a particular application (e.g., small and lightweight for low-power computing devices, fast and working in real time for time critical applications such as rocket launches, or with lots of bells and whistles to make them user friendly for desktop use). The most common distributions, such as Ubuntu and Debian, are geared towards user-friendliness and will look fairly familiar to users of Windows and Macs used to a "point-and-click" interface, while the more specialised ones often make do without any graphical user interface at all, instead using a terminal window only (more on that below).
+Unlike Windows or MacOS, there are many varieties of Linux, so-called distributions, which all share the same fundamental architecture and "kernel" (the program that actually runs the OS), but differ in their look and feel, the software included, and ease of use. The reason Linux is so widely used is in part due to the ease of modifying Linux to create a distribution suitable for a particular application (e.g., small and lightweight for low-power computing devices, fast and working in real time for time critical applications such as rocket launches or [[https://github.com/readme/featured/nasa-ingenuity-helicopter|flying helicopters on Mars]], or with lots of bells and whistles to make them user friendly for desktop use). The most common distributions, such as Ubuntu and Debian, are geared towards user-friendliness and will look fairly familiar to users of Windows and Macs used to a "point-and-click" interface, while the more specialised ones often make do without any graphical user interface at all, instead using a terminal window only (more on that below).
 ===== Logging in to a remote Linux server =====
-  *SSH
+UNDER CONSTRUCTION
+***//SSH//**
-SSH stands for Secure Shell. It is a command-line protocol which provides strong password authentication and public key authentication, as well as encrypted data communications between two computers connecting over an open network, such as the internet. To be able to connect to the server //psycomp.rhul.ac.uk// over SSH, open the command-line terminal and use the following command:
+SSH stands for Secure Shell. It is a command-line protocol which provides strong password authentication and public key authentication, as well as encrypted data communications between two computers connecting over an open network, such as the internet. It is run from the command line as follows:
+<del>
+To be able to connect to the server //psycomp.rhul.ac.uk// over SSH, open the command-line terminal and use the following command:
@@ Line 24: / Line 30: @@
-  *RDP
+  ***//RDP//**
 RDP stands for Remote Desktop Protocol. The //psycomp.rhul.ac.uk// can be accessed for GUI (Graphical User Interface) tools on the cluster. This node is provided with high-performance graphics drivers for this. This //psycomp.rhul.ac.uk// can be accessed directly while operating from a local PC on campus and for off-campus access is granted only through VPN (details of VPN access is provided [[https://intranet.royalholloway.ac.uk/students/help-support/it-services/access-off-campus/vpn/vpn.aspx|here]]). After connecting to the VPN use the Remote Desktop Connection software (Windows RDP / Ubuntu Remmina) and connect to the //psycomp// Server( server address: //psycomp.rhul.ac.uk// ). Use the [user_ID] and password and domain as cc to log on to the server.
-  *VNCViewer
+  ***//VNCViewer//**
+Open the VNCViewer app and enter //psycomp.rhul.ac.uk//:[VNC_session_number] in the VNC server dropdown box. The [VNC_session_number] is an unique session code that you should have received from the IT team (If you haven't been provided one, please email the psychology IT Team [[Can.Keles@rhul.ac.uk]]). After clicking Connect enter your VNC password and log-on to the cluster.</del>
-Open the VNCViewer app and enter psyclogin.rhul.ac.uk:[VNC_session_number] in the VNC server dropdown box. The [VNC_session_number] is an unique session code that you should have received from the IT team (If you haven't been provided one, please email the psychology IT Team here). After clicking Connect enter your VNC password and log-on to the cluster.
 ===== The Linux desktop =====
 There are a variety of desktops available on Linux distributions but most common distributions (such as Ubuntu or Debian) have a desktop that looks fairly familiar to a Windows or Mac user - there will be a menu bar at the top or the bottom of the screen which you can use to launch different programs, and a desktop with some folders in it that you can click on to open and navigate through the file system. For our purposes we will focus upon the Plasma KDE desktop.
 ===== The Linux terminal =====
-**//The shell: //**bash, sh, csh
+Although modern Linux versions have a look and feel that is similar to Windows or Mac, a lot of the power of Linux resides in the use of the //terminal// - the software tool used for running programmes from the command line. Almost everything in Linux can be done from the command line, and although it can seem daunting and cumbersome at first to have to enter commands by typing rather than pointing and clicking you will soon realise that it is both much faster and allows automating (scripting) tasks that would take a very long time to do using the mouse. Once you get the hang of it it is straightforward to write scripts and even programmes without having to be a seasoned software engineer.
-**//Command-line completion//**
+Below are some useful tips and tricks for using the terminal that will help you use its full potential.
-  *Writing a shell script
+==== The shell ====
-  *Navigating the file system
+The terminal window runs a program called the //shell// to interpret the commands you write. There are different types of shell programs but the most common ones that you'll encounter are //sh// or //bash//, or less commonly, //csh//. The information below refers to bash, which is the standard in most modern Linux distributions.
-===== The Linux file system =====
+=== Special characters ===
-**// The home directory:  //**/home/myusername
+The command prompt:
-**// The root directory: //**/
+The terminal prints a "prompt" at the beginning of a line when it expects command input. This is typically of the format "username@computer:current folder$", so if you are logged into the computer psych01 and your user name is jonas and you're in the folder /MRIWork04, the prompt would look like this:
+<code sh>
+jonas@psych01:/MRIWork04$
+# Change to home folder - note you only input "cd" below
+jonas@psych01:/MRIWork04$ cd
+jonas@psych01:~$
+</code>
-**//System directories: //**/usr, /bin, /etc, /lib, /media
+In this guide, the command prompt is not shown in any of the examples below to make it easy for you to copy and paste commands into the terminal.
-**//Mounting external file systems//**
+#: In the terminal, any line that starts with a hash character # is treated as a comment and ignored.
+<code sh>
+# You can write anything after a hash character, it is ignored by the shell
+</code>
-**//File permissions //**
+&: The ampersand character & //after// a command means "run the program in the background".
+Normally, when you run a program that doesn't immediately return a result (for example if you open a browser by typing
+<code sh>
+firefox
+</code>
+the terminal is locked (you can't enter any new commands) until that command has finished (in this case, when you close the browser window).
+If instead you type
+<code sh>
+firefox &
+</code>
+the terminal will keep the browser running in the "background" so you can keep using the terminal window without having to wait for the browser window to close (it will only stop running when you close it yourself, or you logout.
-**// Recursive commands //**
+If you've started a program without the & character you can put it in the background by pressing **CTRL+Z** - this will halt (suspend) the program. You can then start it again as a background process by typing
-===== Essential commands =====
-**//Finding where you are //** pwd
+<code sh>bg</code>
-**//List files://** ls
+which means "run in background". If for some reason you want to return the program to run in the "foreground", so your terminal is blocked, just type
-**//Change directory://** cd
+<code sh>fg</code>
-**//Rename or move files: //** mv
+CTRL+C: if you are running a program in foreground mode, you can stop it by pressing CTRL+C - it will stop executing more or less immediately and not gracefully (i.e. it wont save any files or make a controlled exit; it will just stop).
-**//Create files and directories: //** touch, mkdir
+/ The forward slash is used to describe the location of files and folders and is used to delimit folders. In Linux the top-level directory is / and directories below are located by relation to this top level directory. For example, programs commonly used by users (rather than the system itseld) will be in a folder called //bin// which is located within a subfolder of the top-level directory called //usr//. The full path to this folder is therefore written
-**//Delete files and directories: //**rm, rmdir
+/usr/bin
-**//Find files and directories: //**find
-**//Check CPU usage: //**top, ps
+. A single period (.) means "here" - the current directory. So if you're say copying a file from /usr/local/games/mygame to where you are located, you would write
-**//Text editing: //**pluma, kate, nano
+<code sh>cp /usr/local/games/mygame .</code>
-**//PDF viewer: //**atril
+.. Two periods mean "the folder above your current location". So if your current location is /home/jonas, ..  would mean /home. This is especially useful when changing directories, for example:
-**//Desktop applications: //**LibreOffice, Firefox
+<code sh>
+# Change directory to the folder above
+cd ..
+</code>
-**//Common neuroimaging applications://** FSL, Matlab, AFNI
+~ The tilde means your home directory. If your user name is jonas, your home directory will be /home/jonas, but you could use ~ instead.
+<code sh>
+# List contents of home directory
+ls /home/jonas
+# List contents of home directory
+ls ~
+</code>
-**//copying files and directories: //** cp, scp
-**// Getting help: //** man
+=== Wildcards ===
+You often want to be able to find or list files of a particular type or name but don't want to have to type out each file individually. What you can use then is what is known as //wildcards// - special characters that can stand for any text. To make this clearer, suppose you want to list all files that end with the extension IMA in your current folder (this is how the raw data from the scanner is stored). If you go into the directory /MRIWork/MRIWork04/jl/msc2425/functional/20250218/20030215AAPU/scan01 and type ls:
+<code sh>
+cd /MRIWork/MRIWork04/jl/msc2425/functional/20250218/20030215AAPU/scan01
+ls
+</code>
+you will see the following:
+<code sh>
+20030215AAPU.MR.HOLLOWAY_JONAS.0001.0001.2025.02.18.12.47.03.0.136922180.IMA  20250218_120320s001a1001A.hdr  20250218_120320s001a1001.hdr
+20030215AAPU.MR.HOLLOWAY_JONAS.0001.0003.2025.02.18.12.47.03.0.136922202.IMA  20250218_120320s001a1001A.img  20250218_120320s001a1001.img
+</code>
+Suppose you just want to list the IMA files. You can then type
+<code sh>
+ls *IMA
+</code>
+and the output will now be
+<code sh>
+20030215AAPU.MR.HOLLOWAY_JONAS.0001.0001.2025.02.18.12.47.03.0.136922180.IMA  20030215AAPU.MR.HOLLOWAY_JONAS.0001.0003.2025.02.18.12.47.03.0.136922202.IMA
+</code>
+The asterisk * is a wildcard - it means "match any character" so *IMA means "any file that ends with IMA". You can use multiple wildcards, for example if you type
+<code sh>
+ls *s001a*
+</code>
+you will see
+<code sh>
+20250218_120320s001a1001A.hdr  20250218_120320s001a1001A.img  20250218_120320s001a1001.hdr  20250218_120320s001a1001.img
+</code>
+- that is, every file whose name contains the string "s001a". Wildcards can be very handy when selecting files in scripts, for instance, or if you want to move large numbers of files of a particular type or name from one location to another much more quickly and efficiently than clicking and dragging files on a desktop, for example.
+Most of the time you only need to use the * wildcard, which means "any numbers of characters" (so in the above example, *IMA would match both m.IMA and mmmmmm.IMA). Sometimes though you might want to use a wildcard for just a single character - then you can use a ? (question mark) which means "any single character". For instance, assume there are two sets of files in your folder, one set named m1.IMA, m2.IMA, m3.IMA, .. m9.IMA, and another named mm1.IMA, mm2.IMA,...mm9.IMA. If you want to list only the files with a single m, you couldn't use the * (because m*IMA would list both m1.IMA... and mm1.IMA..). Instead you could write
+<code sh>
+ls m?.IMA
+</code>
+which means "list the files whose names begins with m, followed by an arbitrary single character, followed by .IMA" and this would only list m1.IMA, m2.IMA.. etc but not mm1.IMA etc.
+==== Command and file name completion ====
+If you don't remember the exact name of a command, start typing the first letter(s) of the command and press the Tab key - this will list all the commands that begin with those letters - if nothing appears then there is no command that matches.
+If you start typing a command followed by a space, you will instead be shown the files in the current directory - if there are many files a prompt will appear saying "Display all XX possibilities? (y or n)" and if you press y the files are shown one page at a time (there will be a prompt that says "--More--" at the bottom) and by pressing the space bar you can step through them. You can start typing the first letters of a filename, and it will complete that too, so lets say you have two files called myfile1 and myfile2 in your folder, if you type
+<code sh>
+ls myf
+</code>
+and press the Tab key rather than Enter, you will hear a beep and a list will be shown of the two files, like so:
+<code sh>
+ls myf # pressing Tab
+myfile1
+myfile2
+ls myf
+</code>
+==== Command history ====
+Use the up and down arrows on the keyboard to step through past commands, and hit enter to re-run them (or Ctrl-C to quit).
+To list recent commands, type
+<code sh>
+history
+</code>
+which will produce a list of the last (normally around 1000) commands preceded by a number. To re-run any of these command, just type an exclamation mark followed by the number, e.g. if the command numbered 1500 was ls -ltRr (a recursive full listing of all directoris and subdirectories in reverse time order), typing
+<code sh>
+!1500
+</code>
+will have the same effect as re-typing
+<code sh>
+ls -ltRr
+</code>
+Often it is handy to use the history function to search for a particular command, e.g. you might not remember how you connected to another computer by ssh, so you want to search all commands including ssh. For this you can use the grep command, which searches for a string:
+<code sh>
+history | grep ssh
+</code>
+The vertical bar | is a //pipe// - this means that the output from the first command (history) is sent as input to the following command (grep) rather than  to the terminal output. Pipes are very useful constructs in Linux that allow you to combine multiple commands into one, whcih can be very handy. For example, the following command will find all instances of ssh in the command history, and then print them out sorted according to the name of the server (the third field of the output of history) in reverse order - here, the output of history is piped to grep, which in turn is piped to sort:
+<code sh>
+history |grep ssh|sort -k 3,3 -r
+</code>
+==== Text editors ====
+Because so much in Linux is done by manipulating text files (as well as writing code), you will often need to use a program to edit text - a text editor. Standard word processing programs are unsuitable for this purpose as they save their output in a non-text format and are not intended for writing simple text commands (or computer code). Much better is to use text editors that are designed for this specific purpose. Two of the most popular editors in Linux are emacs and vi (or vim). Emacs is a lot easier to use and has a more intuitive interface - vi seems to be designed with the express purpose of frustrating novice users and tends to lead to a lot of beeping and error messages. Which one you prefer is a personal decision; they both accomplish the same task.
+To run them, just type <code sh>  emacs</code> or <code sh>vi</code> or <code sh>vim</code>
+Two more modern and user-friendly editors are pluma and gedit - these may or may not be installed on your system, but try:
+<code sh>pluma</code> or <code sh>gedit</code>
+==== Useful commands & programs ====
+== cd - change directory ==
+Syntax: cd <path to directory>
+<code sh>
+cd /home/jonas
+# Move up one directory
+cd ..
+# Move to home directory
+cd
+</code>
+== pwd - show current directory ==
+Syntax: pwd
+<code sh>
+cd /home/jonas
+pwd
+/home/jonas
+</code>
+== ls - list files ==
+Syntax: ls [<directory or file names>]
+<code sh>
+# List files in current directory
+ls
+# List files in /home/jonas
+ls /home/jonas
+# List files in current folder in long format showing size and ownership
+ls -l
+# List files in reverse order by time (last first)
+ls -tr
+# List files in long format and reverse order by time (last first)
+ls -ltr
+# List files recursively in current folder and subfolders
+ls -R
+# List files beginning with myf
+ls myf*
+# List files ending with ile
+ls *ile
+# List files containing the string yfi
+ls *yfi*
+</code>
+== mv - move or rename files ==
+Syntax: mv <input file> <output file name or directory>
+<code sh>
+# Rename a file in current directory
+mv myfile1 myfile2
+# Move a file to the directory above
+mv myfile1 ..
+# Move a file to /home/jonas
+mv myfile1 /home/jonas
+</code>
+== cp - copy files ==
+Syntax: cp <input file> <output file name or directory>
+<code sh>
+</code>
+== rm - remove files ==
+Syntax: rm <file/s>
+<code sh>
+# Remove a single file
+rm myfile1
+# Remove multiple files matching pattern myf*
+rm myf*
+# Remove a folder and all of its contents (beware!)
+rm -rf myfolder
+</code>
+== mkdir - make a directory (folder) ==
+Syntax: mkdir <new folder name>
+<code sh>
+# Make a new folder in current directory
+mkdir mynewfolder
+# Make a new folder in another directory
+mkdir /home/jonas/mynewfolder
+# Make a new folder containing a subfolder
+mkdir -p /home/jonas/mynewfolder/mynewsubfolder
+</code>
+== rmdir - remove a directory ==
+Syntax: rmdir <directory name>
+<code sh>
+# Remove an empty folder (folders with contents need to be removed with rm -rf, see above)
+rmdir myfolder
+</code>
+== more and cat - show the content of text files ==
+Syntax: more <filename>
+cat <filename>
+<code sh>
+# List contents of a text file, one page at a time (step through using space, press q to quit)
+more myfile1
+# List full contents of a text file
+cat myfile1
+</code>
+== find - find files ==
+Syntax:
+<code sh>
+</code>
+== grep - find text in a file ==
+Syntax:
+<code sh>
+</code>
+== sort - sort text ==
+Syntax:
+<code sh>
+</code>
+== man - show help for a program ==
+Syntax:
+<code sh>
+</code>
+== ps, top - show CPU usage and running processes ==
+Syntax:
+<code sh>
+</code>
+== kill, pkill, xkill - stop a program or process ==
+If you come from Windows, you will be familiar with the task manager which can be handy when wanting to stop a program that has crashed or is non-responsive. Linux programs tend to be more stable than Windows ones, but you might still from time to time need to stop an out of control program. Unlike the Windows task manager, which seems to meekly ask programs whether they might be so kind as to stop and can take a while to have any effect, the Linux equivalents show no mercy and are instanteneous and uncompromising - the program or process will stop instantly without saving files etc. Beware that if you kill a process used by the system, you could cause unintended consequences for yourself (e.g. if you stop the window system X). However, the inherently secure nature of Linux means you cannot affect the system as a whole or other users, and logging out and logging in will bring you back to a normal state. The bottom line is, in general don't kill processes you don't know what they are or do.
+<code bash>
+# To kill a program if you know it's process number (see above), use kill -9:
+kill -9 <process number>
+#To kill a program whose name you know, use pkill -9:
+pkill -9 firefox
+#To kill a window program you can use xkill; you will then need to click the window of the program you want to stop:
+xkill
+</code>
+== ssh - remote login to another computer ==
+Syntax: pwd
+<code sh>
+</code>
+== scp - copy files to another computer ==
+Syntax:
+<code sh>
+</code>
+== rsync - copy large numbers of files fast and efficiently ==
+Syntax:
+<code sh>
+</code>
+===== The Linux file system =====
+**// The home directory:  //**/home/myusername
+**// The root directory: //**/
+**//System directories: //**/usr, /bin, /etc, /lib, /media
+**//Mounting external file systems//**
+**//File permissions //**
+**// Recursive commands //**