Differences

This shows you the differences between two versions of the page.

--- cluster-file_transfer [2024/05/07 13:03] – gabriele
+++ cluster-file_transfer [2026/09/03 12:58] (current) – gabriele
@@ Line 4: / Line 4: @@
 ====== Transferring files to/from psychp01 ======
 What's on this page:\\
-  - Transferring data
+  - Transferring data internally (e.g., from your computer in the office)
+  - Transferring data externally (e.g., from a website or an external institution)
   - Transferring code
   - File Transfer Clients
-===== Transferring data =====
+===== 1. Transferring data internally =====
 You need to upload your data into the folder that was created by the IT for you within the ''/MRIWork'' folder on the cluster. Your folder name will be something like ''MRIWork#'' where ''#'' stands for a number assigned to you. Hence, if your number is “25”, your path name to upload your data on the cluster will be:
@@ Line 17: / Line 18: @@
 To know more about ''/MRIWork'', folder structure, data archiving, data backup and data sharing, have a look at the [[data-storage|CUBIC wiki]].
-Please NOTE: theoretically, you can upload your data also into your home directory (see Transferring code below) and save your analysis results there. However, the home directory has limited amount of storage space and is not periodically backed up. Hence, it is advisable that you save your data and analysis results in your /MRIWork/MRIWork# folder, even if you do not have MRI data and do not think your data occupy too much space. This will avoid problems with storage space and minimize risks of data loss.
+Please NOTE: theoretically, you can upload your data also into your home directory (see Transferring code below) and save your analysis results there. However, the home directory has limited amount of storage space and is **not** periodically backed up. Hence, it is advisable that you save your data and analysis results in your ''/MRIWork/MRIWork#'' folder, even if you do not have MRI data and do not think your data occupy too much space. This will avoid problems with storage space and minimize risks of data loss.
-Transferring code
+===== 2. Transferring data externally =====
+To transfer data from outside RHUL, you'd need to first move the data to a separate "graphics" server before moving them to ''/MRIWork''. This is for security reasons. Data will scanned by an antivirus as you move them. From this "graphics" server, you can then easily copy them in your folder within ''/MRIWork''.
-You can upload your scripts into a folder within your home directory on the cluster. The IT will have probably created a folder in the home directory with the first letter of your name preceding your surname (e.g., gbellucci). Hence, if your name is Gabriele Bellucci, your path name to upload your data on the cluster will be:
+This "graphics" server has IP: 134.219.34.200 and is called ''graphics'' because it has a graphics card that allows users to connect via any remote desktop app of their choice (e.g., [[https://apps.microsoft.com/detail/9n1f85v9t8bn?hl=en-US&gl=GB|Windows App]]). The "graphics" server is managed by Jonas Larsson (jonas.larsson@rhul.ac.uk). Please contact Jonas to have him create an account for you on this server.
-/home/gbellucci
-Your folder name in the home directory has the same name as the username you use to access the cluster. This name was provided to you by the IT when you asked for access. Be aware that there might be deviations on how your folder name in the home directory has been created (especially if you were granted access to the cluster before 2024). If you have access to the cluster, just log in using ssh and type pwd to see your folder name in the home directory. Something like the line above will pop up on your command line.
+==== Example 1: ====
-The most secure way to transfer code is using File Transfer Clients like sftp and scp via the command line. Please see below for how to use these clients and a demonstration video.
+Suppose you need to download a dataset from a website that does not require a data sharing agreement and is a simple browser-based download, like [[https://datadryad.org/dataset/doi:10.5061/dryad.2v6wwpzt3|UCL - Release of cognitive and multimodal MRI data including real-world tasks and hippocampal subfield segmentations]].
+  - Open your remote desktop app.
+  - Enter the host name (the server "name"): 134.219.34.200
+  - Enter your username: gabriele (for your it's the name Jonas gave you, most likely first letter of your name + your surname)
+  - Enter your password: **** (the one Jonas gave you to connect the first time, which need to be changed at first login)
+  - Once connected, click on the World icon on the bar at the bottom of the screen. This will open a Firefox webpage
+  - Search your dataset page like you'd do it normally on your browser
+  - Locate your dataset.
+   - If the size of your dataset is below 1TB, just click on the download link. The data will be stored in Downloads.
+   - If the dataset is bigger:
+     - In Firefox, open the menu (three lines, top right) and choose ''Settings'' (a shortcut: typing ''about:preferences#general'' in the address bar takes you straight to that panel).
+     - Stay on the ''General'' panel and scroll down to the ''Downloads'' section (it's a fair way down, under ''Files and Applications'').
+     - There you'll see ''Save files to'' with your current folder next to it. Click ''Browse...''
+     - A new window opens titled ''Choose Download Folder''.
+     - On top of that window, you'll see two tabs, one with your username and a small house, and the other labelled ''Downloads''.
+     - On the left next to the tab with your username, click the back arrows.
+     - This opens a new tab with a square and circle within it. Click it.
+     - You'll then see a list of folders. Search for ''Storage 1'' or ''Storage 2''. They have roughly 10 TB each. Pick one of them.
+     - The change takes effect right away, no need to save or restart.
+  - Click on the download link and download it.
+  - Once downloaded, you can extract the files locally. Right-click and choose ''Open with File Roller''.
+  - Choose your local folder where to extract the files. I suggest you create here the folder structure you'd like your data to have in ''/MRIWork'' for your future analyses.
-File Transfer Clients
+From the graphics server you can't log into psychp01 directly, but the same storage is mounted read-only. You'll see a ''/MRIWork'' folder with exactly the same contents. It isn't a copy, it's the same files, accessed over the network. Because the mount is read-only, you can't create, edit, or delete anything on psychp01 from the "graphics" server. You can only read from it and copy data across to the "graphics" server's own storage. This is convenient, as you can create and test your environment here before moving your data onto psychp01. You can also test-run some analyses, as you'll have the same software (Jonas is taking care that the "graphics" server has all the software psychp01 has), but please be aware of the warning below.
-Primary access to psychp01 is via ssh based tools (on the command line). To upload or download data and code, File Transfer Clients such as scp and sftp can be used.
+Once you have downloaded all your data, created your folder structure and potentially tested some analyses, you can pull the data from psychp01 for storage and additional analyses.
-To transfer data to and from psychp01 use the following address:
+== To move the data from the "graphics" server onto psychp01. ==
+Suppose you have saved all your data on the "graphics" server in a folder with path: ''/home/gabriele/Downloads/new_dataset'' and you want to move the ''new_dataset'' folder to '/MRIWork/MRIWork25/gb/gabriele_bellucci/' on psychp01.
+ - Establish an ssh connection with pyshcp01 via the terminal (see [[cluster-access#log_in_with_ssh|SSH]]).
+ - Establish an sftp connection with the "graphics" server (see [[cluster-file_transfer#sftp|SFTP]] below):
+  sftp gabriele@134.219.34.200
+ - You'll be prompted to enter your password.
+ - Move the data:
+  get -r /home/gabriele/Downloads/new_dataset /MRIWork/MRIWork25/gb/gabriele_bellucci/
+The ''new_dataset'' folder will be under '/MRIWork/MRIWork25/gb/gabriele_bellucci/'.
-psychp01.rhul.ac.uk
+**WARNING**
+Remember: the "graphics" server is simply a computer sitting in Wolfson, which is NOT backed up and should NOT be used for long-term storage or important data analyses. If there's an outage (which unfortunately happens pretty often), the "graphics" server will be severely impacted and you may use all your data.
-sftp
-sftp, which stands for Secure File Transfer Protocol, is an encrypted protocol built into SSH that can implement commands for transferring files between two remote systems over a secure connection. There are many resources on the web on how to use sftp (e.g., here). Here, example applications to transfer data onto psychp01 will be shown.
-First, you need to establish a secure connection with the server. This is very similar to how you would connect with the server using ssh (see here).
+==== Example 2: ====
-sftp username@psychp01.rhul.ac.uk
+Suppose you need to download a dataset that **cannot** be downloaded by simply clicking a link in the browser, because it is hosted on a repository that uses ''git-annex''. A good example is the [[https://doi.gin.g-node.org/10.12751/g-node.5mv3bf/|Welsh Advanced Neuroimaging Database (WAND)]], hosted on [[https://gin.g-node.org/CUBRIC/WAND|GIN]] by Cardiff University: 170 participants, 8 imaging sessions, roughly **1.95 TB** in total.
-Like for the ssh connection, “username” is the username provided to you by the IT when you asked for access to the cluster. Hit enter and you will be required to enter your password. Once you are connected, at the beginning of your command line, you will see that an connection has been established:
+Repositories like GIN do not store the data in one downloadable archive. Instead, you first download a small "skeleton" of the dataset (the folder structure and the file names, a few hundred MB), and then ask for the content of only the files you actually need. This is very convenient: you can look at the whole dataset, decide what you want, and download only that.
-sftp>
+Everything below is done **on the "graphics" server** (see above for how to connect), following the same logic as Example 1: external data must land on "graphics" first, and can then be moved to ''/MRIWork'' on psychp01.
-Now, you can ftp commands to (among others) upload, download, remove, and move files. Type help to check all commands available.
+== Before you start ==
-sftp> help
+Three things to know, because none of them are obvious and none of them are in GIN's own documentation:
-The sftp connection puts you on the cluster. Here, you can use all common commands you would use on your local machine to get the current directory, change the current directory and so on. If you would like to use the same commands on your local computer, you can do that by adding an “l” in front of the command you want to use. This “l” stands for “local” and tells sftp to use the command on the local machine as opposed to the remote one.
+  * You **cannot** download this dataset over HTTPS. GIN serves the folder structure over HTTPS, but the actual imaging files only over ''ssh''. You therefore need a (free) GIN account with your own ''ssh'' key. There is no way around this.
+  * The ''ssh'' address needs a **leading slash** after the colon. Without it, the server replies ''GIN: Invalid repository path''.
+  * ''git'', ''git-annex'' and the ''gin'' client are already installed on the "graphics" server for all users. You do not need to install anything.
-For instance, when you establish an sftp connection, you will find yourself in your home directory. Hence, if your home directory path is /home/gbellucci, when you type pwd, you will see the second line of the code below appearing:
+== 1. Set your git identity ==
-sftp> pwd
+''git-annex'' writes small commits as it works and refuses to run without a name and an email address:
-Remote working directory: /home/gbellucci
-On the contrary, if you type lpwd, you will see the second line of the code below appearing:
+<code>
+git config --global user.name "Your Name"
+git config --global user.email "your.name@rhul.ac.uk"
+</code>
-sftp> lpwd
+These are only labels written into your local copy. They are not checked against anything, and they do not give anyone access to anything.
-Local working directory: /Users/Gab
-where /Users/Gab is my (local) current directory on my computer. Type help to see the difference in the commands for remote and local implementations.
+== 2. Register a GIN account ==
-To download data from the cluster onto your local directory, you need to use the get command, like this:
+This step is done in the browser (click on the World icon at the bottom of the screen to open Firefox).
-sftp> get remote_filename_path local_dirpath
+  - Go to [[https://gin.g-node.org|https://gin.g-node.org]]
+  - Click ''Register'' (top right)
+  - Choose a username, enter your RHUL email and a password
+  - Confirm your account by clicking the link in the email you receive
-For example, if you have to get a file called results_matrix.mat from the folder results in your home directory /home/gbellucci and download it in your folder project_results on your local directory /Users/Gab, you will do:
+Registration is free and immediate; there is no approval process and no data sharing agreement for this dataset.
-sftp> get /home/gbellucci/results/results_matrix.mat /Users/Gab/project_results
+== 3. Create an ''ssh'' key and upload it to GIN ==
-Alternatively, you can cd to results (on the cluster), lcd to project_results (on your local machine), and then just type get results_matrix.mat, like this:
+An ''ssh'' key comes in two halves: a **private** key that never leaves the "graphics" server, and a **public** key that you give to GIN so it can recognise you.
-sftp> cd /home/gbellucci/results
+First check whether you already have one:
-sftp> lcd /Users/Gab/project_results
-sftp> get results_matrix.mat
-NOTE: If you have folder names that contain spaces, sftp would fail. For instance, something like that: sftp> lcd /Users/Gab/project results, (i.e., your results folder named “project result” with a space) would not work!
+<code>
+ls ~/.ssh/id_*.pub
+</code>
-If you have to download a folder, you will need to use the -r argument like that:
+If nothing is listed, create one:
-sftp> get -r remote_dirpath local_dirpath
+<code>
+ssh-keygen -t ed25519 -C "your.name@rhul.ac.uk"
+</code>
-On the contrary, if you have to upload data from your local machine to the cluster, you will need to use the put command:
+Press Enter to accept the default file name. You can leave the passphrase empty; if you set one, you will be asked for it every time, which is inconvenient for downloads that run for hours.
-sftp> put local_filename_path remote_dirpath
+Now print the **public** key (note the ''.pub'' — never share the file without it):
-In this video, you will see how to transfer a Python code and a bash file to psychp01 using put.
+<code>
+cat ~/.ssh/id_ed25519.pub
+</code>
+You will see a single long line starting with ''ssh-ed25519''. Select and copy the **whole** line, including the email at the end.
+Then, in the browser, logged into GIN:
+  - Click your avatar (top right) and choose ''Your Settings''
+  - Choose ''SSH Keys'' in the menu on the left
+  - Click ''Add Key''
+  - Give it a name, e.g. ''graphics server''
+  - Paste the line into the ''Content'' box and save
+Check that it worked:
+<code>
+ssh -T git@gin.g-node.org
+</code>
+The first time, you will be asked whether you trust the server. Type ''yes''. You should then see:
+<code>
+Hi there, You've successfully authenticated, but GIN does not provide shell access.
+</code>
+This message means **success**. GIN only allows ''git'' operations, never an interactive login, so "no shell access" is the expected and correct answer.
+== 4. Choose where the data will go ==
+**Do not download into your home directory.** Use ''Storage 1'' or ''Storage 2'' on the "graphics" server (roughly 10 TB each), exactly as in Example 1.
+Check the free space before you start:
+<code>
+df -h /path/to/storage
+cd /path/to/storage
+</code>
+Note that ''/MRIWork'' is mounted **read-only** on the "graphics" server, so it cannot be used as the download destination. The data go into ''Storage 1'' or ''Storage 2'' first, and are moved to ''/MRIWork'' afterwards (see below).
+== 5. Download the dataset skeleton ==
+<code>
+git clone git@gin.g-node.org:/CUBRIC/WAND.git
+cd WAND
+git annex init "graphics server"
+</code>
+**Note the slash** immediately after the colon, before ''CUBRIC''. This is the single most common cause of failure and it is not documented by GIN.
+This step is quick and small. You now have the complete folder structure with the real file names, all the metadata files (''.json'', ''.tsv'', the README, ''participants.tsv''), and //placeholders// where the imaging files will go.
+You can check the overall picture with:
-scp
+<code>
+git annex info
+</code>
-scp (secure copy) is a command-line utility that allows you to securely copy files and directories between two locations. scp use requires a password, and both the files and password are encrypted so as to securely transfer data from one location to the other. scp uses the ssh protocol for both authentication and encryption. See here for my information.
+which reports the total size of the dataset, how much you currently have locally (zero at this point), and how much disk space is available.
-When transferring data, scp takes on two main arguments:
+== 6. Check how big your request is, before downloading ==
-scp source destination
+''git-annex'' knows the size of every file without downloading anything. This lets you find out in advance whether your selection will fit. For example, to add up sessions 02, 03 and 06 across all participants:
-The first argument is the address of the source file to transfer, the second the address where it has to be transferred to. A good way to memorize it is to think that scp needs to know what to send where to.
+<code>
+for s in 02 03 06; do
+  printf "ses-%s: " "$s"
+  git annex find sub-*/ses-$s --format='${bytesize}\n' 2>/dev/null \
+    | awk '{t+=$1} END {printf "%.1f GB\n", t/1e9}'
+done
+</code>
-For example, to transfer files from the remote cluster (source) to your local machine (destination), use:
+Change the list ''02 03 06'' to the sessions you need. Compare the result with the output of ''df -h .'' **before** starting a transfer, not halfway through one.
-scp username@address_name:pathname_remote_src pathname_local_dest
+== 7. Download the data you need ==
-Suppose my username (the one given to you by the IT when you got access to the cluster) is gbellucci, the filename of the file (e.g., a MATLAB file .m) I need to transfer is best_analysis.m, the pathname to that file on my local computer is /Users/Gab, and the pathname of the remote folder on the cluster I need to send my file to is /home/gbellucci/coolest_project. The line on terminal I need to transfer my file will be
+Always test with a single participant first, so you can see how much one subject costs:
-To transfer files from your local machine to the remote cluster, use:
-scp <space> pathname_local_src <space> username@address_name:pathname_remote_dest
+<code>
+git annex get sub-00395/ses-03/anat
+du -sh .
+</code>
-Suppose the filename of the file (e.g., a MATLAB file .m) I need to transfer is best_analysis.m, the pathname to that file on my local computer is again /Users/Gab, and the pathname of the remote folder on the cluster I need to send my file to is /home/gbellucci/coolest_project. The command on the command line will be:
+Then start the real download. Use ''screen'' so that the transfer survives a lost connection or a closed remote desktop session:
-scp /Users/Gab/best_analysis.m gbellucci@psychp01.rhul.ac.uk:/home/gbellucci/coolest_project
+<code>
+screen -S wand
+git annex get -J4 sub-*/ses-0{2,3,6}
+</code>
-Remember, your data will not be in your folder in the home directory but in your MRIWork# folder in /MRIWork. Hence, to upload a data file (say, data.mat), you’d need to type:
+''-J4'' runs four downloads in parallel. Detach from the ''screen'' session with ''Ctrl-A'' then ''D'', and come back to it later with ''screen -r wand''.
-scp /Users/Gab/data.mat gbellucci@psychp01.rhul.ac.uk:/MRIWork/MRIWork25/data_coolest_project
+If the transfer is interrupted, simply run the same command again: ''git-annex'' keeps track of what it already has and continues where it stopped.
-If you have to upload or download multiple files or a file that contains multiple file (e.g., a folder), now you’ll have a directory path (and not a file path), and you can use the -r argument to reiterate the sending over all files like that:
+You will see error messages for participants who do not have a given session. This is normal: not every volunteer took part in every session (the 7 T and TMS sessions in particular had much smaller sub-samples).
-scp -r dirpath_local_src username@psychp01.rhul.ac.uk:dirpath_remote_dest
+To download only some participants, put their IDs in a text file, one per line:
-For example, if your directory path is to the folder called analyses_folder, you can type the following:
+<code>
+while read s; do
+  git annex get -J4 "$s"/ses-0{2,3,6}
+done < subjects.txt
+</code>
-scp -r /Users/Gab/analyses_folder gbellucci@psychp01.rhul.ac.uk:/home/gbellucci/coolest_project
+== 8. Move the data onto psychp01 ==
-If you have a whole data folder to transfer, you will upload it into your /MRIWork/MRIWork# folder like that:
+Once the download is finished and you are happy with your folder structure, move the data to ''/MRIWork'' exactly as described above in [[cluster-file_transfer#to_move_the_data_from_the_graphics_server_onto_psychp01|To move the data from the "graphics" server onto psychp01]].
-scp -r /Users/Gab/data gbellucci@psychp01.rhul.ac.uk:/MRIWork/MRIWork25/data_coolest_project
+Note that the downloaded dataset contains a hidden ''.git'' folder holding the ''git-annex'' machinery. If you copy the whole ''WAND'' folder, it comes along and roughly doubles the space used. If you only want the imaging files on psychp01, copy the ''sub-*'' folders and the metadata files, and leave the repository behind on the "graphics" server.
-You would swap the two arguments if the folder is on the cluster, and you would need to get it onto your local computer:
+== Structure of the WAND dataset ==
-scp -r username@psychp01.rhul.ac.uk:dirpath_remote_src dirpath_local_dest
+The data follow the [[https://bids.neuroimaging.io/|BIDS]] standard: first participant, then session, then data type, i.e. ''sub-<ID>/ses-<NN>/<datatype>/''.
-For example, if your directory path is to the folder on the cluster called results_folder that you need to download into your analyses_folder on your local computer, you can type the following:
+^ Session ^ Content ^
+| ses-01 | MEG (CTF ''.ds'' folders) |
+| ses-02 | Connectom 3 T, ultra-strong gradients: diffusion and quantitative MRI |
+| ses-03 | Prisma 3 T: structural, functional, perfusion |
+| ses-04 | 7 T spectroscopy |
+| ses-05 | 3 T GABA-edited spectroscopy (MEGA-PRESS) |
+| ses-06 | 7 T structural and functional |
+| ses-07 | 3 T metabolic (subset, about 39 participants) |
+| ses-08 | TMS (subset, about 40 participants) |
-scp -r gbellucci@psychp01.rhul.ac.uk:/home/gbellucci/results_folder /Users/Gab/analyses_folder
+''ses-02'' and ''ses-06'' are by far the largest.
-rsync
+== Common pitfalls ==
-rsync, which stands for remote sync, is a remote and local file synchronization tool. It uses an algorithm to minimize the amount of data copied by only moving the portions of files that have changed. Please see here for more information.
+  * **The files look like they are already there, but they are not.** After cloning, you can see and browse every file name, including the MEG ''.ds'' folders. Those are placeholders until you run ''git annex get''. Analysis software pointed at a dataset that has not been downloaded will usually report //corrupt data// rather than //missing files//, which is confusing. To list what is still missing in the current folder: ''git annex find . %%--%%not %%--%%in here''
+  * **Never run ''git annex get'' without a path.** With no path it means "download everything", i.e. 1.95 TB. Do that only if you want to download the full dataset. The same applies to ''gin sync %%--%%content''.
+  * **The download links on the GIN wiki are dead.** The ''gin'' client is now distributed through [[https://github.com/G-Node/gin-cli/releases|GitHub releases]]. It is already installed on the "graphics" server, and in practice you can do everything with plain ''git'' and ''git annex'' anyway.
+  * **Remember the warning above**: the "graphics" server is not backed up. Do not leave the only copy of anything there.
-sshfs
-sshfs allows you to mount the file system on your local machine. See here for more details. Basic usage for Linux users:
-sshfs username@psychp01.rhul.ac.uk:dirpath mountpoint [options]
+===== 3. Transferring code =====
+You can upload your scripts into a folder within your home directory on the cluster. The IT will have probably created a folder in the home directory with the first letter of your name preceding your surname (e.g., ''gbellucci''). Hence, if your name is Gabriele Bellucci, your path name to upload your data on the cluster will be:
+  /home/gbellucci
-FileZilla
+Your folder name in the home directory has the same name as the username you use to access the cluster. This name was provided to you by the IT when you asked for access. Be aware that there might be deviations on how your folder name in the home directory has been created (especially if you were granted access to the cluster before 2024). If you have access to the cluster, just log in using ''ssh'' and type ''pwd'' to see your folder name in the home directory. Something like the line above will pop up on your command line.
-FileZilla is a free and open-source File Transfer Protocol (FTP) client that supports ftp, ftps and sftp protocols. It allows the implementation of the above command-line programs through a graphical interface. Please have a look at this step-by-step guide on how to use FileZilla.
+The most secure way to transfer code is using File Transfer Clients like ''sftp'' and ''scp'' via the command line. Please see below for how to use these clients and page 20 of the pdf file on the [[cluster-guide|home page]] for a demonstration video.
-ExpanDrive
+===== 4. File Transfer Clients =====
+Primary access to psychp01 is via ssh based tools (on the command line). To upload or download data and code, File Transfer Clients such as ''scp'' and ''sftp'' can be used.
-An alternative to File Transfer Clients like the one mentioned above is ExpanDrive. ExpanDrive is a network filesystem client for MacOS, Microsoft Windows and Linux that facilitates mapping of local volume to many different types of cloud storage. It is different from the above File Transfer Clients because it is integrated into all applications on the operating system and does not require a file to be downloaded onto the local machine. On the contrary, remote files can be accessed, managed and changed as if they were stored locally.
+To transfer data to and from psychp01 use the following address:
+  psychp01.rhul.ac.uk
+==== SFTP ====
+''sftp'', which stands for Secure File Transfer Protocol, is an encrypted protocol built into SSH that can implement commands for transferring files between two remote systems over a secure connection. There are many resources on the web on how to use ''sftp'' (e.g., [[https://www.digitalocean.com/community/tutorials/how-to-use-sftp-to-securely-transfer-files-with-a-remote-server|here]]). Here, example applications to transfer data onto psychp01 will be shown.\\
+First, you need to establish a secure connection with the server. This is very similar to how you would connect with the server using ''ssh'' (see [[cluster-access|here]]).
+  sftp username@psychp01.rhul.ac.uk
+Like for the ssh connection, “username” is the username provided to you by the IT when you asked for access to the cluster. Hit enter and you will be required to enter your password. Once you are connected, at the beginning of your command line, you will see that an connection has been established:
+  sftp>
+Now, you can use ''ftp'' commands to (among others) upload, download, remove, and move files. Type ''help'' to check all commands available.
+  sftp> help
+The ''sftp'' connection puts you on the cluster. Here, you can use all [[cluster-linux|common commands]] you would use on your local machine to get the current directory, change the current directory and so on. If you would like to use the same commands on //your local computer//, you can do that by adding an “l” in front of the command you want to use. This “l” stands for “local” and tells ''sftp'' to use the command on the local machine as opposed to the remote one.\\
+For instance, when you establish an ''sftp'' connection, you will find yourself in your home directory. Hence, if your home directory path is ''/home/gbellucci'', when you type ''pwd'', you will see the second line of the code below appearing:
+  sftp> pwd
+  Remote working directory: /home/gbellucci
+On the contrary, if you type ''lpwd'', you will see the second line of the code below appearing:
+  sftp> lpwd
+  Local working directory: /Users/Gab
+where ''/Users/Gab'' is my (local) current directory on my computer. Type ''help'' to see the difference in the commands for remote and local implementations.
+To download data from the cluster onto your local directory, you need to use the ''get'' command, like this:
+  sftp> get remote_filename_path local_dirpath
+For example, if you have to get a file called ''results_matrix.mat'' from the folder ''results'' in your home directory ''/home/gbellucci'' and download it in your folder ''project_results'' on your local directory ''/Users/Gab'', you will do:
+  sftp> get /home/gbellucci/results/results_matrix.mat /Users/Gab/project_results
+Alternatively, you can ''cd'' to ''results'' (on the cluster), ''lcd'' to ''project_results'' (on your local machine), and then just type ''get results_matrix.mat'', like this:
+  sftp> cd /home/gbellucci/results
+  sftp> lcd /Users/Gab/project_results
+  sftp> get results_matrix.mat
+__NOTE__: If you have folder names that contain spaces, ''sftp'' would fail. For instance, something like that: ''sftp> lcd /Users/Gab/project results'', (i.e., your results folder named “project result” with a space) would not work!\\
+If you have to download a folder, you will need to use the ''-r'' argument like that:
+  sftp> get -r remote_dirpath local_dirpath
+On the contrary, if you have to upload data from your local machine to the cluster, you will need to use the put command:
+  sftp> put local_filename_path remote_dirpath
+In the video on page 20 of the pdf file on the [[cluster-guide|main page]], you will see how to transfer a Python code and a bash file to psychp01 using ''put''.
+==== SCP ====
+''scp'' (secure copy) is a command-line utility that allows you to securely copy files and directories between two locations. ''scp'' use requires a password, and both the files and password are encrypted so as to securely transfer data from one location to the other. ''scp'' uses the ''ssh'' protocol for both authentication and encryption. See [[https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/|here]] for more information.
+When transferring data, ''scp'' takes on two main arguments:
+  scp source destination
+The first argument is the address of the source file to transfer, the second the address where it has to be transferred to. A good way to memorize it is to think that ''scp'' needs to know ''what'' to send ''where to''.\\
+For example, to transfer files from the remote cluster (source) to your local machine (destination), use:
+  scp username@address_name:pathname_remote_src pathname_local_dest
+To transfer files from your local machine (source) to the remote cluster (destination), use:
+  scp pathname_local_src username@address_name:pathname_remote_dest
+Suppose my username (the one given to you by the IT when you got access to the cluster) is ''gbellucci'', the filename of the file (e.g., a MATLAB file .m) I need to transfer is ''best_analysis.m'', the pathname to that file on my local computer is ''/Users/Gab'', and the pathname of the remote folder on the cluster I need to send my file to is ''/home/gbellucci/coolest_project''. The line I need on terminal to transfer my file will be:
+  scp /Users/Gab/best_analysis.m gbellucci@psychp01.rhul.ac.uk:/home/gbellucci/coolest_project
+Remember, your data will not be in your folder in the home directory but in your ''MRIWork#'' folder in ''/MRIWork''. Hence, to upload a data file (say, ''data.mat''), you’d need to type:
+  scp /Users/Gab/data.mat gbellucci@psychp01.rhul.ac.uk:/MRIWork/MRIWork25/data_coolest_project
+If you have to upload or download multiple files or a file that contains multiple file (e.g., a folder), now you’ll have a directory path (and not a file path), and you can use the ''-r'' argument to reiterate the sending over all files like that:
+  scp -r dirpath_local_src username@psychp01.rhul.ac.uk:dirpath_remote_dest
+For example, if your directory path is to the folder called ''analyses_folder'', you can type the following:
+  scp -r /Users/Gab/analyses_folder gbellucci@psychp01.rhul.ac.uk:/home/gbellucci/coolest_project
+If you have a whole data folder to transfer, you will upload it into your ''/MRIWork/MRIWork#'' folder like that:
+  scp -r /Users/Gab/data gbellucci@psychp01.rhul.ac.uk:/MRIWork/MRIWork25/data_coolest_project
+You would swap the two arguments if the folder is on the cluster, and you would need to get it onto your local computer:
+  scp -r username@psychp01.rhul.ac.uk:dirpath_remote_src dirpath_local_dest
+For example, if your directory path is to the folder on the cluster called ''results_folder'' that you need to download into your ''analyses_folder'' on your local computer, you can type the following:
+  scp -r gbellucci@psychp01.rhul.ac.uk:/home/gbellucci/results_folder /Users/Gab/analyses_folder
+==== RSYNC ====
+''rsync'', which stands for //remote sync//, is a remote and local file synchronization tool. It uses an algorithm to minimize the amount of data copied by only moving the portions of files that have changed. Please see [[https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories|here]] for more information.
+==== SSHFS ====
+''sshfs'' allows you to mount the file system on your local machine. See [[https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh|here]] for more details. Basic usage for Linux users:
+  sshfs username@psychp01.rhul.ac.uk:dirpath mountpoint [options]
+==== FileZilla ====
+''FileZilla'' is a free and open-source //File Transfer Protocol (FTP)// client that supports ''ftp'', ''ftps'' and ''sftp'' protocols. It allows the implementation of the above command-line programs through a graphical interface. Please have a look at [[http://54.236.43.240/doku.php?id=data-download|this step-by-step guide]] on how to use FileZilla.
+==== ExpanDrive ====
+An alternative to File Transfer Clients like the one mentioned above is [[https://www.expandrive.com/|ExpanDrive]]. ExpanDrive is a network filesystem client for MacOS, Microsoft Windows and Linux that facilitates mapping of local volume to many different types of cloud storage. It is different from the above File Transfer Clients because it is integrated into all applications on the operating system and does not require a file to be downloaded onto the local machine. On the contrary, remote files can be accessed, managed and changed as if they were stored locally.
 The downside is that it is a non-free commercial tool.
+[[{:backward_arrow.png?40|width: 12em}cluster-analyses|Running analyses on psychp01]][[{:forward_arrow.png?40|width: 12em}cluster-batch|Bash files and Batch system]]\\
+[[{:toc.png?40|width: 12em}cluster-toc|Return to Table of Contents]][[{:main_page.png?40|width: 12em}cluster-guide|Return to main page]]
+~~DISCUSSION|Discussion~~