| Both sides previous revisionPrevious revision | |
| cluster-file_transfer [2026/09/03 12:54] – gabriele | cluster-file_transfer [2026/09/03 12:58] (current) – gabriele |
|---|
| </code> | </code> |
| |
| == 8. If the data do not fit == | == 8. Move the data onto psychp01 == |
| | |
| Three ways to reduce the size, in order of usefulness: | |
| | |
| * **Fewer participants**: use the ''subjects.txt'' loop above. | |
| * **Fewer modalities**: ask for ''sub-*/ses-02/dwi'' instead of the whole of ''ses-02''. | |
| * **Download, analyse, release**: once you have finished with a participant, free the space again with ''git annex drop <path>''. The file name stays in place and can be downloaded again at any time. | |
| | |
| == 9. Move the data onto psychp01 == | |
| |
| Once the download is finished and you are happy with your folder structure, move the data to ''/MRIWork'' exactly as described above in [[cluster-file_transfer#to_move_the_data_from_the_graphics_server_onto_psychp01|To move the data from the "graphics" server onto psychp01]]. | Once the download is finished and you are happy with your folder structure, move the data to ''/MRIWork'' exactly as described above in [[cluster-file_transfer#to_move_the_data_from_the_graphics_server_onto_psychp01|To move the data from the "graphics" server onto psychp01]]. |
| |
| * **The files look like they are already there, but they are not.** After cloning, you can see and browse every file name, including the MEG ''.ds'' folders. Those are placeholders until you run ''git annex get''. Analysis software pointed at a dataset that has not been downloaded will usually report //corrupt data// rather than //missing files//, which is confusing. To list what is still missing in the current folder: ''git annex find . %%--%%not %%--%%in here'' | * **The files look like they are already there, but they are not.** After cloning, you can see and browse every file name, including the MEG ''.ds'' folders. Those are placeholders until you run ''git annex get''. Analysis software pointed at a dataset that has not been downloaded will usually report //corrupt data// rather than //missing files//, which is confusing. To list what is still missing in the current folder: ''git annex find . %%--%%not %%--%%in here'' |
| * **Never run ''git annex get'' without a path.** With no path it means "download everything", i.e. 1.95 TB. The same applies to ''gin sync %%--%%content''. | * **Never run ''git annex get'' without a path.** With no path it means "download everything", i.e. 1.95 TB. Do that only if you want to download the full dataset. The same applies to ''gin sync %%--%%content''. |
| * **The download links on the GIN wiki are dead.** The ''gin'' client is now distributed through [[https://github.com/G-Node/gin-cli/releases|GitHub releases]]. It is already installed on the "graphics" server, and in practice you can do everything with plain ''git'' and ''git annex'' anyway. | * **The download links on the GIN wiki are dead.** The ''gin'' client is now distributed through [[https://github.com/G-Node/gin-cli/releases|GitHub releases]]. It is already installed on the "graphics" server, and in practice you can do everything with plain ''git'' and ''git annex'' anyway. |
| * **Remember the warning above**: the "graphics" server is not backed up. Do not leave the only copy of anything there. | * **Remember the warning above**: the "graphics" server is not backed up. Do not leave the only copy of anything there. |
| |
| == Other datasets == | |
| |
| The same procedure works for any dataset hosted on GIN, and, with ''datalad'' in place of ''git annex'', for [[https://openneuro.org|OpenNeuro]] and other ''git-annex'' based repositories. Only the repository address changes. | |
| |
| |
