This document describes how you can transfer files between Savio, UC Berkeley's high performance computing cluster, and your UC Berkeley bDrive account, which uses Google’s Google Drive cloud storage service.
As noted in the instructions below, you'll perform these file transfers using the
gdrive utility on Savio.
To set up the gdrive tool to connect to your account on bDrive:
1. Log into Savio's Data Transfer Node (DTN), dtn.brc.berkeley.edu
For example, at your computer's terminal or command-line prompt, enter:
myusername is the name of your user account on Savio.
Note: the gdrive tool is only accessible when you're logged into the DTN.
2. At a command prompt, enter the following command:
3. Follow the instructions shown by that command.
Doing so will obtain a "token" from Google, and (by default) save it in a "
.gdrive" directory within your home directory on Savio. Once that token is present, you can then transfer files at will between your Savio account and your bDrive account.
Note: you'll usually need to perform this setup only once, unless you revoke this token, either by deleting the token file from your Savio account, or revoking permission for the
gdrive app in the settings for your Google Drive account.
Transferring files between Savio and bDrive
To transfer files between Savio and your account on bDrive, use the
gdrive tool, which is available on Savio's Data Transfer Node.
To see what commands are available when using this tool, enter:
To view help for a specific command, enter "gdrive help" followed by the name of that command. For instance, to view help for gdrive's "upload" command, enter:
gdrive help upload
To view all of gdrive's help information on a single page, please see the "Usage" section of the current gdrive documentation ("README" file). (Note: this documentation could at times reflect a different version of gdrive than is currently installed on Savio.)
Copying ("uploading") files and directories from Savio to bDrive
By default, when you copy files and directories (aka "folders") from Savio to your account on bDrive, they will initially be copied to your "My Drive" area on bDrive. (You can later move or copy them elsewhere.)
Uploading a file
To copy (“upload”) a single file from Savio to bDrive, enter the command:
gdrive upload filename
filename represents the name of the file on Savio that you want to copy to bDrive; e.g.:
gdrive upload testfile.hdf5
Uploading a folder
To copy (“upload”) a single directory on Savio, with all of its contents included, to create a corresponding folder on bDrive, enter the command:
gdrive upload --recursive directoryname
directoryname represents the name of the directory on Savio that you want to copy to bDrive; e.g.:
gdrive upload --recursive my_tensorflow_output
Copying ("downloading") files and directories from bDrive to Savio
By default, when you copy files and directories (aka "folders") from your account on bDrive to Savio, they will be copied to your current directory on Savio. (You can later move or copy them elsewhere.)
Each of the following commands requires that you supply a
fileID, the unique identifier of the file or folder on bDrive.
The simplest way to obtain a fileID for a single file or folder on bDrive, is to open the relevant file or folder on the bDrive website and copy the long set of numbers and letters contained within your browser's URL. These unique identifiers (fileIDs) will look similar to those in boldface in these examples:
Unique identifier (fileID) for a file (a non-existent example, for illustration only):
Unique identifier (fileID) for a folder (a non-existent example, for illustration only):
See Finding fileIDs for multiple files and folders on bDrive, below, for another way to obtain
fileIDs in bulk, via
Downloading a single file
To copy (“download”) a single file from bDrive to Savio, enter the command:
gdrive download fileID
fileID represents the unique identifier of the file on bDrive that you want to copy to Savio; e.g.:
gdrive download 1EA4UInWV5A3SFvWR4ETjBgb0AGhhLB83EIJYe5w
Downloading a folder
To copy (“download”) a single folder on bDrive, with all of its contents included, to create a corresponding directory on Savio, enter the command:
gdrive download --recursive fileID
fileID represents the name of the folder on bDrive that you want to copy to Savio; e.g.:
gdrive download --recursive 0BF39FU3k1XRVhl0FNrsgYLX
Finding fileIDs for multiple files and folders on bDrive
When you need to copy multiple files and/or folders from bDrive to Savio, it can be time consuming to navigate the graphical interface of the bDrive website to obtain their individual
fileIDs. Fortunately, there are
gdrive commands you can use instead to obtain these unique identifiers for many files and/or folders all at once.
gdrive list --query command will allow you to search for items that are visible to you on your bDrive account - both those owned by you and those others have shared with you - and return information about them such as their names, sizes, creation dates, and
fileIDs. This command uses the search operators of the Google Drive REST APIs.
A few examples:
Search for and list a single item (file or folder) by exact name:
gdrive list -q "name = 'AM_SIM_Abaqus_Extend.zip'"
Search for and list all items (files or folders) containing "aba" in their names (case insensitive):
gdrive list --query "name contains 'aba'"
Search for and list all items (files or folders) containing "aba" in their names (case insensitive), for which you have (or at least share) ownership, and ignoring files in the bDrive (Google Drive) Trash:
gdrive list --query "trashed = false and 'me' in owners and name contains 'aba'"
Search for and list all items (files or folders) within the folder whose
0BF39FU3k1XRVhl0FNrsgYLX (a non-existent example, for illustration only):
gdrive list --query "'0BF39FU3k1XRVhl0FNrsgYLX' in parents"
Removing access to your bDrive account from Savio
By performing the one-time setup steps listed above, the gdrive application stores a token in a file within your home directory. This token permits anyone who has the ability to log into your Savio account to also access your bDrive account from there.
When and if you want to disable that access, from Savio to your bDrive account, you can simply delete the "
token_v2.json" file, which (by default) can be found in the
".gdrive" directory within your home directory on Savio; e.g.:
(Note that the "
.gdrive" directory’s name begins with a period ("
."), so to use the '
ls' utility to view its contents, you’ll need to use
'ls -l $HOME/.gdrive' (where the letter after the dash is a lowercase letter "
When you want to re-enable that access, you can once again follow the steps described above in Perform setup.