Transferring Data Between Savio and Your bDrive (Google Drive) Account

This document describes how you can transfer files between Savio, UC Berkeley's high performance computing cluster, and your UC Berkeley bDrive account, which uses Google’s Google Drive cloud storage service.

As noted in the instructions below, you'll perform these file transfers using the gdrive utility on Savio.

Perform setup

To set up the gdrive tool to connect to your account on bDrive:

1. Log into Savio's Data Transfer Node (DTN), dtn.brc.berkeley.edu

For example, at your computer's terminal or command-line prompt, enter:

ssh myusername@dtn.brc.berkeley.edu

where myusername is the name of your user account on Savio.

Note: the gdrive tool is only accessible when you're logged into the DTN.

2. At a command prompt, enter the following command:

gdrive about

3. Follow the instructions shown by that command.

Doing so will obtain a "token" from Google, and (by default) save it in a ".gdrive" directory within your home directory on Savio. Once that token is present, you can then transfer files at will between your Savio account and your bDrive account.

Note: you'll usually need to perform this setup only once, unless you revoke this token, either by deleting the token file from your Savio account, or revoking permission for the gdrive app in the settings for your Google Drive account.

Transferring files between Savio and bDrive

To transfer files between Savio and your account on bDrive, use the gdrive tool, which is available on Savio's Data Transfer Node.

To see what commands are available when using this tool, enter:

gdrive help

To view help for a specific command, enter "gdrive help" followed by the name of that command. For instance, to view help for gdrive's "upload" command, enter:

gdrive help upload

To view all of gdrive's help information on a single page, please see the "Usage" section of the current gdrive documentation ("README" file). (Note: this documentation could at times reflect a different version of gdrive than is currently installed on Savio.)

Copying ("uploading") files and directories from Savio to bDrive

By default, when you copy files and directories (aka "folders") from Savio to your account on bDrive, they will initially be copied to your "My Drive" area on bDrive. (You can later move or copy them elsewhere.)

Uploading a file

To copy (“upload”) a single file from Savio to bDrive, enter the command:

gdrive upload filename

where filename represents the name of the file on Savio that you want to copy to bDrive; e.g.:

gdrive upload testfile.hdf5

Uploading a folder

To copy (“upload”) a single directory on Savio, with all of its contents included, to create a corresponding folder on bDrive, enter the command:

gdrive upload --recursive directoryname

where directoryname represents the name of the directory on Savio that you want to copy to bDrive; e.g.:

gdrive upload --recursive my_tensorflow_output

Copying ("downloading") files and directories from bDrive to Savio

By default, when you copy files and directories (aka "folders") from your account on bDrive to Savio, they will be copied to your current directory on Savio. (You can later move or copy them elsewhere.)

Each of the following commands requires that you supply a fileID, the unique identifier of the file or folder on bDrive.

Finding fileIDs

The simplest way to obtain a fileID for a single file or folder on bDrive, is to open the relevant file or folder on the bDrive website and copy the long set of numbers and letters contained within your browser's URL. These unique identifiers (fileIDs) will look similar to those in boldface in these examples:

Unique identifier (fileID) for a file (a non-existent example, for illustration only):
https://docs.google.com/document/d/1EA4UInWV5A3SFvWR4ETjBgb0AGhhLB83EIJYe5w/edit

Unique identifier (fileID) for a folder (a non-existent example, for illustration only):
https://drive.google.com/drive/u/1/folders/0BF39FU3k1XRVhl0FNrsgYLXc

See Finding fileIDs for multiple files and folders on bDrive, below, for another way to obtain fileIDs in bulk, via gdrive commands.

Downloading a single file

To copy (“download”) a single file from bDrive to Savio, enter the command:

gdrive download fileID

where fileID represents the unique identifier of the file on bDrive that you want to copy to Savio; e.g.:

gdrive download 1EA4UInWV5A3SFvWR4ETjBgb0AGhhLB83EIJYe5w

Downloading a folder

To copy (“download”) a single folder on bDrive, with all of its contents included, to create a corresponding directory on Savio, enter the command:

gdrive download --recursive fileID

where fileID represents the name of the folder on bDrive that you want to copy to Savio; e.g.:

gdrive download --recursive 0BF39FU3k1XRVhl0FNrsgYLX

Finding fileIDs for multiple files and folders on bDrive

When you need to copy multiple files and/or folders from bDrive to Savio, it can be time consuming to navigate the graphical interface of the bDrive website to obtain their individual fileIDs. Fortunately, there are gdrive commands you can use instead to obtain these unique identifiers for many files and/or folders all at once.

The gdrive list --query command will allow you to search for items that are visible to you on your bDrive account - both those owned by you and those others have shared with you - and return information about them such as their names, sizes, creation dates, and fileIDs. This command uses the search operators of the Google Drive REST APIs.

A few examples:

Search for and list a single item (file or folder) by exact name:
gdrive list -q "name = 'AM_SIM_Abaqus_Extend.zip'"

Search for and list all items (files or folders) containing "aba" in their names (case insensitive):
gdrive list --query "name contains 'aba'"

Search for and list all items (files or folders) containing "aba" in their names (case insensitive), for which you have (or at least share) ownership, and ignoring files in the bDrive (Google Drive) Trash:
gdrive list --query "trashed = false and 'me' in owners and name contains 'aba'"

Search for and list all items (files or folders) within the folder whose fileID is 0BF39FU3k1XRVhl0FNrsgYLX (a non-existent example, for illustration only):
gdrive list --query "'0BF39FU3k1XRVhl0FNrsgYLX' in parents"

Removing access to your bDrive account from Savio

By performing the one-time setup steps listed above, the gdrive application stores a token in a file within your home directory. This token permits anyone who has the ability to log into your Savio account to also access your bDrive account from there.

When and if you want to disable that access, from Savio to your bDrive account, you can simply delete the "token_v2.json" file, which (by default) can be found in the ".gdrive" directory within your home directory on Savio; e.g.:

rm $HOME/.gdrive/token_v2.json

(Note that the ".gdrive" directory’s name begins with a period ("."), so to use the 'ls' utility to view its contents, you’ll need to use 'ls -l $HOME/.gdrive' (where the letter after the dash is a lowercase letter "L").)

When you want to re-enable that access, you can once again follow the steps described above in Perform setup.