Data management and sharing

Quick start: Rich data management & sharing

Using command line utility to manage data

foldershare is a command line client to manage data on this site, see instructions for using it on your computer and clusters like Comet, Stampede2 & Maverick.

1. Setup for your computer

A. PHP check

FolderShare command line client requires PHP, check if your system already has PHP7.2.x+ installed

1. Linux: Few Linux distributions ship with PHP, if not available see section B
2. Mac OS: Recent versions of Mac OS already have PHP7.2+ installed
3. Windows: PHP needs to be installed, see section B

Check php availability on your computer as follows

# On terminal issue the following command
php -v

---

B. Installing PHP 7.2.x+

If PHP version 7.2.x+ is not available on your system, visit how to install php.

---

C. Download foldershare php client

foldershare supports REST web services for command-line access. You can use this to upload, download, and change files and folders from a terminal window or script on a Mac, Windows, or Linux desktop, laptop, or supercomputer.

1. You must have PHP 7.2.x+ on your computer, see section A & B above 
2. Download and unzip the foldershare php client (requires PHP 7.2.x+)

3. Fetch foldershare php client from project website
   curl -OL https://seedmelab.org/sites/default/files/downloads/foldershare_client.1.3.zip 
   unzip foldershare_client.1.3.zip
   cd foldershare_client.1.3
   

4. Type "./foldershare --help" in a terminal window to get help. You must have php in your path or type

    /full/path/to/php ./foldershare --help

5. Now follow the Command Set section described on Usage from Expanse page.

2. Setup on Expanse cluster at SDSC

Use the foldershare command line client to upload/download data from Expanse's file system. The client is already installed on the expanse cluster.

1. Login to Expanse
2. Create an authorization file in your home directory
3. Set up your terminal environment

   export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules 
   module load foldershare
   fs="foldershare --authfile ~/myauth.json"
   # Add above to your ~/.bashrc file for convenience
   # Alternatively login to hpcshare.sdsc.edu as follows   
   # foldershare --host hostname --username YOUR_USERNAME --password  YOUR_PASSWORD

4. Type "foldershare --help" in a terminal window to get help. 

    foldershare --help
    # Alternatively
    fs --help

5. See command below

3. FolderShare client command set

Many Linux-style file and folder commands are supported, such as:

• ls to list a folder's contents.
• stat to get the status of a file or folder.
• mkdir to make a new folder.
• rm to remove a file or folder tree.
• rmdir to remove an empty folder.
• cp to copy a file or folder.
• mv to move a file or folder.

Additional commands for data transfer and update:

• download to download a file or folder.
• upload to upload a file or folder.
• update to add or change a file or folder description.

 

4. Command line usage 

Following description provides sample interaction via the command line interface

Create a variable to hold the following information

# Complete step A1 which loads the foldershare module
fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"

Create a new folder 

# foldershare mkdir OPTIONS FULL_REMOTE_PATH
$fs mkdir --help
$fs mkdir /test

Upload a local file and folder to remote location and display progress

# foldershare upload OPTIONS LOCAL_PATH FULL_REMOTE_PATH
$fs upload --help
$fs upload -v /expanse/community/seedmelab/data/123.zip /test
$fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data / 

List contents of remote folder

$fs ls --help
$fs ls /sample-structured-data
$fs ls -l /sample-structured-data
$fs ls -lR /

Download a remote folder and its contents and display progress

# Create temporary directory for downloading content
download_dir=`mktemp -d -t hpcshare-XXXXXX`
# foldershare download OPTIONS FULL_REMOTE_PATH LOCAL_PATH 
$fs download --help
$fs download -v --sync /test $download_dir
# Check download on local path
ls -l $download_dir 

Update description of remote file/folder 

# foldershare update description 'arbitrary text' FULL_REMOTE_PATH
$fs update --help
$fs update description 'Hello world!' /test/123.zip 

Get description of remote file/folder with stat command and different output format

# foldershare --format TYPE command 
# Supported TYPE: auto(default) full key value linux text
$fs --format text stat /test/123.zip 

Copy remote file/folder to another remote location

# foldershare cp OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs cp --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs cp --wait /test/child /test/subfolder

Move remote folder to another remote location

# foldershare mv OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs mv --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs mv --wait /test/child /test/subfolder 

Remove remote file/folder 

# foldershare rm OPTIONS /FULL_REMOTE_PATH 
$fs rm --help
$fs rm -rf /test/subfolder