Data management and sharing

Quick start: Rich data management & sharing

  1. Login to this site with your username and password.
  2. Visit Data page.
  3. From File menu File menu icon select the following actions to manage data.
    1. UploadDownload and other commands to manage your data.
    2. Edit Description command to add metadata to any file or folder.
    3. Share or Invite commands to grant access to your data with others.
  4. From ancestor menu ancestor menu icon navigate folder and list hierarchy.
  5. Review videos and follow instructions for command line use noted below.

See usage videos

Usage via Web browser
Using REST Client
Customizing SeedMeLab

Using command line utility to manage data

foldershare is a command line client to manage data on this site, see instructions for using it on your computer and clusters like Comet, Stampede2 & Maverick.

1. Setup for your computer

A. PHP check

FolderShare command line client requires PHP, check if your system already has PHP7.2.x+ installed

  1. Linux: Few Linux distributions ship with PHP, if not available see section B
  2. Mac OS: Recent versions of Mac OS already have PHP7.2+ installed
  3. Windows: PHP needs to be installed, see section B

Check php availability on your computer as follows

# On terminal issue the following command
php -v

B. Installing PHP 7.2.x+

If PHP version 7.2.x+ is not available on your system, visit how to install php.


C. Download foldershare php client

foldershare supports REST web services for command-line access. You can use this to upload, download, and change files and folders from a terminal window or script on a Mac, Windows, or Linux desktop, laptop, or supercomputer.

  1. You must have PHP 7.2.x+ on your computer, see section A & B above 
  2. Download and unzip the foldershare php client (requires PHP 7.2.x+)
  3. Fetch foldershare php client from project website
    curl -OL https://seedmelab.org/sites/default/files/downloads/foldershare_client.1.3.zip 
    unzip foldershare_client.1.3.zip
    cd foldershare_client.1.3
    
  4. Type "./foldershare --help" in a terminal window to get help. You must have php in your path or type
     /full/path/to/php ./foldershare --help
  5. Now follow the Command Set section described on Usage from Expanse page.

2. Setup on Expanse cluster at SDSC

Use the foldershare command line client to upload/download data from Expanse's file system. The client is already installed on the expanse cluster.

  1. Login to Expanse
  2. Create an authorization file in your home directory
  3. Set up your terminal environment
    export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules 
    module load foldershare
    fs="foldershare --authfile ~/myauth.json"
    # Add above to your ~/.bashrc file for convenience
    # Alternatively login to hpcshare.sdsc.edu as follows   
    # foldershare --host hostname --username YOUR_USERNAME --password  YOUR_PASSWORD
  4. Type "foldershare --help" in a terminal window to get help. 
     foldershare --help
     # Alternatively
     fs --help
  5. See command below

3. FolderShare client command set

Many Linux-style file and folder commands are supported, such as:

  • ls to list a folder's contents.
  • stat to get the status of a file or folder.
  • mkdir to make a new folder.
  • rm to remove a file or folder tree.
  • rmdir to remove an empty folder.
  • cp to copy a file or folder.
  • mv to move a file or folder.

Additional commands for data transfer and update:

  • download to download a file or folder.
  • upload to upload a file or folder.
  • update to add or change a file or folder description.

 

4. Command line usage 

Following description provides sample interaction via the command line interface

Create a variable to hold the following information

# Complete step A1 which loads the foldershare module
fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"

Create a new folder 

# foldershare mkdir OPTIONS FULL_REMOTE_PATH
$fs mkdir --help
$fs mkdir /test

Upload a local file and folder to remote location and display progress

# foldershare upload OPTIONS LOCAL_PATH FULL_REMOTE_PATH
$fs upload --help
$fs upload -v /expanse/community/seedmelab/data/123.zip /test
$fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data / 

List contents of remote folder

$fs ls --help
$fs ls /sample-structured-data
$fs ls -l /sample-structured-data
$fs ls -lR /

Download a remote folder and its contents and display progress

# Create temporary directory for downloading content
download_dir=`mktemp -d -t hpcshare-XXXXXX`
# foldershare download OPTIONS FULL_REMOTE_PATH LOCAL_PATH 
$fs download --help
$fs download -v --sync /test $download_dir
# Check download on local path
ls -l $download_dir 

Update description of remote file/folder 

# foldershare update description 'arbitrary text' FULL_REMOTE_PATH
$fs update --help
$fs update description 'Hello world!' /test/123.zip 

Get description of remote file/folder with stat command and different output format

# foldershare --format TYPE command 
# Supported TYPE: auto(default) full key value linux text
$fs --format text stat /test/123.zip 

Copy remote file/folder to another remote location

# foldershare cp OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs cp --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs cp --wait /test/child /test/subfolder

Move remote folder to another remote location

# foldershare mv OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs mv --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs mv --wait /test/child /test/subfolder 

Remove remote file/folder 

# foldershare rm OPTIONS /FULL_REMOTE_PATH 
$fs rm --help
$fs rm -rf /test/subfolder