Quick start: Rich data management & sharing
- Login to this site with your username and password.
- Visit Data page.
- From File menu select the following actions to manage data.
- Upload, Download and other commands to manage your data.
- Edit Description command to add metadata to any file or folder.
- Share or Invite commands to grant access to your data with others.
- From ancestor menu navigate folder and list hierarchy.
- Review videos and follow instructions for command line use noted below.
See usage videos
Using command line utility to manage data
foldershare is a command line client to manage data on this site, see instructions for using it on your computer and clusters like Comet, Stampede2 & Maverick.
1. Setup for your computer
A. PHP check
FolderShare command line client requires PHP, check if your system already has PHP7.2.x+ installed
- Linux: Few Linux distributions ship with PHP, if not available see section B
- Mac OS: Recent versions of Mac OS already have PHP7.2+ installed
- Windows: PHP needs to be installed, see section B
Check php availability on your computer as follows
# On terminal issue the following command php -v
B. Installing PHP 7.2.x+
If PHP version 7.2.x+ is not available on your system, visit how to install php.
C. Download foldershare php client
foldershare supports REST web services for command-line access. You can use this to upload, download, and change files and folders from a terminal window or script on a Mac, Windows, or Linux desktop, laptop, or supercomputer.
- You must have PHP 7.2.x+ on your computer, see section A & B above
- Download and unzip the foldershare php client (requires PHP 7.2.x+)
Fetch foldershare php client from project website curl -OL https://seedmelab.org/sites/default/files/downloads/foldershare_client.1.3.zip unzip foldershare_client.1.3.zip cd foldershare_client.1.3
- Type "./foldershare --help" in a terminal window to get help. You must have php in your path or type
/full/path/to/php ./foldershare --help
- Now follow the Command Set section described on Usage from Expanse page.
2. Setup on Expanse cluster at SDSC
Use the foldershare command line client to upload/download data from Expanse's file system. The client is already installed on the expanse cluster.
- Login to Expanse
- Create an authorization file in your home directory
- Set up your terminal environment
export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules module load foldershare fs="foldershare --authfile ~/myauth.json" # Add above to your ~/.bashrc file for convenience # Alternatively login to hpcshare.sdsc.edu as follows # foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD
- Type "foldershare --help" in a terminal window to get help.
foldershare --help # Alternatively fs --help
- See command below
3. FolderShare client command set
Many Linux-style file and folder commands are supported, such as:
- ls to list a folder's contents.
- stat to get the status of a file or folder.
- mkdir to make a new folder.
- rm to remove a file or folder tree.
- rmdir to remove an empty folder.
- cp to copy a file or folder.
- mv to move a file or folder.
Additional commands for data transfer and update:
- download to download a file or folder.
- upload to upload a file or folder.
- update to add or change a file or folder description.
4. Command line usage
Following description provides sample interaction via the command line interface
Create a variable to hold the following information
# Complete step A1 which loads the foldershare module fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"
Create a new folder
# foldershare mkdir OPTIONS FULL_REMOTE_PATH $fs mkdir --help $fs mkdir /test
Upload a local file and folder to remote location and display progress
# foldershare upload OPTIONS LOCAL_PATH FULL_REMOTE_PATH $fs upload --help $fs upload -v /expanse/community/seedmelab/data/123.zip /test $fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data /
List contents of remote folder
$fs ls --help $fs ls /sample-structured-data $fs ls -l /sample-structured-data $fs ls -lR /
Download a remote folder and its contents and display progress
# Create temporary directory for downloading content download_dir=`mktemp -d -t hpcshare-XXXXXX` # foldershare download OPTIONS FULL_REMOTE_PATH LOCAL_PATH $fs download --help $fs download -v --sync /test $download_dir # Check download on local path ls -l $download_dir
Update description of remote file/folder
# foldershare update description 'arbitrary text' FULL_REMOTE_PATH $fs update --help $fs update description 'Hello world!' /test/123.zip
Get description of remote file/folder with stat command and different output format
# foldershare --format TYPE command # Supported TYPE: auto(default) full key value linux text $fs --format text stat /test/123.zip
Copy remote file/folder to another remote location
# foldershare cp OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH $fs cp --help # --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation $fs cp --wait /test/child /test/subfolder
Move remote folder to another remote location
# foldershare mv OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH $fs mv --help # --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation $fs mv --wait /test/child /test/subfolder
Remove remote file/folder
# foldershare rm OPTIONS /FULL_REMOTE_PATH $fs rm --help $fs rm -rf /test/subfolder