Guide for using Pelican Software to Upload and Download Datasets

The Pelican Platform is an open source software designed to help researchers store and access their datasets.  It is foundation of the Open Source Data Federation (OSDF) and is the preferred tool for clients wanting to use the computing capacity of the OSPool.

For KSU researchers with a namespace (public or private) in the OSDF can use Pelican to Upload, List and Download their dataset files and directories.  If you are interested in OSDF storage for your project, contact Ramazan Aygun to get started.

 


Using Pelican

  • On the installing Pelican page, choose the latest Pelican version, your Operating System and architecture to download the software.

    Follow the installation instructions for you operating system.

    To check your installation, open a terminal window and type the command below to see if version of the software is reportd.

    • pelican --version

  • In this example, our project has a name space of osdf:///kennesaw/DemoProject and we want to upload a single file demodata.csv.  In a terminal window, navigated to where the file exists, use the following command to upload the file.

    • pelican object put demodata.csv osdf:///kennesaw/DemoProject

    This will upload the one file at the home directory of the project.

    In this example, our private project has a name space of osdf:///kennesaw-priv/DemoProject and we want to upload an entire directory of files to a new directory in the project.

    • pelican object put data_folder osdf:///kennesaw-priv/DemoProject/new_dir/?recursive

    This upload all the contents of data_folder and upload them to the new_dir within this private project.

    Look at the next section for help accepting these uploads.

  • Following the use of the pelican object put command, users will be prompted to provide a password.   This password should be retained and will be used when these files need to be shared.

    In order to approve the credentials you supplied, a token is generated that requires you to validate with a successfull KSU NetID logon. 

    • Copy the link presented after you type the password.
    • Paste the link into a browser
    • In the center of the page, click ORCID and begin typing kennesaw.
    • Select KSU and click the Log On button.
    • Complete the KSU login process.
    • Your upload should begin.

  • Use the ls command to see the contents within a directory of your project.

    • pelican object ls -l osdf:///kennesaw/ProjectName

    Use the ls command to see the contents within a nested directory of your project.

    • pelican object ls -l osdf:///kennesaw/ProjectName/dir_name/

  • Use the get command to download datasets from an OSDF project namespace.  

    • pelican object get osdf:///kennesaw/ProjectName/file_name local_directory

    Note: in your terminal, navigate to where you intend the file to be stored locally, you don't have to supply the local path as it will default to the current location.

    Use the get command to download an entire directory from a OSDF project namespace.

    • pelican object get osdf:///kennesaw/ProjectName/dir_name/?recursive local_directory