Data Access and Storage Interface (DASI)

DASI is a metadata-driven data store. It is a semantic interface for data, where the data is indexed and uniquely identified by sets of scientifically-meaningful metadata keys. DASI is modular and is compatible with multiple backends (i.e., object stores or POSIX) through diverse frontends (Python, C++, C).

Warning This project is BETA and will be experimental for the foreseeable future. Interfaces and functionality are likely to change. DO NOT use this software in any project/software that is operational.

Introduction

DASI is built on top of FDB [1], which has been developed at ECMWF [2] for previous EU projects (see Acknowledgements) and has been adapted to be highly configurable for different domains. Using FDB allows DASI to use various backends, such as POSIX, Ceph and Cortx-Motr object-storage, and NVRAM backend.

Configuration

An example configuration:

---
schema: path/to/schema/file
catalogue: toc
store: file
spaces:
   - handler: Default
   roots:
   - path: path/to/data/output1
   - path: path/to/data/output2

Schema

The schema defines the metadata keys which index and identify the data within a domain.

An example schema describing a hierarchical taxonomy of metadata keys:

[ User, Laboratory?, Project
   [ DateTime, Processing
      [ Type, Object ]]]

Installation

DASI Library (C/C++)

The supported way to install DASI library is by building from source.

Dependencies

• C/C++ compiler
• CMake
• ecbuild
• eckit
• metkit
• fdb

Build and Install

git clone https://github.com/ecmwf-projects/dasi
cd dasi

# Setup environment variables (edit as needed)
SRC_DIR=$(pwd)
BUILD_DIR=build
INSTALL_DIR=$HOME/local
export eckit_DIR=$ECKIT_DIR
export metkit_DIR=$METKIT_DIR
export fdb5_DIR=$FDB_DIR

# Create the the build directory
mkdir $BUILD_DIR
cd $BUILD_DIR

# Run ecbuild (CMake)
ecbuild --prefix=$INSTALL_DIR -- $SRC_DIR

# Build and install
make -j10
make test      # optional
make install

# Check installation
$INSTALL_DIR/bin/dasi --version

Install pydasi

The Python interface to DASI is called pydasi. It uses the cffi Python package for interfacing with the DASI C API.

Dependencies

• DASI Library (C/C++)
• cffi

Optional: Python Environment Setup

It is advised to create a virtual Python environment:

$ cd project_dir
$ python -m venv .venv
$ source .venv/bin/activate

Installation

pydasi can be installed using pip command:

$ cd project_dir
$ source .venv/bin/activate
(.venv) $ pip install cffi
(.venv) $ pip install pydasi

Contributions

Have any feedback / questions / comments / issues ? You can post them here.

The main repository is hosted on GitHub; testing, bug reports and contributions are highly welcomed and appreciated.

See also the contributors for a more complete list.

Contacts:

• James Hawkes [2]
• Simon Smart [2]
• Tiago Quintino [2]

Acknowledgements

Past and current funding and support for this project are listed in the Acknowledgements.

License

This software is licensed under the terms of the Apache License Version 2.0 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.

In applying this license, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.

Footnotes

[1]	"Fields DataBase (FDB) is a domain-specific object store"

[2]	(1, 2, 3, 4) "European Centre for Medium-Range Weather Forecasts (ECMWF)"