1Introduction

HDFql stands for “Hierarchical Data Format query language” and is the first tool that enables users to manage HDF5¹ files through a high-level (declarative) language. This language was designed to be simple to use and similar to SQL, thus dramatically reducing the learning effort. HDFql can be seen as a solid alternative to the C API (which contains more than 400 low-level functions that are far from easy to use!) and to existing wrappers for C++, Java, Python, C#, Fortran, R, and MATLAB for manipulating HDF5 files. In addition, whenever possible, it automatically employs parallelism by leveraging all available cores and, in MPI clusters, nodes to speed up operations and process large volumes of data, hiding the inherent complexity from the user.

As an example, imagine that one needs to create an HDF5 file named “my_file.h5” and, inside it, a group named “my_group” containing a one-dimensional (size 3) dataset named “my_dataset” of data type integer. Additionally, the dataset is compressed using ZLIB and initialized with the values 4, 8, and 6. In HDFql, this can be easily implemented as follows:

create and use file my_file.h5
create dataset my_group/my_dataset as int(3) enable zlib values(4, 8, 6)

In contrast, using the C API for the same example is quite cumbersome:

hid_t file;
hid_t group;
hid_t dataspace; 
hid_t property;
hid_t dataset;
hsize_t dimension;
int value[3];
file = H5Fcreate("my_file.h5", H5F_ACC_EXCL, H5P_DEFAULT, H5P_DEFAULT);
group = H5Gcreate(file, "my_group", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
dimension = 3;
dataspace = H5Screate_simple(1, &dimension, NULL); 
property = H5Pcreate(H5P_DATASET_CREATE); 
H5Pset_chunk(property, 1, &dimension); 
H5Pset_deflate(property, 9);
dataset = H5Dcreate(group, "my_dataset", H5T_NATIVE_INT, dataspace, H5P_DEFAULT, property, H5P_DEFAULT);
value[0] = 4; 
value[1] = 8;
value[2] = 6;
H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, &value);