6Language
HDFql is a high-level (declarative) language to manage HDF5 files in a simple and natural way. It was designed to be similar to SQL (wherever possible), so that its learning effort is kept to a minimum while still providing great power and flexibility to the programmer. This chapter describes data types, processing options to further transform result sets, redirecting options to read/write data/result sets from/into disparate input/output sources, and operations (i.e. the language itself) available in HDFql. It also introduces the text formatting conventions used throughout this chapter to describe HDFql operations (Table 6.1), as well as a summary of existing operations (Table 6.2). Before continuing, it is recommended to first read the HDF5 User’s Guide available at https://support.hdfgroup.org/documentation/hdf5/latest/_u_g.html to facilitate understanding of the current chapter.
| Convention | Description | Example |
|---|---|---|
| Bold | Keyword that must be typed exactly as shown | CREATE |
| Italic | Value that the programmer must supply | dataset_name |
| Between brackets ([]) | Optional keyword/value | [DATASET] |
| Between braces ({}) | Logical grouping of keywords/values (to ease understanding) | {[TRUNCATE] BINARY FILE file_name} |
| Separated with a pipe (|) | Set of keywords/values from which one must be chosen | GROUP | DATASET | ATTRIBUTE |
| Asterisk (*) | Keyword/value that may be supplied zero or more times | group_name [, group_name]* |
| Plus (+) | Keyword/value that must be supplied one or more times | {, [chunk_dim]}+ |
Table 6.1 – HDFql operations text formatting conventions
| Operation | Description |
|---|---|
| CREATE DIRECTORY | Create a directory |
| CREATE FILE | Create an HDF5 file |
| CREATE GROUP | Create an HDF5 group |
| CREATE DATASET | Create an HDF5 dataset |
| CREATE ATTRIBUTE | Create an HDF5 attribute |
| CREATE [SOFT | HARD] LINK | Create an HDF5 soft or hard link |
| CREATE EXTERNAL LINK | Create an HDF5 external link |
| ALTER DIMENSION | Alter (i.e. change) the dimensions of an existing HDF5 dataset |
| RENAME DIRECTORY | Rename (or move) an existing directory |
| RENAME FILE | Rename (or move) an existing file |
| RENAME | Rename (or move) an existing HDF5 group, dataset, attribute, (soft) link, or external link |
| COPY FILE | Copy an existing file |
| COPY | Copy an existing HDF5 group, dataset, attribute, (soft) link, or external link |
| DROP DIRECTORY | Drop (i.e. delete) an existing directory |
| DROP FILE | Drop (i.e. delete) an existing file |
| DROP | Drop (i.e. delete) an existing HDF5 group, dataset, attribute, (soft) link, or external link |
| INSERT | Insert (i.e. write) data into an HDF5 dataset or attribute |
| SELECT | Select (i.e. read) data from an HDF5 dataset or attribute |
| SHOW FILE VALIDITY | Show (i.e. get) the validity of a file (i.e. whether it is a valid HDF5 file or not) |
| SHOW USE DIRECTORY | Show (i.e. get) the working directory currently in use |
| SHOW USE FILE | Show (i.e. get) the HDF5 file currently in use or check if a certain HDF5 file is used (i.e. opened) |
| SHOW ALL USE FILE | Show (i.e. get) all HDF5 files in use (i.e. open) |
| SHOW USE GROUP | Show (i.e. get) the HDF5 group currently in use |
| SHOW | Show (i.e. get) HDF5 objects (i.e. groups, datasets, attributes, (soft) links, and external links) or check for the existence of an object |
| SHOW TYPE | Show (i.e. get) the type of an HDF5 object (i.e. group, dataset, or attribute) |
| SHOW DATA TYPE | Show (i.e. get) the data type of an HDF5 dataset or attribute or of its members |
| SHOW MEMBER | Show (i.e. get) the members of an HDF5 dataset or attribute |
| SHOW MASK | Show (i.e. get) the (filter) mask of an HDF5 dataset |
| SHOW ENDIANNESS | Show (i.e. get) the endianness of an HDF5 dataset or attribute or of its members |
| SHOW CHARSET | Show (i.e. get) the charset of an HDF5 dataset or attribute or of its members |
| SHOW STORAGE TYPE | Show (i.e. get) the storage type (layout) of an HDF5 dataset |
| SHOW STORAGE ALLOCATION | Show (i.e. get) the storage allocation of an HDF5 dataset |
| SHOW STORAGE DIMENSION | Show (i.e. get) the storage dimensions of an HDF5 dataset |
| SHOW DIMENSION | Show (i.e. get) the dimensions of an HDF5 dataset or attribute |
| SHOW ORDER | Show (i.e. get) the (creation) order strategy of an HDF5 group or dataset |
| SHOW PAD | Show (i.e. get) the padding of an HDF5 dataset or attribute or of its members |
| SHOW TAG | Show (i.e. get) the tag of an HDF5 dataset or attribute or of its members |
| SHOW OFFSET | Show (i.e. get) the member offsets of an HDF5 dataset or attribute |
| SHOW FILL TYPE | Show (i.e. get) the fill type of an HDF5 dataset |
| SHOW FILL | Show (i.e. get) the fill values of an HDF5 dataset |
| SHOW FILE SIZE | Show (i.e. get) the size (in bytes) of a file or of the HDF5 file currently in use |
| SHOW USERBLOCK SIZE | Show (i.e. get) the size (in bytes) of a user-defined block of data stored within an HDF5 file or within the HDF5 file currently in use |
| SHOW USERBLOCK | Show (i.e. get) the user-defined block of data stored within an HDF5 file or within the HDF5 file currently in use |
| SHOW ALIGNMENT | Show (i.e. get) the alignment parameter values for accessing HDF5 files |
| SHOW SIZE | Show (i.e. get) the size (in bytes) of an HDF5 dataset or attribute |
| SHOW HDFQL VERSION | Show (i.e. get) the version of the HDFql library |
| SHOW HDF5 VERSION | Show (i.e. get) the version of the HDF5 library used by HDFql |
| SHOW MPI VERSION | Show (i.e. get) the version of the MPI library used by HDFql |
| SHOW DIRECTORY | Show (i.e. get) the directories within a directory or check for the existence of a directory |
| SHOW FILE | Show (i.e. get) the files within a directory or check for the existence of a file |
| SHOW EXECUTE STATUS | Show (i.e. get) the status of the last executed operation |
| SHOW LIBRARY BOUNDS | Show (i.e. get) the library bound values for creating or opening HDF5 files |
| SHOW CACHE | Show (i.e. get) the cache parameter values for accessing HDF5 files or datasets |
| SHOW ATOMIC | Show (i.e. get) the atomicity for accessing HDF5 files in an MPI environment |
| SHOW MPI SIZE | Show (i.e.get) the number (i.e. size) of processes associated with the default MPI communicator (MPI_COMM_WORLD) or with a user-defined MPI communicator |
| SHOW MPI RANK | Show (i.e. get) the rank of the calling process associated with the default MPI communicator (MPI_COMM_WORLD) or with a user-defined MPI communicator |
| SHOW MPI ROOT | Show (i.e. get) the MPI root process rank (for gathering processed (HDF5) data by other MPI processes) |
| SHOW MPI HINT | Show (i.e. get) MPI hints for accessing HDF5 files in an MPI environment |
| SHOW EXTERNAL LINK PREFIX | Show (i.e. get) the prefix to prepend to file names stored in HDF5 external links |
| SHOW FLUSH | Show (i.e. get) the status of automatic flushing |
| SHOW THREAD | Show (i.e. get) the number of (CPU) threads to use when executing operations that support parallelism |
| SHOW PLUGIN PATH | Show (i.e. get) the path where plugins (in the form of shared libraries) are searched for and dynamically loaded by HDFql/HDF5 library |
| SHOW DECIMAL SCALE | Show (i.e. get) the decimal scale (i.e. number of decimal places) to use when writing fractional numbers into a text or Excel file |
| SHOW DEBUG | Show (i.e. get) the status of the debug mechanism |
| USE DIRECTORY | Use (i.e. open) a directory for subsequent operations |
| USE FILE | Use (i.e. open) an HDF5 file for subsequent operations |
| USE GROUP | Use (i.e. open) an HDF5 group for subsequent operations |
| FLUSH | Flush the entire virtual HDF5 file (global) or the specific HDF5 file (local) currently in use |
| CLOSE FILE | Close the HDF5 file currently in use or a certain HDF5 file in use (i.e. opened) |
| CLOSE ALL FILE | Close all HDF5 files in use (i.e. opened) |
| CLOSE GROUP | Close the HDF5 group currently in use (i.e. opened) |
| SET USERBLOCK | Set (i.e. write) a user-defined block of data stored within an HDF5 file or within the HDF5 file currently in use |
| SET ALIGNMENT | Set the alignment for accessing HDF5 files |
| SET LIBRARY BOUNDS | Set the library bound values for creating and opening HDF5 files |
| SET CACHE | Set the cache parameter values for accessing HDF5 files or datasets |
| SET ATOMIC | Set the atomicity for accessing HDF5 files in an MPI environment to enabled or disabled |
| SET MPI ROOT | Set the MPI root process rank for gathering processed (HDF5) data by other MPI processes |
| SET MPI HINT | Set MPI hints for accessing HDF5 files in an MPI environment |
| SET EXTERNAL LINK PREFIX | Set the prefix to prepend to file names stored in HDF5 external links |
| SET FLUSH | Set the automatic flushing of the entire virtual HDF5 file (global) or only the HDF5 file (local) currently in use to enabled or disabled |
| SET THREAD | Set the number of (CPU) threads to use when executing operations that support parallelism |
| SET PLUGIN PATH | Set the path where plugins (in the form of shared libraries) are searched for and dynamically loaded by HDFql/HDF5 library |
| SET DECIMAL SCALE | Set the decimal scale (i.e. number of decimal places) to use when writing fractional numbers into a text or Excel file |
| SET DEBUG | Set the debug mechanism to enabled or disabled |
Table 6.2 – HDFql operations