6.4.4CREATE DATASET

#Syntax

#Description

Create an HDF5 dataset named dataset_name. Multiple datasets can be created at once by separating their names with a comma (,). If dataset_name already exists, it is not overwritten, no subsequent datasets are created, and an error is raised. To overwrite an existing dataset, specify the keyword TRUNCATE (all data stored in the dataset will be permanently lost). In case dataset_name includes intermediate groups that do not exist, they are created automatically along with dataset_name (e.g. when creating the dataset “my_group/my_subgroup/my_dataset”, in addition to “my_dataset” being created as a dataset, “my_group” and “my_subgroup” are also created as groups if they do not already exist). By default, dataset_name does not track attributes stored within it by their creation order. To track the creation order of attributes stored in dataset_name, the keyword ATTRIBUTE ORDER TRACKED must be specified. In case the keyword ATTRIBUTE ORDER INDEXED is specified, attributes stored within dataset_name are also tracked by their creation order and using an index (to speed up retrieval of attribute names). By default, if no storage type (layout) is specified and (1) the dataset is not extendible and (2) no HDF5 predefined filter is used, the dataset will be created with a contiguous storage type (layout). To specify a certain storage type (layout), one of the following keywords may be employed:

• CONTIGUOUS – the data is stored in the HDF5 file in a single contiguous block.
• COMPACT – the data is stored in the object header of the dataset. This storage type (layout) should only be used for data no larger than 65520 bytes (otherwise an error is raised).
• CHUNKED – the data is stored in equal-sized blocks (or chunks) of a predefined size. This storage type (layout) should be used when the dataset is extendible and/or HDF5 predefined filters are specified (otherwise an error is raised).

By default, if no storage allocation is specified, the dataset will have an early, incremental, or late storage allocation depending on whether its storage type (layout) is compact, chunked, or contiguous, respectively. To specify a certain storage allocation, one of the following keywords may be employed:

• EARLY – the space required to store the entire dataset is allocated (i.e. reserved) immediately in the HDF5 file.
• INCREMENTAL – the space required to store the dataset is allocated (i.e. reserved) incrementally in the HDF5 file as needed.
• LATE – the space required to store the entire dataset is allocated (i.e. reserved) in the HDF5 file when data is written into the dataset for the first time.

To create an extendible dataset³⁵, the keyword TO may be employed when specifying the dimensions that are extendible (i.e. that can grow), along with the initial size of each dimension (dataset_dim) and the maximum size (dataset_max_dim) to which it may grow. If a dimension is expected to grow infinitely, the keyword UNLIMITED should be specified. Of note, when a dimension has an initial size of one and is expected to grow infinitely, the keyword TO, together with dataset_dim and dataset_max_dim, may simply be replaced by the keyword UNLIMITED.

In case the keyword ENABLE is specified, one or more HDF5 predefined filters³⁶ may be used to create a (linear) pipeline by additionally specifying one or more of the following keywords:

• NBIT – Compresses the data of an n-bit data type (including arrays and the n-bit fields of compound data types) by packing n‐bit data on output (i.e. stripping off all unused bits) and unpacking on input (i.e. restoring the extra bits required for computation). This filter may only be used for integer and floating-point data types (otherwise an error is raised).
• SCALEOFFSET – Compresses the data by applying a scale and/or offset operation to each element and truncating the result to a minimum number of bits. This filter may only be used for integer and floating-point data types (otherwise an error is raised).
• SHUFFLE – Rearranges the bytes in the chunk by de-interlacing a block of data, which may improve the compression ratio. This filter is usually used in conjunction with the ZLIB filter.
• ZLIB – Compresses the data using the ZLIB library, which is based on the Deflate lossless data compression algorithm.
• FLETCHER32 – Adds a checksum to each chunk to detect data corruption. In case a chunk becomes corrupted, any subsequent attempt to read it will raise an error.

#Parameter(s)

chunk_dim – optional integer that specifies the chunk size of the dimension in question. If chunk_dim is specified, it must be equal to or greater than one (otherwise an error is raised). If not specified and in case the keyword CHUNKED is specified, HDFql will automatically calculate an appropriate value³⁷ and assign it to chunk_dim.

file_name – optional string that specifies the name of the HDF5 file in which the dataset is to be created. If file_name is specified, the file is opened on the fly, the dataset is created within it, and the file is subsequently closed. If not specified, the file currently in use is used instead. As a general rule, in case file_name contains spaces or special characters, it should be enclosed in double quotes (“). Otherwise, if it is not enclosed in double quotes, the dataset is not created, and an error is raised. This rule also applies to any other HDFql operation that works with file names (e.g. RENAME FILE).

dataset_name – mandatory string that specifies the name of the HDF5 dataset to create. As a general rule, in case dataset_name contains spaces or special characters, it should be enclosed in double quotes (“). Otherwise, if it is not enclosed in double quotes, the dataset is not created, and an error is raised. This rule also applies to any other HDFql operation that works with dataset names (e.g. RENAME DATASET).

data_type – optional keyword that specifies the data type of the HDF5 dataset (or of the member that composes it) to create. If not specified, HDFql will automatically infer an appropriate data type for the dataset based on the characteristics (i.e. type and dimensions) of the data to be written into it (if neither direct values nor an input redirecting option is specified, the dataset is not created, and an error is raised).

member_name – mandatory string that specifies the name of the member that composes the HDF5 dataset of data type HDFQL_ENUMERATION or HDFQL_COMPOUND. As a general rule, in case member_name contains spaces or special characters, it should be enclosed in double quotes (“). Otherwise, if it is not enclosed in double quotes, the dataset is not created, and an error is raised. This rule also applies to any other HDFql operation that works with member names (e.g. SHOW MEMBER).

member_value – optional integer that specifies the value to assign to the member that composes the HDF5 dataset of data type HDFQL_ENUMERATION. If not specified, its value defaults to the value assigned to the previous member incremented by one. Of note, the default value assigned to the first member (of the enumeration) is zero (unless explicitly specified).

member_dim – optional integer that specifies the size of the dimension of the member that composes the HDF5 dataset of data type HDFQL_COMPOUND.

dataset_dim – optional integer that specifies the size of the dimension of the HDF5 dataset.

dataset_max_dim – optional integer that specifies the maximum size of the dimension of the HDF5 dataset. To specify an unlimited size, the keyword UNLIMITED should be specified for this purpose. If dataset_max_dim is specified, it must be equal to or greater than dataset_dim and the keyword CHUNKED should be specified (otherwise an error is raised).

compound_size – optional integer that specifies the size (in bytes) of the HDF5 (nested) compound dataset. If not specified, HDFql will automatically calculate the size by either (1) summing the size (in bytes) of all members of the compound if member_offset is not specified or (2) taking the highest sum of the member’s offset with its size (in bytes) if member_offset is specified. Of note, the specification of a size is only available for a dataset of data type HDFQL_COMPOUND (any other data type will raise an error).

member_offset – optional integer that specifies the (memory) member offsets that compose the HDF5 dataset. If specified, the dataset is assumed to store a C padded struct data type (i.e. its members may not be contiguous in memory due to padding between them) and is used as such by HDFql. If not specified, the dataset is assumed to store a C primitive or packed struct data type (i.e. its members are contiguous in memory and have no padding between them) and is used as such by HDFql. Of note, the specification of an offset is only available for a dataset of data type HDFQL_COMPOUND (any other data type will raise an error).

tag_value – optional string that specifies the value of a tag attached to the HDF5 dataset or to its member(s). Of note, the specification of a tag is only available for a dataset or a member of data type HDFQL_OPAQUE (any other data type will raise an error).

fill_value – optional integer, float, or string that specifies the (default) value to write into the HDF5 dataset upon creating it or to return when reading the dataset if no data has ever been written into it. If not specified, the default fill value will be zero or empty, depending on whether the dataset is a number or a string, respectively. In case the keyword FILL UNDEFINED is specified, the value to return will be undefined when reading the dataset if no data has ever been written into it. In case the keyword FILL NEVER is specified, the fill value is never written into the dataset upon creating it, and the value to return will be undefined when reading the dataset if no data has ever been written into it.

attribute_max_compact – optional integer that specifies the maximum number of attributes to store in the compact format. In case the number of attributes (stored in dataset_name) exceeds attribute_max_compact, the storage of attributes switches to the dense format. If not specified, its default value is 8 (defined by the HDF5 library).

attribute_min_dense – optional integer that specifies the minimum number of attributes to store in the dense format. In case the number of attributes (stored in dataset_name) falls below attribute_min_dense, the storage of attributes switches to the compact format. If not specified, its default value is 6 (defined by the HDF5 library).

nbit_precision_value – optional integer that specifies the precision of the N-bit filter. If nbit_precision_value is specified, it must be greater than zero (otherwise an error is raised).

nbit_offset_value – optional integer that specifies the N-bit filter offset. If nbit_offset_value is specified, it must be equal to or greater than zero (otherwise an error is raised).

scaleoffset_value – optional integer that specifies the scale-offset filter offset. The scaleoffset_value must be equal to or greater than zero (otherwise an error is raised). In case the HDF5 dataset is of integer data type, scaleoffset_value specifies the number of bits to retain (of note, if scaleoffset_value is zero, the HDF5 library will automatically calculate the number of bits required for lossless compression). In case the dataset is of floating-point data type, scaleoffset_value specifies the number of digits after the decimal point to retain.

zlib_level – optional integer that specifies the compression level of the ZLIB filter. The zlib_level must be between 0 (no compression) and 9 (best compression) (otherwise an error is raised). If not specified and in case the keyword ZLIB is specified, its default value is 9.

initial_value – optional integer, float, or string to write into the HDF5 dataset to create. In case initial_value is not specified, the element in question will be zeroed or emptied, depending on whether the dataset is a number or a string, respectively.

input_redirecting_option – optional option that specifies the cursor in use, a (text, binary, or Excel) file, or memory (i.e. a user-defined variable) to read data from in order to write it into the HDF5 dataset to create (please refer to the subsection FROM for additional information).

#Return

Nothing

#Example(s)

# create an HDF5 dataset named "my_dataset0" of data type int (the dataset will not be overwritten if it already exists)
CREATE DATASET my_dataset0 AS INT

# create an HDF5 dataset named "my_dataset1" of data type char in a root group named "my_group" (the dataset will not be overwritten if it already exists)
CREATE DATASET /my_group/my_dataset1 AS CHAR

# create two HDF5 datasets named "my_dataset2" and "my_dataset3" of data type short (both datasets will be overwritten if they already exist)
CREATE TRUNCATE DATASET my_dataset2, my_dataset3 AS SMALLINT 

# create an HDF5 dataset named "this is a long dataset name" of data type float (the dataset will not be overwritten if it already exists)
CREATE DATASET "this is a long dataset name" AS FLOAT

# create an HDF5 dataset named "my_dataset4" of data type unsigned long long using the big endian representation
CREATE DATASET my_dataset4 AS BIG ENDIAN UNSIGNED BIGINT

# create an HDF5 dataset named "my_dataset5" of data type int using the little endian representation with an initial value of 80178
CREATE DATASET my_dataset5 AS LITTLE ENDIAN INT VALUES(80178)

# create an HDF5 dataset named "my_dataset6" of data type char using an ASCII representation 
CREATE DATASET my_dataset6 AS ASCII CHAR

# create an HDF5 dataset named "my_dataset7" of data type float of one dimension (size 1024) with a fill value of 85.2
CREATE DATASET my_dataset7 AS FLOAT(1024) FILL(85.2)

# create a compact HDF5 dataset named "my_dataset8" of data type double of three dimensions (size 2x5x10)
CREATE COMPACT DATASET my_dataset8 AS DOUBLE(2, 5, 10)

# create a chunked (size 20x100) HDF5 dataset named "my_dataset9" of data type unsigned char of two dimensions (size 500x1000)
CREATE CHUNKED(20, 100) DATASET my_dataset9 AS UNSIGNED TINYINT(500, 1000)

# create an HDF5 dataset named "my_dataset10" of data type int of two dimensions (size 20x400) using the N-bit data compression filter
CREATE DATASET my_dataset10 AS INT(20, 400) ENABLE NBIT PRECISION 16 OFFSET 4

# create an HDF5 dataset named "my_dataset11" of data type float of one dimension (size 500000) using both the ZLIB data compression and Fletcher32 checksum error detection filters
CREATE DATASET my_dataset11 AS FLOAT(500000) ENABLE ZLIB 5 FLETCHER32

# create an HDF5 dataset named "my_dataset12" of data type variable-length float 
CREATE DATASET my_dataset12 AS VARFLOAT

# create an HDF5 dataset named "my_dataset13" of data type variable-length short of one dimension (size 5) with initial values from a text file named "my_file.txt"
CREATE DATASET my_dataset13 AS VARSMALLINT(5) VALUES FROM FILE my_file.txt

# create an HDF5 dataset named "my_dataset14" of data type variable-length char with an initial value of "Hierarchical Data Format"
CREATE DATASET my_dataset14 AS VARCHAR VALUES("Hierarchical Data Format")

# create an HDF5 dataset named "my_dataset15" of data type opaque 
CREATE DATASET my_dataset15 AS OPAQUE

# create an HDF5 dataset named "my_dataset16" of data type opaque of one dimension (size 6) with initial (ASCII) values of 72, 68, 70, 0, 113, and 108 (i.e. "HDF\0ql")
CREATE DATASET my_dataset16 AS OPAQUE(6) VALUES(72, 68, 70, 0, 113, 108)

# create an HDF5 dataset named "my_dataset17" of data type opaque of two dimensions (size 10x1024) with a tag value "Raw data"
CREATE DATASET my_dataset17 AS OPAQUE(10, 1024) TAG "Raw data"

# create a chunked (size 2) HDF5 dataset named "my_dataset18" of data type float of one dimension (size 5 and extendible up to 10)
CREATE CHUNKED(2) DATASET my_dataset18 AS FLOAT(5 TO 10)

# create a chunked (with an automatically calculated size) HDF5 dataset named "my_dataset19" of data type variable-length int of one dimension (size 1 and extendible to an unlimited size)
CREATE CHUNKED DATASET my_dataset19 AS VARINT(UNLIMITED)

# create a chunked HDF5 dataset named "my_dataset20" (with an automatically calculated chunk size) of data type double of three dimensions (first dimension with size 3 and extendible up to 5; second dimension with size 7; third dimension with size 20 and extendible to an unlimited size)
CREATE CHUNKED DATASET my_dataset20 AS DOUBLE(3 TO 5, 7, 20 TO UNLIMITED)

# create an HDF5 dataset named "my_dataset21" of data type enumeration composed of three members named "Lisbon" (with value 0), "New York" (with value 1), and "Tokyo" (with value 2)
CREATE DATASET my_dataset21 AS ENUMERATION(Lisbon, "New York", Tokyo)

# create an HDF5 dataset named "my_dataset22" of data type enumeration composed of three members named "red" (with value 0), "green" (with value 15), and "blue" (with value 16)
CREATE DATASET my_dataset22 AS ENUMERATION(red, green AS 15, blue)

# create an HDF5 dataset named "my_dataset23" of data type enumeration of one dimension (size 4) composed of two members named "car" (with value 100) and "plane" (with value 200), with a fill value of "plane" (i.e. 200)
CREATE DATASET my_dataset23 AS ENUMERATION(car AS 100, plane AS 200)(4) FILL(plane)

# create an HDF5 dataset named "my_dataset24" of data type compound composed of three members named "person" (of data type variable-length char), "age" (of data type unsigned int), and "weight" (of data type float)
CREATE DATASET my_dataset24 AS COMPOUND(person AS VARCHAR, age AS UNSIGNED INT, weight AS FLOAT)

# create an HDF5 dataset named "my_dataset25" of data type compound composed of four members named "id" (of data type long long), "description" (of data type variable-length char), "position" (of data type compound composed of two members named "x" (of data type short) and "y" (of data type short)), and "temperature" (of data type enumeration composed of three members named "cold" (with value 0), "warm" (with value 1), and "hot" (with value 10))
CREATE DATASET my_dataset25 AS COMPOUND(id AS BIGINT, description AS VARCHAR, position AS COMPOUND(x AS SMALLINT, y AS SMALLINT), temperature AS ENUMERATION(cold, warm, hot AS 10))

# create an HDF5 dataset named "my_dataset26" of data type compound of one dimension (size 5) composed of three members named "state" (of data type enumeration composed of two members named "off" (with value 0) and "on" (with value 1)), "readings" (of data type int of two dimensions (size 3x4)), and "factors" (of data type compound composed of two members named "first" (of data type float) and "second" (of data type double))
CREATE DATASET my_dataset26 AS COMPOUND(state AS ENUMERATION(off, on), readings AS INT(3, 4), factors AS COMPOUND(first AS FLOAT, second AS DOUBLE))(5)

# create an HDF5 dataset named "my_dataset27" of data type compound of one dimension (size 1 and extendible to an unlimited size) composed of two members named "m0" (of data type double) and "m1" (of data type float)
CREATE DATASET my_dataset27 AS COMPOUND(m0 AS DOUBLE, m1 AS FLOAT)(UNLIMITED)

# use (i.e. open) an HDF5 file named "my_file.h5" 
USE FILE my_file.h5

# create an HDF5 dataset named "my_dataset28" of data type double in the HDF5 file currently in use (i.e. file "my_file.h5")
CREATE DATASET my_dataset28 AS DOUBLE

# close HDF5 file currently in use (i.e. file "my_file.h5") 
CLOSE FILE 

# create an HDF5 dataset named "my_dataset29" of data type int in file "my_file.h5" 
CREATE DATASET my_file.h5 my_dataset29 AS INT