# 6.4.4 CREATE DATASET

Part of: [6 Language](6-language.md) › [6.4 Data Definition Language (DDL)](6-4-data-definition-language-ddl.md)

### Syntax

**CREATE** [**TRUNCATE**] [**EARLY** | **INCREMENTAL** | **LATE**] [**CONTIGUOUS** | **COMPACT** | {**CHUNKED** [**(**_chunk_dim_ | {[_chunk_dim_] {**,** [_chunk_dim_]}+}**)**]}] **DATASET** [_file_name_] _dataset_name_ [**,** [_file_name_] _dataset_name_]\*\
[**AS** _data_type_ [**(UNLIMITED** | {_dataset_dim_ [**TO** {_dataset_max_dim_ | **UNLIMITED**}]} [**,** **UNLIMITED** | {_dataset_dim_ [**TO** {_dataset_max_dim_ | **UNLIMITED**}]}]\***)**] [**SIZE** _compound_size_]]\
[**TAG** _tag_value_]\
[**FILL** {**(**[_fill_value_] [**,** [_fill_value_]]\***)** | **UNDEFINED** | **NEVER** | **DEFAULT**}]\
[**ATTRIBUTE** [**ORDER** {**TRACKED** | **INDEXED**}] [**STORAGE** **COMPACT** _attribute_max_compact_ **DENSE** _attribute_min_dense_]]\
[**ENABLE** [**NBIT** **PRECISION** _nbit_precision_value_ **OFFSET** _nbit_offset_value_] [**SCALEOFFSET** _scaleoffset_value_] [**SHUFFLE**] [**ZLIB** [_zlib_level_]] [**FLETCHER32**]]\
[**VALUES** {**(**[_initial_value_] [**,** [_initial_value_]]\***)** | _input_redirecting_option_}]\
_data_type_ := [**NATIVE** | **LITTLE** **ENDIAN** | **BIG** **ENDIAN** | **ASCII** | **UTF8**] {**TINYINT** | **UNSIGNED** **TINYINT** | **SMALLINT** | **UNSIGNED** **SMALLINT** | **INT** | **UNSIGNED** **INT** | **BIGINT** | **UNSIGNED** **BIGINT** | **FLOAT** | **DOUBLE** | {**CHAR** [**NULLTERM** | **NULLPAD** | **SPACEPAD**]} | **VARTINYINT** | **UNSIGNED** **VARTINYINT** | **VARSMALLINT** | **UNSIGNED** **VARSMALLINT** | **VARINT** | **UNSIGNED** **VARINT** | **VARBIGINT** | **UNSIGNED** **VARBIGINT** | **VARFLOAT** | **VARDOUBLE** | {**VARCHAR** [**NULLTERM** | **NULLPAD** | **SPACEPAD**]} | **OPAQUE** | _enumeration_ | _compound_}\
_enumeration_ := [**TINYINT** | **SMALLINT** | **INT** | **BIGINT**] **ENUMERATION(**_member_name_ [**AS** _member_value_] [**,** _member_name_ [**AS** _member_value_]]\***)**\
_compound_ := **COMPOUND(**_member_name_ **AS** _data_type_ [**(**_member_dim_ [**,** _member_dim_]\***)**] [**SIZE** _compound_size_] [**OFFSET** _member_offset_] [**TAG** _tag_value_] [**,** _member_name_ **AS** _data_type_ [**(**_member_dim_ [**,** _member_dim_]\***)**] [**SIZE** _compound_size_] [**OFFSET** _member_offset_] [**TAG** _tag_value_]]\***)**

### Description

Create an HDF5 dataset named _dataset_name_. Multiple datasets can be created at once by separating their names with a comma (,). If _dataset_name_ already exists, it is not overwritten, no subsequent datasets are created, and an error is raised. To overwrite an existing dataset, specify the keyword TRUNCATE (all data stored in the dataset will be permanently lost). In case _dataset_name_ includes intermediate groups that do not exist, they are created automatically along with _dataset_name_ (e.g. when creating the dataset “my_group/my_subgroup/my_dataset”, in addition to “my_dataset” being created as a dataset, “my_group” and “my_subgroup” are also created as groups if they do not already exist). By default, _dataset_name_ does not track attributes stored within it by their creation order. To track the creation order of attributes stored in _dataset_name_, the keyword ATTRIBUTE ORDER TRACKED must be specified. In case the keyword ATTRIBUTE ORDER INDEXED is specified, attributes stored within _dataset_name_ are also tracked by their creation order and using an index (to speed up retrieval of attribute names). By default, if no storage type (layout) is specified and (1) the dataset is not extendible and (2) no HDF5 predefined filter is used, the dataset will be created with a contiguous storage type (layout). To specify a certain storage type (layout), one of the following keywords may be employed:

- CONTIGUOUS – the data is stored in the HDF5 file in a single contiguous block.
- COMPACT – the data is stored in the object header of the dataset. This storage type (layout) should only be used for data no larger than 65520 bytes (otherwise an error is raised).
- CHUNKED – the data is stored in equal-sized blocks (or chunks) of a predefined size. This storage type (layout) should be used when the dataset is extendible and/or HDF5 predefined filters are specified (otherwise an error is raised).

By default, if no storage allocation is specified, the dataset will have an early, incremental, or late storage allocation depending on whether its storage type (layout) is compact, chunked, or contiguous, respectively. To specify a certain storage allocation, one of the following keywords may be employed:

- EARLY – the space required to store the entire dataset is allocated (i.e. reserved) immediately in the HDF5 file.
- INCREMENTAL – the space required to store the dataset is allocated (i.e. reserved) incrementally in the HDF5 file as needed.
- LATE – the space required to store the entire dataset is allocated (i.e. reserved) in the HDF5 file when data is written into the dataset for the first time.

To create an extendible dataset[^35], the keyword TO may be employed when specifying the dimensions that are extendible (i.e. that can grow), along with the initial size of each dimension (_dataset_dim_) and the maximum size (_dataset_max_dim_) to which it may grow. If a dimension is expected to grow infinitely, the keyword UNLIMITED should be specified. Of note, when a dimension has an initial size of one and is expected to grow infinitely, the keyword TO, together with _dataset_dim_ and _dataset_max_dim_, may simply be replaced by the keyword UNLIMITED.

In case the keyword ENABLE is specified, one or more HDF5 predefined filters[^36] may be used to create a (linear) pipeline by additionally specifying one or more of the following keywords:

- NBIT – Compresses the data of an n-bit data type (including arrays and the n-bit fields of compound data types) by packing n‐bit data on output (i.e. stripping off all unused bits) and unpacking on input (i.e. restoring the extra bits required for computation). This filter may only be used for integer and floating-point data types (otherwise an error is raised).
- SCALEOFFSET – Compresses the data by applying a scale and/or offset operation to each element and truncating the result to a minimum number of bits. This filter may only be used for integer and floating-point data types (otherwise an error is raised).
- SHUFFLE – Rearranges the bytes in the chunk by de-interlacing a block of data, which may improve the compression ratio. This filter is usually used in conjunction with the ZLIB filter.
- ZLIB – Compresses the data using the ZLIB library, which is based on the Deflate lossless data compression algorithm.
- FLETCHER32 – Adds a checksum to each chunk to detect data corruption. In case a chunk becomes corrupted, any subsequent attempt to read it will raise an error.

### Parameter(s)

_chunk_dim_ – optional integer that specifies the chunk size of the dimension in question. If _chunk_dim_ is specified, it must be equal to or greater than one (otherwise an error is raised). If not specified and in case the keyword CHUNKED is specified, HDFql will automatically calculate an appropriate value[^37] and assign it to _chunk_dim_.

_file_name_ – optional string that specifies the name of the HDF5 file in which the dataset is to be created. If _file_name_ is specified, the file is opened on the fly, the dataset is created within it, and the file is subsequently closed. If not specified, the file currently in use is used instead. As a general rule, in case _file_name_ contains spaces or special characters, it should be enclosed in double quotes (“). Otherwise, if it is not enclosed in double quotes, the dataset is not created, and an error is raised. This rule also applies to any other HDFql operation that works with file names (e.g. [RENAME FILE](6-4-10-rename-file.md)).

_dataset_name_ – mandatory string that specifies the name of the HDF5 dataset to create. As a general rule, in case _dataset_name_ contains spaces or special characters, it should be enclosed in double quotes (“). Otherwise, if it is not enclosed in double quotes, the dataset is not created, and an error is raised. This rule also applies to any other HDFql operation that works with dataset names (e.g. [RENAME DATASET](6-4-11-rename.md)).

_data_type_ – optional keyword that specifies the data type of the HDF5 dataset (or of the member that composes it) to create. If not specified, HDFql will automatically infer an appropriate data type for the dataset based on the characteristics (i.e. type and dimensions) of the data to be written into it (if neither direct values nor an input redirecting option is specified, the dataset is not created, and an error is raised).

_member_name_ – mandatory string that specifies the name of the member that composes the HDF5 dataset of data type [HDFQL_ENUMERATION](5-1-constants.md#hdfql_enumeration) or [HDFQL_COMPOUND](5-1-constants.md#hdfql_compound). As a general rule, in case _member_name_ contains spaces or special characters, it should be enclosed in double quotes (“). Otherwise, if it is not enclosed in double quotes, the dataset is not created, and an error is raised. This rule also applies to any other HDFql operation that works with member names (e.g. [SHOW MEMBER](6-7-9-show-member.md)).

_member_value_ – optional integer that specifies the value to assign to the member that composes the HDF5 dataset of data type [HDFQL_ENUMERATION](5-1-constants.md#hdfql_enumeration). If not specified, its value defaults to the value assigned to the previous member incremented by one. Of note, the default value assigned to the first member (of the enumeration) is zero (unless explicitly specified).

_member_dim_ – optional integer that specifies the size of the dimension of the member that composes the HDF5 dataset of data type [HDFQL_COMPOUND](5-1-constants.md#hdfql_compound).

_dataset_dim_ – optional integer that specifies the size of the dimension of the HDF5 dataset.

_dataset_max_dim_ – optional integer that specifies the maximum size of the dimension of the HDF5 dataset. To specify an unlimited size, the keyword UNLIMITED should be specified for this purpose. If _dataset_max_dim_ is specified, it must be equal to or greater than _dataset_dim_ and the keyword CHUNKED should be specified (otherwise an error is raised).

_compound_size_ – optional integer that specifies the size (in bytes) of the HDF5 (nested) compound dataset. If not specified, HDFql will automatically calculate the size by either (1) summing the size (in bytes) of all members of the compound if _member_offset_ is not specified or (2) taking the highest sum of the member’s offset with its size (in bytes) if _member_offset_ is specified. Of note, the specification of a size is only available for a dataset of data type [HDFQL_COMPOUND](5-1-constants.md#hdfql_compound) (any other data type will raise an error).

_member_offset_ – optional integer that specifies the (memory) member offsets that compose the HDF5 dataset. If specified, the dataset is assumed to store a C padded struct data type (i.e. its members may not be contiguous in memory due to padding between them) and is used as such by HDFql. If not specified, the dataset is assumed to store a C primitive or packed struct data type (i.e. its members are contiguous in memory and have no padding between them) and is used as such by HDFql. Of note, the specification of an offset is only available for a dataset of data type [HDFQL_COMPOUND](5-1-constants.md#hdfql_compound) (any other data type will raise an error).

_tag_value_ – optional string that specifies the value of a tag attached to the HDF5 dataset or to its member(s). Of note, the specification of a tag is only available for a dataset or a member of data type [HDFQL_OPAQUE](5-1-constants.md#hdfql_opaque) (any other data type will raise an error).

_fill_value_ – optional integer, float, or string that specifies the (default) value to write into the HDF5 dataset upon creating it or to return when reading the dataset if no data has ever been written into it. If not specified, the default fill value will be zero or empty, depending on whether the dataset is a number or a string, respectively. In case the keyword FILL UNDEFINED is specified, the value to return will be undefined when reading the dataset if no data has ever been written into it. In case the keyword FILL NEVER is specified, the fill value is never written into the dataset upon creating it, and the value to return will be undefined when reading the dataset if no data has ever been written into it.

_attribute_max_compact_ – optional integer that specifies the maximum number of attributes to store in the compact format. In case the number of attributes (stored in _dataset_name_) exceeds _attribute_max_compact_, the storage of attributes switches to the dense format. If not specified, its default value is 8 (defined by the HDF5 library).

_attribute_min_dense_ – optional integer that specifies the minimum number of attributes to store in the dense format. In case the number of attributes (stored in _dataset_name_) falls below _attribute_min_dense_, the storage of attributes switches to the compact format. If not specified, its default value is 6 (defined by the HDF5 library).

_nbit_precision_value_ – optional integer that specifies the precision of the N-bit filter. If _nbit_precision_value_ is specified, it must be greater than zero (otherwise an error is raised).

_nbit_offset_value_ – optional integer that specifies the N-bit filter offset. If _nbit_offset_value_ is specified, it must be equal to or greater than zero (otherwise an error is raised).

_scaleoffset_value_ – optional integer that specifies the scale-offset filter offset. The _scaleoffset_value_ must be equal to or greater than zero (otherwise an error is raised). In case the HDF5 dataset is of integer data type, _scaleoffset_value_ specifies the number of bits to retain (of note, if _scaleoffset_value_ is zero, the HDF5 library will automatically calculate the number of bits required for lossless compression). In case the dataset is of floating-point data type, _scaleoffset_value_ specifies the number of digits after the decimal point to retain.

_zlib_level_ – optional integer that specifies the compression level of the ZLIB filter. The _zlib_level_ must be between 0 (no compression) and 9 (best compression) (otherwise an error is raised). If not specified and in case the keyword ZLIB is specified, its default value is 9.

_initial_value_ – optional integer, float, or string to write into the HDF5 dataset to create. In case _initial_value_ is not specified, the element in question will be zeroed or emptied, depending on whether the dataset is a number or a string, respectively.

_input_redirecting_option_ – optional option that specifies the cursor in use, a (text, binary, or Excel) file, or memory (i.e. a user-defined variable) to read data from in order to write it into the HDF5 dataset to create (please refer to the subsection [FROM](6-3-1-from.md) for additional information).

### Return

Nothing

### Example(s)

```hdfql
# create an HDF5 dataset named "my_dataset0" of data type int (the dataset will not be overwritten if it already exists)
CREATE DATASET my_dataset0 AS INT

# create an HDF5 dataset named "my_dataset1" of data type char in a root group named "my_group" (the dataset will not be overwritten if it already exists)
CREATE DATASET /my_group/my_dataset1 AS CHAR

# create two HDF5 datasets named "my_dataset2" and "my_dataset3" of data type short (both datasets will be overwritten if they already exist)
CREATE TRUNCATE DATASET my_dataset2, my_dataset3 AS SMALLINT 

# create an HDF5 dataset named "this is a long dataset name" of data type float (the dataset will not be overwritten if it already exists)
CREATE DATASET "this is a long dataset name" AS FLOAT
```

```hdfql
# create an HDF5 dataset named "my_dataset4" of data type unsigned long long using the big endian representation
CREATE DATASET my_dataset4 AS BIG ENDIAN UNSIGNED BIGINT

# create an HDF5 dataset named "my_dataset5" of data type int using the little endian representation with an initial value of 80178
CREATE DATASET my_dataset5 AS LITTLE ENDIAN INT VALUES(80178)

# create an HDF5 dataset named "my_dataset6" of data type char using an ASCII representation 
CREATE DATASET my_dataset6 AS ASCII CHAR
```

```hdfql
# create an HDF5 dataset named "my_dataset7" of data type float of one dimension (size 1024) with a fill value of 85.2
CREATE DATASET my_dataset7 AS FLOAT(1024) FILL(85.2)

# create a compact HDF5 dataset named "my_dataset8" of data type double of three dimensions (size 2x5x10)
CREATE COMPACT DATASET my_dataset8 AS DOUBLE(2, 5, 10)

# create a chunked (size 20x100) HDF5 dataset named "my_dataset9" of data type unsigned char of two dimensions (size 500x1000)
CREATE CHUNKED(20, 100) DATASET my_dataset9 AS UNSIGNED TINYINT(500, 1000)
```

```hdfql
# create an HDF5 dataset named "my_dataset10" of data type int of two dimensions (size 20x400) using the N-bit data compression filter
CREATE DATASET my_dataset10 AS INT(20, 400) ENABLE NBIT PRECISION 16 OFFSET 4

# create an HDF5 dataset named "my_dataset11" of data type float of one dimension (size 500000) using both the ZLIB data compression and Fletcher32 checksum error detection filters
CREATE DATASET my_dataset11 AS FLOAT(500000) ENABLE ZLIB 5 FLETCHER32

# create an HDF5 dataset named "my_dataset12" of data type variable-length float 
CREATE DATASET my_dataset12 AS VARFLOAT

# create an HDF5 dataset named "my_dataset13" of data type variable-length short of one dimension (size 5) with initial values from a text file named "my_file.txt"
CREATE DATASET my_dataset13 AS VARSMALLINT(5) VALUES FROM FILE my_file.txt

# create an HDF5 dataset named "my_dataset14" of data type variable-length char with an initial value of "Hierarchical Data Format"
CREATE DATASET my_dataset14 AS VARCHAR VALUES("Hierarchical Data Format")
```

```hdfql
# create an HDF5 dataset named "my_dataset15" of data type opaque 
CREATE DATASET my_dataset15 AS OPAQUE

# create an HDF5 dataset named "my_dataset16" of data type opaque of one dimension (size 6) with initial (ASCII) values of 72, 68, 70, 0, 113, and 108 (i.e. "HDF\0ql")
CREATE DATASET my_dataset16 AS OPAQUE(6) VALUES(72, 68, 70, 0, 113, 108)

# create an HDF5 dataset named "my_dataset17" of data type opaque of two dimensions (size 10x1024) with a tag value "Raw data"
CREATE DATASET my_dataset17 AS OPAQUE(10, 1024) TAG "Raw data"
```

```hdfql
# create a chunked (size 2) HDF5 dataset named "my_dataset18" of data type float of one dimension (size 5 and extendible up to 10)
CREATE CHUNKED(2) DATASET my_dataset18 AS FLOAT(5 TO 10)

# create a chunked (with an automatically calculated size) HDF5 dataset named "my_dataset19" of data type variable-length int of one dimension (size 1 and extendible to an unlimited size)
CREATE CHUNKED DATASET my_dataset19 AS VARINT(UNLIMITED)

# create a chunked HDF5 dataset named "my_dataset20" (with an automatically calculated chunk size) of data type double of three dimensions (first dimension with size 3 and extendible up to 5; second dimension with size 7; third dimension with size 20 and extendible to an unlimited size)
CREATE CHUNKED DATASET my_dataset20 AS DOUBLE(3 TO 5, 7, 20 TO UNLIMITED)
```

```hdfql
# create an HDF5 dataset named "my_dataset21" of data type enumeration composed of three members named "Lisbon" (with value 0), "New York" (with value 1), and "Tokyo" (with value 2)
CREATE DATASET my_dataset21 AS ENUMERATION(Lisbon, "New York", Tokyo)

# create an HDF5 dataset named "my_dataset22" of data type enumeration composed of three members named "red" (with value 0), "green" (with value 15), and "blue" (with value 16)
CREATE DATASET my_dataset22 AS ENUMERATION(red, green AS 15, blue)

# create an HDF5 dataset named "my_dataset23" of data type enumeration of one dimension (size 4) composed of two members named "car" (with value 100) and "plane" (with value 200), with a fill value of "plane" (i.e. 200)
CREATE DATASET my_dataset23 AS ENUMERATION(car AS 100, plane AS 200)(4) FILL(plane)

# create an HDF5 dataset named "my_dataset24" of data type compound composed of three members named "person" (of data type variable-length char), "age" (of data type unsigned int), and "weight" (of data type float)
CREATE DATASET my_dataset24 AS COMPOUND(person AS VARCHAR, age AS UNSIGNED INT, weight AS FLOAT)

# create an HDF5 dataset named "my_dataset25" of data type compound composed of four members named "id" (of data type long long), "description" (of data type variable-length char), "position" (of data type compound composed of two members named "x" (of data type short) and "y" (of data type short)), and "temperature" (of data type enumeration composed of three members named "cold" (with value 0), "warm" (with value 1), and "hot" (with value 10))
CREATE DATASET my_dataset25 AS COMPOUND(id AS BIGINT, description AS VARCHAR, position AS COMPOUND(x AS SMALLINT, y AS SMALLINT), temperature AS ENUMERATION(cold, warm, hot AS 10))

# create an HDF5 dataset named "my_dataset26" of data type compound of one dimension (size 5) composed of three members named "state" (of data type enumeration composed of two members named "off" (with value 0) and "on" (with value 1)), "readings" (of data type int of two dimensions (size 3x4)), and "factors" (of data type compound composed of two members named "first" (of data type float) and "second" (of data type double))
CREATE DATASET my_dataset26 AS COMPOUND(state AS ENUMERATION(off, on), readings AS INT(3, 4), factors AS COMPOUND(first AS FLOAT, second AS DOUBLE))(5)

# create an HDF5 dataset named "my_dataset27" of data type compound of one dimension (size 1 and extendible to an unlimited size) composed of two members named "m0" (of data type double) and "m1" (of data type float)
CREATE DATASET my_dataset27 AS COMPOUND(m0 AS DOUBLE, m1 AS FLOAT)(UNLIMITED)
```

```hdfql
# use (i.e. open) an HDF5 file named "my_file.h5" 
USE FILE my_file.h5

# create an HDF5 dataset named "my_dataset28" of data type double in the HDF5 file currently in use (i.e. file "my_file.h5")
CREATE DATASET my_dataset28 AS DOUBLE

# close HDF5 file currently in use (i.e. file "my_file.h5") 
CLOSE FILE 

# create an HDF5 dataset named "my_dataset29" of data type int in file "my_file.h5" 
CREATE DATASET my_file.h5 my_dataset29 AS INT
```

[^35]: An extendible HDF5 dataset is one whose dimensions can grow. These dimensions start with an initial size and may be increased later. To create an extendible dataset, the storage type (layout) of the dataset must be chunked (otherwise an error is raised). In case the storage type (layout) is not specified, HDFql will automatically set it to chunked and calculate an appropriate chunk size.
[^36]: To be able to use HDF5 predefined filters, the storage type (layout) of the HDF5 dataset must be chunked (otherwise an error is raised). In case the storage type (layout) is not specified, HDFql will automatically set it to chunked and calculate an appropriate chunk size.
[^37]: The calculated value may not be optimal, as it is based on a best-guess approach with the aim of alleviating the programmer from specifying it. In case performance is critical, the chunk size of the dimension in question should be explicitly specified, considering how the data (stored in the HDF5 dataset) is accessed, as it greatly influences performance (HDFql does not have enough information to determine the data access pattern).