Installation

Installation on Linux is recommended, but it is also known to work on Windows and macOS. On macOS the command line interface should be used with bash, not the default zsh, which is not compatible with the syntax.

Requirements:

Python 3, or a Python distribution such as Anaconda

Default Python distribution on Linux

Install the required system packages. On Debian-derived distributions (Ubuntu, Devuan, …):
```
apt install python3-full python3-pip pipx
```
On Fedora:
```
sudo yum install python3 python3-pip pipx
```
Install ds format. If you indend to use the command-line interface, you can install ds format with pipx:
```
pipx install ds-format==4.0.0
ln -s ~/.local/pipx/venvs/ds-format/share/man/man1/ds*.1 ~/.local/share/man/man1/
```
You might have to add $HOME/.local/bin to the PATH environment variable if not present already in order to access the ds command.

If you indend to use the Python interface, you can install in the home directory with pip3:
```
pip3 install ds-format==4.0.0
```
Replace pip3 with pip if pip3 is not available. Add --break-system-packages if your distribution does not allow installing into home directory but you want to anyway.

Alternatively, install into a Python virtual environment with:
```
python3 -m venv venv
. venv/bin/activate
pip3 install ds-format==4.0.0
```
You can then use the ds format Python interface from within the virtual environment. Deactivate the environment with deactivate.

Anaconda on Linux, Windows or macOS

Install Anaconda.
Install ds format in the terminal (Linux and macOS) or Anaconda Prompt (Windows):
```
pip install ds-format==4.0.0
```

Uninstallation

To uninstall if installed with pipx:

pipx uninstall ds-format
rm ~/.local/pipx/venvs/ds-format/share/man/man1/ds*.1

To uninstall if installed with pip3 or pip:

pip3 uninstall ds-format

Replace pip3 with pip if pip3 is not available.

Release notes

4.1.2 (2024-09-10)

Only import IO modules in drivers as needed. This is a workaround for conflicting shared libraries lined by the netCDF4 and h5py packages.

4.1.1 (2024-06-03)

Fixed compatibility with older Python versions.
Fixed JD_CALENDAR constant in the documentation.

4.1.0 (2024-02-16)

ds.var: Fixed handling of missing values.
csv: Missing values are not written as empty strings.
ds cat: Fixed showing scalar vars along empty vars.
Fixed handling of missing values in time variables.
Support for .he5 file extension of HDF5 files.
Improvements in the documentation.

4.0.1 (2023-09-06)

Fix support for Windows and macOS.

4.0.0 (2023-09-06)

The ds command is now installed using setuptools entry_points, which also works on Windows.
ds.var: Normalize variable data when setting the variable.
Imporved documentation.
CSV: Fixes in file reading when some columns are not defined.
ds.split: New function for splitting datasets.
netcdf: Fixed missing JD_UNITS and JD_CALENDAR.
ds cat: New -n option for disabling header.
Support for .tab/.tsv format.

3.7.0 (2023-04-24)

ds rename, ds rename_dim: Fix renaming multiple vars, attrs or dims at a time.
ds.rename_m, ds.rename_attr_m, ds.rename_dim_m: New function.
ds.rename: Also remove variable metadata.
ds.readdir: Fix warnings.
Fixes in the documentation.

3.6.1 (2023-04-21)

ds.readdir: Fixed handling of unreadable files.
ds.attr: Fixed getting attr when var does not exist.
Fixes in the documentation.

3.6.0 (2023-04-01)

Support for parallel processing in readdir.
Fixed merging of time variables by ds merge.
Do not show SIGPIPE error message when sending output to another command.
ds stats: NaNs are now ignored in the calculation of statistics.
ds stats: Output standard deviation and confidence intervals.
Improved documentation.

3.5.2 (2023-03-06)

Fixed an issue with a new version of NumPy.

3.5.1 (2022-11-30)

Fixed installation location for manual pages.

3.5.0 (2022-11-26)

merge: Fixed handling of missing variables.
type: Fixed handling of string variables.
hdf: Fixed handling of type of string variables.
Minor fixes and improvements in documentation.

3.4.0 (2022-11-22)

readdir: Added recursive option.
Improved documentation.

3.3.3 (2022-11-13)

Fixed storing of missing values.

3.3.2 (2022-11-13)

Fixed setting of dataset attributes in the ds storage format.

3.3.1 (2022-11-10)

Fixed package imports in new Python.
Fixed missing required package.
Minor improvements in the documentation.

3.3.0 (2022-10-16)

Basic support for CSV.
Improved documentation.
meta: Argument naming more consistent with the rest of the API.
group_by: Fixed argument checking.
Fixed JSON encoding of NumPy arrays.

3.2.0 (2022-10-09)

Support for HDF5.
Fixes in ds cat JSON output.
Fixes in the NetCDF driver.

3.1.0 (2022-10-05)

Support for JSON output in commands.

3.0.0 (2022-09-30)

New commands and Python API functions.
Version 1.0 of the storage format (incompatible with the previous experimental format).
Support for empty variables.
pst output in commands instead of JSON.
Improved argument checking.
Improved documentation and command help.
Support for variables beginning with a dot (in the ds storage format).
Improved support for missing values.
Glob pattern matching of variable, attribute and dimension names in commands.
Support for error reporting modes.

2.0.1 (2022-09-26)

Fixed missing package in setup.py (Markdown).

2.0.0 (2022-07-31)

Command line documentation and man pages.
Command output in PST format.
ds ls: Added a: attrs option.
New ds file format (experimental).
Documentation website.

1.1.2 (2022-01-01)

Fixed handling of NetCDF time variables.
Fixed merge function definition and implementation.

1.1.1 (2021-12-11)

Dataset validation on write.
Dropped support for Python 2.
merge: new variables option.

1.1.0 (2021-03-31)

Improved reading of NetCDF time variables.
Documented readdir function.

1.0.1 (2020-08-12)

Dependencies installed from PyPI.

1.0.0 (2020-04-28)

Initial release.