comdas.codecs.base

Base classes for compression codecs.

Classes

Codec()

Base class for all codecs.

CompressedPayload(original_shape, dtype)

Base container for a codec's compressed payload (representation of an array).
class comdas.codecs.base.CompressedPayload(original_shape, dtype)

Bases: object

Base container for a codec’s compressed payload (representation of an array).

Codecs should subclass this to hold the arrays/parameters the method requires.

Variables:

  • original_shape (tuple[int, ...]) – The shape of the array before compression.

  • dtype (numpy.dtype) – The data type of the array before compression.

property compression_params: dict

The parameters used for compression.

The default implementation returns an empty dict. Codecs with parameters relevant to re-encoding should override this.

Return type:

dict

class comdas.codecs.base.Codec

Bases: ABC

Base class for all codecs.

A codec is responsible for compressing and decompressing arrays. It knows how to turn a dense NumPy array into a CompressedPayload and back, and how to (de)serialize that payload to/from an HDF5 file.

Subclasses are auto-registered by name on class creation. This allows the reader to automatically recognize and use the appropriate codec by name, without having to import every codec.

Most codecs do not support random access writes and algorithms can be inefficient at performing many small writes, so writes to the payload are buffered until max_pending_writes is reached. Once this value is exceeded, the entire array will be decompressed, the writes applied, and the array recompressed. By default, this value is set to 10,000.

If a codec supports random access writes efficiently, it should implement the partial_write() method. If this method is implemented, it will always be immediately used for applying writes, bypassing max_pending_writes entirely.

Variables:

  • name (str) – Unique, human-readable name for the codec.

  • version (str) – Codec format version, bumped if the codec’s payload structure changes in a way that is not backward-compatible.

  • max_pending_writes (int) – Maximum number of pending writes before automatic consolidation. This is only used by codecs that do not implement partial_write().

classmethod get_registered(name)

Look up a codec class by name.

Parameters:

name (str) – The codec’s name attribute.

Returns:

The codec class registered under name.

Return type:

type[Codec]

Raises:

KeyError – If no codec is registered under name.

abstractmethod encode(array, **kwargs)

Compress array into a CompressedPayload.

Parameters:

  • array (ndarray) – The dense array to compress.

  • kwargs – Codec-specific compression parameters.

Returns:

The compressed payload.

Return type:

CompressedPayload

abstractmethod decode(payload)

Fully decompress the payload.

Parameters:

payload (CompressedPayload) – A payload previously produced by encode().

Returns:

The decompressed array.

Return type:

ndarray

decode_partial(payload, key)

Partially decompress the payload, reconstructing only the elements addressed by key.

The default implementation uses a full decode. Codecs that support partial decompression or random access read/write should overwrite this method.

Parameters:
Returns:

The requested subset of the reconstructed array.

Return type:

ndarray

partial_write(payload, key, value)

Write value into the compressed representation at key, without a full decode -> update -> re-encode round trip.

This is an optional method, most codecs do not support random access writes. The default implementation always raises a NotImplementedError, which DuckArray interprets as “this codec or write attempt doesn’t support true partial writes” and falls back to its overlay + default_overlay_flush_threshold strategy instead.

A codec should implement this for the cases it can handle cheaply and raise NotImplementedError for cases it can’t. The fallback applies per call, not just per codec. See supports_partial_write().

Implementations must apply exactly the same assignment semantics as dense_array[key] = value would on the fully decompressed array, including NumPy’s broadcasting rules.

Parameters:

  • payload (CompressedPayload) – The payload before the write.

  • key – A NumPy-style index/slice key.

  • value – The value(s) to write, broadcast against the shape implied by key exactly as plain NumPy assignment would.

Returns:

The payload reflecting the write.

Return type:

CompressedPayload

Raises:

NotImplementedError – Always, unless overridden.

supports_partial_write()

Whether this codec has overridden partial_write().

Returns:

True if partial_write() is overridden.

Return type:

bool

abstractmethod compressed_size_bytes(payload)

Get the total size, in bytes, of everything stored in payload.

Parameters:

payload (CompressedPayload) – A payload previously produced by encode().

Returns:

Total size in bytes of the payload’s stored arrays.

Return type:

int

compression_ratio(payload)

Helper to determine the compression ratio of the payload.

Parameters:

payload (CompressedPayload) – A payload previously produced by encode().

Returns:

Original bytes divided by compressed bytes.

Return type:

float

abstractmethod payload_to_group(payload, group)

Write this codec’s own arrays/params into an open HDF5 group.

Only codec-specific data belongs here. The general COMDAS container already writes original_shape, dtype, codec_name, and codec_version at the container level before calling this. Don’t duplicate those fields.

Parameters:

  • payload (CompressedPayload) – The payload to serialize.

  • group (h5py.Group) – An open, writable h5py.Group dedicated to this one patch.

Return type:

None

abstractmethod payload_from_group(group, *, original_shape, dtype)

Reconstruct this codec’s payload from an HDF5 group.

Parameters:

  • group (h5py.Group) – An open, readable h5py.Group previously written by payload_to_group().

  • original_shape (tuple[int, ...]) – The array shape, as already read from the container’s own (codec-independent) attributes.

  • dtype (dtype) – The array dtype, as already read from the container’s own (codec-independent) attributes.

Returns:

The reconstructed payload.

Return type:

CompressedPayload