Skip to content

Metadata

The src/flowerpower_io/metadata.py module collects and manages detailed metadata during I/O operations: schema information, row and column counts, file counts, partition columns, and timestamps. Readers compute it lazily via _get_metadata() and return it when metadata=True; writers return it from write().

Functions

get_serializable_schema

get_serializable_schema(data) -> dict[str, str]

Generates a JSON-serializable schema representation of the input data (Pandas DataFrame, Polars DataFrame, PyArrow Table, and other supported frame types).

  • Returns: dict[str, str] mapping column names to their data types.

get_dataframe_metadata

get_dataframe_metadata(
    df,
    path: str | list[str] | None = None,
    format: str | None = None,
    topic: str | None = None,
    num_files: int | None = None,
    partition_columns: list[str] | None = None,
    fs: AbstractFileSystem | None = None,
    **kwargs,
) -> dict

Extracts comprehensive metadata from a DataFrame or PyArrow Table (or a list of them): schema, row count, column count, file count, and a generation timestamp. This is the function readers and writers use internally.

  • Parameters:
  • df: Input frame(s) — Pandas, Polars (eager or lazy), or PyArrow.
  • path: Source path(s); used to derive num_files via glob when not given explicitly.
  • format: Source format (e.g. "csv", "parquet").
  • num_files: Explicit file count; computed from path/fs or list_files() otherwise.
  • partition_columns: Columns excluded from the schema (dataset partitions).
  • fs: Filesystem used to resolve num_files from path.
  • Returns: dict with path, format, timestamp, schema, num_columns, num_rows, num_files (None-valued keys omitted).

get_duckdb_metadata

get_duckdb_metadata(
    rel: duckdb.DuckDBPyRelation,
    path: str,
    format: str,
    fs: AbstractFileSystem | None = None,
    include_shape: bool = False,
    include_num_files: bool = False,
    partition_columns: list[str] | None = None,
    **kwargs,
) -> dict

Extracts metadata from a DuckDB relation.

  • Parameters:
  • rel: The DuckDB relation.
  • path: Source path of the data.
  • format: Source format.
  • include_shape / include_num_files: Additionally compute row/column counts / file count (requires evaluating the relation / globbing path).
  • Returns: dict containing the relation metadata.

get_pyarrow_dataset_metadata

get_pyarrow_dataset_metadata(
    ds: pds.Dataset,
    path: str,
    format: str,
    **kwargs,
) -> dict

Extracts metadata from a PyArrow Dataset: schema, partition columns, and file count. This is the canonical dataset-level metadata shape cached by BaseDatasetReader._get_metadata().

  • Parameters:
  • ds: The PyArrow Dataset.
  • path: Dataset root path.
  • format: Dataset format (e.g. "parquet", "csv", "json").
  • Returns: dict with path, format, timestamp, schema, partition_columns, num_columns, num_rows (None — counting rows requires a scan), num_files.

get_delta_metadata

get_delta_metadata(
    dtable: DeltaTable,
    path: str,
    **kwargs,
) -> dict

Extracts metadata from a Delta Lake table: schema, version, and table properties.

  • Parameters:
  • dtable: A deltalake.DeltaTable object.
  • path: Path of the Delta table.
  • Returns: dict containing the Delta table metadata.

get_mqtt_metadata

get_mqtt_metadata(
    payload: bytes | dict,
    topic: str | None = None,
    **kwargs,
) -> dict

Extracts metadata from an MQTT message payload (JSON bytes or an already-parsed dict).

  • Parameters:
  • payload: Raw JSON bytes or a parsed message dict.
  • topic: The MQTT topic the payload was received on.
  • Returns: dict with topic, format ("mqtt"), timestamp, and schema information.