parq_blockmodel.io.ingest_writer.IngestWriter

parq_blockmodel.io.ingest_writer.IngestWriter#

class parq_blockmodel.io.ingest_writer.IngestWriter(input_path, output_path, geometry, schema=None, engine_initializer=None, compression='fast')[source]#

Stateful writer for streaming Parquet data into canonical .pbm files.

This class manages the ingestion of data from a source Parquet file into a canonical .pbm (Parquet block model) file with automatic block_id derivation, spatial column coercion, and optional schema validation and df-eval operations.

The writer handles: - Streaming batches from source Parquet files - Deriving block_id from positional columns (block_id, world_id, ijk, or xyz) - Validating consistency between block_id and xyz if both present - Ensuring all spatial columns are present in output - Applying schema validation and df-eval operations if schema provided - Atomic file writing via parq_tools

input_path#

Path to the source Parquet file.

Type:

Path

output_path#

Path to the output .pbm file.

Type:

Path

geometry#

Geometry of the block model (defines block size, shape, and orientation).

Type:

RegularGeometry

schema#

Optional pandera schema for validation and df-eval operations.

Type:

Optional[DataFrameSchema]

engine_initializer#

Optional callable to configure the df-eval Engine. Signature: Callable[[Engine], Engine].

Type:

Optional[Callable]

__init__(input_path, output_path, geometry, schema=None, engine_initializer=None, compression='fast')[source]#

Initialize the IngestWriter with input/output paths and geometry.

Parameters:

  • input_path (str or Path) – Path to the source Parquet file. Use None for DataFrame-based writes.

  • output_path (str or Path) – Path to the output .pbm file.

  • geometry (RegularGeometry) – Geometry of the block model.

  • schema (Optional[DataFrameSchema], default None) – Optional pandera schema for validation and df-eval operations.

  • engine_initializer (Optional[Callable], default None) – Optional callable to configure the df-eval Engine.

Raises:

ValueError – If input_path is provided but does not exist or is not a valid Parquet file.

Methods

__init__(input_path, output_path, geometry)

Initialize the IngestWriter with input/output paths and geometry.

write([columns, chunk_size, tol])

Stream data from source Parquet file into canonical .pbm file.

write_dataframe(dataframe[, columns, tol])

Write a DataFrame directly into canonical .pbm file.
write(columns=None, chunk_size=1000000, tol=1e-06)[source]#

Stream data from source Parquet file into canonical .pbm file.

Orchestrates the streaming write process: 1. Determines source columns and output columns 2. Builds world_id_encoding if needed (using geometry extents) 3. Iterates through batches of the source Parquet file 4. For each batch:

  • Derives block_id from positional columns

  • Validates consistency between block_id and xyz if both present

  • Ensures all spatial columns are present

  • Applies schema validation and df-eval operations if schema provided

  • Writes to output Parquet

Parameters:

  • columns (Optional[list[str]], default None) – Optional subset of columns to persist. If None, all columns in the source file plus any missing special columns are persisted.

  • chunk_size (int, default 1_000_000) – Batch size for reading Parquet data.

  • tol (float, default 1e-6) – Tolerance for xyz to ijk conversion when deriving block_id.

Raises:

ValueError – If source file lacks required positional columns, if block_id values are inconsistent with xyz, if block_id values are outside geometry bounds, if canonical constraint (unique block_ids) is violated, or if requested columns are not present in source file.

Return type:

None

write_dataframe(dataframe, columns=None, tol=1e-06)[source]#

Write a DataFrame directly into canonical .pbm file.

Unlike write() which streams from a Parquet file, this method processes a pandas DataFrame that’s already in memory. The entire DataFrame is processed in one pass (not chunked).

Parameters:

  • dataframe (pd.DataFrame) – Input DataFrame with positional columns (xyz, ijk, block_id, or world_id).

  • columns (Optional[list[str]], default None) – Optional subset of columns to persist. If None, all columns are persisted.

  • tol (float, default 1e-6) – Tolerance for xyz to ijk conversion when deriving block_id.

Raises:

ValueError – If DataFrame lacks required positional columns or other validation fails.

Return type:

None