parq_tools.parq_concat.concat_parquet_file_with_dataframe

parq_tools.parq_concat.concat_parquet_file_with_dataframe(parquet_path, df, output_path=None, *, index_columns, batch_size=100000, allow_overwrite=False, show_progress=False, **pq_write_kwargs)[source]

Rewrite a Parquet file by appending columns from an in-memory DataFrame.

The source Parquet file is streamed in row batches, aligned against the provided DataFrame using index_columns, and written to a new Parquet file. This avoids materializing the DataFrame to disk first.

Validation is content-based, so Parquet files with non-standard extensions (for example .pbm) are accepted if they are valid Parquet files.

Parameters:

  • parquet_path (Path) – Path to the source Parquet file.

  • df (DataFrame) – In-memory DataFrame containing the extra columns to append.

  • output_path (Optional[Path]) – Destination path for the rewritten Parquet file. If omitted, the source file is rewritten in place.

  • index_columns (List[str]) – Column names used to align rows between the source file and df.

  • batch_size (int) – Number of source rows to process per batch.

  • allow_overwrite (bool) – If True, allow replacing an existing output file, including the source file when rewriting in place.

  • show_progress (bool) – If True, display a progress bar if tqdm is available.

  • **pq_write_kwargs (object) – Extra keyword arguments forwarded to pyarrow.parquet.ParquetWriter.

Raises:

  • FileNotFoundError – If the source file does not exist.

  • ValueError – If the file is not valid Parquet, batch_size is invalid, or key columns are duplicated/misaligned.

  • FileExistsError – If the target exists and overwrite is not allowed.

  • KeyError – If required index columns are missing from either input.

Return type:

None