.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/06_profiling.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_06_profiling.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_06_profiling.py:


Parquet Profiling
=================

`ydata-profiling` provides a convenient way to profile Parquet files using the `ProfileReport` class.
Their documentation provides options for profiling large datasets.  This example describes an alternative approach.

In cases where a parquet may be very wide, and you want to profile it column by column, you can use the
`ParquetProfileReport` class from the `parq_tools.utils.profile_utils` module.
This allows you to generate a profile report by loading columns in batches, reducing memory consumption.

.. GENERATED FROM PYTHON SOURCE LINES 12-19

.. code-block:: Python

    import tempfile

    import pandas as pd
    from pathlib import Path

    from parq_tools import ParquetProfileReport








.. GENERATED FROM PYTHON SOURCE LINES 20-22

Create a Parquet file for profiling
-----------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 22-35

.. code-block:: Python


    temp_dir = Path(tempfile.gettempdir()) / "profile_parquet_example"
    temp_dir.mkdir(parents=True, exist_ok=True)

    # Create a sample DataFrame and save as Parquet
    df = pd.DataFrame({
        "col1": range(100),
        "col2": ["a"] * 100,
        "col3": [True, False] * 50,
    })
    parquet_path = temp_dir / "example.parquet"
    df.to_parquet(parquet_path)








.. GENERATED FROM PYTHON SOURCE LINES 36-41

Profile by column
-----------------
The `ParquetProfileReport` class allows you to profile a Parquet file by loading columns in batches.

While we are profiling 3 columns, the 4th progress step is used to capture the merging process.

.. GENERATED FROM PYTHON SOURCE LINES 41-52

.. code-block:: Python


    report = ParquetProfileReport(
        parquet_path=parquet_path,
        columns=None,  # None means all columns
        batch_size=1,  # Process 1 column at a time
        show_progress=True,
    )
    report.profile()

    report.show()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Profiling columns:   0%|          | 0/4 [00:00<?, ?it/s]
    Profiling columns:  25%|██▌       | 1/4 [00:00<00:01,  2.32it/s]
    Profiling columns:  50%|█████     | 2/4 [00:00<00:00,  3.37it/s]
    Profiling columns:  75%|███████▌  | 3/4 [00:00<00:00,  3.78it/s]
    Profiling columns: 100%|██████████| 4/4 [00:01<00:00,  3.98it/s]
    Profiling columns: 100%|██████████| 4/4 [00:01<00:00,  3.66it/s]

      0%|          | 0/3 [00:00<?, ?it/s]
    100%|██████████| 3/3 [00:00<00:00, 807.32it/s]




.. GENERATED FROM PYTHON SOURCE LINES 53-56

Run native ydata-profiling
--------------------------
As expected the native report runs faster, and only requires 3 steps.

.. GENERATED FROM PYTHON SOURCE LINES 56-63

.. code-block:: Python


    report = ParquetProfileReport(
        parquet_path=parquet_path,
        batch_size=None,  # None batch size will run standard ydata-profiling ProfileReport
        show_progress=True,
    )
    report.profile().show()




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/3 [00:00<?, ?it/s]
    100%|██████████| 3/3 [00:00<00:00, 100663.30it/s]





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 2.447 seconds)


.. _sphx_glr_download_auto_examples_06_profiling.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: 06_profiling.ipynb <06_profiling.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: 06_profiling.py <06_profiling.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_