.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/04_renaming.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_04_renaming.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_04_renaming.py:


Renaming and Metadata
=====================

A simple example to demonstrate how to rename columns in a parquet file.
Additionally, we can update the metadata in the file - in this case we add column descriptions.

.. GENERATED FROM PYTHON SOURCE LINES 10-21

.. code-block:: Python

    import json
    import tempfile

    import pandas as pd
    import pyarrow as pa
    import pyarrow.parquet as pq
    from pathlib import Path

    from parq_tools import rename_and_update_metadata









.. GENERATED FROM PYTHON SOURCE LINES 22-26

Create a Parquet file
---------------------

Create a temporary parquet file for demonstration

.. GENERATED FROM PYTHON SOURCE LINES 27-50

.. code-block:: Python



    def create_parquet_file(file_path: Path):
        # Define the dataset
        data = {
            "x": range(1, 11),  # Index column
            "y": range(11, 21),  # Index column
            "z": range(21, 31),  # Index column
            "a": [f"val{i}" for i in range(1, 11)],  # Supplementary column
            "b": [i * 2 for i in range(1, 11)],  # Supplementary column
            "c": [i % 3 for i in range(1, 11)],  # Supplementary column
        }

        # Create a DataFrame
        df = pa.Table.from_pydict(data)

        # Write the DataFrame to a Parquet file
        pq.write_table(df, file_path)


    parquet_file_path = Path(tempfile.gettempdir()) / "example_data.parquet"
    create_parquet_file(parquet_file_path)








.. GENERATED FROM PYTHON SOURCE LINES 51-52

View the file as a DataFrame

.. GENERATED FROM PYTHON SOURCE LINES 52-55

.. code-block:: Python

    df = pd.read_parquet(parquet_file_path)
    df






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>x</th>
          <th>y</th>
          <th>z</th>
          <th>a</th>
          <th>b</th>
          <th>c</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1</td>
          <td>11</td>
          <td>21</td>
          <td>val1</td>
          <td>2</td>
          <td>1</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2</td>
          <td>12</td>
          <td>22</td>
          <td>val2</td>
          <td>4</td>
          <td>2</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3</td>
          <td>13</td>
          <td>23</td>
          <td>val3</td>
          <td>6</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4</td>
          <td>14</td>
          <td>24</td>
          <td>val4</td>
          <td>8</td>
          <td>1</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5</td>
          <td>15</td>
          <td>25</td>
          <td>val5</td>
          <td>10</td>
          <td>2</td>
        </tr>
        <tr>
          <th>5</th>
          <td>6</td>
          <td>16</td>
          <td>26</td>
          <td>val6</td>
          <td>12</td>
          <td>0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>7</td>
          <td>17</td>
          <td>27</td>
          <td>val7</td>
          <td>14</td>
          <td>1</td>
        </tr>
        <tr>
          <th>7</th>
          <td>8</td>
          <td>18</td>
          <td>28</td>
          <td>val8</td>
          <td>16</td>
          <td>2</td>
        </tr>
        <tr>
          <th>8</th>
          <td>9</td>
          <td>19</td>
          <td>29</td>
          <td>val9</td>
          <td>18</td>
          <td>0</td>
        </tr>
        <tr>
          <th>9</th>
          <td>10</td>
          <td>20</td>
          <td>30</td>
          <td>val10</td>
          <td>20</td>
          <td>1</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 56-61

Rename columns
--------------

We can rename a selection of columns.  Here we assume we don't want to rename the `index` columns.
Assuming we have no knowledge of the column names, we'll read them from the file schema.

.. GENERATED FROM PYTHON SOURCE LINES 61-66

.. code-block:: Python


    index_cols = ["x", "y", "z"]
    col_names = pq.ParquetFile(parquet_file_path).schema.names
    col_names





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    ['x', 'y', 'z', 'a', 'b', 'c']



.. GENERATED FROM PYTHON SOURCE LINES 67-68

Create a mapping and rename the columns

.. GENERATED FROM PYTHON SOURCE LINES 68-73

.. code-block:: Python


    new_col_names: dict[str, str] = {col: f"new_{col}" for col in col_names if col not in index_cols + ['c']}
    output_file_path = parquet_file_path.parent / "renamed_data.parquet"
    rename_and_update_metadata(input_path=parquet_file_path, output_path=output_file_path,
                               rename_map=new_col_names, show_progress=True)




.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    Processing:   0%|          | 0/10 [00:00<?, ?rows/s]
    Processing: 100%|██████████| 10/10 [00:00<00:00, 9358.11rows/s]




.. GENERATED FROM PYTHON SOURCE LINES 74-75

Read the renamed file and display it

.. GENERATED FROM PYTHON SOURCE LINES 75-79

.. code-block:: Python


    df_renamed = pd.read_parquet(output_file_path)
    df_renamed






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>x</th>
          <th>y</th>
          <th>z</th>
          <th>new_a</th>
          <th>new_b</th>
          <th>c</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1</td>
          <td>11</td>
          <td>21</td>
          <td>val1</td>
          <td>2</td>
          <td>1</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2</td>
          <td>12</td>
          <td>22</td>
          <td>val2</td>
          <td>4</td>
          <td>2</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3</td>
          <td>13</td>
          <td>23</td>
          <td>val3</td>
          <td>6</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4</td>
          <td>14</td>
          <td>24</td>
          <td>val4</td>
          <td>8</td>
          <td>1</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5</td>
          <td>15</td>
          <td>25</td>
          <td>val5</td>
          <td>10</td>
          <td>2</td>
        </tr>
        <tr>
          <th>5</th>
          <td>6</td>
          <td>16</td>
          <td>26</td>
          <td>val6</td>
          <td>12</td>
          <td>0</td>
        </tr>
        <tr>
          <th>6</th>
          <td>7</td>
          <td>17</td>
          <td>27</td>
          <td>val7</td>
          <td>14</td>
          <td>1</td>
        </tr>
        <tr>
          <th>7</th>
          <td>8</td>
          <td>18</td>
          <td>28</td>
          <td>val8</td>
          <td>16</td>
          <td>2</td>
        </tr>
        <tr>
          <th>8</th>
          <td>9</td>
          <td>19</td>
          <td>29</td>
          <td>val9</td>
          <td>18</td>
          <td>0</td>
        </tr>
        <tr>
          <th>9</th>
          <td>10</td>
          <td>20</td>
          <td>30</td>
          <td>val10</td>
          <td>20</td>
          <td>1</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 80-83

Update metadata
---------------
We can also update the metadata in the file.  In this case we add descriptions to the renamed columns.

.. GENERATED FROM PYTHON SOURCE LINES 83-94

.. code-block:: Python

    metadata = {
        "description": "This is the description of the dataset",
        "version": "0.1.0",
    }
    column_descriptions = {'new_a': {'description': "This is the a column renamed"},
                           'new_b': {'description': "This is the b column renamed"},
                           'c': {'description': "This is the original c column",
                                 'unit_of_measure': "unitless"}}
    rename_and_update_metadata(input_path=output_file_path, output_path=output_file_path,
                               rename_map=new_col_names, table_metadata=metadata, column_metadata=column_descriptions)








.. GENERATED FROM PYTHON SOURCE LINES 95-96

First the file metadata

.. GENERATED FROM PYTHON SOURCE LINES 96-99

.. code-block:: Python

    d_metadata = pq.read_metadata(output_file_path)
    d_metadata





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <pyarrow._parquet.FileMetaData object at 0x7f8089d6e6b0>
      created_by: parquet-cpp-arrow version 20.0.0
      num_columns: 6
      num_rows: 10
      num_row_groups: 1
      format_version: 2.6
      serialized_size: 1911



.. GENERATED FROM PYTHON SOURCE LINES 100-101

Now the table metadata

.. GENERATED FROM PYTHON SOURCE LINES 101-106

.. code-block:: Python

    pf: pq.ParquetFile = pq.ParquetFile(output_file_path)
    table_metadata = pf.schema.to_arrow_schema().metadata
    decoded = {k.decode(): v.decode() for k, v in table_metadata.items()}
    print(json.dumps(decoded, indent=2))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "description": "This is the description of the dataset",
      "version": "0.1.0"
    }




.. GENERATED FROM PYTHON SOURCE LINES 107-108

Now the column metadata

.. GENERATED FROM PYTHON SOURCE LINES 108-118

.. code-block:: Python

    arrow_schema = pf.schema.to_arrow_schema()
    # Extract column metadata
    column_metadata = {col: arrow_schema.field(col).metadata for col in pf.schema.names}
    # Decode metadata
    column_metadata_decoded = {
        col: {k.decode(): v.decode() for k, v in meta.items()} if meta else {}
        for col, meta in column_metadata.items()
    }
    print(json.dumps(column_metadata_decoded, indent=2))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {
      "x": {},
      "y": {},
      "z": {},
      "new_a": {
        "description": "This is the a column renamed"
      },
      "new_b": {
        "description": "This is the b column renamed"
      },
      "c": {
        "description": "This is the original c column",
        "unit_of_measure": "unitless"
      }
    }




.. GENERATED FROM PYTHON SOURCE LINES 119-121

Persisting only renamed columns
-------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 121-129

.. code-block:: Python


    new_col_names = {"x": "x", "a": "new_a"}
    output_file_path_renamed_only = parquet_file_path.parent / "renamed_data_only.parquet"
    rename_and_update_metadata(input_path=parquet_file_path, output_path=output_file_path_renamed_only,
                               rename_map=new_col_names, return_all_columns=False)

    df_renamed_only = pd.read_parquet(output_file_path_renamed_only)
    df_renamed_only





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>x</th>
          <th>new_a</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>1</td>
          <td>val1</td>
        </tr>
        <tr>
          <th>1</th>
          <td>2</td>
          <td>val2</td>
        </tr>
        <tr>
          <th>2</th>
          <td>3</td>
          <td>val3</td>
        </tr>
        <tr>
          <th>3</th>
          <td>4</td>
          <td>val4</td>
        </tr>
        <tr>
          <th>4</th>
          <td>5</td>
          <td>val5</td>
        </tr>
        <tr>
          <th>5</th>
          <td>6</td>
          <td>val6</td>
        </tr>
        <tr>
          <th>6</th>
          <td>7</td>
          <td>val7</td>
        </tr>
        <tr>
          <th>7</th>
          <td>8</td>
          <td>val8</td>
        </tr>
        <tr>
          <th>8</th>
          <td>9</td>
          <td>val9</td>
        </tr>
        <tr>
          <th>9</th>
          <td>10</td>
          <td>val10</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.018 seconds)


.. _sphx_glr_download_auto_examples_04_renaming.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: 04_renaming.ipynb <04_renaming.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: 04_renaming.py <04_renaming.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_