Simulating a Network in Parallel

While the ultimate objective is to process multiple fractionated samples together (with sample as a dim), this pattern may be useful in the mean-time. It demonstrates how to process multiple samples in parallel, with a progressbar to provide feedback.

The function my_simulator and the class TqdmParallel is defined in simulating_networks_tools.py, and are imported here to demonstrate.

import pandas as pd
import plotly
from joblib import delayed

from elphick.mass_composition import MassComposition
from elphick.mass_composition.datasets.sample_data import sample_data
from elphick.mass_composition.flowsheet import Flowsheet
from elphick.mass_composition.utils.parallel import TqdmParallel
from examples._simulating_network_functions import my_simulator

Execute multiple simulations

df_data: pd.DataFrame = sample_data()
obj_mc: MassComposition = MassComposition(df_data, name='sample')
d_inputs: dict[int, MassComposition] = {1: obj_mc, 2: obj_mc.add(obj_mc), 3: obj_mc.add(obj_mc).add(obj_mc)}

results: list[tuple[int, Flowsheet]] = TqdmParallel(n_jobs=3, prefer="processes", total=len(d_inputs))(
    delayed(my_simulator)(item) for item in d_inputs.items()
)

d_results = {sid: fs for sid, fs in results}

  0%|          | 0/3 [00:00<?, ?it/s]
 33%|###3      | 1/3 [00:06<00:13,  6.66s/it]
 67%|######6   | 2/3 [00:07<00:03,  3.23s/it]
100%|##########| 3/3 [00:07<00:00,  2.53s/it]

Print the results

print(d_results)

{1: <elphick.mass_composition.flowsheet.Flowsheet object at 0x7f3959bc0460>, 2: <elphick.mass_composition.flowsheet.Flowsheet object at 0x7f3959bc3160>, 3: <elphick.mass_composition.flowsheet.Flowsheet object at 0x7f39580e71f0>}

View the network for a sample

fig = d_results[1].table_plot()
plotly.io.show(fig)