Problems to run parallel processing with WNTR / EPANET

[marinhosg]

Dear colleagues,

I'm having some problems when try I to run my WNTR simulations with parallel processing. In sum, I'm trying to run a Water Net (".inp" EPANET file) and compute some parameters from this simulation, as bridge_density, central_point_dominance, todini_index, etc. However, I need to run this simulations several times, considering n hypothesis, and the total of simulations is around 6,000. So, I knew that using parallel processing would be an alternative to speeding up my simulations and save time. But, when I try to use multiprocessing or concurrent.futures tools, the simulations don't run or take more time than the thread process (normal analysis, one by one). Is that a way to solve this problem? I'll attach an example of what I'm working with. Sometimes, EPANET error 200 appears on screen. Thank you.

import wntr
import numpy as np
import pandas as pd
import networkx as nx
import copy
import matplotlib.pyplot as plt
import concurrent.futures
import multiprocessing
import time

[...]
# Import and create the network model from the original INP file
inp_file = "C:/Users/gabri/Desktop/dissertacao/SAO_RAFAEL.inp" #"C:/Users/ianca/Desktop/DADOS DE ENTRADA/REDE_COMPLETA.inp"
original_wn = wntr.network.WaterNetworkModel(inp_file)

[...]

# Obtaining tube roughness
pipe_roughness = original_wn.query_link_attribute('roughness')

# Defining Failure Probability Based on Roughness
failure_probability = pipe_roughness / pipe_roughness.sum()

# Desired pour sizes
tamanho_vazamento =[0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0] 
# DataFrame vazio para armazenar os resultados da simulação
resultados_df = pd.DataFrame()

np.random.seed(67823)  # semente

def simulacao_rede(tamanho_vazamento, failure_probability,i):
    # Create a copy of the original network for this simulation
    wn = copy.deepcopy(original_wn)
    
    # Desired failure percentage
    porcentagem_of_failure = 0.02 #0.015 # 0.001 em 0.001
    
    # Calculating the number of tubes to be damaged
    num_links = len(wn.link_name_list)
    tubos_failure = int(porcentagem_of_failure * num_links)
    
    # Randomly choosing tubes to fail based on probability
    pipes_to_fail =  np.random.choice(failure_probability.index, tubos_failure, replace=False, p=failure_probability.values)
    
    [...]
    
    # Simulation with the modified network
    sim = wntr.sim.EpanetSimulator(wn)
    wn.options.hydraulic.demand_model = 'PDA'
    wn.options.hydraulic.required_pressure = 50
    wn.options.hydraulic.minimum_pressure = 10
    results = sim.run_sim(file_prefix=generate_file_prefix(i))

[...]
# PARALLELIZATION PROCESS HERE:

for aa in tamanho_vazamento:
    if __name__ == '__main__':
        with concurrent.futures.ProcessPoolExecutor() as executor:
            resultss = [executor.submit(simulacao_rede, aa, failure_probability, i) for i in range(num_simulacoes)]
            
            # Collect results
            results_list = []
            for future in concurrent.futures.as_completed(resultss):
                index = resultss.index(future)
                result = future.result()
                resultss_list.append([index, result])
                print(result)  # Print result (optional)
                
            # Append current results to the all_results list with context of b
            all_resultss.extend(((b, index), result) for index, result in results_list)

Environment

• Operating system: Windows
• Python version: 3.9.13
• WNTR version: 1.1
• Running on Spyder (Anaconda)

[ucchejbb]

Have you tried the multiprocessing package?

See the mp_run_epanet and runepanet functions in https://github.com/USEPA/PPMtools/blob/master/PPMtools.py

I have used multiprocessing package to run simulations in parallel in the past. I have not used the concurrent.futures packages in the past.

[kaklise]

See https://usepa.github.io/WNTR/advancedsim.html#multiple-processors for more information on running WNTR with multiple processors. This uses the treading package.

Error 200 indicates that there is an error in the INP file. If you load the INP file that throws the error into the EPANET GUI, you should get additional information on the problem.