The Advanced Data Import - Importing a CSV file article demonstrated the power and flexibility of PIPE-FLO® Professional's new Data Import feature by taking advantage of Python™ scripting to generate or retrieve data. One flaw in the presented approach is the difficulty of locating and correcting bad data in the .csv file being read. If it is run with invalid data types or empty fields it would abort the script and return an error for the line of code where the exception was encountered. Fortunately, Python™ provides some avenues for us to catch these problems in advance and deal with them in a more expedient manner. Making this extra effort results in a Data Import script that is robust, accurate and time saving across multiple files and users.
The example below reads flow demand data (name, flow, and elevation) from a CSV file, checks the data for validity, and imports the valid items into a PIPE-FLO Professional model. Where exceptions are encountered, they are tallied and reported in the Data Import Output window. The flow demand names correspond to Ultra-pure water system.pipe, which can also be found in My Engineering Files>Projects>Sample Systems. The text below can also be copied into a text editor to create the and CSV_ErrorCheck.py files necessary to run this example in PIPE-FLO.
#The first step is to load some standard Python modules to read the .csv file and perform the file operations:
The file consists of flow demand name, flow rate, and elevation in comma delimited rows. Notice that there are some erroneous or empty values. This block of text can be copied into a text editor and saved as :
The Data Import file CSV_ErrorCheck.py locates UPW_Data.csv and uses a loop to read, test, and set the values in PIPE-FLO. Errors and update counts are printed to the Output window where they occur. The individual pieces of this script will be discussed and the complete version included at the end. This script has been updated for use with PIPE-FLO Professional 15, see the comments in the script on lines 63 & 64 for use with PIPE-FLO 14 and 14.1.
Notice that the script prints the test failures and update/error counts in the Output tab of the Data Import dialog, and the Errors tab is empty. It should look like this:
Invalid flow value type: Device: GS-124-10 , Value: null
Invalid elevation value type: Device: GS-124-13 , Value:
Did not find device GS-124-16
21 items updated.
The real value in this script is not the time saved to input the values from the csv file for a single project, but the flexibility to drop any .csv file in this folder with the appropriate column format into the folder and rerun the script at any time. This could be for future revisions to this model or be applied to any PIPE-FLO model and .csv file pair as long as the flow demand names match. In addition to the time saved, the reduction in manual data transcription leads to better data integrity. Essentially, the time investment to develop a flexible script can provide savings and accuracy on all current and future models. By modifying the variable to data row mapping, as well as the import commands, this script can also be modified to work with any ,pipe file, .csv file and devices.