Batch Processing in GIS

Going through the same process over and over again can be a big set back. Not only on your time but it could result in some human error and also repetitive strain injury (RSI).

Repetitive strain injury (RSI) is a general term used to describe the pain felt in muscles, nerves and tendons caused by repetitive movement and overuse

NHS

Take for example: You have a set of 50 raster images and you need to:

  • perform a raster projection
  • clip the raster to a certain extend
  • and finally extract cell values from this set of data.

Imagine doing all this one after the other until you finish all 50 images? Basic mathematics tells you that you have to go through the same process 150 times provided you do not make mistakes of course.

Well just like in every problem, there is always a solution for that. I believe when ESRI developed their software, even QGIS too, they actually thought of that and provided tools that would cater for all that and avoid straining the users which cause some huge setbacks.

Allow me to introduce you to Model Builder. In short Model Builder is a visual programming language for building geoprocessing workflows. So instead of you having to click, click and continue clicking for the rest of the day, all you have to do is design a model using ArcGIS software or QGIS. I am not going to mention about other Geographic Information Systems software because I haven’t used them yet.

Image Source: ResearchGate – an example of a geoprocessing workflow built with model builder.

I must say I am a bit jealous too about this model builder tool. For developers like myself, we often leverage people’s inabilities to perform batch tasks by writing code for them and charging a few for that. With Model builder, anyone who understands how a geoprocessing workflow is supposed to look like can basically build their own models and execute their tasks.

Building Our Own Model

Let’s build our own model instead and get to see how it is really done using ArcGIS software. I am a fan of ArcGIS Pro so that’s what I will be using. I am sure if you are still in ArcMap you will be able to follow along too.

First thing to do is adding your data by connecting a folder to your working directory.

We are going to be using Raster images for this guide. You can use your own data too.

In case you need some data you can download some from the CHRS Data Portal.

You can get precipitation data from the 20th century all for free.

Now that we have our data loaded (make sure to add some defining polygon tool for the clip) let us start building the model.

Within your Project in the Catalogue section navigate to the Toolboxes folder (with a briefcase icon). A toolbox with the same name as your project name (if you had named your project in ArcGIS Pro) will appear.

Right-Click on the toolbox (.tbx), click on New and final click on Model.

create a Model

The model builder window should now be displayed with a model named Model.

Let’s drag and drop a process from the system toolbox. We are going to start by projecting our raster images to make sure we are all in the same coordinate system or you might want to change from one system to the other that’s still possible.

To add a process, on the top ribbon, under ModelBuilder, within the Insert group, click on the drop-down arrow beside toolbox. A search box will appear. Type the name of the tool you need to use (in this case Project Raster which converts a raster file from one coordinate system to the other). Drag and drop the tool into the ModelBuilder workspace.

Adding a tool

A visual (white in color) with a process name and a processed output name will appear.

Let’s add some data to make this first initial process complete. Return to you catalogue tab and open the location where your images are stored. Mine were stored in the Mozambique folder as visualized above.

In order to process a batch of raster files, this process needs to be iterative hence in programming we call it a For Loop. We need to loop through each and every single image contained within our Mozambique folder which is why we dragged the folder instead of the image.

To perform iterations, still in the Insert group, click on Iterators. A list of all possible iterative processes will appear. Here we will use the Iterate Raster tool instead. Click on it and it will automatically occupy the Model Builder workspace.

Model iterating functions

Its time for us to connect the dots now. All the three visuals we need to set up our first process are all within our workspace.

Click and drag the folder input that we dragged early on and map it to the Iterate Rasters process. A pop-up will appear and click on Workspace. The iteration process will change color.

ProcessesOrange
InputsBlue
OutputsGreen
The color combination meanings

Now click and drag the output Raster (in green) and map it to the Project Raster process. On the option pop-up click on Input raster.

Notice that the Project tool did not change color this time? This means there are some required parameters for the process that are still missing and these need to stated for the model to be complete.

Hint: As long as the model is still greyish/white, it will not work. 

Double click on the Project Raster process to see what needs to be filled out.

See the blank fields marked with an asterisk (*)? The output coordinate system needs to be filled out in order for the model to be complete. Click the drop down and select the desired coordinate system for the projection process. Once you are done click on OK. The color of the process will change to show that it has been complete.

But there is still one more thing here.

Look at the output of the projection process, it adopted the name of the first raster file which was found in the folder. Would you want every output raster file to have that same name with a number at the end?

Thought so too.

Naming tip

In order to name your files using the same name as they were before any process is performed on them we use the %name% expression. This is called the model variable substitution. We are telling the iterative process to adopt the original name of the file as the output name too.

To implement this, double click on the output file for the process and replace the name with the %name% expression. Make sure you do not delete the whole folder path and always navigate to the intended path you wish to store these. ArcGIS normally saves all processed data into the Default File Geodatabase.

Our small model is ready to execute and project all those 50 raster files to a different coordinate system.

Our Iterative Model

Your model (if you followed these steps) should look like the image above.

Of course this is not all you need to know about creating batch processes. There is still a couple of things we need to do like:

  • properly naming our input and outputs
  • documenting the process
  • saving and naming our tool
  • creating model variables (this allows other people to use the same model but using their own data)
  • sharing models
  • having more than one process to the model.

All this will be featured in the next article where we will be looking at the Good Practices for Model Building in GIS.

I hope you will be able to reduce the workload from now on and implement these smart ways and tools of working in a GIS environment.

About the Author

surveyorjr

Kumbirai is a 4th year Surveying & Geomatics students at the Midlands State University in Zimbabwe. He is also the Founder and Team Lead at African Surveyors Connect. He enjoys spending time writing web-applications and is more of a GIS enthusiast who mainly focuses and grew interests in Health and Land Administrative Applications of Geographic Systems.

Leave a Reply

Your email address will not be published.

You may also like these