Last year, we were asked to work with the Dial Before You Dig (South Australia/Northern Territory) association to come up with a vision for the future of underground asset damage prevention.
In this post I focus on some of the opportunities made available through taking a geospatial approach to asset protection and the technical challenges we addressed through the pilot. Moving to a solution that builds on the concepts of spatial data discovery and sharing whilst maintaining the integrity of an enquiry-first service will have relevance for anyone looking to expose your spatial data to a wider audience.
Dial Before You Dig
Note: The views and opinions expressed in this article are those of the authors and do not necessarily reflect those of DBYD SA/NT or AADBYDS.
If you’re not aware of Dial Before You Dig (DBYD), it’s a not-for-profit organisation for assisting in preventing damage and disruption to Australia’s vast infrastructure networks including essential services such as electricity, gas, water, and telecommunications. A recent study commissioned by the Association of Australian Dial Before You Dig Services (AADBYDS) estimated that there are 740,000 kilometres of underground assets with a value of more than $340B nationwide.
Stuart Burdack (CEO, AADBYDS) describes it most clearly when he says that DBYD exists to achieve “Zero harm, zero damage” in relation to underground assets.
Today, Dial Before You Dig provides the 1100 referral serviceallowing anyone who’s going to be involved in digging up a road or even their own garden to lodge an enquiry once and have it forwarded to each of the underground asset owners who have assets in the vicinity of the dig. In return the asset owners will send you maps with the indicative location of any infrastructure near where you intend to excavate.
Similar referral services are used in many countries including services such as 811 (USA) or Click Before You Dig (Canada). Elsewhere in the world, such as in the UK and Singapore, referral services are more fragmented requiring the excavator to research who the possible asset owners are in an area to request plans.
Today’s Referral Service
Dial Before You Dig in Australia is very successful in terms of the number of asset owners that are registered with the service and the very high levels of awareness among the community of the need to perform a DBYD before doing any excavations.
Having said that there are a number of obstacles and challenges presented by the technical one call solution for those making enquiries and for the asset owners responsible for responding with their plans.
When we mapped out the user journey for the enquirer the most striking challenge is the resulting influx of emails from submitting a request. Keeping track of all of the email responses returned by each asset owner (sometimes up to 16 or more asset owners for a single enquiry!) and ensuring you have all the information you need can be a complicated process. We found a number of examples where the excavator had not considered every asset owner’s information before digging.
The enquirer can also struggle to read the asset plans and correctly orientate themselves spatially alongside the maps they are looking at. We found one example where the contractor was looking at maps that were in fact for the opposite carriageway from where they were working. This comprehension challenge is exacerbated by each asset owner returning their infrastructure plans in different formats with different maps and symbology.
For the asset owner, joining DBYD creates a requirement to put in place a method for replying to the referrals generated for each dig enquiry in their region. the costs of generating and returning these maps can be far larger than the fees to be a member of the DBYD service itself. With double digit growth in referral volumes each year, this growth flows straight through to the asset owner in terms of the demand for producing ever more plans.
The current service struggles to innovate due to a decentralised model of referral that puts the requirement to deliver improved technical solutions on all of the members individually rather than through the Dial Before You Dig organisations. A closed application architecture actively prevents third-parties from innovating on top of the service.
For the pilot program we worked with DBYD SA/NT to look at how innovation could improve this technical solution and deliver a Next Generation service.
Next Generation Pilot
For us, Next Generation DBYD was about moving from the current enquiry-forwarding referral model to a complete enquiry-response solution that acts as a broker for supporting the discovery and sharing of underground asset data in a secure and consistent manner.
The concept of lodging an enquiry remains important to effective protection of underground infrastructure. The asset owners want to know who’s planning on digging in proximity to their pipes, cables and fibre so that they can ensure effective communication of safe work methods. However, this doesn’t mean that the responsibility of DBYD should end with capturing and forwarding on the enquiries to the relevant utilities.
The principles that underpinned the Next Gen Pilot were:
- Deliver world’s best practice for asset and life protection;
- Facilitate future expansion in the range of services offered by DBYD;
- Ensure the asset owner remains in control of its data and representation.
In addition to these principles it was a requirement of the solution that it would reduce the impost placed on asset owners through a shared centralised response automation system and that the enquirer’s experience must be improved to strengthen understanding of the risks and safety practices when performing works.
The solution itself was developed to be a platform allowing for multiple methods for participating in the service (upload data, connect to web services, integrate with custom APIs or even sketching asset data on a web map for the smallest members). An open API allows for new services to be developed on top of the platform by third-parties whilst ensuring members remain in complete control of their data.
For the rest of this post I look not so much at the Next Gen DBYD solution but rather at some of the interesting challenges we came across and the approach we took to solving them. I’m certain there will be more than one way to approach each challenge and look forward to hearing any ideas you may have in the comments section on this post.
Ownership of Spatial Data and Representation
Changing from a referral service to an instant-response solution involves consideration of who is responsible for ownership of the spatial data and its representation. Is it DBYD as the service provider? SmarterWX as the solution developer? Or the asset owner themselves?
A key tenet of our solution was to ensure the asset owner is responsible for maintaining their own data and the representation of their data. To achieve this each asset owner is “sandboxed” to isolate their data, symbology and output preferences. The data does not get transformed into a common data model and maps are not generated using a solution-defined representation. The asset owner chooses how their data is stored and how it is presented to the enquirer.
Alternative solutions using a “big bucket” approach that forces all asset owners’ data into a common data model with a common representation are not feasible for such a diverse set of utilities, local governments, petroleum companies and other underground asset owners. Any approach to transforming the asset data can lead to an increased transfer of risk from the owner of the data to the service provider – something we aimed to avoid.
Many organisations are already exposing their asset data through geospatial web services either internally for their own use or with the public. For these organisations we looked at how they could take advantage of their existing services within the solution from each of these options.
The pilot demonstrated three different approaches to a federated services model.
- Spatial data services – the asset owner can register their OGC WFS, WMS or ArcGIS server services with the solution and for each enquiry we fetch the necessary data or images. The DBYD service uses this to generate the necessary response documentation centrally and shares it with the enquirer.
- Map production services – the asset owner registers a print service with the solution. For each enquiry we pass the request to the registered print service on the asset owner’s server and a PDF map is returned before being stored centrally and shared with the enquirer through the DBYD web portal.
- Custom APIs – a custom HTTP request is defined that will be called for every enquiry. The asset owner has complete control of the production of the necessary responses which are returned to the enquirer through the Next Gen UI.
Each of these has pros and cons with a key difference being the amount of processing load that is being handled by the Next Gen DBYD solution versus the asset owners’ own systems. To reduce the impost on the asset owners through growing demand you can take advantage of the centralised processing service.
Later in this post I discuss how we apply a security layer between the enquirer and the asset owners’ data to maintain the integrity of the enquiry first approach to sharing.
The users of Dial Before You Dig are varied and each needs to be considered when building a solution. Some users will be making dozens of enquiries a day with their needs being for a low-friction quick-to-use application. Other users may only use Dial Before You Dig a handful of times a year and will not have the same level of familiarity with the system.
As with any web application the best way to encourage adoption of your solution is to provide the best user experience. We ran a number of workshops with a variety of user types to understand how to deliver a fast, intuitive and frictionless interface for enquiry lodgement.
A key outcome was that users struggle with sketching a dig site on a map. As someone who’s been in the GIS industry for approaching a decade it’s easy to forget how the wider community can struggle to use the web maps we are working with every day.
First up, don’t make the user do all the hard work to zoom, pan and search the map for their dig sites. The solution to that was to start by clearly offering a text-location search with fuzzy logic that caters for misspellings, incorrect suburb names and so on. Also, it was seen that a user will typically be working in roughly the same area from job-to-job, so it’s also useful to open the map to the last location the user worked at.
These minor adjustments, when used together, provide a much-improved experience for users.
Data quality is an important consideration when it comes to sharing plans. Quality can mean positional accuracy, completeness of data, classification correctness or how up-to-date the data is.
Australian Standard AS-5488 provides a classification system for subsurface utility information, rating the quality levels from A to D based on a combination of how the information was generated, the available attributes and positional accuracy of the data. Level A requires a horizontal and vertical accuracy of less than 5cm. The data being served through DBYD in Australia is level D as it is based on existing records rather than any sort of local site survey.
The quality of each asset owner’s data is going to be different. And within one asset owner’s data the quality will be variable, especially since some of the assets were buried many decades ago.
The challenge for a solution like this is how do you represent data that is known to lack positional accuracy on a map. Some of the things to watch out for: –
- Showing multiple asset classes on a map may lead the viewer to assume relative positioning. In the diagram below, once you’ve found the water pipe do you assume the electricity cables are a couple of metres to the east?
- Multiple layers of information can hide each other when viewed on a single map. A line representing a low voltage cable might obscure another line representing a high voltage cable running under the same lane.
- Not all infrastructure being captured. For example, a lot of plans miss the final service connection to each house ending in the street. In the diagram the gas main is shown but connections to individual houses are not.
Our approach to this was to (a) limit the number of asset classes shown in any one view and (b) ensuring the asset owner is in complete control of the representation of their data. The asset owner decides on the layer ordering, symbology and so on regardless of whether we are providing a digital or printed map.
Whilst it’s tempting to show all asset owners’ data together on one map this can make it more difficult for the viewer to comprehend the different information and encourages the misperception relating to relative positions.
I believe that the future state for underground asset identification should be the augmented reality viewer allowing the excavator to stare through their iPad to see what lies beneath. Today the available data doesn’t allow it but the Next Generation DBYD pilot we built supports this model ensuring the solution is future proofed ready for improved data.
Map Production at Scale
Dial Before You Dig Australia handled over 1.9 million enquiries last year resulting in over 10 million referrals. With a single instant-response service running across the working day that could lead to three thousand PDF map documents an hour being generated. And that’s before you consider the 10% per year growth in DBYD enquiries!
Using ArcGIS Pro for map production, the solution takes advantage of the horizontal-scaling capabilities of Amazon Web Services to spin up additional instances to meet any increase in demand. As a powerful multi-threaded application each instance of ArcGIS Pro easily takes up the load of multiple map production requests in parallel.
Since the requests for DBYD are not uniform throughout the day this architecture allows us to ramp up to meet the demand at the busiest time of day (9am to 12pm) and then wind back down during the quieter times (9pm to 6am).
Exposing Spatial Data Dig by Dig
Allowing users to integrate DBYD into their own GIS workflows was a key requirement for our replacement solution. Many organisations are using digital mapping to share their work instructions and corporate asset information to their field workers and DBYD information should be available as a part of that.
An important consideration here is that the asset owner doesn’t want to just make a geospatial feature layer of its network available for enquirers to view as this removes the tracking information they have about who’s digging where and when. Asset data should only be made available for the area of a proposed dig site; not for the full extent.
To achieve this our solution included a proxy interface in front of the feature layer which required a unique enquiry token to be passed along with each request. This token is used to crop any feature queries to a maximum size of the original enquiry’s extent. This approach ensures that an excavator wishing to view the data must first record their intent to dig and will only be shown data once for that dig area.
Keeping an Audit Trail of Map Views
When you’re sending PDFs of maps to people it’s easy to keep a repository of the sent emails as an archive of the information that has been provided. In the case of DBYD this can form an important audit trail in the case of any legal dispute arising from damage caused by an asset strike.
When asset data is provided through a digital map we have to take a different approach to this audit trail.
Firstly, for any enquiry we need to know what data was shown for that particular time. An enquiry must be linked to both a version of the data and a version of the representation (symbology) in use at the time of that enquiry. This allows us to step back in time and view the map exactly as it was seen by the enquirer.
And, secondly, we keep an audit trail of the specific portions of the map that a user looked at when viewing the asset information. Each time the user zoomed in or panned around the application recorded the extent and scale to allow us to record how a user interacted with the map. This allows, if necessary, the application to answer questions such as, “Did the user zoom in far enough to see all the detail?” or “Did they move around to see assets across the dig site?”
(As well as recording how the maps embedded inside of the POC solution are viewed, we also record queries against the GIS feature services in the case where users were viewing data on their own web map.)
Open Platform to Drive Innovation
I am a huge believer in the view that an open-platform will drive innovation as third-parties will come up with ideas you would have never dreamt of. The platform model drives the Apple App Store where millions of developers are working on apps that could never have been developed by Apple alone.
When the US Weather Service opened up their data for third-parties to develop on top of it created a five-billion-dollar industry in the use of meteorological data for a huge range of applications.
The existing DBYD solution is a closed-system and prohibits this style of innovation. For the next generation pilot, we built it to be an open-platform allowing the same underground asset and enquiry-response data to be extended by third-parties. This platform is designed to be a data broker enabling the discovery of asset owners and the sharing of their data.
As the data improves I can see the machinery manufacturers including real-time integration in to the cab alerting the operator to nearby hazards or actively preventing the back-hoe from breaking earth when there’s a gas pipeline underneath.
Permitting solutions can be integrated into the enquiry-response process so that the lodgement of your dig response kicks off the permitting workflow with your preferred solution. Each part of the solution included web-hooks to allow for this style of open integration.
Three sixty degree data sharing will see locators sharing the verified asset locations back to the utilities to assist in improving positional accuracy.
As a by-product of the Next Generation DBYD pilot, SmarterWX has taken parts of the solution we developed and turned them into stand-alone products to support the existing members and enquirers of the existing 1100.com.au one-call service.
Extending our existing SmarterWX range of software-as-a-service solutions, SmarterWX Automate and SmarterWX Locate are both available as part of an early-access program with general availability planned for later this year.
SmarterWX Automate is your automated response service for asset owners needing to handle the enquiry referrals generated by 1100.com.au. A fully cloud-based service, Automate generates the asset plans and related documentation and returns them to the enquirer. Built to integrate with your existing GIS investments, Automate will synchronise to your existing asset data removing the need for you to move data around.
For anyone who is making lots of DBYD enquiries, SmarterWX Locate takes away the pain of tracking the huge volume of emails that result. Locate collates the responses as they are returned into a user-friendly web interface with just a single email notification to let you know when all of the responses are ready. This is a great productivity tool for individuals and for sharing across your team.
Note: The views and opinions expressed in this article are those of the authors and do not necessarily reflect those of DBYD SA/NT or AADBYDS.