Automatic Validation of Hydstra Time Series Data
ALS and Kisters (commissioned by ALS) have together built a ‘HYADJUST’ AUTO VALIDATION TOOL (HAV) for first pass validation and logic checking of time series data prior to archiving. ALS clients have access to a Custom Pre-processor that displays a first pass visual (manual procedure only) and populates the correct checks according to the site and parameter. A number of failsafe techniques have been adopted to ensure that all checks are effective, from the perspective of both data accuracy and cost.
The HAV acts on processed data in Hydstra work (or archive) files, and applies calibration adjustments to each dataset according to pre-set criteria. Typically the data adjusted is raw telemetered data. Data comments are applied and quality codes automatically assigned according to error ranges and contract specifications.
The HAV is designed to work automatically with the ALS Hydrographics LiveData™ app. Alternatively it can be run manually by users.
Automated Procedure (using ALS LiveData™ app)
- Telemetered data is imported to Hydstra via SVRIMP Task Server Module. Data is continuously appended to site specific Hydstra work files.
- Following a site visit the LiveData™ app brings in field readings in near-real time that show any sensor drift and the magnitude thereof. This field data is then automatically passed to the HAV and relevant checks run on the applicable time series datasets.
- The checks run are specific to the logged parameter types and contract requirements. For example, the expected conductivity range may be unique to a site and a particular client contract may have specific quality coding requirements.
- Apply field check value calibration and assign data quality according to drift magnitude.
- Lift bad quality values from zero to the next good point and alter the quality code accordingly.
- Check for and remove single data spikes (Refer to Figure 2).
- Run a second pass for spikes and report.
- Check for and recode rate of change exceedances as per site specific limits derived from historic data.
- Check for acceptable site and parameter specific maxima and minima. Apply appropriate quality coding outside expected ranges. Checks are based on site specific historic trends for normal dry weather conditions, anything outside of these ranges will be flagged for manual checking.
- Automatically insert standard data comments to the Hydstra file.
- Close any gap/s with quality coding appropriate to gap size.
- Deliver the resultant Hydstra file for review and archiving within the Hydstra Workbench.
Manual Procedure (using ALS Custom Pre-processor)
The manual process for running the HAV is the same as the automated procedure, only differing as follows:
- Field collected data may be processed via HYGENMAN and the HAV process applied (rather than directly to telemetered data).
- The ALS Custom Pre-processor builds TELVIEW preconfigured plots for requested date/times output to screen. This allows the user an easy initial visual overview.
- Site and parameter specific scripts for running the HAV are generated to screen for easy cut/paste. This allows the user to only run selected parameter checks if required (rather than all parameter checks as per the automated procedure).
Outputs from the HAV process:
- An automated data validation report with embedded hyperlinks for each site (Refer to Figure 1).
- A record of each logger value adjustment and the assigned data quality code, as specified within configuration settings (accessible via the hyperlink in Figure 1).
- A record of any data spikes that were removed.
- A record of any additional spikes that were located but not able to be removed.
- Inclusion of standard data comments in the Hydstra.
- A record of any issues still requiring attention.
- If the final HAV process passed, files are created for the Hydstra Workbench. This includes validated data appended to a copy of the site work file for that site, as well as preservation of the raw work file for comparison.
The HAV is designed to check and adjust data within normal operating ranges, which on average applies to an estimated 90% of data requiring validation. Data is adjusted to field calibration readings and quality coded according to calibration discrepancies. Rogue data spikes are automatically removed and appropriate data comments and quality codes inserted. Gaps caused by missing data are filled and appropriately coded (according to the size of gap). The verifying Hydrographer then only has to check what has been done, and if in agreement, archive the data.
A successful data validation run eliminates, on average, 50% of the required manual validation work. ALS estimates that 80% of standard parameter sites in normal operating conditions will pass validation checks without any user intervention. Refer to Figure 3 below for estimated data validation time savings for various site number scenarios.
by Andrew Kaar (Group Data Manager)
Groundwater Sampling Techniques
ALS has extensive hydrographics experience in the sampling and analysis of groundwater, and can specifically tailor a sampling project to meet client requirements.
There are a number of groundwater sampling techniques available, with the two methods used most widely by ALS being Low Flow Micropurge and the relatively new HydraSleeve sampling.
Low Flow Micropurge
Micropurge sampling is one of the most commonly used techniques to sample groundwater and has many distinct advantages. This sampling technique involves a low flow bladder pump being lowered to the depth of the bore screen, taking a representative sample directly from the aquifer, with minimal disturbance to the water column. There is no need for complete bore volume purging using compressors, pumps or bailers. Nor is it necessary to allow the bore to recharge and stabilise to ensure the sample is reflective of the aquifer, as typically required by the purging method. Thus, micropurge sampling is less labour intensive.
Micropurge sampling only requires inert gas, a controller unit and a pump which eliminates multiple visits to bores for sampling, reduces manual handling from bailing and the need for petrol driven compressors for bore purging. Micropurge sampling is a safe, cost effective method of sampling.
The HydraSleeve is a relatively new method of groundwater sampling. HydraSleeves are ‘no purge’ discrete sampling devices that are lowered to the known screen depth of the bore then, at the time of sampling, collect a sample from the aquifer as the device is drawn back up through the water column. These sampling devices are single use samplers, with a one way sample intake, so they can be deployed days or weeks prior to sampling and will only take a sample when drawn upwards through the aquifer.
HydraSleeves are available in a number of sizes, making them suitable for smaller diameter bores where conventional pumps and methods cannot be used. These sampling devices are also capable of collecting samples from bores of any yield and are especially suited to bores with a slow recharge, where other sampling methods cannot be reliably used due to their resultant disturbance of the standing water level as groundwater is purged from the bore.
HydraSleeves are one of the most cost effective ways of sampling groundwater. Benefits of the HydraSleeve include reduced setup and sampling times, no ongoing equipment maintenance, easier to transport and store, in addition to less manual handling requirements improving sampler safety in the field.
by Steven Abercromby, Team Leader Mackay
Australian ADCP Regatta, March 16-17 2017
The Regatta consisted of a field day on the Thredbo River at Paddy’s Corner, followed by a short session on bathymetry techniques on the Murrumbidgee in Cooma before wrapping up with a discussion session at the Snowy Hydro Discovery Centre, Cooma. The event gave participants a great opportunity to explore a range of objectives.
Overall the Regatta was attended by fifty four participants, representing five states and territories, along with a representative from New Zealand’s Marlborough District Council. Overall the participating organisations represented a good mix of private and public hydrometric service providers, consultants and vendors.
Participating Organisations were:
- Water NSW;
- Bureau of Metrology;
- Department of Environment, Land, Water and Planning (DELWP);
- Department of Water WA (now Department of Water and Environmental Regulation);
- Manly Hydraulics Lab;
- Marlborough District Council, New Zealand;
- Snowy Hydro Limited;
- Ventia; and
Vendors onsite included:
- HyQuest Solutions;
- Xylem Analytics; and
- Blue Zone Group.
This article is a summary of the events, results and outcomes from the regatta with recommendations and suggestions for future events.
As per the 2015 Regatta the event aimed to:
- Provide an opportunity for practitioners/organisations to ‘pressure test’ and discuss the application of ADCP discharge measurement techniques and processes against the National Industry Guidelines for hydrometric monitoring, Part 8.
- Enable opportunity for continuous improvement discussions between practitioners and suppliers with regard to the effectiveness/practicalities of the Guideline, application of techniques and instrument and gauging hardware applications.
- Validate ADCP performance by conducting comparison measurements and provide information to enable assessment of repeatability of data outputs.
- Provide opportunities to compare and share knowledge on data collection techniques, procedures, instruments and application in a potential variety of field conditions amongst practitioners and peers.
- Compile a report on results and outcomes for knowledge sharing amongst the National Hydrometric Industry.
In addition the 2017 Regatta identified two additional areas for further exploration in line with recommendations made following the 2015 Regatta:
- Location: Challenge processes/SOP’s and technology with a mix of good and bad sections; and
- Guidelines: Test more aspects of National Industry Guidelines for hydrometric monitoring, Part 8, Application of Acoustic Doppler Current Profilers to Measure Discharge in Open Channels under a wider range of conditions.
The potential to have a session on basic bathymetry applications (Bathymetry 101) with off the shelf products was also explored as a part of the two day program.
The Regatta comprised of a mix of onsite tasks followed by an information and/or discussion session on the previous day’s results.
Prior to the event a survey was sent to all the registered participants, in order to collect background information in relation to the organizations represented.
The participants were also asked to complete a post regatta survey, providing feedback on the workshop, content covered, topics for future consideration and the usefulness of such an event. Participants were also asked to provide their thoughts on future events and workshops.
The workshop was run by ALS under the mentorship of Snowy Hydro, who conducted the first regatta in 2015. This concept of the incumbent providing guidance to the next host worked well and should be considered for any future events.
The regatta was held on the Thredbo River near Jindabyne. The area was considered suitable as it contained:
- A channel stretch of approx. 30m to 40m wide with sections of fast moving shallow water on a rocky bed transitioning to a large sandy pool with distinct flow patterns due to its location on a bend;
- Natural hazards such as an island upstream and several riffle sections, creating a range of hydraulic conditions;
- A Permanent SHL traveller line, set up in the main body of the channel, and SHL Hydrographer onsite to assist with site knowledge;
- Natural flows in the 2-10 cumecs range. These were recognised as safe wading flows at this site.
Regatta Process and Results
Deployment on the day saw Snowy Hydro Ltd setup five dedicated taglines. These five sites represented the five different types of sections observed at this location, and providing a range of challenges from island in centre to large backwater sections (Refer to Figure 8 and Figure 9).
The Snowy Hydro rating for site indicated an expected flow of 1.07 cumecs. This represented an extremely low flow and necessitated serious consideration as to the methods and procedures to apply in order to produce repeatable data collection. Due to the volume of attendees, many groups elected to “choose their own adventure” both upstream and downstream of the dedicated sections. These sections were identified as “Other”.
Some Statistics collected during the day:
- 54 Participants/Observers;
- 7 organisations from five states (NSW, Vic, SA, WA, Tas);
- ADCP’s used included S5, M9, RiverRay, River Pro, Stream Pro;
- Other technology used - Oss B, FlowTracker;
- 47 measurements were collected;
- Methods used – Stationary 23, Moving 18, Wading 6;
- Mean Q = 1.086 cumecs (mean of all measures);
- Rated Q = 1.07 cumecs;
- 5 Set sections 18-25 m wide – 28 measures;
- Choose your own adventure – 19 measures;
- Wading gaugings were conducted downstream of the control;
- No moving boats were attempted in site 1.
All participants conducted measurements based on their own organisational requirements and procedures. The required information was collected from each gauging, and tabulated to form the basis of the reporting, based on the following information:
- Equipment / Model
- Moving Boat Test
- Q Raw
- Sensor Head Depth
- Exposure Time (secs)
Post processing techniques were not applied to any data collected, it is suggested they should be considered and discussed at future regattas.
SonTek and RDI equipment was used on the day with a larger percentage of organisations using the SonTek M9 equipment.
The adopted mean has been taken from all flow gauging’s performed. Overall results show a 1.086 cumec with the breakdown from moving boat, stationary and wading indicated in Figure 10.
Observations/commentary on the results
- Some large scatter in the deviation. These outliers could be easily explained with comments provided on the field sheets. Many organizations trialling various methods in the low flow conditions to prove that results can vary.
- The low Q on day contributed to the scatter – larger flows at a uniform site in theory should see the scatter reduced (as demonstrated at 2015 Regatta).
- During the regatta participants were experimented with the operational limits of ADCP equipment. Some of the larger scatters may be result of participants experimenting with differentmodes/techniques, as well as testing ADCPs in conditions where they would not normally operate effectively.
- An instance of two ADCP’s being tethered together (side by side) yielded exactly same result. In this instance it is not clear if frequency overlap/receiving issues contributed to this result. The participant organisation will investigate the results further.
- Large scatter occurred around measurements at Section 1. This was to be expected as it was considered the poorest section with depths encroaching on blanking distance, estimates required at shallow sections etc. It should be noted that no moving boat measurements were attempted at this section.
- Measurement scatter from Regatta Mean by section:
- Section 1: +20% to -24%
- Section 2: +14% to -2%
- Section 3: +7% to -3.5%
- Section 4: +24% to -13%
- Section 5: +3.5% to -12.5%
- Alternate sections: +14% to -20%
The minimum acceptable time of exposure for ADCPS gauging’s, outlined in the quality assessment guide in the National Guidelines, is 800 seconds. This is further broken into Stationary and Moving boat. Stationary being for the sum of the times at each vertical and Moving Boat is the sum of the exposure times for the accepted transect used to calculate the final discharge. An exposure time of less than 800 seconds should incur a downgrade in the quality ranking.
Exposure times were not provided for all gauging’s completed, however, from those that were provided the average exposure time was 1015.
The exposure comparison between moving boat and stationary indicate that there is very little advantage in either method with regard to time saved. This was also evident in the 2015 Regatta.
Some other notes from the 2015 Regatta that participants agreed were still relevant, with regard to exposure time:
- It is important to keep the boat speed at less than the water speed. As a result extended sections of slower flows resulted in reduced boat speeds;
- Some larger times in the results would likely be due to visiting groups being unfamiliar with the section and flow conditions. It was found that as they became familiar with the transects, they became more efficient.
- It became apparent during the general discussion that 40 seconds minimums were being used at each vertical with some using 60 seconds.
- The number of verticals measured in the cross section can vary with stream width (aligning with aspects of AS3778).
- Spacing of the verticals through the section was done such that the participants could define the stream bed (rather than using fixed spacing).
Post Regatta Discussion
Some general observations
- The Bureau of Meteorology National Industry Guidelines for hydrometric monitoring, Parts 8 to 10 are due for renewal;
- JRGWI and/or the Bureau of Meteorology have been asked to provide a list of who is on this committee. Each of the agencies/participants should push information and queries through to them via the WaMSTeC processes and/or their state JRGWI representatives
- Technical Reference Group AHA (ADCP) – Contact is Simon Cruickshank (AHA committee);
- It was suggested that Google Drive could be used for the sharing of documents between organizations. Further feedback on this suggestion is sought; and
- Regattas are a great way to QA/QC your device against other units. This is a similar observation to those noted at the New Zealand Regattas.
- The site selected proved challenging due to the low velocities, most agreed this was a good test of equipment capabilities;
- While site selection is key in normal gauging processes participants thought it may not have been the largest contributor to the scatter in the resultant data but rather the low flow conditions pushed the operational limits of equipment;
- There were too many crews to work within the five designated sites. This resulted in many gauging’s taken at alternate locations;
- The site was difficult compared with the first Regatta, however the ability to alter and control flows at future Regattas would be an advantage;
- Does it matter that the regatta site has an existing rating – most agree yes;
- Post processing of data – session on this and the changes it makes to dataset would be good for “classroom discussion”.
- Bathymetry, requires more in depth discussion and perhaps a regatta in its own right.
This article first appeared in the Australasian Hydrographer August 2017, the journal of the Australian Hydrographers Association.
by Anthony Skinner, NSW Operations Manager