ArcGIS Server and GDA2020

Background

Esri has been working for to incorporate time-based geographic transformations into the software. As you can imagine, these transformations are hugely complex, since the process involves the actual position of the features on the surface of the earth, taking continental drift, and plate tectonic motion into consideration. These time-based transformations are based on a so-called “epoch”, which is the date on which all features are at located in relation to an absolute location.

There are several moving parts involved here.

1] The computer programming to perform the math to implement these geographic transformations.

2] The transformation parameters (14 for time based transformations) to transform data between an original, static datum like GDA94, which assumes that the features of the earth are in one position and not moving and a datum like GDA2020.  Later datums will be dynamic so as to take  plate motion into consideration, this is discussed here, software vendors are already planning for this new approach.

3] The transformation parameters to transform between one time-based datum, and another more current time-based transformation – GDA2020 to say, GDA2030 for example if such a thing were to exist someday.

What the ArcGIS Software does not do is calculate these transformation parameters. ArcGIS checks them and verifies that the transformation parameters provided by the national government agency perform to the accuracy provided, but does not calculate them.

Therefore, even if ArcGIS has the transformation method in place to perform a 14-parameter time-based transformation, ArcGIS needs the transformation parameters, from the national government agency that has created the specific transformation for their area of interest.

ArcGIS software does support GDA2020 Z point transformations ‘out of the box’ but not the NTv2 grid files for the mathematical and mathematical plus distortion transformations.  The NTv2 grid files will have to downloaded (which include some content from Geoscience Australia) for the appropriate ArcGIS version from;

 My Esri > Product Components > Data and Content > ArcGIS Coordinate Systems Data

Description:
Contains the data files required for the … transformation method and vertical transformation files for … the world … for use with either ArcMap, ArcGIS Enterprise, ArcGIS Engine, or an ArcGIS Pro per machine install

and install on each ArcGIS machine, this will install to C:\Program Files (x86)\ArcGIS\CoordinateSystemsData.

This installer is specific to the ArcGIS version, eg:

  • ArcGIS_Coordinate_Systems_Data_Windows_1061_163829.exe
  • ArcGIS_Coordinate_Systems_Data_Windows_1071_169282.exe

The NTv2 transformation between GDA 1994 and GDA 2020 is one small moving part of this major upheaval in the world of geodesy, and the pursuit of ever-more-accurate and ever-more-precise coordinates for features on the earth’s surface.
Esri publish documentation specific to each ArcGIS version which have included GDA2020 for some time, for instance in 10.5.1 they put the transformation GDA_1994_to_GDA_2020_NTv2_CD in 10.5.1 assuming that they knew the names of ntv2 files, then Geoscience Australia changed the file names so this transformation was deprecated in the _C versions. As far as this naming is concerned _C is for Conformal and _CD is Conformal_and_Distortion. Each of these transformations will get a unique Well Known ID (WKID) value each time the transformation changes. For example:

geographic_transformations_1060

geotran_1071_24

How To: Select the correct geographic (datum) transformation when projecting between datums

Esri Australia have been publishing blogs on GDA2020 for a while;

https://esriaustraliatechblog.wordpress.com/tag/gda2020/

Up to version 10.6.1 for some situations you needed to rename the GSB file as mentioned here;

https://esriaustraliatechblog.wordpress.com/2018/01/04/gda2020-arcgis/

The transformation 108065 was added to the Projection Engine at 10.6.0, but was removed from 10.6.1, so that transformation never existed at 10.6.1. The same transformation, but with a new name for the GSB file was added again at 10.7.1.
The conclusion here is that you should upgrade to the latest version of ArcGIS to get the most GDA2020 transformations.
A map of the approximate shifts from GDA94 to GDA2020 locations across Australia is here; https://www.icsm.gov.au/gda2020

ArcGIS Server (does not support time-dependent transformations yet)

Expect in 2020 many agencies will be publishing services natively GDA2020.
If all data is stored in GDA2020 and published in GDA2020 there should not be any issues, apart from later on taking into account the epoch of the data, as mentioned here;

https://esriaustraliatechblog.wordpress.com/2018/12/24/understanding-gda2020-and-its-relationship-with-web-gis/

If you are upgrading your servers and need to change the coordinate systems of your data or services, please republish the services. The version of the ArcGIS Coordinate Systems Data should be identical on the desktop client and server should be the same.

You can test a server’s support for a particular transformation if the GeometryServer is running;

https://localhost/server/rest/services/Utilities/Geometry/GeometryServer/project?inSR=4283&outSR=7856&geometries={“geometryType” : “esriGeometryPoint”,”geometries”: [{“x”: 153.30141143,”y”: -27.90831298}]}&transformation=108447&transformForward=true&vertical=false&f=html

should return (which is less than 1mm away from the expected ordinates);

{“geometries”: [{ “x”: 529661.2445906086, “y”: 6912919.071889246}]}

For instance, trying;

url1will return the message;

Invalid or missing input parameters.
the specified wkid or wkt is not a geotransformation

As;

url2

is at ArcGIS version 10.4.1 so is not capable of doing GDA2020 transformations.

 

 

 

 
If you access a more recent ArcGIS Server version like;

url4

 

and you get an error like;

Unable to complete operation.
Cannot load the data file Dataset_australia/GDA94_GDA2020_conformal_and_distortion for the geographic transformation GDA_1994_To_GDA2020_NTv2_2_Conformal_and_Distortion.

Then the server is capable of that transformation but the ArcGIS Coordinate Systems Data installer has not been run on the server.
What this means is that if you query the following to add a layer to a web map of yours which is going to be in a particular coordinate system like outSR=7856 https://epsg.io/7856 and you get geometry from this layer;

 

url5

{“displayFieldName”:”PROP.QLD_SURVEYCONTROL_SCDB.MRK_ID”,”fieldAliases”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:”GDA latitude”,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:”GDA longitude”,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:”AHD height”},”geometryType”:”esriGeometryPoint”,”spatialReference”:{“wkid”:7856,”latestWkid”:7856},”fields”:[{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA latitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA longitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”,”type”:”esriFieldTypeDouble”,”alias”:”AHD height”}],”features”:[{“attributes”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:-27.9083129838,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:153.3014114276,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:30.34},”geometry”:{“x”:529660.65561161563,”y”:6912917.6855818881}}]}

Or this (where you also asked for the datum transformation);

url6

{“displayFieldName”:”PROP.QLD_SURVEYCONTROL_SCDB.MRK_ID”,”fieldAliases”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:”GDA latitude”,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:”GDA longitude”,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:”AHD height”},”geometryType”:”esriGeometryPoint”,”spatialReference”:{“wkid”:7856,”latestWkid”:7856},”fields”:[{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA latitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA longitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”,”type”:”esriFieldTypeDouble”,”alias”:”AHD height”}],”features”:[{“attributes”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:-27.9083129838,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:153.3014114276,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:30.34},”geometry”:{“x”:529660.65561161563,”y”:6912917.6855818881}}]}

Then you are not going to see the points transformed onto your map as you expect (as the server does not have the NTv2 files) and the layer will be not shifted as needed.

Make it your 2020 resolution to bring your ArcGIS Servers up to date, install ArcGIS Coordinate Systems Data and discuss GDA2020 with your users!

 

 

 

 

ArcGIS Online Security Changes – Is Your Organisation Prepared?

Earlier last month Esri published a blog post titled Prepare for Next Major ArcGIS Online Security Advancement Now. I have to admit when I first read it; I didn’t pause for long – thinking it was related to another recent web-security related change by Esri – the switch to TLS 1.2

If your eyes are glazing over already, hang in there. There was more to this post than I first thought, and it’s something you should be thinking about now if you’re administering an ArcGIS Online organisation that has been in place for some time. To be specific, if you created your ArcGIS Online site before September 2018, then you should read on.

What’s going to happen in 2020 is that ArcGIS Online will no longer work with external references to resources that use HTTP in the referring URL. Only HTTPS references will be supported. This is the way the web is moving and Esri is simply following best practice.

If you created your ArcGIS Online organisational after September 2018, then you will have been subject to this restriction from the get-go and won’t have a problem (it has been the default position since that time). If the organisational site is older than that, then there’s a chance you could have these less secure references to resources lurking in your web maps, web scenes and other items. If you do nothing, a bunch of things may stop working at some point in 2020, and you’ll be scurrying to try and fix them in a hurry.

For any of you utilising Story Maps you may have already encountered this. In 2018 the Story Map team implemented HTTPS only compliant web apps. This meant not only did the story map have to be secured with HTTPS but also any referenced site in a story map  also had to be secured in the same fashion. In 2020 ArcGIS Online in its entirety will follow suite. Details on that earlier Story Map change here.

How would you know you’ve got a problem to solve? Esri has created a tool called the ArcGIS Online Security Advisor that will scan all the items in your organisation, looking for the issue. You’ll need to be logged in as an administrator of the organisation to do this. The HTTP Check component of that tool is in Beta right now with new capabilities being added regularly.

It can’t directly fix the issues because a simple replacement of HTTP with HTTPS in the reference may not work  if the target server doesn’t support HTTPS. However, it will give you the feedback you need to go triage each of the problems it flags.

A typical scenario that could impact many users is where a GIS service is coming from an older version of ArcGIS Server and added as an item to an ArcGIS Online organisation. That older ArcGIS Server version would have allowed the specification of just HTTP, or both HTTP and HTTPS when exposing services.

Here’s an example. Way back, I created a web map in my ArcGIS Online organisation that refers to the Australian Coastal Sediment Compartments web service from GeoScience Australia.  The link here is for the HTTPS version (since this is all about best practice), but when I created that map, I used the HTTP version. Both forms are currently supported by the GeoScience Australia GIS Server.

GAMap

The map displays the GA map service on top of the Esri Oceans basemap and works fine. Come 2020; this won’t be the case. If I use the beta HTTP Check tool in the ArcGIS Online Security Advisor, it picks up two problems with the web map.

GAMapDetection

 
In scanning the web map item, it detected that I have the URL of the GA web service in the Description. While that won’t break the map when HTTP is no longer supported, it still needs attention given the resource it refers to will change.

GAMapItem

The second pickup by the tool was the actual URL to the GA service in the JSON data describing the web map.

GAMapItemData

Equally, if you’ve created items in your portal that refer to resources coming from external sources  – say a web service from an external agency that you collaborate with, then you may come up against the same issue if they’re using HTTP.

When you interact with a web site that doesn’t use HTTPS to encrypt traffic these days, you get to know it. It’s no longer just a small broken padlock icon – mainstream browsers are now calling it out and telling you that the site is not secure. That’s a good thing, and Esri is just doing its part to ensure web security best practices are adhered to.

Read the original blog post here and use the ArcGIS Online Security Advisor tool to determine whether you need to take any action.

Imagery Best Practices

Supporting Imagery and Lidar in the ArcGIS platform has been around for a long time. In the ArcGIS Server Space, Image Server became available at 9.3.1. Since then it has evolved to Mosaic Datasets, Image Services, Raster Functions and now raster analytics. Now imagery is really an integral part of the ArcGIS platform. However, it is only as performant when the imagery is managed and configured optimally.

I often get asked,

  • what format should I store my imagery in?,
  • How many images can be in a Mosaic Dataset?
  • How should I structure my imagery? and
  • what is the maximum number of images per folder?

When answering these questions I have drawn on past experience and advice from Esri. Now though Esri have compiled all this information into an Excellent Centralised resource Imagery Workflows – Best Practices https://doc.arcgis.com/en/imagery/workflows/best-practices/what-are-best-practices.htm

In the Imagery formats and Performance section it details topics such as:

  • File format suitability
  • Recommended imagery formats
  • Reformatting imagery
  • Pyramids
  • Statistics
  • Working with large mosaics
  • Storage system performance

If you’re going to be managing imagery and lidar I recommend you reads these documents. They are comprehensive and invaluable. I must admit I have been doing this for 12 years now and there is information on Lidar management that I did not know about.

This is just one component of the ArcGIS Imagery Workflows documentation Esri has just produced.

Gordon

ArcGIS 10.2.1 for Utilities supports GDA2020

Esri recently released an update for the ArcGIS 10.2.1 for Utilities and Telecom that has included support for GDA2020 projections and transformations.
The update includes

  • support for the mathematical transformation between GDA94 and GDA2020
  • Support for the NtV2.0 Grid file transformations
  • Also included are the Conformal and Conform + Distortion grid files.

If you intend on migrating to GDA2020 at 10.2.1 then we recommend you update to the latest Patch #9 available from
https://support.esri.com/en/download/7680

Important Security Updates to the ArcGIS Platform

January, 2019

Esri have recently announced upcoming improvements to ArcGIS Online in order to maintain the highest industry standards for data integrity and network security. Starting on 16 April 2019, ArcGIS Online will only accept TLS 1.2 connections for ArcGIS Online services. Some software, like ArcGIS Pro, are already TLS 1.2 enabled. Other Esri software, such as ArcGIS Desktop, uses TLS 1.0—this software requires a patch or configuration change to support TLS 1.2 connections. Esri is releasing patches and instructions to update existing software to support these connections.

What is TLS?
TLS or  “Transport Layer Security” is a widely deployed network security protocol. It provides privacy and data integrity between communicating applications over a network. You use TLS whenever accessing ArcGIS Online services, such as basemaps, geoprocessing services, and the Living Atlas, from ArcGIS Desktop, ArcGIS Enterprise, and other applications.

Continue reading

Understanding GDA2020 and it’s relationship with Web GIS

Background

Australia sits on one of the Earth’s fastest moving tectonic plates which has been moving 70 millimetres per year. By 2020, Australia will have moved 1.8 metres north east of it’s location in 1994. To effectively map the earth, representations, known as datums are used to model and identify locations. Australia’s national datum; Geocentric Datum of Australia 1994 (GDA94) will soon be replaced by a new datum Geodetic Datum of Australia (GDA2020). GDA2020 will align with current positioning technology.

Geoscience Australia and the Intergovernmental Committee on Surveying and Mapping (ICSM) have released a new Geodetic datum GDA2020. The datum provides higher positional accuracy and will be able to represent locations dynamically rather than the stationary GDA94. .

In the coming two to three years most organisations will be transforming their spatial data from the GDA94 datum to the new GDA2020 datum. Esri’s, ArcGIS Software has been updated to accommodate these new datums across Australia and will support all of our customers rigorous requirements for locational and positional accuracy.

Continue reading

Important Portal for ArcGIS Security Alert

17 December 2018

Esri has announced that they have discovered a critical security vulnerability in Portal for ArcGIS when specially constructed steps are taken by authenticated users. This results in a privilege escalation issue where the user can elevate themselves to become administrators of the portal.

This issue is present in all supported versions of Portal for ArcGIS, on both Windows and Linux operating systems. Esri has released patches for all versions of Portal for ArcGIS, from version 10.3 through 10.6.1.

Esri have published the following Knowledge Base article relating to this issue: Problem: Warning of security vulnerability in Portal for ArcGIS

Continue reading