Category Archives: General

Important Portal for ArcGIS Security Alert

4 June 2020

Esri has announced that they have detected a critical security vulnerability in the Portal for ArcGIS component of ArcGIS Enterprise when special steps are taken by persons with network access to the ArcGIS Enterprise portal to exploit Server-Side Request Forgery (SSRF), which can result in access to and control over other infrastructure resources by unauthenticated persons.

This can affect deployments running in Amazon Web Services (AWS) in particular which makes this issue particularly urgent for those deployments.

This security issue affects all supported versions prior to ArcGIS Enterprise 10.8 on both Windows and Linux operating systems.

What you need to do

Esri has released patches for versions of ArcGIS Enterprise from 10.5 through to 10.7.1.

Esri strongly recommends installing the Portal for ArcGIS Security 2020 Update 1 Patch at your earliest opportunity. ArcGIS Enterprise 10.8 already contains these fixes and is not affected.

More information

Esri have published the following Blog and Knowledge Base article relating to this issue:

KB Article: Problem: Warning of security vulnerability in ArcGIS Enterprise

Blog: Critical Security Patch for ArcGIS Enterprise portal Released

Getting Help

If you have any questions or concerns, please contact your local Esri distributor’s support team (for Esri Australia clients, use My Esrisupport@esriaustralia.com.au, call 1800 447 111; or book a Consultation with a Technical specialist). For any email requests please add “Portal for ArcGIS Security Patch June 2020” to the subject line so that it can be addressed.

 

Survey123 for ArcGIS: Securing surveys and results

Survey123 for ArcGIS is a complete, form-centric solution for creating, sharing and analyzing surveys. It is used to create smart forms and can be submitted from a web browser or dedicated Survey123 for ArcGIS native app by a defined audience.

Surveys can be designed on the Survey123 website or via the desktop with Survey123 Connect. Once designed, the survey is published, and in this process a survey form and service layers are created in the designer’s portal (ArcGIS Online or ArcGIS Enterprise). The survey form item represents the questions and survey settings, and the survey layers storing the submitted data.

After publishing you can collaborate with other users via the Survey123 website. Collaboration options apply to Submitters and Viewers, giving control on who can submit, what they submit and then also who can view the results. As these collaboration settings are applied in the Survey123 website the settings on the items in portal are updated. In the example below, updating ‘Who can submit to this survey’ to ‘Everyone (Public)’ changed the sharing on the survey form and fieldworker service layer to ‘Everyone (Public)’:

Survey123 Collaboration settings show survey is shared to Everyone (Public)
Survey123 Collaboration settings
The Everyone (Public) settings on the Survey123 website push to the portal item settings
ArcGIS Online item settings: the survey form (questions) and service layer (where the submitted data is stored) are shared to Everyone (Public)

This highlights several things:

  • Survey123 leverages the storage, sharing and security model of the ArcGIS platform
  • You can access the content directly to apply granular settings or use the layers in other maps and web mapping applications
  • The Survey123 platform is reliant on these items. Care needs to be taken not to make sharing or item setting changes that would break the platform’s ability to function.

In general it’s best to configure settings through the Survey123 website, as this will ensure that all items are updated as necessary while retaining platform functionality. A somewhat common mistake is sharing only the ‘form’ via the portal Content screen. This allows users to access the survey form but will fail on submission as they don’t have access to view/edit the associated feature layer. Sharing through the Survey123 website ensured that both items were shared at the appropriate level.

When working with public surveys, greater control is often desired over who can submit and view survey data. If not secured, public survey results containing private or sensitive information could be viewed or manipulated. By configuring the securing options for the underying survey layers that support the survey, you can allow public users to submit surveys without exposing previously collected data. Other stakeholders can be provided access to the data, in part or its entirety, through Feature Layer Views.

Esri recently released two key resources to help you with securing your survey data:

If you’re a Survey123 publisher or organization administrator, you’ll want to take a look at the above two resources to ensure that you understand the level of access to your survey data. It’s important to remember that some surveys intentionally share their results publicly. If you have any additional questions about implementing these best practices, please contact Tech Support.

Additional information:

Important ArcGIS Server Security Alert

3 April 2020

Esri has announced that they have detected a critical security vulnerability in ArcGIS Server when specially constructed steps are taken by persons with network access to the ArcGIS deployment to exploit Server-Side Request Forgery (SSRF), which can potentially be used to obtain access to sensitive internal system information by unauthorized individuals.

This issue is present in ArcGIS for Server, on both Windows and Linux operating systems. Esri has released patches for 10.7.1 and prior supported versions of ArcGIS for Server here. ArcGIS Server 10.8 is unaffected by this issue.

Esri have published the following Blog and Knowledge Base article relating to this issue:

Critical Security patch for ArcGIS Server Released.

Problem: Warning of security vulnerability in ArcGIS Server

Esri strongly recommends installing the relevant patch at your earliest possible opportunity. All patches can be downloaded from the Esri Support website:

ArcGIS Server Security 2020 Update 1 Patch.

If you have any questions or concerns, please contact your local Esri distributor’s support team (for Esri Australia clients, use My Esri, support@esriaustralia.com.au or 1800 447 111) For any email requests please add “ArcGIS Server Security Patch, April 2020” to the subject line so that it can be addressed.

ArcGIS Server and GDA2020

Background

Esri has been working for to incorporate time-based geographic transformations into the software. As you can imagine, these transformations are hugely complex, since the process involves the actual position of the features on the surface of the earth, taking continental drift, and plate tectonic motion into consideration. These time-based transformations are based on a so-called “epoch”, which is the date on which all features are at located in relation to an absolute location.

There are several moving parts involved here.

1] The computer programming to perform the math to implement these geographic transformations.

2] The transformation parameters (14 for time based transformations) to transform data between an original, static datum like GDA94, which assumes that the features of the earth are in one position and not moving and a datum like GDA2020.  Later datums will be dynamic so as to take  plate motion into consideration, this is discussed here, software vendors are already planning for this new approach.

3] The transformation parameters to transform between one time-based datum, and another more current time-based transformation – GDA2020 to say, GDA2030 for example if such a thing were to exist someday.

What the ArcGIS Software does not do is calculate these transformation parameters. ArcGIS checks them and verifies that the transformation parameters provided by the national government agency perform to the accuracy provided, but does not calculate them.

Therefore, even if ArcGIS has the transformation method in place to perform a 14-parameter time-based transformation, ArcGIS needs the transformation parameters, from the national government agency that has created the specific transformation for their area of interest.

ArcGIS software does support GDA2020 Z point transformations ‘out of the box’ but not the NTv2 grid files for the mathematical and mathematical plus distortion transformations.  The NTv2 grid files will have to downloaded (which include some content from Geoscience Australia) for the appropriate ArcGIS version from;

 My Esri > Product Components > Data and Content > ArcGIS Coordinate Systems Data

Description:
Contains the data files required for the … transformation method and vertical transformation files for … the world … for use with either ArcMap, ArcGIS Enterprise, ArcGIS Engine, or an ArcGIS Pro per machine install

and install on each ArcGIS machine, this will install to C:\Program Files (x86)\ArcGIS\CoordinateSystemsData.

This installer is specific to the ArcGIS version, eg:

  • ArcGIS_Coordinate_Systems_Data_Windows_1061_163829.exe
  • ArcGIS_Coordinate_Systems_Data_Windows_1071_169282.exe

The NTv2 transformation between GDA 1994 and GDA 2020 is one small moving part of this major upheaval in the world of geodesy, and the pursuit of ever-more-accurate and ever-more-precise coordinates for features on the earth’s surface.
Esri publish documentation specific to each ArcGIS version which have included GDA2020 for some time, for instance in 10.5.1 they put the transformation GDA_1994_to_GDA_2020_NTv2_CD in 10.5.1 assuming that they knew the names of ntv2 files, then Geoscience Australia changed the file names so this transformation was deprecated in the _C versions. As far as this naming is concerned _C is for Conformal and _CD is Conformal_and_Distortion. Each of these transformations will get a unique Well Known ID (WKID) value each time the transformation changes. For example:

geographic_transformations_1060

geotran_1071_24

How To: Select the correct geographic (datum) transformation when projecting between datums

Esri Australia have been publishing blogs on GDA2020 for a while;

https://esriaustraliatechblog.wordpress.com/tag/gda2020/

Up to version 10.6.1 for some situations you needed to rename the GSB file as mentioned here;

https://esriaustraliatechblog.wordpress.com/2018/01/04/gda2020-arcgis/

The transformation 108065 was added to the Projection Engine at 10.6.0, but was removed from 10.6.1, so that transformation never existed at 10.6.1. The same transformation, but with a new name for the GSB file was added again at 10.7.1.
The conclusion here is that you should upgrade to the latest version of ArcGIS to get the most GDA2020 transformations.
A map of the approximate shifts from GDA94 to GDA2020 locations across Australia is here; https://www.icsm.gov.au/gda2020

ArcGIS Server (does not support time-dependent transformations yet)

Expect in 2020 many agencies will be publishing services natively GDA2020.
If all data is stored in GDA2020 and published in GDA2020 there should not be any issues, apart from later on taking into account the epoch of the data, as mentioned here;

https://esriaustraliatechblog.wordpress.com/2018/12/24/understanding-gda2020-and-its-relationship-with-web-gis/

If you are upgrading your servers and need to change the coordinate systems of your data or services, please republish the services. The version of the ArcGIS Coordinate Systems Data should be identical on the desktop client and server should be the same.

You can test a server’s support for a particular transformation if the GeometryServer is running;

https://localhost/server/rest/services/Utilities/Geometry/GeometryServer/project?inSR=4283&outSR=7856&geometries={“geometryType” : “esriGeometryPoint”,”geometries”: [{“x”: 153.30141143,”y”: -27.90831298}]}&transformation=108447&transformForward=true&vertical=false&f=html

should return (which is less than 1mm away from the expected ordinates);

{“geometries”: [{ “x”: 529661.2445906086, “y”: 6912919.071889246}]}

For instance, trying;

url1will return the message;

Invalid or missing input parameters.
the specified wkid or wkt is not a geotransformation

As;

url2

is at ArcGIS version 10.4.1 so is not capable of doing GDA2020 transformations.

 

 

 

 
If you access a more recent ArcGIS Server version like;

url4

 

and you get an error like;

Unable to complete operation.
Cannot load the data file Dataset_australia/GDA94_GDA2020_conformal_and_distortion for the geographic transformation GDA_1994_To_GDA2020_NTv2_2_Conformal_and_Distortion.

Then the server is capable of that transformation but the ArcGIS Coordinate Systems Data installer has not been run on the server.
What this means is that if you query the following to add a layer to a web map of yours which is going to be in a particular coordinate system like outSR=7856 https://epsg.io/7856 and you get geometry from this layer;

 

url5

{“displayFieldName”:”PROP.QLD_SURVEYCONTROL_SCDB.MRK_ID”,”fieldAliases”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:”GDA latitude”,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:”GDA longitude”,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:”AHD height”},”geometryType”:”esriGeometryPoint”,”spatialReference”:{“wkid”:7856,”latestWkid”:7856},”fields”:[{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA latitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA longitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”,”type”:”esriFieldTypeDouble”,”alias”:”AHD height”}],”features”:[{“attributes”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:-27.9083129838,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:153.3014114276,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:30.34},”geometry”:{“x”:529660.65561161563,”y”:6912917.6855818881}}]}

Or this (where you also asked for the datum transformation);

url6

{“displayFieldName”:”PROP.QLD_SURVEYCONTROL_SCDB.MRK_ID”,”fieldAliases”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:”GDA latitude”,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:”GDA longitude”,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:”AHD height”},”geometryType”:”esriGeometryPoint”,”spatialReference”:{“wkid”:7856,”latestWkid”:7856},”fields”:[{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA latitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”,”type”:”esriFieldTypeDouble”,”alias”:”GDA longitude”},{“name”:”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”,”type”:”esriFieldTypeDouble”,”alias”:”AHD height”}],”features”:[{“attributes”:{“PROP.QLD_SURVEYCONTROL_SCDB.GDALATITUDE”:-27.9083129838,”PROP.QLD_SURVEYCONTROL_SCDB.GDALONGITUDE”:153.3014114276,”PROP.QLD_SURVEYCONTROL_SCDB.AHDHEIGHT”:30.34},”geometry”:{“x”:529660.65561161563,”y”:6912917.6855818881}}]}

Then you are not going to see the points transformed onto your map as you expect (as the server does not have the NTv2 files) and the layer will be not shifted as needed.

Make it your 2020 resolution to bring your ArcGIS Servers up to date, install ArcGIS Coordinate Systems Data and discuss GDA2020 with your users!

 

 

 

 

ArcGIS Online Security Changes – Is Your Organisation Prepared?

Earlier last month Esri published a blog post titled Prepare for Next Major ArcGIS Online Security Advancement Now. I have to admit when I first read it; I didn’t pause for long – thinking it was related to another recent web-security related change by Esri – the switch to TLS 1.2

If your eyes are glazing over already, hang in there. There was more to this post than I first thought, and it’s something you should be thinking about now if you’re administering an ArcGIS Online organisation that has been in place for some time. To be specific, if you created your ArcGIS Online site before September 2018, then you should read on.

What’s going to happen in 2020 is that ArcGIS Online will no longer work with external references to resources that use HTTP in the referring URL. Only HTTPS references will be supported. This is the way the web is moving and Esri is simply following best practice.

If you created your ArcGIS Online organisational after September 2018, then you will have been subject to this restriction from the get-go and won’t have a problem (it has been the default position since that time). If the organisational site is older than that, then there’s a chance you could have these less secure references to resources lurking in your web maps, web scenes and other items. If you do nothing, a bunch of things may stop working at some point in 2020, and you’ll be scurrying to try and fix them in a hurry.

For any of you utilising Story Maps you may have already encountered this. In 2018 the Story Map team implemented HTTPS only compliant web apps. This meant not only did the story map have to be secured with HTTPS but also any referenced site in a story map  also had to be secured in the same fashion. In 2020 ArcGIS Online in its entirety will follow suite. Details on that earlier Story Map change here.

How would you know you’ve got a problem to solve? Esri has created a tool called the ArcGIS Online Security Advisor that will scan all the items in your organisation, looking for the issue. You’ll need to be logged in as an administrator of the organisation to do this. The HTTP Check component of that tool is in Beta right now with new capabilities being added regularly.

It can’t directly fix the issues because a simple replacement of HTTP with HTTPS in the reference may not work  if the target server doesn’t support HTTPS. However, it will give you the feedback you need to go triage each of the problems it flags.

A typical scenario that could impact many users is where a GIS service is coming from an older version of ArcGIS Server and added as an item to an ArcGIS Online organisation. That older ArcGIS Server version would have allowed the specification of just HTTP, or both HTTP and HTTPS when exposing services.

Here’s an example. Way back, I created a web map in my ArcGIS Online organisation that refers to the Australian Coastal Sediment Compartments web service from GeoScience Australia.  The link here is for the HTTPS version (since this is all about best practice), but when I created that map, I used the HTTP version. Both forms are currently supported by the GeoScience Australia GIS Server.

GAMap

The map displays the GA map service on top of the Esri Oceans basemap and works fine. Come 2020; this won’t be the case. If I use the beta HTTP Check tool in the ArcGIS Online Security Advisor, it picks up two problems with the web map.

GAMapDetection

 
In scanning the web map item, it detected that I have the URL of the GA web service in the Description. While that won’t break the map when HTTP is no longer supported, it still needs attention given the resource it refers to will change.

GAMapItem

The second pickup by the tool was the actual URL to the GA service in the JSON data describing the web map.

GAMapItemData

Equally, if you’ve created items in your portal that refer to resources coming from external sources  – say a web service from an external agency that you collaborate with, then you may come up against the same issue if they’re using HTTP.

When you interact with a web site that doesn’t use HTTPS to encrypt traffic these days, you get to know it. It’s no longer just a small broken padlock icon – mainstream browsers are now calling it out and telling you that the site is not secure. That’s a good thing, and Esri is just doing its part to ensure web security best practices are adhered to.

Read the original blog post here and use the ArcGIS Online Security Advisor tool to determine whether you need to take any action.

Imagery Best Practices

Supporting Imagery and Lidar in the ArcGIS platform has been around for a long time. In the ArcGIS Server Space, Image Server became available at 9.3.1. Since then it has evolved to Mosaic Datasets, Image Services, Raster Functions and now raster analytics. Now imagery is really an integral part of the ArcGIS platform. However, it is only as performant when the imagery is managed and configured optimally.

I often get asked,

  • what format should I store my imagery in?,
  • How many images can be in a Mosaic Dataset?
  • How should I structure my imagery? and
  • what is the maximum number of images per folder?

When answering these questions I have drawn on past experience and advice from Esri. Now though Esri have compiled all this information into an Excellent Centralised resource Imagery Workflows – Best Practices https://doc.arcgis.com/en/imagery/workflows/best-practices/what-are-best-practices.htm

In the Imagery formats and Performance section it details topics such as:

  • File format suitability
  • Recommended imagery formats
  • Reformatting imagery
  • Pyramids
  • Statistics
  • Working with large mosaics
  • Storage system performance

If you’re going to be managing imagery and lidar I recommend you reads these documents. They are comprehensive and invaluable. I must admit I have been doing this for 12 years now and there is information on Lidar management that I did not know about.

This is just one component of the ArcGIS Imagery Workflows documentation Esri has just produced.

Gordon

ArcGIS 10.2.1 for Utilities supports GDA2020

Esri recently released an update for the ArcGIS 10.2.1 for Utilities and Telecom that has included support for GDA2020 projections and transformations.
The update includes

  • support for the mathematical transformation between GDA94 and GDA2020
  • Support for the NtV2.0 Grid file transformations
  • Also included are the Conformal and Conform + Distortion grid files.

If you intend on migrating to GDA2020 at 10.2.1 then we recommend you update to the latest Patch #9 available from
https://support.esri.com/en/download/7680

Important Security Updates to the ArcGIS Platform

January, 2019

Esri have recently announced upcoming improvements to ArcGIS Online in order to maintain the highest industry standards for data integrity and network security. Starting on 16 April 2019, ArcGIS Online will only accept TLS 1.2 connections for ArcGIS Online services. Some software, like ArcGIS Pro, are already TLS 1.2 enabled. Other Esri software, such as ArcGIS Desktop, uses TLS 1.0—this software requires a patch or configuration change to support TLS 1.2 connections. Esri is releasing patches and instructions to update existing software to support these connections.

What is TLS?
TLS or  “Transport Layer Security” is a widely deployed network security protocol. It provides privacy and data integrity between communicating applications over a network. You use TLS whenever accessing ArcGIS Online services, such as basemaps, geoprocessing services, and the Living Atlas, from ArcGIS Desktop, ArcGIS Enterprise, and other applications.

Continue reading

Understanding GDA2020 and it’s relationship with Web GIS

Background

Australia sits on one of the Earth’s fastest moving tectonic plates which has been moving 70 millimetres per year. By 2020, Australia will have moved 1.8 metres north east of it’s location in 1994. To effectively map the earth, representations, known as datums are used to model and identify locations. Australia’s national datum; Geocentric Datum of Australia 1994 (GDA94) will soon be replaced by a new datum Geodetic Datum of Australia (GDA2020). GDA2020 will align with current positioning technology.

Geoscience Australia and the Intergovernmental Committee on Surveying and Mapping (ICSM) have released a new Geodetic datum GDA2020. The datum provides higher positional accuracy and will be able to represent locations dynamically rather than the stationary GDA94. .

In the coming two to three years most organisations will be transforming their spatial data from the GDA94 datum to the new GDA2020 datum. Esri’s, ArcGIS Software has been updated to accommodate these new datums across Australia and will support all of our customers rigorous requirements for locational and positional accuracy.

Continue reading

Important Portal for ArcGIS Security Alert

17 December 2018

Esri has announced that they have discovered a critical security vulnerability in Portal for ArcGIS when specially constructed steps are taken by authenticated users. This results in a privilege escalation issue where the user can elevate themselves to become administrators of the portal.

This issue is present in all supported versions of Portal for ArcGIS, on both Windows and Linux operating systems. Esri has released patches for all versions of Portal for ArcGIS, from version 10.3 through 10.6.1.

Esri have published the following Knowledge Base article relating to this issue: Problem: Warning of security vulnerability in Portal for ArcGIS

Continue reading