Updating ArcGIS Online hosted feature layer with Smartsheet using ArcGIS API for Python

Introduction

Smartsheet is a dynamic workspace that empowers teams to manage projects, automate workflows, and rapidly build new solutions. Smartsheet uses spreadsheets, referred to as sheets, as the basis of everything it does, but the difference between Smartsheet and spreadsheet programs like Microsoft Excel or Google Sheets is that Smartsheet has all sorts of collaboration functionality incorporated into it. The screenshot below is a sample Smartsheet data.

In this blog, we will update a point feature layer in ArcGIS Online (AGOL) with a Smartsheet dataset using ArcGIS API for Python. In the process, first we will convert this Smartsheet into a Pandas Data Frame, remove rows with no coordinates and then update the feature layer. It is notable that the data contains longitude and latitude in X and Y columns.

The following screenshot is the feature layer that has already been published as a hosted feature layer from the Smartsheet into AGOL before and after updates. As such both the hosted feature layer and the Smartsheet have the same schema and fields. This Smartsheet gets updated on a weekly basis. Therefore the hosted feature layer needs to be updated as well to reflect the updates on the web map. Using ArcGIS API for Python, this process can be automated.

Requirements

There are three items that are required before data conversion:

  1. Install Smartsheet library by running this command “pip install smartsheet-python-sdk” in Python Command Prompt (here python is in D drive: “D:\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\envs\arcgispro-py3\Scripts>”)
  2. Smartsheet Access Token – here’s a link which shows how to create one
  3. Sheet ID – to obtain the sheet ID, in Smartsheet go to File > Properties > Sheet ID

Methodology

1. Smartsheet Conversion into a Data Frame

First import the modules.

Input the access token generated from the Smartsheet in the script below. After the Smartsheet authentication, a function is used to implement the Smartsheet to data frame conversion.

Input the sheet ID copied from the properties into the following script, then we call the function to create the data frame.

The rows with no latitudes or longitudes are removed.

Note that X and Y columns are the longitude and latitude coordinates respectively.

2. Export Data Frame into a CSV

Once the data frame is created, it is output as a temporary csv file in a folder. Here, a TEMP folder in C drive is used.

3. Upload CSV to AGOL

Connect to AGOL, then remove any existing csv item with the same title from AGOL.

When properties are set with title, description, and tags, upload the csv file to the Smartsheet’s folder in AGOL. We can skip the description and tags here and can just use the title since this item is temporary and gets deleted when a new csv is added.

4. Truncate and Append

After that we get the hosted feature layer by id and we truncate it. This hosted feature layer becomes empty but still retains all the properties and settings. Finally, we use the Append method to update this feature layer with the newly added csv file.

The good thing about using Append method is that the feature layer’s id does not change. This means that if the feature layer is already used in a web map or a web application, it will not break.

Now the hosted feature layer updated with the Smartsheet is displayed in the web map.

There are some points to note:

  • The hosted feature layer has initially been created from the same Smartsheet. Hence the schema, field names and types of both data are the same.
  • Append is currently only available in AGOL and cannot be used in Portal for ArcGIS.
  • I did not talk about the data used here as it can be any data as long as it contains coordinates since the emphasis is predominantly on the methodology.

This script can be used in e.g., Windows Task Scheduler to run on a regular basis for automation of the whole process.

Summary

Smartsheet integration with AGOL has been made easier and more efficient using ArcGIS API for Python with just a few lines of code. Therefore, when there are some updates in Smartsheet, these updates will automatically occur in the hosted feature layer whenever the script is run. This ensures the web maps and web applications such as Operations Dashboard display the latest features.

ArcGIS Field Maps: Getting Started with Non-Spatial Tables

Why should you and your organisation start using non-spatial tables in ArcGIS Field Maps. Continue reading to find out how you can add immense value to your organisation by using non-spatial tables with ArcGIS Fields Maps. Your field workers will be thanking you as you are about to make their job (yours as well) a lot easier! Continue reading to learn how!

Why should I use a non-spatial table?

Let’s look at an example of asset management, your department is responsible for maintaining a number electrical assets around the state. You already have access to all of the spatial data for these assets. this includes their location, ID, type, year_installed and owner. Currently, your field workers will inspect the assets every month and fill out a asset inspection via paper form and then when they return to the office manually input the data into a spreadsheet… Not very productive is it? What if there was a way to navigate to the asset and then complete the asset inspection in ArcGIS Field Maps. Well today is your lucky day! keep reading and we will take you through the process from start to finish.

Step 1: Create a feature class and table

  • Open a new project in ArcGIS Pro
  • Create a new feature class with the steps here
    • At this time please add all desired fields
    • Additionally, you could use a feature class with existing features
  • Create a new table with the steps above but instead of selecting feature class select Table
    • At this time please add all desired fields
  • Right click on your feature class > Manage > Add Global ID’s
  • Right click on your Table > Manage > Add Global ID’s

Your Geodatabase should now look like this

Step 2: Create a relationship

Now we have to create a relationship between our asset feature class and our Asset_inpsections Table. The reason for this is so that when we complete our asset inspections that will be linked to a specific asset. To create a relationship please follow the below steps

  • Open the Geoprocessing tool Create Relationship Class
  • Input the following parameters as seen in the below screen shot;
    • Origin Table: Assets
    • Destination Table: Asset Inspections
    • Output Relationship Class: Asset_inspections_RC
    • Cardinality: One to Many
    • Origin Primary Key: Asset_ID
    • Origin Foreign Key: Asset_ID
  • All other defaults can remain

Step 3: Publish Data to your Portal

Now that we have created a relationship between our Table and feature class we are required to publish them to our Portal as a Web Map. Once published we can view the Web Map in ArcGIS Field Maps!

To publish your Web Map with editable capabilities please follow the below steps

  1. Navigate to the Share tab in the ribbon and select Web Map
  2. Complete the required details as necessary
    • More information on Sharing Web Maps can be found here
  3. For Select a Configuration point to “Copy all data: editable”
  4. Select Configuration in the Share as web Map Pane
  5. Select the Web Feature Layer
  6. Select Properties
    • This will take you back to a very similar Pane (but most importantly different)
  7. Select Configuration
  8. Select the desired type of editing privileges allowed on the web map
    • This can be changed at a later time
  9. Ensure both Enable Sync and Export Data are selected
  10. Select the back button > Map to return to the first Share As Web Map Pane
  11. Select Analyse
    • Resolve any errors as needed
  12. Select Share

Step 4: Starting up ArcGIS Field Maps for the first time

Now that we have successfully Published our Web Map to our Portal we can now access the Web Map via ArcGIS Field Maps.

  1. Download ArcGIS Field Maps from the Apple App Store or Google Play Store
  2. Sign into ArcGIS Field Maps with an account that has permissions to access the Web Map and layers contained
  3. Search for your Web Map Name and select to open

Step 5: Creating assets with ArcGIS Field Maps

Once you have the Web Map open we can go ahead and collect the location of the assets, if we already have the assets location we can jump to the next step!

To collect out please follow the below steps;

  1. Select add point to create the features spatial location
  2. Select the blue circle in the bottom right corner and enter the fields as necessary
  3. Once happy with the inputs select submit
  4. Repeat the process as many times as necessary to collect your assets

Step 6: Undertake asset inspections

Once we have added assets to ArcGIS Field Maps whether that is via manually collecting them in ArcGIS Field Maps or adding an existing layer, we can undertake asset inspections.

To undertake asset inspections please follow the below steps;

  1. Open your asset inspections Web Map
  2. Select the desired feature that you want to collect an asset inspection on
  3. Navigate and select the related table
  4. Select Add to create a new record
  5. Once completed select submit in the top right corner

Please note that you can collect as many records/assets inspections for a single feature as needed

How to Install ArcGIS Pro and ArcGIS for Desktop (ArcMap)

The following blog will explain how to download and install the two flagship ArcGIS applications for desktop:

The following steps require that you have purchased an ArcGIS license with the appropriate authorizations.

Downloading from My.Esri.com:

  1. Head to my.esri.com and sign in using your My Esri credentials
  2. Select: My Organizations
  3. Select: Downloads
  4. For both ArcGIS Pro and ArcGIS for Desktop, you can then proceed to View Downloads and select the latest version of the installation files to proceed.
  5. If you would prefer to download a previously released version, you may find them within the All Versions tab or the Download ISO Files tab. See the below image for reference.

Note: Within the downloads page for both products, you may find other optional extensions, support files, and license manager server installation files. 

You can also download ArcGIS Pro from arcgis.com if you are a member of an active ArcGIS Online Organization

  1. Sign in to arcgis.com
  2. In the top-right corner, select your User Profile and head to My Settings
  3. Select the Licenses tab
  4. Scroll down to find a prompt to Download ArcGIS Pro

For resources on how to license your software, please refer to the following resources:

ArcGIS Pro: Managing licences in ArcGIS Pro | Esri Australia

ArcGIS Desktop: Authorizing your software—ArcMap | Documentation (arcgis.com)

Congratulations, you may now proceed to enjoy using ArcMap or ArcGIS Pro.

ArcGIS Developer Subscriptions : What’s Included, What’s Not?

What is an ArcGIS Developer Subscription ?

The ArcGIS Developer Subscription provides developers with access to Esri API’s and SDKs, as well as developer tools and services that supplement the process of application development with Esri technology. As with numerous other Esri offerings, the developer subscription is available via a variety of plans, starting with the “Essentials” level. The Essentials developer subscription is obtained by signing up for a developer account at https://developers.arcgis.com/sign-up.  – This is a freely available Subscription, and provides the developer account holder with access to the inclusions outlined below.

Beyond the “Essentials” level, there are a further four purchasable ArcGIS Developer Subscription (ADS) plans available, each with increasing  inclusions, beyond those included with Essentials. These plans, in order of increasing  product inclusions, are : “Builder”, “Professional”, “Premium” and “Enterprise”. Inclusions are cumulative, meaning that each plan will include all entitlements specified in the level beneath it; So, as an example, all Professional inclusions will also be found amongst the Premium inclusions.


The key difference between a free Subscription and a purchased Subscription is that a purchased subscription comes with its own set of core, off-the-shelf Esri products, such as ArcGIS Pro or ArcGIS Enterprise (eligibility for which depends on the purchased level). The inclusion of such core products allows for the setting up of distinct Esri development and testing environments, where various workflow and deployment scenarios may be configured and evaluated.

ArcGIS Developer Subscription Inclusions

The ArcGIS Developer’s pricing page, “Building ArcGIS Solutions” tables the Developer Subscription inclusions, increasing from the “Builder” plan, through to the highest “Enterprise” plan. What though, of the distinction between free Essentials inclusions, and Builder inclusions? – Tabled below is an elaboration of the above “Building ArcGIS Solutions” link; It incorporates inclusions at the Essentials level, therefore providing a clear summary of distinctions between the free and purchased subscription levels.

In general, all ADS plans provide access to all Esri API’s, SDKs and app builders, excepting the ArcGIS Pro and ArcGIS Enterprise SDK’s which a developer would only require, should they already have licenses for ArcGIS Pro or ArcGIS Enterprise. The ArcGIS Developer’s Downloads page is used to access respective API/SDK download links.
There is also an evident distinction between the Essentials ADS and the Builder ADS, where a developer seeks to use App Studio and deploy developed apps via the Apple App Store, or Google Play. – This is not possible with an Essentials plan, but is possible for purchased ADS plans, where a “Developer” AppStudio license is provided.
Another notable difference, the Essentials subscription includes Esri Community support which comes in the form of access to various online Esri forums. Beyond this, more personal one-on-one support is available via “Esri Technical Support”; This though, is only available to holders of a purchased ADS.
Further, all purchased ADS plans include access to a distinct 5 member ArcGIS Online (AGOL) Dev and Test organisation, which exists for the purposes of emulating a production AGOL environment, and testing planned workflows. This is distinct from the single member AGOL account which comes with all developer subscription plans; This single member AGOL organisation could primarily be seen as a web GIS content creation and hosting facility.

In terms of accessing license files for core products provided under an ArcGIS Developer Subscription, Esri’s MyEsri user portal is used by the developer assigned to the subject ADS. Since these inclusions are only applicable to purchased ArcGIS Developer Subscriptions, using MyEsri in this way is not applicable to the Essentials level developer.

The concept of the ArcGIS Developer Subscription, where developers have access to a full suite of Esri product, for “dev and test” purposes at a heavily reduced price, has been around for many years; Some readers may even remember the “Esri Developer Network” which was a similar “predecessor” to the ADS. Esri developers will also likely know, the experience of the ADS holder is fundamentally based around tools and resources found within the ArcGIS Developer’s website : https://developers.arcgis.com.
Within the last 12 months though, the release of “ArcGIS Platform” saw the availability of many new capabilities for ArcGIS developers1, and consequently the ArcGIS Developer’s website was extensively overhauled to accommodate these changes. – This is the reason that https://developers.arcgis.com took on a very new appearance in late January of this year, and therefore, that ADS holders (whether holding a free or purchased subscription), had access to a whole a new world of ArcGIS developer-centric services and tooling.
1 : New capabilities that became available to ADS holders when ArcGIS Platform was released are denoted with an * in the below table.

ArcGIS Developer Subscription Inclusions, by Subscription :

Further to the above table, Esri core products (eg. ArcGIS Pro, ArcGIS Enterprise), ArcGIS user type extensions (eg. Utility Network), ArcGIS premium apps (eg. ArcGIS Collector), and extension products for ArcGIS Desktop and ArcGIS Enterprise  (eg. Spatial Analyst, GeoEvent Server) are *only* available to holders of purchased ArcGIS Developer Subscriptions, depending on the purchased level, the variation of which is listed here.

Summary

The ArcGIS Developer Subscription is accessible to all developers, making available a broad and sophisticated range of location based developer capabilities, APIs and tools. Location aware applications may take many forms – the ArcGIS Developer Subscription, by virtue of its 5 distinct “levels” of product inclusion, seeks to accommodate all these forms, as well as developers from varied backgrounds. If you haven’t already become involved, sign-up, dive in and have a go. It’s never been easier!

Developing & Licensing ArcGIS Runtime Applications, Demystifying the Requirements

This post describes ArcGIS Runtime licensing and the developer’s requirements, for developing and deploying an ArcGIS Runtime application.

If you’ve wondered about building an ArcGIS Runtime app and associated licensing requirements, navigating the abundance of ArcGIS Developer information, can be a little daunting. Coupled with this, is the January 2021 release of ArcGIS Platform, and the associated aspects that it brings to the table.

If you are new to Esri technology, the first thing you need to understand is what ArcGIS Runtime is. Simply put, ArcGIS Runtime is an Esri SDK product that allows developers to build standalone native applications for a variety of platforms and devices. As such, there are multiple different types of ArcGIS Runtime SDK available :
ArcGIS Runtime SDK for .NET
ArcGIS Runtime SDK for Android
ArcGIS Runtime SDK for iOS
ArcGIS Runtime SDK for Java
ArcGIS Runtime SDK for Qt
Depending on the platform that the developed application is to run on, the developer works with the corresponding Runtime SDK.

In order to license and deploy an ArcGIS Runtime application, there are two key items that must be considered :
1. Which ArcGIS Runtime functionality is to be used by the Runtime application ?
2. From where will the ArcGIS Runtime application access data and services ?
Foremost though, the developer will need to :
3. Access and download their ArcGIS Runtime SDK of choice.

The remainder of this post describes the above requirements and associated considerations. 

1. Access the ArcGIS Runtime SDK

First off, the developer requires access to their required ArcGIS Runtime SDK. To download and freely develop and test against any of the aforementioned ArcGIS Runtime SDKs, the developer needs to have an ArcGIS Developer Subscription (ADS).

A developer can sign up for a freely available ADS at https://developers.arcgis.com/sign-up/. They then gain access to https://developers.arcgis.com via their Developer account, and may subsequently download the required ArcGIS Runtime SDK, by navigating to the “Downloads” tab, from the Developers “Dashboard” page.

Note too, Developer Subscriptions are also available as purchased subscriptions; Irrespective of this, any type of ArcGIS Developer Subscription, free or purchased, will allow you to download and develop with an ArcGIS Runtime SDK *1.

*1 If you would like to know more about ArcGIS Developer Subscriptions, the types that are available, and the entitlements that they include, you may be interested in the further blog post : “The ArcGIS Developer Subscription : Varieties and Inclusions”

2. Which ArcGIS Runtime Functionality is to be Used by the ArcGIS Runtime Application ?

ArcGIS Runtime functionality is available at four different license levels : “Lite”, “Basic”, “Standard” and “Advanced”, as specified at https://developers.arcgis.com/net/license-and-deployment/license/#licensing-capabilities. These levels comprise capabilities that are cumulative, moving from the Lite level up to the Advanced level. The ArcGIS Runtime functionality used by a subject application will govern the required ArcGIS Runtime license level that the developer needs to acquire.

There are 2 ways that the license is granted on application initiation. Either :

  • via a license code, which is available at a “Lite”, “Basic”, “Standard” and “Advanced” level, and embedded within an application’s initiation code.

or

  • via a “Named User” login, for ArcGIS Online organisation members,  or ArcGIS Enterprise Portal organisation members.

2.1 License codes

If the application is utilising capabilities at just the “Lite” license level, the developer can obtain a “Lite” license code from the ArcGIS Runtime documentation, available by logging into the ArcGIS Developer’s website. That is, on logging in, go to the applicable ArcGIS Runtime documentation page topic : “License and Deployment” > “License”,and scroll down to the “License String” sub-heading, where the Developer account Lite license string may be accessed, as shown below.

Example link for iOS :
https://developers.arcgis.com/ios/license-and-deployment/license/#license-string
Example link for .NET :
https://developers.arcgis.com/net/license-and-deployment/license/#license-string
The “Lite” license level is free of charge, and may be used to deploy an unlimited number of ArcGIS Runtime applications.

For “Basic”, “Standard” and “Advanced” license codes, developers need to purchase ArcGIS Runtime deployment packs, which are purchased in bundles of “x” deployment licenses (Basic level deployment packs are purchased in bundles of 50 licenses; Standard level packs are purchased in bundles of 25 licenses, and Advanced deployment packs are purchased in bundles of 5 licenses.).
Each install of the subject application represents a single deployment. So, for example, if the developer expects that there will be 100 installs for a subject Standard level ArcGIS Runtime app, 4 Standard level deployment packs would need to be purchased.

Licensing via a license string is required when a subject application will be either permanently offline, or offline for periods greater than 30 days.

2.2 Named Users

User types defined within ArcGIS Online organisations and on-premises ArcGIS Enterprise Portal organisations may be used to license ArcGIS Runtime applications. Depending on the user type of organisation members, a varying Runtime license level is available, as defined in the below table.
Hence, named users logging into an ArcGIS Runtime application will be able to access coded functionality, corresponding with their user type’s Runtime license level.

Source : https://enterprise.arcgis.com/en/portal/latest/administer/windows/licensing-arcgis-runtime-through-portal.htm

3. From where will the ArcGIS Runtime Application access Data and Services ?

In terms of an ArcGIS Runtime application accessing data and services, if the data is hosted and shared publicly, and the service freely available, there is no cost implication.

Alternative to this, the application may be accessing secured services within either an ArcGIS Online organisation, or an on-premises ArcGIS Enterprise Portal organisation. – In this circumstance, members of the “organisation” login to the subject application using their “organisation’s” Named User account credentials. In this way, data and services are accessible via the same sharing constraints as if those members were logged into the subject organisation.

Similarly, if ArcGIS Online premium services are accessed by the application, credits are consumed on a per user basis, attributed against the Named user, and charged as credits to the organisation.

Further to the above, an ArcGIS Developers Subscription (free or purchased) makes available to the developer a suite of freely available data and services, also known as ArcGIS Platform “Location Services. These include ready-to-use services such as geocoding, routing, basemaps, spatial analysis and data hosting capabilities, thus giving developers a means of accessing spatial service capabilities that is independent of the need for further ArcGIS infrastructure (- an option that will likely be attractive for applications that do not sit atop large ArcGIS implementations).

Location Services are made available via a pay-as-you-go costing model that includes a generous monthly “free tier” of service usage, before charges apply; Permanent access tokens, known as API Keys are used to secure these services. API Keys, Location Services and the costing model are managed via the Developer’s subscription and the ArcGIS Developers website.

A Note About Revenue Generating ArcGIS Runtime Applications

Prior to the January 2021 release of “ArcGIS Platform”, developers writing revenue generating apps were required to have an ArcGIS Developer Subscription that was purchased.

The release of ArcGIS Platform though, saw changes in respect of revenue generating apps and Developer Subscription requirements. Essentially, a prime intent of ArcGIS Platform was to reduce the developer’s requirements in terms of developing and deploying applications written with Esri APIs and SDKs. That being said, ArcGIS Platform removed the requirement for revenue generating apps to have an associated purchased ArcGIS Developers subscription. Instead, a freely obtainable ArcGIS Developer Subscription may be used.

Summary

In summary then, there are potentially 2 payment points involved with developing and licensing ArcGIS Runtime applications :

If the application is using beyond the “Lite” level of ArcGIS Runtime capabilities, deployed applications must be licensed via a Basic, Standard or Advanced Runtime license. This license is supplied either  as a license code (purchased in a deployment pack) or via a named user account for users that are members of an ArcGIS Online Organisation or Enterprise Portal Organisation. – Named users gain a license to use a Runtime application at the Lite, Basic, Standard or Advanced level, via the user type that is assigned to them.

The license code option is required where the application is required to execute entirely offline, or offline for periods of greater than 30 days.
 

If the application is using data and services that are accessed via an API Key, managed through the developer’s ArcGIS Developer Account, the developer needs to pay for these data and services via a monthly “pay-as-you-go” model, *when* the free tier of allocated monthly data and services have been consumed. If, however, the application is using ArcGIS Online premium services or data content from an ArcGIS Online organisation or an ArcGIS Enterprise Portal, credits are consumed on a per user basis, as if the application user were working in ArcGIS Online or Portal.

Going vertical in ArcGIS

With the integration of 2D Maps and 3D Scenes in ArcGIS Pro, coupled with broader use of high-accuracy GNSS, there is a growing need to consider vertical datums when working with GIS data. This blog seeks to unpack some key ideas around vertical datums within an Australian context, as well as provide some example workflows you might utilise when working with z-enabled data.

Continue reading

Working with Portal for ArcGIS authentication: Enterprise groups, the Portal Administrator Directory, and SAML

A brief background

When managing authentication with your chosen identity store in Portal for ArcGIS, authentication can be configured at the Portal tier, the Web tier, or the External tier. The primary differences from a user experience perspective being Portal tier requires credentials to be provided to sign in and it does not support a Single Sign On (SSO) experience. Web tier can be configured to utilise the ArcGIS Web Adaptor, Microsoft Internet Information Services (IIS), and Integrated Windows Authentication (IWA) for a SSO user experience.

If using an enterprise identity store with Portal tier authentication, by its nature it requires a closer integration with your Microsoft Active Directory (AD) or Lightweight Directory Access Protocol (LDAP) compatible identity store. When a user enters their credentials to the Portal for ArcGIS sign in page, the credentials are passed to the portal application which then handles the process of authenticating the credentials with the configured enterprise identity store before signing them in.

If using an enterprise identity store with Web tier authentication, the ArcGIS Web Adaptor and IIS configured with IWA enabled handle the authentication of credentials rather than the portal application, enabling a SSO experience. It can also be configured with the Java ArcGIS Web Adaptor and LDAP, for utilisation in a Linux environment for example.

Becoming a popular option, Portal for ArcGIS supports integration with identity providers compatible with Security Assertion Markup Language (SAML) 2.0 authentication, a form of external tier authentication method. Similar to web tier, authentication is handled on a web server but usually an external web server to the ArcGIS Enterprise deployment. Common scenarios include the utilisation of external identity providers such as Microsoft Azure Active Directory, Microsoft Active Directory Federation Services (ADFS), Okta, and others. Comparative user experiences would include utilising a Google, Facebook, Apple, or GitHub account to sign in and access other websites or services such as ArcGIS Online.

SAML is becoming a popular and prevalent security setting for Portal for ArcGIS due to the various security benefits, enabling a seamless Single Sign On (SSO) experience, and the ability to support multiple identity providers (both built-in and enterprise accounts).

Implications of SAML on the Portal Administrator Directory

Let us assume we have set up our Portal for ArcGIS with SAML authentication to an external identity provider. We have our enterprise users and groups configured as per Configure a SAML-compliant identity provider with a portal—Portal for ArcGIS | Documentation for ArcGIS Enterprise. Now we are looking at how we can interact with our enterprise users and groups through the Portal Administrator Directory endpoint. However, attempting to utilise the “Get Enterprise Groups for Users” operation through <portalurl>/webadaptor/portaladmin /security/groups returns error code 500: ” An enterprise group may not have been configured or we were unable to connect to it. Please check the logs for more information.”

We also attempt to use a Python script to automate the process of adding SAML users to portal, we are using an administrator account but the Generate Token request to the ArcGIS REST API still returns a 400 error! What could be the reason for all these errors?

Why? Simply, this is expected behaviour by external tier authentication.

When we configure SAML authentication with our enterprise identity provider, login requests to portal are redirected to the external identity provider via SAML. Upon successful login the SAML response is used by Portal for ArcGIS as authorised and trusted credentials to sign in. The external identity provider cannot be integrated via SAML as the Portal for ArcGIS user store and group store configurations.

The “Get Enterprise Groups for Users” and “Get Users Within Enterprise Group” only operate on configured user and group store configurations. These back-end operations only work with portal tier and web tier authentication scenarios. It does not currently support external tier authentication configuration such as SAML.


The following conceptual workflow demonstrates how the login requests are managed between Portal for ArcGIS and SAML (Step 2).

Ref: ArcGIS Server and Portal for ArcGIS: An Introduction to Security (esri.com)

If we explore our Portal for ArcGIS security configuration through <portal url>/webadaptor/portaladmin/security/config we will see the group store configuration in our Portal for ArcGIS identity store are still set to “BUILTIN” type even though we have set up SAML enterprise logins.  Basically, Portal for ArcGIS defers users to the SAML identity provider and trusts the assertion provided on behalf of the user. Rather than the username / login being handled by the Portal for ArcGIS built-in identity store, it is handled by the SAML identity provider.

Groups and Users operation on the Portal Administrator Directory endpoint are designed to interact with enterprise authentication at the web tier and portal tier, where we need to update our Portal for ArcGIS identity store with LDAP or AD users and groups. With this configuration, we will need to update group store configuration directly, using the JSON text mentioned in the following link: Use your portal with LDAP or Active Directory and portal-tier authentication—Portal for ArcGIS | Documentation for ArcGIS Enterprise.


In this scenario our Portal for ArcGIS uses the Microsoft Windows AD as its group store identity. Therefore, the “Get Enterprise Groups for Users” and “Get Users Within Enterprise Group” operations will return a valid response. Step 4 on the following picture demonstrates conceptually how Portal for ArcGIS accesses AD or LDAP identity stores to authenticate the login requests.

Authors: Razi Mosadeghi and Gregory Hibbett

Reference and Relevant resources:

Portal authentication configuration—Portal for ArcGIS | Documentation for ArcGIS Enterprise

Configure a SAML-compliant identity provider with a portal—Portal for ArcGIS | Documentation for ArcGIS Enterprise

Create groups—Portal for ArcGIS | Documentation for ArcGIS Enterprise

Use your portal with LDAP or Active Directory and portal-tier authentication—Portal for ArcGIS | Documentation for ArcGIS Enterprise

Configure Azure Active Directory—Portal for ArcGIS | Documentation for ArcGIS Enterprise

Tutorial: Azure Active Directory integration with ArcGIS Enterprise | Microsoft Docs

Does Portal inherit AD configured membership groups and will users be added to these groups upon login to Portal with their Enterprise Login? | Esri Australia Technical Blog (wordpress.com)

Using PowerShell to automatically find and package log files for ArcGIS Server, Portal for ArcGIS, and ArcGIS DataStore.

Have you ever submitted a support case and, as a part of the troubleshooting process, been asked to provide logs for one (or multiple) components of a base ArcGIS Enterprise deployment?

Esri provides documentation on where the default logging location is for various ArcGIS Enterprise components, but this is a user-configurable path.

Default Logging Locations

  • ArcGIS Server
    • C:\arcgisserver\logs – as per here.
  • Portal for ArcGIS
    • C:\arcgisportal\logs – as per here.
  • ArcGIS DataStore
    • C:\arcgisdatastore\logs – as per here.

Maybe you go to check these default directories and you find that the directory doesn’t exist, or that there aren’t any logs present? You ask yourself, “Am I working with an ArcGIS Enterprise environment that has had logging locations modified from its defaults? What’s the process for figuring out where the logs might be?

Continue reading

Esri Basemaps – October update for Tasmania (Part 1)

Thanks to Land Tasmania, we have started to bring some of their authoratative datasets (theList) into the Esri Basemaps.

This first iteration included:

  • National Parks
  • State Reserves
  • Forests
  • Ioenspaces
  • Local Parks
  • Sports Fields
  • Tennis Courts/Hardcourts
  • Golf Courses
  • Building Footprints
  • Refresh of the Parcels

A second iteration will focus in on additional layers such as the coastline, estuaries, hydrographic features and more. This will come at a later time.

Take a look at the Esri Basemap and do some zooming around Tasmania to see the new content. These basemaps support offline workflows if you require these basemaps in the field.

Continue reading

Does Portal inherit AD configured membership groups and will users be added to these groups upon login to Portal with their Enterprise Login?

Portal for ArcGIS may be configured for Enterprise logins (eg SAML/Active Directory).   An organisation may require their Portal content to be managed based on Active Directory Group membership.  The below answers the question “Does Portal automatically create groups to match Active Directory Groups and will users automatically be added to these groups when first logging in to Portal using their Enterprise Logins?”

The answer is yes, we simply need to configure portal groups and bind them to  the active directory group using the below steps.

Does Portal inherit Active Directory Configured Membership groups?
 
Portal for ArcGIS does not automatically create groups to match what is available in Active Directory.  The GIS Administrator will need to create and configure groups in Portal for ArcGIS for each of the Active Directory Groups that they want to allow membership of.
 
How to configure Portal to enable Enterprise Group Membership

User will need to Manually create the Portal Groups then bind them to the Active Directory Group

First, configure the organisation  SAML settings to enable SAML based Group Membership. This may be done via Organisations > Settings > Security >  Logins > Configure > Advanced Settings



You will then have the ability to create Portal groups with the setting “Enable SAML based group membership”


Here is where you will need to configure the enterprise group name.  This name may not be a recognisable name, it may be a group ID or SID.  Members will only be added to the group once they have logged in and if there is a group in the SAML assertion response which matches the enterprise group name.  



Can SAML/Active Directory users be automatically added to configured Enterprise groups when signing into Portal?
 
Yes.  Once you have configured the Portal Groups and associated them with their respective Active Directory  groups you do not need to manage membership of those groups within Portal.  When a user logs in with  their enterprise account, the groups to which they are members in Active Directory is returned in the SAML response and ArcGIS reflects that by allowing the user membership to the matching groups you have defined. 

Relevant resources

Use your portal with LDAP or Active Directory and portal-tier authentication https://enterprise.arcgis.com/en/portal/latest/administer/windows/use-your-portal-with-ldap-and-portal-tier-authentication.htm

Link enterprise groups from an IDP https://enterprise.arcgis.com/en/portal/latest/administer/windows/create-groups.htm#ESRI_SECTION1_5E3FFFAA1B7E443FBB1E483E070B1979