Smartsheet is a dynamic workspace that empowers teams to manage projects, automate workflows, and rapidly build new solutions. Smartsheet uses spreadsheets, referred to as sheets, as the basis of everything it does, but the difference between Smartsheet and spreadsheet programs like Microsoft Excel or Google Sheets is that Smartsheet has all sorts of collaboration functionality incorporated into it. The screenshot below is a sample Smartsheet data.
In this blog, we will update a point feature layer in ArcGIS Online (AGOL) with a Smartsheet dataset using ArcGIS API for Python. In the process, first we will convert this Smartsheet into a Pandas Data Frame, remove rows with no coordinates and then update the feature layer. It is notable that the data contains longitude and latitude in X and Y columns.
The following screenshot is the feature layer that has already been published as a hosted feature layer from the Smartsheet into AGOL before and after updates. As such both the hosted feature layer and the Smartsheet have the same schema and fields. This Smartsheet gets updated on a weekly basis. Therefore the hosted feature layer needs to be updated as well to reflect the updates on the web map. Using ArcGIS API for Python, this process can be automated.
There are three items that are required before data conversion:
Install Smartsheet library by running this command “pip install smartsheet-python-sdk” in Python Command Prompt (here python is in D drive: “D:\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\envs\arcgispro-py3\Scripts>”)
Smartsheet Access Token – here’s a link which shows how to create one
Sheet ID – to obtain the sheet ID, in Smartsheet go to File > Properties > Sheet ID
1. Smartsheet Conversion into a Data Frame
First import the modules.
Input the access token generated from the Smartsheet in the script below. After the Smartsheet authentication, a function is used to implement the Smartsheet to data frame conversion.
Input the sheet ID copied from the properties into the following script, then we call the function to create the data frame.
The rows with no latitudes or longitudes are removed.
Note that X and Y columns are the longitude and latitude coordinates respectively.
2. Export Data Frame into a CSV
Once the data frame is created, it is output as a temporary csv file in a folder. Here, a TEMP folder in C drive is used.
3. Upload CSV to AGOL
Connect to AGOL, then remove any existing csv item with the same title from AGOL.
When properties are set with title, description, and tags, upload the csv file to the Smartsheet’s folder in AGOL. We can skip the description and tags here and can just use the title since this item is temporary and gets deleted when a new csv is added.
4. Truncate and Append
After that we get the hosted feature layer by id and we truncate it. This hosted feature layer becomes empty but still retains all the properties and settings. Finally, we use the Append method to update this feature layer with the newly added csv file.
The good thing about using Append method is that the feature layer’s id does not change. This means that if the feature layer is already used in a web map or a web application, it will not break.
Now the hosted feature layer updated with the Smartsheet is displayed in the web map.
There are some points to note:
The hosted feature layer has initially been created from the same Smartsheet. Hence the schema, field names and types of both data are the same.
Append is currently only available in AGOL and cannot be used in Portal for ArcGIS.
I did not talk about the data used here as it can be any data as long as it contains coordinates since the emphasis is predominantly on the methodology.
This script can be used in e.g., Windows Task Scheduler to run on a regular basis for automation of the whole process.
Smartsheet integration with AGOL has been made easier and more efficient using ArcGIS API for Python with just a few lines of code. Therefore, when there are some updates in Smartsheet, these updates will automatically occur in the hosted feature layer whenever the script is run. This ensures the web maps and web applications such as Operations Dashboard display the latest features.
Why should you and your organisation start using non-spatial tables in ArcGIS Field Maps. Continue reading to find out how you can add immense value to your organisation by using non-spatial tables with ArcGIS Fields Maps. Your field workers will be thanking you as you are about to make their job (yours as well) a lot easier! Continue reading to learn how!
Why should I use a non-spatial table?
Let’s look at an example of asset management, your department is responsible for maintaining a number electrical assets around the state. You already have access to all of the spatial data for these assets. this includes their location, ID, type, year_installed and owner. Currently, your field workers will inspect the assets every month and fill out a asset inspection via paper form and then when they return to the office manually input the data into a spreadsheet… Not very productive is it? What if there was a way to navigate to the asset and then complete the asset inspection in ArcGIS Field Maps. Well today is your lucky day! keep reading and we will take you through the process from start to finish.
Additionally, you could use a feature class with existing features
Create a new table with the steps above but instead of selecting feature class select Table
At this time please add all desired fields
Right click on your feature class > Manage > Add Global ID’s
Right click on your Table > Manage > Add Global ID’s
Your Geodatabase should now look like this
Step 2: Create a relationship
Now we have to create a relationship between our asset feature class and our Asset_inpsections Table. The reason for this is so that when we complete our asset inspections that will be linked to a specific asset. To create a relationship please follow the below steps
The ArcGIS Developer Subscription provides developers with access to Esri API’s and SDKs, as well as developer tools and services that supplement the process of application development with Esri technology. As with numerous other Esri offerings, the developer subscription is available via a variety of plans, starting with the “Essentials” level. The Essentials developer subscription is obtained by signing up for a developer account at https://developers.arcgis.com/sign-up. – This is a freely available Subscription, and provides the developer account holder with access to the inclusions outlined below.
Beyond the “Essentials” level, there are a further four purchasable ArcGIS Developer Subscription (ADS) plans available, each with increasing inclusions, beyond those included with Essentials. These plans, in order of increasing product inclusions, are : “Builder”, “Professional”, “Premium” and “Enterprise”. Inclusions are cumulative, meaning that each plan will include all entitlements specified in the level beneath it; So, as an example, all Professional inclusions will also be found amongst the Premium inclusions.
The key difference between a free Subscription and a purchased Subscription is that a purchased subscription comes with its own set of core, off-the-shelf Esri products, such as ArcGIS Pro or ArcGIS Enterprise (eligibility for which depends on the purchased level). The inclusion of such core products allows for the setting up of distinct Esri development and testing environments, where various workflow and deployment scenarios may be configured and evaluated.
ArcGIS Developer Subscription Inclusions
The ArcGIS Developer’s pricing page, “Building ArcGIS Solutions” tables the Developer Subscription inclusions, increasing from the “Builder” plan, through to the highest “Enterprise” plan. What though, of the distinction between free Essentials inclusions, and Builder inclusions? – Tabled below is an elaboration of the above “Building ArcGIS Solutions” link; It incorporates inclusions at the Essentials level, therefore providing a clear summary of distinctions between the free and purchased subscription levels.
In general, all ADS plans provide access to all Esri API’s, SDKs and app builders, excepting the ArcGIS Pro and ArcGIS Enterprise SDK’s which a developer would only require, should they already have licenses for ArcGIS Pro or ArcGIS Enterprise. The ArcGIS Developer’s Downloads page is used to access respective API/SDK download links. There is also an evident distinction between the Essentials ADS and the Builder ADS, where a developer seeks to use App Studio and deploy developed apps via the Apple App Store, or Google Play. – This is not possible with an Essentials plan, but is possible for purchased ADS plans, where a “Developer” AppStudio license is provided. Another notable difference, the Essentials subscription includes Esri Community support which comes in the form of access to various online Esri forums. Beyond this, more personal one-on-one support is available via “Esri Technical Support”; This though, is only available to holders of a purchased ADS. Further, all purchased ADS plans include access to a distinct 5 member ArcGIS Online (AGOL) Dev and Test organisation, which exists for the purposes of emulating a production AGOL environment, and testing planned workflows. This is distinct from the single member AGOL account which comes with all developer subscription plans; This single member AGOL organisation could primarily be seen as a web GIS content creation and hosting facility.
In terms of accessing license files for core products provided under an ArcGIS Developer Subscription, Esri’s MyEsri user portal is used by the developer assigned to the subject ADS. Since these inclusions are only applicable to purchased ArcGIS Developer Subscriptions, using MyEsri in this way is not applicable to the Essentials level developer.
The concept of the ArcGIS Developer Subscription, where developers have access to a full suite of Esri product, for “dev and test” purposes at a heavily reduced price, has been around for many years; Some readers may even remember the “Esri Developer Network” which was a similar “predecessor” to the ADS. Esri developers will also likely know, the experience of the ADS holder is fundamentally based around tools and resources found within the ArcGIS Developer’s website : https://developers.arcgis.com. Within the last 12 months though, the release of “ArcGIS Platform” saw the availability of many new capabilities for ArcGIS developers1, and consequently the ArcGIS Developer’s website was extensively overhauled to accommodate these changes. – This is the reason that https://developers.arcgis.com took on a very new appearance in late January of this year, and therefore, that ADS holders (whether holding a free or purchased subscription), had access to a whole a new world of ArcGIS developer-centric services and tooling. 1 :New capabilities that became available to ADS holders when ArcGIS Platform was released are denoted with an * in the below table.
ArcGIS Developer Subscription Inclusions, by Subscription :
Further to the above table, Esri core products (eg. ArcGIS Pro, ArcGIS Enterprise), ArcGIS user type extensions (eg. Utility Network), ArcGIS premium apps (eg. ArcGIS Collector), and extension products for ArcGIS Desktop and ArcGIS Enterprise (eg. Spatial Analyst, GeoEvent Server) are *only* available to holders of purchased ArcGIS Developer Subscriptions, depending on the purchased level, the variation of which is listed here.
The ArcGIS Developer Subscription is accessible to all developers, making available a broad and sophisticated range of location based developer capabilities, APIs and tools. Location aware applications may take many forms – the ArcGIS Developer Subscription, by virtue of its 5 distinct “levels” of product inclusion, seeks to accommodate all these forms, as well as developers from varied backgrounds. If you haven’t already become involved, sign-up, dive in and have a go. It’s never been easier!
This post describes ArcGIS Runtime licensing and the developer’s requirements, for developing and deploying an ArcGIS Runtime application.
If you’ve wondered about building an ArcGIS Runtime app and associated licensing requirements, navigating the abundance of ArcGIS Developer information, can be a little daunting. Coupled with this, is the January 2021 release of ArcGIS Platform, and the associated aspects that it brings to the table.
In order to license and deploy an ArcGIS Runtime application, there are two key items that must be considered : 1. Which ArcGIS Runtime functionality is to be used by the Runtime application ? 2. From where will the ArcGIS Runtime application access data and services ? Foremost though, the developer will need to : 3. Access and download their ArcGIS Runtime SDK of choice.
The remainder of this post describes the above requirements and associated considerations.
1. Access the ArcGIS Runtime SDK
First off, the developer requires access to their required ArcGIS Runtime SDK. To download and freely develop and test against any of the aforementioned ArcGIS Runtime SDKs, the developer needs to have an ArcGIS Developer Subscription (ADS).
Note too, Developer Subscriptions are also available as purchased subscriptions; Irrespective of this, any type of ArcGIS Developer Subscription, free or purchased, will allow you to download and develop with an ArcGIS Runtime SDK *1.
*1 If you would like to know more about ArcGIS Developer Subscriptions, the types that are available, and the entitlements that they include, you may be interested in the further blog post : “The ArcGIS Developer Subscription : Varieties and Inclusions”
2. Which ArcGIS Runtime Functionality is to be Used by the ArcGIS Runtime Application ?
ArcGIS Runtime functionality is available at four different license levels : “Lite”, “Basic”, “Standard” and “Advanced”, as specified at https://developers.arcgis.com/net/license-and-deployment/license/#licensing-capabilities. These levels comprise capabilities that are cumulative, moving from the Lite level up to the Advanced level. The ArcGIS Runtime functionality used by a subject application will govern the required ArcGIS Runtime license level that the developer needs to acquire.
There are 2 ways that the license is granted on application initiation. Either :
via a license code, which is available at a “Lite”, “Basic”, “Standard” and “Advanced” level, and embedded within an application’s initiation code.
via a “Named User” login, for ArcGIS Online organisation members, or ArcGIS Enterprise Portal organisation members.
2.1 License codes
If the application is utilising capabilities at just the “Lite” license level, the developer can obtain a “Lite” license code from the ArcGIS Runtime documentation, available by logging into the ArcGIS Developer’s website. That is, on logging in, go to the applicable ArcGIS Runtime documentation page topic : “License and Deployment” > “License”,and scroll down to the “License String” sub-heading, where the Developer account Lite license string may be accessed, as shown below.
For “Basic”, “Standard” and “Advanced” license codes, developers need to purchase ArcGIS Runtime deployment packs, which are purchased in bundles of “x” deployment licenses (Basic level deployment packs are purchased in bundles of 50 licenses; Standard level packs are purchased in bundles of 25 licenses, and Advanced deployment packs are purchased in bundles of 5 licenses.). Each install of the subject application represents a single deployment. So, for example, if the developer expects that there will be 100 installs for a subject Standard level ArcGIS Runtime app, 4 Standard level deployment packs would need to be purchased.
Licensing via a license string is required when a subject application will be either permanently offline, or offline for periods greater than 30 days.
2.2 Named Users
User types defined within ArcGIS Online organisations and on-premises ArcGIS Enterprise Portal organisations may be used to license ArcGIS Runtime applications. Depending on the user type of organisation members, a varying Runtime license level is available, as defined in the below table. Hence, named users logging into an ArcGIS Runtime application will be able to access coded functionality, corresponding with their user type’s Runtime license level.
3. From where will the ArcGIS Runtime Application access Data and Services ?
In terms of an ArcGIS Runtime application accessing data and services, if the data is hosted and shared publicly, and the service freely available, there is no cost implication.
Alternative to this, the application may be accessing secured services within either an ArcGIS Online organisation, or an on-premises ArcGIS Enterprise Portal organisation. – In this circumstance, members of the “organisation” login to the subject application using their “organisation’s” Named User account credentials. In this way, data and services are accessible via the same sharing constraints as if those members were logged into the subject organisation.
Similarly, if ArcGIS Online premium services are accessed by the application, credits are consumed on a per user basis, attributed against the Named user, and charged as credits to the organisation.
Further to the above, an ArcGIS Developers Subscription (free or purchased) makes available to the developer a suite of freely available data and services, also known as ArcGIS Platform “Location Services“. These include ready-to-use services such as geocoding, routing, basemaps, spatial analysis and data hosting capabilities, thus giving developers a means of accessing spatial service capabilities that is independent of the need for further ArcGIS infrastructure (- an option that will likely be attractive for applications that do not sit atop large ArcGIS implementations).
Location Services are made available via a pay-as-you-go costing model that includes a generous monthly “free tier” of service usage, before charges apply; Permanent access tokens, known as API Keys are used to secure these services. API Keys, Location Services and the costing model are managed via the Developer’s subscription and the ArcGIS Developers website.
A Note About Revenue Generating ArcGIS Runtime Applications
Prior to the January 2021 release of “ArcGIS Platform”, developers writing revenue generating apps were required to have an ArcGIS Developer Subscription that was purchased.
The release of ArcGIS Platform though, saw changes in respect of revenue generating apps and Developer Subscription requirements. Essentially, a prime intent of ArcGIS Platform was to reduce the developer’s requirements in terms of developing and deploying applications written with Esri APIs and SDKs. That being said, ArcGIS Platform removed the requirement for revenue generating apps to have an associated purchased ArcGIS Developers subscription. Instead, a freely obtainable ArcGIS Developer Subscription may be used.
In summary then, there are potentially 2 payment points involved with developing and licensing ArcGIS Runtime applications :
If the application is using beyond the “Lite” level of ArcGIS Runtime capabilities, deployed applications must be licensed via a Basic, Standard or Advanced Runtime license. This license is supplied either as a license code (purchased in a deployment pack) or via a named user account for users that are members of an ArcGIS Online Organisation or Enterprise Portal Organisation. – Named users gain a license to use a Runtime application at the Lite, Basic, Standard or Advanced level, via the user type that is assigned to them.
The license code option is required where the application is required to execute entirely offline, or offline for periods of greater than 30 days.
If the application is using data and services that are accessed via an API Key, managed through the developer’s ArcGIS Developer Account, the developer needs to pay for these data and services via a monthly “pay-as-you-go” model, *when* the free tier of allocated monthly data and services have been consumed. If, however, the application is using ArcGIS Online premium services or data content from an ArcGIS Online organisation or an ArcGIS Enterprise Portal, credits are consumed on a per user basis, as if the application user were working in ArcGIS Online or Portal.
With the integration of 2D Maps and 3D Scenes in ArcGIS Pro, coupled with broader use of high-accuracy GNSS, there is a growing need to consider vertical datums when working with GIS data. This blog seeks to unpack some key ideas around vertical datums within an Australian context, as well as provide some example workflows you might utilise when working with z-enabled data.
When managing authentication with your chosen identity store in Portal for ArcGIS, authentication can be configured at the Portal tier, the Web tier, or the External tier. The primary differences from a user experience perspective being Portal tier requires credentials to be provided to sign in and it does not support a Single Sign On (SSO) experience. Web tier can be configured to utilise the ArcGIS Web Adaptor, Microsoft Internet Information Services (IIS), and Integrated Windows Authentication (IWA) for a SSO user experience.
If using an enterprise identity store with Portal tier authentication, by its nature it requires a closer integration with your Microsoft Active Directory (AD) or Lightweight Directory Access Protocol (LDAP) compatible identity store. When a user enters their credentials to the Portal for ArcGIS sign in page, the credentials are passed to the portal application which then handles the process of authenticating the credentials with the configured enterprise identity store before signing them in.
If using an enterprise identity store with Web tier authentication, the ArcGIS Web Adaptor and IIS configured with IWA enabled handle the authentication of credentials rather than the portal application, enabling a SSO experience. It can also be configured with the Java ArcGIS Web Adaptor and LDAP, for utilisation in a Linux environment for example.
Becoming a popular option, Portal for ArcGIS supports integration with identity providers compatible with Security Assertion Markup Language (SAML) 2.0 authentication, a form of external tier authentication method. Similar to web tier, authentication is handled on a web server but usually an external web server to the ArcGIS Enterprise deployment. Common scenarios include the utilisation of external identity providers such as Microsoft Azure Active Directory, Microsoft Active Directory Federation Services (ADFS), Okta, and others. Comparative user experiences would include utilising a Google, Facebook, Apple, or GitHub account to sign in and access other websites or services such as ArcGIS Online.
SAML is becoming a popular and prevalent security setting for Portal for ArcGIS due to the various security benefits, enabling a seamless Single Sign On (SSO) experience, and the ability to support multiple identity providers (both built-in and enterprise accounts).
Implications of SAML on the Portal Administrator Directory
Let us assume we have set up our Portal for ArcGIS with SAML authentication to an external identity provider. We have our enterprise users and groups configured as per Configure a SAML-compliant identity provider with a portal—Portal for ArcGIS | Documentation for ArcGIS Enterprise. Now we are looking at how we can interact with our enterprise users and groups through the Portal Administrator Directory endpoint. However, attempting to utilise the “Get Enterprise Groups for Users” operation through <portalurl>/webadaptor/portaladmin /security/groups returns error code 500:” An enterprise group may not have been configured or we were unable to connect to it. Please check the logs for more information.”
We also attempt to use a Python script to automate the process of adding SAML users to portal, we are using an administrator account but the Generate Token request to the ArcGIS REST API still returns a 400 error! What could be the reason for all these errors?
Why? Simply, this is expected behaviour by external tier authentication.
When we configure SAML authentication with our enterprise identity provider, login requests to portal are redirected to the external identity provider via SAML. Upon successful login the SAML response is used by Portal for ArcGIS as authorised and trusted credentials to sign in. The external identity provider cannot be integrated via SAML as the Portal for ArcGIS user store and group store configurations.
The “Get Enterprise Groups for Users” and “Get Users Within Enterprise Group” only operate on configured user and group store configurations. These back-end operations only work with portal tier and web tier authentication scenarios. It does not currently support external tier authentication configuration such as SAML.
The following conceptual workflow demonstrates how the login requests are managed between Portal for ArcGIS and SAML (Step 2).
If we explore our Portal for ArcGIS security configuration through <portal url>/webadaptor/portaladmin/security/config we will see the group store configuration in our Portal for ArcGIS identity store are still set to “BUILTIN” type even though we have set up SAML enterprise logins. Basically, Portal for ArcGIS defers users to the SAML identity provider and trusts the assertion provided on behalf of the user. Rather than the username / login being handled by the Portal for ArcGIS built-in identity store, it is handled by the SAML identity provider.
In this scenario our Portal for ArcGIS uses the Microsoft Windows AD as its group store identity. Therefore, the “Get Enterprise Groups for Users” and “Get Users Within Enterprise Group” operations will return a valid response. Step 4 on the following picture demonstrates conceptually how Portal for ArcGIS accesses AD or LDAP identity stores to authenticate the login requests.
Maybe you go to check these default directories and you find that the directory doesn’t exist, or that there aren’t any logs present? You ask yourself, “Am I working with an ArcGIS Enterprise environment that has had logging locations modified from its defaults? What’s the process for figuring out where the logs might be?“
Portal for ArcGIS may be configured for Enterprise logins (eg SAML/Active Directory). An organisation may require their Portal content to be managed based on Active Directory Group membership. The below answers the question “Does Portal automatically create groups to match Active Directory Groups and will users automatically be added to these groups when first logging in to Portal using their Enterprise Logins?”
The answer is yes, we simply need to configure portal groups and bind them to the active directory group using the below steps.
Does Portal inherit Active Directory Configured Membership groups?
Portal for ArcGIS does not automatically create groups to match what is available in Active Directory. The GIS Administrator will need to create and configure groups in Portal for ArcGIS for each of the Active Directory Groups that they want to allow membership of.
How to configure Portal to enable Enterprise Group Membership
User will need to Manually create the Portal Groups then bind them to the Active Directory Group
First, configure the organisation SAML settings to enable SAML based Group Membership. This may be done via Organisations > Settings > Security > Logins > Configure > Advanced Settings
You will then have the ability to create Portal groups with the setting “Enable SAML based group membership”
Here is where you will need to configure the enterprise group name. This name may not be a recognisable name, it may be a group ID or SID. Members will only be added to the group once they have logged in and if there is a group in the SAML assertion response which matches the enterprise group name.
Can SAML/Active Directory users be automatically added to configured Enterprise groups when signing into Portal?
Yes. Once you have configured the Portal Groups and associated them with their respective Active Directory groups you do not need to manage membership of those groups within Portal. When a user logs in with their enterprise account, the groups to which they are members in Active Directory is returned in the SAML response and ArcGIS reflects that by allowing the user membership to the matching groups you have defined.