Part I – ArcGIS Image Dedicated Capabilities, Components and Configurations
Introduction
ArcGIS Image Dedicated or AID introduces raster analytics and streaming features directly to imagery within your Amazon Web Services or Microsoft Azure cloud setup. This service entails Esri overseeing dedicated servers positioned alongside your imagery, aiming to minimize egress costs and facilitate scalability.
ArcGIS Image Dedicated (AID) features a distinct collection of geoprocessing tools that enable users to publish and manage imagery services.
Unlocking Capabilities with ArcGIS Image Dedicated (AID)
With AID, you can accomplish the following:
- Retain Imagery in Current Location: AID is designed for organizations with imagery stored in cloud repositories. It caters to those required to maintain imagery within specific regions or those aiming to process imagery in proximity to existing storage, thereby reducing egress costs and enabling scalability.
- Specify Cloud Storage: Inform Esri about the location of your imagery and raster collections, indicating whether they reside in Amazon Web Services (AWS) or Microsoft Azure.
- Interface with Image Dedicated: Utilize ArcGIS Pro as the interface to author image services, streamlining the process of managing and manipulating imagery.
- Serve Imagery: Transform your imagery into tiled and dynamic image services compatible with ArcGIS. Additionally, leverage the capability to stream Open Geospatial Consortium (OGC) compliant Web Map Service (WMS) and Web Coverage Service (WCS) endpoints.
- Enable Analysis: Unlock a comprehensive suite of raster functions, tools, templates, and APIs within ArcGIS. This empowers users to undertake sophisticated image analysis projects and develop solutions tailored to their needs.
AID Setup
To utilize AID, an ArcGIS Online organization is mandatory. The administrator of this ArcGIS Online organization can visit the ArcGIS Image Dedicated website to register. Upon the initial login, users will be directed to the user registration page, where the contact details and email information will automatically populate with the corresponding ArcGIS Online organizational account information.
During the registration process, you will be requested to authorize AID to generate an administrative access item in your ArcGIS Online account. This item is essential for managing privileges associated with your organization’s subscriptions to AID. The AID Admin item will appear as an application in the list of items on the Content tab of ArcGIS Online.
Following registration, the organization’s administrator can share full administrator access to all AID services with other members. Access to the designated item should be limited to those requiring comprehensive administrative privileges, while more specific and detailed services will have separate access items. Here’s a screenshot of the home page:
Data Access
There are two primary methods for accessing raster data stored in cloud storage:
- ACS connections
- Raster proxies
ACS (ArcGIS Cloud Storage) connection files in ArcGIS enable the creation of a connection to cloud storage, allowing access to hosted data. Users can utilize the Create Cloud Storage Connection File geoprocessing tool, entering security credentials to store in an encrypted ACS file. Multiple cloud security profiles can be employed from a single machine.
Additionally, the custom AID Service Management geoprocessing toolbox is necessary when using ACS files, available for download from the ArcGIS Image Dedicated website. The toolbox facilitates the reading of raster datasets, such as cloud raster format (CRF), stored in the cloud. To expedite repeated access to rasters, local raster caches can be created by enabling the pixel cache option. Alternatively, raster proxies, small XML files containing information about raster files in cloud storage, can be used. ArcGIS treats raster proxies as virtual rasters, accessing actual data from the cloud as needed. This method caches previously read pixels and tile indices locally for faster pixel requests.
AID Components
AID components are as follow:
- Dynamic Imagery
- Tiled Imagery
- Processing and Analysis
- Pro Machine
- Data Storage Accounts
- Reports
- Downloads
- Documentation
Dynamic Imagery
You can manage and add dynamic imagery servers for large imagery collections that need on-the-fly processing and dynamic mosaicking. To set up a dynamic imagery server follow the steps below:
- Go to the home page of ArcGIS Image Dedicated > Active Servers
- Click + New Server. The following form opens:
Order ID: The ID of the order purchased by the user.
Cloud Provider: The cloud based storage provider that will be used by this server.
Designated Cloud Region: This should be the same region where data is stored. If you are using public data look at the region where the bucket is located and select the same region for the server.
Data Storage Account Alias: The data storage account determines the data bucket, output bucket, and log bucket for the server.
DNS Prefix: This string will be part of the server URL that will be used to access the services (if the string is “xyz” the server url will be https://di-xyz.img.arcgis.com/).
Server Alias: Alias for public access of the server.
You can also add additional comments and turn on the option to create a Server Access itme on ArcGIS Online or Portal for ArcGIS.
c. When the form is filled out, hit Submit Request button. The server will be created with the specifications you provided on the Active Servers list.
Tiled Imagery
You can manage and add tiled imagery servers for individual images, rasters, and tile cache.
To set up a tiled imagery server follow the steps below:
- Go to the home page of ArcGIS Image Dedicated > Active Servers
- Click + New Server. The following form opens:
The form parameters are exactly the same as the dynamic imagery server parameters but it creates the server for the tiled imagery.
c. When the form is filled out, hit Submit Request button. The server will be created with the specifications you provided on the Active Servers list.
Processing and Analysis
Under the processing and analysis subscription, users can activate tasks to process new imagery, generating fresh data representations, and conduct analyses to extract information. These tasks encompass activities such as format conversion, automated creation of mosaic datasets, classification, and deep learning.
Pro Machine
Within the processing and analysis subscription, users have the option to utilize virtual Pro Machines. These machines can be deployed in the same cloud and region as the imagery data, facilitating the publication and management of image services, as well as analysis tasks. Pro Machines are centrally managed through the ArcGIS Image Dedicated website and can be accessed using Remote Desktop. It’s essential to note that users must provide their own ArcGIS Pro license (Basic, Standard, or Advanced) to use ArcGIS Pro on a Pro Machine, and these licenses can include necessary extensions like Image Analyst for comprehensive image analysis workflows.
To set up a Pro Machine:
- Click the Pro Machine tile on the home page.
- Click + New Machine.
- The following form open, populate the parameters.
- Submit the form by clicking the Submit Request.
Cloud Provider: The cloud based storage provider that will be used by this machine.
Designated Cloud Region: Select the same region where your data is located. If you are using public buckets, identify the storage location and select the same.
Instance Type: The instance type determines the computing power and memory of the Pro machine. Choose a type that suits your computing requirements. Click here for more information on instance types.
Apply Template: Templates contain pre-installed components for the Pro machine. Some templates may only be available for certain purposes or instance types. Default template will be applied if left blank.
Additional Disk Storage: Storage space that will be allocated to in the Pro Machine.
Inbound access IP address preferences: Enables RDP connection to the provided IP address. Enter up to 3 IP addresses.
Pro Version: The ArcGIS Pro version that is required for deployment.
Pro Machine Alias: Provide a name to identify your Pro machine.
To access the Pro Machine:
- Go to the ArcGIS Image Dedicated home page.
- Select the Pro Machine tile.
- Select the Cloud Provider in the Active Pro Machines
- Pro machine(s) now created as shown in the screenshot below.
- Before starting the machine, you need to register you IP address. To do that, click the Machine Alias name and enter your IP address in the Inbound Access IP Address in the Update Machine Properties pane and click Update. The tool also includes options of password reset, start/stop machine and delete machine. The update will take around 10 to 15 minutes to complete.
- After IP address update, click the START button in the Star/Stop Machine; this will take up around 5 minutes to start up the machine.
- Use the down arrow button in the Retrieve Credentials to open the Credentials window. Copy the IP address, username, and password or you can download the RDP file and save it on your machine.
- Use the downloaded RDP file to connect to the Pro Machine.
- In case of an issue, you can report it to Esri via the Report an Issue button on the right of the pane.
Control Access to Cloud Storage
AID enables the creation of image services on Esri-managed infrastructure, requiring permissions in the cloud storage infrastructure for server access to read or write data, with current support limited to AWS and Azure.
Access to Data in AWS:
To securely access data in the AWS cloud infrastructure, the process involves setting up;
- an Amazon Resource Name (ARN) through Python scripting or
- AWS CloudFormation templates.
These resources can be obtained by downloading the Create Storage Account zipped folder from the ArcGIS Image Dedicated website. The Python script automates the creation of a secret in AWS Secrets Manager, encrypts it using Amazon’s Key Management Service (KMS), and generates an ARN, which is then shared only with authorized machines. This ARN grants specific access permissions to S3 buckets for reading imagery, writing analysis output, and storing ArcGIS server logs. Users have full control over access permissions and can modify them as needed. Alternatively, the CloudFormation template can be uploaded to the AWS Console for a quick and templatized allocation of services. After successfully implementing the ARN using either method, the “Secret ID” contains the shared secret’s ARN, which can be used to create data storage accounts in ArcGIS Image Dedicated. It’s important to note that the Python script requires Python 3 with boto3 installed, and AWS CLI configured with keys.
Access to Data in Azure:
To secure access to data in the Azure cloud infrastructure, two methods are available:
- access keys
- and shared access signatures (SAS) tokens.
Access keys, authorized through Shared Key authorization, provide full permissions to ArcGIS Image Dedicated servers, allowing operations such as creating, reading, updating, and deleting files, queues, blobs, and tables. On the other hand, SAS tokens are URLs that grant restricted access rights to Azure storage resources. By distributing an SAS URL, access is granted for a specified period, with customized permissions. Once a data storage account request is successful, the Azure storage account key or SAS is encrypted using Advanced Encryption Standard 256, making it usable by ArcGIS Image Dedicated.
Reports
Usage reports for dynamic imagery servers, tile servers, Pro Machines, and other processing and analysis services can be generated through the ArcGIS Image Dedicated website. These reports offer insights into resource utilization and associated costs.
To access the reports, users can select the Reports tile on the website homepage, specify the service of interest, and fill out the form with the necessary details.
Additionally, users can monitor their remaining Image Dedicated credits on the website by clicking on their account name in the main ribbon, selecting the option to view credit usage, and obtaining information on the remaining credits and the last update timestamp.
Downloads
You can use the Downloads tile to download geoprocessing toolboxes to process imagery data associated with the processing and analysis services and manage dynamic and tiled image services. These tools can be added and used in ArcGIS Pro.
Documentation
There are some documents provided by Esri to provide more insights into ArcGIS Image Dedicated and its components which can be accessed via the Documentation tile.
In the next blog, I will go through the Pro machine, connect to cloud, AID specific tools and will show you how to create a dynamic image service.
Written by Mehdi Pira @Esri Australia 2023
Esri Australia Technical Blog 