Optimise ArcGIS Enterprise to publish large data packages

1 Reply

With the increasing quality of high-resolution imagery, 3D data, and large amounts of geographic information being developed, there are several tweaks and optimisations to consider when publishing large datasets into ArcGIS Enterprise.

We will explore hosted scene layers in particular, as scene layer packages (slpk) are able to easily accumulate to exceptionally large file sizes.

A scene layer is a type of layer that is optimised for displaying large amounts of 3D data in a scene. Visualising your information in 3D makes it easier to intuitively understand and experience geographic information in a realistic way. Even audiences unfamiliar with GIS can understand complex and detailed data by viewing it in a 3D scene on the web.

Any package over 90GB will be considered large, as this is a typical threshold for when your ArcGIS Enterprise settings may require optimisation to maintain reliability for publishing services. However, your mileage may vary. Failure to optimise ArcGIS Enterprise settings for large packages may result in errors during publishing, which may come in the form of one or more of the following:

  • An error has occurred
  • File not found
  • Token invalid
  • Failed to publish web layer. Failed to execute (SharePackage)

The optimisations and considerations that we will explore can be broken down into the following two categories:

  1. Timeouts
  2. Allocate sufficient disk space

Timeouts

This section explores several ArcGIS Enterprise system parameters and is split into the following three subsections:

  • Increase Token Expiration
  • PublishingTools Timeout
  • PublishingTools service instances

Increase Token Expiration

If your publishing jobs happen to fail at the 12-hour mark, there is a token expiration parameter in ArcGIS Enterprise that can be adjusted. Follow the workflow below to increase the token expiration timer:

  1. On your Portal for ArcGIS machine, browse to Program Files\ArcGIS\Portal\framework\webapps\arcgis#sharing\WEB-INF\classes\resources
  2. Open gw-config.properties with a text editor
  3. Search for the parameter config.pub-token-expirations-mins and increase the value. The default value is 720 minutes (12 hours). This value will depend on the size of your file, but as an example, you could try 5000 minutes (>3 days). No adverse effects around system stability or performance have been reported even when increasing this value to over 7 days (>10000 minutes).

PublishingTools Timeout

PublishingTools is a system geoprocessing service utilised by ArcGIS Enterprise to publish services. Sometimes, the timeout value needs to be increased to cater for larger data packages. Follow the workflow below to increase the PublishingTools timeout value:

  1. Navigate to Server Manager > Services > System > PublishingTools > Pooling
  2. Increase the maximum time a client can use a service. The unit of measurement for this value is in seconds. For consistency, you can match this value with the configured value for config.pub-token-expiration-mins mentioned in the previous section. i.e. If token expiration is 5000 minutes, set maximum time client can use a service to 300000 seconds. As with the token expiration parameter, no adverse effects around system stability or performance have been reported even when raising this to higher values.

PublishingTools service instances

To process service requests such as panning/zooming around a map or publishing layers to your Enterprise Portal, the ArcGIS Server machine uses an instance of the ESRI server process, ArcSOC.exe. Refer to the following documentation for more information about ArcSOC.exe:

A publishingTools ArcSOC.exe can only process one publishing job request at a time. Publishing large packages may engage the publishingTools ArcSOC.exe process for a long period of time. Therefore, you may need to consider increasing the maximum number of ArcSOC.exe instances available in order to be able to process other publishing jobs simultaneously.

Be wary when tweaking this setting, as your ArcGIS Server must be able to accommodate the increased utilisation of CPU and memory resources. Each ArcSOC.exe instance typically consumes 100-200MB of memory. General practice dictates that your server machine’s resources should remain under 75% load to avoid resource contention issues. Your server machine resource usage should be monitored before and after tweaking this setting. Follow the workflow below to increase the maximum number of ArcSOC.exe instances for PublishingTools:

  1. Navigate to Server Manager > Services > System > PublishingTools > Pooling
  2. Increase Maximum number of instances per machine value
  3. Our general recommendation is limiting the maximum instances to n + 1, where n is the number of server cores. However, this value can be increased or decreased as needed. Monitor the stability and performance of your Server machine(s) and ArcGIS Enterprise deployment after adjusting this parameter.

Allocate sufficient disk space

The publishing process in ArcGIS Enterprise makes use of temporary files and storage before storing the final data in your Portal and Data Store content directories. These temporary files are typically not immediately cleared during or in between publishing jobs, thus need to be accounted for when calculating required available disk space. There are 3 main storage locations to look at when publishing large hosted scene layers:

  • ArcGIS Server arcgisjobs directory
  • Portal for ArcGIS content directory
  • ArcGIS Data Store tile cache data directory

ArcGIS Server arcgisjobs directory

ArcGIS Enterprise will utilise this directory to temporarily store files needed by the publishing process. For further information about this directory and the automated server directory cleaning process, refer to the document below:

The available disk space for this directory should be at least 2 times the original .slpk file size. The location of the arcgisjobs directory can be found by following the workflow below:

  1. Navigate to Server Manager > Site > Directories
  2. Locate the Jobs directory

Portal for ArcGIS content directory

ArcGIS Enterprise utilises this directory to store content saved and uploaded by users of your Portal. Depending on the workflow used when publishing hosted scene layers, a copy of the slpk file may be stored in this directory.

The available disk space for this directory should be at least 2 times the original .slpk file size. The location of the Portal content directory can be found by following the workflow below:

  1. Navigate to Portaladmin > System > Directories > content
  2. Locate the path for content directory

ArcGIS Data Store tile cache data directory

ArcGIS Enterprise utilises this directory to store your hosted tile cache data.

The available disk space for this directory should be at least 3 times the original .slpk file size. The location of the Data Store tile cache data directory can be found by following the workflow below:

  1. On the ArcGIS tile cache Data Store machine, open an elevated Command Prompt
  2. Change directory to DataStore tools (cd “C:\Program Files\ArcGIS\DataStore\tools”)
  3. Run describedatastore tool.
  4. Locate Data location in the Information for tile cache datastore section.

If you have a single-machine deployment of ArcGIS Enterprise with all the above storage locations sharing the same disk drive, you will effectively require at least 7 times the original .slpk file size (not including the original .slpk file itself) in available disk space to maintain system reliability when publishing a large hosted scene layer.

Related Links

1 thought on “Optimise ArcGIS Enterprise to publish large data packages”

Got something to say?