Determining the interdependencies of items in Portal for ArcGIS


With almost any Portal there comes a time when you have dozens of items, created over time by different people, and you may be looking to perform a bit of a clean-up.

The problem now arises that you do not know if the Web Map made by a colleague, who since left the organisation, can be deleted or if it is being used by any other items? You might find that deleting an insignificant-looking item brings your organisation’s most-used application down.

The best approach is to avoid ending up in this situation, by making sure your organisation has good conventions in place to leverage the following to keep your content structured and clean:

  • Tags
  • Categories
  • Delete Protection
  • Mark data as authoritative
  • Mark data as deprecated
  • Naming conventions
  • Folders
  • Groups
  • Thumbnails

Unfortunately, if you are already in the scenario of having dozens or hundreds of items, it can be very difficult to make sense of which items relate to others. The below Python script will assist in extracting the dependencies between items from your Portal and present them in an interactive HTML diagram. The script is written for Python 3.7.

It makes use of the following libraries that you may need to install into your Python environment:

  1. NetworkX
  2. MatPlotLib
  3. PyVis

To install these packages, use the pip installer to add them to your environment:

  1. Open command prompt and enter cd C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3
  2. Run pip network
  3. Run pip matplotlib
  4. Run pip pyvis

Once these have been installed, copy the below script into your Python IDE of choice.

Modify the line

source = GIS(r’https://<machine fqdn>/<web adaptor name>’, ‘<portaladmin login>’, ‘<portaladmin password>’)

to reflect your Portal’s url and Administrator login. Administrator level is required to be able to list items that are owned by other users. Currently, ArcGIS Online does not support the item.dependent_upon() function which is required by this script.

Now when you run the script (updated 8 Sept 2021 for refactoring and adding arrows), you should get an output like below which graphically shows the different items in your Portal and their relationships to each other.

import arcpy
import requests

import pandas as pd
import networkx as nx
import matplotlib.pyplot as plt

from matplotlib.pyplot import figure
from urllib3.exceptions import InsecureRequestWarning
from pyvis.network import Network
from pyvis.options import Options

# Suppress only the single warning from urllib3 needed.
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)

from arcgis.gis import GIS
from IPython.display import display

def addDependency(item, dependency, pdArray, includeUrlDependencies= False):
    guidMap[item.itemid] = str.format("{2} \r\n {0} \r\n {1}", item["title"], item.itemid, item["type"])
    if dependency['dependencyType'] == 'id':
        pdArray.loc[len(pdArray)] = [item.itemid,dependency['id']]
    elif (includeUrlDependencies == True and dependency['dependencyType'] == 'url'):
        pdSource.loc[len(pdSource)] = [item.itemid,dependency['url']]



source = GIS(r'https://<fqdn>/<webAdaptorName>/', 'portaladmin', '<password>')

pdSource = pd.DataFrame(columns=['source','dependent_on'])
guidMap = {}

source_users = source.users.search(query='', sort_field='username', sort_order='asc', max_users=10000, outside_org=False, exclude_system=True)

arcpy.AddMessage(source_users)
print('User list: Role')
for user in source_users:
    print(user.username + "\t:\t" + str(user.role))
    arcpy.AddMessage(user.username + "\t:\t" + str(user.role))
for user in source_users:
    num_items = 0
    num_folders = 0
    print("Collecting item ids for {}".format(user.username))
    arcpy.AddMessage("Collecting item ids for {}".format(user.username))

    user_content = user.items()
    source_items_by_id = {}

    # Get item ids from root folder first
    for item in user_content:
        print(item.itemid)
        #guidMap[item.itemid] = str.format("{2} \r\n {0} \r\n {1}", item["title"], item.itemid, item["type"])
        source_items_by_id[item.itemid] = item
        if (item.dependent_upon()['total'] > 0):
            print("\n\n Item: \t {2} \t\t\t id {0} is dependent on these items: \t {1}".format(item.itemid,item.dependent_upon(), item.title))
            for dependency in item.dependent_upon()['list']:
                addDependency(item, dependency, pdSource, False)

    # Get item ids from each of the folders next
    folders = user.folders
    for folder in folders:
        num_folders += 1
        folder_items = user.items(folder=folder['title'])
        for item in folder_items:
            num_items += 1
            #guidMap[item.itemid] = str.format("{0} ({1})", item["title"], item.itemid)
            source_items_by_id[item.itemid] = item
            if (item.dependent_upon()['total'] > 0):
                print("\n\n Item: \t {2} \t\t\t id {0} is dependent on these items: \t {1}".format(item.itemid,item.dependent_upon(), item.title))
                for dependency in item.dependent_upon()['list']:
                    addDependency(item, dependency, pdSource, False)


    print("Number of folders for {} {} # Number of items checked {} ".format(user.username,str(num_folders), str(num_items)))

print(pdSource)

figure(figsize=(10,8))

G = nx.DiGraph()
G = nx.from_pandas_edgelist(pdSource, 'source', 'dependent_on', create_using=nx.DiGraph)
G = nx.relabel_nodes(G, guidMap)

nx.draw_networkx(G)

import json

options = {
  "edges": {
    "arrows": {
      "to": {
        "enabled": True
      }
    },
    "color": {
      "inherit": True
    },
    "font": {
      "size": 8
    },
    "smooth": False
  },
  "interaction": {
    "hover": True
  },
  "manipulation": {
    "enabled": False,
    "initiallyActive": False
  },
  "physics": {
    "barnesHut": {
      "gravitationalConstant": -20700,
      "springLength": 110,
      "avoidOverlap": 0.4
    },
    "minVelocity": 0.75
  }
}

net = Network(height="100%", width="100%")
net.set_options(json.dumps(options))
#net.show_buttons(True)
net.from_nx(G)
net.show("Dependencies.html")

5 thoughts on “Determining the interdependencies of items in Portal for ArcGIS

  1. Chelsea

    Looks like an awesome script! Will be super helpful for us. I’m trying to get it to work, but getting “IndexError: list index out of range” at the line “figure(figsize=(10,8))”. Do you know how to fix this?

    Reply
      1. Chelsea

        Thanks for the suggestion. I also had to comment out the line “nx.draw_networkx(G)” as it was then throwing the same index out of range error. After that, though, it worked! Thanks!

  2. Jesper

    HI, I see many dependencies with only th UUID and no name. I assume it is the datasources, do you have an idea why?

    Reply
    1. Eduard B Post author

      Yes, it might be that other item types are picked up which do not have those same attributes. I only exclude url dependencies because they tend to all have AGOL services in common which makes for a very confusing graph.

      Reply

Got something to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s