Bringing Your Data from the Cloud Back to Earth – How to extract your data from ArcGIS Online.


In my blog post series so far this year, I have considered and written up common workflows for preparing data for ‘in field’ collection capture using the Collector for ArcGIS app.

If you’ve missed any of these blogs, you can find them using the Collector tag on this page.

So what’s the next step? Well, now that you’ve published the data, and used the Collector app to collect new features and you’re back at the office, you’d probably like to get the data from the “cloud” and do some good old editing or analysis in ArcGIS for Desktop.

This brings us to my current blog post! Here I’ll cover a few simple techniques that you may use to extract the data from a feature service that’s running on ArcGIS Online, and use it in ArcGIS Desktop. To illustrate this workflow I will use the same feature service representing traffic accidents that I’ve used to demonstrate the Collector’s “offline editing” workflows.

Workflow #1.: Extract the data in a form of a shapefile or a CSV table.

ArcGIS Online for Organisations allows you to extract the data that you’ve collected using the Collector App as shapefile or a CSV file.

Steps:

1. Log in to ArcGIS Online for Organisations and navigate to MyContents

2. Locate the feature service that stores the points collected by the mobile app and open its properties (left-click on the name of the feature service).

01_AGOL2DESKTP

 

3. Click the arrow next to the layer that you’d like to export and choose the output format

02_AGOL2DESKTP

The resulting shapefile will be saved to the My Content folder (root). You can click the drop-down arrow and download it.

03_AGOL2DESKTP

Now we’ve got our data from the cloud. There are, however, some shortcomings to consider. The biggest consideration is that shapefiles are not capable of handling attachments. So if you collected any photographs or added any other attachments, such as PDF documents, they will not be exported with the attribute table and the geometries to an output shapefile.

Workflow #2. Open the map in ArcGIS Desktop and create a local copy for editing

Let’s return to the My Content folder on ArcGIS Online and locate the same feature service that we used in workflow one.

Steps:

1. Click on the dropdown arrow on the right and choose the “open in ArcGIS Desktop” option

04_AGOL2DESKTP

 

2. Your browser should download a package. Once completed, click on it and this should launch ArcMap.

05_AGOL2DESKTP

 

A new ArcMap document with the feature service layer added to the table of contents will open.

3. Right-click on the group layer (the feature service layer which may contain several feature layers) and choose Edit Features > Create Local Copy for Editing

06_AGOL2DESKTP

A local copy of the data will be created and saved to the ArcMap’s home folder. By default it’s located at C:\Users\<Current User>\Documents\ArcGIS\FeatureServiceLocalEdits.

07_AGOL2DESKTP

TIP: The location of the Home folder can be easily changed through the Catalog Window Options Dialog.

08_AGOL2DESKTP

The geodatabase will contain attachments and the original feature class- shortcoming averted!!

Step 4 (Optional).

Sometimes you may need to transfer the edits and the new features back to the “master geodatabase” which is the geodatabase that had initially been used to publish a feature service.

To synchronise the changes between a desktop geodatabase and the feature service in the “Cloud” you can consider using one of the following methods:

a. Append Tool. The tool’s documentation is available here: http://resources.arcgis.com/en/help/main/10.2/index.html#//001700000050000000

b. Load Data (or the Simple Data Loader), which may not be the best choice for data with attachments as the relationship class which links attachments to features will not allow the Simple Data Loader to do its job. For non-simple features use the Object Loader. Related documentation is available here: http://resources.arcgis.com/en/help/main/10.2/index.html#//003n0000003r000000

c. Copy and paste features in an edit session.  The copy and paste workflow is described by the following technical article:  http://resources.arcgis.com/en/help/main/10.2/index.html#//01m700000041000000

The second workflow allows you to not only export the data and the attachments that you collected in the field, but it also allows you to edit a local copy of the data and synchronise the changes with the Feature Service running on your ArcGIS Online for Organisations account.

Step 5 Optional.

After you’ve created a local copy of the data, you can go proceed with editing as usual. Once you’ve saved the edits, right click on the group layer (the feature service layer) in the table of contents and choose Edit Features > synchronise Edits with the Server

09_AGOL2DESKTP

I hope the above workflows will help you easily get your data from the Cloud back to earth and use it in your editing and analysis workflows.

Oh, one more thing. Did you know that you can add your ArcGIS for Organisations account as an ArcGIS for Server connection in ArcMap? This little hidden gem could save you time and eliminate the necessity of getting your online data through a web browser.

The workflow is quite simple.

Step 1. Get the URL.

Log in to your ArcGIS Online for Organisations account and open properties of any of the feature services you’d published.

Step 2. Request a service URL

Scroll to layers and request a service URL

10_AGOL2DESKTP

 

The Service URL will appear at the top of your page, in the address bar. It’s a REST URL and the first part of it will contain the URL of your ArcGIS Online for Organisations subscription (highlighted in yellow)

11_AGOL2DESKTP

Step 3. Add an ArcGIS Server Connection.

Open ArcMap and in the Catalog Window scroll down until you see the GIS SERVERS category.

Expand the GIS SERVER folder and double click Add ArcGIS Server

Choose Use Services and enter the server’s URL and your ArcGIS Online username and password.

12_AGOL2DESKTP

 

A new ArcGIS for Server connection will be added to ArcMap and you will see all the feature services and other datasets that have been published on your ArcGIS Online for Organisations account.

13_AGOL2DESKTP

To use your data, simply drag and drop it to ArcMap’s main window.

14 thoughts on “Bringing Your Data from the Cloud Back to Earth – How to extract your data from ArcGIS Online.

  1. Mirjam

    If you have a ArcGIS public account and ArcGIS10 sp5, is there any way to get the data in the cloud back to earth? I made a couple of story maps and used a csv file to upload the map tour points. But I made some changes directly on the maptour (hence in the cloud e.g. arcgis online)
    Now I would love to get this data back. (not so much the images but points and description. And I don’t have an organisations account.

    Reply
  2. Ivan E. Post author

    Hi Mirjam,

    If you created a story map using the online storymap builder (just on the story maps website), then a feature service on ArcGIS Online should’ve been created.

    Unfortunately, it looks like you can get access to that feature service only if you are using an ArcGIS Online for Organisations account.
    However there’s a workaround for the public account owners, which works in ArcGIS 10.1 and 10.2.x:

    1. Go to ArcGIS.com

    2. Sign in using your public account

    3. When you create a storymap a web map and a web mapping application are being created by ArcGIS Online. You will be able to find them under “My Content” section of ArcGIS.com

    Find the web map (not the application) and choose to open it in ArcGIS for Desktop.

    4. The map which contains the basemap and the features that you’d created in the story map will open in ArcMap.

    5. Right-click on the feature layer > Data > export Data and save it as a shapefile or a geodatabase feature class.

    I tested this in ArcGIS Desktop 10.1 and 10.2.1 – worked well. But my 10.0SP5 machine has failed to open the map with the feature layer.

    Reply
  3. Mirjam

    Thanks for your reply. Like you said, the feature layer won’t load in ArcGIS desktop 10.0 sp5. I guess I have to ask for an upgrade 😉
    Thanks anyway for your extensive reply and your blogpost. It was just what I was looking for.
    Perhaps I have another tip (I haven’t read your other posts yet so perhaps this is old news) but I would be interested to know if you can use an editable layer in a maptour template.
    I made an local history story map http://bit.ly/1dH5FVY and it would be lovely if I could add an editable layer so people could add their own piece of local history.
    And will this work with a public account. Thanks anyway. I will keep an eye on your blog.

    Reply
    1. Ivan E. Post author

      Hi Mirjam,

      I’m afraid that is not possible. I don’t think you can add an extra editable feature service to the map tour template if the map is built online. And not with a free AGOL public account, I’m afraid…
      I can think of only one way to achieve something like this:

      1. Download the java script application (the Map Tour template from GitHub) https://github.com/Esri/map-tour-storytelling-template-js

      2. Get ArcGIS Online for Organisations (you can use the 30-day trial first and see if it works for you)

      3. Publish a feature service which will be editable in the web app

      4. Do some mid-level java script coding: basically you will need to reference another feature service within the config.js – the main app file

      5. Publish the web app on a web server. You will need to find a web server (I.e. use one in your organisations if you have it, or purchase a virtual hosting space somewhere).

      I must say that the above steps are more or less theoretical. I haven’t done it myself. But Java Script apps are quite simple and Esri inc do normally provide some meaningful instructions, so that everyone can be a java web app developer now:)

      The biggest issue is the web server. Sometimes it’s quite hard to find one that suits your budget and technical parameters, although, it shouldn’t be a problem in Europe.

      PS
      Your storymap is great! Well done!

      PPS
      I would also recommend you to check on Bern Szukalsly blogs: http://blogs.esri.com/esri/arcgis/author/bszukalski/ he’s the main storymap guy in the World

      Reply
  4. Iain Stuart

    You would think it would be easier to come back down from the cloud. I am sure most businesses and other organisations would want their basic data safe in their own system rather than sitting in some cloud.

    Reply
    1. Ivan E. Post author

      Hi Scott,
      If you are exporting your feature service as a file geodatabase (via AGOL GUI) or creating a local copy for editing, as mentioned in the blog, attachments will be stored inside that file geodatabase. You will not be able to see them a list of JPEG or PNG items as Geodatabase doesn’t support these imagery formats. They will be stored in one of the system tables instead

      Reply
  5. Eloise

    Hi,
    If i used a .sde data source to create my map that was published to use with arcgis Collector, what could be the best way to have my data in my .sde synchronized with to data collected or modified with Collector?
    Thank you,

    Reply
  6. MFR

    Hi, I recognize that this post is a bit old, but perhaps you can answer a related issue?

    If I am conducting local edits on a feature service from an organizational account (in 10.2 desktop) then I don’t sync my local edits yet, but I also make changes from the same username on a different machine (maybe in desktop or collector) then how does this process happen in the background and will it affect my edits? Sometimes I experience missing edits (geometry, attributes, deleted features, etc.) and I think it may be caused by simultaneous editing in the same username? But when I conduct controlled tests on this, nothing seems to be wrong? E.G. I will open two local copies, on two different machines, then make edits, and sync both back up, look for my edits and they persist. I have tried reordering the local copy and sync aspects in a variety of ways and always I find my edits. I am making small changes like add one feature, or edit one attribute.

    The reason I would want to do this is because I have a work machine and a portable laptop as well. Sometimes I forget to sync after moving from home to work, etc. Then when I get home I want to be able to sync my edits from my laptop, while the local copy had been sitting there all day, but I was editing in another local copy on another machine?

    I would like some specific information on how the sync process happens and what would potentially interfere with this. Missing data is a HUGE issue that I need to avoid. I am not the only user in my group editing the service and many others often find unsynced local copies too. It happens.

    Reply
  7. Kara M

    When I log into my AGO account as the administrator and go through the steps to make a feature layer exportable, I don’t have the shapefile or csv options – my drop down just has one option and it’s the “service URL”. What am I missing here? Thank you!

    Reply
  8. Bitje

    hmm, I am able to make a ‘local copy for editing’ but alas no attachments (photo’s in my case) are copied locally. Any idea what I’m doing wrong? I’m logged on as the owner of the data and

    Reply

Got something to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s