Viewing Imagery in a different light

Imagery is the most commonly requested basemap of all the Esri base maps available. It provides the spatial context for your authoritative data that vector base maps cannot. Many questions can be easily answered by visually inspecting the imagery, for example how far is that tree away from the power line? or is that a new swimming pool that does not have a building permit?

There is though more to imagery though than visually inspecting in the Red, Green and Blue visual spectrum. Many of today’s satellites capture information in the Yellow, Near Infrared and Far Infrared portions of the electromagnetic spectrum. In fact the latest World View satellite has 16 bands. So how can this extra analytical data be accessed in a web service?

Image services provide an underutilised feature called functions. These are algorithms that can be applied to an image service on-the-fly to reveal hidden information in the imagery. When an image service published, it is not just the visible band that can be published but all bands from a satellite image can be made available. In its simplest form a function can swap these bands being used to display. For example from Red, Green, Blue to Infrared, Red, Green. Another common analysis is Vegetation health through an NDVI analysis. The difference in chlorophyll reluctance and absorbance of plants in the Infrared and Red portion of the spectrum.  The best part about using functions is that they are only calculated on the portion of image shown on the screen at the resolution displayed and no new image is created, just an interpretation of the image, all performed server side on-the-fly and then sent to the client. There are some very good examples on the Amazon Landsat 8 demonstrator site found at http://www.esri.com/landing-pages/software/landsat/unlock-earths-secrets

Continue reading

2016: The Year of the Parcel Fabric

One of the many new additions to ArcMap 10 was the introduction of Parcel Editor. Some of you may remember this as a replacement for the Survey Analyst Cadastral Editor product. With this new addition the cadastral fabric dataset was replaced with the new parcel fabric. At the time many just looked at this with a passing interest, especially those with cadastral systems already in place.

So why should 2016 be the year of the Parcel Fabric for you? Here are my top three reasons why it should be. Continue reading

Landscape

Mapping Mars with ArcGIS Pro

I only discovered relatively recently that the Desktop suite is not limited to mapping Earth. Hidden amongst the thousands of coordinate systems available, you can find those for mapping planets within the Solar System.

Just for kicks, I downloaded all the geological data from Mars to have a look in ArcGIS Pro and see how it would fare in the hybrid 2D/3D interface.

You can try this out for yourselves by downloading the data from the USGS here. The datasets are conveniently provided in a file geodatabase and the layers come with some standard symbology in MXDs, so it’s an easy feat to import into ArcGIS Pro.

Continue reading

Versioned Workflows a True Story

As a long time GIS trainer working for Esri Australia, I have taught a lot of courses over the past 10 or so years. In my opinion, one of the best would have to be Implementing Versioned Workflows in a Multiuser Geodatabase. A three day course designed to cover the version related content in Esri’s current course curriculum.

I know the title sounds a little intimidating and I will not lie, some of the content is quite advanced.

However the way topics are introduced and explained visually, the continuity between the scenarios and the way we are taken on exploratory journey through the repository of the multiuser geodatabase leave students with a thorough understanding of its mechanics.

Forgetting that the scenarios are based in and around Fargo, North Dakota, the course progressively and comprehensively takes the students through a number of various real life workflows exposing how the functionality provided in a geodatabase has been implemented. After students perform functions from ArcMap as a user they then explore the changes to the SDE repository as an administrator.

After sitting this course for the first time not only did I walk away with a better understanding of alternative methods of implementing a multiuser geodatabase, but also the impact of the various approaches including the advantages and disadvantages of each.

The next Implementing Versioned Workflows in a Multiuser Geodatabase will be in Brisbane on the 8-10/3/2016.

If you are interested in learning more about this course or any courses offered by Esri Australia please contact the Training Coordinator on 1800 447 111 or email training@esriaustralia.com.au.

Capitalising on momentum in training with an Esri Australia Technical Adviser

In recent months, many of Esri Australia’s largest clients have embraced wide-ranging technical change with the deployment of the latest release of the ArcGIS Platform, comprising a suite of wholly new or improved technologies. These customers see real change in the first days of the deployment of the ArcGIS Platform. This is particularly true of clients who pair software deployment simultaneously with the delivery of Esri Australia’s technical training for their staff throughout an organisation – analyst to executive; spatial staff or staff new to location analytics.

In a recent engagement with a client, I have personally seen the deployment, configuration, integration and adoption of Esri’s Production Mapping and Workflow Manager technologies. These technologies have directly contributed to improving and streamlining analysts’ workflows, expanded the visibility and resource management capabilities of team leaders, and have provided executive staff with high level understanding of the synergies and friction points in business practices across the organisation. Continue reading

Is ArcSDE dead? Part II

The term ArcSDE/SDE is being gradually replaced with the term “Multiuser Geodatabase” and this is probably creating some confusion among the users of the ArcGIS platform. In order to understand the difference between ArcSDE and multiuser geodatabase, let’s start clarifying what is and what is not ArcSDE.

What is not ArcSDE:

ArcSDE is not a product. It’s a technology. In the same way ArcGIS is not a product, is a platform. Only prior to ArcGIS 9.2, ArcSDE was a standalone software product. At the ArcGIS 9.2 release, ArcSDE was integrated into both ArcGIS for Desktop and ArcGIS for Server.

ArcSDE is not only the application sever connection, a piece of software that packs the giomgr.exe and a gsrv.exe processes. This is a common mistake that I have mentioned in my previous post but from a more business perspective. I’ll try to do the same exercise now but from a technical perspective.

So, what is ArcSDE  – also called “the database enhanced with the ArcSDE technology” or simply the “ArcSDE geodatabase” or the “Multiuser geodatabase”?

Let’s answer this question step by step.

First of all, ArcSDE is a hybrid technology built up from different components. These components are implemented in two places, the client (ArcGIS for Desktop/Pro or ArcGIS Server, mobile, etc.) and in the relational database management system (any of the supported ones).

The components that make the ArcSDE technology are those shown in the slide number 3 and 4 below.

cmgd

Note that, the above are the slides of the official geodatabase training course version 10.2, taught all around the world therefore, at least until version 10.2, we have been using, explaining and discussing what ArcSDE is.

The slides below are from the same course at version 10.3. Note that the content is the same although the term ArcSDE has been replaced by “Multiuser geodatabase” (except that the translator component is only in the client side reflecting the deprecation of the application server connection).

CMGD_103

What are the components of the ArcSDE/Multiuser geodatabase technology? – Note that the lack of any of the components described below would mean that the multiuser geodatabase wouldn’t work.

  • The client (ArcGIS for Desktop (32 bits) , ArcGIS Pro and ArcGIS Server (64bits), etc.)
  • The repository of ArcSDE/multiuser geodatabase, the geodatabase system tables, stored procedures and functions.
  • The management tools:
    • Geoprocessing tools
    • Database tools – for example, the functionality to backup and restore is considered part of the ArcSDE technology.
    • And until version 10.2.x, the command line tools. Deprecated in version 10.3.
  • And finally, the translator, this is the component that translates every map interaction (clicks on the map performed by the user) into SQL statements – Since the database doesn’t understand “clicks on a map”, it only understands the SQL language, we need something that acts as an “interpreter” between the user and the database.

Notice that in the slide above (version 10.2) the translator component is floating in the middle of the client and the server. There is a reason why this is so… the reason is related with how the client connects to the multiuser geodatabase. Up to version 10.2, there were two different connection types, application server connection and direct connections – first released with version 9.2 -. Let’s review some of the particularities of each connection type:

  • Direct connections (two tier connection): If the client connects using direct connections, the translator component is located in the client side and requires to have the database client installed in the machine where the ArcGIS client is installed -. This is the default connection method.

transl

  • Application Service connections (3-tiers connection). The application server connection has been the traditional way to connect to an enterprise geodatabase. It requires a service (commonly called “sde” or “arcsde”) which is normally installed and configured in the same machine where the database server is installed.

transl22

The sde service packs two processes– the giomgr and the gsvr – the giomgr.exe is in charge of listening for incoming connections (clients creating a connection to the enterprise geodatabase using the sde service), when the connection is accepted, the giomgr fires up the “gsvr.exe”, one per client connection. Each gsvr.exe will then translates the clicks on the map performed by the user into SQL statements that are then passed to the database, the database interprets the SQL, generates the answer which is finally sent to the client through the gsvr.exe process, as it is shown below.

With this connection type all the translation load is placed on the server side.

sde_service

As I mentioned before, the application server connection method has been deprecated in version 10.3 and with it, the command line tools.

I’m sure there are many reasons why the application server connection has been deprecated. From my point of view there is one reason that deserves to be discussed more in detail. Performance.

When the application server connection was released the processing capacity of the servers was much higher than the processing capacity of the workstations. The application server connection was a good approach to put all the translator processing load into the server machine and so to use the server processing capacity to perform the translation.

Nowadays however, there are reasons why we wouldn’t want to use application server connections:

  • Today, we can use client’s machines processing capacity to perform the translation and in this way, to share the translation load among all the clients connecting to the multiuser geodatabase. We just need to install the database client and to connect to the multiuser geodatabase using direct connections. In this way, the translation will happen in the client machine, leaving the server to use the processing capacity for other purposes.
  • Application server connections also create single points of failure, if the sde service fails, no one can connect to the geodatabase whereas with the direct connection this risk doesn’t exist.
  • Application server connections put much more load on the server if we compare it with direct connections.

With regards to the last point, the capacity planning tool is an excellent tool to model each connection type and to see how much server processing capacity is required. Below I have done a quick exercise that shows the amount of server processing power required to perform the same workflow using application server connections versus direct connections.

I have modelled a simple workflow, ArcGIS for Desktop 10.2 connecting to a multiuser geodatabase using direct connection [1] and application server connections [2]. Both workflows generate 50.000 transactions per hour (140 concurrent users approx. with a productivity of 6 displays per minutes as average per user (a standard user).

DC

[1] Workflow: ArcGIS Desktop connecting to a multiuser geodatabase using direct connection (DC).

ASC

[2] Workflow: ArcGIS Desktop connecting to a multiuser geodatabase using application server connection (ASC).

Notice that using direct connection the workflow requires 14,7% of the server processing capacity whereas using application service connections it requires 29,3% of the server processing capacity, almost double. This is another reason  to reconsider the use of application server connections, if you are still using them. 

I hope I was able to bring some light into this topic. If you are interested in expanding your understanding about enterprise geodatabases, I highly recommend to attend the following training courses (in particular the second one):

Deploying and Maintaining a Multiuser Geodatabase (10.3)

Implementing Versioned Workflows in a Multiuser Geodatabase (10.3)

Regards, Walter Simonazzi

 

Ever wondered what layers reside in all my MXD’s? Is there a way to summarize each layers’ properties?

Here’s one way forward utilizing ArcGIS for Desktop and Python….

Working at Esri Australia in the Support and Training realm, we listen to numerous client’s requests, concerns and workflow issues. To keep abreast of how our technology is heading, one source I frequent is Continue reading