In my previous blogs I have described how to make imagery available in both image and maps services and how you may then consume these services in cloud image processing analytics. But what about the actual image storage.
We know that imagery from ground up is always big! Everything about an image represents volumes and capacity. If you want to create image services and host them on your own infrastructure then you need to ensure that you have suitable attached storage but server storage is expensive and can be prohibitive.
Now we have all heard of cloud computing, how about cloud storage. Many of us would have seen by now the new cloud storage solutions Dropbox, OneDrive or Box. Can I put my imagery here? Well not really as ArcGIS for Server cannot reference these storage and the storage required for imagery would get expensive. Amazon and Microsoft do provide cheap storage options in the S3 disks and Azure Blobs. These are ideal storage mechanisms for imagery as they are relatively cheap and imagery is relatively static. Even better we can configure cloud servers to read this imagery and serve it fast!
Esri has partnered with USGS and Amazon to place Landsat imagery in the Amazon cloud and have the imagery readily accessible to everyone. Already there is 120 Terabyte of Landsat data and this is growing by 400 Gigabytes every day. Each location on the earth has a new scene added once every two weeks. You can analyse this imagery on the fly for crop health, geology, fire and time series the imagery. To take a look go to http://www.esri.com/landing-pages/software/landsat/unlock-earths-secrets I think you’ll be impressed.
Those of us who understand cloud storage will already be saying that these drives are not fast and that access to them cannot be as fast as what is seen in the above example. Well ESRI has extended its ArcGIS platform to include a new image format called MRF. Originally developed by NASA this format is made up of a number of files:
- Auxiliary files
The index file is the clever file as it is actually a cache of the image file. When imagery is loaded to the cloud storage it is moved to this format. This format then enables ArcGIS to server the imagery as if it was on the local the machine.
Each Amazon instance that is configured comes with 100gb of EBS storage. On this attached storage is an index of all the images on the S3 drives. Image extension simply reads the index, calls the image it requires from S3 and copies it locally to the attached storage. Now here is the clever part. It only copies the part of the image it requires at the resolution it requires, therefore providing the speed required to serve imagery.
This then makes an image management solution on Amazon very affordable as all the imagery is stored cheaply while still being readily accessible. As the end user it is possible to store your imagery off-site and coupled with ArcGIS for Server make it available to everyone in your organisation at a respectable speed regardless of the application being used.
This functionality is currently available through special request from Esri & Esri Australia but will be part of the standard product in an upcoming release of ArcGIS.
Enjoy the freedom of large image stores