The Data Quality Assessment
The Data Quality Assessment shows OSM map coverage by comparing the number of OSM object counts (number/km2) to the local population density (population/km2). It is simple, yet efficient in showing in which areas spatial objects are already mapped and whether it has low or high population density. It is thus a good proxy to understand the spatial completeness of OSM data without comparing each feature to the real-world from satellite images.
Why assessing Data Quality
CommonSensing uses data from multiple sources, including government departments from CS partner countries, international organisations, and non-governmental organisations. OpenStreetMap (OSM) data is created mainly by citizens without formal qualifications. The quality of OSM datasets has not been assessed. Map shows areas where people are, but data is not available.
Methodology to calculate estimates of completeness
This is the process to estimate the completeness of Open Street Map coverage of building footprints in areas where the map is known to be incomplete.
Building Data from OpenStreetMap
Building footprints are sourced from OpenStreetMap contributors on HEX.
High-resolution population data from Facebook
High-resolution population data from Facebook Connectivity Lab and Center for International Earth Science Information Network
Constructed data by machine learning techniques to identify buildings from commercially available satellite images. Then, overlay with general population estimates and other population statistics. See more
STEP1: Create hexagonal grids covering target area
Why Hexagonal grids?
Better fit to curved surfaces: when dealing with large areas, where the curvature of the earth becomes important, hexagons are better able to fit this curvature than squares. Therefore, soccer balls are constructed of hexagonal panels
Reduced edge effects: a hexagonal grid gives the lowest perimeter to area ratio of any regular tessellation of the plane
All neighbors are identical: a hexagonal grid cell has six identical neighboring cells, each sharing one of the six equal length sides. Furthermore, the distance between centroids is the same for all neighbors
All hexagonal grids in the Data Quality Assessment application is created using Create Hexagon Tessellation toolbox which can be downloaded from here. For more information, see this blog post.
STEP2: Aggregate population data into hex grids
High-resolution population data from Facebook can be downloaded from the following links below:
- Fiji >> population_fji_2018-10-01.csv.zip
- Solomon Islands >> population_slb_2018-10-01.csv.zip
- Vanuatu >> population_vut_2018-10-01.csv.zip
After downloaded data, you can use Spatial Join (Analysis) toolbox to summary population in each hexagonal grid.
STEP3: Count number of buildings in to hex grids
Building data can be downloaded from OpenStreetMap website. Then, you must convert building from polygon data to point to count how many building in each hexagonal grid. This can be done with Feature To Point toolbox and make sure you checked INSIDE in point_location parameter before running the tool. After your building point is ready, you can again use Spatial Join (Analysis) to count the number of building in each hexagonal grid.
STEP4: Publish your result to ArcGIS Online/Portal
When your data is ready, you can now publish to ArcGIS Online/Portal to be used in the Data Quality Assessment application. For more information, see Publish hosted feature layers.
STEP5: Make Relationship Map
Relationship maps are a new way to visualize and compare your data within ArcGIS Online. They allow you to map two patterns within a single map and help you see if two things are related. This is done using a technique known as bivariate choropleth mapping, where two color ramps combine into a grid-like legend showing all pattern combinations. See How to Make a Relationship Map in ArcGIS Online.
Configuring the application to work with your data
The DQA application is developed using Interactive Legend Configurable App Template built with the 4.x ArcGIS API for JavaScript. It is a configurable app template where the core functionality revolves around an interactive filter tool. This configurable app template provides end-users with an experience to filter features on a web map via selectable legend elements.
Once you have a feature service, you will have to modify some of the configuration settings in before deploying the application.
The following settings must be modified:
- Go to application folder: <<your web server path>>\DSS\FJI\DQA\app\config\applicationBase.json
- Open the json file in a text editor and replace the default web map ID in line 54.
- Find your web map id, navigate to ArcGIS Online or your Portal, and find the map you want to display. If it is one of your maps, make sure it's shared with everyone (public). View the map details and copy the ID from the URL in the top of your browser. The section you need to copy is bolded in the following URL: unosat-geodrr.cern.ch/portal/home/webmap/viewer.html?webmap= d7ecee5a3a2842bd85d5909418ef23df
In addition, you may need to change the following settings:
- Go to application folder: <<your web server path>>\DSS\FJI\DQA\app\config\application.json
- Open the json file in a text editor and edit the following parameters:
- Set portalURL to the URL of your ArcGIS Online organisation or your portal if you are not using ArcGIS Online.
Follow your web server's instructions to publish the contents of application folder as a web site or virtual directory.
For more information, see