Geocoding in the Wild: Comparing Mapbox, Google, Esri, and HERE

Right after I launched PickYourPlace, I started collecting user feedback. One way I did this was by demoing the functionality to friends. A few weeks ago, after my radio interview, we had some family friends over. When my wife mentioned my project and radio appearance, they were curious to learn more. I gave them my pitch and offered to show them a demo. They eagerly agreed, so I connected my iPad to the TV and walked them through everything from the landing page to the explore page.

I was using random addresses to demonstrate different functions when I suggested trying their home address. They were excited about this idea, but things got interesting when I entered their street name. The address was "Dallyn St NW," but the search only showed "Dailyn St NW" in the suggestions. There was no "Dallyn" at all. When I checked the map, it also showed "Dailyn." Puzzled by this discrepancy, I moved on with the demo using my own address instead.

PickYourPlace uses Mapbox for maps and geocoding services, which relies on OpenStreetMap (OSM) data. When I investigated the issue on the OSM website and other OSM-based providers, I discovered they all had the incorrect spelling. While this was frustrating, finding the root cause meant I knew how to fix it.

OSM is an open location data provider—anyone can contribute data to the platform, and all that data is freely available globally. Think of it as Wikipedia for maps and location data. I've used OSM extensively over the past decade to create base maps, points of interest databases, and routing engines for various projects. While I love its open nature and excellent toolchain, this same openness led to my demo mishap. Someone had contributed incorrect data to OSM, affecting everything built on top of it.

In the geospatial world, data quality is paramount. Poor data quality—whether in completeness or freshness—impacts everything: base maps show wrong information, geocoders return incorrect addresses and coordinates, and routing engines produce faulty results. No matter how great the software is, bad data leads to poor outcomes.

Fortunately, contributing to OSM is straightforward. You can create an account on openstreetmap.org and start editing the world map immediately. Unlike other providers like Google, Esri, or HERE—where change requests can take months if they're even possible—OSM changes can go live in minutes to days. After I fixed the street names and residential property addresses, Mapbox incorporated my updates within a week. That's faster than most companies' typical two-week sprint cycle for releases. The images below show the before and after states of the corrected street name.

This experience highlighted the importance of data quality in location-based products. Bad data makes any product less usable or unusable entirely. It also prompted me to examine geocoding more closely. Those familiar with geospatial technology understand the process of choosing a service provider. Geocoding providers typically fall into two categories: those using OSM or other open data, like Mapbox, Stadia Maps, and OpenCage, and proprietary solutions like Google, Esri, and HERE. The ongoing debate about which provider to choose often comes down to several factors, with data quality at the forefront.

Should I switch providers? To answer this objectively, I decided to conduct an experiment. Last year, while working on the Amazon Location Service team at AWS, I performed a similar analysis comparing Esri and HERE geocoders. This time, I wanted to evaluate how Mapbox performs against Esri, HERE, and Google.

What is Geocoding?

Geocoding converts addresses into geographic coordinates, enabling us to place locations on a map. The process involves three main steps:

1. Parsing input data
2. Querying the reference database
3. Assigning geographic coordinates

The primary outputs are geographic coordinates (longitude and latitude pairs). The process also produces normalized address components like street number, street name, city, postal code, region, and country, which help with consistent formatting and data enrichment.

Different providers may produce varying results based on how they collect and maintain their data. They might combine proprietary sources, open data, and AI-assisted mapping, updating their databases on different schedules. All these factors affect output quality.

Evaluation Framework

Let's examine the key metrics for comparing geocoders:

Match Rate

The percentage of successfully geocoded addresses relative to total submitted addresses. A higher match rate indicates better address resolution.

Accuracy

Compares geocoding results to baseline data in two ways:

• Positional accuracy: How close each geocoded point is to the "true" location (baseline/ground truth), measured by spatial distance
• Lexical accuracy: How closely returned address labels match the "true" address labels, measured using Levenshtein distance (minimum single-character edits needed to transform one string into another)

Similarity

Performs pairwise comparisons between geocoders:

• Positional similarity: How similar geocoded points are spatially
• Lexical similarity: How similar address formats and spellings are

Your use case determines which metrics matter most. If you only need geographic coordinates, focus on match rate and positional metrics, ignoring lexical ones.

Test Scenarios

I tested the geocoders using three scenarios with 1,000 randomly selected Canadian addresses from the City of Calgary's open data portal:

Scenario 1: Basic Input

Used raw input data with just address fields (no city, province, postal code, or country). Example:

id,address,latitude,longitude
85,25 EVERGREEN CR SW,50.9215815275547,-114.100897916366

Scenario 2: Enriched Input

Added city, province, and country to addresses. Example:

25 EVERGREEN CR SW, Calgary, AB, Canada

Scenario 3: Misspelled Input

Modified Scenario 2 addresses by removing "E" and "W" from quadrant indicators. Example:

25 EVERGREEN CR S, Calgary, AB, Canada

Methods

I used Python 3.9 in a Jupyter Notebook environment and:

• Downloaded residential address and city boundary datasets
• Obtained API keys from:
  • Amazon Location Service (for Esri and HERE)
  • Google Maps Platform
  • Mapbox
  • Canada Post (for address verification)

For each scenario, I:

1. Geocoded addresses using all providers
2. Collected Canada Post-verified addresses as baseline
3. Analyzed results for match rate, accuracy, and similarity

Results and Analysis

Match Rate

In Scenario 1, HERE and Mapbox geocoders demonstrated superior performance with match rates exceeding 99%, with over 75% of geocoded addresses falling within city boundaries. Esri showed the weakest performance, achieving only a 60% match rate, with just 3% of geocoded addresses within city limits. This indicates that HERE, Mapbox, and Google handle incomplete addresses more effectively than Esri.

For Scenarios 2 and 3, all providers showed marked improvement. Esri, Google, and Mapbox performed similarly, achieving the highest match rates and slightly outperforming HERE by approximately 3%.

Positional Accuracy

In Scenario 1, Google emerged as the leader with a median error distance of 10.8m, with 49% of its geocoded points falling within 10m of the baseline. HERE and Mapbox showed similar performance, with approximately 60% of their geocoded points within 100m of the baseline. Esri's results largely deviated by more than 10km from the baseline.

All providers showed significant improvement in the next two scnearios. Esri and Google performed similarly, achieving the best results with median error distances of 6m and 7m in Scenario 2, and 7m and 8m in Scenario 3, respectively. About 60% of their geocoded points fell within 10m of the baseline.

Detailed statistics showed that approximately 60% of points from Esri and Google were within 10m of the baseline, while about 70% from HERE and Mapbox were within 100m.

Positional Similarity

HERE and Mapbox demonstrated the highest degree of similarity across all scenarios, with nearly half their geocoded points within 10m of each other. Their similarity was strongest in Scenario 2, showing a median pairwise distance of 8m.

The similarity analysis revealed:

• Strong correlation between HERE and Mapbox results
• Consistent performance across different input formats
• Greater divergence in results when handling incomplete addresses

Lexical Accuracy

In Scenario 1, HERE, Google, and Mapbox performed similarly, with median similarity scores between 0.5 and 0.6. For Scenarios 2 and 3, Esri stood out with over 20% of results achieving the highest lexical accuracy score. The other providers maintained consistent performance, with approximately 80% of their results achieving a 0.75 lexical accuracy score.

Lexical Similarity

Analysis showed Mapbox's address labels closely aligned with HERE's results in Scenario 1, with even stronger similarities in subsequent scenarios (median pairwise scores above 0.82). Mapbox showed the least similarity to Esri across all scenarios.

The results demonstrated:

• High consistency between HERE and Mapbox in address formatting
• Strong correlation in normalized address components
• Significant variations in how different providers handle address normalization

Key Findings

1. Match Rate Performance:
   • HERE and Mapbox excel with incomplete addresses
   • All providers perform well with complete addresses
   • Esri requires more complete address information for optimal results
2. Positional Accuracy:
   • Google leads in accuracy with incomplete addresses
   • Esri and Google show best results with complete addresses
   • Address completeness significantly impacts accuracy across all providers
3. Address Label Quality:
   • HERE, Google, and Mapbox maintain consistent performance
   • Esri excels with complete address information
   • All providers handle common misspellings effectively

Conclusion

For PickYourPlace, match rate and lexical accuracy are crucial metrics. While Mapbox's current performance meets our needs, this analysis reveals the strengths of different providers. Each project should evaluate geocoders using representative data specific to their use case, as unique characteristics like address formats or geographic coverage can significantly impact performance.

Why not build our own geocoder? Because it's complex:

• Parsing varied input formats is challenging
• Maintaining current reference data is resource-intensive
• Processing multiple data sources is costly and time-consuming

Maybe someday, but for now, we'll stick with proven solutions. I've shared the code for this analysis in a GitHub repository, though it currently only includes two geocoders. You can use it as a template to evaluate other providers and geographies based on your specific needs.