Case Research: Unstructured Location Descriptions of Automobile Accidents
Knowledge Assortment and Preparation
To check out and quantify the geocoding capabilities of LLMs, an inventory of 100 unstructured location descriptions of car accidents in Minnesota had been randomly chosen from a dataset that was scraped from the web. The bottom reality coordinates for all 100 accidents had been manually created via using varied mapping purposes like Google Maps and the Minnesota Division of Transportation’s Traffic Mapping Application (TMA).
Some pattern location descriptions are featured beneath.
US Hwy 71 at MN Hwy 60 , WINDOM, Cottonwood County
EB Freeway 10 close to Joplin St NW, ELK RIVER, Sherburne County
EB I 90 / HWY 22, FOSTER TWP, Faribault County
Freeway 75 milepost 403, SAINT VINCENT TWP, Kittson County
65 Freeway / King Highway, BRUNSWICK TWP, Kanabec County
As seen within the examples above, there are large number of prospects for a way the outline will be structured, in addition to what defines the placement. One instance of that is the fourth description, which includes a mile marker quantity, which is way much less more likely to be matched in any type of geocoding course of, since that info sometimes isn’t included in any type of reference information. Discovering the bottom reality coordinates for descriptions like this one relied closely on using the Minnesota Division of Transportation’s Linear Referencing System (LRS) which supplies a standardized method of how roads are measured via out the State, with which mile markers play an important function in. This information will be accessed via the TMA utility talked about beforehand.
Methodology & Geocoding Methods
After getting ready the information, 5 separate notebooks had been set as much as check out totally different geocoding processes. Their configurations are as follows.
1. Google Geocoding API, used on the uncooked location description
2. Esri Geocoding API, used on the uncooked location description
3. Google Geocoding API, used on an OpenAI GPT 3.5 standardized location description
4. Esri Geocoding API, used on an OpenAI GPT 3.5 standardized location description
5. OpenAI GPT 3.5, used as a geocoder itself
To summarize, the Google and Esri geocoding APIs had been used on each the uncooked descriptions in addition to descriptions that had been standardized utilizing a brief immediate that was handed into the OpenAI GPT 3.5 mannequin. The Python code for this standardization course of will be seen beneath.
def standardize_location(df, description_series):
df["ai_location_description"] = df[description_series].apply(_gpt_chat)return df
def _gpt_chat(input_text):
immediate = """Standardize the next location description into textual content
that could possibly be fed right into a Geocoding API. When responding, solely
return the output textual content."""
response = openai.ChatCompletion.create(
mannequin="gpt-3.5-turbo",
messages=[
{"role": "system", "content": prompt},
{"role": "user", "content": input_text},
],
temperature=0.7,
n=1,
max_tokens=150,
cease=None,
)
return response.decisions[0].message.content material.strip().break up("n")[-1]
The 4 check circumstances utilizing geocoding APIs used the code beneath to make API requests to their respective geocoders and return the ensuing coordinates for all 100 descriptions.
# Esri Geocoder
def geocode_esri(df, description_series):
df["xy"] = df[description_series].apply(
_single_esri_geocode
)df["x"] = df["xy"].apply(
lambda row: row.break up(",")[0].strip()
)
df["y"] = df["xy"].apply(
lambda row: row.break up(",")[1].strip()
)
df["x"] = pd.to_numeric(df["x"], errors="coerce")
df["y"] = pd.to_numeric(df["y"], errors="coerce")
df = df[df["x"].notna()]
df = df[df["y"].notna()]
return df
def _single_esri_geocode(input_text):
base_url = "https://geocode-api.arcgis.com/arcgis/relaxation/companies/World/GeocodeServer/findAddressCandidates"
params = {
"f": "json",
"singleLine": input_text,
"maxLocations": "1",
"token": os.environ["GEOCODE_TOKEN"],
}
response = requests.get(base_url, params=params)
information = response.json()
attempt:
x = information["candidates"][0]["location"]["x"]
y = information["candidates"][0]["location"]["y"]
besides:
x = None
y = None
return f"{x}, {y}"
# Google Geocoder
def geocode_google(df, description_series):
df["xy"] = df[description_series].apply(
_single_google_geocode
)df["x"] = df["xy"].apply(
lambda row: row.break up(",")[0].strip()
)
df["y"] = df["xy"].apply(
lambda row: row.break up(",")[1].strip()
)
df["x"] = pd.to_numeric(df["x"], errors="coerce")
df["y"] = pd.to_numeric(df["y"], errors="coerce")
df = df[df["x"].notna()]
df = df[df["y"].notna()]
return df
def _single_google_geocode(input_text):
base_url = "https://maps.googleapis.com/maps/api/geocode/json"
params = {
"deal with": input_text,
"key": os.environ["GOOGLE_MAPS_KEY"],
"bounds": "43.00,-97.50 49.5,-89.00",
}
response = requests.get(base_url, params=params)
information = response.json()
attempt:
x = information["results"][0]["geometry"]["location"]["lng"]
y = information["results"][0]["geometry"]["location"]["lat"]
besides:
x = None
y = None
return f"{x}, {y}"
Moreover, one ultimate course of examined was to make use of GPT 3.5 because the geocoder itself, with out the assistance of any geocoding API. The code for this course of appeared almost an identical to the standardization code used above, however featured a distinct immediate, proven beneath.
Geocode the next deal with. Return a latitude (Y) and longitude (X) as precisely as doable. When responding, solely return the output textual content within the following format: X, Y
Efficiency Metrics and Insights
After the varied processes had been developed, every course of was run and several other efficiency metrics had been calculated, each when it comes to execution time and geocoding accuracy. These metrics are listed beneath.
| Geocoding Course of | Imply | StdDev | MAE | RMSE |
| ------------------- | ------ | ------ | ------ | ------ |
| Google with GPT 3.5 | 0.1012 | 1.8537 | 0.3698 | 1.8565 |
| Google with Uncooked | 0.1047 | 1.1383 | 0.2643 | 1.1431 |
| Esri with GPT 3.5 | 0.0116 | 0.5748 | 0.0736 | 0.5749 |
| Esri with Uncooked | 0.0001 | 0.0396 | 0.0174 | 0.0396 |
| GPT 3.5 Geocoding | 2.1261 | 80.022 | 45.416 | 80.050 |
| Geocoding Course of | 75% ET | 90% ET | 95% ET | Run Time |
| ------------------- | ------ | ------ | ------ | -------- |
| Google with GPT 3.5 | 0.0683 | 0.3593 | 3.3496 | 1m 59.9s |
| Google with Uncooked | 0.0849 | 0.4171 | 3.3496 | 0m 23.2s |
| Esri with GPT 3.5 | 0.0364 | 0.0641 | 0.1171 | 2m 22.7s |
| Esri with Uncooked | 0.0362 | 0.0586 | 0.1171 | 0m 51.0s |
| GPT 3.5 Geocoding | 195.54 | 197.86 | 199.13 | 1m 11.9s |
The metrics are defined in additional element right here. Imply represents the imply error (when it comes to Manhattan distance, or the full of X and Y distinction from the bottom reality, in decimal levels). StdDev represents the usual deviation of error (when it comes to Manhattan distance, in decimal levels). MAE represents the imply absolute error (when it comes to Manhattan distance, in decimal levels). RMSE represents the foundation imply sq. error (when it comes to Manhattan distance, in decimal levels). 75%, 90%, 95% ET represents the error threshold for that given % (when it comes to Euclidean distance, in decimal levels), that means that for a given proportion, that proportion of information falls throughout the ensuing worth’s distance from the bottom reality. Lastly, run time merely represents the full time taken to run the geocoding course of on 100 information.
Clearly, GPT 3.5 performs far worse by itself. Though, if a pair outliers are taken out of the image (which had been labelled by the mannequin as being situated in different continents), for essentially the most half the outcomes of that course of don’t look too misplaced, visually not less than.
It’s also attention-grabbing to see that the LLM-standardization course of truly decreased accuracy, which I personally discovered a bit shocking, since my complete intention of introducing that part was to hopefully barely enhance the general accuracy of the geocoding course of. It’s price noting that the prompts themselves might have been part of the issue right here, and it’s price additional exploring the function of “immediate engineering” in geospatial contexts.
The final fundamental takeaway from this evaluation is the execution time variations, with which any course of that features using GPT 3.5 performs considerably slower. Esri’s geocoding API can also be slower than Google’s on this setting too. Rigorous testing was not carried out, nonetheless, so these outcomes ought to be taken with that into consideration.
