OSM PBF Loader¶
OSMPbfLoader can really quickly parse full OSM extract in the form of *.osm.pbf file.
It can download and parse a lot of features much faster than the OSMOnlineLoader, but it's much more useful when a lot of different features are required at once (like when using predefined filters).
When only a single or few features are needed, OSMOnlineLoader might be a better choice, since OSMPbfLoader will use a full extract of all features in a given region and will have to iterate over all of them.
Run this notebook in Google Colab:
Remember to install the srai library before running the notebook:
%pip install srai[all]
In [1]:
Copied!
import geopandas as gpd
from shapely.geometry import Point, box
from srai.constants import REGIONS_INDEX, WGS84_CRS
from srai.geometry import buffer_geometry
from srai.loaders.osm_loaders import OSMPbfLoader
from srai.loaders.osm_loaders.filters import GEOFABRIK_LAYERS, HEX2VEC_FILTER
from srai.loaders.osm_loaders.filters.popular import get_popular_tags
from srai.regionalizers import geocode_to_region_gdf
import geopandas as gpd
from shapely.geometry import Point, box
from srai.constants import REGIONS_INDEX, WGS84_CRS
from srai.geometry import buffer_geometry
from srai.loaders.osm_loaders import OSMPbfLoader
from srai.loaders.osm_loaders.filters import GEOFABRIK_LAYERS, HEX2VEC_FILTER
from srai.loaders.osm_loaders.filters.popular import get_popular_tags
from srai.regionalizers import geocode_to_region_gdf
Using OSMPbfLoader to download data for a specific area¶
Download all features from HEX2VEC_FILTER in Warsaw, Poland¶
In [2]:
Copied!
loader = OSMPbfLoader()
warsaw_gdf = geocode_to_region_gdf("Warsaw, Poland")
warsaw_features_gdf = loader.load(warsaw_gdf, HEX2VEC_FILTER)
warsaw_features_gdf
loader = OSMPbfLoader()
warsaw_gdf = geocode_to_region_gdf("Warsaw, Poland")
warsaw_features_gdf = loader.load(warsaw_gdf, HEX2VEC_FILTER)
warsaw_features_gdf
0%| | 0.00/91.3M [00:00<?, ?B/s]
0%| | 32.8k/91.3M [00:00<06:06, 249kB/s]
0%| | 81.9k/91.3M [00:00<04:46, 318kB/s]
0%| | 197k/91.3M [00:00<02:41, 563kB/s]
0%|▏ | 426k/91.3M [00:00<01:29, 1.02MB/s]
1%|▎ | 885k/91.3M [00:00<00:48, 1.88MB/s]
2%|▋ | 1.80M/91.3M [00:00<00:25, 3.56MB/s]
4%|█▍ | 3.62M/91.3M [00:00<00:12, 6.82MB/s]
7%|██▌ | 6.44M/91.3M [00:01<00:07, 11.3MB/s]
10%|███▊ | 9.32M/91.3M [00:01<00:05, 14.4MB/s]
13%|████▉ | 12.2M/91.3M [00:01<00:04, 16.5MB/s]
17%|██████ | 15.1M/91.3M [00:01<00:04, 17.9MB/s]
20%|███████▎ | 18.0M/91.3M [00:01<00:03, 18.9MB/s]
23%|████████▍ | 20.9M/91.3M [00:01<00:03, 19.6MB/s]
26%|█████████▌ | 23.7M/91.3M [00:01<00:03, 20.1MB/s]
29%|██████████▊ | 26.6M/91.3M [00:02<00:03, 20.4MB/s]
32%|███████████▉ | 29.5M/91.3M [00:02<00:02, 20.6MB/s]
35%|█████████████▏ | 32.4M/91.3M [00:02<00:02, 20.8MB/s]
39%|██████████████▎ | 35.3M/91.3M [00:02<00:02, 20.9MB/s]
42%|███████████████▍ | 38.2M/91.3M [00:02<00:02, 21.0MB/s]
45%|████████████████▋ | 41.0M/91.3M [00:02<00:02, 21.0MB/s]
48%|█████████████████▊ | 43.9M/91.3M [00:02<00:02, 21.1MB/s]
51%|██████████████████▉ | 46.8M/91.3M [00:02<00:02, 21.1MB/s]
54%|████████████████████▏ | 49.7M/91.3M [00:03<00:01, 21.1MB/s]
58%|█████████████████████▎ | 52.6M/91.3M [00:03<00:01, 21.1MB/s]
61%|██████████████████████▍ | 55.5M/91.3M [00:03<00:01, 21.1MB/s]
64%|███████████████████████▋ | 58.3M/91.3M [00:03<00:01, 21.1MB/s]
67%|████████████████████████▊ | 61.2M/91.3M [00:03<00:01, 21.1MB/s]
70%|█████████████████████████▉ | 64.1M/91.3M [00:03<00:01, 21.1MB/s]
73%|███████████████████████████▏ | 67.0M/91.3M [00:03<00:01, 21.1MB/s]
77%|████████████████████████████▎ | 69.9M/91.3M [00:04<00:01, 21.1MB/s]
80%|█████████████████████████████▍ | 72.8M/91.3M [00:04<00:00, 21.1MB/s]
83%|██████████████████████████████▋ | 75.6M/91.3M [00:04<00:00, 21.1MB/s]
86%|███████████████████████████████▊ | 78.5M/91.3M [00:04<00:00, 21.1MB/s]
89%|████████████████████████████████▉ | 81.4M/91.3M [00:04<00:00, 21.2MB/s]
92%|██████████████████████████████████▏ | 84.3M/91.3M [00:04<00:00, 21.2MB/s]
95%|███████████████████████████████████▎ | 87.2M/91.3M [00:04<00:00, 21.2MB/s]
99%|████████████████████████████████████▍| 90.1M/91.3M [00:05<00:00, 21.2MB/s]
0%| | 0.00/91.3M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 91.3M/91.3M [00:00<00:00, 117GB/s]
Finished operation in 0:01:00
Out[2]:
| geometry | aeroway | amenity | building | healthcare | historic | landuse | leisure | military | natural | office | shop | sport | tourism | water | waterway | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | ||||||||||||||||
| node/1849727484 | POINT (20.9936 52.23076) | None | bank | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/1851381884 | POINT (21.07414 52.12393) | None | bench | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/1853996701 | POINT (21.04548 52.1492) | None | bicycle_rental | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/1855433181 | POINT (20.97508 52.20378) | None | None | None | None | None | None | None | None | None | None | convenience | None | None | None | None |
| node/1855479857 | POINT (20.96181 52.2113) | None | None | None | None | None | None | None | None | None | None | convenience | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| way/1369235724 | POLYGON ((21.0455 52.17972, 21.04556 52.17972,... | None | parking | None | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1369235725 | POLYGON ((21.04549 52.17967, 21.04556 52.17967... | None | parking | None | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1369235726 | POLYGON ((21.04557 52.18017, 21.04564 52.18017... | None | parking | None | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1369235732 | POLYGON ((20.9433 52.23652, 20.94339 52.23654,... | None | None | yes | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1369368948 | POLYGON ((21.02536 52.22657, 21.02547 52.22646... | None | None | None | None | None | commercial | None | None | None | None | None | None | None | None | None |
349973 rows × 16 columns
Plot features¶
Inspired by prettymaps
In [3]:
Copied!
clipped_features_gdf = warsaw_features_gdf.clip(warsaw_gdf.geometry.union_all())
clipped_features_gdf = warsaw_features_gdf.clip(warsaw_gdf.geometry.union_all())
In [4]:
Copied!
ax = warsaw_gdf.plot(color="lavender", figsize=(16, 16))
# plot water
clipped_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="deepskyblue"
)
# plot greenery
clipped_features_gdf[
clipped_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="mediumseagreen")
# plot buildings
clipped_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="dimgray", markersize=0.1
)
xmin, ymin, xmax, ymax = warsaw_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = warsaw_gdf.plot(color="lavender", figsize=(16, 16))
# plot water
clipped_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="deepskyblue"
)
# plot greenery
clipped_features_gdf[
clipped_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="mediumseagreen")
# plot buildings
clipped_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="dimgray", markersize=0.1
)
xmin, ymin, xmax, ymax = warsaw_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Download all features from popular tags based on OSMTagInfo in Vienna, Austria¶
In [5]:
Copied!
popular_tags = get_popular_tags(in_wiki_only=True)
num_keys = len(popular_tags)
f"Unique keys: {num_keys}."
popular_tags = get_popular_tags(in_wiki_only=True)
num_keys = len(popular_tags)
f"Unique keys: {num_keys}."
Out[5]:
'Unique keys: 380.'
In [6]:
Copied!
{k: popular_tags[k] for k in list(popular_tags)[:10]}
{k: popular_tags[k] for k in list(popular_tags)[:10]}
Out[6]:
{'4wd_only': ['yes'],
'LandPro08:reviewed': ['no'],
'abandoned': ['yes'],
'abandoned:railway': ['rail'],
'abutters': ['residential'],
'access': ['agricultural',
'customers',
'delivery',
'designated',
'destination',
'forestry',
'military',
'no',
'permissive',
'permit',
'private',
'unknown',
'yes'],
'access_control': ['full', 'partial'],
'addr:TW:dataset': ['137998'],
'addr:city': ['London', '臺北市'],
'addr:country': ['CZ', 'DE', 'RU', 'TW', 'US']}
In [7]:
Copied!
vienna_center_circle = buffer_geometry(Point(16.37009, 48.20931), meters=1000)
vienna_center_circle_gdf = gpd.GeoDataFrame(
geometry=[vienna_center_circle],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["Vienna"], name=REGIONS_INDEX),
)
vienna_center_circle = buffer_geometry(Point(16.37009, 48.20931), meters=1000)
vienna_center_circle_gdf = gpd.GeoDataFrame(
geometry=[vienna_center_circle],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["Vienna"], name=REGIONS_INDEX),
)
In [8]:
Copied!
loader = OSMPbfLoader()
vienna_features_gdf = loader.load(vienna_center_circle_gdf, popular_tags)
vienna_features_gdf
loader = OSMPbfLoader()
vienna_features_gdf = loader.load(vienna_center_circle_gdf, popular_tags)
vienna_features_gdf
0%| | 0.00/49.1M [00:00<?, ?B/s]
0%| | 8.19k/49.1M [00:00<12:29, 65.6kB/s]
0%| | 38.9k/49.1M [00:00<04:50, 169kB/s]
0%| | 93.2k/49.1M [00:00<02:51, 286kB/s]
0%|▏ | 208k/49.1M [00:00<01:32, 527kB/s]
1%|▎ | 437k/49.1M [00:00<00:49, 984kB/s]
2%|▋ | 892k/49.1M [00:00<00:25, 1.86MB/s]
4%|█▎ | 1.79M/49.1M [00:00<00:13, 3.55MB/s]
7%|██▋ | 3.60M/49.1M [00:01<00:06, 6.91MB/s]
15%|█████▍ | 7.19M/49.1M [00:01<00:03, 13.5MB/s]
21%|███████▉ | 10.5M/49.1M [00:01<00:02, 17.1MB/s]
28%|██████████▌ | 14.0M/49.1M [00:01<00:01, 20.1MB/s]
35%|████████████▉ | 17.1M/49.1M [00:01<00:01, 21.4MB/s]
40%|██████████████▉ | 19.8M/49.1M [00:01<00:01, 21.3MB/s]
48%|█████████████████▋ | 23.5M/49.1M [00:01<00:01, 23.0MB/s]
55%|████████████████████▎ | 27.0M/49.1M [00:01<00:00, 24.3MB/s]
62%|██████████████████████▉ | 30.5M/49.1M [00:02<00:00, 25.2MB/s]
67%|████████████████████████▉ | 33.1M/49.1M [00:02<00:00, 23.6MB/s]
74%|███████████████████████████▍ | 36.4M/49.1M [00:02<00:00, 24.3MB/s]
81%|█████████████████████████████▉ | 39.8M/49.1M [00:02<00:00, 25.0MB/s]
88%|████████████████████████████████▋ | 43.5M/49.1M [00:02<00:00, 26.1MB/s]
96%|███████████████████████████████████▍ | 47.1M/49.1M [00:02<00:00, 26.7MB/s]
0%| | 0.00/49.1M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 49.1M/49.1M [00:00<00:00, 53.8GB/s]
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/quackosm/pb f_file_reader.py:2612: UserWarning: Select clause contains more than 100 columns (found 380 columns). Query might fail with insufficient memory resources. Consider applying more restrictive OsmTagsFilter for parsing. warnings.warn(
Finished operation in 0:00:33
Out[8]:
| geometry | access | admin_level | advertising | amenity | area:highway | artwork_type | atm | barrier | bench | ... | tunnel | type | usage | vehicle | vending | waste | water | water_source | waterway | wheelchair | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | |||||||||||||||||||||
| node/33182886 | POINT (16.36631 48.20782) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33182965 | POINT (16.37033 48.20359) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33344361 | POINT (16.36484 48.2159) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33344362 | POINT (16.36413 48.21558) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/34591548 | POINT (16.36967 48.20235) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| way/1429924649 | POLYGON ((16.37356 48.20983, 16.37356 48.20983... | None | None | None | fountain | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1429925120 | POLYGON ((16.37346 48.20991, 16.3734 48.20999,... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1429932987 | POLYGON ((16.36836 48.20717, 16.36828 48.20711... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1429933399 | POLYGON ((16.36815 48.20726, 16.36816 48.20726... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1429937063 | POLYGON ((16.36844 48.20997, 16.3685 48.20992,... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
23653 rows × 195 columns
Plot features¶
Uses default preset colours from prettymaps
In [9]:
Copied!
clipped_vienna_features_gdf = vienna_features_gdf.clip(vienna_center_circle)
clipped_vienna_features_gdf = vienna_features_gdf.clip(vienna_center_circle)
In [10]:
Copied!
ax = vienna_center_circle_gdf.plot(color="#F2F4CB", figsize=(16, 16))
# plot water
clipped_vienna_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="#a8e1e6"
)
# plot streets
clipped_vienna_features_gdf.dropna(subset=["highway"], how="all").plot(
ax=ax, color="#475657", markersize=0.1
)
# plot buildings
clipped_vienna_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="#FF5E5B"
)
# plot parkings
clipped_vienna_features_gdf[
(clipped_vienna_features_gdf["amenity"] == "parking")
| (clipped_vienna_features_gdf["highway"] == "pedestrian")
].plot(ax=ax, color="#2F3737", markersize=0.1)
# plot greenery
clipped_vienna_features_gdf[
clipped_vienna_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="#8BB174")
xmin, ymin, xmax, ymax = vienna_center_circle_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = vienna_center_circle_gdf.plot(color="#F2F4CB", figsize=(16, 16))
# plot water
clipped_vienna_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="#a8e1e6"
)
# plot streets
clipped_vienna_features_gdf.dropna(subset=["highway"], how="all").plot(
ax=ax, color="#475657", markersize=0.1
)
# plot buildings
clipped_vienna_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="#FF5E5B"
)
# plot parkings
clipped_vienna_features_gdf[
(clipped_vienna_features_gdf["amenity"] == "parking")
| (clipped_vienna_features_gdf["highway"] == "pedestrian")
].plot(ax=ax, color="#2F3737", markersize=0.1)
# plot greenery
clipped_vienna_features_gdf[
clipped_vienna_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="#8BB174")
xmin, ymin, xmax, ymax = vienna_center_circle_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Download all grouped features based on Geofabrik layers in New York, USA¶
In [11]:
Copied!
manhattan_bbox = box(-73.994551, 40.762396, -73.936872, 40.804239)
manhattan_bbox_gdf = gpd.GeoDataFrame(
geometry=[manhattan_bbox],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["New York"], name=REGIONS_INDEX),
)
manhattan_bbox = box(-73.994551, 40.762396, -73.936872, 40.804239)
manhattan_bbox_gdf = gpd.GeoDataFrame(
geometry=[manhattan_bbox],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["New York"], name=REGIONS_INDEX),
)
In [12]:
Copied!
loader = OSMPbfLoader()
new_york_features_gdf = loader.load(manhattan_bbox_gdf, GEOFABRIK_LAYERS)
new_york_features_gdf
loader = OSMPbfLoader()
new_york_features_gdf = loader.load(manhattan_bbox_gdf, GEOFABRIK_LAYERS)
new_york_features_gdf
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/quackosm/osm_extracts/__init__.py:592: GeometryNotCoveredWarning: Skipping extract because of low IoU value (bbbike_newyork, 0.000187). warnings.warn(
0%| | 0.00/98.0M [00:00<?, ?B/s]
0%| | 8.19k/98.0M [00:00<24:18, 67.2kB/s]
0%| | 38.9k/98.0M [00:00<09:24, 174kB/s]
0%| | 93.2k/98.0M [00:00<05:33, 294kB/s]
0%| | 208k/98.0M [00:00<03:00, 542kB/s]
0%|▏ | 437k/98.0M [00:00<01:36, 1.01MB/s]
1%|▎ | 892k/98.0M [00:00<00:50, 1.91MB/s]
2%|▋ | 1.80M/98.0M [00:00<00:26, 3.67MB/s]
4%|█▎ | 3.62M/98.0M [00:00<00:13, 7.13MB/s]
7%|██▍ | 6.42M/98.0M [00:01<00:07, 11.9MB/s]
9%|███▏ | 8.45M/98.0M [00:01<00:06, 13.3MB/s]
12%|████▎ | 11.4M/98.0M [00:01<00:05, 16.3MB/s]
14%|█████▏ | 13.6M/98.0M [00:01<00:05, 16.8MB/s]
17%|██████▏ | 16.5M/98.0M [00:01<00:04, 18.8MB/s]
19%|███████ | 18.8M/98.0M [00:01<00:04, 18.6MB/s]
22%|████████▏ | 21.6M/98.0M [00:01<00:03, 19.8MB/s]
24%|█████████ | 24.0M/98.0M [00:01<00:03, 19.6MB/s]
27%|██████████ | 26.7M/98.0M [00:02<00:03, 20.2MB/s]
30%|███████████ | 29.2M/98.0M [00:02<00:03, 20.2MB/s]
32%|████████████ | 31.8M/98.0M [00:02<00:03, 20.5MB/s]
35%|████████████▉ | 34.3M/98.0M [00:02<00:03, 20.3MB/s]
38%|█████████████▉ | 36.9M/98.0M [00:02<00:03, 20.3MB/s]
40%|██████████████▉ | 39.4M/98.0M [00:02<00:02, 20.4MB/s]
43%|███████████████▊ | 42.0M/98.0M [00:02<00:02, 20.4MB/s]
45%|████████████████▊ | 44.5M/98.0M [00:02<00:02, 20.5MB/s]
48%|█████████████████▋ | 47.0M/98.0M [00:03<00:02, 20.3MB/s]
51%|██████████████████▋ | 49.6M/98.0M [00:03<00:02, 20.4MB/s]
53%|███████████████████▋ | 52.1M/98.0M [00:03<00:02, 20.2MB/s]
56%|████████████████████▌ | 54.6M/98.0M [00:03<00:02, 20.2MB/s]
58%|█████████████████████▌ | 57.1M/98.0M [00:03<00:02, 20.2MB/s]
61%|██████████████████████▌ | 59.6M/98.0M [00:03<00:01, 20.2MB/s]
63%|███████████████████████▍ | 62.1M/98.0M [00:03<00:01, 20.2MB/s]
66%|████████████████████████▍ | 64.6M/98.0M [00:03<00:01, 20.1MB/s]
68%|█████████████████████████▎ | 67.1M/98.0M [00:04<00:01, 19.9MB/s]
71%|██████████████████████████▎ | 69.6M/98.0M [00:04<00:01, 20.0MB/s]
73%|███████████████████████████▏ | 72.0M/98.0M [00:04<00:01, 19.8MB/s]
76%|████████████████████████████▏ | 74.5M/98.0M [00:04<00:01, 20.0MB/s]
79%|█████████████████████████████ | 77.0M/98.0M [00:04<00:01, 20.0MB/s]
81%|█████████████████████████████▉ | 79.4M/98.0M [00:04<00:00, 19.8MB/s]
84%|██████████████████████████████▉ | 82.1M/98.0M [00:04<00:00, 20.1MB/s]
86%|███████████████████████████████▉ | 84.5M/98.0M [00:04<00:00, 19.9MB/s]
89%|████████████████████████████████▊ | 87.1M/98.0M [00:05<00:00, 20.2MB/s]
91%|█████████████████████████████████▊ | 89.5M/98.0M [00:05<00:00, 19.9MB/s]
94%|██████████████████████████████████▊ | 92.1M/98.0M [00:05<00:00, 20.4MB/s]
96%|███████████████████████████████████▋ | 94.5M/98.0M [00:05<00:00, 20.0MB/s]
99%|████████████████████████████████████▋| 97.2M/98.0M [00:05<00:00, 20.4MB/s]
0%| | 0.00/98.0M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 98.0M/98.0M [00:00<00:00, 161GB/s]
Finished operation in 0:00:38
Out[12]:
| geometry | accommodation | buildings | catering | education | fuel_parking | health | highway_links | landuse | leisure | ... | public | railways | shopping | tourism | traffic | transport | very_small_roads | water | water_traffic | waterways | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | |||||||||||||||||||||
| node/1748991481 | POINT (-73.94931 40.78784) | None | None | None | None | None | None | None | None | None | ... | None | None | shop=convenience | None | None | None | None | None | None | None |
| node/1748991484 | POINT (-73.94863 40.78925) | None | None | amenity=restaurant | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/1748991487 | POINT (-73.94757 40.79013) | None | None | None | None | None | None | None | None | None | ... | None | None | shop=greengrocer | None | None | None | None | None | None | None |
| node/1748991489 | POINT (-73.94864 40.78879) | None | None | amenity=restaurant | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/1749000951 | POINT (-73.9486 40.78885) | None | None | None | None | None | None | None | None | None | ... | None | None | shop=convenience | None | None | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| way/1418249946 | LINESTRING (-73.95286 40.80421, -73.95286 40.8... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1418253086 | LINESTRING (-73.98241 40.77388, -73.98232 40.7... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1418253087 | LINESTRING (-73.98226 40.77382, -73.9822 40.77... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1418253088 | LINESTRING (-73.98241 40.77404, -73.98232 40.7... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1418253089 | LINESTRING (-73.98227 40.77397, -73.9822 40.77... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
51040 rows × 27 columns
Plot features¶
Inspired by https://snazzymaps.com/style/14889/flat-pale
In [13]:
Copied!
ax = manhattan_bbox_gdf.plot(color="#e7e7df", figsize=(16, 16))
# plot greenery
new_york_features_gdf[new_york_features_gdf["leisure"] == "leisure=park"].plot(
ax=ax, color="#bae5ce"
)
# plot water
new_york_features_gdf.dropna(subset=["water", "waterways"], how="all").plot(
ax=ax, color="#c7eced"
)
# plot streets
new_york_features_gdf.dropna(subset=["paths_unsuitable_for_cars"], how="all").plot(
ax=ax, color="#e7e7df", linewidth=1
)
new_york_features_gdf.dropna(
subset=["very_small_roads", "highway_links", "minor_roads"], how="all"
).plot(ax=ax, color="#fff", linewidth=2)
new_york_features_gdf.dropna(subset=["major_roads"], how="all").plot(
ax=ax, color="#fac9a9", linewidth=3
)
# plot buildings
new_york_features_gdf.dropna(subset=["buildings"], how="all").plot(
ax=ax, color="#cecebd"
)
xmin, ymin, xmax, ymax = manhattan_bbox_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = manhattan_bbox_gdf.plot(color="#e7e7df", figsize=(16, 16))
# plot greenery
new_york_features_gdf[new_york_features_gdf["leisure"] == "leisure=park"].plot(
ax=ax, color="#bae5ce"
)
# plot water
new_york_features_gdf.dropna(subset=["water", "waterways"], how="all").plot(
ax=ax, color="#c7eced"
)
# plot streets
new_york_features_gdf.dropna(subset=["paths_unsuitable_for_cars"], how="all").plot(
ax=ax, color="#e7e7df", linewidth=1
)
new_york_features_gdf.dropna(
subset=["very_small_roads", "highway_links", "minor_roads"], how="all"
).plot(ax=ax, color="#fff", linewidth=2)
new_york_features_gdf.dropna(subset=["major_roads"], how="all").plot(
ax=ax, color="#fac9a9", linewidth=3
)
# plot buildings
new_york_features_gdf.dropna(subset=["buildings"], how="all").plot(
ax=ax, color="#cecebd"
)
xmin, ymin, xmax, ymax = manhattan_bbox_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Using OSMPbfLoader to download data for a specific area and transforming it to GeoParquet file¶
Download all grouped features based on Geofabrik layers in Reykjavík, Iceland¶
In [14]:
Copied!
loader = OSMPbfLoader()
reykjavik_gdf = geocode_to_region_gdf("Reykjavík, IS")
reykjavik_features_gpq = loader.load_to_geoparquet(reykjavik_gdf, GEOFABRIK_LAYERS)
reykjavik_features_gpq
loader = OSMPbfLoader()
reykjavik_gdf = geocode_to_region_gdf("Reykjavík, IS")
reykjavik_features_gpq = loader.load_to_geoparquet(reykjavik_gdf, GEOFABRIK_LAYERS)
reykjavik_features_gpq
0%| | 0.00/62.9M [00:00<?, ?B/s]
0%| | 12.3k/62.9M [00:00<13:11, 79.5kB/s]
0%| | 41.0k/62.9M [00:00<07:26, 141kB/s]
0%| | 98.3k/62.9M [00:00<04:17, 244kB/s]
0%|▏ | 209k/62.9M [00:00<02:27, 427kB/s]
1%|▎ | 438k/62.9M [00:00<01:18, 801kB/s]
1%|▌ | 893k/62.9M [00:00<00:40, 1.52MB/s]
3%|█ | 1.80M/62.9M [00:01<00:20, 2.92MB/s]
6%|██▏ | 3.62M/62.9M [00:01<00:10, 5.69MB/s]
10%|███▋ | 6.31M/62.9M [00:01<00:06, 9.28MB/s]
14%|█████▎ | 9.03M/62.9M [00:01<00:04, 11.8MB/s]
19%|██████▊ | 11.7M/62.9M [00:01<00:03, 13.4MB/s]
23%|████████▍ | 14.3M/62.9M [00:01<00:03, 14.3MB/s]
27%|█████████▉ | 16.8M/62.9M [00:02<00:03, 14.9MB/s]
31%|███████████▍ | 19.3M/62.9M [00:02<00:02, 15.3MB/s]
35%|████████████▉ | 21.9M/62.9M [00:02<00:02, 15.6MB/s]
39%|██████████████▎ | 24.4M/62.9M [00:02<00:02, 15.8MB/s]
43%|███████████████▊ | 27.0M/62.9M [00:02<00:02, 15.9MB/s]
47%|█████████████████▎ | 29.5M/62.9M [00:02<00:02, 15.9MB/s]
51%|██████████████████▊ | 32.0M/62.9M [00:02<00:01, 16.0MB/s]
55%|████████████████████▎ | 34.6M/62.9M [00:03<00:01, 16.1MB/s]
59%|█████████████████████▊ | 37.1M/62.9M [00:03<00:01, 16.1MB/s]
63%|███████████████████████▎ | 39.7M/62.9M [00:03<00:01, 16.3MB/s]
67%|████████████████████████▊ | 42.2M/62.9M [00:03<00:01, 16.2MB/s]
71%|██████████████████████████▎ | 44.7M/62.9M [00:03<00:01, 16.2MB/s]
75%|███████████████████████████▊ | 47.3M/62.9M [00:03<00:00, 16.1MB/s]
79%|█████████████████████████████▎ | 49.8M/62.9M [00:04<00:00, 16.1MB/s]
83%|██████████████████████████████▊ | 52.3M/62.9M [00:04<00:00, 16.2MB/s]
87%|████████████████████████████████▎ | 54.9M/62.9M [00:04<00:00, 16.2MB/s]
91%|█████████████████████████████████▋ | 57.4M/62.9M [00:04<00:00, 16.1MB/s]
95%|███████████████████████████████████▏ | 59.9M/62.9M [00:04<00:00, 16.2MB/s]
99%|████████████████████████████████████▋| 62.4M/62.9M [00:04<00:00, 16.1MB/s]
0%| | 0.00/62.9M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 62.9M/62.9M [00:00<00:00, 80.9GB/s]
Finished operation in 0:00:23
Out[14]:
PosixPath('files/372cc10dc2cb70fec59ce6093baa282a6c709187d7106ba7ccc97660a0426039_098931824b94bfe02e01bc4987422a39ec47bd3f6924e5325ddae0846badafba_exploded.parquet')
Read those features using DuckDB¶
In [15]:
Copied!
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_features_gpq))
features_relation
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_features_gpq))
features_relation
Out[15]:
┌─────────────────┬───────────────┬─────────────┬───────────┬──────────┬───────────┬──────────────┬─────────┬───────────────┬─────────┬─────────┬─────────────┬─────────────┬─────────┬─────────┬─────────┬───────────────────────────┬─────────┬─────────┬──────────┬──────────────┬─────────┬───────────────────────────┬───────────┬──────────────────┬─────────┬───────────────┬───────────┬────────────────────────────────┐ │ feature_id │ accommodation │ air_traffic │ buildings │ catering │ education │ fuel_parking │ health │ highway_links │ landuse │ leisure │ major_roads │ minor_roads │ miscpoi │ money │ natural │ paths_unsuitable_for_cars │ pofw │ public │ railways │ shopping │ tourism │ traffic │ transport │ very_small_roads │ water │ water_traffic │ waterways │ geometry │ │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ geometry │ ├─────────────────┼───────────────┼─────────────┼───────────┼──────────┼───────────┼──────────────┼─────────┼───────────────┼─────────┼─────────┼─────────────┼─────────────┼─────────┼─────────┼─────────┼───────────────────────────┼─────────┼─────────┼──────────┼──────────────┼─────────┼───────────────────────────┼───────────┼──────────────────┼─────────┼───────────────┼───────────┼────────────────────────────────┤ │ node/738189872 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8004768 64.1272348) │ │ node/739058158 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7970908 64.1236694) │ │ node/754499254 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ shop=alcohol │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8480784 64.1403193) │ │ node/754499261 │ tourism=hotel │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9385928 64.1482053) │ │ node/761664757 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=motorway_junction │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.82538 64.124046) │ │ node/761681461 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9404569 64.14286) │ │ node/765786389 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9090665 64.1365761) │ │ node/765796009 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9085977 64.1377917) │ │ node/769892993 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9329523 64.1492124) │ │ node/786031184 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.947746 64.1479759) │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ node/2004244218 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8481031 64.1070553) │ │ node/2004244220 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8410716 64.107062) │ │ node/2004244221 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8486821 64.1070732) │ │ node/2004244223 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8389556 64.1070758) │ │ node/2004244224 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8425917 64.1070868) │ │ node/2004244226 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8374445 64.1071434) │ │ node/2004244227 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8335792 64.1071457) │ │ node/2004244229 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8314298 64.1071526) │ │ node/2004244230 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.843039 64.1071542) │ │ node/2004244232 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.839698 64.1071736) │ ├─────────────────┴───────────────┴─────────────┴───────────┴──────────┴───────────┴──────────────┴─────────┴───────────────┴─────────┴─────────┴─────────────┴─────────────┴─────────┴─────────┴─────────┴───────────────────────────┴─────────┴─────────┴──────────┴──────────────┴─────────┴───────────────────────────┴───────────┴──────────────────┴─────────┴───────────────┴───────────┴────────────────────────────────┤ │ ? rows (>9999 rows, 20 shown) 29 columns │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Count all buildings¶
In [16]:
Copied!
features_relation.filter("buildings IS NOT NULL").count("feature_id")
features_relation.filter("buildings IS NOT NULL").count("feature_id")
Out[16]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 25372 │ └───────────────────┘
You can see more examples of how to use PbfFileReader from the QuackOSM library in the docs: https://kraina-ai.github.io/quackosm/latest/examples/pbf_file_reader/