OSM PBF Loader¶
OSMPbfLoader can really quickly parse full OSM extract in the form of *.osm.pbf file.
It can download and parse a lot of features much faster than the OSMOnlineLoader, but it's much more useful when a lot of different features are required at once (like when using predefined filters).
When only a single or few features are needed, OSMOnlineLoader might be a better choice, since OSMPbfLoader will use a full extract of all features in a given region and will have to iterate over all of them.
Run this notebook in Google Colab:
Remember to install the srai library before running the notebook:
%pip install srai[all]
In [1]:
Copied!
import geopandas as gpd
from shapely.geometry import Point, box
from srai.constants import REGIONS_INDEX, WGS84_CRS
from srai.geometry import buffer_geometry
from srai.loaders.osm_loaders import OSMPbfLoader
from srai.loaders.osm_loaders.filters import GEOFABRIK_LAYERS, HEX2VEC_FILTER
from srai.loaders.osm_loaders.filters.popular import get_popular_tags
from srai.regionalizers import geocode_to_region_gdf
import geopandas as gpd
from shapely.geometry import Point, box
from srai.constants import REGIONS_INDEX, WGS84_CRS
from srai.geometry import buffer_geometry
from srai.loaders.osm_loaders import OSMPbfLoader
from srai.loaders.osm_loaders.filters import GEOFABRIK_LAYERS, HEX2VEC_FILTER
from srai.loaders.osm_loaders.filters.popular import get_popular_tags
from srai.regionalizers import geocode_to_region_gdf
Using OSMPbfLoader to download data for a specific area¶
Download all features from HEX2VEC_FILTER in Warsaw, Poland¶
In [2]:
Copied!
loader = OSMPbfLoader()
warsaw_gdf = geocode_to_region_gdf("Warsaw, Poland")
warsaw_features_gdf = loader.load(warsaw_gdf, HEX2VEC_FILTER)
warsaw_features_gdf
loader = OSMPbfLoader()
warsaw_gdf = geocode_to_region_gdf("Warsaw, Poland")
warsaw_features_gdf = loader.load(warsaw_gdf, HEX2VEC_FILTER)
warsaw_features_gdf
0%| | 0.00/90.8M [00:00<?, ?B/s]
0%| | 32.8k/90.8M [00:00<06:09, 246kB/s]
0%| | 81.9k/90.8M [00:00<04:48, 314kB/s]
0%| | 197k/90.8M [00:00<02:42, 557kB/s]
0%|▏ | 426k/90.8M [00:00<01:29, 1.00MB/s]
1%|▎ | 885k/90.8M [00:00<00:48, 1.86MB/s]
2%|▋ | 1.80M/90.8M [00:00<00:25, 3.52MB/s]
4%|█▍ | 3.62M/90.8M [00:00<00:12, 6.74MB/s]
7%|██▌ | 6.36M/90.8M [00:01<00:07, 10.9MB/s]
10%|███▊ | 9.24M/90.8M [00:01<00:05, 14.1MB/s]
13%|████▉ | 12.1M/90.8M [00:01<00:04, 16.2MB/s]
17%|██████ | 15.0M/90.8M [00:01<00:04, 17.7MB/s]
20%|███████▎ | 17.9M/90.8M [00:01<00:03, 18.6MB/s]
23%|████████▍ | 20.8M/90.8M [00:01<00:03, 19.3MB/s]
26%|█████████▋ | 23.7M/90.8M [00:01<00:03, 19.8MB/s]
29%|██████████▊ | 26.5M/90.8M [00:02<00:03, 20.1MB/s]
32%|███████████▉ | 29.3M/90.8M [00:02<00:03, 20.2MB/s]
36%|█████████████▏ | 32.3M/90.8M [00:02<00:02, 20.6MB/s]
39%|██████████████▎ | 35.2M/90.8M [00:02<00:02, 20.7MB/s]
42%|███████████████▌ | 38.1M/90.8M [00:02<00:02, 20.7MB/s]
45%|████████████████▋ | 41.0M/90.8M [00:02<00:02, 20.8MB/s]
48%|█████████████████▊ | 43.8M/90.8M [00:02<00:02, 20.8MB/s]
51%|███████████████████ | 46.7M/90.8M [00:03<00:02, 20.8MB/s]
55%|████████████████████▏ | 49.6M/90.8M [00:03<00:01, 20.8MB/s]
58%|█████████████████████▍ | 52.5M/90.8M [00:03<00:01, 20.8MB/s]
61%|██████████████████████▌ | 55.3M/90.8M [00:03<00:01, 20.8MB/s]
64%|███████████████████████▋ | 58.2M/90.8M [00:03<00:01, 20.8MB/s]
67%|████████████████████████▉ | 61.1M/90.8M [00:03<00:01, 20.9MB/s]
71%|██████████████████████████ | 64.0M/90.8M [00:03<00:01, 20.9MB/s]
74%|███████████████████████████▎ | 66.9M/90.8M [00:03<00:01, 20.9MB/s]
77%|████████████████████████████▍ | 69.8M/90.8M [00:04<00:01, 20.9MB/s]
80%|█████████████████████████████▋ | 72.7M/90.8M [00:04<00:00, 20.9MB/s]
83%|██████████████████████████████▊ | 75.6M/90.8M [00:04<00:00, 20.9MB/s]
86%|███████████████████████████████▉ | 78.4M/90.8M [00:04<00:00, 20.9MB/s]
90%|█████████████████████████████████▏ | 81.3M/90.8M [00:04<00:00, 20.9MB/s]
93%|██████████████████████████████████▎ | 84.2M/90.8M [00:04<00:00, 20.9MB/s]
96%|███████████████████████████████████▌ | 87.1M/90.8M [00:04<00:00, 20.9MB/s]
99%|████████████████████████████████████▋| 90.0M/90.8M [00:05<00:00, 20.9MB/s]
0%| | 0.00/90.8M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 90.8M/90.8M [00:00<00:00, 160GB/s]
Finished operation in 0:01:00
Out[2]:
| geometry | aeroway | amenity | building | healthcare | historic | landuse | leisure | military | natural | office | shop | sport | tourism | water | waterway | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | ||||||||||||||||
| node/59815899 | POINT (21.09668 52.17673) | None | None | None | None | None | None | None | None | None | None | None | None | None | None | dam |
| node/61253022 | POINT (21.00581 52.17944) | None | parking_entrance | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/95003080 | POINT (20.96723 52.24442) | None | fuel | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/95100197 | POINT (20.93535 52.25315) | None | fountain | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/96657908 | POINT (20.92989 52.26453) | None | parking | None | None | None | None | None | None | None | None | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| way/1253599642 | POLYGON ((21.05555 52.15791, 21.05553 52.15791... | None | shelter | None | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1253730265 | POLYGON ((20.86831 52.18313, 20.86831 52.18309... | None | None | service | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1253730929 | POLYGON ((20.86814 52.18513, 20.86808 52.18513... | None | None | service | None | None | None | None | None | None | None | None | None | None | None | None |
| way/1253747709 | LINESTRING (21.08047 52.29275, 21.08005 52.292... | None | None | None | None | None | None | None | None | None | None | None | None | None | None | ditch |
| way/1253747710 | LINESTRING (21.08057 52.29287, 21.08047 52.29275) | None | None | None | None | None | None | None | None | None | None | None | None | None | None | ditch |
348063 rows × 16 columns
Plot features¶
Inspired by prettymaps
In [3]:
Copied!
clipped_features_gdf = warsaw_features_gdf.clip(warsaw_gdf.geometry.union_all())
clipped_features_gdf = warsaw_features_gdf.clip(warsaw_gdf.geometry.union_all())
In [4]:
Copied!
ax = warsaw_gdf.plot(color="lavender", figsize=(16, 16))
# plot water
clipped_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="deepskyblue"
)
# plot greenery
clipped_features_gdf[
clipped_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="mediumseagreen")
# plot buildings
clipped_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="dimgray", markersize=0.1
)
xmin, ymin, xmax, ymax = warsaw_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = warsaw_gdf.plot(color="lavender", figsize=(16, 16))
# plot water
clipped_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="deepskyblue"
)
# plot greenery
clipped_features_gdf[
clipped_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="mediumseagreen")
# plot buildings
clipped_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="dimgray", markersize=0.1
)
xmin, ymin, xmax, ymax = warsaw_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Download all features from popular tags based on OSMTagInfo in Vienna, Austria¶
In [5]:
Copied!
popular_tags = get_popular_tags(in_wiki_only=True)
num_keys = len(popular_tags)
f"Unique keys: {num_keys}."
popular_tags = get_popular_tags(in_wiki_only=True)
num_keys = len(popular_tags)
f"Unique keys: {num_keys}."
Out[5]:
'Unique keys: 378.'
In [6]:
Copied!
{k: popular_tags[k] for k in list(popular_tags)[:10]}
{k: popular_tags[k] for k in list(popular_tags)[:10]}
Out[6]:
{'4wd_only': ['yes'],
'LandPro08:reviewed': ['no'],
'abandoned': ['yes'],
'abandoned:railway': ['rail'],
'abutters': ['residential'],
'access': ['agricultural',
'customers',
'delivery',
'designated',
'destination',
'forestry',
'military',
'no',
'permissive',
'permit',
'private',
'unknown',
'yes'],
'access_control': ['full', 'partial'],
'addr:TW:dataset': ['137998'],
'addr:city': ['London', '臺北市'],
'addr:country': ['CZ', 'DE', 'RU', 'TW', 'US']}
In [7]:
Copied!
vienna_center_circle = buffer_geometry(Point(16.37009, 48.20931), meters=1000)
vienna_center_circle_gdf = gpd.GeoDataFrame(
geometry=[vienna_center_circle],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["Vienna"], name=REGIONS_INDEX),
)
vienna_center_circle = buffer_geometry(Point(16.37009, 48.20931), meters=1000)
vienna_center_circle_gdf = gpd.GeoDataFrame(
geometry=[vienna_center_circle],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["Vienna"], name=REGIONS_INDEX),
)
In [8]:
Copied!
loader = OSMPbfLoader()
vienna_features_gdf = loader.load(vienna_center_circle_gdf, popular_tags)
vienna_features_gdf
loader = OSMPbfLoader()
vienna_features_gdf = loader.load(vienna_center_circle_gdf, popular_tags)
vienna_features_gdf
0%| | 0.00/49.0M [00:00<?, ?B/s]
0%| | 8.19k/49.0M [00:00<12:43, 64.1kB/s]
0%| | 38.9k/49.0M [00:00<04:55, 165kB/s]
0%| | 93.2k/49.0M [00:00<02:54, 280kB/s]
0%|▏ | 208k/49.0M [00:00<01:34, 516kB/s]
1%|▎ | 437k/49.0M [00:00<00:50, 963kB/s]
2%|▋ | 892k/49.0M [00:00<00:26, 1.82MB/s]
4%|█▎ | 1.80M/49.0M [00:00<00:13, 3.49MB/s]
7%|██▋ | 3.60M/49.0M [00:01<00:06, 6.77MB/s]
15%|█████▍ | 7.16M/49.0M [00:01<00:03, 13.1MB/s]
21%|███████▉ | 10.4M/49.0M [00:01<00:02, 16.5MB/s]
27%|█████████▊ | 13.0M/49.0M [00:01<00:02, 17.5MB/s]
34%|████████████▌ | 16.6M/49.0M [00:01<00:01, 20.1MB/s]
41%|███████████████▏ | 20.0M/49.0M [00:01<00:01, 21.4MB/s]
47%|█████████████████▌ | 23.2M/49.0M [00:01<00:01, 22.3MB/s]
54%|███████████████████▉ | 26.4M/49.0M [00:01<00:00, 23.0MB/s]
61%|██████████████████████▌ | 29.9M/49.0M [00:02<00:00, 23.9MB/s]
68%|█████████████████████████ | 33.2M/49.0M [00:02<00:00, 24.4MB/s]
75%|███████████████████████████▋ | 36.6M/49.0M [00:02<00:00, 24.8MB/s]
82%|██████████████████████████████▏ | 39.9M/49.0M [00:02<00:00, 25.1MB/s]
89%|████████████████████████████████▊ | 43.4M/49.0M [00:02<00:00, 25.4MB/s]
95%|███████████████████████████████████▎ | 46.7M/49.0M [00:02<00:00, 25.5MB/s]
0%| | 0.00/49.0M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 49.0M/49.0M [00:00<00:00, 48.0GB/s]
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/quackosm/pb f_file_reader.py:2612: UserWarning: Select clause contains more than 100 columns (found 378 columns). Query might fail with insufficient memory resources. Consider applying more restrictive OsmTagsFilter for parsing. warnings.warn(
Finished operation in 0:00:33
Out[8]:
| geometry | access | admin_level | advertising | amenity | area:highway | artwork_type | atm | barrier | bench | ... | tunnel | type | usage | vehicle | vending | waste | water | water_source | waterway | wheelchair | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | |||||||||||||||||||||
| node/33182886 | POINT (16.36631 48.20782) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33182965 | POINT (16.37033 48.20359) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33344361 | POINT (16.36484 48.2159) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33344362 | POINT (16.36413 48.21558) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/34591548 | POINT (16.36967 48.20235) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| relation/6155723 | POLYGON ((16.36869 48.20096, 16.3688 48.20117,... | None | None | None | None | None | None | None | None | yes | ... | yes | multipolygon | None | None | None | None | None | None | None | yes |
| relation/6528503 | POLYGON ((16.37127 48.21546, 16.37135 48.21538... | None | None | None | None | None | None | None | None | None | ... | None | multipolygon | None | None | None | None | None | None | None | None |
| relation/7362473 | POLYGON ((16.35977 48.211, 16.35979 48.21101, ... | None | None | None | fountain | None | None | None | None | None | ... | None | multipolygon | None | None | None | None | None | None | None | None |
| relation/6528506 | POLYGON ((16.37002 48.21476, 16.36996 48.21474... | None | None | None | None | None | None | None | None | None | ... | None | multipolygon | None | None | None | None | None | None | None | None |
| relation/6528511 | POLYGON ((16.37114 48.2149, 16.37116 48.21489,... | None | None | None | None | None | None | None | None | None | ... | None | multipolygon | None | None | None | None | None | None | None | None |
23607 rows × 194 columns
Plot features¶
Uses default preset colours from prettymaps
In [9]:
Copied!
clipped_vienna_features_gdf = vienna_features_gdf.clip(vienna_center_circle)
clipped_vienna_features_gdf = vienna_features_gdf.clip(vienna_center_circle)
In [10]:
Copied!
ax = vienna_center_circle_gdf.plot(color="#F2F4CB", figsize=(16, 16))
# plot water
clipped_vienna_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="#a8e1e6"
)
# plot streets
clipped_vienna_features_gdf.dropna(subset=["highway"], how="all").plot(
ax=ax, color="#475657", markersize=0.1
)
# plot buildings
clipped_vienna_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="#FF5E5B"
)
# plot parkings
clipped_vienna_features_gdf[
(clipped_vienna_features_gdf["amenity"] == "parking")
| (clipped_vienna_features_gdf["highway"] == "pedestrian")
].plot(ax=ax, color="#2F3737", markersize=0.1)
# plot greenery
clipped_vienna_features_gdf[
clipped_vienna_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="#8BB174")
xmin, ymin, xmax, ymax = vienna_center_circle_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = vienna_center_circle_gdf.plot(color="#F2F4CB", figsize=(16, 16))
# plot water
clipped_vienna_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="#a8e1e6"
)
# plot streets
clipped_vienna_features_gdf.dropna(subset=["highway"], how="all").plot(
ax=ax, color="#475657", markersize=0.1
)
# plot buildings
clipped_vienna_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="#FF5E5B"
)
# plot parkings
clipped_vienna_features_gdf[
(clipped_vienna_features_gdf["amenity"] == "parking")
| (clipped_vienna_features_gdf["highway"] == "pedestrian")
].plot(ax=ax, color="#2F3737", markersize=0.1)
# plot greenery
clipped_vienna_features_gdf[
clipped_vienna_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="#8BB174")
xmin, ymin, xmax, ymax = vienna_center_circle_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Download all grouped features based on Geofabrik layers in New York, USA¶
In [11]:
Copied!
manhattan_bbox = box(-73.994551, 40.762396, -73.936872, 40.804239)
manhattan_bbox_gdf = gpd.GeoDataFrame(
geometry=[manhattan_bbox],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["New York"], name=REGIONS_INDEX),
)
manhattan_bbox = box(-73.994551, 40.762396, -73.936872, 40.804239)
manhattan_bbox_gdf = gpd.GeoDataFrame(
geometry=[manhattan_bbox],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["New York"], name=REGIONS_INDEX),
)
In [12]:
Copied!
loader = OSMPbfLoader()
new_york_features_gdf = loader.load(manhattan_bbox_gdf, GEOFABRIK_LAYERS)
new_york_features_gdf
loader = OSMPbfLoader()
new_york_features_gdf = loader.load(manhattan_bbox_gdf, GEOFABRIK_LAYERS)
new_york_features_gdf
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/quackosm/osm_extracts/__init__.py:592: GeometryNotCoveredWarning: Skipping extract because of low IoU value (bbbike_newyork, 0.000187). warnings.warn(
0%| | 0.00/97.9M [00:00<?, ?B/s]
0%| | 8.19k/97.9M [00:00<25:58, 62.8kB/s]
0%| | 38.9k/97.9M [00:00<10:02, 162kB/s]
0%| | 93.2k/97.9M [00:00<05:55, 275kB/s]
0%| | 208k/97.9M [00:00<03:12, 507kB/s]
0%|▏ | 437k/97.9M [00:00<01:42, 947kB/s]
1%|▎ | 892k/97.9M [00:00<00:54, 1.79MB/s]
2%|▋ | 1.80M/97.9M [00:00<00:27, 3.44MB/s]
4%|█▎ | 3.62M/97.9M [00:01<00:14, 6.68MB/s]
6%|██▎ | 6.25M/97.9M [00:01<00:08, 10.8MB/s]
9%|███▏ | 8.39M/97.9M [00:01<00:07, 12.4MB/s]
11%|████▏ | 11.0M/97.9M [00:01<00:05, 14.5MB/s]
14%|█████ | 13.3M/97.9M [00:01<00:05, 15.4MB/s]
16%|█████▉ | 15.8M/97.9M [00:01<00:04, 16.6MB/s]
19%|██████▉ | 18.4M/97.9M [00:01<00:04, 17.4MB/s]
21%|███████▉ | 21.0M/97.9M [00:01<00:04, 18.0MB/s]
24%|████████▉ | 23.5M/97.9M [00:02<00:04, 18.2MB/s]
27%|█████████▉ | 26.2M/97.9M [00:02<00:03, 18.8MB/s]
29%|██████████▊ | 28.7M/97.9M [00:02<00:03, 18.8MB/s]
32%|███████████▊ | 31.3M/97.9M [00:02<00:03, 19.2MB/s]
35%|████████████▊ | 33.9M/97.9M [00:02<00:03, 19.1MB/s]
37%|█████████████▊ | 36.5M/97.9M [00:02<00:03, 19.2MB/s]
40%|██████████████▋ | 38.9M/97.9M [00:02<00:03, 19.0MB/s]
42%|███████████████▋ | 41.5M/97.9M [00:03<00:02, 19.0MB/s]
45%|████████████████▋ | 44.0M/97.9M [00:03<00:02, 19.1MB/s]
48%|█████████████████▌ | 46.6M/97.9M [00:03<00:02, 19.1MB/s]
50%|██████████████████▌ | 49.2M/97.9M [00:03<00:02, 19.3MB/s]
53%|███████████████████▌ | 51.8M/97.9M [00:03<00:02, 19.3MB/s]
56%|████████████████████▌ | 54.4M/97.9M [00:03<00:02, 19.4MB/s]
58%|█████████████████████▌ | 57.0M/97.9M [00:03<00:02, 19.5MB/s]
61%|██████████████████████▌ | 59.7M/97.9M [00:03<00:01, 19.7MB/s]
64%|███████████████████████▌ | 62.4M/97.9M [00:04<00:01, 19.8MB/s]
66%|████████████████████████▌ | 65.0M/97.9M [00:04<00:01, 19.8MB/s]
69%|█████████████████████████▌ | 67.7M/97.9M [00:04<00:01, 19.9MB/s]
72%|██████████████████████████▌ | 70.3M/97.9M [00:04<00:01, 19.9MB/s]
75%|███████████████████████████▌ | 73.0M/97.9M [00:04<00:01, 19.9MB/s]
77%|████████████████████████████▌ | 75.6M/97.9M [00:04<00:01, 19.9MB/s]
80%|█████████████████████████████▌ | 78.3M/97.9M [00:04<00:00, 19.9MB/s]
83%|██████████████████████████████▌ | 81.0M/97.9M [00:05<00:00, 20.0MB/s]
86%|███████████████████████████████▋ | 83.8M/97.9M [00:05<00:00, 20.5MB/s]
89%|████████████████████████████████▊ | 86.9M/97.9M [00:05<00:00, 21.1MB/s]
92%|██████████████████████████████████ | 90.1M/97.9M [00:05<00:00, 22.0MB/s]
96%|███████████████████████████████████▎ | 93.5M/97.9M [00:05<00:00, 23.1MB/s]
99%|████████████████████████████████████▌| 96.8M/97.9M [00:05<00:00, 23.7MB/s]
0%| | 0.00/97.9M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 97.9M/97.9M [00:00<00:00, 125GB/s]
Finished operation in 0:00:39
Out[12]:
| geometry | accommodation | buildings | catering | education | fuel_parking | health | highway_links | landuse | leisure | ... | public | railways | shopping | tourism | traffic | transport | very_small_roads | water | water_traffic | waterways | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | |||||||||||||||||||||
| node/462004948 | POINT (-73.95553 40.77949) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | railway=station | None | None | None | None |
| node/480491548 | POINT (-73.96358 40.78135) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=crossing | None | None | None | None | None |
| node/480492539 | POINT (-73.96863 40.77474) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=crossing | None | None | None | None | None |
| node/480492646 | POINT (-73.96828 40.77501) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=crossing | None | None | None | None | None |
| node/480492947 | POINT (-73.9665 40.7781) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=crossing | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| relation/10925251 | POLYGON ((-73.99006 40.77396, -73.98999 40.774... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| relation/9771573 | POLYGON ((-73.97507 40.76441, -73.97512 40.764... | tourism=hotel | building=hotel | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| relation/10078100 | POLYGON ((-73.99363 40.77349, -73.99362 40.773... | None | None | None | None | None | None | None | landuse=grass | None | ... | None | None | None | None | None | None | None | None | None | None |
| relation/7459068 | POLYGON ((-73.97729 40.77214, -73.97732 40.772... | None | None | amenity=restaurant | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| relation/4623434 | POLYGON ((-73.98243 40.77731, -73.98245 40.777... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
50955 rows × 27 columns
Plot features¶
Inspired by https://snazzymaps.com/style/14889/flat-pale
In [13]:
Copied!
ax = manhattan_bbox_gdf.plot(color="#e7e7df", figsize=(16, 16))
# plot greenery
new_york_features_gdf[new_york_features_gdf["leisure"] == "leisure=park"].plot(
ax=ax, color="#bae5ce"
)
# plot water
new_york_features_gdf.dropna(subset=["water", "waterways"], how="all").plot(
ax=ax, color="#c7eced"
)
# plot streets
new_york_features_gdf.dropna(subset=["paths_unsuitable_for_cars"], how="all").plot(
ax=ax, color="#e7e7df", linewidth=1
)
new_york_features_gdf.dropna(
subset=["very_small_roads", "highway_links", "minor_roads"], how="all"
).plot(ax=ax, color="#fff", linewidth=2)
new_york_features_gdf.dropna(subset=["major_roads"], how="all").plot(
ax=ax, color="#fac9a9", linewidth=3
)
# plot buildings
new_york_features_gdf.dropna(subset=["buildings"], how="all").plot(
ax=ax, color="#cecebd"
)
xmin, ymin, xmax, ymax = manhattan_bbox_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = manhattan_bbox_gdf.plot(color="#e7e7df", figsize=(16, 16))
# plot greenery
new_york_features_gdf[new_york_features_gdf["leisure"] == "leisure=park"].plot(
ax=ax, color="#bae5ce"
)
# plot water
new_york_features_gdf.dropna(subset=["water", "waterways"], how="all").plot(
ax=ax, color="#c7eced"
)
# plot streets
new_york_features_gdf.dropna(subset=["paths_unsuitable_for_cars"], how="all").plot(
ax=ax, color="#e7e7df", linewidth=1
)
new_york_features_gdf.dropna(
subset=["very_small_roads", "highway_links", "minor_roads"], how="all"
).plot(ax=ax, color="#fff", linewidth=2)
new_york_features_gdf.dropna(subset=["major_roads"], how="all").plot(
ax=ax, color="#fac9a9", linewidth=3
)
# plot buildings
new_york_features_gdf.dropna(subset=["buildings"], how="all").plot(
ax=ax, color="#cecebd"
)
xmin, ymin, xmax, ymax = manhattan_bbox_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Using OSMPbfLoader to download data for a specific area and transforming it to GeoParquet file¶
Download all grouped features based on Geofabrik layers in Reykjavík, Iceland¶
In [14]:
Copied!
loader = OSMPbfLoader()
reykjavik_gdf = geocode_to_region_gdf("Reykjavík, IS")
reykjavik_features_gpq = loader.load_to_geoparquet(reykjavik_gdf, GEOFABRIK_LAYERS)
reykjavik_features_gpq
loader = OSMPbfLoader()
reykjavik_gdf = geocode_to_region_gdf("Reykjavík, IS")
reykjavik_features_gpq = loader.load_to_geoparquet(reykjavik_gdf, GEOFABRIK_LAYERS)
reykjavik_features_gpq
0%| | 0.00/62.9M [00:00<?, ?B/s]
0%| | 12.3k/62.9M [00:00<13:18, 78.7kB/s]
0%| | 41.0k/62.9M [00:00<07:30, 139kB/s]
0%| | 98.3k/62.9M [00:00<04:19, 242kB/s]
0%|▏ | 209k/62.9M [00:00<02:28, 424kB/s]
1%|▎ | 438k/62.9M [00:00<01:18, 795kB/s]
1%|▌ | 893k/62.9M [00:00<00:41, 1.50MB/s]
3%|█ | 1.80M/62.9M [00:01<00:21, 2.90MB/s]
6%|██ | 3.61M/62.9M [00:01<00:10, 5.62MB/s]
10%|███▋ | 6.30M/62.9M [00:01<00:06, 9.19MB/s]
14%|█████▏ | 8.92M/62.9M [00:01<00:04, 11.4MB/s]
18%|██████▊ | 11.5M/62.9M [00:01<00:03, 12.9MB/s]
22%|████████▏ | 14.0M/62.9M [00:01<00:03, 13.9MB/s]
26%|█████████▋ | 16.6M/62.9M [00:02<00:03, 14.5MB/s]
30%|███████████▏ | 19.0M/62.9M [00:02<00:02, 14.8MB/s]
34%|████████████▌ | 21.4M/62.9M [00:02<00:02, 14.9MB/s]
38%|██████████████ | 23.8M/62.9M [00:02<00:02, 15.1MB/s]
42%|███████████████▍ | 26.3M/62.9M [00:02<00:02, 15.3MB/s]
46%|████████████████▉ | 28.7M/62.9M [00:02<00:02, 15.3MB/s]
50%|██████████████████▎ | 31.2M/62.9M [00:02<00:02, 15.4MB/s]
53%|███████████████████▊ | 33.6M/62.9M [00:03<00:01, 15.4MB/s]
57%|█████████████████████▏ | 36.1M/62.9M [00:03<00:01, 15.4MB/s]
61%|██████████████████████▋ | 38.5M/62.9M [00:03<00:01, 15.3MB/s]
65%|████████████████████████ | 40.9M/62.9M [00:03<00:01, 15.3MB/s]
69%|█████████████████████████▍ | 43.3M/62.9M [00:03<00:01, 15.3MB/s]
73%|██████████████████████████▊ | 45.7M/62.9M [00:03<00:01, 15.3MB/s]
76%|████████████████████████████▎ | 48.1M/62.9M [00:04<00:00, 15.2MB/s]
80%|█████████████████████████████▋ | 50.5M/62.9M [00:04<00:00, 15.2MB/s]
84%|███████████████████████████████ | 52.9M/62.9M [00:04<00:00, 15.2MB/s]
88%|████████████████████████████████▌ | 55.3M/62.9M [00:04<00:00, 15.2MB/s]
92%|█████████████████████████████████▉ | 57.7M/62.9M [00:04<00:00, 15.2MB/s]
96%|███████████████████████████████████▎ | 60.1M/62.9M [00:04<00:00, 15.2MB/s]
99%|████████████████████████████████████▊| 62.5M/62.9M [00:05<00:00, 15.3MB/s]
0%| | 0.00/62.9M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 62.9M/62.9M [00:00<00:00, 87.1GB/s]
Finished operation in 0:00:23
Out[14]:
PosixPath('files/372cc10dc2cb70fec59ce6093baa282a6c709187d7106ba7ccc97660a0426039_098931824b94bfe02e01bc4987422a39ec47bd3f6924e5325ddae0846badafba_exploded.parquet')
Read those features using DuckDB¶
In [15]:
Copied!
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_features_gpq))
features_relation
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_features_gpq))
features_relation
Out[15]:
┌─────────────────┬───────────────┬─────────────┬───────────┬──────────┬───────────┬──────────────┬─────────┬───────────────┬─────────┬─────────┬─────────────┬─────────────┬─────────┬─────────┬─────────┬───────────────────────────┬─────────┬─────────┬──────────┬──────────┬─────────┬────────────────────────┬───────────┬──────────────────┬─────────┬───────────────┬───────────┬────────────────────────────────┐ │ feature_id │ accommodation │ air_traffic │ buildings │ catering │ education │ fuel_parking │ health │ highway_links │ landuse │ leisure │ major_roads │ minor_roads │ miscpoi │ money │ natural │ paths_unsuitable_for_cars │ pofw │ public │ railways │ shopping │ tourism │ traffic │ transport │ very_small_roads │ water │ water_traffic │ waterways │ geometry │ │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ geometry │ ├─────────────────┼───────────────┼─────────────┼───────────┼──────────┼───────────┼──────────────┼─────────┼───────────────┼─────────┼─────────┼─────────────┼─────────────┼─────────┼─────────┼─────────┼───────────────────────────┼─────────┼─────────┼──────────┼──────────┼─────────┼────────────────────────┼───────────┼──────────────────┼─────────┼───────────────┼───────────┼────────────────────────────────┤ │ node/59644197 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8708536 64.1513895) │ │ node/59644347 │ NULL │ NULL │ NULL │ NULL │ NULL │ amenity=fuel │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8948347 64.1464783) │ │ node/59645175 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9070835 64.1470059) │ │ node/59981446 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=turning_circle │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8259246 64.1312551) │ │ node/59981804 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7926296 64.1366613) │ │ node/59982025 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8002203 64.1338684) │ │ node/59984122 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7943347 64.1403517) │ │ node/59984489 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7940459 64.1367439) │ │ node/59985139 │ NULL │ NULL │ NULL │ NULL │ NULL │ amenity=fuel │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.807943 64.1370366) │ │ node/60711683 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7891611 64.1520116) │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ node/2004267014 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8491985 64.1433871) │ │ node/2004267018 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8631412 64.1433962) │ │ node/2004267022 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8509876 64.1434015) │ │ node/2004267025 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8639426 64.1434017) │ │ node/2004267029 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.853323 64.143407) │ │ node/2004267033 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8815954 64.1434148) │ │ node/2004267072 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8536937 64.1434155) │ │ node/2004267073 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8822272 64.1434165) │ │ node/2004267074 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.854326 64.1434261) │ │ node/2004267075 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8562568 64.1434394) │ ├─────────────────┴───────────────┴─────────────┴───────────┴──────────┴───────────┴──────────────┴─────────┴───────────────┴─────────┴─────────┴─────────────┴─────────────┴─────────┴─────────┴─────────┴───────────────────────────┴─────────┴─────────┴──────────┴──────────┴─────────┴────────────────────────┴───────────┴──────────────────┴─────────┴───────────────┴───────────┴────────────────────────────────┤ │ ? rows (>9999 rows, 20 shown) 29 columns │ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Count all buildings¶
In [16]:
Copied!
features_relation.filter("buildings IS NOT NULL").count("feature_id")
features_relation.filter("buildings IS NOT NULL").count("feature_id")
Out[16]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 25372 │ └───────────────────┘
You can see more examples of how to use PbfFileReader from the QuackOSM library in the docs: https://kraina-ai.github.io/quackosm/latest/examples/pbf_file_reader/