OSM PBF Loader¶
OSMPbfLoader can really quickly parse full OSM extract in the form of *.osm.pbf file.
It can download and parse a lot of features much faster than the OSMOnlineLoader, but it's much more useful when a lot of different features are required at once (like when using predefined filters).
When only a single or few features are needed, OSMOnlineLoader might be a better choice, since OSMPbfLoader will use a full extract of all features in a given region and will have to iterate over all of them.
Run this notebook in Google Colab:
Remember to install the srai library before running the notebook:
%pip install srai[all]
In [1]:
Copied!
import geopandas as gpd
from shapely.geometry import Point, box
from srai.constants import REGIONS_INDEX, WGS84_CRS
from srai.geometry import buffer_geometry
from srai.loaders.osm_loaders import OSMPbfLoader
from srai.loaders.osm_loaders.filters import GEOFABRIK_LAYERS, HEX2VEC_FILTER
from srai.loaders.osm_loaders.filters.popular import get_popular_tags
from srai.regionalizers import geocode_to_region_gdf
import geopandas as gpd
from shapely.geometry import Point, box
from srai.constants import REGIONS_INDEX, WGS84_CRS
from srai.geometry import buffer_geometry
from srai.loaders.osm_loaders import OSMPbfLoader
from srai.loaders.osm_loaders.filters import GEOFABRIK_LAYERS, HEX2VEC_FILTER
from srai.loaders.osm_loaders.filters.popular import get_popular_tags
from srai.regionalizers import geocode_to_region_gdf
Using OSMPbfLoader to download data for a specific area¶
Download all features from HEX2VEC_FILTER in Warsaw, Poland¶
In [2]:
Copied!
loader = OSMPbfLoader()
warsaw_gdf = geocode_to_region_gdf("Warsaw, Poland")
warsaw_features_gdf = loader.load(warsaw_gdf, HEX2VEC_FILTER)
warsaw_features_gdf
loader = OSMPbfLoader()
warsaw_gdf = geocode_to_region_gdf("Warsaw, Poland")
warsaw_features_gdf = loader.load(warsaw_gdf, HEX2VEC_FILTER)
warsaw_features_gdf
0%| | 0.00/91.3M [00:00<?, ?B/s]
0%| | 32.8k/91.3M [00:00<05:00, 304kB/s]
0%| | 81.9k/91.3M [00:00<03:55, 387kB/s]
0%| | 197k/91.3M [00:00<02:13, 684kB/s]
0%|▏ | 426k/91.3M [00:00<01:13, 1.23MB/s]
1%|▎ | 885k/91.3M [00:00<00:39, 2.28MB/s]
2%|▋ | 1.80M/91.3M [00:00<00:20, 4.32MB/s]
4%|█▍ | 3.62M/91.3M [00:00<00:10, 8.28MB/s]
7%|██▌ | 6.46M/91.3M [00:00<00:06, 13.7MB/s]
10%|███▋ | 9.08M/91.3M [00:00<00:04, 16.8MB/s]
13%|████▊ | 12.0M/91.3M [00:01<00:04, 19.5MB/s]
16%|██████ | 14.8M/91.3M [00:01<00:03, 21.3MB/s]
19%|███████▏ | 17.7M/91.3M [00:01<00:03, 22.6MB/s]
22%|████████▎ | 20.5M/91.3M [00:01<00:03, 23.4MB/s]
26%|█████████▌ | 23.5M/91.3M [00:01<00:02, 24.2MB/s]
29%|██████████▋ | 26.4M/91.3M [00:01<00:02, 24.6MB/s]
32%|███████████▊ | 29.3M/91.3M [00:01<00:02, 24.8MB/s]
35%|█████████████ | 32.1M/91.3M [00:01<00:02, 25.0MB/s]
38%|██████████████▏ | 35.0M/91.3M [00:02<00:02, 25.1MB/s]
42%|███████████████▎ | 37.9M/91.3M [00:02<00:02, 25.2MB/s]
45%|████████████████▌ | 40.8M/91.3M [00:02<00:01, 25.3MB/s]
48%|█████████████████▋ | 43.7M/91.3M [00:02<00:01, 25.3MB/s]
51%|██████████████████▊ | 46.6M/91.3M [00:02<00:01, 25.4MB/s]
54%|████████████████████ | 49.4M/91.3M [00:02<00:01, 25.4MB/s]
57%|█████████████████████▏ | 52.3M/91.3M [00:02<00:01, 25.4MB/s]
60%|██████████████████████▍ | 55.2M/91.3M [00:02<00:01, 25.4MB/s]
64%|███████████████████████▌ | 58.0M/91.3M [00:02<00:01, 25.4MB/s]
67%|████████████████████████▋ | 60.9M/91.3M [00:03<00:01, 25.5MB/s]
70%|█████████████████████████▊ | 63.6M/91.3M [00:03<00:01, 25.2MB/s]
73%|██████████████████████████▉ | 66.5M/91.3M [00:03<00:00, 25.3MB/s]
76%|████████████████████████████ | 69.4M/91.3M [00:03<00:00, 25.4MB/s]
79%|█████████████████████████████▎ | 72.3M/91.3M [00:03<00:00, 25.5MB/s]
82%|██████████████████████████████▍ | 75.1M/91.3M [00:03<00:00, 25.5MB/s]
85%|███████████████████████████████▌ | 78.0M/91.3M [00:03<00:00, 25.6MB/s]
89%|████████████████████████████████▊ | 80.9M/91.3M [00:03<00:00, 25.8MB/s]
92%|█████████████████████████████████▉ | 83.8M/91.3M [00:03<00:00, 25.9MB/s]
95%|███████████████████████████████████ | 86.7M/91.3M [00:04<00:00, 25.9MB/s]
98%|████████████████████████████████████▎| 89.5M/91.3M [00:04<00:00, 25.9MB/s]
0%| | 0.00/91.3M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 91.3M/91.3M [00:00<00:00, 134GB/s]
Finished operation in 0:00:56
Out[2]:
| geometry | aeroway | amenity | building | healthcare | historic | landuse | leisure | military | natural | office | shop | sport | tourism | water | waterway | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | ||||||||||||||||
| node/1849727484 | POINT (20.9936 52.23076) | None | bank | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/1851381884 | POINT (21.07414 52.12393) | None | bench | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/1853996701 | POINT (21.04548 52.1492) | None | bicycle_rental | None | None | None | None | None | None | None | None | None | None | None | None | None |
| node/1855433181 | POINT (20.97508 52.20378) | None | None | None | None | None | None | None | None | None | None | convenience | None | None | None | None |
| node/1855479857 | POINT (20.96181 52.2113) | None | None | None | None | None | None | None | None | None | None | convenience | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| relation/13473201 | POLYGON ((20.968 52.26484, 20.96809 52.26496, ... | None | None | apartments | None | None | None | None | None | None | None | None | None | None | None | None |
| relation/12274648 | MULTIPOLYGON (((21.13682 52.24065, 21.13683 52... | None | None | apartments | None | None | None | None | None | None | None | None | None | None | None | None |
| relation/13155345 | POLYGON ((21.01428 52.21057, 21.01427 52.2106,... | None | None | None | None | None | None | track | None | None | None | None | running | None | None | None |
| relation/14028630 | MULTIPOLYGON (((21.10168 52.20114, 21.10167 52... | None | None | None | None | None | None | None | None | scrub | None | None | None | None | None | None |
| relation/11876626 | POLYGON ((20.99684 52.12547, 20.99676 52.12551... | None | None | None | None | None | None | None | None | scrub | None | None | None | None | None | None |
349973 rows × 16 columns
Plot features¶
Inspired by prettymaps
In [3]:
Copied!
clipped_features_gdf = warsaw_features_gdf.clip(warsaw_gdf.geometry.union_all())
clipped_features_gdf = warsaw_features_gdf.clip(warsaw_gdf.geometry.union_all())
In [4]:
Copied!
ax = warsaw_gdf.plot(color="lavender", figsize=(16, 16))
# plot water
clipped_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="deepskyblue"
)
# plot greenery
clipped_features_gdf[
clipped_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="mediumseagreen")
# plot buildings
clipped_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="dimgray", markersize=0.1
)
xmin, ymin, xmax, ymax = warsaw_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = warsaw_gdf.plot(color="lavender", figsize=(16, 16))
# plot water
clipped_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="deepskyblue"
)
# plot greenery
clipped_features_gdf[
clipped_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="mediumseagreen")
# plot buildings
clipped_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="dimgray", markersize=0.1
)
xmin, ymin, xmax, ymax = warsaw_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Download all features from popular tags based on OSMTagInfo in Vienna, Austria¶
In [5]:
Copied!
popular_tags = get_popular_tags(in_wiki_only=True)
num_keys = len(popular_tags)
f"Unique keys: {num_keys}."
popular_tags = get_popular_tags(in_wiki_only=True)
num_keys = len(popular_tags)
f"Unique keys: {num_keys}."
Out[5]:
'Unique keys: 380.'
In [6]:
Copied!
{k: popular_tags[k] for k in list(popular_tags)[:10]}
{k: popular_tags[k] for k in list(popular_tags)[:10]}
Out[6]:
{'4wd_only': ['yes'],
'LandPro08:reviewed': ['no'],
'abandoned': ['yes'],
'abandoned:railway': ['rail'],
'abutters': ['residential'],
'access': ['agricultural',
'customers',
'delivery',
'designated',
'destination',
'forestry',
'military',
'no',
'permissive',
'permit',
'private',
'unknown',
'yes'],
'access_control': ['full', 'partial'],
'addr:TW:dataset': ['137998'],
'addr:city': ['London', '臺北市'],
'addr:country': ['CZ', 'DE', 'RU', 'TW', 'US']}
In [7]:
Copied!
vienna_center_circle = buffer_geometry(Point(16.37009, 48.20931), meters=1000)
vienna_center_circle_gdf = gpd.GeoDataFrame(
geometry=[vienna_center_circle],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["Vienna"], name=REGIONS_INDEX),
)
vienna_center_circle = buffer_geometry(Point(16.37009, 48.20931), meters=1000)
vienna_center_circle_gdf = gpd.GeoDataFrame(
geometry=[vienna_center_circle],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["Vienna"], name=REGIONS_INDEX),
)
In [8]:
Copied!
loader = OSMPbfLoader()
vienna_features_gdf = loader.load(vienna_center_circle_gdf, popular_tags)
vienna_features_gdf
loader = OSMPbfLoader()
vienna_features_gdf = loader.load(vienna_center_circle_gdf, popular_tags)
vienna_features_gdf
0%| | 0.00/49.1M [00:00<?, ?B/s]
0%| | 8.19k/49.1M [00:00<10:17, 79.5kB/s]
0%| | 38.9k/49.1M [00:00<03:55, 208kB/s]
0%| | 93.2k/49.1M [00:00<02:18, 354kB/s]
0%|▏ | 179k/49.1M [00:00<01:29, 544kB/s]
1%|▎ | 364k/49.1M [00:00<00:49, 993kB/s]
2%|▌ | 749k/49.1M [00:00<00:25, 1.93MB/s]
3%|█▏ | 1.52M/49.1M [00:00<00:12, 3.74MB/s]
6%|██▏ | 2.88M/49.1M [00:00<00:06, 6.77MB/s]
11%|████▏ | 5.59M/49.1M [00:00<00:03, 12.9MB/s]
19%|██████▉ | 9.24M/49.1M [00:01<00:02, 19.9MB/s]
24%|████████▉ | 11.9M/49.1M [00:01<00:01, 21.6MB/s]
30%|███████████▏ | 14.8M/49.1M [00:01<00:01, 23.7MB/s]
37%|█████████████▌ | 18.0M/49.1M [00:01<00:01, 25.8MB/s]
42%|███████████████▋ | 20.8M/49.1M [00:01<00:01, 26.5MB/s]
48%|█████████████████▊ | 23.6M/49.1M [00:01<00:00, 26.7MB/s]
54%|████████████████████ | 26.7M/49.1M [00:01<00:00, 27.6MB/s]
60%|██████████████████████▎ | 29.6M/49.1M [00:01<00:00, 27.8MB/s]
66%|████████████████████████▍ | 32.5M/49.1M [00:01<00:00, 27.8MB/s]
72%|██████████████████████████▋ | 35.4M/49.1M [00:01<00:00, 28.1MB/s]
78%|████████████████████████████▊ | 38.3M/49.1M [00:02<00:00, 28.2MB/s]
84%|███████████████████████████████ | 41.2M/49.1M [00:02<00:00, 28.3MB/s]
90%|█████████████████████████████████▎ | 44.3M/49.1M [00:02<00:00, 29.0MB/s]
96%|███████████████████████████████████▌ | 47.2M/49.1M [00:02<00:00, 28.2MB/s]
0%| | 0.00/49.1M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 49.1M/49.1M [00:00<00:00, 76.3GB/s]
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/quackosm/pb f_file_reader.py:2612: UserWarning: Select clause contains more than 100 columns (found 380 columns). Query might fail with insufficient memory resources. Consider applying more restrictive OsmTagsFilter for parsing. warnings.warn(
Finished operation in 0:00:32
Out[8]:
| geometry | access | admin_level | advertising | amenity | area:highway | artwork_type | atm | barrier | bench | ... | tunnel | type | usage | vehicle | vending | waste | water | water_source | waterway | wheelchair | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | |||||||||||||||||||||
| node/33182886 | POINT (16.36631 48.20782) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33182965 | POINT (16.37033 48.20359) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33344361 | POINT (16.36484 48.2159) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/33344362 | POINT (16.36413 48.21558) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/34591548 | POINT (16.36967 48.20235) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| way/1443039524 | POLYGON ((16.36997 48.21788, 16.37 48.21789, 1... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1443039525 | POLYGON ((16.37003 48.21783, 16.37006 48.21784... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1443039526 | POLYGON ((16.36986 48.21798, 16.36989 48.21799... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1443039527 | POLYGON ((16.36992 48.21793, 16.36995 48.21794... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1443039528 | POLYGON ((16.3698 48.21803, 16.36983 48.21805,... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
23653 rows × 195 columns
Plot features¶
Uses default preset colours from prettymaps
In [9]:
Copied!
clipped_vienna_features_gdf = vienna_features_gdf.clip(vienna_center_circle)
clipped_vienna_features_gdf = vienna_features_gdf.clip(vienna_center_circle)
In [10]:
Copied!
ax = vienna_center_circle_gdf.plot(color="#F2F4CB", figsize=(16, 16))
# plot water
clipped_vienna_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="#a8e1e6"
)
# plot streets
clipped_vienna_features_gdf.dropna(subset=["highway"], how="all").plot(
ax=ax, color="#475657", markersize=0.1
)
# plot buildings
clipped_vienna_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="#FF5E5B"
)
# plot parkings
clipped_vienna_features_gdf[
(clipped_vienna_features_gdf["amenity"] == "parking")
| (clipped_vienna_features_gdf["highway"] == "pedestrian")
].plot(ax=ax, color="#2F3737", markersize=0.1)
# plot greenery
clipped_vienna_features_gdf[
clipped_vienna_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="#8BB174")
xmin, ymin, xmax, ymax = vienna_center_circle_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = vienna_center_circle_gdf.plot(color="#F2F4CB", figsize=(16, 16))
# plot water
clipped_vienna_features_gdf.dropna(subset=["water", "waterway"], how="all").plot(
ax=ax, color="#a8e1e6"
)
# plot streets
clipped_vienna_features_gdf.dropna(subset=["highway"], how="all").plot(
ax=ax, color="#475657", markersize=0.1
)
# plot buildings
clipped_vienna_features_gdf.dropna(subset=["building"], how="all").plot(
ax=ax, color="#FF5E5B"
)
# plot parkings
clipped_vienna_features_gdf[
(clipped_vienna_features_gdf["amenity"] == "parking")
| (clipped_vienna_features_gdf["highway"] == "pedestrian")
].plot(ax=ax, color="#2F3737", markersize=0.1)
# plot greenery
clipped_vienna_features_gdf[
clipped_vienna_features_gdf["landuse"].isin(
["grass", "orchard", "flowerbed", "forest", "greenfield", "meadow"]
)
].plot(ax=ax, color="#8BB174")
xmin, ymin, xmax, ymax = vienna_center_circle_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Download all grouped features based on Geofabrik layers in New York, USA¶
In [11]:
Copied!
manhattan_bbox = box(-73.994551, 40.762396, -73.936872, 40.804239)
manhattan_bbox_gdf = gpd.GeoDataFrame(
geometry=[manhattan_bbox],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["New York"], name=REGIONS_INDEX),
)
manhattan_bbox = box(-73.994551, 40.762396, -73.936872, 40.804239)
manhattan_bbox_gdf = gpd.GeoDataFrame(
geometry=[manhattan_bbox],
crs=WGS84_CRS,
index=gpd.pd.Index(data=["New York"], name=REGIONS_INDEX),
)
In [12]:
Copied!
loader = OSMPbfLoader()
new_york_features_gdf = loader.load(manhattan_bbox_gdf, GEOFABRIK_LAYERS)
new_york_features_gdf
loader = OSMPbfLoader()
new_york_features_gdf = loader.load(manhattan_bbox_gdf, GEOFABRIK_LAYERS)
new_york_features_gdf
/opt/hostedtoolcache/Python/3.10.18/x64/lib/python3.10/site-packages/quackosm/osm_extracts/__init__.py:592: GeometryNotCoveredWarning: Skipping extract because of low IoU value (bbbike_newyork, 0.000187). warnings.warn(
0%| | 0.00/98.0M [00:00<?, ?B/s]
0%| | 8.19k/98.0M [00:00<19:58, 81.8kB/s]
0%| | 38.9k/98.0M [00:00<07:44, 211kB/s]
0%| | 93.2k/98.0M [00:00<04:34, 356kB/s]
0%| | 207k/98.0M [00:00<02:29, 653kB/s]
0%|▏ | 436k/98.0M [00:00<01:19, 1.22MB/s]
1%|▎ | 891k/98.0M [00:00<00:42, 2.31MB/s]
2%|▋ | 1.80M/98.0M [00:00<00:21, 4.45MB/s]
4%|█▎ | 3.62M/98.0M [00:00<00:10, 8.66MB/s]
7%|██▋ | 7.20M/98.0M [00:00<00:05, 16.9MB/s]
11%|███▉ | 10.3M/98.0M [00:01<00:04, 20.7MB/s]
13%|████▉ | 13.2M/98.0M [00:01<00:03, 22.8MB/s]
16%|██████ | 16.0M/98.0M [00:01<00:03, 24.3MB/s]
20%|███████▏ | 19.1M/98.0M [00:01<00:03, 25.7MB/s]
23%|████████▍ | 22.4M/98.0M [00:01<00:02, 27.1MB/s]
26%|█████████▌ | 25.3M/98.0M [00:01<00:02, 27.2MB/s]
29%|██████████▋ | 28.4M/98.0M [00:01<00:02, 27.9MB/s]
32%|███████████▊ | 31.4M/98.0M [00:01<00:02, 27.9MB/s]
35%|████████████▉ | 34.2M/98.0M [00:01<00:02, 27.3MB/s]
38%|█████████████▉ | 36.9M/98.0M [00:01<00:02, 27.1MB/s]
40%|██████████████▉ | 39.6M/98.0M [00:02<00:02, 26.7MB/s]
43%|███████████████▉ | 42.3M/98.0M [00:02<00:02, 26.4MB/s]
46%|████████████████▉ | 44.9M/98.0M [00:02<00:02, 26.0MB/s]
49%|█████████████████▉ | 47.5M/98.0M [00:02<00:01, 25.9MB/s]
51%|██████████████████▉ | 50.1M/98.0M [00:02<00:01, 25.3MB/s]
54%|███████████████████▉ | 52.7M/98.0M [00:02<00:01, 25.2MB/s]
56%|████████████████████▊ | 55.2M/98.0M [00:02<00:01, 24.7MB/s]
59%|█████████████████████▊ | 57.7M/98.0M [00:02<00:01, 24.7MB/s]
61%|██████████████████████▋ | 60.1M/98.0M [00:02<00:01, 24.3MB/s]
64%|███████████████████████▌ | 62.6M/98.0M [00:03<00:01, 24.2MB/s]
66%|████████████████████████▌ | 65.0M/98.0M [00:03<00:01, 24.0MB/s]
69%|█████████████████████████▍ | 67.4M/98.0M [00:03<00:01, 24.0MB/s]
71%|██████████████████████████▎ | 69.8M/98.0M [00:03<00:01, 23.8MB/s]
74%|███████████████████████████▎ | 72.3M/98.0M [00:03<00:01, 24.1MB/s]
76%|████████████████████████████▏ | 74.7M/98.0M [00:03<00:00, 24.0MB/s]
79%|█████████████████████████████▏ | 77.2M/98.0M [00:03<00:00, 24.4MB/s]
83%|██████████████████████████████▋ | 81.3M/98.0M [00:03<00:00, 28.8MB/s]
86%|███████████████████████████████▊ | 84.2M/98.0M [00:03<00:00, 28.4MB/s]
89%|████████████████████████████████▊ | 87.0M/98.0M [00:03<00:00, 27.9MB/s]
92%|██████████████████████████████████ | 90.1M/98.0M [00:04<00:00, 28.7MB/s]
95%|███████████████████████████████████▏ | 93.3M/98.0M [00:04<00:00, 29.3MB/s]
98%|████████████████████████████████████▎| 96.2M/98.0M [00:04<00:00, 29.0MB/s]
0%| | 0.00/98.0M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 98.0M/98.0M [00:00<00:00, 105GB/s]
Finished operation in 0:00:35
Out[12]:
| geometry | accommodation | buildings | catering | education | fuel_parking | health | highway_links | landuse | leisure | ... | public | railways | shopping | tourism | traffic | transport | very_small_roads | water | water_traffic | waterways | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| feature_id | |||||||||||||||||||||
| node/4246602052 | POINT (-73.96684 40.77487) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/4682935365 | POINT (-73.99105 40.7727) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=stop | None | None | None | None | None |
| node/4688127189 | POINT (-73.96528 40.80134) | None | None | amenity=restaurant | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| node/4689234438 | POINT (-73.97461 40.78319) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=crossing | None | None | None | None | None |
| node/4689234439 | POINT (-73.97465 40.78332) | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | highway=crossing | None | None | None | None | None |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| way/1420126997 | LINESTRING (-73.98283 40.76684, -73.98288 40.7... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1420126998 | LINESTRING (-73.98288 40.76686, -73.98292 40.7... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1420129934 | LINESTRING (-73.98417 40.76989, -73.9842 40.76... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1420129935 | LINESTRING (-73.9842 40.76991, -73.98421 40.76... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | None | None | None | None |
| way/1420272260 | LINESTRING (-73.96935 40.79472, -73.96903 40.7... | None | None | None | None | None | None | None | None | None | ... | None | None | None | None | None | None | highway=service | None | None | None |
51040 rows × 27 columns
Plot features¶
Inspired by https://snazzymaps.com/style/14889/flat-pale
In [13]:
Copied!
ax = manhattan_bbox_gdf.plot(color="#e7e7df", figsize=(16, 16))
# plot greenery
new_york_features_gdf[new_york_features_gdf["leisure"] == "leisure=park"].plot(
ax=ax, color="#bae5ce"
)
# plot water
new_york_features_gdf.dropna(subset=["water", "waterways"], how="all").plot(
ax=ax, color="#c7eced"
)
# plot streets
new_york_features_gdf.dropna(subset=["paths_unsuitable_for_cars"], how="all").plot(
ax=ax, color="#e7e7df", linewidth=1
)
new_york_features_gdf.dropna(
subset=["very_small_roads", "highway_links", "minor_roads"], how="all"
).plot(ax=ax, color="#fff", linewidth=2)
new_york_features_gdf.dropna(subset=["major_roads"], how="all").plot(
ax=ax, color="#fac9a9", linewidth=3
)
# plot buildings
new_york_features_gdf.dropna(subset=["buildings"], how="all").plot(
ax=ax, color="#cecebd"
)
xmin, ymin, xmax, ymax = manhattan_bbox_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax = manhattan_bbox_gdf.plot(color="#e7e7df", figsize=(16, 16))
# plot greenery
new_york_features_gdf[new_york_features_gdf["leisure"] == "leisure=park"].plot(
ax=ax, color="#bae5ce"
)
# plot water
new_york_features_gdf.dropna(subset=["water", "waterways"], how="all").plot(
ax=ax, color="#c7eced"
)
# plot streets
new_york_features_gdf.dropna(subset=["paths_unsuitable_for_cars"], how="all").plot(
ax=ax, color="#e7e7df", linewidth=1
)
new_york_features_gdf.dropna(
subset=["very_small_roads", "highway_links", "minor_roads"], how="all"
).plot(ax=ax, color="#fff", linewidth=2)
new_york_features_gdf.dropna(subset=["major_roads"], how="all").plot(
ax=ax, color="#fac9a9", linewidth=3
)
# plot buildings
new_york_features_gdf.dropna(subset=["buildings"], how="all").plot(
ax=ax, color="#cecebd"
)
xmin, ymin, xmax, ymax = manhattan_bbox_gdf.total_bounds
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
Using OSMPbfLoader to download data for a specific area and transforming it to GeoParquet file¶
Download all grouped features based on Geofabrik layers in Reykjavík, Iceland¶
In [14]:
Copied!
loader = OSMPbfLoader()
reykjavik_gdf = geocode_to_region_gdf("Reykjavík, IS")
reykjavik_features_gpq = loader.load_to_geoparquet(reykjavik_gdf, GEOFABRIK_LAYERS)
reykjavik_features_gpq
loader = OSMPbfLoader()
reykjavik_gdf = geocode_to_region_gdf("Reykjavík, IS")
reykjavik_features_gpq = loader.load_to_geoparquet(reykjavik_gdf, GEOFABRIK_LAYERS)
reykjavik_features_gpq
0%| | 0.00/62.9M [00:00<?, ?B/s]
0%| | 12.3k/62.9M [00:00<10:52, 96.3kB/s]
0%| | 41.0k/62.9M [00:00<06:08, 170kB/s]
0%| | 98.3k/62.9M [00:00<03:32, 295kB/s]
0%|▏ | 209k/62.9M [00:00<02:01, 516kB/s]
1%|▎ | 438k/62.9M [00:00<01:04, 969kB/s]
1%|▌ | 893k/62.9M [00:00<00:33, 1.83MB/s]
3%|█ | 1.80M/62.9M [00:00<00:17, 3.52MB/s]
6%|██▏ | 3.62M/62.9M [00:01<00:08, 6.88MB/s]
10%|███▋ | 6.28M/62.9M [00:01<00:05, 11.1MB/s]
14%|█████▏ | 8.87M/62.9M [00:01<00:03, 13.9MB/s]
18%|██████▊ | 11.5M/62.9M [00:01<00:03, 15.9MB/s]
22%|████████▎ | 14.2M/62.9M [00:01<00:02, 17.3MB/s]
27%|█████████▊ | 16.7M/62.9M [00:01<00:02, 18.1MB/s]
31%|███████████▎ | 19.3M/62.9M [00:01<00:02, 18.6MB/s]
35%|████████████▊ | 21.9M/62.9M [00:01<00:02, 18.9MB/s]
39%|██████████████▎ | 24.4M/62.9M [00:02<00:02, 19.2MB/s]
43%|███████████████▊ | 27.0M/62.9M [00:02<00:01, 19.3MB/s]
47%|█████████████████▎ | 29.5M/62.9M [00:02<00:01, 19.3MB/s]
51%|██████████████████▊ | 32.0M/62.9M [00:02<00:01, 19.4MB/s]
55%|████████████████████▏ | 34.4M/62.9M [00:02<00:01, 19.1MB/s]
59%|█████████████████████▋ | 36.8M/62.9M [00:02<00:01, 19.0MB/s]
62%|███████████████████████ | 39.3M/62.9M [00:02<00:01, 19.0MB/s]
66%|████████████████████████▍ | 41.7M/62.9M [00:02<00:01, 18.8MB/s]
70%|█████████████████████████▉ | 44.0M/62.9M [00:03<00:01, 18.6MB/s]
74%|███████████████████████████▎ | 46.4M/62.9M [00:03<00:00, 18.7MB/s]
78%|████████████████████████████▊ | 48.9M/62.9M [00:03<00:00, 18.8MB/s]
82%|██████████████████████████████▏ | 51.4M/62.9M [00:03<00:00, 18.7MB/s]
86%|███████████████████████████████▋ | 53.8M/62.9M [00:03<00:00, 18.9MB/s]
89%|█████████████████████████████████ | 56.3M/62.9M [00:03<00:00, 18.9MB/s]
93%|██████████████████████████████████▌ | 58.7M/62.9M [00:03<00:00, 18.8MB/s]
97%|███████████████████████████████████▉ | 61.2M/62.9M [00:04<00:00, 19.0MB/s]
0%| | 0.00/62.9M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 62.9M/62.9M [00:00<00:00, 78.7GB/s]
Finished operation in 0:00:23
Out[14]:
PosixPath('files/372cc10dc2cb70fec59ce6093baa282a6c709187d7106ba7ccc97660a0426039_098931824b94bfe02e01bc4987422a39ec47bd3f6924e5325ddae0846badafba_exploded.parquet')
Read those features using DuckDB¶
In [15]:
Copied!
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_features_gpq))
features_relation
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_features_gpq))
features_relation
Out[15]:
┌─────────────────┬───────────────┬─────────────┬───────────┬────────────────────┬───────────┬──────────────┬─────────┬───────────────┬─────────┬─────────┬─────────────┬─────────────┬─────────┬─────────┬─────────┬───────────────────────────┬─────────┬─────────┬──────────┬──────────────────┬─────────────────────┬─────────────────────┬───────────┬──────────────────┬─────────┬───────────────┬───────────┬────────────────────────────────┐ │ feature_id │ accommodation │ air_traffic │ buildings │ catering │ education │ fuel_parking │ health │ highway_links │ landuse │ leisure │ major_roads │ minor_roads │ miscpoi │ money │ natural │ paths_unsuitable_for_cars │ pofw │ public │ railways │ shopping │ tourism │ traffic │ transport │ very_small_roads │ water │ water_traffic │ waterways │ geometry │ │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │ geometry │ ├─────────────────┼───────────────┼─────────────┼───────────┼────────────────────┼───────────┼──────────────┼─────────┼───────────────┼─────────┼─────────┼─────────────┼─────────────┼─────────┼─────────┼─────────┼───────────────────────────┼─────────┼─────────┼──────────┼──────────────────┼─────────────────────┼─────────────────────┼───────────┼──────────────────┼─────────┼───────────────┼───────────┼────────────────────────────────┤ │ node/288037466 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9592124 64.1386272) │ │ node/288037479 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9633126 64.1389854) │ │ node/288037485 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9623437 64.140269) │ │ node/288037508 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.964693 64.1407153) │ │ node/288037511 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9520576 64.1367153) │ │ node/288037537 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.9587812 64.1362093) │ │ node/304852646 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ tourism=picnic_site │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7698095 64.1114289) │ │ node/305004175 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ shop=convenience │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8528399 64.1302889) │ │ node/306002513 │ NULL │ NULL │ NULL │ amenity=restaurant │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8107534 64.1289449) │ │ node/306333990 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=crossing │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8749093 64.1297008) │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ · │ │ node/2004218796 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8055134 64.1068747) │ │ node/2004218798 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8059629 64.1068772) │ │ node/2004218799 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8064683 64.1068834) │ │ node/2004218801 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8069884 64.1068858) │ │ node/2004218803 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8260288 64.1068908) │ │ node/2004218804 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8076694 64.1068996) │ │ node/2004218806 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.8173134 64.1069132) │ │ node/2004218807 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.785184 64.1069136) │ │ node/2004218809 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7946755 64.106921) │ │ node/2004218810 │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ NULL │ highway=street_lamp │ NULL │ NULL │ NULL │ NULL │ NULL │ POINT (-21.7916464 64.1069214) │ ├─────────────────┴───────────────┴─────────────┴───────────┴────────────────────┴───────────┴──────────────┴─────────┴───────────────┴─────────┴─────────┴─────────────┴─────────────┴─────────┴─────────┴─────────┴───────────────────────────┴─────────┴─────────┴──────────┴──────────────────┴─────────────────────┴─────────────────────┴───────────┴──────────────────┴─────────┴───────────────┴───────────┴────────────────────────────────┤ │ ? rows (>9999 rows, 20 shown) 29 columns │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Count all buildings¶
In [16]:
Copied!
features_relation.filter("buildings IS NOT NULL").count("feature_id")
features_relation.filter("buildings IS NOT NULL").count("feature_id")
Out[16]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 25372 │ └───────────────────┘
You can see more examples of how to use PbfFileReader from the QuackOSM library in the docs: https://kraina-ai.github.io/quackosm/latest/examples/pbf_file_reader/