PBF File Reader¶
PBFFileReader can really quickly parse full OSM extract in the form of *.osm.pbf file.
It uses DuckDB with spatial extension to convert pbf files into geoparquet files without GDAL dependency.
Reader can filter objects by geometry and by OSM tags with option to split tags into columns or keep it as a single dictionary.
Caching strategy is implemented to reduce computations, but it can be overriden using ignore_cache parameter.
Download all buildings in Reykjavík, Iceland¶
Filtering the data with geometry and by tags, with tags in exploded form
Run this notebook in Google Colab:
Remember to install the quackosm library before running the notebook:
%pip install quackosm[cli]
In [1]:
Copied!
import urllib.request
from quackosm import PbfFileReader, geocode_to_geometry
import urllib.request
from quackosm import PbfFileReader, geocode_to_geometry
In [2]:
Copied!
iceland_pbf_url = "https://download.geofabrik.de/europe/iceland-latest.osm.pbf"
iceland_pbf_file = "iceland.osm.pbf"
urllib.request.urlretrieve(iceland_pbf_url, iceland_pbf_file)
iceland_pbf_url = "https://download.geofabrik.de/europe/iceland-latest.osm.pbf"
iceland_pbf_file = "iceland.osm.pbf"
urllib.request.urlretrieve(iceland_pbf_url, iceland_pbf_file)
Out[2]:
('iceland.osm.pbf', <http.client.HTTPMessage at 0x7fe32138a060>)
In [3]:
Copied!
reykjavik_geometry = geocode_to_geometry("Reykjavík, IS")
reykjavik_geometry
reykjavik_geometry = geocode_to_geometry("Reykjavík, IS")
reykjavik_geometry
Out[3]:
To filter out buildings, we will utilize format used also in the osmnx library: a dictionary with keys representing tag keys and values that could be a bool, string or a list of string.
By default, QuackOSM will return only the tags that are present in the passed filter.
In this example we will select all the buildings using { "building": True } filter and only building tag values will be present in the result.
In [4]:
Copied!
reader = PbfFileReader(
geometry_filter=reykjavik_geometry, tags_filter={"building": True}
)
reykjavik_buildings_gpq = reader.convert_pbf_to_parquet("iceland.osm.pbf")
reykjavik_buildings_gpq
reader = PbfFileReader(
geometry_filter=reykjavik_geometry, tags_filter={"building": True}
)
reykjavik_buildings_gpq = reader.convert_pbf_to_parquet("iceland.osm.pbf")
reykjavik_buildings_gpq
Finished operation in 0:00:11
Out[4]:
PosixPath('files/iceland_ae99e3d9_4e8a6f65_exploded_sorted.parquet')
Read those features using DuckDB¶
In [5]:
Copied!
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_buildings_gpq))
features_relation
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_buildings_gpq))
features_relation
Out[5]:
┌───────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────┐
│ feature_id │ geometry │ building │
│ varchar │ geometry('ogc:crs84') │ varchar │
├───────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────┤
│ way/671607155 │ POLYGON ((-21.7893832 64.0513494, -21.7892828 64.0513882, -21.7895674 64.0515295, -21.7896664 64.0514913, -21.7895539 64.0514354, -21.7896311 64.0514057, -21.7895352 64.0513581, -21.7894594 64.0513873, -21.7893832 64.0513494)) │ yes │
│ way/671607153 │ POLYGON ((-21.785565 64.0518193, -21.7854812 64.0518449, -21.7855606 64.0518945, -21.7856443 64.0518689, -21.785565 64.0518193)) │ yes │
│ way/668334022 │ POLYGON ((-21.7290748 64.0677094, -21.7289315 64.0677504, -21.7290516 64.0678308, -21.7291949 64.0677898, -21.7290748 64.0677094)) │ yes │
│ way/668337377 │ POLYGON ((-21.7443484 64.0735764, -21.7442615 64.0735948, -21.7443022 64.0736317, -21.7443891 64.0736134, -21.7443484 64.0735764)) │ yes │
│ way/670007221 │ POLYGON ((-21.7658076 64.0756955, -21.7658067 64.0757663, -21.7660254 64.0757668, -21.7660263 64.075696, -21.7658076 64.0756955)) │ yes │
│ way/366744730 │ POLYGON ((-21.7724858 64.0767213, -21.7723384 64.0767463, -21.7722604 64.0766586, -21.7724077 64.0766336, -21.7724313 64.07666, -21.7724713 64.0766532, -21.7724978 64.076683, -21.7724578 64.0766898, -21.7724858 64.0767213)) │ yes │
│ way/508499993 │ POLYGON ((-21.7744719 64.078054, -21.7744139 64.0780667, -21.7743643 64.0780233, -21.7744222 64.0780106, -21.7744719 64.078054)) │ yes │
│ way/366744729 │ POLYGON ((-21.7766302 64.0789397, -21.7764969 64.0789688, -21.7764483 64.0789263, -21.7765815 64.0788972, -21.7766302 64.0789397)) │ yes │
│ way/508500012 │ POLYGON ((-21.787494 64.0805525, -21.7875199 64.0805852, -21.7874234 64.0805998, -21.7873974 64.0805672, -21.787494 64.0805525)) │ yes │
│ way/508499994 │ POLYGON ((-21.7870689 64.0808515, -21.7870851 64.0808399, -21.7871417 64.0808549, -21.7870593 64.0809141, -21.786971 64.0808904, -21.7868658 64.0808625, -21.786932 64.0808151, -21.7870689 64.0808515)) │ yes │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ way/109031203 │ POLYGON ((-21.8522376 64.1291558, -21.8520485 64.1292501, -21.8519548 64.1292143, -21.8519879 64.1291978, -21.8519595 64.129187, -21.8519019 64.1292157, -21.8517894 64.1291728, -21.8518468 64.1291441, -21.8518603 64.1291493, -21.8518789 64.12914, -21.8518095 64.1291135, -21.851838 64.1290993, -21.8518065 64.1290873, -21.8519157 64.1290329, -21.8522376 64.1291558)) │ residential │
│ way/105016207 │ POLYGON ((-21.9488127 64.1290946, -21.9487064 64.1290693, -21.9485738 64.1291754, -21.9487466 64.1292165, -21.948812 64.1291642, -21.9487455 64.1291484, -21.9488127 64.1290946)) │ yes │
│ way/844013582 │ POLYGON ((-21.8467021 64.1290744, -21.8466086 64.1291559, -21.8468957 64.1292148, -21.8469742 64.1291395, -21.8467021 64.1290744)) │ yes │
│ way/104964231 │ POLYGON ((-21.9500135 64.1292308, -21.9496476 64.1291427, -21.9497089 64.1290942, -21.9497362 64.1291008, -21.9497717 64.1290727, -21.9501103 64.1291542, -21.9500135 64.1292308)) │ yes │
│ way/108362257 │ POLYGON ((-21.8840584 64.1291849, -21.8843592 64.1292328, -21.8844535 64.12912, -21.8841528 64.1290721, -21.8840584 64.1291849)) │ residential │
│ way/107649857 │ POLYGON ((-21.7589277 64.1291994, -21.7587444 64.1292132, -21.7587057 64.1291154, -21.758889 64.1291016, -21.7589277 64.1291994)) │ residential │
│ way/109031111 │ POLYGON ((-21.8580266 64.1291974, -21.8579459 64.1291675, -21.8579863 64.1291468, -21.8577817 64.129071, -21.8577173 64.1291041, -21.8576916 64.1290946, -21.8576641 64.1291088, -21.857671 64.1291114, -21.8576212 64.1291369, -21.8576373 64.1291429, -21.8575992 64.1291626, -21.8577028 64.129201, -21.8577347 64.1291846, -21.8578337 64.1292212, -21.8578452 64.1292153, -21.8579304 64.1292468, -21.8580266 64.1291974)) │ residential │
│ way/102590049 │ POLYGON ((-21.951944 64.1291842, -21.9519783 64.129159, -21.9519578 64.1291537, -21.9520099 64.1291154, -21.9520475 64.1291251, -21.9520958 64.1290897, -21.9522674 64.1291341, -21.952171 64.129205, -21.9521535 64.1292004, -21.9521152 64.1292286, -21.951944 64.1291842)) │ yes │
│ way/107756907 │ POLYGON ((-21.7535666 64.1292208, -21.753405 64.1291989, -21.7533962 64.1292111, -21.7533226 64.1292012, -21.7533321 64.1291879, -21.753277 64.1291805, -21.753292 64.1291594, -21.7532639 64.1291556, -21.7532933 64.1291142, -21.7533033 64.1291002, -21.7534113 64.1291149, -21.7534004 64.1291302, -21.7536108 64.1291587, -21.7535666 64.1292208)) │ residential │
│ way/516193601 │ POLYGON ((-21.9309649 64.1293002, -21.9309693 64.129321, -21.930907 64.1293235, -21.9308935 64.1293185, -21.9307978 64.129283, -21.930774 64.1292742, -21.9307271 64.129055, -21.9308431 64.1290503, -21.9308363 64.1290182, -21.9313146 64.1289987, -21.9313302 64.1290717, -21.9314664 64.1290662, -21.9315116 64.1292779, -21.9309649 64.1293002)) │ yes │
└───────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┘
? rows (>9999 rows, 20 shown) 3 columns
Count all buildings¶
In [6]:
Copied!
features_relation.count("feature_id")
features_relation.count("feature_id")
Out[6]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 25521 │ └───────────────────┘
Keeping all the tags while filtering the data¶
To keep all of the tags present in the source data, we can use keep_all_tags parameter. That way we will still return only buildings, but with all of the tags attached.
By default, all of those tags will be kept in a single column as a dict.
In [7]:
Copied!
reader.convert_pbf_to_geodataframe("iceland.osm.pbf", keep_all_tags=True)
reader.convert_pbf_to_geodataframe("iceland.osm.pbf", keep_all_tags=True)
Finished operation in 0:00:11
Out[7]:
| tags | geometry | |
|---|---|---|
| feature_id | ||
| way/671607155 | {'building': 'yes', 'roof:colour': 'green', 'r... | POLYGON ((-21.78938 64.05135, -21.78928 64.051... |
| way/671607153 | {'building': 'yes', 'roof:colour': 'green', 'r... | POLYGON ((-21.78556 64.05182, -21.78548 64.051... |
| way/668334022 | {'building': 'yes', 'name': 'Torgeirsstaðir'} | POLYGON ((-21.72907 64.06771, -21.72893 64.067... |
| way/668337377 | {'building': 'yes'} | POLYGON ((-21.74435 64.07358, -21.74426 64.073... |
| way/670007221 | {'building': 'yes', 'roof:colour': 'green', 'r... | POLYGON ((-21.76581 64.0757, -21.76581 64.0757... |
| ... | ... | ... |
| way/499098406 | {'building': 'yes'} | POLYGON ((-21.79544 64.29633, -21.79536 64.296... |
| way/351976584 | {'building': 'yes', 'start_date': '1986'} | POLYGON ((-21.8061 64.29683, -21.80597 64.2967... |
| way/351976596 | {'addr:housename': 'Laufbrekka', 'addr:postcod... | POLYGON ((-21.79557 64.297, -21.79539 64.29697... |
| way/351976595 | {'building': 'yes', 'start_date': '2005'} | POLYGON ((-21.79492 64.29698, -21.79476 64.297... |
| way/812660542 | {'building': 'yes'} | POLYGON ((-21.80953 64.29755, -21.80929 64.297... |
25521 rows × 2 columns
Download main roads for Estonia¶
Filtering the data only by tags, with tags in exploded form
In [8]:
Copied!
highways_filter = {
"highway": [
"motorway",
"trunk",
"primary",
"secondary",
"tertiary",
]
}
highways_filter = {
"highway": [
"motorway",
"trunk",
"primary",
"secondary",
"tertiary",
]
}
In [9]:
Copied!
estonia_pbf_url = "http://download.geofabrik.de/europe/estonia-latest.osm.pbf"
estonia_pbf_file = "estonia.osm.pbf"
urllib.request.urlretrieve(estonia_pbf_url, estonia_pbf_file)
reader = PbfFileReader(geometry_filter=None, tags_filter=highways_filter)
estonia_features_gpq = reader.convert_pbf_to_parquet(estonia_pbf_file)
estonia_features_gpq
estonia_pbf_url = "http://download.geofabrik.de/europe/estonia-latest.osm.pbf"
estonia_pbf_file = "estonia.osm.pbf"
urllib.request.urlretrieve(estonia_pbf_url, estonia_pbf_file)
reader = PbfFileReader(geometry_filter=None, tags_filter=highways_filter)
estonia_features_gpq = reader.convert_pbf_to_parquet(estonia_pbf_file)
estonia_features_gpq
Finished operation in 0:00:14
Out[9]:
PosixPath('files/estonia_87e7b741_noclip_exploded_sorted.parquet')
In [10]:
Copied!
features_relation = connection.read_parquet(str(estonia_features_gpq))
features_relation
features_relation = connection.read_parquet(str(estonia_features_gpq))
features_relation
Out[10]:
┌────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────┐
│ feature_id │ geometry │ highway │
│ varchar │ geometry('ogc:crs84') │ varchar │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────┤
│ way/36872845 │ LINESTRING (27.2145593 57.5437215, 27.2194975 57.5455473, 27.2199616 57.5457464, 27.2203307 57.5459478, 27.2205615 57.546123, 27.2207955 57.5463688, 27.2210439 57.5467076, 27.2222459 57.5487383, 27.2233359 57.5504811, 27.2235732 57.5509167, 27.2236975 57.5512212, 27.2237427 57.5515224, 27.2237557 57.5518294, 27.2234787 57.5535, 27.2234177 57.5536613, 27.2233139 57.5538456, 27.2228174 57.5543309) │ tertiary │
│ way/44077674 │ LINESTRING (26.6829371 57.5449717, 26.6828062 57.5450309, 26.6826495 57.5450835, 26.6824839 57.5451207, 26.682169 57.5451766, 26.6816509 57.5452207, 26.6815987 57.5452259, 26.6810758 57.5452783, 26.6805316 57.5453458, 26.680415 57.5453656, 26.6792501 57.5455918, 26.678372 57.5457753, 26.6775269 57.545982, 26.676824 57.5461784, 26.6761706 57.5463871, 26.6757913 57.5465222, 26.6752945 57.5467094, 26.6743075 57.5471138, 26.6737475 57.5474094, 26.6731038 57.5477122, 26.6727903 57.5478475, 26.6724522 57.5479695, 26.6716688 57.5481974, 26.6708187 57.5484458, 26.670529 57.5485505, 26.6703059 57.5486482, 26.6700868 57.5487791, 26.6695149 57.5491421, 26.6694443 57.5491947, 26.6691873 57.5493864, 26.6690475 57.5495267, 26.6689241 57.5497001, 26.6688722 57.549804, 26.668849 57.5499168, 26.668834 57.5500536, 26.6688437 57.5502059, 26.6690315 57.5509594, 26.6690413 57.5511128, 26.669016 57.5512927, 26.6689245 57.5514913, 26.6688185 57.551648, 26.6687122 57.5517518, 26.6684842 57.551933, 26.6682222 57.5520726, 26.6681643 57.5521024, 26.6675048 57.5523811, 26.6673011 57.5524729, 26.6669947 57.5526641, 26.6667158 57.5529156, 26.6665048 57.5531316, 26.6664155 57.5532663) │ secondary │
│ way/182788869 │ LINESTRING (26.5560824 57.5525312, 26.5563448 57.5523857, 26.5564444 57.5523305, 26.5565409 57.5521687, 26.556553 57.551832, 26.556552 57.5512833, 26.5566496 57.5500773, 26.556722 57.5497795, 26.5568692 57.5495695, 26.5571295 57.5494091, 26.5575561 57.5492913, 26.5612206 57.5486969, 26.5668586 57.5477916, 26.5674258 57.5477333, 26.5681016 57.5477916, 26.5689325 57.5479511, 26.5699398 57.5481445, 26.5717098 57.5484844, 26.5721201 57.5484974, 26.57247 57.5484262, 26.5736647 57.5477009, 26.5758127 57.5464576) │ tertiary │
│ way/123588635 │ LINESTRING (26.3624165 57.5804431, 26.3626787 57.5802237, 26.3630697 57.5799531, 26.3638655 57.5794878, 26.3647737 57.5790056, 26.3655597 57.5785865, 26.3662321 57.5782166, 26.3668959 57.5778444, 26.3680224 57.5772427, 26.3687778 57.5768002, 26.3691053 57.5765661, 26.3695376 57.5762079, 26.3697385 57.5759386, 26.3698738 57.5757068, 26.3700922 57.5753018, 26.3702974 57.5749178, 26.370507 57.5745549, 26.3708738 57.573897, 26.3711532 57.57341, 26.3715288 57.5728152, 26.371865 57.5725179, 26.3725331 57.5720847, 26.373354 57.571586, 26.3739085 57.5712371, 26.374712 57.5707477, 26.375332 57.570366, 26.3760263 57.5699375, 26.3766507 57.5695534, 26.3777511 57.5689259, 26.3782707 57.5686425, 26.3791353 57.568132, 26.3797685 57.5677526, 26.3808033 57.5673194, 26.3814714 57.5670431, 26.3831482 57.5663944, 26.3844014 57.5659049, 26.3856327 57.5654318, 26.386327 57.5651087, 26.3866589 57.5649307, 26.3872309 57.5644951, 26.3880256 57.5638651, 26.3889557 57.563132, 26.3891435 57.5629259, 26.3892133 57.5627596, 26.3892308 57.5625582, 26.3892221 57.5623966, 26.3891347 57.5621787, 26.3887636 57.5616143, 26.388423 57.5611084, 26.3874187 57.5596421, 26.3872047 57.5592252, 26.3871698 57.5589909, 26.387196 57.5587825, 26.3873663 57.5585506, 26.387637 57.5583374, 26.3883356 57.5579861, 26.3891302 57.557604, 26.3892963 57.5575105, 26.3896238 57.5572107, 26.3897591 57.5569741, 26.3898683 57.5565642, 26.3900037 57.5556459, 26.3901347 57.5551376, 26.3902831 57.5547979, 26.3906325 57.5543949, 26.3909643 57.5539592, 26.3912132 57.5536734, 26.3916106 57.5531674, 26.3918769 57.5528371, 26.3921782 57.5524505, 26.3924795 57.5520335, 26.3925799 57.5518133, 26.3928943 57.5513049, 26.3930559 57.5509722, 26.3934838 57.5501334, 26.393689 57.5497023, 26.3942392 57.5485917, 26.3948767 57.5473054, 26.3952741 57.5464267, 26.395523 57.5459909, 26.3957128 57.5457922, 26.395785 57.5457167, 26.3960251 57.5455176, 26.39706 57.5448005, 26.3973962 57.5445334, 26.3974879 57.5444326, 26.3976233 57.5441866, 26.3976233 57.5433289, 26.3976146 57.5424361, 26.3976277 57.5417495, 26.3975884 57.5413089, 26.3974355 57.5408285, 26.3973307 57.540409, 26.397226 57.5400668, 26.3971212 57.5398676, 26.3968048 57.5396315, 26.3965055 57.5394176, 26.3961131 57.539116, 26.3960724 57.5387188, 26.3961247 57.5384622, 26.3964466 57.5375665, 26.3967672 57.5369824, 26.3968939 57.536691, 26.3970383 57.5364666, 26.3971358 57.5355897, 26.3971801 57.5351919, 26.3971927 57.5350783, 26.3972805 57.5349728, 26.3982904 57.5337584, 26.3989111 57.533012, 26.399245 57.5328025, 26.3997513 57.5325656, 26.4002393 57.5322957, 26.400666 57.5320233, 26.4009456 57.5317338, 26.4010901 57.5312535, 26.4011316 57.5305976, 26.4010418 57.5298309, 26.4009973 57.5292818, 26.4009757 57.5283642, 26.4007008 57.5273207, 26.4002781 57.5257159, 26.399904 57.524158, 26.3995969 57.5229545, 26.3994853 57.5226434, 26.3994353 57.5225603, 26.3992176 57.5223293, 26.3989872 57.522109, 26.3987544 57.5218999, 26.3987178 57.5218572) │ secondary │
│ way/182788870 │ LINESTRING (26.5555031 57.5528161, 26.5560824 57.5525312) │ tertiary │
│ way/1034798159 │ LINESTRING (27.2216939 57.5623301, 27.2216059 57.562141, 27.2213297 57.5611197, 27.2211675 57.5609002, 27.2207224 57.5604574, 27.2206186 57.5602662, 27.2204822 57.5599092, 27.2204895 57.5597448, 27.2205741 57.5595847, 27.2206868 57.5594828, 27.2212142 57.5591889, 27.2213629 57.5590695, 27.2214871 57.5589501, 27.2216715 57.5585371, 27.2217014 57.5583466, 27.2216249 57.5581666, 27.2214507 57.557956, 27.2206895 57.5568372, 27.2206009 57.5566134, 27.2205773 57.5564244, 27.2206381 57.5562008, 27.2210526 57.5555192, 27.2213197 57.5552709, 27.221992 57.5548525, 27.2223582 57.5546554, 27.2228174 57.5543309) │ tertiary │
│ way/349875145 │ LINESTRING (26.6618721 57.5644661, 26.6625061 57.5642172, 26.6646336 57.5634051, 26.6654207 57.5631114, 26.665748 57.5629672, 26.6660607 57.5627876, 26.6662781 57.562612, 26.6664435 57.5624169, 26.6666821 57.5620397, 26.6669135 57.5615823, 26.6669898 57.5613778, 26.667044 57.5611146, 26.6670496 57.5607521, 26.6669618 57.5603699, 26.6667163 57.5594662, 26.6666543 57.5591751, 26.666606 57.5589291, 26.6665894 57.5584941, 26.6667944 57.5575564, 26.6672396 57.5558269, 26.6672481 57.5555388, 26.6671943 57.5552393, 26.6671186 57.5550027, 26.6666077 57.5539711, 26.6665383 57.5538288, 26.6664505 57.5536489, 26.6664132 57.5535318, 26.666408 57.5534957, 26.6663924 57.5533868, 26.6664069 57.5532868, 26.6664155 57.5532663) │ secondary │
│ way/1034798156 │ LINESTRING (27.222079 57.5626282, 27.2218649 57.5624974, 27.2216939 57.5623301) │ tertiary │
│ way/182788871 │ LINESTRING (26.5235403 57.5744752, 26.5240664 57.5743289, 26.5244656 57.5742227, 26.5278912 57.5732265, 26.5282267 57.5731289, 26.5288162 57.5728634, 26.5289813 57.5726421, 26.5290285 57.5722944, 26.5295119 57.5717696, 26.5337067 57.5680778, 26.5348009 57.5671148, 26.5353308 57.5666484, 26.5384974 57.564783, 26.5415577 57.562862, 26.5417749 57.5626484, 26.5418835 57.5623636, 26.5418835 57.5618976, 26.5417387 57.561684, 26.541511 57.5613969, 26.5414429 57.5612742, 26.5413887 57.5610561, 26.5413887 57.5608943, 26.5414298 57.5603936, 26.5413284 57.5599751, 26.541358 57.5598002, 26.5414836 57.5596822, 26.5419197 57.559399, 26.5461916 57.5567966, 26.5471932 57.5563888, 26.5473966 57.5562937, 26.5480379 57.5559939, 26.548593 57.5558191, 26.5528763 57.5551484, 26.5531333 57.555094, 26.5533454 57.5550106, 26.5535298 57.5549199, 26.5536354 57.5548216, 26.5537467 57.5546775, 26.5540475 57.5541423, 26.5542631 57.5538532, 26.5546267 57.5533654, 26.5551773 57.5529974, 26.5555031 57.5528161) │ tertiary │
│ way/1034798157 │ LINESTRING (27.2225597 57.5651254, 27.2225259 57.5649716, 27.2225475 57.564821, 27.2226864 57.5645998, 27.2229599 57.5643868, 27.2230116 57.5643094, 27.2230206 57.5638593, 27.22304 57.5637813, 27.2231025 57.5636746, 27.2232978 57.5634294, 27.2233378 57.5633206, 27.2233033 57.563125, 27.2231102 57.5629799, 27.222079 57.5626282) │ tertiary │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ way/1011712751 │ LINESTRING (24.4902866 58.409069, 24.4911676 58.4090315) │ secondary │
│ way/1011712743 │ LINESTRING (24.4935999 58.4089796, 24.4936217 58.4091317) │ secondary │
│ way/360550433 │ LINESTRING (24.4933609 58.4091401, 24.4933332 58.4089862) │ secondary │
│ way/360550431 │ LINESTRING (24.4933609 58.4091401, 24.4931851 58.4091463, 24.4929228 58.40915, 24.4926616 58.4091491, 24.4924699 58.4091462, 24.4917378 58.4091177, 24.4902866 58.409069) │ secondary │
│ way/261722732 │ LINESTRING (24.4936217 58.4091317, 24.4933609 58.4091401) │ secondary │
│ way/701057648 │ LINESTRING (22.7808876 58.4099208, 22.7815414 58.4097553, 22.7822548 58.4095712, 22.7827107 58.4094536, 22.7839132 58.4091441, 22.7843126 58.4090421, 22.7845514 58.4089812, 22.7849091 58.4088886, 22.7862469 58.4085449, 22.7867512 58.4084194, 22.7868477 58.4083974) │ tertiary │
│ way/35343362 │ LINESTRING (26.6458257 58.4099123, 26.6470296 58.409613, 26.6475703 58.4094632, 26.6484381 58.4091776, 26.6493514 58.4088105, 26.650173 58.4084216) │ trunk │
│ way/1033628276 │ LINESTRING (26.7080932 58.408788, 26.7079186 58.4095776) │ trunk │
│ way/33361318 │ LINESTRING (24.34918 58.4121498, 24.3465763 58.4115737, 24.3315144 58.4082991, 24.3301504 58.4080025, 24.3289188 58.4077347, 24.3287436 58.4076966, 24.3283411 58.4076091, 24.3275456 58.4074018, 24.3267243 58.4073451, 24.3255083 58.407292, 24.3241684 58.407284, 24.3220596 58.4072058, 24.3217319 58.4071937, 24.3211993 58.4071495, 24.3200855 58.407057, 24.3174202 58.4067479, 24.312723 58.4066008, 24.3125493 58.4065863, 24.3120941 58.4065386, 24.3111048 58.4064037) │ tertiary │
│ way/1032843354 │ LINESTRING (24.6701002 58.4089287, 24.6706621 58.4091945, 24.671201 58.4094465, 24.6718597 58.4097073) │ secondary │
└────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────┘
? rows (>9999 rows, 20 shown) 3 columns
Count loaded roads¶
In [11]:
Copied!
features_relation.count("feature_id")
features_relation.count("feature_id")
Out[11]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 30940 │ └───────────────────┘
In [12]:
Copied!
length_in_meters = (
features_relation.project(
"ST_Length(ST_Transform(geometry, 'EPSG:4326', 'EPSG:3301')) AS road_length"
)
.sum("road_length")
.fetchone()[0]
)
length_in_km = length_in_meters / 1000
length_in_km
length_in_meters = (
features_relation.project(
"ST_Length(ST_Transform(geometry, 'EPSG:4326', 'EPSG:3301')) AS road_length"
)
.sum("road_length")
.fetchone()[0]
)
length_in_km = length_in_meters / 1000
length_in_km
Out[12]:
29793.98977410875
Plot the roads using GeoPandas¶
With fast loading of geoparquet files using geoarrow.pyarrow library.
In [13]:
Copied!
import geoarrow.pyarrow as ga
from geoarrow.pyarrow import io
from quackosm._constants import GEOMETRY_COLUMN
parquet_table = io.read_geoparquet_table(estonia_features_gpq)
ga.to_geopandas(parquet_table.column(GEOMETRY_COLUMN)).plot()
import geoarrow.pyarrow as ga
from geoarrow.pyarrow import io
from quackosm._constants import GEOMETRY_COLUMN
parquet_table = io.read_geoparquet_table(estonia_features_gpq)
ga.to_geopandas(parquet_table.column(GEOMETRY_COLUMN)).plot()
Out[13]:
<Axes: xlabel='Geodetic longitude [degree]', ylabel='Geodetic latitude [degree]'>
Download all data for Liechtenstein¶
Without filtering, with tags in a compact form
In [14]:
Copied!
liechtenstein_pbf_url = "https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf"
liechtenstein_pbf_file = "liechtenstein.osm.pbf"
urllib.request.urlretrieve(liechtenstein_pbf_url, liechtenstein_pbf_file)
# Here explode_tags is set to False explicitly,
# but it would set automatically when not filtering the data
reader = PbfFileReader(geometry_filter=None, tags_filter=None)
liechtenstein_features_gpq = reader.convert_pbf_to_parquet(
liechtenstein_pbf_file, explode_tags=False
)
liechtenstein_features_gpq
liechtenstein_pbf_url = "https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf"
liechtenstein_pbf_file = "liechtenstein.osm.pbf"
urllib.request.urlretrieve(liechtenstein_pbf_url, liechtenstein_pbf_file)
# Here explode_tags is set to False explicitly,
# but it would set automatically when not filtering the data
reader = PbfFileReader(geometry_filter=None, tags_filter=None)
liechtenstein_features_gpq = reader.convert_pbf_to_parquet(
liechtenstein_pbf_file, explode_tags=False
)
liechtenstein_features_gpq
Finished operation in 0:00:06
Out[14]:
PosixPath('files/liechtenstein_nofilter_noclip_compact_sorted.parquet')
In [15]:
Copied!
features_relation = connection.read_parquet(str(liechtenstein_features_gpq))
features_relation
features_relation = connection.read_parquet(str(liechtenstein_features_gpq))
features_relation
Out[15]:
┌──────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ feature_id │ tags │ geometry │
│ varchar │ map(varchar, varchar) │ geometry('ogc:crs84') │
├──────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ node/2186283697 │ {power=tower} │ POINT (9.5414282 46.9784404) │
│ node/1110282277 │ {power=tower} │ POINT (9.5405011 46.9821126) │
│ node/540308216 │ {line_attachment=suspension, power=tower} │ POINT (9.5400257 46.9839628) │
│ node/540308211 │ {power=tower} │ POINT (9.5372399 46.9862797) │
│ node/1338490716 │ {power=tower} │ POINT (9.5342839 46.9887559) │
│ node/540308206 │ {power=tower} │ POINT (9.5313353 46.9912103) │
│ node/540308180 │ {power=tower} │ POINT (9.5285455 46.9936492) │
│ node/540308170 │ {power=tower} │ POINT (9.5257615 46.9960867) │
│ node/540308159 │ {line_attachment=anchor, power=tower} │ POINT (9.5231821 46.998349) │
│ node/1338490728 │ {line_attachment=suspension, power=tower} │ POINT (9.5207374 47.0011402) │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ node/13686145193 │ {name=Schwemmiwald, place=locality} │ POINT (9.5718599 47.1038687) │
│ way/298855656 │ {bridge=yes, highway=track, layer=1, 'piste:difficulty'=easy, 'piste:name'=Haita, 'piste:ref'=2, 'piste:type'=downhill, surface=concrete, tracktype=grade3} │ LINESTRING (9.6059749 47.1038262, 9.6059853 47.1039143) │
│ way/490968742 │ {landuse=retail} │ POLYGON ((9.5248339 47.10408, 9.5249613 47.1040731, 9.5255286 47.1040421, 9.5254849 47.103671, 9.5254198 47.1036746, 9.5247903 47.1037089, 9.5248339 47.10408)) │
│ way/1057550375 │ {'addr:city'=Triesen, 'addr:country'=LI, 'addr:housenumber'=66b, 'addr:postcode'=9495, 'addr:street'=Feldstrasse, building=yes} │ POLYGON ((9.5293101 47.1039481, 9.5292929 47.1038108, 9.5293743 47.1038067, 9.52938 47.1038519, 9.5294272 47.1038492, 9.5294335 47.1038997, 9.5293831 47.1039027, 9.5293883 47.1039436, 9.5293562 47.1039454, 9.5293101 47.1039481)) │
│ node/5140945767 │ {natural=tree} │ POINT (9.5236049 47.103889) │
│ way/439955219 │ {'piste:grooming'=classic;skating, 'piste:type'=nordic} │ LINESTRING (9.5788027 47.1067065, 9.5789223 47.1058676, 9.579081 47.1054124, 9.5792283 47.1050112, 9.5794834 47.1045945, 9.5796988 47.1042512, 9.5798461 47.1038576, 9.5800309 47.1034454, 9.5800615 47.1027851, 9.5801862 47.1023028, 9.5802486 47.1020212, 9.580549 47.1012496, 9.5806131 47.1010807) │
│ way/210655086 │ {'addr:city'=Triesenberg, 'addr:country'=LI, 'addr:housenumber'=63, 'addr:postcode'=9497, 'addr:street'=Malbunstrasse, building=yes} │ POLYGON ((9.6088719 47.1039341, 9.6088566 47.1038724, 9.6089179 47.1038654, 9.6089332 47.1039271, 9.6088719 47.1039341)) │
│ way/271900134 │ {'addr:city'=Triesen, 'addr:country'=LI, 'addr:housenumber'=3, 'addr:postcode'=9495, 'addr:street'=Saxweg, building=yes} │ POLYGON ((9.5347968 47.103903, 9.5348244 47.103884, 9.5347381 47.103826, 9.5347132 47.1038431, 9.5346911 47.1038283, 9.5345793 47.1039053, 9.5346896 47.1039795, 9.5347987 47.1039043, 9.5347968 47.103903)) │
│ node/5058544516 │ {natural=tree} │ POINT (9.5286685 47.103903) │
│ way/1057580546 │ {'addr:city'=Triesen, 'addr:country'=LI, 'addr:housenumber'=3, 'addr:postcode'=9495, 'addr:street'=Lawenastrasse, building=yes} │ POLYGON ((9.5341221 47.1039378, 9.5341432 47.1038654, 9.5342685 47.1038842, 9.5343276 47.1038903, 9.5343065 47.1039627, 9.5341221 47.1039378)) │
└──────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
? rows (>9999 rows, 20 shown) 3 columns
Return data as GeoDataFrame¶
PbfFileReader can also return the data in the GeoDataFrame form.
Here the caching strategy will be utilized - file won't be transformed again.
In [16]:
Copied!
features_gdf = reader.convert_pbf_to_geodataframe(liechtenstein_pbf_file)
features_gdf
features_gdf = reader.convert_pbf_to_geodataframe(liechtenstein_pbf_file)
features_gdf
Out[16]:
| tags | geometry | |
|---|---|---|
| feature_id | ||
| node/2186283697 | {'power': 'tower'} | POINT (9.54143 46.97844) |
| node/1110282277 | {'power': 'tower'} | POINT (9.5405 46.98211) |
| node/540308216 | {'line_attachment': 'suspension', 'power': 'to... | POINT (9.54003 46.98396) |
| node/540308211 | {'power': 'tower'} | POINT (9.53724 46.98628) |
| node/1338490716 | {'power': 'tower'} | POINT (9.53428 46.98876) |
| ... | ... | ... |
| way/1147542974 | {'natural': 'shingle'} | POLYGON ((9.65257 47.51371, 9.65329 47.51328, ... |
| way/636965953 | {'natural': 'wetland', 'wetland': 'reedbed'} | POLYGON ((9.65609 47.51547, 9.65575 47.5156, 9... |
| way/1147583577 | {'natural': 'wetland', 'wetland': 'marsh'} | POLYGON ((9.65764 47.51459, 9.65754 47.51469, ... |
| way/1147583578 | {'natural': 'wetland', 'wetland': 'marsh'} | POLYGON ((9.65496 47.51582, 9.65567 47.51564, ... |
| way/640953866 | {'natural': 'wetland', 'source:outline': 'VoGI... | POLYGON ((9.65471 47.5159, 9.65445 47.51598, 9... |
62179 rows × 2 columns
Plot the forests using GeoPandas¶
Filter all polygons and features with landuse=forest.
In [17]:
Copied!
features_gdf[
features_gdf.geom_type.isin(("Polygon", "MultiPolygon"))
& features_gdf.tags.apply(lambda x: "landuse" in x and x["landuse"] == "forest")
].plot(color="green")
features_gdf[
features_gdf.geom_type.isin(("Polygon", "MultiPolygon"))
& features_gdf.tags.apply(lambda x: "landuse" in x and x["landuse"] == "forest")
].plot(color="green")
Out[17]:
<Axes: xlabel='Geodetic longitude [degree]', ylabel='Geodetic latitude [degree]'>
