PBF File Reader¶
PBFFileReader can really quickly parse full OSM extract in the form of *.osm.pbf file.
It uses DuckDB with spatial extension to convert pbf files into geoparquet files without GDAL dependency.
Reader can filter objects by geometry and by OSM tags with option to split tags into columns or keep it as a single dictionary.
Caching strategy is implemented to reduce computations, but it can be overriden using ignore_cache parameter.
Download all buildings in Reykjavík, Iceland¶
Filtering the data with geometry and by tags, with tags in exploded form
Run this notebook in Google Colab:
Remember to install the quackosm library before running the notebook:
%pip install quackosm[cli]
In [1]:
Copied!
import urllib.request
from quackosm import PbfFileReader, geocode_to_geometry
import urllib.request
from quackosm import PbfFileReader, geocode_to_geometry
In [2]:
Copied!
iceland_pbf_url = "https://download.geofabrik.de/europe/iceland-latest.osm.pbf"
iceland_pbf_file = "iceland.osm.pbf"
urllib.request.urlretrieve(iceland_pbf_url, iceland_pbf_file)
iceland_pbf_url = "https://download.geofabrik.de/europe/iceland-latest.osm.pbf"
iceland_pbf_file = "iceland.osm.pbf"
urllib.request.urlretrieve(iceland_pbf_url, iceland_pbf_file)
Out[2]:
('iceland.osm.pbf', <http.client.HTTPMessage at 0x7fe15e293c50>)
In [3]:
Copied!
reykjavik_geometry = geocode_to_geometry("Reykjavík, IS")
reykjavik_geometry
reykjavik_geometry = geocode_to_geometry("Reykjavík, IS")
reykjavik_geometry
Out[3]:
To filter out buildings, we will utilize format used also in the osmnx library: a dictionary with keys representing tag keys and values that could be a bool, string or a list of string.
By default, QuackOSM will return only the tags that are present in the passed filter.
In this example we will select all the buildings using { "building": True } filter and only building tag values will be present in the result.
In [4]:
Copied!
reader = PbfFileReader(
geometry_filter=reykjavik_geometry, tags_filter={"building": True}
)
reykjavik_buildings_gpq = reader.convert_pbf_to_parquet("iceland.osm.pbf")
reykjavik_buildings_gpq
reader = PbfFileReader(
geometry_filter=reykjavik_geometry, tags_filter={"building": True}
)
reykjavik_buildings_gpq = reader.convert_pbf_to_parquet("iceland.osm.pbf")
reykjavik_buildings_gpq
Finished operation in 0:00:12
Out[4]:
PosixPath('files/iceland_ae99e3d9_4e8a6f65_exploded_sorted.parquet')
Read those features using DuckDB¶
In [5]:
Copied!
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_buildings_gpq))
features_relation
import duckdb
connection = duckdb.connect()
connection.load_extension("parquet")
connection.load_extension("spatial")
features_relation = connection.read_parquet(str(reykjavik_buildings_gpq))
features_relation
Out[5]:
┌───────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────┐
│ feature_id │ geometry │ building │
│ varchar │ geometry('ogc:crs84') │ varchar │
├───────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────┤
│ way/671607155 │ POLYGON ((-21.7893832 64.0513494, -21.7892828 64.0513882, -21.7895674 64.0515295, -21.7896664 64.0514913, -21.7895539 64.0514354, -21.7896311 64.0514057, -21.7895352 64.0513581, -21.7894594 64.0513873, -21.7893832 64.0513494)) │ yes │
│ way/671607153 │ POLYGON ((-21.785565 64.0518193, -21.7854812 64.0518449, -21.7855606 64.0518945, -21.7856443 64.0518689, -21.785565 64.0518193)) │ yes │
│ way/668334022 │ POLYGON ((-21.7290748 64.0677094, -21.7289315 64.0677504, -21.7290516 64.0678308, -21.7291949 64.0677898, -21.7290748 64.0677094)) │ yes │
│ way/668337377 │ POLYGON ((-21.7443484 64.0735764, -21.7442615 64.0735948, -21.7443022 64.0736317, -21.7443891 64.0736134, -21.7443484 64.0735764)) │ yes │
│ way/670007221 │ POLYGON ((-21.7658076 64.0756955, -21.7658067 64.0757663, -21.7660254 64.0757668, -21.7660263 64.075696, -21.7658076 64.0756955)) │ yes │
│ way/366744730 │ POLYGON ((-21.7724858 64.0767213, -21.7723384 64.0767463, -21.7722604 64.0766586, -21.7724077 64.0766336, -21.7724313 64.07666, -21.7724713 64.0766532, -21.7724978 64.076683, -21.7724578 64.0766898, -21.7724858 64.0767213)) │ yes │
│ way/508499993 │ POLYGON ((-21.7744719 64.078054, -21.7744139 64.0780667, -21.7743643 64.0780233, -21.7744222 64.0780106, -21.7744719 64.078054)) │ yes │
│ way/366744729 │ POLYGON ((-21.7766302 64.0789397, -21.7764969 64.0789688, -21.7764483 64.0789263, -21.7765815 64.0788972, -21.7766302 64.0789397)) │ yes │
│ way/508500012 │ POLYGON ((-21.787494 64.0805525, -21.7875199 64.0805852, -21.7874234 64.0805998, -21.7873974 64.0805672, -21.787494 64.0805525)) │ yes │
│ way/508499994 │ POLYGON ((-21.7870689 64.0808515, -21.7870851 64.0808399, -21.7871417 64.0808549, -21.7870593 64.0809141, -21.786971 64.0808904, -21.7868658 64.0808625, -21.786932 64.0808151, -21.7870689 64.0808515)) │ yes │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ way/104964231 │ POLYGON ((-21.9500135 64.1292308, -21.9496476 64.1291427, -21.9497089 64.1290942, -21.9497362 64.1291008, -21.9497717 64.1290727, -21.9501103 64.1291542, -21.9500135 64.1292308)) │ yes │
│ way/108362257 │ POLYGON ((-21.8840584 64.1291849, -21.8843592 64.1292328, -21.8844535 64.12912, -21.8841528 64.1290721, -21.8840584 64.1291849)) │ residential │
│ way/107649857 │ POLYGON ((-21.7589277 64.1291994, -21.7587444 64.1292132, -21.7587057 64.1291154, -21.758889 64.1291016, -21.7589277 64.1291994)) │ residential │
│ way/109031111 │ POLYGON ((-21.8580266 64.1291974, -21.8579459 64.1291675, -21.8579863 64.1291468, -21.8577817 64.129071, -21.8577173 64.1291041, -21.8576916 64.1290946, -21.8576641 64.1291088, -21.857671 64.1291114, -21.8576212 64.1291369, -21.8576373 64.1291429, -21.8575992 64.1291626, -21.8577028 64.129201, -21.8577347 64.1291846, -21.8578337 64.1292212, -21.8578452 64.1292153, -21.8579304 64.1292468, -21.8580266 64.1291974)) │ residential │
│ way/102590049 │ POLYGON ((-21.951944 64.1291842, -21.9519783 64.129159, -21.9519578 64.1291537, -21.9520099 64.1291154, -21.9520475 64.1291251, -21.9520958 64.1290897, -21.9522674 64.1291341, -21.952171 64.129205, -21.9521535 64.1292004, -21.9521152 64.1292286, -21.951944 64.1291842)) │ yes │
│ way/107756907 │ POLYGON ((-21.7535666 64.1292208, -21.753405 64.1291989, -21.7533962 64.1292111, -21.7533226 64.1292012, -21.7533321 64.1291879, -21.753277 64.1291805, -21.753292 64.1291594, -21.7532639 64.1291556, -21.7532933 64.1291142, -21.7533033 64.1291002, -21.7534113 64.1291149, -21.7534004 64.1291302, -21.7536108 64.1291587, -21.7535666 64.1292208)) │ residential │
│ way/516193601 │ POLYGON ((-21.9309649 64.1293002, -21.9309693 64.129321, -21.930907 64.1293235, -21.9308935 64.1293185, -21.9307978 64.129283, -21.930774 64.1292742, -21.9307271 64.129055, -21.9308431 64.1290503, -21.9308363 64.1290182, -21.9313146 64.1289987, -21.9313302 64.1290717, -21.9314664 64.1290662, -21.9315116 64.1292779, -21.9309649 64.1293002)) │ yes │
│ way/108362204 │ POLYGON ((-21.8790925 64.1292192, -21.8787687 64.1292121, -21.8787739 64.1291668, -21.8788121 64.1291677, -21.8788156 64.1291376, -21.8789062 64.1291396, -21.8789099 64.1291076, -21.8791351 64.1291126, -21.8791306 64.1291517, -21.8790477 64.1291499, -21.8790442 64.1291802, -21.8790969 64.1291814, -21.8790925 64.1292192)) │ residential │
│ way/110555796 │ POLYGON ((-21.9039443 64.1291069, -21.9038348 64.1292087, -21.903739 64.1291891, -21.9036651 64.1292578, -21.903505 64.1292251, -21.9035933 64.1291429, -21.903636 64.1291516, -21.9037241 64.1290697, -21.9038233 64.12909, -21.9038303 64.1290836, -21.9039443 64.1291069)) │ yes │
│ way/109031190 │ POLYGON ((-21.8564044 64.1292388, -21.8562607 64.1291305, -21.8564117 64.1290924, -21.8565555 64.1292007, -21.8564044 64.1292388)) │ residential │
└───────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────┘
? rows (>9999 rows, 20 shown) 3 columns
Count all buildings¶
In [6]:
Copied!
features_relation.count("feature_id")
features_relation.count("feature_id")
Out[6]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 25509 │ └───────────────────┘
Keeping all the tags while filtering the data¶
To keep all of the tags present in the source data, we can use keep_all_tags parameter. That way we will still return only buildings, but with all of the tags attached.
By default, all of those tags will be kept in a single column as a dict.
In [7]:
Copied!
reader.convert_pbf_to_geodataframe("iceland.osm.pbf", keep_all_tags=True)
reader.convert_pbf_to_geodataframe("iceland.osm.pbf", keep_all_tags=True)
Finished operation in 0:00:12
Out[7]:
| tags | geometry | |
|---|---|---|
| feature_id | ||
| way/671607155 | {'building': 'yes', 'roof:colour': 'green', 'r... | POLYGON ((-21.78938 64.05135, -21.78928 64.051... |
| way/671607153 | {'building': 'yes', 'roof:colour': 'green', 'r... | POLYGON ((-21.78556 64.05182, -21.78548 64.051... |
| way/668334022 | {'building': 'yes', 'name': 'Torgeirsstaðir'} | POLYGON ((-21.72907 64.06771, -21.72893 64.067... |
| way/668337377 | {'building': 'yes'} | POLYGON ((-21.74435 64.07358, -21.74426 64.073... |
| way/670007221 | {'building': 'yes', 'roof:colour': 'green', 'r... | POLYGON ((-21.76581 64.0757, -21.76581 64.0757... |
| ... | ... | ... |
| way/499098406 | {'building': 'yes'} | POLYGON ((-21.79544 64.29633, -21.79536 64.296... |
| way/351976584 | {'building': 'yes', 'start_date': '1986'} | POLYGON ((-21.8061 64.29683, -21.80597 64.2967... |
| way/351976596 | {'addr:housename': 'Laufbrekka', 'addr:postcod... | POLYGON ((-21.79557 64.297, -21.79539 64.29697... |
| way/351976595 | {'building': 'yes', 'start_date': '2005'} | POLYGON ((-21.79492 64.29698, -21.79476 64.297... |
| way/812660542 | {'building': 'yes'} | POLYGON ((-21.80953 64.29755, -21.80929 64.297... |
25509 rows × 2 columns
Download main roads for Estonia¶
Filtering the data only by tags, with tags in exploded form
In [8]:
Copied!
highways_filter = {
"highway": [
"motorway",
"trunk",
"primary",
"secondary",
"tertiary",
]
}
highways_filter = {
"highway": [
"motorway",
"trunk",
"primary",
"secondary",
"tertiary",
]
}
In [9]:
Copied!
estonia_pbf_url = "http://download.geofabrik.de/europe/estonia-latest.osm.pbf"
estonia_pbf_file = "estonia.osm.pbf"
urllib.request.urlretrieve(estonia_pbf_url, estonia_pbf_file)
reader = PbfFileReader(geometry_filter=None, tags_filter=highways_filter)
estonia_features_gpq = reader.convert_pbf_to_parquet(estonia_pbf_file)
estonia_features_gpq
estonia_pbf_url = "http://download.geofabrik.de/europe/estonia-latest.osm.pbf"
estonia_pbf_file = "estonia.osm.pbf"
urllib.request.urlretrieve(estonia_pbf_url, estonia_pbf_file)
reader = PbfFileReader(geometry_filter=None, tags_filter=highways_filter)
estonia_features_gpq = reader.convert_pbf_to_parquet(estonia_pbf_file)
estonia_features_gpq
Finished operation in 0:00:15
Out[9]:
PosixPath('files/estonia_87e7b741_noclip_exploded_sorted.parquet')
In [10]:
Copied!
features_relation = connection.read_parquet(str(estonia_features_gpq))
features_relation
features_relation = connection.read_parquet(str(estonia_features_gpq))
features_relation
Out[10]:
┌────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────┐
│ feature_id │ geometry │ highway │
│ varchar │ geometry('ogc:crs84') │ varchar │
├────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────┤
│ way/36872845 │ LINESTRING (27.2145593 57.5437215, 27.2194975 57.5455473, 27.2199616 57.5457464, 27.2203307 57.5459478, 27.2205615 57.546123, 27.2207955 57.5463688, 27.2210439 57.5467076, 27.2222459 57.5487383, 27.2233359 57.5504811, 27.2235732 57.5509167, 27.2236975 57.5512212, 27.2237427 57.5515224, 27.2237557 57.5518294, 27.2234787 57.5535, 27.2234177 57.5536613, 27.2233139 57.5538456, 27.2228174 57.5543309) │ tertiary │
│ way/44077674 │ LINESTRING (26.6829371 57.5449717, 26.6828062 57.5450309, 26.6826495 57.5450835, 26.6824839 57.5451207, 26.682169 57.5451766, 26.6816509 57.5452207, 26.6815987 57.5452259, 26.6810758 57.5452783, 26.6805316 57.5453458, 26.680415 57.5453656, 26.6792501 57.5455918, 26.678372 57.5457753, 26.6775269 57.545982, 26.676824 57.5461784, 26.6761706 57.5463871, 26.6757913 57.5465222, 26.6752945 57.5467094, 26.6743075 57.5471138, 26.6737475 57.5474094, 26.6731038 57.5477122, 26.6727903 57.5478475, 26.6724522 57.5479695, 26.6716688 57.5481974, 26.6708187 57.5484458, 26.670529 57.5485505, 26.6703059 57.5486482, 26.6700868 57.5487791, 26.6695149 57.5491421, 26.6694443 57.5491947, 26.6691873 57.5493864, 26.6690475 57.5495267, 26.6689241 57.5497001, 26.6688722 57.549804, 26.668849 57.5499168, 26.668834 57.5500536, 26.6688437 57.5502059, 26.6690315 57.5509594, 26.6690413 57.5511128, 26.669016 57.5512927, 26.6689245 57.5514913, 26.6688185 57.551648, 26.6687122 57.5517518, 26.6684842 57.551933, 26.6682222 57.5520726, 26.6681643 57.5521024, 26.6675048 57.5523811, 26.6673011 57.5524729, 26.6669947 57.5526641, 26.6667158 57.5529156, 26.6665048 57.5531316, 26.6664155 57.5532663) │ secondary │
│ way/182788869 │ LINESTRING (26.5560824 57.5525312, 26.5563448 57.5523857, 26.5564444 57.5523305, 26.5565409 57.5521687, 26.556553 57.551832, 26.556552 57.5512833, 26.5566496 57.5500773, 26.556722 57.5497795, 26.5568692 57.5495695, 26.5571295 57.5494091, 26.5575561 57.5492913, 26.5612206 57.5486969, 26.5668586 57.5477916, 26.5674258 57.5477333, 26.5681016 57.5477916, 26.5689325 57.5479511, 26.5699398 57.5481445, 26.5717098 57.5484844, 26.5721201 57.5484974, 26.57247 57.5484262, 26.5736647 57.5477009, 26.5758127 57.5464576) │ tertiary │
│ way/123588635 │ LINESTRING (26.3624165 57.5804431, 26.3626787 57.5802237, 26.3630697 57.5799531, 26.3638655 57.5794878, 26.3647737 57.5790056, 26.3655597 57.5785865, 26.3662321 57.5782166, 26.3668959 57.5778444, 26.3680224 57.5772427, 26.3687778 57.5768002, 26.3691053 57.5765661, 26.3695376 57.5762079, 26.3697385 57.5759386, 26.3698738 57.5757068, 26.3700922 57.5753018, 26.3702974 57.5749178, 26.370507 57.5745549, 26.3708738 57.573897, 26.3711532 57.57341, 26.3715288 57.5728152, 26.371865 57.5725179, 26.3725331 57.5720847, 26.373354 57.571586, 26.3739085 57.5712371, 26.374712 57.5707477, 26.375332 57.570366, 26.3760263 57.5699375, 26.3766507 57.5695534, 26.3777511 57.5689259, 26.3782707 57.5686425, 26.3791353 57.568132, 26.3797685 57.5677526, 26.3808033 57.5673194, 26.3814714 57.5670431, 26.3831482 57.5663944, 26.3844014 57.5659049, 26.3856327 57.5654318, 26.386327 57.5651087, 26.3866589 57.5649307, 26.3872309 57.5644951, 26.3880256 57.5638651, 26.3889557 57.563132, 26.3891435 57.5629259, 26.3892133 57.5627596, 26.3892308 57.5625582, 26.3892221 57.5623966, 26.3891347 57.5621787, 26.3887636 57.5616143, 26.388423 57.5611084, 26.3874187 57.5596421, 26.3872047 57.5592252, 26.3871698 57.5589909, 26.387196 57.5587825, 26.3873663 57.5585506, 26.387637 57.5583374, 26.3883356 57.5579861, 26.3891302 57.557604, 26.3892963 57.5575105, 26.3896238 57.5572107, 26.3897591 57.5569741, 26.3898683 57.5565642, 26.3900037 57.5556459, 26.3901347 57.5551376, 26.3902831 57.5547979, 26.3906325 57.5543949, 26.3909643 57.5539592, 26.3912132 57.5536734, 26.3916106 57.5531674, 26.3918769 57.5528371, 26.3921782 57.5524505, 26.3924795 57.5520335, 26.3925799 57.5518133, 26.3928943 57.5513049, 26.3930559 57.5509722, 26.3934838 57.5501334, 26.393689 57.5497023, 26.3942392 57.5485917, 26.3948767 57.5473054, 26.3952741 57.5464267, 26.395523 57.5459909, 26.3957128 57.5457922, 26.395785 57.5457167, 26.3960251 57.5455176, 26.39706 57.5448005, 26.3973962 57.5445334, 26.3974879 57.5444326, 26.3976233 57.5441866, 26.3976233 57.5433289, 26.3976146 57.5424361, 26.3976277 57.5417495, 26.3975884 57.5413089, 26.3974355 57.5408285, 26.3973307 57.540409, 26.397226 57.5400668, 26.3971212 57.5398676, 26.3968048 57.5396315, 26.3965055 57.5394176, 26.3961131 57.539116, 26.3960724 57.5387188, 26.3961247 57.5384622, 26.3964466 57.5375665, 26.3967672 57.5369824, 26.3968939 57.536691, 26.3970383 57.5364666, 26.3971358 57.5355897, 26.3971801 57.5351919, 26.3971927 57.5350783, 26.3972805 57.5349728, 26.3982904 57.5337584, 26.3989111 57.533012, 26.399245 57.5328025, 26.3997513 57.5325656, 26.4002393 57.5322957, 26.400666 57.5320233, 26.4009456 57.5317338, 26.4010901 57.5312535, 26.4011316 57.5305976, 26.4010418 57.5298309, 26.4009973 57.5292818, 26.4009757 57.5283642, 26.4007008 57.5273207, 26.4002781 57.5257159, 26.399904 57.524158, 26.3995969 57.5229545, 26.3994853 57.5226434, 26.3994353 57.5225603, 26.3992176 57.5223293, 26.3989872 57.522109, 26.3987544 57.5218999, 26.3987178 57.5218572) │ secondary │
│ way/182788870 │ LINESTRING (26.5555031 57.5528161, 26.5560824 57.5525312) │ tertiary │
│ way/1034798159 │ LINESTRING (27.2216939 57.5623301, 27.2216059 57.562141, 27.2213297 57.5611197, 27.2211675 57.5609002, 27.2207224 57.5604574, 27.2206186 57.5602662, 27.2204822 57.5599092, 27.2204895 57.5597448, 27.2205741 57.5595847, 27.2206868 57.5594828, 27.2212142 57.5591889, 27.2213629 57.5590695, 27.2214871 57.5589501, 27.2216715 57.5585371, 27.2217014 57.5583466, 27.2216249 57.5581666, 27.2214507 57.557956, 27.2206895 57.5568372, 27.2206009 57.5566134, 27.2205773 57.5564244, 27.2206381 57.5562008, 27.2210526 57.5555192, 27.2213197 57.5552709, 27.221992 57.5548525, 27.2223582 57.5546554, 27.2228174 57.5543309) │ tertiary │
│ way/349875145 │ LINESTRING (26.6618721 57.5644661, 26.6625061 57.5642172, 26.6646336 57.5634051, 26.6654207 57.5631114, 26.665748 57.5629672, 26.6660607 57.5627876, 26.6662781 57.562612, 26.6664435 57.5624169, 26.6666821 57.5620397, 26.6669135 57.5615823, 26.6669898 57.5613778, 26.667044 57.5611146, 26.6670496 57.5607521, 26.6669618 57.5603699, 26.6667163 57.5594662, 26.6666543 57.5591751, 26.666606 57.5589291, 26.6665894 57.5584941, 26.6667944 57.5575564, 26.6672396 57.5558269, 26.6672481 57.5555388, 26.6671943 57.5552393, 26.6671186 57.5550027, 26.6666077 57.5539711, 26.6665383 57.5538288, 26.6664505 57.5536489, 26.6664132 57.5535318, 26.666408 57.5534957, 26.6663924 57.5533868, 26.6664069 57.5532868, 26.6664155 57.5532663) │ secondary │
│ way/1034798156 │ LINESTRING (27.222079 57.5626282, 27.2218649 57.5624974, 27.2216939 57.5623301) │ tertiary │
│ way/182788871 │ LINESTRING (26.5235403 57.5744752, 26.5240664 57.5743289, 26.5244656 57.5742227, 26.5278912 57.5732265, 26.5282267 57.5731289, 26.5288162 57.5728634, 26.5289813 57.5726421, 26.5290285 57.5722944, 26.5295119 57.5717696, 26.5337067 57.5680778, 26.5348009 57.5671148, 26.5353308 57.5666484, 26.5384974 57.564783, 26.5415577 57.562862, 26.5417749 57.5626484, 26.5418835 57.5623636, 26.5418835 57.5618976, 26.5417387 57.561684, 26.541511 57.5613969, 26.5414429 57.5612742, 26.5413887 57.5610561, 26.5413887 57.5608943, 26.5414298 57.5603936, 26.5413284 57.5599751, 26.541358 57.5598002, 26.5414836 57.5596822, 26.5419197 57.559399, 26.5461916 57.5567966, 26.5471932 57.5563888, 26.5473966 57.5562937, 26.5480379 57.5559939, 26.548593 57.5558191, 26.5528763 57.5551484, 26.5531333 57.555094, 26.5533454 57.5550106, 26.5535298 57.5549199, 26.5536354 57.5548216, 26.5537467 57.5546775, 26.5540475 57.5541423, 26.5542631 57.5538532, 26.5546267 57.5533654, 26.5551773 57.5529974, 26.5555031 57.5528161) │ tertiary │
│ way/1034798157 │ LINESTRING (27.2225597 57.5651254, 27.2225259 57.5649716, 27.2225475 57.564821, 27.2226864 57.5645998, 27.2229599 57.5643868, 27.2230116 57.5643094, 27.2230206 57.5638593, 27.22304 57.5637813, 27.2231025 57.5636746, 27.2232978 57.5634294, 27.2233378 57.5633206, 27.2233033 57.563125, 27.2231102 57.5629799, 27.222079 57.5626282) │ tertiary │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ way/1442710221 │ LINESTRING (26.7635872 58.4236535, 26.7635496 58.4235911) │ tertiary │
│ way/977629471 │ LINESTRING (24.3151484 58.4237233, 24.3151571 58.4237893) │ tertiary │
│ way/1033618850 │ LINESTRING (26.763782 58.4240183, 26.7636933 58.4238423, 26.7635872 58.4236535) │ tertiary │
│ way/34149667 │ LINESTRING (26.0360267 58.4268881, 26.0360319 58.4266872, 26.0361327 58.4257623, 26.0364658 58.4249042, 26.0364884 58.4246484, 26.0364442 58.4243758, 26.0362883 58.424073, 26.0361698 58.4238713, 26.0361195 58.4236617, 26.0361681 58.4235066, 26.0363059 58.4232997, 26.0365509 58.4229799, 26.0367781 58.4227068, 26.0370029 58.42239, 26.0370449 58.4222999, 26.0371319 58.4221137, 26.0372187 58.4217834, 26.0372428 58.421439, 26.0372101 58.4212022, 26.0371629 58.4211061, 26.0370946 58.4210178, 26.0370333 58.4209689, 26.0368555 58.4208485) │ tertiary │
│ way/440712723 │ LINESTRING (23.7589741 58.4220674, 23.7589533 58.42208, 23.7586051 58.4223968, 23.7583283 58.4227317, 23.75811 58.4230983, 23.75718 58.4255633, 23.7570799 58.425732, 23.75673 58.4263217) │ tertiary │
│ way/1033618849 │ LINESTRING (26.7640458 58.4251332, 26.763942 58.4246157, 26.7639124 58.4244685, 26.7638811 58.4243123, 26.7638642 58.4242561, 26.7638493 58.4242068, 26.7638136 58.4240882, 26.763782 58.4240183) │ tertiary │
│ way/1015002765 │ LINESTRING (23.001532 58.4285514, 23.0016535 58.4281708, 23.0017471 58.4279014, 23.001812 58.427779, 23.0021301 58.4272485, 23.0022104 58.4271141, 23.0023465 58.4269192, 23.0024855 58.4267383, 23.0026661 58.4265184, 23.0026868 58.4264665, 23.0029728 58.4261278, 23.0041393 58.4247325, 23.0043829 58.4244469, 23.0047896 58.42397, 23.00567 58.422815, 23.0061778 58.4219912, 23.0065736 58.4212296, 23.0068056 58.4206498) │ tertiary │
│ way/440715401 │ LINESTRING (24.0070171 58.4306809, 24.0067663 58.4295542, 24.0066576 58.4288018, 24.0066433 58.4283277, 24.0066286 58.4278441, 24.006555 58.4265254, 24.0063632 58.4250248, 24.0063557 58.4246504, 24.0064978 58.4243385, 24.0079717 58.4224753, 24.0082277 58.4221207, 24.0089254 58.4209419, 24.009771 58.4193795) │ tertiary │
│ way/99058291 │ LINESTRING (26.4883763 58.42845, 26.4895436 58.4278078, 26.4916946 58.4267212, 26.4931384 58.4260474, 26.4947611 58.4253343, 26.4958629 58.4248873, 26.4977298 58.4241801, 26.4994601 58.4235905, 26.5000972 58.4233808, 26.5014066 58.4229738, 26.5036275 58.4223483, 26.5053693 58.4218993) │ trunk │
│ way/1427945596 │ LINESTRING (25.5980343 58.4249023, 25.597795 58.4255297) │ secondary │
└────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────┘
? rows (>9999 rows, 20 shown) 3 columns
Count loaded roads¶
In [11]:
Copied!
features_relation.count("feature_id")
features_relation.count("feature_id")
Out[11]:
┌───────────────────┐ │ count(feature_id) │ │ int64 │ ├───────────────────┤ │ 30514 │ └───────────────────┘
In [12]:
Copied!
length_in_meters = (
features_relation.project(
"ST_Length(ST_Transform(geometry, 'EPSG:4326', 'EPSG:3301')) AS road_length"
)
.sum("road_length")
.fetchone()[0]
)
length_in_km = length_in_meters / 1000
length_in_km
length_in_meters = (
features_relation.project(
"ST_Length(ST_Transform(geometry, 'EPSG:4326', 'EPSG:3301')) AS road_length"
)
.sum("road_length")
.fetchone()[0]
)
length_in_km = length_in_meters / 1000
length_in_km
Out[12]:
29869.477378091367
Plot the roads using GeoPandas¶
With fast loading of geoparquet files using geoarrow.pyarrow library.
In [13]:
Copied!
import geoarrow.pyarrow as ga
from geoarrow.pyarrow import io
from quackosm._constants import GEOMETRY_COLUMN
parquet_table = io.read_geoparquet_table(estonia_features_gpq)
ga.to_geopandas(parquet_table.column(GEOMETRY_COLUMN)).plot()
import geoarrow.pyarrow as ga
from geoarrow.pyarrow import io
from quackosm._constants import GEOMETRY_COLUMN
parquet_table = io.read_geoparquet_table(estonia_features_gpq)
ga.to_geopandas(parquet_table.column(GEOMETRY_COLUMN)).plot()
Out[13]:
<Axes: >
Download all data for Liechtenstein¶
Without filtering, with tags in a compact form
In [14]:
Copied!
liechtenstein_pbf_url = "https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf"
liechtenstein_pbf_file = "liechtenstein.osm.pbf"
urllib.request.urlretrieve(liechtenstein_pbf_url, liechtenstein_pbf_file)
# Here explode_tags is set to False explicitly,
# but it would set automatically when not filtering the data
reader = PbfFileReader(geometry_filter=None, tags_filter=None)
liechtenstein_features_gpq = reader.convert_pbf_to_parquet(
liechtenstein_pbf_file, explode_tags=False
)
liechtenstein_features_gpq
liechtenstein_pbf_url = "https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf"
liechtenstein_pbf_file = "liechtenstein.osm.pbf"
urllib.request.urlretrieve(liechtenstein_pbf_url, liechtenstein_pbf_file)
# Here explode_tags is set to False explicitly,
# but it would set automatically when not filtering the data
reader = PbfFileReader(geometry_filter=None, tags_filter=None)
liechtenstein_features_gpq = reader.convert_pbf_to_parquet(
liechtenstein_pbf_file, explode_tags=False
)
liechtenstein_features_gpq
Finished operation in 0:00:07
Out[14]:
PosixPath('files/liechtenstein_nofilter_noclip_compact_sorted.parquet')
In [15]:
Copied!
features_relation = connection.read_parquet(str(liechtenstein_features_gpq))
features_relation
features_relation = connection.read_parquet(str(liechtenstein_features_gpq))
features_relation
Out[15]:
┌──────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ feature_id │ tags │ geometry │
│ varchar │ map(varchar, varchar) │ geometry('ogc:crs84') │
├──────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ node/2186283697 │ {power=tower} │ POINT (9.5414282 46.9784404) │
│ node/1110282277 │ {power=tower} │ POINT (9.5405011 46.9821126) │
│ node/540308216 │ {line_attachment=suspension, power=tower} │ POINT (9.5400257 46.9839628) │
│ node/540308211 │ {power=tower} │ POINT (9.5372399 46.9862797) │
│ node/1338490716 │ {power=tower} │ POINT (9.5342839 46.9887559) │
│ node/540308206 │ {power=tower} │ POINT (9.5313353 46.9912103) │
│ node/540308180 │ {power=tower} │ POINT (9.5285455 46.9936492) │
│ node/540308170 │ {power=tower} │ POINT (9.5257615 46.9960867) │
│ node/540308159 │ {line_attachment=anchor, power=tower} │ POINT (9.5231821 46.998349) │
│ node/1338490728 │ {line_attachment=suspension, power=tower} │ POINT (9.5207374 47.0011402) │
│ · │ · │ · │
│ · │ · │ · │
│ · │ · │ · │
│ node/5140946027 │ {natural=tree} │ POINT (9.5252458 47.1042461) │
│ way/1409307267 │ {footway=sidewalk, highway=footway} │ LINESTRING (9.5270999 47.1045595, 9.5271021 47.1043293, 9.5271108 47.1041416, 9.5271313 47.103933) │
│ way/1409389508 │ {crossing=unmarked, 'crossing:markings'=no, footway=crossing, highway=footway, surface=asphalt} │ LINESTRING (9.5289307 47.1042762, 9.528965 47.1042456, 9.5289975 47.1042167) │
│ way/490995223 │ {landuse=grass} │ POLYGON ((9.527169 47.1041783, 9.5271744 47.1050108, 9.5275106 47.1050077, 9.5275096 47.1049768, 9.527809 47.1049702, 9.5278074 47.1047844, 9.5276733 47.1047844, 9.5277054 47.1045361, 9.5280281 47.1045638, 9.5280538 47.1043679, 9.5280881 47.1039753, 9.5277698 47.1039628, 9.5277859 47.1036452, 9.5274694 47.1034991, 9.5274426 47.1038058, 9.5272709 47.1038204, 9.5272074 47.1038482, 9.5271859 47.1038993, 9.527169 47.1041783)) │
│ node/4831613369 │ {natural=tree} │ POINT (9.5278401 47.1042608) │
│ node/4814220673 │ {amenity=bench} │ POINT (9.5341102 47.104262) │
│ node/4831613374 │ {natural=tree} │ POINT (9.528001 47.1042644) │
│ node/13752787506 │ {direction=forward, highway=give_way} │ POINT (9.5339886 47.104266) │
│ way/455405904 │ {footway=sidewalk, highway=footway, lit=yes, surface=asphalt} │ LINESTRING (9.5249948 47.105839, 9.5249893 47.1056401, 9.5250016 47.1055242, 9.5249815 47.1051883, 9.5249193 47.1045375, 9.5248783 47.1043115, 9.5247784 47.1036889, 9.5247307 47.1033334, 9.5247126 47.1031041, 9.5247085 47.1028697, 9.5247075 47.1027595, 9.5247107 47.1026984) │
│ way/529160629 │ {highway=footway} │ LINESTRING (9.5251413 47.1040446, 9.5253344 47.1040362, 9.525742 47.1042207, 9.5257633 47.1045018, 9.5256693 47.1045054) │
└──────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
? rows (>9999 rows, 20 shown) 3 columns
Return data as GeoDataFrame¶
PbfFileReader can also return the data in the GeoDataFrame form.
Here the caching strategy will be utilized - file won't be transformed again.
In [16]:
Copied!
features_gdf = reader.convert_pbf_to_geodataframe(liechtenstein_pbf_file)
features_gdf
features_gdf = reader.convert_pbf_to_geodataframe(liechtenstein_pbf_file)
features_gdf
Out[16]:
| tags | geometry | |
|---|---|---|
| feature_id | ||
| node/2186283697 | {'power': 'tower'} | POINT (9.54143 46.97844) |
| node/1110282277 | {'power': 'tower'} | POINT (9.5405 46.98211) |
| node/540308216 | {'line_attachment': 'suspension', 'power': 'to... | POINT (9.54003 46.98396) |
| node/540308211 | {'power': 'tower'} | POINT (9.53724 46.98628) |
| node/1338490716 | {'power': 'tower'} | POINT (9.53428 46.98876) |
| ... | ... | ... |
| way/1147542974 | {'natural': 'shingle'} | POLYGON ((9.65257 47.51371, 9.65329 47.51328, ... |
| way/636965953 | {'natural': 'wetland', 'wetland': 'reedbed'} | POLYGON ((9.65609 47.51547, 9.65575 47.5156, 9... |
| way/1147583577 | {'natural': 'wetland', 'wetland': 'marsh'} | POLYGON ((9.65764 47.51459, 9.65754 47.51469, ... |
| way/1147583578 | {'natural': 'wetland', 'wetland': 'marsh'} | POLYGON ((9.65496 47.51582, 9.65567 47.51564, ... |
| way/640953866 | {'natural': 'wetland', 'source:outline': 'VoGI... | POLYGON ((9.65471 47.5159, 9.65445 47.51598, 9... |
61674 rows × 2 columns
Plot the forests using GeoPandas¶
Filter all polygons and features with landuse=forest.
In [17]:
Copied!
features_gdf[
features_gdf.geom_type.isin(("Polygon", "MultiPolygon"))
& features_gdf.tags.apply(lambda x: "landuse" in x and x["landuse"] == "forest")
].plot(color="green")
features_gdf[
features_gdf.geom_type.isin(("Polygon", "MultiPolygon"))
& features_gdf.tags.apply(lambda x: "landuse" in x and x["landuse"] == "forest")
].plot(color="green")
Out[17]:
<Axes: >
