Load and save
In [1]:
Copied!
from pytorch_lightning import seed_everything
from srai.embedders import Hex2VecEmbedder
from srai.joiners import IntersectionJoiner
from srai.loaders import OSMOnlineLoader
from srai.neighbourhoods import H3Neighbourhood
from srai.plotting import plot_regions
from srai.regionalizers import H3Regionalizer, geocode_to_region_gdf
from pytorch_lightning import seed_everything
from srai.embedders import Hex2VecEmbedder
from srai.joiners import IntersectionJoiner
from srai.loaders import OSMOnlineLoader
from srai.neighbourhoods import H3Neighbourhood
from srai.plotting import plot_regions
from srai.regionalizers import H3Regionalizer, geocode_to_region_gdf
In [2]:
Copied!
SEED = 71
seed_everything(SEED)
SEED = 71
seed_everything(SEED)
Seed set to 71
Out[2]:
71
Load data from OSM¶
First use geocoding to get the area
In [3]:
Copied!
area_gdf = geocode_to_region_gdf("Wrocław, Poland")
plot_regions(area_gdf, tiles_style="CartoDB positron")
area_gdf = geocode_to_region_gdf("Wrocław, Poland")
plot_regions(area_gdf, tiles_style="CartoDB positron")
Out[3]:
Make this Notebook Trusted to load map: File -> Trust Notebook
Next, download the data for the selected region and the specified tags. We're using OSMOnlineLoader here, as it's faster for low numbers of tags. In a real life scenario with more tags, you would likely want to use the OSMPbfLoader.
In [4]:
Copied!
tags = {
"leisure": "park",
"landuse": "forest",
"amenity": ["bar", "restaurant", "cafe"],
"water": "river",
"sport": "soccer",
}
loader = OSMOnlineLoader()
features_gdf = loader.load(area_gdf, tags)
folium_map = plot_regions(area_gdf, colormap=["rgba(0,0,0,0)"], tiles_style="CartoDB positron")
features_gdf.explore(m=folium_map)
tags = {
"leisure": "park",
"landuse": "forest",
"amenity": ["bar", "restaurant", "cafe"],
"water": "river",
"sport": "soccer",
}
loader = OSMOnlineLoader()
features_gdf = loader.load(area_gdf, tags)
folium_map = plot_regions(area_gdf, colormap=["rgba(0,0,0,0)"], tiles_style="CartoDB positron")
features_gdf.explore(m=folium_map)
0%| | 0/7 [00:00<?, ?it/s]
Downloading leisure: park : 0%| | 0/7 [00:00<?, ?it/s]
Downloading leisure: park : 14%|█▍ | 1/7 [00:00<00:01, 4.52it/s]
Downloading landuse: forest : 14%|█▍ | 1/7 [00:00<00:01, 4.52it/s]
Downloading landuse: forest : 29%|██▊ | 2/7 [00:00<00:01, 3.93it/s]
Downloading amenity: bar : 29%|██▊ | 2/7 [00:00<00:01, 3.93it/s]
Downloading amenity: bar : 43%|████▎ | 3/7 [00:00<00:00, 4.82it/s]
Downloading amenity: restaurant: 43%|████▎ | 3/7 [00:00<00:00, 4.82it/s]
Downloading amenity: restaurant: 57%|█████▋ | 4/7 [00:00<00:00, 5.16it/s]
Downloading amenity: cafe : 57%|█████▋ | 4/7 [00:00<00:00, 5.16it/s]
Downloading amenity: cafe : 71%|███████▏ | 5/7 [00:00<00:00, 5.47it/s]
Downloading water: river : 71%|███████▏ | 5/7 [00:00<00:00, 5.47it/s]
Downloading water: river : 86%|████████▌ | 6/7 [00:01<00:00, 5.20it/s]
Downloading sport: soccer : 86%|████████▌ | 6/7 [00:01<00:00, 5.20it/s]
Downloading sport: soccer : 100%|██████████| 7/7 [00:01<00:00, 5.48it/s]
Downloading sport: soccer : 100%|██████████| 7/7 [00:01<00:00, 5.15it/s]
Out[4]:
Make this Notebook Trusted to load map: File -> Trust Notebook
Prepare the data for embedding¶
After downloading the data, we need to prepare it for embedding. Namely - we need to regionalize the selected area, and join the features with regions.
In [5]:
Copied!
regionalizer = H3Regionalizer(resolution=9)
regions_gdf = regionalizer.transform(area_gdf)
plot_regions(regions_gdf, tiles_style="CartoDB positron")
regionalizer = H3Regionalizer(resolution=9)
regions_gdf = regionalizer.transform(area_gdf)
plot_regions(regions_gdf, tiles_style="CartoDB positron")
Out[5]:
Make this Notebook Trusted to load map: File -> Trust Notebook
In [6]:
Copied!
joiner = IntersectionJoiner()
joint_gdf = joiner.transform(regions_gdf, features_gdf)
joint_gdf
joiner = IntersectionJoiner()
joint_gdf = joiner.transform(regions_gdf, features_gdf)
joint_gdf
Out[6]:
| region_id | feature_id |
|---|---|
| 891e2040417ffff | node/7189010475 |
| 891e20552b7ffff | way/217674535 |
| way/349170556 | |
| 891e2042837ffff | way/1056180351 |
| 891e20472d7ffff | way/56467850 |
| ... | ... |
| 891e204081bffff | node/3176881769 |
| node/6387855437 | |
| node/5436080752 | |
| node/6107114912 | |
| node/5454974225 |
3978 rows × 0 columns
Embedding¶
After preparing the data we can proceed with generating embeddings for the regions.
In [7]:
Copied!
import warnings
neighbourhood = H3Neighbourhood(regions_gdf)
embedder = Hex2VecEmbedder([15, 10])
with warnings.catch_warnings():
warnings.simplefilter("ignore")
embeddings = embedder.fit_transform(
regions_gdf,
features_gdf,
joint_gdf,
neighbourhood,
trainer_kwargs={"max_epochs": 5, "accelerator": "cpu"},
batch_size=100,
)
embeddings
import warnings
neighbourhood = H3Neighbourhood(regions_gdf)
embedder = Hex2VecEmbedder([15, 10])
with warnings.catch_warnings():
warnings.simplefilter("ignore")
embeddings = embedder.fit_transform(
regions_gdf,
features_gdf,
joint_gdf,
neighbourhood,
trainer_kwargs={"max_epochs": 5, "accelerator": "cpu"},
batch_size=100,
)
embeddings
0%| | 0/3168 [00:00<?, ?it/s]
100%|██████████| 3168/3168 [00:00<00:00, 32918.10it/s]
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
| Name | Type | Params --------------------------------------- 0 | encoder | Sequential | 280 --------------------------------------- 280 Trainable params 0 Non-trainable params 280 Total params 0.001 Total estimated model params size (MB)
`Trainer.fit` stopped: `max_epochs=5` reached.
Out[7]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
|---|---|---|---|---|---|---|---|---|---|---|
| region_id | ||||||||||
| 891e2040417ffff | 0.320989 | -0.184270 | 0.083933 | 0.461437 | 0.242006 | -0.224296 | -0.019514 | -0.166369 | -0.035694 | 0.354379 |
| 891e204e517ffff | 0.330435 | -0.223019 | -0.023478 | 0.298219 | 0.226723 | -0.125847 | -0.260601 | -0.110452 | -0.212940 | 0.377845 |
| 891e20552b7ffff | -0.446851 | 0.119537 | -0.362874 | -0.625829 | -0.026286 | 0.257514 | -0.312796 | 0.395354 | -0.112199 | -0.151935 |
| 891e2040357ffff | 0.330435 | -0.223019 | -0.023478 | 0.298219 | 0.226723 | -0.125847 | -0.260601 | -0.110452 | -0.212940 | 0.377845 |
| 891e2042837ffff | 0.095645 | 0.185171 | 0.441401 | 0.626441 | -0.151440 | -0.050300 | 0.316299 | -0.395272 | 0.361151 | 0.101272 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 891e204e0cfffff | 0.330435 | -0.223019 | -0.023478 | 0.298219 | 0.226723 | -0.125847 | -0.260601 | -0.110452 | -0.212940 | 0.377845 |
| 891e2042d83ffff | 0.079767 | -0.209288 | 0.139320 | -0.408071 | -0.241863 | 0.145487 | 0.148131 | 0.087076 | -0.023642 | -0.451293 |
| 891e205196fffff | -0.446851 | 0.119537 | -0.362874 | -0.625829 | -0.026286 | 0.257514 | -0.312796 | 0.395354 | -0.112199 | -0.151935 |
| 891e204081bffff | -0.048383 | 0.039810 | 0.040379 | 0.958419 | 0.507441 | -0.968734 | 1.050971 | 0.244109 | 0.857836 | 0.181167 |
| 891e204e127ffff | 0.330435 | -0.223019 | -0.023478 | 0.298219 | 0.226723 | -0.125847 | -0.260601 | -0.110452 | -0.212940 | 0.377845 |
3168 rows × 10 columns
Visualizing the embeddings' similarity¶
In [8]:
Copied!
embedder.save("./modello")
embedder.save("./modello")
In [9]:
Copied!
embedder_loaded = Hex2VecEmbedder.load("./modello")
embedder_loaded
embedder_loaded = Hex2VecEmbedder.load("./modello")
embedder_loaded
Out[9]:
<srai.embedders.hex2vec.embedder.Hex2VecEmbedder at 0x7f5c8c5e6680>
In [10]:
Copied!
from srai.embedders import Highway2VecEmbedder
from srai.loaders import OSMNetworkType, OSMWayLoader
d = OSMWayLoader(OSMNetworkType.DRIVE).load(area_gdf)
from srai.embedders import Highway2VecEmbedder
from srai.loaders import OSMNetworkType, OSMWayLoader
d = OSMWayLoader(OSMNetworkType.DRIVE).load(area_gdf)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/srai/loaders/osm_way_loader/osm_way_loader.py:229: FutureWarning: The clean_periphery argument has been deprecated and will be removed in the v2.0.0 release. Future behavior will be as though clean_periphery=True. See the OSMnx v2 migration guide: https://github.com/gboeing/osmnx/issues/1123 G_directed = ox.graph_from_polygon( /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/osmnx/_overpass.py:359: FutureWarning: `settings.timeout` is deprecated and will be removed in the v2.0.0 release: use `settings.requests_timeout` instead. See the OSMnx v2 migration guide: https://github.com/gboeing/osmnx/issues/1123 overpass_settings = _make_overpass_settings()
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/osmnx/_overpass.py:369: FutureWarning: `settings.timeout` is deprecated and will be removed in the v2.0.0 release: use `settings.requests_timeout` instead. See the OSMnx v2 migration guide: https://github.com/gboeing/osmnx/issues/1123
yield _overpass_request(data={"data": query_str})
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/srai/loaders/osm_way_loader/osm_way_loader.py:237: FutureWarning: The `get_undirected` function is deprecated and will be removed in the v2.0.0 release. Replace it with `convert.to_undirected` instead. See the OSMnx v2 migration guide: https://github.com/gboeing/osmnx/issues/1123 G_undirected = ox.utils_graph.get_undirected(G_directed)
In [11]:
Copied!
joint = joiner.transform(regions_gdf, d[1])
joint = joiner.transform(regions_gdf, d[1])
In [12]:
Copied!
highway2vec = Highway2VecEmbedder()
highway2vec.fit(regions_gdf, d[1], joint)
highway2vec = Highway2VecEmbedder()
highway2vec.fit(regions_gdf, d[1], joint)
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
| Name | Type | Params --------------------------------------- 0 | encoder | Sequential | 16.0 K 1 | decoder | Sequential | 16.2 K --------------------------------------- 32.1 K Trainable params 0 Non-trainable params 32.1 K Total params 0.128 Total estimated model params size (MB)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance.
`Trainer.fit` stopped: `max_epochs=10` reached.
In [13]:
Copied!
highway2vec.save("highway2vec")
highway2vec.save("highway2vec")
In [14]:
Copied!
vars(highway2vec)
vars(highway2vec)
Out[14]:
{'_model': Highway2VecModel(
(encoder): Sequential(
(0): Linear(in_features=218, out_features=64, bias=True)
(1): ReLU()
(2): Linear(in_features=64, out_features=30, bias=True)
)
(decoder): Sequential(
(0): Linear(in_features=30, out_features=64, bias=True)
(1): ReLU()
(2): Linear(in_features=64, out_features=218, bias=True)
)
),
'_hidden_size': 64,
'_embedding_size': 30,
'_is_fitted': True}
In [15]:
Copied!
Highway2VecEmbedder.load("highway2vec")
Highway2VecEmbedder.load("highway2vec")
Out[15]:
<srai.embedders.highway2vec.embedder.Highway2VecEmbedder at 0x7f5c943f0220>
In [16]:
Copied!
import geopandas as gpd
import pandas as pd
from shapely.geometry import Polygon
from srai.constants import REGIONS_INDEX
from srai.embedders import GTFS2VecEmbedder
features_gdf = gpd.GeoDataFrame(
{
"trip_count_at_6": [1, 0, 0],
"trip_count_at_7": [1, 1, 0],
"trip_count_at_8": [0, 0, 1],
"directions_at_6": [
{"A", "A1"},
{"B", "B1"},
{"C"},
],
},
geometry=gpd.points_from_xy([1, 2, 5], [1, 2, 2]),
index=pd.Index(name="stop_id", data=[1, 2, 3]),
)
regions_gdf = gpd.GeoDataFrame(
geometry=[
Polygon([(0, 0), (0, 3), (3, 3), (3, 0)]),
Polygon([(4, 0), (4, 3), (7, 3), (7, 0)]),
Polygon([(8, 0), (8, 3), (11, 3), (11, 0)]),
],
index=pd.Index(name=REGIONS_INDEX, data=["ff1", "ff2", "ff3"]),
)
joint_gdf = gpd.GeoDataFrame()
joint_gdf.index = pd.MultiIndex.from_tuples(
[("ff1", 1), ("ff1", 2), ("ff2", 3)],
names=[REGIONS_INDEX, "stop_id"],
)
embedder = GTFS2VecEmbedder(hidden_size=2, embedding_size=4)
embedder.fit(regions_gdf, features_gdf, joint_gdf)
res = embedder.transform(regions_gdf, features_gdf, joint_gdf)
res
import geopandas as gpd
import pandas as pd
from shapely.geometry import Polygon
from srai.constants import REGIONS_INDEX
from srai.embedders import GTFS2VecEmbedder
features_gdf = gpd.GeoDataFrame(
{
"trip_count_at_6": [1, 0, 0],
"trip_count_at_7": [1, 1, 0],
"trip_count_at_8": [0, 0, 1],
"directions_at_6": [
{"A", "A1"},
{"B", "B1"},
{"C"},
],
},
geometry=gpd.points_from_xy([1, 2, 5], [1, 2, 2]),
index=pd.Index(name="stop_id", data=[1, 2, 3]),
)
regions_gdf = gpd.GeoDataFrame(
geometry=[
Polygon([(0, 0), (0, 3), (3, 3), (3, 0)]),
Polygon([(4, 0), (4, 3), (7, 3), (7, 0)]),
Polygon([(8, 0), (8, 3), (11, 3), (11, 0)]),
],
index=pd.Index(name=REGIONS_INDEX, data=["ff1", "ff2", "ff3"]),
)
joint_gdf = gpd.GeoDataFrame()
joint_gdf.index = pd.MultiIndex.from_tuples(
[("ff1", 1), ("ff1", 2), ("ff2", 3)],
names=[REGIONS_INDEX, "stop_id"],
)
embedder = GTFS2VecEmbedder(hidden_size=2, embedding_size=4)
embedder.fit(regions_gdf, features_gdf, joint_gdf)
res = embedder.transform(regions_gdf, features_gdf, joint_gdf)
res
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
| Name | Type | Params --------------------------------------- 0 | encoder | Sequential | 16 1 | decoder | Sequential | 13 --------------------------------------- 29 Trainable params 0 Non-trainable params 29 Total params 0.000 Total estimated model params size (MB)
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py:298: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
`Trainer.fit` stopped: `max_epochs=10` reached.
Out[16]:
| 0 | 1 | 2 | 3 | |
|---|---|---|---|---|
| region_id | ||||
| ff1 | 0.687915 | 0.153115 | -0.648070 | -0.417944 |
| ff2 | 0.849668 | 0.173882 | -0.952428 | -0.055791 |
| ff3 | 0.913412 | 0.186434 | -1.057311 | 0.080431 |
In [17]:
Copied!
embedder.save("gtfs2vec")
embedder.save("gtfs2vec")
In [18]:
Copied!
a = embedder.load("gtfs2vec")
a = embedder.load("gtfs2vec")
In [19]:
Copied!
a = embedder.transform(regions_gdf, features_gdf, joint_gdf)
a = embedder.transform(regions_gdf, features_gdf, joint_gdf)
In [20]:
Copied!
a
a
Out[20]:
| 0 | 1 | 2 | 3 | |
|---|---|---|---|---|
| region_id | ||||
| ff1 | 0.687915 | 0.153115 | -0.648070 | -0.417944 |
| ff2 | 0.849668 | 0.173882 | -0.952428 | -0.055791 |
| ff3 | 0.913412 | 0.186434 | -1.057311 | 0.080431 |