Hex2vec embedder
In [1]:
Copied!
import warnings
import matplotlib.pyplot as plt
import pandas as pd
from pytorch_lightning import seed_everything
from pytorch_lightning.loggers import CSVLogger
from srai.embedders import Hex2VecEmbedder
from srai.joiners import IntersectionJoiner
from srai.loaders import OSMOnlineLoader
from srai.neighbourhoods import H3Neighbourhood
from srai.plotting import plot_numeric_data, plot_regions
from srai.regionalizers import H3Regionalizer, geocode_to_region_gdf
import warnings
import matplotlib.pyplot as plt
import pandas as pd
from pytorch_lightning import seed_everything
from pytorch_lightning.loggers import CSVLogger
from srai.embedders import Hex2VecEmbedder
from srai.joiners import IntersectionJoiner
from srai.loaders import OSMOnlineLoader
from srai.neighbourhoods import H3Neighbourhood
from srai.plotting import plot_numeric_data, plot_regions
from srai.regionalizers import H3Regionalizer, geocode_to_region_gdf
In [2]:
Copied!
SEED = 71
seed_everything(SEED)
SEED = 71
seed_everything(SEED)
Seed set to 71
Out[2]:
71
Load data from OSM¶
First use geocoding to get the area
In [3]:
Copied!
area_gdf = geocode_to_region_gdf("Wrocław, Poland")
plot_regions(area_gdf, tiles_style="CartoDB positron")
area_gdf = geocode_to_region_gdf("Wrocław, Poland")
plot_regions(area_gdf, tiles_style="CartoDB positron")
Out[3]:
Make this Notebook Trusted to load map: File -> Trust Notebook
Next, download the data for the selected region and the specified tags. We're using OSMOnlineLoader here, as it's faster for low numbers of tags. In a real life scenario with more tags, you would likely want to use the OSMPbfLoader.
In [4]:
Copied!
tags = {
"leisure": "park",
"landuse": "forest",
"amenity": ["bar", "restaurant", "cafe"],
"water": "river",
"sport": "soccer",
}
loader = OSMOnlineLoader()
features_gdf = loader.load(area_gdf, tags)
folium_map = plot_regions(
area_gdf, colormap=["rgba(0,0,0,0)"], tiles_style="CartoDB positron"
)
features_gdf.explore(m=folium_map)
tags = {
"leisure": "park",
"landuse": "forest",
"amenity": ["bar", "restaurant", "cafe"],
"water": "river",
"sport": "soccer",
}
loader = OSMOnlineLoader()
features_gdf = loader.load(area_gdf, tags)
folium_map = plot_regions(
area_gdf, colormap=["rgba(0,0,0,0)"], tiles_style="CartoDB positron"
)
features_gdf.explore(m=folium_map)
Out[4]:
Make this Notebook Trusted to load map: File -> Trust Notebook
Prepare the data for embedding¶
After downloading the data, we need to prepare it for embedding. Namely - we need to regionalize the selected area, and join the features with regions.
In [5]:
Copied!
regionalizer = H3Regionalizer(resolution=9)
regions_gdf = regionalizer.transform(area_gdf)
plot_regions(regions_gdf, tiles_style="CartoDB positron")
regionalizer = H3Regionalizer(resolution=9)
regions_gdf = regionalizer.transform(area_gdf)
plot_regions(regions_gdf, tiles_style="CartoDB positron")
Out[5]:
Make this Notebook Trusted to load map: File -> Trust Notebook
In [6]:
Copied!
joiner = IntersectionJoiner()
joint_gdf = joiner.transform(regions_gdf, features_gdf)
joint_gdf
joiner = IntersectionJoiner()
joint_gdf = joiner.transform(regions_gdf, features_gdf)
joint_gdf
Out[6]:
| region_id | feature_id |
|---|---|
| 891e2040897ffff | node/280727473 |
| 891e2040d4bffff | node/300461026 |
| node/300461036 | |
| 891e2040d5bffff | node/300461042 |
| 891e2040887ffff | node/300461045 |
| ... | ... |
| 891e204098fffff | way/1437139462 |
| 891e20409bbffff | way/1437139462 |
| 891e2042447ffff | way/1442991568 |
| 891e204240bffff | way/1442991568 |
| 891e204241bffff | way/1442991568 |
4196 rows × 0 columns
Embedding¶
After preparing the data we can proceed with generating embeddings for the regions.
In [7]:
Copied!
neighbourhood = H3Neighbourhood(regions_gdf)
embedder = Hex2VecEmbedder([15, 10])
csv_logger = CSVLogger(save_dir="hex2vec_logs")
with warnings.catch_warnings():
warnings.simplefilter("ignore")
embeddings = embedder.fit_transform(
regions_gdf,
features_gdf,
joint_gdf,
neighbourhood,
trainer_kwargs={"max_epochs": 5, "accelerator": "cpu", "logger": csv_logger},
batch_size=100,
)
embeddings
neighbourhood = H3Neighbourhood(regions_gdf)
embedder = Hex2VecEmbedder([15, 10])
csv_logger = CSVLogger(save_dir="hex2vec_logs")
with warnings.catch_warnings():
warnings.simplefilter("ignore")
embeddings = embedder.fit_transform(
regions_gdf,
features_gdf,
joint_gdf,
neighbourhood,
trainer_kwargs={"max_epochs": 5, "accelerator": "cpu", "logger": csv_logger},
batch_size=100,
)
embeddings
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
| Name | Type | Params | Mode ----------------------------------------------- 0 | encoder | Sequential | 280 | train ----------------------------------------------- 280 Trainable params 0 Non-trainable params 280 Total params 0.001 Total estimated model params size (MB) 4 Modules in train mode 0 Modules in eval mode
`Trainer.fit` stopped: `max_epochs=5` reached.
Out[7]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
|---|---|---|---|---|---|---|---|---|---|---|
| region_id | ||||||||||
| 891e20446dbffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 |
| 891e2050ab7ffff | -0.314627 | -0.030717 | -0.267098 | -0.431051 | 0.007139 | 0.250178 | -0.417783 | 0.342695 | -0.114699 | -0.008824 |
| 891e2047517ffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 |
| 891e2043093ffff | -0.609562 | 0.242022 | -0.447098 | -0.750125 | -0.089139 | 0.296732 | -0.304271 | 0.461081 | -0.192829 | -0.324114 |
| 891e20452c3ffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 891e2041b9bffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 |
| 891e2041c2fffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 |
| 891e205186fffff | -0.804173 | 0.620528 | -0.015068 | -0.315197 | -0.349738 | 0.330779 | 0.146608 | 0.139494 | 0.288151 | -0.356280 |
| 891e20471d7ffff | 0.070379 | 0.208213 | 0.443370 | 0.580281 | -0.148675 | 0.003799 | 0.303462 | -0.377255 | 0.382121 | 0.156455 |
| 891e2041d47ffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 |
3168 rows × 10 columns
In [8]:
Copied!
metrics_df = pd.read_csv(csv_logger.log_dir + "/metrics.csv").dropna(
subset="train_f1_epoch"
)
fig, ax1 = plt.subplots(1, 1, figsize=(10, 5))
ax2 = ax1.twinx()
line1 = ax1.plot(metrics_df["epoch"], metrics_df["train_f1_epoch"])
line2 = ax2.plot(metrics_df["epoch"], metrics_df["train_loss_epoch"], color="orange")
ax1.legend(line1 + line2, ["F1", "Loss"], loc=7)
ax1.set_title("Training metrics")
ax1.set_ylabel("F1")
ax2.set_ylabel("Loss")
ax1.set_xlabel("Training epoch")
plt.show()
metrics_df = pd.read_csv(csv_logger.log_dir + "/metrics.csv").dropna(
subset="train_f1_epoch"
)
fig, ax1 = plt.subplots(1, 1, figsize=(10, 5))
ax2 = ax1.twinx()
line1 = ax1.plot(metrics_df["epoch"], metrics_df["train_f1_epoch"])
line2 = ax2.plot(metrics_df["epoch"], metrics_df["train_loss_epoch"], color="orange")
ax1.legend(line1 + line2, ["F1", "Loss"], loc=7)
ax1.set_title("Training metrics")
ax1.set_ylabel("F1")
ax2.set_ylabel("Loss")
ax1.set_xlabel("Training epoch")
plt.show()
Visualizing the embeddings' similarity¶
In [9]:
Copied!
from sklearn.cluster import KMeans
clusterizer = KMeans(n_clusters=5, random_state=SEED)
clusterizer.fit(embeddings)
embeddings["cluster"] = clusterizer.labels_
embeddings
from sklearn.cluster import KMeans
clusterizer = KMeans(n_clusters=5, random_state=SEED)
clusterizer.fit(embeddings)
embeddings["cluster"] = clusterizer.labels_
embeddings
Out[9]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | cluster | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| region_id | |||||||||||
| 891e20446dbffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 | 2 |
| 891e2050ab7ffff | -0.314627 | -0.030717 | -0.267098 | -0.431051 | 0.007139 | 0.250178 | -0.417783 | 0.342695 | -0.114699 | -0.008824 | 0 |
| 891e2047517ffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 | 2 |
| 891e2043093ffff | -0.609562 | 0.242022 | -0.447098 | -0.750125 | -0.089139 | 0.296732 | -0.304271 | 0.461081 | -0.192829 | -0.324114 | 0 |
| 891e20452c3ffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 | 2 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 891e2041b9bffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 | 2 |
| 891e2041c2fffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 | 2 |
| 891e205186fffff | -0.804173 | 0.620528 | -0.015068 | -0.315197 | -0.349738 | 0.330779 | 0.146608 | 0.139494 | 0.288151 | -0.356280 | 0 |
| 891e20471d7ffff | 0.070379 | 0.208213 | 0.443370 | 0.580281 | -0.148675 | 0.003799 | 0.303462 | -0.377255 | 0.382121 | 0.156455 | 1 |
| 891e2041d47ffff | 0.347264 | -0.222077 | -0.002131 | 0.311417 | 0.220451 | -0.120041 | -0.267070 | -0.100752 | -0.194564 | 0.363015 | 2 |
3168 rows × 11 columns
In [10]:
Copied!
plot_numeric_data(regions_gdf, "cluster", embeddings)
plot_numeric_data(regions_gdf, "cluster", embeddings)
Out[10]:
Make this Notebook Trusted to load map: File -> Trust Notebook