### Install Black Source: https://pyrosm.readthedocs.io/en/stable/_sources/contributions.rst Install the Black formatter via pip. ```bash pip install black ``` -------------------------------- ### List downloads output Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Example output from listing downloaded files. ```text ['/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/Helsinki.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/Paris.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/estonia-latest.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/finland-latest.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/malta-latest.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/newyorkcity.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/south-america-latest.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/spain-latest.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/tallinn-estonia.osm.pbf', '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/us-west-latest.osm.pbf'] ``` -------------------------------- ### Install Benchmark Dependencies Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Commands to install Python, Java, and R dependencies required for the benchmark suite. ```bash # Python tools (conda-forge recommended for the geo-stack) # ipywidgets is needed so QuackOSM/DuckDB can manage their progress bar inside Jupyter. mamba install -c conda-forge pyrosm osmnx quackosm pyosmium osmium-tool geopandas matplotlib pandas ipywidgets # Osmosis (Java command-line tool) + a Java runtime mamba install -c conda-forge openjdk # then download Osmosis from https://github.com/openstreetmap/osmosis/releases # and make sure the `osmosis` launcher is on your PATH. # osmextract is an R package (run via the companion osmextract_benchmark_scaling.ipynb). # Install R + sf (GDAL) + the Jupyter R kernel from conda-forge, then osmextract from CRAN: mamba install -c conda-forge r-base r-sf r-irkernel R -e 'install.packages("osmextract", repos="https://cloud.r-project.org")' R -e 'IRkernel::installspec()' # registers the "R" Jupyter kernel ``` -------------------------------- ### Install Pyrosm Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb The recommended way to install the library is via pip. ```bash pip install pyrosm ``` -------------------------------- ### Download and crop output Source: https://pyrosm.readthedocs.io/en/stable/downloading_data.html Example output showing the file paths for cropped and uncropped downloads. ```text Geofabrik extract covering the area: 'England' (id: england) Cropped filename: /var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/bbox_-0.245_50.798_-0.016_50.892.osm.pbf Geofabrik extract covering the area: 'England' (id: england) No cropping: /var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/england-latest.osm.pbf ``` -------------------------------- ### Example benchmark output logs Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/osmextract_benchmark_scaling.html Sample console output showing the detection of the input file and the version information for the packages used. ```text The input place was matched with: Helsinki The chosen file was already detected in the download directory. Skip downloading. ``` ```text osmextract 0.6.0 | sf 1.1.0 PBF: /Users/tenkanh2/Library/Application Support/org.R-project.R/R/osmextract/bbbike_Helsinki.osm.pbf ``` -------------------------------- ### Example benchmark output Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Displays the console output showing the performance results for the PBF cropping task. ```text Task 3 - crop Finland to the Helsinki region (buildings = count read back): pyrosm 98.6 s 59.6 MB 208,632 buildings pyosmium 226.3 s 59.6 MB 208,632 buildings osmium-tool 14.8 s 59.7 MB 208,632 buildings ``` -------------------------------- ### Setup and Benchmark Utilities for OSMExtract Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/osmextract_benchmark_scaling.html Initializes the environment, defines PBF resolution logic, and provides functions to measure peak memory and execution time for osmextract tasks. ```R # Buildings setup: read the manifest, define the per-area benchmark (time + peak memory, measured # in a separate process like the Python tools), and accumulate results so each area can be run -- # and re-run -- in its own cell below. areas <- read.csv("scaling_areas.csv", stringsAsFactors = FALSE) worker <- normalizePath("bench_worker.R", mustWork = FALSE) if (!file.exists(worker)) stop("bench_worker.R must sit next to this notebook") time_flag <- if (Sys.info()[["sysname"]] == "Darwin") "-l" else "-v" time_bin <- if (file.exists("/usr/bin/time")) "/usr/bin/time" else "time" rscript <- file.path(R.home("bin"), "Rscript") if (!file.exists(rscript)) rscript <- Sys.which("Rscript") if (!nzchar(rscript) || !file.exists(rscript)) stop("Could not find Rscript for this R installation") find_python <- function() { candidates <- c(Sys.which("python"), file.path(dirname(dirname(R.home())), "bin", "python")) candidates <- candidates[nzchar(candidates) & file.exists(candidates)] if (length(candidates) == 0) return(NA_character_) candidates[1] } rebuild_kamppi_pbf <- function() { out <- file.path(tempdir(), "kamppi.osm.pbf") if (file.exists(out)) return(normalizePath(out)) if (!exists("pbf") || !file.exists(pbf)) { stop("Kamppi PBF is missing and the Helsinki PBF from oe_get() is not available; re-run the first cell") } python <- find_python() if (is.na(python)) { stop("Kamppi PBF is missing and Python was not found; re-run the setup cell in benchmarks_scaling.ipynb") } script <- paste( "from pyrosm import OSM", "import sys", "OSM(sys.argv[1], bounding_box=[24.920, 60.162, 24.942, 60.172]).to_pbf(sys.argv[2])", sep = "\n" ) log <- suppressWarnings(system2(python, c("-c", script, pbf, out), stdout = TRUE, stderr = TRUE)) if (!file.exists(out)) { detail <- paste(tail(log[nzchar(log)], 8), collapse = " | ") stop(sprintf("Could not rebuild Kamppi PBF: %s", detail)) } out } resolve_area_pbf <- function(area_label, pbf_path) { if (file.exists(pbf_path)) return(normalizePath(pbf_path)) if (identical(area_label, "Kamppi")) return(rebuild_kamppi_pbf()) stop(sprintf("PBF for '%s' not found: %s. Re-run the setup cell in benchmarks_scaling.ipynb.", area_label, pbf_path)) } parse_peak_rss_mb <- function(output_lines) { text <- paste(output_lines, collapse = "\n") # macOS `/usr/bin/time -l`: " maximum resident set size" m <- regmatches(text, regexpr("[0-9]+ +maximum resident set size", text)) if (length(m) == 1) return(as.numeric(sub(" .*", "", m)) / 1e6) # bytes -> MB # GNU `/usr/bin/time -v`: "Maximum resident set size (kbytes): " m <- regmatches(text, regexpr("Maximum resident set size \\(kbytes\\): [0-9]+", text)) if (length(m) == 1) return(as.numeric(sub(".*: ", "", m)) / 1e3) # kB -> MB NA_real_ } bench_oe <- function(worker_task, csv_task, area, pbf, repeats) { runtimes <- numeric(0); peaks <- numeric(0); n <- NA_integer_; status <- "ok" for (i in seq_len(repeats)) { out <- tryCatch(suppressWarnings(system2(time_bin, c(time_flag, rscript, worker, worker_task, pbf), stdout = TRUE, stderr = TRUE)), error = function(e) { status <<- paste("error:", conditionMessage(e)); character(0) }) if (length(out) == 0) break data_line <- grep("^[0-9.]+,[0-9]+$", out, value = TRUE) # "seconds,features" from the worker if (length(data_line) == 0) { detail <- paste(tail(out[nzchar(out)], 8), collapse = " | ") status <- if (nzchar(detail)) paste("error:", detail) else "error: no worker output" break } parts <- as.numeric(strsplit(data_line[length(data_line)], ",")[[1]]) runtimes <- c(runtimes, parts[1]); n <- as.integer(parts[2]) peaks <- c(peaks, parse_peak_rss_mb(out)) } if (length(runtimes) == 0) return(data.frame(task = csv_task, area = area, tool = "osmextract", seconds = NA_real_, peak_mb = NA_real_, features = NA_integer_, status = status, stringsAsFactors = FALSE)) data.frame(task = csv_task, area = area, tool = "osmextract", seconds = round(median(runtimes), 2), peak_mb = round(median(peaks), 1), features = n, status = "ok", stringsAsFactors = FALSE) } ``` -------------------------------- ### Benchmark Output Log Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Example output showing file paths and sizes for the defined benchmark areas. ```text Region PBF : /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/Helsinki.osm.pbf (62.9 MB) Finland PBF: /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/finland-latest.osm.pbf (725.8 MB) Buildings area ladder: Kamppi /var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/kamppi.osm.pbf ( 6.1 MB) Helsinki region /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/Helsinki.osm.pbf ( 62.9 MB) New York City /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/newyorkcity.osm.pbf ( 151.6 MB) Paris /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/Paris.osm.pbf ( 238.4 MB) Finland /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/finland-latest.osm.pbf ( 725.8 MB) Spain /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/spain-latest.osm.pbf ( 1456.1 MB) South America /private/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/south-america-latest.osm.pbf ( 4027.3 MB) ``` -------------------------------- ### Benchmark Output Example Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Sample output showing performance metrics for the Kamppi area. ```text Kamppi (6 MB, up to 10x, timeout 120s): pyrosm (in-memory) 0.95 s 190 MB 4,717 lines pyrosm (out-of-core) 0.92 s 205 MB 4,717 lines quackosm (with columns) 13.84 s 1173 MB 4,711 lines quackosm (no columns) 13.93 s 538 MB 4,711 lines pyosmium 0.74 s 169 MB 4,771 lines osmium-tool 1.00 s 1541 MB 4,717 lines osmnx 26.11 s 481 MB 13,112 lines ``` -------------------------------- ### Install pyrosm via conda-forge Source: https://pyrosm.readthedocs.io/en/stable/_sources/faq.md Recommended installation command using mamba or conda. ```bash mamba install -c conda-forge pyrosm ``` -------------------------------- ### Benchmark results output Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/osmextract_benchmark_scaling.html Example output format showing task, area, tool, execution time, peak memory, feature count, and status. ```text task area tool seconds peak_mb features status 1 roads Kamppi osmextract 0.07 350.9 4716 ok ``` ```text task area tool seconds peak_mb features status 1 roads Helsinki region osmextract 3.14 1022.3 296845 ok ``` ```text task area tool seconds peak_mb features status 1 roads New York City osmextract 6.65 1584.9 669037 ok ``` ```text task area tool seconds peak_mb features status 1 roads Paris osmextract 7.34 1503.3 620227 ok ``` ```text task area tool seconds peak_mb features status 1 roads Finland osmextract 32.57 4657.9 2178539 ok ``` ```text task area tool seconds peak_mb features status 1 roads Spain osmextract 114.19 10010.6 5716619 ok ``` -------------------------------- ### Parallel Cropping Benchmark Output Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Example console output showing the performance results of the parallel cropping tasks. ```text Task 3 in parallel - more workers / threads: pyrosm, 10 workers (fast-mode) 24.9 s 83.7 MB 208,632 buildings pyrosm, 10 workers (compact) 64.5 s 59.6 MB 208,632 buildings pyosmium, 10 workers 228.0 s 59.6 MB 208,632 buildings osmium-tool, 10 threads 2.9 s 59.7 MB 208,632 buildings ``` -------------------------------- ### Example output of PBF export Source: https://pyrosm.readthedocs.io/en/stable/saving_and_cropping.html Displays the console output showing file sizes and layer counts after performing the export operations. ```text whole dataset 54.4 MiB buildings only 12.8 MiB combined: 176900 buildings, 255130 network edges ``` -------------------------------- ### Benchmark output for Kamppi Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Example output showing performance metrics for the Kamppi area. ```text Kamppi (6 MB, up to 10x, timeout 120s): pyrosm (in-memory) 0.72 s 179 MB 665 polys pyrosm (out-of-core) 0.83 s 177 MB 665 polys quackosm (with columns) 10.81 s 705 MB 663 polys quackosm (no columns) 10.25 s 525 MB 663 polys pyosmium 0.53 s 144 MB 663 polys osmium-tool 0.65 s 1203 MB 619 polys osmnx 4.38 s 307 MB 666 polys ``` -------------------------------- ### Install optional dependency for out-of-core reading Source: https://pyrosm.readthedocs.io/en/stable/_sources/installation.ipynb Required for caching layer results to GeoParquet files when using the out-of-core engine. ```bash $ pip install pyarrow ``` -------------------------------- ### Install optional dependencies for graph export Source: https://pyrosm.readthedocs.io/en/stable/_sources/installation.ipynb Required for exporting street networks to graphs using specific backends. ```bash $ pip install igraph networkx pandarm ``` -------------------------------- ### Setup benchmark worker and area manifest Source: https://pyrosm.readthedocs.io/en/stable/_sources/benchmarks/osmextract_benchmark_scaling.ipynb Loads the area manifest and configures the external worker process for measuring execution time and memory usage. ```r # Buildings setup: read the manifest, define the per-area benchmark (time + peak memory, measured # in a separate process like the Python tools), and accumulate results so each area can be run -- # and re-run -- in its own cell below. areas <- read.csv("scaling_areas.csv", stringsAsFactors = FALSE) worker <- normalizePath("bench_worker.R", mustWork = FALSE) if (!file.exists(worker)) stop("bench_worker.R must sit next to this notebook") time_flag <- if (Sys.info()[["sysname"]] == "Darwin") "-l" else "-v" time_bin <- if (file.exists("/usr/bin/time")) "/usr/bin/time" else "time" rscript <- file.path(R.home("bin"), "Rscript") if (!file.exists(rscript)) rscript <- Sys.which("Rscript") if (!nzchar(rscript) || !file.exists(rscript)) stop("Could not find Rscript for this R installation") find_python <- function() { candidates <- c(Sys.which("python"), file.path(dirname(dirname(R.home())), "bin", "python")) candidates <- candidates[nzchar(candidates) & file.exists(candidates)] if (length(candidates) == 0) return(NA_character_) candidates[1] } rebuild_kamppi_pbf <- function() { ``` -------------------------------- ### Initialize osmextract benchmark environment Source: https://pyrosm.readthedocs.io/en/stable/_sources/benchmarks/osmextract_benchmark_scaling.ipynb Configures the R environment, sets download timeouts, and retrieves the Helsinki PBF file for benchmarking. ```r # Same input as the Python benchmark (get_data("helsinki")): the latest Helsinki extract from # BBBike. osmextract fetches it itself via oe_get(provider = "bbbike") -- one of its core # capabilities. download_only + skip_vectortranslate just download the .osm.pbf and return its # path; it is used for the version banner and as a fallback source for clipping Kamppi. suppressPackageStartupMessages(library(osmextract)) REPEATS <- 3L options(timeout = max(600, getOption("timeout"))) # the ~62 MB download can exceed R's 60 s default pbf <- oe_get("Helsinki", provider = "bbbike", download_only = TRUE, skip_vectortranslate = TRUE, quiet = FALSE) cat("osmextract", as.character(packageVersion("osmextract")), "| sf", as.character(packageVersion("sf")), "\nPBF:", pbf, "\n") ``` -------------------------------- ### Install pyrosm via package managers Source: https://pyrosm.readthedocs.io/en/stable/_sources/installation.ipynb Use these commands to install pyrosm from the conda-forge channel or via pip. ```bash $ mamba install -c conda-forge pyrosm ``` ```bash $ micromamba install -c conda-forge pyrosm ``` ```bash $ pip install pyrosm ``` -------------------------------- ### Graph Simplification Output Source: https://pyrosm.readthedocs.io/en/stable/graphs.html Example output showing the reduction in graph complexity. ```text Full: 768 nodes, 1490 edges Simplified: 268 nodes, 577 edges ``` -------------------------------- ### Build documentation locally Source: https://pyrosm.readthedocs.io/en/stable/_sources/contributions.rst Generate HTML documentation from the docs folder using Sphinx. ```bash make html ``` -------------------------------- ### Initialize OSM reader with history file Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Download a sample history file and initialize the OSM reader. The reader automatically detects the .osh.pbf format. ```python from pyrosm import OSM, get_data # Download a small sample history file for Helsinki fp = get_data("helsinki_history_pbf") # Initialise the reader; pyrosm detects the .osh.pbf file as a history file osm = OSM(fp) ``` -------------------------------- ### Get boundary geometry Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Retrieve the boundary of a specific area to use as a spatial filter. ```python # Get the borough of Camden as our bounding box bounding_box = osm.get_boundaries(name="London Borough of Camden") bounding_box.plot() ``` -------------------------------- ### Initialize Benchmark Environment Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Imports and configuration constants used to manage benchmark execution and output readability. ```python import os import time import shutil import statistics import subprocess import tempfile import warnings from collections import namedtuple from pathlib import Path import pandas as pd import geopandas as gpd import matplotlib.pyplot as plt warnings.filterwarnings("ignore") # keep the benchmark output readable # Median window for the single-area road task. The buildings task sets its own per-area # repeats via the AREAS ladder below (small areas 10x; the big country files 3x). REPEATS = 3 # Cropping reads/writes a multi-hundred-MB file; once is enough. CROP_REPEATS = 1 ``` -------------------------------- ### Geofabrik extract URL output Source: https://pyrosm.readthedocs.io/en/stable/downloading_data.html Example output showing the URL of the identified Geofabrik extract. ```text 'https://download.geofabrik.de/europe/united-kingdom/england-latest.osm.pbf' ``` -------------------------------- ### Perform Editable Build Source: https://pyrosm.readthedocs.io/en/stable/_sources/contributions.rst Command to install pyrosm in editable mode, compiling Cython extensions in-place. ```bash pip install -e . --no-build-isolation ``` -------------------------------- ### Initialize OSM reader with test data Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Retrieve a test PBF dataset path using get_data and initialize the OSM reader instance. ```python import pyrosm # Get filepath to test PBF dataset fp = pyrosm.get_data("test_pbf") print("Filepath to test data:", fp) # Initialize the OSM object osm = pyrosm.OSM(fp) # See the type print("Type of 'osm' instance: ", type(osm)) ``` -------------------------------- ### Access to_graph documentation Source: https://pyrosm.readthedocs.io/en/stable/graphs.html Displays help documentation for the to_graph method. ```python from pyrosm import OSM, get_data osm = OSM(get_data("test_pbf")) # To see all available parameters and their explanations, simply call help help(osm.to_graph) ``` -------------------------------- ### Filter Tags to Keep Source: https://pyrosm.readthedocs.io/en/stable/_sources/tags_and_columns.ipynb Limit the columns created from OSM tags by specifying a subset in the get method. ```python from pyrosm import OSM, get_data osm = OSM(get_data("test_pbf")) buildings = osm.get_buildings(tags_to_keep=["building", "addr:city"]) buildings.columns.tolist() ``` -------------------------------- ### Initialize OSM with out-of-core engine Source: https://pyrosm.readthedocs.io/en/stable/api/pyrosm.OSM.html Demonstrates initializing the OSM reader with the out-of-core engine and parallel workers, requiring a main guard on certain operating systems. ```python if __name__ == "__main__": osm = OSM(fp, engine="out_of_core", workers="auto") buildings = osm.get_buildings() ``` -------------------------------- ### Create Development Environment Source: https://pyrosm.readthedocs.io/en/stable/_sources/contributions.rst Commands to create a development environment using mamba or micromamba from provided configuration files. ```bash mamba env create -f ci/314-conda.yaml ``` ```bash micromamba create -f ci/314-conda.yaml ``` -------------------------------- ### Initialize OSM reader with dataset Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Download and initialize the OSM reader for a specific region. ```python from pyrosm import OSM, get_data # Download a dataset for Greater London (update if exists in the temp already) fp = get_data("Greater London", update=True) osm = OSM(fp) ``` -------------------------------- ### Geofabrik extract lookup output Source: https://pyrosm.readthedocs.io/en/stable/downloading_data.html Example output showing the identified Geofabrik extract for a given bounding box. ```text Geofabrik extract covering the area: 'England' (id: england) ``` -------------------------------- ### Initialize OSM and Retrieve Network Data Source: https://pyrosm.readthedocs.io/en/stable/graphs.html Load Helsinki region data and extract the driving network and restaurant POIs. ```python from pyrosm import OSM, get_data import numpy as np import matplotlib.pyplot as plt osm = OSM(get_data("helsinki")) nodes, edges = osm.get_network(network_type="driving", nodes=True) restaurants = osm.get_pois(custom_filter={"amenity": ["restaurant"]}) ``` -------------------------------- ### Activate Development Environment Source: https://pyrosm.readthedocs.io/en/stable/_sources/contributions.rst Command to activate the created development environment. ```bash mamba activate test ``` -------------------------------- ### Compact and repack PBF output Source: https://pyrosm.readthedocs.io/en/stable/_sources/saving_and_cropping.ipynb Demonstrates using compact and repack flags to reduce the output file size at the cost of processing speed. ```python default = OSM(fp, bounding_box=bbox).to_pbf() compact = OSM(fp, bounding_box=bbox).to_pbf(compact=True) repack = OSM(fp, bounding_box=bbox).to_pbf(repack=True) for label, path in [("default", default), ("compact", compact), ("repack", repack)]: size_mb = os.path.getsize(path) / (1024 * 1024) print(f"{label:8s} {size_mb:.1f} MiB") ``` -------------------------------- ### Initialize and Run OSMExtract Benchmarks Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/osmextract_benchmark_scaling.html Initializes the results data frame and defines helper functions to execute and persist benchmark results for specific areas. ```R if (!exists("oe_scaling_results")) oe_scaling_results <- data.frame() # Upsert one (task, area) result into oe_scaling_results and write the CSV immediately -- so each # tool/area is exported as it finishes, and re-running an area replaces just that row. run_oe_for <- function(worker_task, csv_task, area_label) { row <- areas[areas$area == area_label, ] if (nrow(row) == 0) stop(sprintf("area '%s' not in scaling_areas.csv", area_label)) pbf_path <- resolve_area_pbf(row$area[1], row$path[1]) result <- bench_oe(worker_task, csv_task, row$area[1], pbf_path, row$repeats[1]) keep <- !(oe_scaling_results$task == csv_task & oe_scaling_results$area == area_label) oe_scaling_results <<- rbind(oe_scaling_results[keep, ], result) write.csv(oe_scaling_results, "osmextract_scaling_results.csv", row.names = FALSE) print(result) } run_oe_buildings <- function(area_label) run_oe_for("buildings", "buildings", area_label) run_oe_network <- function(area_label) run_oe_for("network", "roads", area_label) ``` -------------------------------- ### Initialize Pyrosm with an OSM file Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb The primary entry point for using the library is creating an OSM object with a path to a PBF file. ```python from pyrosm import OSM # Initialize the OSM object with the path to your PBF file fp = "path/to/your/data.osm.pbf" osm = OSM(fp) ``` -------------------------------- ### Download and crop OSM data by bounding box Source: https://pyrosm.readthedocs.io/en/stable/downloading_data.html Demonstrates downloading data with default cropping enabled and disabling cropping to keep the full extract. ```python # Download the covering extract and crop it to the box fp = get_data_by_bbox([-0.245, 50.798, -0.016, 50.892]) print("Cropped filename:", fp) # crop=False keeps the full extract instead fp = get_data_by_bbox([-0.245, 50.798, -0.016, 50.892], crop=False) print("No cropping:", fp) ``` -------------------------------- ### Configure and execute building parsing benchmarks Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Defines the reader configurations and the execution logic for benchmarking tools against PBF files of varying sizes. ```python SCALE_READERS = [ # label, worker tool, only on these areas (None = all) ("pyrosm (in-memory)", "pyrosm-in-memory", {"Kamppi", "Helsinki region", "New York City", "Paris", "Finland"}), ("pyrosm (out-of-core)", "pyrosm-out-of-core", None), ("quackosm (with columns)", "quackosm-with-columns", None), ("quackosm (no columns)", "quackosm-no-columns", None), ("pyosmium", "pyosmium", {"Kamppi", "Helsinki region", "New York City", "Paris"}), ("osmium-tool", "osmium-tool", {"Kamppi", "Helsinki region", "New York City", "Paris", "Finland"}), ("osmnx", "osmnx", {"Kamppi", "Helsinki region", "New York City", "Paris"}), ] SMALL_AREAS = {"Kamppi", "Helsinki region"} AREA = {area.label: area for area in AREAS} # OSMnx queries Overpass for a bounding box rather than reading the file; use the same extent as # the local data (Kamppi's clip box; Helsinki's data extent, read once and cached). _OSMNX_BBOX = {"Kamppi": KAMPPI_BBOX} def _osmnx_bbox(area): if area.label not in _OSMNX_BBOX: _OSMNX_BBOX[area.label] = list(OSM(area.pbf).get_buildings().total_bounds) return _OSMNX_BBOX[area.label] def run_buildings_for_area(area): """Benchmark every applicable tool on one area and record the rows. Re-running replaces this area's earlier buildings rows, so each area cell can be run on its own.""" print(f"{area.label} ({os.path.getsize(area.pbf)/1e6:.0f} MB, up to {area.repeats}x, " f"timeout {area.timeout}s):") for label, worker_tool, only_areas in SCALE_READERS: if only_areas is not None and area.label not in only_areas: continue extra_args = _osmnx_bbox(area) if worker_tool == "osmnx" else None if worker_tool == "osmnx": reps = 1 # one live Overpass query is enough elif worker_tool in ["pyosmium", "osmium-tool"] and area.label not in SMALL_AREAS: reps = 1 # cap the slow streaming reader on big files else: reps = area.repeats runs = [] for _ in range(reps): run = monitor_parse(worker_tool, area.pbf, area.timeout, extra_args=extra_args) runs.append(run) if run["status"] != "ok": break # don't repeat an OOM / timeout / crash ok_runs = [r for r in runs if r["status"] == "ok"] if ok_runs: row = {"task": "buildings", "area": area.label, "tool": label, "seconds": round(statistics.median(r["seconds"] for r in ok_runs), 2), "peak_mb": round(statistics.median(r["peak_mb"] for r in ok_runs), 1), "features": ok_runs[-1]["features"], "status": "ok"} print(f" {label:22} {row['seconds']:8.2f} s {row['peak_mb']:8.0f} MB " f"{row['features']:>9,} polys") else: row = {"task": "buildings", "area": area.label, "tool": label, "seconds": None, "peak_mb": max(r["peak_mb"] for r in runs), "features": None, "status": runs[-1]["status"]} print(f" {label:22} {runs[-1]['status']:>22} {row['peak_mb']:8.0f} MB peak") record_result(row) ``` -------------------------------- ### OSM Class Constructor Source: https://pyrosm.readthedocs.io/en/stable/_sources/reference.rst The OSM class is the main entry point for the library. It is initialized with a path to an OpenStreetMap PBF file. ```APIDOC ## OSM(filepath) ### Description Initializes the OSM reader with a PBF file. ### Parameters - **filepath** (str) - Required - Path to the PBF file. ``` -------------------------------- ### Import math library Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Imports the standard math library. ```python import math ``` -------------------------------- ### Display benchmark output Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Prints a confirmation message for the loaded osmextract data. ```text Loaded 12 osmextract row(s) from osmextract_scaling_results.csv ``` -------------------------------- ### Initialize Pyrosm and extract network Source: https://pyrosm.readthedocs.io/en/stable/graphs.html Load OSM data and retrieve walkable roads as nodes and edges. ```python from pyrosm import OSM, get_data import osmnx as ox # Initialize the reader osm = OSM(get_data("helsinki_pbf")) # Get all walkable roads and the nodes nodes, edges = osm.get_network(nodes=True) # Check first rows in the edge edges.head() ``` -------------------------------- ### Initialize the OSM reader object Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Import the pyrosm library and access the help documentation for the OSM reader initialization. ```python # Import the library import pyrosm # Print information about the basic usage of the `OSM` reader object help(pyrosm.OSM.__init__) ``` -------------------------------- ### OSM Class Constructor Source: https://pyrosm.readthedocs.io/en/stable/reference.html Initializes the OpenStreetMap PBF reader object. ```APIDOC ## OSM(filepath, bounding_box=None, keep_metadata=False, ...) ### Description Constructs an OSM reader object from a PBF file. ### Parameters - **filepath** (str) - Required - Path to the PBF file. - **bounding_box** (list/tuple) - Optional - Bounding box to filter data. - **keep_metadata** (bool) - Optional - Whether to keep OSM metadata. ``` -------------------------------- ### Initialize OSM reader and inspect columns Source: https://pyrosm.readthedocs.io/en/stable/_sources/tags_and_columns.ipynb Initializes the OSM reader with test data and displays existing columns and unique tag attributes. ```python from pyrosm import OSM, get_data # Get test data fp = get_data("test_pbf") # Initialize the reader osm = OSM(fp) buildings = osm.get_buildings() # Print info print("Existing columns:\n", buildings.columns) print("\nAdditional attributes in the 'tags': \n", buildings.tags.unique()) ``` -------------------------------- ### Initialize OSM with bounding box Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Apply a geometric filter to the OSM reader instance. ```python # Get the shapely geometry from GeoDataFrame bbox_geom = bounding_box['geometry'].values[0] # Initiliaze with bounding box osm = OSM(fp, bounding_box=bbox_geom) ``` -------------------------------- ### Download Data with Flexible Naming Source: https://pyrosm.readthedocs.io/en/stable/downloading_data.html Demonstrates that the library can resolve variations in place name formatting. ```python # Passing names in slightly different style does not matter fp = get_data("Rio de Janeiro") print(fp) ``` ```text Downloaded Protobuf data 'RiodeJaneiro.osm.pbf' (35.3 MB) to: '/var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/RiodeJaneiro.osm.pbf' /var/folders/f2/pgp09jl542zffhtrt2hx8zhh0000gp/T/pyrosm/RiodeJaneiro.osm.pbf ``` -------------------------------- ### Read OSM data from a file Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb Initialize the OSM reader with a path to an OSM PBF file. ```python from pyrosm import OSM # Initialize the OSM reader # Replace 'path/to/your/file.osm.pbf' with the actual path to your file osm = OSM("path/to/your/file.osm.pbf") ``` -------------------------------- ### Fork and Clone the Repository Source: https://pyrosm.readthedocs.io/en/stable/_sources/contributions.rst Commands to clone the repository and link it to the upstream project. ```bash git clone git@github.com:your-user-name/pyrosm.git pyrosm-yourname cd pyrosm-yourname git remote add upstream https://github.com/pyrosm/pyrosm.git ``` -------------------------------- ### Initialize Pyrosm Reader Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb Initialize the OSM reader with a path to a PBF file. ```python from pyrosm import OSM # Initialize the reader osm = OSM("path/to/your/data.osm.pbf") ``` -------------------------------- ### Read OSM data from a file Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb Initialize the OSM reader with a path to an OpenStreetMap PBF file. ```python from pyrosm import OSM # Initialize the OSM object with the path to the PBF file osm = OSM("path/to/your/data.osm.pbf") ``` -------------------------------- ### Configure visualization styling Source: https://pyrosm.readthedocs.io/en/stable/benchmarks/benchmarks_scaling.html Sets up matplotlib parameters for consistent chart styling and defines color constants for tool highlighting. ```python # ---- Styling for the result charts ---- plt.rcParams.update({ "figure.facecolor": "white", "savefig.facecolor": "white", "savefig.dpi": 200, "savefig.bbox": "tight", "axes.edgecolor": "#888888", "axes.linewidth": 0.8, "axes.spines.top": False, "axes.spines.right": False, "axes.grid": True, "axes.axisbelow": True, "grid.color": "#ECECF1", "grid.linewidth": 1.0, "axes.titlesize": 13, "axes.titleweight": "bold", "axes.labelsize": 11, "axes.labelcolor": "#333333", "xtick.color": "#333333", "ytick.color": "#333333", }) # pyrosm is highlighted; every other tool shares one muted colour. PYROSM_COLOR = "#2A849D" OTHER_COLOR = "#BC778F" def bar_colors(tools): return [PYROSM_COLOR if tool == "pyrosm" else OTHER_COLOR for tool in tools] # Result figures are written here (relative to this notebook). FIGURE_DIR = "figures" os.makedirs(FIGURE_DIR, exist_ok=True) ``` -------------------------------- ### Run osmextract for specific area Source: https://pyrosm.readthedocs.io/en/stable/_sources/benchmarks/osmextract_benchmark_scaling.ipynb Retrieves area configuration and prepares for benchmark execution. ```R # Upsert one (task, area) result into oe_scaling_results and write the CSV immediately -- so each # tool/area is exported as it finishes, and re-running an area replaces just that row. run_oe_for <- function(worker_task, csv_task, area_label) { row <- areas[areas$area == area_label, ] if (nrow(row) == 0) stop(sprintf("area '%s' not in scaling_areas.csv", area_label)) pbf_path <- resolve_area_pbf(row$area[1], row$path[1]) ``` -------------------------------- ### OSM.list_downloads() Source: https://pyrosm.readthedocs.io/en/stable/api/pyrosm.OSM.list_downloads.html Lists the PBF files downloaded by get_data in the default download directory. ```APIDOC ## OSM.list_downloads() ### Description List the PBF files downloaded by `get_data` in the default download directory (`/pyrosm`). ### Returns - **list of str**: The downloaded files’ paths. ``` -------------------------------- ### Initialize OSM reader and extract driving network Source: https://pyrosm.readthedocs.io/en/stable/quickstart.html Initializes the OSM reader with a file path and retrieves the driving network. Use the out-of-core engine for large datasets to manage memory usage. ```python %matplotlib inline osm = OSM(fp) drive = osm.get_network("driving") drive.head(3) ``` -------------------------------- ### Initialize Pyrosm with OSM data Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb Load an OSM PBF file into a Pyrosm object to begin processing. ```python from pyrosm import OSM # Initialize the OSM object with the path to your PBF file osm = OSM("path/to/your/data.osm.pbf") ``` -------------------------------- ### Execute osmextract benchmark Source: https://pyrosm.readthedocs.io/en/stable/_sources/benchmarks/osmextract_benchmark_scaling.ipynb Runs a benchmark task multiple times and returns a data frame containing median runtime, peak memory, and feature count. ```R bench_oe <- function(worker_task, csv_task, area, pbf, repeats) { runtimes <- numeric(0); peaks <- numeric(0); n <- NA_integer_; status <- "ok" for (i in seq_len(repeats)) { out <- tryCatch(suppressWarnings(system2(time_bin, c(time_flag, rscript, worker, worker_task, pbf), stdout = TRUE, stderr = TRUE)), error = function(e) { status <<- paste("error:", conditionMessage(e)); character(0) }) if (length(out) == 0) break data_line <- grep("^[0-9.]+,[0-9]+$", out, value = TRUE) # "seconds,features" from the worker if (length(data_line) == 0) { detail <- paste(tail(out[nzchar(out)], 8), collapse = " | ") status <- if (nzchar(detail)) paste("error:", detail) else "error: no worker output" break } parts <- as.numeric(strsplit(data_line[length(data_line)], ",")[[1]]) runtimes <- c(runtimes, parts[1]); n <- as.integer(parts[2]) peaks <- c(peaks, parse_peak_rss_mb(out)) } if (length(runtimes) == 0) return(data.frame(task = csv_task, area = area, tool = "osmextract", seconds = NA_real_, peak_mb = NA_real_, features = NA_integer_, status = status, stringsAsFactors = FALSE)) data.frame(task = csv_task, area = area, tool = "osmextract", seconds = round(median(runtimes), 2), peak_mb = round(median(peaks), 1), features = n, status = "ok", stringsAsFactors = FALSE) } ``` -------------------------------- ### Read OSM data from a file Source: https://pyrosm.readthedocs.io/en/stable/_sources/quickstart.ipynb Initialize the OSM object by pointing to a PBF file. ```python from pyrosm import OSM # Initialize the OSM object # Provide the path to the PBF file fp = "path/to/your/data.osm.pbf" osm = OSM(fp) ``` -------------------------------- ### Apply a custom filter to OSM data Source: https://pyrosm.readthedocs.io/en/stable/custom_filter.html Demonstrates initializing the OSM reader and applying a custom filter to extract specific building types. ```python from pyrosm import OSM, get_data # Get test data fp = get_data("test_pbf") # Initialize the reader osm = OSM(fp) # Read buildings with custom filter my_filter = {"building": ["residential", "retail"]} buildings = osm.get_buildings(custom_filter=my_filter) # Plot title = "Filtered buildings: " + ", ".join(buildings["building"].unique()) ax = buildings.plot(column="building", cmap="RdBu", legend=True) ax.set_title(title); ``` -------------------------------- ### OSM.to_pbf() Source: https://pyrosm.readthedocs.io/en/stable/api/pyrosm.OSM.to_pbf.html Crops the source PBF file by the object's bounding box and writes a valid, re-readable *.osm.pbf file to disk. ```APIDOC ## OSM.to_pbf(output_path=None, keep_relations=True, workers=1, compact=False, repack=False) ### Description Crop the source PBF by this object’s bounding_box and write a valid, re-readable *.osm.pbf to disk. The process streams the source file blob-by-blob to maintain memory efficiency. ### Parameters - **output_path** (str, optional) - Where to write the cropped PBF. If None, a temporary file is created. - **keep_relations** (bool) - Whether to include relations referencing kept nodes or ways. Defaults to True. - **workers** (int) - Number of worker processes for parallel processing. Defaults to 1. - **compact** (bool) - Whether to prune the string table in each output block to reduce file size. Defaults to False. - **repack** (bool) - Whether to re-chunk elements into canonical, densely packed blocks. Defaults to False. ### Returns - **str** - The path of the written PBF file. ``` -------------------------------- ### List Available Cities Source: https://pyrosm.readthedocs.io/en/stable/_sources/downloading_data.ipynb Prints the list of all cities currently available for download. ```python # Prints a list of all cities that can be downloaded print(sources.cities.available) ``` -------------------------------- ### Read all boundaries Source: https://pyrosm.readthedocs.io/en/stable/reading_osm_data.html Initializes the OSM object and reads all administrative boundaries from the PBF file. ```python from pyrosm import OSM from pyrosm import get_data fp = get_data("helsinki_region_pbf") osm = OSM(fp) # Read all boundaries using the default settings boundaries = osm.get_boundaries() boundaries.plot(facecolor="none", edgecolor="blue") ``` -------------------------------- ### Download OSM data by name Source: https://pyrosm.readthedocs.io/en/stable/_sources/downloading_data.ipynb Use get_data to fetch PBF datasets for specific regions or cities directly to a temporary directory. ```python from pyrosm import get_data # Download data for the city of Helsinki fp = get_data("Helsinki") print(fp) ```