### Add Spark Connector Dependency (Spark 3/Scala 2.12) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this command to start the Spark shell with the shaded connector dependency for Spark 3 and Scala 2.12. This resolves Protobuf version conflicts. ```bash bin/spark-shell --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' ``` -------------------------------- ### Add Spark Connector Dependency (PySpark) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this command to start PySpark with the shaded connector dependency for Spark 3 and Scala 2.12. This resolves Protobuf version conflicts. ```bash bin/pyspark --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' ``` -------------------------------- ### Add Spark Connector Dependency (Spark SQL) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this command to start Spark SQL with the shaded connector dependency for Spark 3 and Scala 2.12. This resolves Protobuf version conflicts. ```bash bin/spark-sql --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' ``` -------------------------------- ### Create DataFrame from OSM PBF (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Read OSM PBF data into a Spark DataFrame for analysis. This example counts primitives by type. ```scala import com.acervera.osm4scala.spark.OsmSqlEntity import org.apache.spark.sql.SparkSession object PrimitivesCounter { def main(args: Array[String]): Unit = { val spark = SparkSession .builder() .appName("Primitives counter") .getOrCreate() spark.read .format("osm.pbf") .load(args(0)) .groupBy(OsmSqlEntity.FIELD_TYPE) .count .show() } } ``` -------------------------------- ### Create DataFrame from OSM PBF (PySpark) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Read OSM PBF data into a Spark DataFrame for analysis using PySpark. This example counts primitives by type. ```python from pyspark.sql import SparkSession import sys if __name__ == '__main__': spark = SparkSession.builder.appName("Primitives counter").getOrCreate() spark.read.format("osm.pbf") \ .load(sys.argv[1]) \ .groupBy("type") \ .count() \ .show() ``` -------------------------------- ### Load OSM Data into DataFrame (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Load an OSM PBF file into a Spark DataFrame using the osm4scala connector in Scala. This example shows how to select and filter traffic signals. ```Scala val osmDF = spark.read.format("osm.pbf").load("/home/jovyan/work/monaco-anonymized.osm.pbf") osmDF.select("latitude", "longitude") .where("element_at(tags, 'highway') == 'traffic_signals'") .show ``` -------------------------------- ### Load OSM Data into DataFrame (Python) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Load an OSM PBF file into a Spark DataFrame using the osm4scala connector in Python. Note the '%%python' header required when using a Scala kernel. This example filters for traffic signals. ```Python %%python osm_df = spark.read.format("osm.pbf").load("/home/jovyan/work/monaco-anonymized.osm.pbf") osm_df.select("latitude", "longitude").where("element_at(tags, 'highway') == 'traffic_signals'").show() ``` -------------------------------- ### Install osm4scala Package in Jupyter Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this command in a Jupyter Notebook cell to add the osm4scala package for Spark v3.1.1 and Scala 2.12. This is required before you can use the osm4scala connector. ```Scala %%init_spark launcher.packages = ["com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11"] ``` -------------------------------- ### Run Tests for Scala 2.11 Source: https://simplexspatial.github.io/osm4scala/docs/contributing Execute the clean and test commands for Scala version 2.11. ```bash PATCH_211=false sbt clean +test ``` -------------------------------- ### Run Tests for Scala 2.11 with Patch Source: https://simplexspatial.github.io/osm4scala/docs/contributing Execute the clean and test commands for Scala version 2.11, applying a patch. ```bash PATCH_211=true sbt clean +test ``` -------------------------------- ### Create SQL Temporal View (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Load OSM PBF data and create a temporary SQL view for querying. ```scala val osmDF = spark.sqlContext.read.format("osm.pbf").load("") osmDF.createOrReplaceTempView("osm") ``` -------------------------------- ### Submit Spark Application (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Submit a Scala Spark application to your cluster, including the osm4scala package. ```bash bin/spark-submit \ --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' \ examples/spark-documentation/target/scala-2.12/osm4scala-examples-spark-documentation_2.12-1.0.11.jar \ /tmp/osm/monaco-anonymized.osm.pbf ``` -------------------------------- ### Submit Spark Application (PySpark) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Submit a PySpark application to your cluster, including the osm4scala package. ```bash bin/spark-submit \ --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' \ examples/spark-documentation/src/main/scala/com/acervera/osm4scala/examples/spark/documentation/PrimiriveCounter.py \ /tmp/osm/monaco-anonymized.osm.pbf ``` -------------------------------- ### Create Table from OSM PBF Data (SQL) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Creates a Spark SQL table using the osm.pbf format, pointing to the location of your OSM PBF files. ```sql CREATE TABLE osm USING osm.pbf LOCATION ''; ``` -------------------------------- ### Count Node Primitives in a PBF File Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library Demonstrates counting all node primitives in a PBF file using EntityIterator. Requires importing the library. ```scala import org.openstreetmap.osm4scala.core.EntityIterator import org.openstreetmap.osm4scala.core.OSMTypes EntityIterator.fromPbf(inputStream).count(_.osmModel == OSMTypes.Node) ``` -------------------------------- ### Add Osm4scala Bintray Repository Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library Optional: Add this resolver if you encounter issues resolving dependencies. This points to the Bintray repository for osm4scala. ```scala resolvers += "osm4scala repo" at "http://dl.bintray.com/angelcervera/maven" ``` -------------------------------- ### Shade Rule for Protobuf Dependency Conflict (SBT) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this SBT shade rule to rename the 'com.google.protobuf' package to 'shadeproto' to resolve conflicts with older versions used by Spark and Hadoop. ```scala assemblyShadeRules in assembly := Seq( ShadeRule .rename("com.google.protobuf.**" -> "shadeproto.@1") .inAll ) ``` -------------------------------- ### Count OSM Primitives (SQL) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Counts the number of different OSM primitives (nodes, ways, relations) using SQL. ```sql select type, count(type) from osm group by type ``` -------------------------------- ### Count Primitives by Type using SQL (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Query the 'osm' SQL view to count primitives grouped by their type. ```scala spark.sql("select type, count(*) as num_primitives from osm group by type").show() ``` -------------------------------- ### Add Osm4scala Core Dependency with sbt Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library Instructions for adding the osm4scala-core library to your sbt project. Replace '' with the desired library version. ```scala libraryDependencies += "com.acervera.osm4scala" %% "osm4scala-core" % "" ``` -------------------------------- ### Add Osm4scala Core Dependency with Maven Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library Instructions for adding the osm4scala-core library to your Maven project. Replace '${scala-version}' and '${version}' with your specific versions. ```xml com.acervera.osm4scala osm4scala-core_${scala-version} ${version} ``` -------------------------------- ### Parallel Primitive Counting with AKKA Source Source: https://simplexspatial.github.io/osm4scala/docs/performance Processes OSM PBF data in parallel using AKKA Sources. This approach offers better memory management, with usage directly related to the number of actors, and allows for distributed execution. ```text This example show how to process data in parallel, using AKKA The implementation is not complex at all, but it is necessary a little (a really little bit) of knowledge about AKKA to understand it. Two big advantage respect the Future.traverse version: * The memory used depends directly on the number of actor used, so you can process the full planet with no more of few GB of RAM. * It is possible distribute the execution in different nodes. ``` -------------------------------- ### Extract Traffic Lights as POIs (PySpark) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Selects and displays latitude, longitude, and tags for elements where the 'highway' tag is 'traffic_signals'. Shows the top 10 results. ```python osmDF.select("latitude", "longitude", "tags").where("element_at(tags, 'highway') == 'traffic_signals'").show(10,False) ``` -------------------------------- ### Extract Ways with Nodes and Tags Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this query to select the id, nodes, and tags from all ways (type = 1) in the OSM dataset. ```scala spark.sql("select id, nodes, tags from osm where type = 1").show() ``` -------------------------------- ### Add osm4scala Spark Dependency (SBT) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Include the osm4scala Spark connector as a dependency in your SBT project. ```scala libraryDependencies += "com.acervera.osm4scala" % "osm4scala-spark3-shaded_2.12" % "1.0.11" ``` -------------------------------- ### Create Table for OSM PBF with Split Option Disabled (Spark SQL) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Define a table using the osm.pbf format with the 'split' option set to 'false' for controlling parallelization. ```sql CREATE TABLE osm USING osm.pbf OPTIONS ( 'split' = 'false' ) LOCATION '' ``` -------------------------------- ### Extract All Tag Keys using SQL (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Query the 'osm' SQL view to extract and display all unique tag keys used in OSM data. ```scala spark.sql("select distinct(explode(map_keys(tags))) as tag_key from osm order by tag_key asc").show() ``` -------------------------------- ### Count Primitives (Single Thread) Source: https://simplexspatial.github.io/osm4scala/docs/performance Counts primitives in an OSM PBF file using a single thread. Memory usage is negligible as it processes data in blocks. ```text Found [67,976,861] primitives in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 32.44 sec. Found [4,839,505] primitives of type [Way] in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 31.72 sec. Found [63,006,432] primitives of type [Node] in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 32.70 sec. Found [130,924] primitives of type [Relation] in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 32.66 sec. ``` ```text Found [3,976,885,170] primitives in /media/angelcervera/My Passport/osm/planet-latest.osm.pbf in 2,566.11 sec. ``` -------------------------------- ### Load OSM PBF Data into DataFrame (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Reads OSM PBF data from the specified path into a Spark DataFrame. Ensure the path points to your PBF file or directory. ```scala val osmDF = spark.read.format("osm.pbf").load("") ``` -------------------------------- ### Extract Traffic Lights as POIs (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Selects and displays latitude, longitude, and tags for elements where the 'highway' tag is 'traffic_signals'. Shows the top 10 results. ```scala osmDF.select("latitude", "longitude", "tags").where("element_at(tags, 'highway') == 'traffic_signals'").show(10,false) ``` -------------------------------- ### Load OSM PBF Data into DataFrame (PySpark) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Reads OSM PBF data from the specified path into a Spark DataFrame. Ensure the path points to your PBF file or directory. ```python osmDF = spark.read.format("osm.pbf").load("") ``` -------------------------------- ### Extract Unique Tags (Single Thread) Source: https://simplexspatial.github.io/osm4scala/docs/performance Extracts a list of unique tags from an OSM PBF file using a single thread. Processing time is efficient for large datasets. ```text Found [4,166] different tags in /home/angelcervera/projects/osm/spain-latest.osm.pbf. List stored in /home/angelcervera/projects/osm/spain-latest.tags.txt. Time to process: 39.22 sec. Found [2,451] different tags in primitives of type [Way] in /home/angelcervera/projects/osm/spain-latest.osm.pbf. List stored in /home/angelcervera/projects/osm/spain-latest.tags.txt. Time to process: 33.47 sec. ``` -------------------------------- ### Extract Node Data using SQL (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Query the 'osm' SQL view to select id, latitude, longitude, and tags for all nodes (type = 0). ```scala spark.sql("select id, latitude, longitude, tags from osm where type = 0").show() ``` -------------------------------- ### Extract Relations with Members and Tags Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this query to select the id, relations, and tags from all relations (type = 2) in the OSM dataset. ```scala spark.sql("select id, relations, tags from osm where type = 2").show() ``` -------------------------------- ### Add osm4scala Spark Dependency (Maven) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Include the osm4scala Spark connector as a dependency in your Maven project. ```xml com.acervera.osm4scala osm4scala-spark3-shaded_2.12 1.0.11 ``` -------------------------------- ### Read OSM PBF with Split Option Disabled (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this option to disable file splitting and control parallelization per file when reading OSM PBF data in Scala. ```scala val osmDF = spark.read.format("osm.pbf").option("split", "false").load("") ``` -------------------------------- ### Parallel Primitive Counting with Scala Future.traverse Source: https://simplexspatial.github.io/osm4scala/docs/performance Processes OSM PBF data in parallel using Scala's Future.traverse. This method can consume significant memory as it creates a Future for each element, potentially loading entire blocks into memory. ```scala import java.util.concurrent.atomic.AtomicLong import scala.concurrent.Future import scala.concurrent.Await import scala.concurrent.duration.Duration import java.io.InputStream val counter = new AtomicLong() def count(pbfIS: InputStream): Long = { val result = Future.traverse(BlobTupleIterator.fromPbf(pbfIS))(tuple => Future { counter.addAndGet( count(tuple._2) ) }) Await.result(result, Duration.Inf) counter.longValue() } ``` -------------------------------- ### Query Traffic Signals in Spark Shell Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use SQL to filter and retrieve data for traffic signals from the osm table. This query selects latitude, longitude, and tags for rows where the type is 0 and the 'highway' tag is 'traffic_signals', limiting the results to the top 10. ```sql spark-sql> select latitude, longitude, tags from osm where type = 0 and element_at(tags, "highway") == 'traffic_signals' limit 10; 40.42125 -3.6844500000000004 {"crossing":"traffic_signals","crossing_ref":"zebra","highway":"traffic_signals"} 40.41779000000001 -3.6241199999999996 {"highway":"traffic_signals"} 40.41473000000003 -3.627109999999999 {"highway":"traffic_signals"} 40.414200000000015 -3.6282099999999993 {"highway":"traffic_signals"} 40.42635999999994 -3.727220000000005 {"crossing":"traffic_signals","highway":"traffic_signals"} 40.41937999999995 -3.688820000000004 {"highway":"traffic_signals"} 40.426489999999944 -3.687640000000004 {"highway":"traffic_signals","traffic_signals":"signal"} 40.421339999999944 -3.683020000000004 {"highway":"traffic_signals"} 40.41797999999994 -3.669340000000004 {"highway":"traffic_signals"} 40.418319999999945 -3.6762200000000043 {"highway":"traffic_signals","traffic_signals":"signal"} Time taken: 0.128 seconds, Fetched 10 row(s) ``` -------------------------------- ### Read OSM PBF with Split Option Disabled (PySpark) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Use this option to disable file splitting and control parallelization per file when reading OSM PBF data in PySpark. ```python osmDF = spark.read.format("osm.pbf").option("split", "false").load("") ``` -------------------------------- ### Count OSM Primitives (Scala) Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Counts the number of different OSM primitives (nodes, ways, relations) in the DataFrame. Type 0 represents nodes, 1 represents ways, and 2 represents relations. ```scala osmDF.groupBy("type").count().show() ``` -------------------------------- ### Extract Relation Metadata Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector Select specific metadata fields like version, userId, userName, and formatted timestamp from relations where userId is not null. The timestamp is formatted to 'dd-MMM-y kk:mm:ss z'. ```scala spark.sql("select id, type, info.version, info.userId, info.userName, date_format(info.timestamp, \"dd-MMM-y kk:mm:ss z\") as timestamp from osm where info.userId IS NOT NULL").show(5, false) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.