osm4scala

https://simplexspatial.github.io/osm4scala/
Admin
osm4scala is a Scala library and Spark Polyglot connector for OpenStreetMap Pbf files, offering easy...
Tokens:6,398
Snippets:84
Trust Score:9.7
Update:4 months ago
Show doc for...
Tokens:
Raw
### Add Spark Connector Dependency (Spark 3/Scala 2.12)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this command to start the Spark shell with the shaded connector dependency for Spark 3 and Scala 2.12. This resolves Protobuf version conflicts.

```bash
bin/spark-shell --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'
```

--------------------------------

### Add Spark Connector Dependency (PySpark)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this command to start PySpark with the shaded connector dependency for Spark 3 and Scala 2.12. This resolves Protobuf version conflicts.

```bash
bin/pyspark --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'
```

--------------------------------

### Add Spark Connector Dependency (Spark SQL)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this command to start Spark SQL with the shaded connector dependency for Spark 3 and Scala 2.12. This resolves Protobuf version conflicts.

```bash
bin/spark-sql --packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11'
```

--------------------------------

### Create DataFrame from OSM PBF (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Read OSM PBF data into a Spark DataFrame for analysis. This example counts primitives by type.

```scala
import com.acervera.osm4scala.spark.OsmSqlEntity
import org.apache.spark.sql.SparkSession
object PrimitivesCounter {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.appName("Primitives counter")
.getOrCreate()

spark.read
.format("osm.pbf")
.load(args(0))
.groupBy(OsmSqlEntity.FIELD_TYPE)
.count
.show()
}
}
```

--------------------------------

### Create DataFrame from OSM PBF (PySpark)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Read OSM PBF data into a Spark DataFrame for analysis using PySpark. This example counts primitives by type.

```python
from pyspark.sql import SparkSession
import sys

if __name__ == '__main__':
    spark = SparkSession.builder.appName("Primitives counter").getOrCreate()
    spark.read.format("osm.pbf") \
        .load(sys.argv[1]) \
        .groupBy("type") \
        .count() \
        .show()
```

--------------------------------

### Load OSM Data into DataFrame (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Load an OSM PBF file into a Spark DataFrame using the osm4scala connector in Scala. This example shows how to select and filter traffic signals.

```Scala
val osmDF = spark.read.format("osm.pbf").load("/home/jovyan/work/monaco-anonymized.osm.pbf")
osmDF.select("latitude", "longitude")
.where("element_at(tags, 'highway') == 'traffic_signals'")
.show
```

--------------------------------

### Load OSM Data into DataFrame (Python)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Load an OSM PBF file into a Spark DataFrame using the osm4scala connector in Python. Note the '%%python' header required when using a Scala kernel. This example filters for traffic signals.

```Python
%%python
osm_df = spark.read.format("osm.pbf").load("/home/jovyan/work/monaco-anonymized.osm.pbf")
osm_df.select("latitude", "longitude").where("element_at(tags, 'highway') == 'traffic_signals'").show()
```

--------------------------------

### Install osm4scala Package in Jupyter

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this command in a Jupyter Notebook cell to add the osm4scala package for Spark v3.1.1 and Scala 2.12. This is required before you can use the osm4scala connector.

```Scala
%%init_spark
launcher.packages = ["com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11"]
```

--------------------------------

### Run Tests for Scala 2.11

Source: https://simplexspatial.github.io/osm4scala/docs/contributing

Execute the clean and test commands for Scala version 2.11.

```bash
PATCH_211=false sbt clean +test
```

--------------------------------

### Run Tests for Scala 2.11 with Patch

Source: https://simplexspatial.github.io/osm4scala/docs/contributing

Execute the clean and test commands for Scala version 2.11, applying a patch.

```bash
PATCH_211=true sbt clean +test
```

--------------------------------

### Create SQL Temporal View (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Load OSM PBF data and create a temporary SQL view for querying.

```scala
val osmDF = spark.sqlContext.read.format("osm.pbf").load("<osm files path here>")
osmDF.createOrReplaceTempView("osm")
```

--------------------------------

### Submit Spark Application (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Submit a Scala Spark application to your cluster, including the osm4scala package.

```bash
bin/spark-submit \
--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' \
examples/spark-documentation/target/scala-2.12/osm4scala-examples-spark-documentation_2.12-1.0.11.jar \
/tmp/osm/monaco-anonymized.osm.pbf
```

--------------------------------

### Submit Spark Application (PySpark)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Submit a PySpark application to your cluster, including the osm4scala package.

```bash
bin/spark-submit \
--packages 'com.acervera.osm4scala:osm4scala-spark3-shaded_2.12:1.0.11' \
examples/spark-documentation/src/main/scala/com/acervera/osm4scala/examples/spark/documentation/PrimiriveCounter.py \
/tmp/osm/monaco-anonymized.osm.pbf
```

--------------------------------

### Create Table from OSM PBF Data (SQL)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Creates a Spark SQL table using the osm.pbf format, pointing to the location of your OSM PBF files.

```sql
CREATE TABLE osm USING osm.pbf LOCATION '<osm files path here>';
```

--------------------------------

### Count Node Primitives in a PBF File

Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library

Demonstrates counting all node primitives in a PBF file using EntityIterator. Requires importing the library.

```scala
import org.openstreetmap.osm4scala.core.EntityIterator
import org.openstreetmap.osm4scala.core.OSMTypes

EntityIterator.fromPbf(inputStream).count(_.osmModel == OSMTypes.Node)
```

--------------------------------

### Add Osm4scala Bintray Repository

Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library

Optional: Add this resolver if you encounter issues resolving dependencies. This points to the Bintray repository for osm4scala.

```scala
resolvers += "osm4scala repo" at "http://dl.bintray.com/angelcervera/maven"
```

--------------------------------

### Shade Rule for Protobuf Dependency Conflict (SBT)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this SBT shade rule to rename the 'com.google.protobuf' package to 'shadeproto' to resolve conflicts with older versions used by Spark and Hadoop.

```scala
assemblyShadeRules in assembly := Seq(
ShadeRule
.rename("com.google.protobuf.**" -> "shadeproto.@1")
.inAll
)
```

--------------------------------

### Count OSM Primitives (SQL)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Counts the number of different OSM primitives (nodes, ways, relations) using SQL.

```sql
select type, count(type) from osm group by type
```

--------------------------------

### Count Primitives by Type using SQL (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Query the 'osm' SQL view to count primitives grouped by their type.

```scala
spark.sql("select type, count(*) as num_primitives from osm group by type").show()
```

--------------------------------

### Add Osm4scala Core Dependency with sbt

Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library

Instructions for adding the osm4scala-core library to your sbt project. Replace '<version>' with the desired library version.

```scala
libraryDependencies += "com.acervera.osm4scala" %% "osm4scala-core" % "<version>"
```

--------------------------------

### Add Osm4scala Core Dependency with Maven

Source: https://simplexspatial.github.io/osm4scala/docs/standalone-scala-library

Instructions for adding the osm4scala-core library to your Maven project. Replace '${scala-version}' and '${version}' with your specific versions.

```xml
<dependency>
  <groupId>com.acervera.osm4scala</groupId>
  <artifactId>osm4scala-core_${scala-version}</artifactId>
  <version>${version}</version>
</dependency>
```

--------------------------------

### Parallel Primitive Counting with AKKA Source

Source: https://simplexspatial.github.io/osm4scala/docs/performance

Processes OSM PBF data in parallel using AKKA Sources. This approach offers better memory management, with usage directly related to the number of actors, and allows for distributed execution.

```text
This example show how to process data in parallel, using AKKA
The implementation is not complex at all, but it is necessary a little (a really little bit) of knowledge about AKKA to understand it. Two big advantage respect the Future.traverse version:
  * The memory used depends directly on the number of actor used, so you can process the full planet with no more of few GB of RAM.
  * It is possible distribute the execution in different nodes.
```

--------------------------------

### Extract Traffic Lights as POIs (PySpark)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Selects and displays latitude, longitude, and tags for elements where the 'highway' tag is 'traffic_signals'. Shows the top 10 results.

```python
osmDF.select("latitude", "longitude", "tags").where("element_at(tags, 'highway') == 'traffic_signals'").show(10,False)
```

--------------------------------

### Extract Ways with Nodes and Tags

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this query to select the id, nodes, and tags from all ways (type = 1) in the OSM dataset.

```scala
spark.sql("select id, nodes, tags from osm where type = 1").show()
```

--------------------------------

### Add osm4scala Spark Dependency (SBT)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Include the osm4scala Spark connector as a dependency in your SBT project.

```scala
libraryDependencies += "com.acervera.osm4scala" % "osm4scala-spark3-shaded_2.12" % "1.0.11"
```

--------------------------------

### Create Table for OSM PBF with Split Option Disabled (Spark SQL)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Define a table using the osm.pbf format with the 'split' option set to 'false' for controlling parallelization.

```sql
CREATE TABLE osm USING osm.pbf OPTIONS ( 'split' = 'false' ) LOCATION '<osm files path here>'
```

--------------------------------

### Extract All Tag Keys using SQL (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Query the 'osm' SQL view to extract and display all unique tag keys used in OSM data.

```scala
spark.sql("select distinct(explode(map_keys(tags))) as tag_key from osm order by tag_key asc").show()
```

--------------------------------

### Count Primitives (Single Thread)

Source: https://simplexspatial.github.io/osm4scala/docs/performance

Counts primitives in an OSM PBF file using a single thread. Memory usage is negligible as it processes data in blocks.

```text
Found [67,976,861] primitives in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 32.44 sec.
Found [4,839,505] primitives of type [Way] in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 31.72 sec.
Found [63,006,432] primitives of type [Node] in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 32.70 sec.
Found [130,924] primitives of type [Relation] in /home/angelcervera/projects/osm/spain-latest.osm.pbf in 32.66 sec.
```

```text
Found [3,976,885,170] primitives in /media/angelcervera/My Passport/osm/planet-latest.osm.pbf in 2,566.11 sec.
```

--------------------------------

### Load OSM PBF Data into DataFrame (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Reads OSM PBF data from the specified path into a Spark DataFrame. Ensure the path points to your PBF file or directory.

```scala
val osmDF = spark.read.format("osm.pbf").load("<osm files path here>")
```

--------------------------------

### Extract Traffic Lights as POIs (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Selects and displays latitude, longitude, and tags for elements where the 'highway' tag is 'traffic_signals'. Shows the top 10 results.

```scala
osmDF.select("latitude", "longitude", "tags").where("element_at(tags, 'highway') == 'traffic_signals'").show(10,false)
```

--------------------------------

### Load OSM PBF Data into DataFrame (PySpark)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Reads OSM PBF data from the specified path into a Spark DataFrame. Ensure the path points to your PBF file or directory.

```python
osmDF = spark.read.format("osm.pbf").load("<osm files path here>")
```

--------------------------------

### Extract Unique Tags (Single Thread)

Source: https://simplexspatial.github.io/osm4scala/docs/performance

Extracts a list of unique tags from an OSM PBF file using a single thread. Processing time is efficient for large datasets.

```text
Found [4,166] different tags in /home/angelcervera/projects/osm/spain-latest.osm.pbf. List stored in /home/angelcervera/projects/osm/spain-latest.tags.txt. Time to process: 39.22 sec.
Found [2,451] different tags in primitives of type [Way] in /home/angelcervera/projects/osm/spain-latest.osm.pbf. List stored in /home/angelcervera/projects/osm/spain-latest.tags.txt. Time to process: 33.47 sec.
```

--------------------------------

### Extract Node Data using SQL (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Query the 'osm' SQL view to select id, latitude, longitude, and tags for all nodes (type = 0).

```scala
spark.sql("select id, latitude, longitude, tags from osm where type = 0").show()
```

--------------------------------

### Extract Relations with Members and Tags

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this query to select the id, relations, and tags from all relations (type = 2) in the OSM dataset.

```scala
spark.sql("select id, relations, tags from osm where type = 2").show()
```

--------------------------------

### Add osm4scala Spark Dependency (Maven)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Include the osm4scala Spark connector as a dependency in your Maven project.

```xml
<dependency>
<groupId>com.acervera.osm4scala</groupId>
<artifactId>osm4scala-spark3-shaded_2.12</artifactId>
<version>1.0.11</version>
</dependency>
```

--------------------------------

### Read OSM PBF with Split Option Disabled (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this option to disable file splitting and control parallelization per file when reading OSM PBF data in Scala.

```scala
val osmDF = spark.read.format("osm.pbf").option("split", "false").load("<osm files path here>")
```

--------------------------------

### Parallel Primitive Counting with Scala Future.traverse

Source: https://simplexspatial.github.io/osm4scala/docs/performance

Processes OSM PBF data in parallel using Scala's Future.traverse. This method can consume significant memory as it creates a Future for each element, potentially loading entire blocks into memory.

```scala
import java.util.concurrent.atomic.AtomicLong
import scala.concurrent.Future
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import java.io.InputStream

val counter = new AtomicLong()
def count(pbfIS: InputStream): Long = {
  val result = Future.traverse(BlobTupleIterator.fromPbf(pbfIS))(tuple => Future {
    counter.addAndGet( count(tuple._2) )
  })
  Await.result(result, Duration.Inf)
  counter.longValue()
}
```

--------------------------------

### Query Traffic Signals in Spark Shell

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use SQL to filter and retrieve data for traffic signals from the osm table. This query selects latitude, longitude, and tags for rows where the type is 0 and the 'highway' tag is 'traffic_signals', limiting the results to the top 10.

```sql
spark-sql> select latitude, longitude, tags from osm where type = 0 and element_at(tags, "highway") == 'traffic_signals' limit 10;
40.42125 -3.6844500000000004 {"crossing":"traffic_signals","crossing_ref":"zebra","highway":"traffic_signals"}
40.41779000000001 -3.6241199999999996 {"highway":"traffic_signals"}
40.41473000000003 -3.627109999999999 {"highway":"traffic_signals"}
40.414200000000015 -3.6282099999999993 {"highway":"traffic_signals"}
40.42635999999994 -3.727220000000005 {"crossing":"traffic_signals","highway":"traffic_signals"}
40.41937999999995 -3.688820000000004 {"highway":"traffic_signals"}
40.426489999999944 -3.687640000000004 {"highway":"traffic_signals","traffic_signals":"signal"}
40.421339999999944 -3.683020000000004 {"highway":"traffic_signals"}
40.41797999999994 -3.669340000000004 {"highway":"traffic_signals"}
40.418319999999945 -3.6762200000000043 {"highway":"traffic_signals","traffic_signals":"signal"}
Time taken: 0.128 seconds, Fetched 10 row(s)
```

--------------------------------

### Read OSM PBF with Split Option Disabled (PySpark)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Use this option to disable file splitting and control parallelization per file when reading OSM PBF data in PySpark.

```python
osmDF = spark.read.format("osm.pbf").option("split", "false").load("<osm files path here>")
```

--------------------------------

### Count OSM Primitives (Scala)

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Counts the number of different OSM primitives (nodes, ways, relations) in the DataFrame. Type 0 represents nodes, 1 represents ways, and 2 represents relations.

```scala
osmDF.groupBy("type").count().show()
```

--------------------------------

### Extract Relation Metadata

Source: https://simplexspatial.github.io/osm4scala/docs/spark-connector

Select specific metadata fields like version, userId, userName, and formatted timestamp from relations where userId is not null. The timestamp is formatted to 'dd-MMM-y kk:mm:ss z'.

```scala
spark.sql("select id, type, info.version, info.userId, info.userName, date_format(info.timestamp, \"dd-MMM-y kk:mm:ss z\") as timestamp from osm where info.userId IS NOT NULL").show(5, false)
```
Tokens:
Raw