PySpark

https://spark.apache.org/docs/latest/api/python
Admin
PySpark is the Python API for Apache Spark that enables real-time, large-scale distributed data...
Tokens:4,976,219
Snippets:35,925
Trust Score:10
Update:1 week ago
Show doc for...
Tokens:
Raw
### Setup for Index.identical examples

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Index.identical.html

Imports option_context and creates an Index and a MultiIndex for the examples.

```python
>>> from pyspark.pandas.config import option_context
>>> idx = ps.Index(['a', 'b', 'c'])
>>> midx = ps.MultiIndex.from_tuples([('a', 'x'), ('b', 'y'), ('c', 'z')])
```

--------------------------------

### Setup for equals examples

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.CategoricalIndex.equals.html

Initializes an Index and a MultiIndex with names for the equality examples.

```python
>>> from pyspark.pandas.config import option_context
>>> idx = ps.Index(['a', 'b', 'c'])
>>> idx.name = "name"
>>> midx = ps.MultiIndex.from_tuples([('a', 'x'), ('b', 'y'), ('c', 'z')])
>>> midx.names = ("nameA", "nameB")
```

--------------------------------

### DataFrame.xs example with MultiIndex setup

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/frame.html

Creates a DataFrame with a MultiIndex from columns 'class', 'animal', and 'locomotion', then displays it. This setup is used for subsequent xs examples.

```python
>>> d = {'num_legs': [4, 4, 2, 2],
...      'num_wings': [0, 0, 2, 2],
...      'class': ['mammal', 'mammal', 'mammal', 'bird'],
...      'animal': ['cat', 'dog', 'bat', 'penguin'],
...      'locomotion': ['walks', 'walks', 'flies', 'walks']}
>>> df = ps.DataFrame(data=d)
>>> df = df.set_index(['class', 'animal', 'locomotion'])
>>> df  # doctest: +NORMALIZE_WHITESPACE
                           num_legs  num_wings
class  animal  locomotion
mammal cat     walks              4          0
       dog     walks              4          0
       bat     flies              2          2
bird   penguin walks              2          2
```

--------------------------------

### Join examples setup with Row objects

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Sets up three DataFrames (df, df2, df3) used in the subsequent join examples. Imports pyspark.sql.functions and Row.

```python
>>> import pyspark.sql.functions as sf
>>> from pyspark.sql import Row
>>> df = spark.createDataFrame([Row(name="Alice", age=2), Row(name="Bob", age=5)])
>>> df2 = spark.createDataFrame([Row(name="Tom", height=80), Row(name="Bob", height=85)])
>>> df3 = spark.createDataFrame([
...     Row(name="Alice", age=10, height=80),
...     Row(name="Bob", age=5, height=None),
...     Row(name="Tom", age=None, height=None),
...     Row(name=None, age=None, height=None),
... ])
```

--------------------------------

### startTime property example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/core/context.html

Get the epoch time when the SparkContext was started.

```python
>>> _ = sc.startTime
```

--------------------------------

### pyspark.pandas.DataFrame.loc

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.loc.html

The .loc accessor is used for label-based indexing. The examples demonstrate getting values and slicing rows with integer labels, where both start and stop are inclusive.

```APIDOC
## pyspark.pandas.DataFrame.loc

### Description
Label-based indexing and slicing for a pandas-on-Spark DataFrame. Use integer labels to select rows and columns. Slices are inclusive of both start and stop labels.

### Usage
python
df.loc[row_indexer, column_indexer]


### Parameters
- **row_indexer** (slice, list, or scalar) - Required - Row labels to select. Can be a slice, list of labels, or a single label.
- **column_indexer** (slice, list, or scalar) - Optional - Column labels to select. Defaults to all columns if omitted.

### Examples

#### Getting values with integer labels
python
>>> df = ps.DataFrame([[1, 2], [4, 5], [7, 8]],
...                   index=[7, 8, 9],
...                   columns=['max_speed', 'shield'])
>>> df
   max_speed  shield
7          1       2
8          4       5
9          7       8


#### Slicing with integer labels (inclusive)
python
>>> df.loc[7:9]
   max_speed  shield
7          1       2
8          4       5
9          7       8


### Notes
- Both the start and stop labels are included in the slice.
- The index must have integer labels for the examples shown.
```

--------------------------------

### Create a DataFrame with columns x, y, z

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.get.html

Initializes a pandas-on-Spark DataFrame with three columns and an index containing duplicate values. This is the setup for the subsequent get examples.

```python
>>> df = ps.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']},
...                   columns=['x', 'y', 'z'], index=[10, 20, 20])
>>> df
    x  y  z
10  0  a  a
20  1  b  b
20  2  b  b
```

--------------------------------

### show_options() example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/config.html

Shows how to call show_options() to display available options. The comment instructs to copy the output into the documentation.

```python
>>> from pyspark.pandas.config import show_options
>>> show_options()
```

--------------------------------

### Create a DataFrame with columns x, y, z

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.get.html

Creates a PySpark pandas DataFrame with columns 'x', 'y', 'z' and index [10, 20, 20]. This is the setup for the subsequent get examples.

```python
>>> df = ps.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']},
...                   columns=['x', 'y', 'z'], index=[10, 20, 20])
>>> df
    x  y  z
10  0  a  a
20  1  b  b
20  2  b  b

```

--------------------------------

### Setup sample data for customers and orders

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Creates sample DataFrames for customers and orders used in the EXISTS subquery examples. The data includes customer IDs, names, countries, and order details.

```python
>>> data_customers = [
...     (101, "Alice", "USA"), (102, "Bob", "Canada"), (103, "Charlie", "USA"),
...     (104, "David", "Australia")
... ]
>>> data_orders = [
...     (1, 101, "2023-01-15", 250), (2, 102, "2023-01-20", 300),
...     (3, 103, "2023-01-25", 400), (4, 101, "2023-02-05", 150)
... ]
>>> customers = spark.createDataFrame(
...     data_customers, ["customer_id", "customer_name", "country"])
>>> orders = spark.createDataFrame(
...     data_orders, ["order_id", "customer_id", "order_date", "total_amount"])
```

--------------------------------

### Get recent progress updates with recentProgress property

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/streaming/query.html

Creates a rate stream, starts a memory query, and retrieves the most recent progress updates. The example shows the expected output as a list and then stops the query.

```python
>>> sdf = spark.readStream.format("rate").load()
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()

Get an array of the most recent query progress updates for this query

>>> sq.recentProgress
[...]

>>> sq.stop()
```

--------------------------------

### getTable examples: create table, get table, drop table

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/catalog.html

Shows how to create a table, retrieve it with getTable, and drop it. The output shows the Table object with name, catalog, and namespace.

```python
>>> _ = spark.sql("DROP TABLE IF EXISTS tbl1")
>>> _ = spark.sql("CREATE TABLE tbl1 (name STRING, age INT) USING parquet")
>>> spark.catalog.getTable("tbl1")
Table(name='tbl1', catalog='spark_catalog', namespace=['default'], ...
```

```python
>>> _ = spark.sql("DROP TABLE tbl1")
```

--------------------------------

### Get current status of streaming query with status property

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/streaming/query.html

Creates a rate stream, starts a memory query, and retrieves the current status. The example shows the expected output format with placeholders and then stops the query.

```python
>>> sdf = spark.readStream.format("rate").load()
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()

Get the current status of the query

>>> sq.status
{'message': '...', 'isDataAvailable': ..., 'isTriggerActive': ...}

>>> sq.stop()
```

--------------------------------

### Explain plans for a streaming query

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/streaming/query.html

Shows how to create a streaming query, print the schema, and explain the physical and logical plans. The example includes the schema output and the plan output placeholders.

```python
>>> sdf = spark.readStream.format("rate").load()
>>> sdf.printSchema()
root
  |-- timestamp: timestamp (nullable = true)
  |-- value: long (nullable = true)

>>> sq = sdf.writeStream.format('memory').queryName('query_explain').start()
>>> sq.processAllAvailable() # Wait a bit to generate the runtime plans.

Explain the runtime plans

>>> sq.explain()
== Physical Plan ==
...
>>> sq.explain(True)
== Parsed Logical Plan ==
...
== Analyzed Logical Plan ==
...
== Optimized Logical Plan ==
...
== Physical Plan ==
...
>>> sq.stop()
```

--------------------------------

### lead window function examples

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions/builtin.html

Shows how to use sf.lead to get the value from a subsequent row in a window partition. The examples create a DataFrame, define a window, and apply lead with different offsets and defaults, demonstrating the resulting output.

```python
>>> from pyspark.sql import functions as sf
>>> from pyspark.sql import Window
>>> df = spark.createDataFrame(
...     [("a", 1), ("a", 2), ("a", 3), ("b", 8), ("b", 2)], ["c1", "c2"])
>>> df.show()
+---+---+
| c1| c2|
+---+---+
|  a|  1|
|  a|  2|
|  a|  3|
|  b|  8|
|  b|  2|
+---+---+
```

```python
>>> w = Window.partitionBy("c1").orderBy("c2")
>>> df.withColumn("next_value", sf.lead("c2").over(w)).show()
+---+---+----------+
| c1| c2|next_value|
+---+---+----------+
|  a|  1|         2|
|  a|  2|         3|
|  a|  3|      NULL|
|  b|  2|         8|
|  b|  8|      NULL|
+---+---+----------+
```

```python
>>> df.withColumn("next_value", sf.lead("c2", 1, 0).over(w)).show()
+---+---+----------+
| c1| c2|next_value|
+---+---+----------+
|  a|  1|         2|
|  a|  2|         3|
|  a|  3|         0|
|  b|  2|         8|
|  b|  8|         0|
+---+---+----------+
```

```python
>>> df.withColumn("next_value", sf.lead("c2", 2, -1).over(w)).show()
+---+---+----------+
| c1| c2|next_value|
+---+---+----------+
|  a|  1|         3|
|  a|  2|        -1|
|  a|  3|        -1|
|  b|  2|        -1|
|  b|  8|        -1|
+---+---+----------+
```

--------------------------------

### date_range examples: specifying start and periods

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/namespace.html

Example of using ps.date_range with start and periods parameters, generating a daily frequency DatetimeIndex.

```python
>>> ps.date_range(start='1/1/2018', periods=8)  # doctest: +SKIP
DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04',
               '2018-01-05', '2018-01-06', '2018-01-07', '2018-01-08'],
              dtype='datetime64[ns]', freq=None)
```

--------------------------------

### Setup: Create customers and orders DataFrames

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Initializes sample customers and orders DataFrames used by the lateralJoin examples. Requires a SparkSession named 'spark'.

```python
>>> from pyspark.sql import functions as sf
>>> from pyspark.sql import Row
>>> customers_data = [
...     Row(customer_id=1, name="Alice"), Row(customer_id=2, name="Bob"),
...     Row(customer_id=3, name="Charlie"), Row(customer_id=4, name="Diana")
... ]
>>> customers = spark.createDataFrame(customers_data)
>>> orders_data = [
...     Row(order_id=101, customer_id=1, order_date="2024-01-10",
...         items=[Row(product="laptop", quantity=5), Row(product="mouse", quantity=12)]),
...     Row(order_id=102, customer_id=1, order_date="2024-02-15",
...         items=[Row(product="phone", quantity=2), Row(product="charger", quantity=15)]),
...     Row(order_id=105, customer_id=1, order_date="2024-03-20",
...         items=[Row(product="tablet", quantity=4)]),
...     Row(order_id=103, customer_id=2, order_date="2024-01-12",
...         items=[Row(product="tablet", quantity=8)]),
...     Row(order_id=104, customer_id=2, order_date="2024-03-05",
...         items=[Row(product="laptop", quantity=7)]),
...     Row(order_id=106, customer_id=3, order_date="2024-04-05",
...         items=[Row(product="monitor", quantity=1)]),
... ]
>>> orders = spark.createDataFrame(orders_data)
```

--------------------------------

### date_range examples: specifying start and end

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/namespace.html

Example of using ps.date_range with start and end parameters, generating a daily frequency DatetimeIndex.

```python
>>> ps.date_range(start='1/1/2018', end='1/08/2018')  # doctest: +SKIP
DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04',
               '2018-01-05', '2018-01-06', '2018-01-07', '2018-01-08'],
              dtype='datetime64[ns]', freq=None)
```

--------------------------------

### substr without length (defaults to 2147483647)

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.substr.html

Extracts substring from a given position to the end of the string when length is omitted. The example uses sf.substr(df.a, df.b) to get substring starting at position 5, resulting in 'k SQL'.

```python
>>> df.select("*", sf.substr(df.a, df.b)).show()
+---------+---+---+------------------------+
|        a|  b|  c|substr(a, b, 2147483647)|
+---------+---+---+------------------------+
|Spark SQL|  5|  1|                   k SQL|
+---------+---+---+------------------------+

```

--------------------------------

### Create sample DataFrame for partitionBy example

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.Window.partitionBy.html

Creates a DataFrame with columns 'id' and 'category' and displays it. This is the initial setup for demonstrating the partitionBy method.

```python
>>> from pyspark.sql import Window, functions as sf
>>> df = spark.createDataFrame(
...      [(1, "a"), (1, "a"), (2, "a"), (1, "b"), (2, "b"), (3, "b")], ["id", "category"])
>>> df.show()
+---+--------+
| id|category|
+---+--------+
|  1|       a|
|  1|       a|
|  2|       a|
|  1|       b|
|  2|       b|
|  3|       b|
+---+--------+
```

--------------------------------

### Create a DataFrame for __getitem__ examples

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.__getitem__.html

Creates a sample DataFrame with columns 'age' and 'name' used in the subsequent examples.

```python
>>> df = spark.createDataFrame([
...     (2, "Alice"), (5, "Bob")], schema=["age", "name"])

```

--------------------------------

### date_range examples: specifying start, end, and periods

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/namespace.html

Example of using ps.date_range with start, end, and periods parameters, generating a linearly spaced DatetimeIndex.

```python
>>> ps.date_range(start='1/1/2018', end='1/08/2018', periods=8)  # doctest: +SKIP
DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04',
               '2018-01-05', '2018-01-06', '2018-01-07', '2018-01-08'],
              dtype='datetime64[ns]', freq=None)
```

--------------------------------

### Binarizer usage example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/feature.html

Shows how to create a Binarizer, set parameters, transform data, and save/load the model. Includes both single-column and multi-column usage.

```python
>>> df = spark.createDataFrame([(0.5,)], ["values"])
>>> binarizer = Binarizer(threshold=1.0, inputCol="values", outputCol="features")
>>> binarizer.setThreshold(1.0)
Binarizer...
>>> binarizer.setInputCol("values")
Binarizer...
>>> binarizer.setOutputCol("features")
Binarizer...
>>> binarizer.transform(df).head().features
0.0
>>> binarizer.setParams(outputCol="freqs").transform(df).head().freqs
0.0
>>> params = {binarizer.threshold: -0.5, binarizer.outputCol: "vector"}
>>> binarizer.transform(df, params).head().vector
1.0
>>> binarizerPath = temp_path + "/binarizer"
>>> binarizer.save(binarizerPath)
>>> loadedBinarizer = Binarizer.load(binarizerPath)
>>> loadedBinarizer.getThreshold() == binarizer.getThreshold()
True
>>> loadedBinarizer.transform(df).take(1) == binarizer.transform(df).take(1)
True
```

```python
>>> df2 = spark.createDataFrame([(0.5, 0.3)], ["values1", "values2"])
>>> binarizer2 = Binarizer(thresholds=[0.0, 1.0])
>>> binarizer2.setInputCols(["values1", "values2"]).setOutputCols(["output1", "output2"])
Binarizer...
>>> binarizer2.transform(df2).show()
+-------+-------+-------+-------+
|values1|values2|output1|output2|
+-------+-------+-------+-------+
|    0.5|    0.3|    1.0|    0.0|
+-------+-------+-------+-------+
...
```

--------------------------------

### overlay example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions/builtin.html

Overlays a portion of a string with a replacement string starting at a byte position. The example replaces part of 'SPARK_SQL' with 'CORE' starting at position 7.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("SPARK_SQL", "CORE")], ("x", "y"))
>>> df.select("*", sf.overlay("x", df.y, 7)).show()
+---------+----+--------------------+
|        x|   y|overlay(x, y, 7, -1)|
+---------+----+--------------------+
|SPARK_SQL|CORE|          SPARK_CORE|
+---------+----+--------------------+
```

--------------------------------

### BlockMatrix add example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/mllib/linalg/distributed.html

Creates three BlockMatrix instances and demonstrates adding them, showing the resulting DenseMatrix output.

```python
>>> dm1 = Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])
>>> dm2 = Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12])
>>> sm = Matrices.sparse(3, 2, [0, 1, 3], [0, 1, 2], [7, 11, 12])
>>> blocks1 = sc.parallelize([((0, 0), dm1), ((1, 0), dm2)])
>>> blocks2 = sc.parallelize([((0, 0), dm1), ((1, 0), dm2)])
>>> blocks3 = sc.parallelize([((0, 0), sm), ((1, 0), dm2)])
>>> mat1 = BlockMatrix(blocks1, 3, 2)
>>> mat2 = BlockMatrix(blocks2, 3, 2)
>>> mat3 = BlockMatrix(blocks3, 3, 2)

>>> mat1.add(mat2).toLocalMatrix()
DenseMatrix(6, 2, [2.0, 4.0, 6.0, 14.0, 16.0, 18.0, 8.0, 10.0, 12.0, 20.0, 22.0, 24.0], 0)

>>> mat1.add(mat3).toLocalMatrix()
DenseMatrix(6, 2, [8.0, 2.0, 3.0, 14.0, 16.0, 18.0, 4.0, 16.0, 18.0, 20.0, 22.0, 24.0], 0)
```

--------------------------------

### Create DataFrame and show

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/group.html

Shows how to create a DataFrame with age and name columns and display it. This is the setup for the count example.

```python
>>> df = spark.createDataFrame(
...      [(2, "Alice"), (3, "Alice"), (5, "Bob"), (10, "Bob")], ["age", "name"])
>>> df.show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  3|Alice|
|  5|  Bob|
| 10|  Bob|
+---+-----+
```

--------------------------------

### scalar() example: setup sample DataFrame

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Creates a sample DataFrame with employee data including id, name, salary, and department_id, used for scalar subquery examples.

```python
>>> data = [
...     (1, "Alice", 45000, 101), (2, "Bob", 54000, 101), (3, "Charlie", 29000, 102),
...     (4, "David", 61000, 102), (5, "Eve", 48000, 101),
... ]
>>> employees = spark.createDataFrame(data, ["id", "name", "salary", "department_id"])

```

--------------------------------

### Tokenizer example: basic usage and parameter changes

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/feature.html

Shows how to create a Tokenizer, set input/output columns, transform a DataFrame, and change parameters using setParams or transform with a params dict. Includes a TypeError when using positional arguments.

```python
>>> df = spark.createDataFrame([("a b c",)], ["text"])
>>> tokenizer = Tokenizer(outputCol="words")
>>> tokenizer.setInputCol("text")
Tokenizer...
>>> tokenizer.transform(df).head()
Row(text='a b c', words=['a', 'b', 'c'])
>>> # Change a parameter.
>>> tokenizer.setParams(outputCol="tokens").transform(df).head()
Row(text='a b c', tokens=['a', 'b', 'c'])
>>> # Temporarily modify a parameter.
>>> tokenizer.transform(df, {tokenizer.outputCol: "words"}).head()
Row(text='a b c', words=['a', 'b', 'c'])
>>> tokenizer.transform(df).head()
Row(text='a b c', tokens=['a', 'b', 'c'])
>>> # Must use keyword arguments to specify params.
>>> tokenizer.setParams("text")
Traceback (most recent call last):
    ...
TypeError: Method setParams forces keyword arguments.
```

```python
>>> tokenizerPath = temp_path + "/tokenizer"
>>> tokenizer.save(tokenizerPath)
>>> loadedTokenizer = Tokenizer.load(tokenizerPath)
>>> loadedTokenizer.transform(df).head().tokens == tokenizer.transform(df).head().tokens
True
```

--------------------------------

### _ipython_key_completions_ example: get column names

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Shows how to get column names for IPython key completions. The first example returns valid identifiers, while the second shows that illegal identifiers are also returned as-is.

```python
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], ["age", "name"])
>>> df._ipython_key_completions_()
['age', 'name']
```

```python
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], ["age 1", "name?1"])
>>> df._ipython_key_completions_()
['age 1', 'name?1']
```

--------------------------------

### count_if example: counting strings starting with 'a'

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions/builtin.html

Counts rows where the fruit column starts with 'a' using sf.count_if.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame(
...   [("apple",), ("banana",), ("cherry",), ("apple",), ("banana",)], ["fruit"])
>>> df.select(sf.count_if(sf.col('fruit').startswith('a'))).show()
+------------------------------+
|count_if(startswith(fruit, a))|
+------------------------------+
|                             2|
+------------------------------+
```

--------------------------------

### HashingTF example

Source: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.HashingTF.html

Demonstrates creating a HashingTF, setting parameters, transforming data, saving/loading, and using indexOf.

```python
>>> df = spark.createDataFrame([(["a", "b", "c"],)], ["words"])
>>> hashingTF = HashingTF(inputCol="words", outputCol="features")
>>> hashingTF.setNumFeatures(10)
HashingTF...
>>> hashingTF.transform(df).head().features
SparseVector(10, {5: 1.0, 7: 1.0, 8: 1.0})
>>> hashingTF.setParams(outputCol="freqs").transform(df).head().freqs
SparseVector(10, {5: 1.0, 7: 1.0, 8: 1.0})
>>> params = {hashingTF.numFeatures: 5, hashingTF.outputCol: "vector"}
>>> hashingTF.transform(df, params).head().vector
SparseVector(5, {0: 1.0, 2: 1.0, 3: 1.0})
>>> hashingTFPath = temp_path + "/hashing-tf"
>>> hashingTF.save(hashingTFPath)
>>> loadedHashingTF = HashingTF.load(hashingTFPath)
>>> loadedHashingTF.getNumFeatures() == hashingTF.getNumFeatures()
True
>>> loadedHashingTF.transform(df).take(1) == hashingTF.transform(df).take(1)
True
>>> hashingTF.indexOf("b")
5
```

--------------------------------

### cube examples with output

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Three examples showing different ways to use cube: by name, by name and age, and by column ordinals. Each example includes the resulting DataFrame output.

```python
>>> df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], schema=["name", "age"])
```

```python
>>> df.cube("name").count().orderBy("name").show()
+-----+-----+
| name|count|
+-----+-----+
| NULL|    2|
|Alice|    1|
|  Bob|    1|
+-----+-----+
```

```python
>>> df.cube("name", df.age).count().orderBy("name", "age").show()
+-----+----+-----+
| name| age|count|
+-----+----+-----+
| NULL|NULL|    2|
|Alice|NULL|    1|
|Alice|   2|    1|
|  Bob|NULL|    1|
|  Bob|   5|    1|
+-----+----+-----+
```

```python
>>> df.cube(1, 2).count().orderBy(1, 2).show()
+-----+----+-----+
| name| age|count|
+-----+----+-----+
| NULL|NULL|    2|
|Alice|NULL|    1|
|Alice|   2|    1|
|  Bob|NULL|    1|
|  Bob|   5|    1|
+-----+----+-----+
```

--------------------------------

### Getting the SRID of GEOMETRY

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.st_srid.html

Example 2: Getting the SRID of GEOMETRY. Uses st_geomfromwkb to convert WKB to GEOMETRY and then st_srid to get the SRID. The output is 0.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(bytes.fromhex('0101000000000000000000F03F0000000000000040'),)], ['wkb']) # noqa
>>> df.select(sf.st_srid(sf.st_geomfromwkb('wkb'))).collect()
[Row(st_srid(st_geomfromwkb(wkb, 0))=0)]
```

--------------------------------

### take examples on Series, Index, and MultiIndex

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/base.html

Examples showing how to select elements by positional indices using the take method on Series, Index, and MultiIndex objects.

```python
>>> psser = ps.Series([100, 200, 300, 400, 500])
>>> psser
0    100
1    200
2    300
3    400
4    500
dtype: int64

>>> psser.take([0, 2, 4]).sort_index()
0    100
2    300
4    500
dtype: int64
```

```python
>>> psidx = ps.Index([100, 200, 300, 400, 500])
>>> psidx
Index([100, 200, 300, 400, 500], dtype='int64')

>>> psidx.take([0, 2, 4]).sort_values()
Index([100, 300, 500], dtype='int64')
```

```python
>>> psmidx = ps.MultiIndex.from_tuples([("x", "a"), ("x", "b"), ("x", "c")])
>>> psmidx  # doctest: +SKIP
MultiIndex([('x', 'a'),
                    ('x', 'b'),
                    ('x', 'c')],
                   )

>>> psmidx.take([0, 2])  # doctest: +SKIP
MultiIndex([('x', 'a'),
                    ('x', 'c')],
                   )
```

--------------------------------

### Getting the SRID of GEOGRAPHY

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.st_srid.html

Example 1: Getting the SRID of GEOGRAPHY. Uses st_geogfromwkb to convert WKB to GEOGRAPHY and then st_srid to get the SRID. The output is 4326.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(bytes.fromhex('0101000000000000000000F03F0000000000000040'),)], ['wkb']) # noqa
>>> df.select(sf.st_srid(sf.st_geogfromwkb('wkb'))).collect()
[Row(st_srid(st_geogfromwkb(wkb))=4326)]
```

--------------------------------

### sum() resample example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/resample.html

Example for the sum() method of the resampler. Uses the same setup as the min() and max() examples and computes the sum for each 3-day bin, then sorts the index.

```python
>>> import numpy as np
>>> from datetime import datetime
>>> np.random.seed(22)
>>> dates = [
...    datetime(2022, 5, 1, 4, 5, 6),
...    datetime(2022, 5, 3),
...    datetime(2022, 5, 3, 23, 59, 59),
...    datetime(2022, 5, 4),
...    pd.NaT,
...    datetime(2022, 5, 4, 0, 0, 1),
...    datetime(2022, 5, 11),
... ]
>>> df = ps.DataFrame(
...    np.random.rand(len(dates), 2), index=pd.DatetimeIndex(dates), columns=["A", "B"]
... )
>>> df
                                    A         B
2022-05-01 04:05:06  0.208461  0.481681
2022-05-03 00:00:00  0.420538  0.859182
2022-05-03 23:59:59  0.171162  0.338864
2022-05-04 00:00:00  0.270533  0.691041
NaT                  0.220405  0.811951
2022-05-04 00:00:01  0.010527  0.561204
2022-05-11 00:00:00  0.813726  0.745100
>>> df.resample("3D").sum().sort_index()
                           A         B
2022-05-01  0.800161  1.679727
2022-05-04  0.501465  2.064196
2022-05-07       NaN       NaN
2022-05-10  0.813726  0.745100
```

--------------------------------

### RuntimeConfig.get examples

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/conf.html

Shows various uses of the get method: returning a default for a non-existent key, retrieving an explicitly set value, getting a built-in default, and overriding the built-in default with a provided default.

```python
>>> spark.conf.get("non-existent-key", "my_default")
'my_default'
```

```python
>>> spark.conf.set("my_key", "my_value")
>>> spark.conf.get("my_key")
'my_value'
```

```python
>>> spark.conf.unset("spark.sql.sources.partitionOverwriteMode")
>>> spark.conf.get("spark.sql.sources.partitionOverwriteMode")
'STATIC'
```

```python
>>> spark.conf.get("spark.sql.sources.partitionOverwriteMode", "DYNAMIC")
'DYNAMIC'
```

--------------------------------

### try_to_binary examples

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions/builtin.html

Three examples showing how to use sf.try_to_binary: converting a string with encoding specified, converting without encoding, and handling conversion failure with ANSI mode enabled. The third example demonstrates that a malformed input returns NULL.

```python
>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([("abc",)], ["e"])
>>> df.select(sf.try_to_binary(df.e, sf.lit("utf-8")).alias('r')).collect()
[Row(r=b'abc')]
```

```python
>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([("414243",)], ["e"])
>>> df.select(sf.try_to_binary(df.e).alias('r')).collect()
[Row(r=b'ABC')]
```

```python
>>> import pyspark.sql.functions as sf
>>> origin = spark.conf.get("spark.sql.ansi.enabled")
>>> spark.conf.set("spark.sql.ansi.enabled", "true")
>>> try:
...     df = spark.range(1)
...     df.select(sf.try_to_binary(sf.lit("malformed"), sf.lit("hex"))).show()
... finally:
...     spark.conf.set("spark.sql.ansi.enabled", origin)
+-----------------------------+
|try_to_binary(malformed, hex)|
+-----------------------------+
|                         NULL|
+-----------------------------+
```

--------------------------------

### max() resample example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/pandas/resample.html

Example for the max() method of the resampler. Uses the same setup as the min() example and computes the maximum value for each 3-day bin, then sorts the index.

```python
>>> import numpy as np
>>> from datetime import datetime
>>> np.random.seed(22)
>>> dates = [
...    datetime(2022, 5, 1, 4, 5, 6),
...    datetime(2022, 5, 3),
...    datetime(2022, 5, 3, 23, 59, 59),
...    datetime(2022, 5, 4),
...    pd.NaT,
...    datetime(2022, 5, 4, 0, 0, 1),
...    datetime(2022, 5, 11),
... ]
>>> df = ps.DataFrame(
...    np.random.rand(len(dates), 2), index=pd.DatetimeIndex(dates), columns=["A", "B"]
... )
>>> df
                                    A         B
2022-05-01 04:05:06  0.208461  0.481681
2022-05-03 00:00:00  0.420538  0.859182
2022-05-03 23:59:59  0.171162  0.338864
2022-05-04 00:00:00  0.270533  0.691041
NaT                  0.220405  0.811951
2022-05-04 00:00:01  0.010527  0.561204
2022-05-11 00:00:00  0.813726  0.745100
>>> df.resample("3D").max().sort_index()
                           A         B
2022-05-01  0.420538  0.859182
2022-05-04  0.270533  0.691041
2022-05-07       NaN       NaN
2022-05-10  0.813726  0.745100
```

--------------------------------

### Install PySpark 4.0.0.dev2

Source: https://spark.apache.org/docs/latest/api/python/_sources/user_guide/dataprep.ipynb.txt

Installs the PySpark package version 4.0.0.dev2 using pip. This is a prerequisite for the examples on this page.

```bash
!pip install pyspark==4.0.0.dev2
```

--------------------------------

### Create DataFrame for show examples

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/dataframe.html

Creates a DataFrame with age and name columns used in the subsequent show method examples.

```python
>>> df = spark.createDataFrame([
...     (14, "Tom"), (23, "Alice"), (16, "Bob"), (19, "This is a super long name")],
...     ["age", "name"])
```

--------------------------------

### Example: Using spark.conf.set, get, unset

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/conf.html

Shows how to set, get, and unset a Spark SQL configuration property. The get after unset raises a SparkNoSuchElementException because the key is not found.

```python
>>> spark.conf.set("my_key", "my_value")
>>> spark.conf.get("my_key")
'my_value'
>>> spark.conf.unset("my_key")
>>> spark.conf.get("my_key")
Traceback (most recent call last):
   ...
pyspark...SparkNoSuchElementException: ... The SQL config "my_key" cannot be found...
```

--------------------------------

### Create SparkConf and set master and app name

Source: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkConf.html

Shows how to create a SparkConf object, set the master and app name, and verify the values. Also demonstrates creating a SparkContext with the configuration.

```python
>>> from pyspark import SparkConf, SparkContext
>>> conf = SparkConf()
>>> conf.setMaster("local").setAppName("My app")
<pyspark.conf.SparkConf object at ...>
>>> conf.get("spark.master")
'local'
>>> conf.get("spark.app.name")
'My app'
>>> sc = SparkContext(conf=conf)
>>> sc.master
'local'
>>> sc.appName
'My app'
>>> sc.sparkHome is None
True
```

--------------------------------

### substr with explicit length

Source: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.substr.html

Extracts a substring of specified length from a string column. The example creates a DataFrame with columns 'a', 'b', 'c' and uses sf.substr("a", "b", "c") to get a substring starting at position 5 of length 1, resulting in 'k'.

```python
>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([("Spark SQL", 5, 1,)], ["a", "b", "c"])
>>> df.select("*", sf.substr("a", "b", "c")).show()
+---------+---+---+---------------+
|        a|  b|  c|substr(a, b, c)|
+---------+---+---+---------------+
|Spark SQL|  5|  1|              k|
+---------+---+---+---------------+

```

--------------------------------

### Create BlockMatrix instances for add examples

Source: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.mllib.linalg.distributed.BlockMatrix.html

Sets up dense and sparse matrices and parallelizes them into block RDDs to create three BlockMatrix instances used in the add examples.

```python
>>> dm1 = Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])
>>> dm2 = Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12])
>>> sm = Matrices.sparse(3, 2, [0, 1, 3], [0, 1, 2], [7, 11, 12])
>>> blocks1 = sc.parallelize([((0, 0), dm1), ((1, 0), dm2)])
>>> blocks2 = sc.parallelize([((0, 0), dm1), ((1, 0), dm2)])
>>> blocks3 = sc.parallelize([((0, 0), sm), ((1, 0), dm2)])
>>> mat1 = BlockMatrix(blocks1, 3, 2)
>>> mat2 = BlockMatrix(blocks2, 3, 2)
>>> mat3 = BlockMatrix(blocks3, 3, 2)
```

--------------------------------

### schema_of_variant_agg example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions/builtin.html

Shows how to use schema_of_variant_agg to get the merged schema of a variant column. The example returns 'OBJECT<a: BIGINT>'.

```python
>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
>>> df.select(schema_of_variant_agg(parse_json(df.json)).alias("r")).collect()
[Row(r='OBJECT<a: BIGINT>')]
```

--------------------------------

### schema_of_variant example

Source: https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions/builtin.html

Shows how to use schema_of_variant to get the SQL schema of a variant column. The example returns 'OBJECT<a: BIGINT>'.

```python
>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }'''} ])
>>> df.select(schema_of_variant(parse_json(df.json)).alias("r")).collect()
[Row(r='OBJECT<a: BIGINT>')]
```

--------------------------------

### RegexTokenizer usage example

Source: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.RegexTokenizer.html

Shows how to create a RegexTokenizer, set input and output columns, transform a DataFrame, change parameters, and save/load the model. The example also demonstrates that setParams requires keyword arguments.

```python
>>> df = spark.createDataFrame([("A B  c",)], ["text"])
>>> reTokenizer = RegexTokenizer()
>>> reTokenizer.setInputCol("text")
RegexTokenizer...
>>> reTokenizer.setOutputCol("words")
RegexTokenizer...
>>> reTokenizer.transform(df).head()
Row(text='A B  c', words=['a', 'b', 'c'])
>>> # Change a parameter.
>>> reTokenizer.setParams(outputCol="tokens").transform(df).head()
Row(text='A B  c', tokens=['a', 'b', 'c'])
>>> # Temporarily modify a parameter.
>>> reTokenizer.transform(df, {reTokenizer.outputCol: "words"}).head()
Row(text='A B  c', words=['a', 'b', 'c'])
>>> reTokenizer.transform(df).head()
Row(text='A B  c', tokens=['a', 'b', 'c'])
>>> # Must use keyword arguments to specify params.
>>> reTokenizer.setParams("text")
Traceback (most recent call last):
    ...
TypeError: Method setParams forces keyword arguments.
>>> regexTokenizerPath = temp_path + "/regex-tokenizer"
>>> reTokenizer.save(regexTokenizerPath)
>>> loadedReTokenizer = RegexTokenizer.load(regexTokenizerPath)
>>> loadedReTokenizer.getMinTokenLength() == reTokenizer.getMinTokenLength()
True
>>> loadedReTokenizer.getGaps() == reTokenizer.getGaps()
True
>>> loadedReTokenizer.transform(df).take(1) == reTokenizer.transform(df).take(1)
True
```
Tokens:
Raw