### Install pubmed_parser from GitHub

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/install.rst

Installs the pubmed_parser package directly from its GitHub repository using pip.

```bash
pip install git+git://github.com/titipata/pubmed_parser.git
```

--------------------------------

### Clone and Install pubmed_parser Locally

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/install.rst

Clones the pubmed_parser repository and installs it locally using pip.

```bash
git clone https://github.com/titipata/pubmed_parser
pip install ./pubmed_parser
```

--------------------------------

### Build Documentation with Sphinx

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/install.rst

Builds the project documentation using make and Sphinx. Requires sphinx and sphinx-gallery to be installed.

```bash
make html
```

--------------------------------

### Initialize Spark 2.1 with findspark

Source: https://github.com/titipata/pubmed_parser/wiki/Setup-Spark-2.1

This snippet demonstrates how to initialize Spark 2.1 using the `findspark` library in a Jupyter Notebook. It sets the `spark_home` environment variable, which is crucial for locating Spark installations.

```python
import os
import findspark
findspark.init(spark_home="/opt/spark-2.1.0-bin-cdh5.9.0/")
```

--------------------------------

### Test pubmed_parser Installation

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/install.rst

Runs tests for the pubmed_parser installation using pytest, including coverage reporting.

```bash
pytest --cov=pubmed_parser tests/ --verbose
```

--------------------------------

### Install pubmed-parser Package

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Provides instructions for installing the pubmed-parser package from GitHub or PyPI. Includes commands for cloning the repository and installing dependencies.

```bash
pip install git+https://github.com/titipata/pubmed_parser.git
```

```bash
pip install pubmed-parser
```

```bash
git clone https://github.com/titipata/pubmed_parser
pip install ./pubmed_parser
```

--------------------------------

### Configure and Create SparkSession

Source: https://github.com/titipata/pubmed_parser/wiki/Setup-Spark-2.1

This snippet shows how to configure a `SparkSession` with specific application settings, master URL, and resource allocation properties. It's used to create the main entry point for Spark functionality, enabling data processing operations.

```python
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf

conf = SparkConf().\
    setAppName('map').\
    setMaster('local[5]').\
    set('spark.yarn.appMasterEnv.PYSPARK_PYTHON', '~/anaconda3/bin/python').\
    set('spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON', '~/anaconda3/bin/python').\
    set('executor.memory', '8g').\
    set('spark.yarn.executor.memoryOverhead', '16g').\
    set('spark.sql.codegen', 'true').\
    set('spark.yarn.executor.memory', '16g').\
    set('yarn.scheduler.minimum-allocation-mb', '500m').\
    set('spark.dynamicAllocation.maxExecutors', '3').\
    set('spark.driver.maxResultSize', '0')

spark = SparkSession.builder.\
    appName("testing").\
    config(conf=conf).\
    getOrCreate()
```

--------------------------------

### Run PubMed OA Spark Script

Source: https://github.com/titipata/pubmed_parser/blob/master/scripts/README.md

Executes the `pubmed_oa_spark.py` script using `spark-submit` to download and preprocess the PubMed Open-Access dataset. This requires a Spark installation and the specified script path.

```bash
~/spark-2.1.0/bin/spark-submit pubmed_oa_spark.py
```

--------------------------------

### Initialize PySpark

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/spark.rst

Initializes the PySpark environment using `findspark`. This step is crucial for locating Spark installations within a Python environment, especially when running on platforms like Jupyter Notebooks. It requires the `spark_home` path to be correctly set.

```python
import os
import findspark
findspark.init(spark_home="/opt/spark-2.1.0-bin-cdh5.9.0/")
```

--------------------------------

### Run MEDLINE Spark Script

Source: https://github.com/titipata/pubmed_parser/blob/master/scripts/README.md

Executes the `medline_spark.py` script using `spark-submit` to download and preprocess the MEDLINE dataset. Similar to the Open-Access script, this requires a Spark installation and the correct script path.

```bash
~/spark-2.1.0/bin/spark-submit medline_spark.py
```

--------------------------------

### Sync MEDLINE Data via FTP Mount

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

An alternative method to download MEDLINE data by mounting the FTP directory using curlftpfs and then syncing with rsync. This approach requires FUSE and rsync to be installed.

```bash
curlftpfs ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/* .gz ftp_mount/
rsync -r -t  -v --progress ftp_mount/* baseline/
fusermount -u ftp_mount
```

--------------------------------

### Add pubmed_parser Egg to Spark

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Adds a built pubmed_parser Python egg file to the Spark context. This is necessary if the library is not installed in the environment where Spark jobs are run.

```python
spark.sparkContext.addPyFile('pubmed_parser/dist/pubmed_parser-0.1-py3.5.egg')
```

--------------------------------

### Schedule Spark Scripts with Cron

Source: https://github.com/titipata/pubmed_parser/blob/master/scripts/README.md

Configures a cron job to automatically run PySpark scripts for dataset processing. The example schedules the `pubmed_oa_spark.py` script to run every Sunday at 8 AM. This process is applicable to both PubMed OA and MEDLINE scripts.

```bash
#!/bin/bash

0 8 * * Sun source ~/.bash_profile;~/spark-2.1.0/bin/spark-submit pubmed_oa_spark.py
```

--------------------------------

### Build Documentation Page

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/README.md

Command to build the documentation page using Sphinx. The output is generated in the `_build/html/` directory.

```shell
make html
```

--------------------------------

### Configure SparkSession

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/spark.rst

Configures and creates a SparkSession with specific application settings. This includes setting the application name, master URL, Python executable paths, memory allocations, and other Spark configurations for optimal performance. The `SparkSession` is the entry point for Spark SQL functionality.

```python
from pyspark.sql import SparkSession
from pyspark.conf import SparkConf

conf = SparkConf().
    setAppName('map').
    setMaster('local[5]').
    set('spark.yarn.appMasterEnv.PYSPARK_PYTHON', '~/anaconda3/bin/python').
    set('spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON', '~/anaconda3/bin/python').
    set('executor.memory', '8g').
    set('spark.yarn.executor.memoryOverhead', '16g').
    set('spark.sql.codegen', 'true').
    set('spark.yarn.executor.memory', '16g').
    set('yarn.scheduler.minimum-allocation-mb', '500m').
    set('spark.dynamicAllocation.maxExecutors', '3').
    set('spark.driver.maxResultSize', '0')

spark = SparkSession.builder.
    appName("testing").
    config(conf=conf).
    getOrCreate()
```

--------------------------------

### Download Pubmed OA Subset

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-Pubmed-Open-Access-dataset

Downloads the compressed PubMed Open-Access bulk data files from the NCBI FTP server. This command uses `wget` to fetch all `.xml.tar.gz` files from the specified directory.

```bash
wget ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/*.xml.tar.gz
```

--------------------------------

### Download MEDLINE Update Files

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Downloads the MEDLINE update files from NCBI FTP servers using wget. These files contain incremental changes to the dataset.

```bash
wget ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/*.gz
```

--------------------------------

### Download MEDLINE Baseline Data

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Downloads the MEDLINE baseline dataset from NCBI FTP servers using wget. Ensure a 'data' folder exists before execution. The dataset contains approximately 29 million articles.

```bash
mkdir data
wget ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/*.gz
```

--------------------------------

### Pubmed Parser API Documentation

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Provides documentation for the core functions within the pubmed_parser library, detailing their purpose, parameters, and return values. This serves as an API reference for developers using the library.

```APIDOC
parse_pubmed_xml(path: str) -> dict
  Parses a PubMed Open Access XML file or string.
  Parameters:
    path: Path to the XML file or the XML content as a string.
  Returns:
    A dictionary containing article details like title, abstract, authors, affiliations, etc.

parse_pubmed_references(path: str) -> list[dict]
  Processes a PubMed Open Access XML file to extract citation references.
  Parameters:
    path: Path to the XML file.
  Returns:
    A list of dictionaries, each detailing a cited article (PMID, PMC, title, journal, cited PMIDs/DOIs).

parse_pubmed_caption(path: str) -> list[dict]
  Extracts image captions and associated metadata from PubMed Open Access XML files.
  Parameters:
    path: Path to the XML file.
  Returns:
    A list of dictionaries, each containing PMID, PMC ID, caption text, figure ID, figure label, and graphic reference.
```

--------------------------------

### Untar Pubmed OA Data

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-Pubmed-Open-Access-dataset

Extracts the contents of a downloaded PubMed Open-Access XML tarball into a specified directory. This command uses `tar` to decompress the archive and place its contents into the `data/` folder.

```bash
tar -xzf comm_use.A-B.xml.tar.gz --directory data/
```

--------------------------------

### Download PubMed OA Figure Tar Archives

Source: https://github.com/titipata/pubmed_parser/wiki/Download-PubMed-OA-figures

Once the `oa_file_list.csv` is downloaded, you can construct the direct download URL for tar archives containing figures. These archives are organized by accession number and can be accessed via FTP. The `File` column in the CSV provides the relative path within the `oa_package` directory.

```ftp
ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/08/e0/PMC13900.tar.gz
```

--------------------------------

### Parse PubMed OA Dataset XML

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Demonstrates parsing a PubMed Open Access XML file. It lists XML file paths from a directory and parses a single file into a dictionary containing article metadata.

```python
import pubmed_parser as pp

# List all xml paths under directory 'data'
path_xml = pp.list_xml_path('data') 

# Parse the first XML file found
pubmed_dict = pp.parse_pubmed_xml(path_xml[0]) 

# Print the parsed dictionary
print(pubmed_dict)
```

--------------------------------

### Parse MEDLINE Baseline Data with PySpark

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Parses downloaded MEDLINE .gz files using pubmed_parser and PySpark. It reads files, extracts publication data, converts it into a Spark DataFrame, and saves it as a parquet file.

```python
import os
from glob import glob
import pubmed_parser as pp
from pyspark.sql import SparkSession
from pyspark.sql import Row

medline_files_rdd = spark.sparkContext.parallelize(glob('data/*.gz'), numSlices=1000)

parse_results_rdd = medline_files_rdd.
    flatMap(lambda x: [Row(file_name=os.path.basename(x), **publication_dict) 
                       for publication_dict in pp.parse_medline_xml(x)])

medline_df = parse_results_rdd.toDF()
# save to parquet
medline_df.write.parquet('raw_medline.parquet', mode='overwrite')
```

--------------------------------

### Parse from Website Functions

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/api.rst

Includes functions for retrieving and parsing PubMed data directly from the web using eutils. These functions allow fetching XML data based on citations or outgoing citations.

```APIDOC
pubmed_parser.parse_xml_web()
  - Parses XML data fetched from the web.

pubmed_parser.parse_citation_web()
  - Retrieves and parses citation data from the web.

pubmed_parser.parse_outgoing_citation_web()
  - Fetches and parses outgoing citation information from the web.
```

--------------------------------

### Parse MEDLINE XML from E-Utilities

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Retrieves and parses MEDLINE XML data directly from the NCBI E-Utilities website using a PubMed ID. Returns a dictionary with article title, abstract, journal, authors, year, and keywords/MeSH terms.

```python
import pubmed_parser as pp

# Assuming 'pmid' is a valid PubMed ID
dict_out = pp.parse_xml_web(pmid, save_xml=False)
```

--------------------------------

### Parse Pubmed XML with PySpark

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-Pubmed-Open-Access-dataset

Processes PubMed XML files using PySpark to extract key metadata like title, abstract, and DOI. It then saves the results as a Parquet file. Dependencies include `pyspark` and `pubmed_parser`.

```python
import os
import pubmed_parser as pp
from pyspark.sql import Row

path_all = pp.list_xml_path('data/')
parse_results_rdd = pubmed_oa_rdd.map(lambda x: Row(file_name=os.path.basename(x), **pp.parse_pubmed_xml(x)))
pubmed_oa_df = parse_results_rdd.toDF()
pubmed_oa_df_sel = pubmed_oa_df[['full_title', 'abstract', 'doi', 
                                 'file_name', 'pmc', 'pmid', 
                                 'publication_year', 'publisher_id', 'journal', 'subjects']]
pubmed_oa_df_sel.write.parquet('pubmed_oa.parquet')
```

--------------------------------

### Download PubMed OA Figure List CSV

Source: https://github.com/titipata/pubmed_parser/wiki/Download-PubMed-OA-figures

To obtain image data for a given PubMed ID (PMID) or PMC accession number, download the `oa_file_list.csv` file from the NCBI FTP server. This CSV file contains essential columns like `PMID`, `Accession ID` (PMC), and `File` path information for Open Access figures.

```ftp
ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_file_list.csv
```

--------------------------------

### Parse PubMed OA Dataset with PySpark

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Shows how to parse all PubMed Open Access XML files in a directory using PySpark. It parallelizes file processing, converts results to a Spark DataFrame, selects relevant columns, and writes the output as a Parquet file.

```python
import os
import pubmed_parser as pp
from pyspark.sql import Row

# Assuming 'spark' is an existing SparkSession

# List all xml paths under a specified folder
path_all = pp.list_xml_path('/path/to/xml/folder/') 

# Parallelize the list of paths using Spark RDD
path_rdd = spark.sparkContext.parallelize(path_all, numSlices=10000)

# Map each path to a Row object containing parsed data
parse_results_rdd = path_rdd.map(lambda x: Row(file_name=os.path.basename(x), **pp.parse_pubmed_xml(x)))

# Convert RDD to Spark DataFrame
pubmed_oa_df = parse_results_rdd.toDF()

# Select specific columns for the final DataFrame
pubmed_oa_df_sel = pubmed_oa_df[['full_title', 'abstract', 'doi',
                                 'file_name', 'pmc', 'pmid',
                                 'publication_year', 'publisher_id',
                                 'journal', 'subjects']]

# Write the DataFrame to a Parquet file
pubmed_oa_df_sel.write.parquet('pubmed_oa.parquet', mode='overwrite')
```

--------------------------------

### Parse MEDLINE XML Citations from Website

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Fetches citation information for a given PubMed ID or PubMed Central ID from the web. Returns a dictionary containing PMC ID, PMID, DOI, the number of citations, and a list of PMCs that cite the article.

```python
import pubmed_parser as pp

# Assuming 'doc_id' is a PubMed or PMC ID
dict_out = pp.parse_citation_web(doc_id, id_type='PMC')
```

--------------------------------

### Parse MEDLINE Grant ID

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Extracts grant IDs from MEDLINE XML files. Returns a list of dictionaries containing PMID, grant ID, grant acronym, country, and agency for each grant found. Returns None if no grant IDs are present.

```python
import pubmed_parser as pp

# Assuming 'xml_file_path' is the path to a MEDLINE XML file
grant_data = pp.parse_grant_id(xml_file_path)
```

--------------------------------

### Parse PubMed OA Table

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Parses tables from PubMed Open Access XML files. Returns a list of dictionaries with PMID, PMC ID, caption, label, column names, table values, and optionally raw XML.

```python
import pubmed_parser as pp

dicts_out = pp.parse_pubmed_table('data/medline16n0902.xml.gz', return_xml=False)
```

--------------------------------

### Parse PubMed OA Images and Captions

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Extracts image captions and associated metadata from PubMed Open Access XML files. The function returns a list of dictionaries, each containing the PMID, PMC ID, caption text, figure ID, figure label, and a reference to the image file. The path to the XML file is required.

```python
import pubmed_parser as pp
dicts_out = pp.parse_pubmed_caption(path) # return list of dictionary
```

--------------------------------

### BibTeX Citation

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Provides the BibTeX entry for citing the pubmed_parser library in academic work, including DOIs, URLs, authors, and publication details.

```bibtex
@article{Achakulvisut2020,
  doi = {10.21105/joss.01979},
  url = {https://doi.org/10.21105/joss.01979},
  year = {2020},
  publisher = {The Open Journal},
  volume = {5},
  number = {46},
  pages = {1979},
  author = {Titipat Achakulvisut and Daniel Acuna and Konrad Kording},
  title = {Pubmed Parser: A Python Parser for PubMed Open-Access XML Subset and MEDLINE XML Dataset XML Dataset},
  journal = {Journal of Open Source Software}
}
```

--------------------------------

### Parse PubMed OA XML Functions

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/api.rst

Offers functions for parsing data from the PubMed Open Access (OA) XML subset. This includes parsing full articles, references, paragraphs, captions, and tables.

```APIDOC
pubmed_parser.parse_pubmed_xml()
  - Parses PubMed Open Access XML data.

pubmed_parser.parse_pubmed_references()
  - Extracts references from PubMed OA XML.

pubmed_parser.parse_pubmed_paragraph()
  - Parses paragraphs from PubMed OA XML.

pubmed_parser.parse_pubmed_caption()
  - Extracts figure captions from PubMed OA XML.

pubmed_parser.parse_pubmed_table()
  - Parses tables from PubMed OA XML.
```

--------------------------------

### Parse Pubmed Affiliation Data with PySpark

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-Pubmed-Open-Access-dataset

Extracts affiliation details (affiliation ID, affiliation text) from parsed PubMed records using PySpark. It creates a separate DataFrame for affiliation information and saves it as a Parquet file.

```python
def parse_affiliation(p):
    affiliation_list = p.affiliation_list
    affiliation_table = list()
    if len(affiliation_list) >= 1:
        for affil in affiliation_list:
            r = Row(pmc=p.pmc, pmid=p.pmid, 
                    affiliation_id=affil[0], affiliation=affil[1])
            affiliation_table.append(r)
        return affiliation_table
    else:
        return None

parse_affil_rdd = parse_results_rdd.map(lambda x: parse_affiliation(x)).\
    filter(lambda x: x is not None).\
    flatMap(lambda xs: [x for x in xs])

parse_affil_df = parse_affil_rdd.toDF()
parse_name_df.write.parquet('pubmed_oa_affiliation.parquet')
```

--------------------------------

### Parse PubMed OA Paragraph

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Parses text and reference PMIDs from PubMed Open Access XML files. Returns a list of dictionaries, each containing PMID, PMC ID, text content, reference IDs, and section information.

```python
import pubmed_parser as pp

dicts_out = pp.parse_pubmed_paragraph('data/6605965a.nxml', all_paragraph=False)
```

--------------------------------

### Parse MEDLINE XML with Pubmed Parser

Source: https://github.com/titipata/pubmed_parser/blob/master/paper/paper.md

This snippet demonstrates how to use the `pubmed_parser` library to parse a MEDLINE XML file. It shows the function call `pp.parse_medline_xml` with options for year information, NLM category, and author list. The output is a list of parsed articles, typically in Python dictionaries.

```Python
import pubmed_parser as pp
parsed_articles = pp.parse_medline_xml('data/pubmed20n0014.xml.gz',
                                       year_info_only=True,
                                       nlm_category=False,
                                       author_list=False)
```

--------------------------------

### Parse Pubmed Author Data with PySpark

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-Pubmed-Open-Access-dataset

Extracts author details (last name, first name, affiliation ID) from parsed PubMed records using PySpark. It creates a separate DataFrame for author information and saves it as a Parquet file.

```python
def parse_name(p):
    author_list = p.author_list
    author_table = list()
    if len(author_list) >= 1:
        for author in author_list:
            r = Row(pmc=p.pmc, pmid=p.pmid, last_name=author[0], 
                    first_name=author[1], affiliation_id=author[2])
            author_table.append(r)
        return author_table
    else:
        return None

parse_name_rdd = parse_results_rdd.map(lambda x: parse_name(x)).\
    filter(lambda x: x is not None).\
    flatMap(lambda xs: [x for x in xs])

parse_name_df = parse_name_rdd.toDF()
parse_name_df.write.parquet('pubmed_oa_author.parquet')
```

--------------------------------

### Process MEDLINE Updates and Deletes

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Processes the raw MEDLINE data to identify the latest version of each article, considering updates and deletions. It uses Spark window functions to rank records by file name and filters for the most recent, non-deleted entries.

```python
from pyspark.sql import Window
from pyspark.sql.functions import rank, max, sum, desc

medline_df = spark.read.parquet('raw_medline.parquet')

window = Window.partitionBy(['pmid']).orderBy(desc('file_name'))

windowed_df = medline_df.select(
    max('delete').over(window).alias('is_deleted'),
    rank().over(window).alias('pos'),
    '*')

windowed_df.where('is_deleted = False and pos = 1').
    write.
    parquet('medline_lastview.parquet', mode='overwrite')
```

--------------------------------

### Parse MEDLINE Grant ID Table

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Parses the Grant ID information from MEDLINE files using pubmed_parser and PySpark. It extracts grant-related data, creates a DataFrame, and saves it to a parquet file.

```python
grant_df = medline_files_rdd.
    flatMap(lambda x: [Row(file_name=os.path.basename(x), **grant_dict) 
                       for grant_dict in pp.parse_medline_grant_id(x)]).
    toDF()
grant_df.write.parquet('grants_df.parquet')
```

--------------------------------

### Parse MEDLINE XML Functions

Source: https://github.com/titipata/pubmed_parser/blob/master/docs/api.rst

Provides functions for parsing data from MEDLINE XML files. These functions are designed to extract specific information from the MEDLINE XML format.

```APIDOC
pubmed_parser.parse_medline_xml()
  - Parses data from MEDLINE XML files.

pubmed_parser.parse_grant_id()
  - Extracts grant IDs from MEDLINE data.
```

--------------------------------

### Parse PubMed Captions with pubmed_parser

Source: https://github.com/titipata/pubmed_parser/wiki/Download-PubMed-OA-figures

The `pubmed_parser` library offers the `parse_pubmed_caption` function to parse figures, specifically `figure_id`, and their corresponding captions from a manuscript. This function is key for associating textual descriptions with visual elements in PubMed articles.

```python
from pubmed_parser import parse_pubmed_caption

# Example usage (assuming manuscript data is available)
# figure_data = parse_pubmed_caption(manuscript_data)
```

--------------------------------

### Parse PubMed OA XML Information

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Parses a PubMed Open Access XML file or string to extract key article details. It returns a dictionary containing the title, abstract, journal, PMIDs, DOI, author list, affiliations, publication year, and subjects. Requires the path to the XML file.

```python
import pubmed_parser as pp
dict_out = pp.parse_pubmed_xml(path)
```

--------------------------------

### Parse PubMed OA Citation References

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Processes a PubMed Open Access XML file to extract citation references. It returns a list of dictionaries, where each dictionary contains details of a cited article, including its PMCID, title, journal, and the PMIDs/DOIs it cites. The path to the XML file is required.

```python
import pubmed_parser as pp
dicts_out = pp.parse_pubmed_references(path) # return list of dictionary
```

--------------------------------

### Parse Outgoing Citations from Web

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Retrieves articles cited by a given PubMed ID or PubMed Central ID. Returns a dictionary with citation counts, document ID, ID type, and a list of cited PMIDs. Identifiers should be passed as strings.

```python
import pubmed_parser as pp

doc_id = "21810267" # Example PubMed ID
id_type = "PMID" # or "PMC"
dict_out = pp.parse_outgoing_citation_web(doc_id, id_type=id_type)
print(dict_out)
```

--------------------------------

### Parse MEDLINE XML

Source: https://github.com/titipata/pubmed_parser/blob/master/README.md

Parses MEDLINE XML files, which have a different structure than PubMed OA. Extracts detailed article information including PMID, PMC, DOI, title, abstract, authors, MeSH terms, publication types, and more.

```python
import pubmed_parser as pp

dicts_out = pp.parse_medline_xml('data/medline16n0902.xml.gz',
                                 year_info_only=False,
                                 nlm_category=False,
                                 author_list=False,
                                 reference_list=False)
```

--------------------------------

### Compute TF-IDF Features with PySpark ML

Source: https://github.com/titipata/pubmed_parser/wiki/Download-and-preprocess-MEDLINE-dataset

Computes Term Frequency-Inverse Document Frequency (TF-IDF) features for the MEDLINE content. It concatenates relevant fields (author, affiliation, title, abstract), tokenizes, removes stopwords, hashes terms, and applies the IDF model.

```python
from pyspark.sql import functions
from pyspark.ml import Pipeline
from pyspark.ml.feature import HashingTF, Tokenizer, IDF, StopWordsRemover, RegexTokenizer

final_results_df = spark.read.parquet('medline_lastview.parquet')
# we will create a column content that concatenas author, affiliation, title, and abstract
content_df = final_results_df.select(final_results_df.pmid, 
                        functions.trim(functions.concat_ws(' ', 
                            final_results_df.author, 
                            final_results_df.affiliation, 
                            final_results_df.title,
                            final_results_df.abstract)).alias('content'))

tokenizer = Tokenizer(inputCol='content', outputCol='words')
stopwordsremover = StopWordsRemover(inputCol=tokenizer.getOutputCol(), outputCol='words_wo_stop')
hashingTF = HashingTF(inputCol=stopwordsremover.getOutputCol(), outputCol='features', numFeatures=2**18)
idf = IDF(minDocFreq=3, inputCol=hashingTF.getOutputCol(), outputCol='tfidf')

tfidf_pipeline = Pipeline(stages=[tokenizer, stopwordsremover, hashingTF, idf])
tfidf_model = tfidf_pipeline.fit(content_df)
tdidf_df = tfidf_model.transform(content_df)

tdidf_df = tfidf_model.transform(content_df).select('pmid', 'tfidf')

tfidf_model.save('tfidf.model')

tdidf_df.write.parquet('tfidf.parquet')
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.