metascraper

https://metascraper.js.org
Admin
metascraper is a library for extracting unified metadata from websites using Open Graph, Microdata,...
Tokens:4,274
Snippets:22
Trust Score:8.1
Update:1 month ago
Show doc for...
Tokens:
Raw
### metascraper with omitPropNames

Source: https://metascraper.js.org/

Creates a metascraper instance with title, image, and description rules, then extracts metadata from an HTML string and URL. The omitPropNames set removes the 'image' property from the result, which is logged as { title: 'Example', image: null, description: null }.

```javascript
const metascraper = require('metascraper')([
  require('metascraper-title')(),
  require('metascraper-image')(),
  require('metascraper-description')()
])

const html = '<title>Example</title><meta property="og:image" content="image.jpg">'
const url = 'https://example.com'

// Omit the image property
const omitPropNames = new Set(['image'])
const metadata = await metascraper({ url, html, omitPropNames })

console.log(metadata)
// Output: { title: 'Example', image: null, description: null }
```

--------------------------------

### Initialize metascraper with rule bundles

Source: https://metascraper.js.org/

Initializes metascraper with a collection of rule bundles. The order of rules matters: the first rule that resolves a value is applied, and later rules act as fallbacks.

```javascript
const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])
```

--------------------------------

### constructor(rules)

Source: https://metascraper.js.org/

Creates a new metascraper instance with the specified rules bundle.

```APIDOC
## constructor(rules)

### Description
Creates a new metascraper instance with the specified rules bundle.

### Method
constructor

### Parameters
#### rules
- **rules** (Array) - Required - The collection of rules bundle to be loaded.

### Request Example
```js
const metascraper = require('metascraper')([
  require('metascraper-title')(),
  require('metascraper-image')(),
  require('metascraper-description')()
])
```

### Response
#### Success Response
Returns a metascraper instance that can be called with options.

```

--------------------------------

### metascraper(options)

Source: https://metascraper.js.org/

Call the instance for extracting content based on rules bundle provided at the constructor.

```APIDOC
## metascraper(options)

### Description
Call the instance for extracting content based on rules bundle provided at the constructor.

### Method
metascraper

### Parameters
#### options
- **html** (String) - Optional - The HTML markup for extracting the content.
- **htmlDom** (object) - Optional - The DOM representation of the HTML markup. When it's not provided, it's get from the `html` parameter.
- **omitPropNames** (Set) - Optional - Default: `[]` - A set of property names that should be omitted. When specified, these properties will be missing in the returned metadata objects, and rules related to that will not be computed.
- **pickPropNames** (Set) - Optional - Default: `undefined` - A set of property names to pick for the metadata extraction process. When specified, only rules for these properties will be executed, and all other properties will be omitted. Takes precedence over `omitPropNames` when both are specified.
- **rules** (Array) - Optional - You can pass additional rules to add on execution time. These rules will be merged with your loaded rules at the beginning.
- **url** (String) - Required - The URL associated with the HTML markup. It is used for resolve relative links that can be present in the HTML markup. It can be used as fallback field for different rules as well.
- **validateUrl** (boolean) - Optional - Default: `true` - Ensure the URL provided is validated as a WHATWG URL API compliant.

### Request Example
```js
const metascraper = require('metascraper')([
  require('metascraper-title')(),
  require('metascraper-image')(),
  require('metascraper-description')()
])

const html = '<title>Example</title><meta property="og:image" content="image.jpg">'
const url = 'https://example.com'

// Omit the image property
const omitPropNames = new Set(['image'])
const metadata = await metascraper({ url, html, omitPropNames })

console.log(metadata)
// Output: { title: 'Example', image: null, description: null }
```

### Response
#### Success Response
Returns a metadata object with extracted properties.

```

--------------------------------

### Extract metadata with metascraper and html-get

Source: https://metascraper.js.org/

Use this to extract metadata from a live URL. It requires the target URL and its HTML markup, which can be retrieved with html-get and a headless browser. The output shows the metadata object returned by metascraper.

```javascript
const getHTML = require('html-get')

/**
 * `browserless` will be passed to `html-get`
 * as driver for getting the rendered HTML.
 */
const browserless = require('browserless')()

const getContent = async url => {
  // create a browser context inside the main Chromium process
  const browserContext = browserless.createContext()
  const promise = getHTML(url, { getBrowserless: () => browserContext })
  // close browser resources before return the result
  promise.then(() => browserContext).then(browser => browser.destroyContext())
  return promise
}

/**
 * `metascraper` is a collection of tiny packages,
 * so you can just use what you actually need.
 */
const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])

/**
 * The main logic
 */
getContent('https://microlink.io')
  .then(metascraper)
  .then(metadata => console.log(metadata))
  .then(browserless.close)
  .then(process.exit)
```

```json
{
  "author": "Microlink HQ",
  "date": "2022-07-10T22:53:04.856Z",
  "description": "Enter a URL, receive information. Normalize metadata. Get HTML markup. Take a screenshot. Identify tech stack. Generate a PDF. Automate web scraping. Run Lighthouse",
  "image": "https://cdn.microlink.io/logo/banner.jpeg",
  "logo": "https://cdn.microlink.io/logo/logo.png",
  "publisher": "Microlink",
  "title": "Turns websites into data — Microlink",
  "url": "https://microlink.io/"
}
```

--------------------------------

### Pass custom options to a rule bundle

Source: https://metascraper.js.org/

Passes custom options to a specific rule bundle. Here, the metascraper-logo rule is configured with a filter that only accepts URLs ending in '.png'.

```javascript
const metascraper = require('metascraper')([
  require('metascraper-logo')({
    filter: url => url.endsWith('.png')
  })
])
```
Tokens:
Raw