### Setup Integration

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Installs a single integration if it hasn't been installed already, calling its `setupOnce` method.

```typescript
function setupIntegration(e) {
  -1 === exports.installedIntegrations.indexOf(e.name) && (e.setupOnce(hub_1.addGlobalEventProcessor, hub_1.getCurrentHub), exports.installedIntegrations.push(e.name), utils_1.logger.log("Integration installed: " + e.name))
}
```

--------------------------------

### Setup Integrations

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Sets up a list of integrations, ensuring each is installed only once and calling their respective `setupOnce` methods.

```typescript
function setupIntegrations(e) {
  var t = {};
  return getIntegrationsToSetup(e).forEach(function(e) {
    t[e.name] = e,
    setupIntegration(e)
  }), t
}
```

--------------------------------

### Build and Install Stunnel

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/netbits.us.stunnel_rsync.html

Standard make commands to compile and install stunnel after configuration. Ensure you are logged in as root for the installation.

```bash
make
make install
```

--------------------------------

### Get Integrations to Setup

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Determines which integrations should be set up based on default integrations and user-provided integrations. Prioritizes 'Debug' integration if present.

```typescript
function getIntegrationsToSetup(e) {
  var t = e.defaultIntegrations && tslib_1.__spread(e.defaultIntegrations) || [],
    n = e.integrations,
    r = [];
  if (Array.isArray(n)) {
    var s = n.map(function(e) { return e.name }),
      a = [];
    t.forEach(function(e) {
      -1 === s.indexOf(e.name) && -1 === a.indexOf(e.name) && (r.push(e), a.push(e.name))
    }), n.forEach(function(e) {
      -1 === a.indexOf(e.name) && (r.push(e), a.push(e.name))
    })
  } else "function" == typeof n ? (r = n(t), r = Array.isArray(r) ? r : [r]) : r = tslib_1.__spread(t);
  var i = r.map(function(e) { return e.name });
  return -1 !== i.indexOf("Debug") && r.push.apply(r, tslib_1.__spread(r.splice(i.indexOf("Debug"), 1))),
    r
}
```

--------------------------------

### Build and Install Rsync

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/netbits.us.stunnel_rsync.html

Commands to compile, install, and clean up the rsync build process.

```bash
make   su   make install   make distclean
```

--------------------------------

### Install Development Packages

Source: https://github.com/adbar/trafilatura/blob/master/CONTRIBUTING.md

Install the necessary packages for development, including testing and linting tools.

```bash
pip install trafilatura[dev]
```

--------------------------------

### Install SoMaJo for tokenization

Source: https://github.com/adbar/trafilatura/blob/master/docs/tutorial1.md

Install the SoMaJo library for tokenization using pip.

```bash
pip install -U SoMaJo
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/adbar/trafilatura/blob/master/tests/cache/pluralsight.com.python.html

Run `poetry install` to install all specified dependencies into a virtual environment. It ensures validated dependencies are recorded in `poetry.lock`.

```bash
poetry install
```

--------------------------------

### Build and Install OpenSSL

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/netbits.us.stunnel_rsync.html

Compiles the OpenSSL source code using 'make' and installs it to the configured prefix using 'make install' as root. 'make clean' removes intermediate objects.

```bash
make
```

```bash
su
```

```bash
make install
```

```bash
make clean
```

--------------------------------

### Install Python Requirements

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/hackernoon.com.scrape.html

Save this to a requirements.txt file and use pip to install the necessary libraries for web scraping.

```text
requests
bs4
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/adbar/trafilatura/blob/master/docs/tutorial-epsilla.md

Install the pyepsilla client for interacting with the Epsilla database, and langchain and sentence_transformers for embedding models.

```bash
pip install -U pyepsilla
pip install -U langchain sentence_transformers
```

--------------------------------

### Install Optional Functionality

Source: https://github.com/adbar/trafilatura/blob/master/docs/installation.md

Install additional libraries for extended functionality and faster processing, such as language detection or faster encoding detection. Use '[all]' to install all optional features.

```bash
$ pip install faust-cchardet  # single package only
```

```bash
$ pip install trafilatura[all]  # all additional functionality
```

--------------------------------

### Setup for Time Formatting

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Initializes formatting for time-related units.

```javascript
function et(e,t){I(e,0,0,function(){return
```

--------------------------------

### Install Trafilatura with GUI support

Source: https://github.com/adbar/trafilatura/blob/master/docs/usage-gui.md

Install Trafilatura version 1.8.1 with the GUI components. Use this command if you need the graphical interface.

```bash
pip3 install -U trafilatura==1.8.1[gui]
```

--------------------------------

### Slow Docker Build Example with setup.py

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/pythonspeed.com.docker.html

This Dockerfile setup leads to slow builds because any code change invalidates the cache for dependency installation. It installs dependencies and code in a single step.

```python
from setuptools import setup

setup(name='exampleapp',
      packages=["exampleapp"],
      install_requires=["flask", "dateutil"])

```

```Dockerfile
FROM python:3.7
COPY . /tmp/myapp
RUN pip install /tmp/myapp
CMD flask run exampleapp:app

```

--------------------------------

### Get Current Timestamp and Set Session Start

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/agrarheute.com.abfallholz.html

Provides functions to get the current high-resolution timestamp and to set the session start time. Used for timing and session tracking.

```javascript
var r = n(2364), o = (new Date).getTime(), i = o;

function a() {
  return r.G && performance.now ? Math.round(performance.now()) :
    (o = Math.max((new Date).getTime(), o)) - i
}

function c() {
  return o
}

function u(t) {
  i = t
}

function s() {
  return i
}
```

--------------------------------

### Initialize and Setup Logging

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Initializes the logging system with optional site and browser versions. Sets up logging with provided metadata.

```javascript
var baseGet=require(896);function get(e,t,r){var a=null==e?void 0:baseGet(e,t);return void 0===a?r:a}module.exports=get;
```

```javascript
(function (process){ "use strict";const clayLog=require(34),_defaults=require(1045);let sitesLogInstance,navigatorReference;function init(e,s){var n={};"undefined"!=typeof window&&void 0===window.process?sitesLogInstance=null:(e&&(n.sitesVersion=e),s&&(n.browserVersion=navigatorReference.userAgent),clayLog.init({name:"sites",meta:n}),sitesLogInstance=clayLog.getLogger())}function setup(e){return e=_defaults({},e,{file:"File not specified! Please declare a file"}),sitesLogInstance?clayLog.meta(e,sitesLogInstance):console.log}process.versions&&process.versions.node||(navigatorReference=navigator,init(null,!0)),module.exports.init=init,module.exports.setup=setup,module.exports.assignNavigator=function(e){navigatorReference=e},module.exports.assignLogInstance=function(e){sitesLogInstance=e}; }).call(this,require(11))
```

--------------------------------

### Install and Load Trafilatura in R

Source: https://github.com/adbar/trafilatura/blob/master/docs/usage-r.md

Install the 'reticulate' package and load the Trafilatura module for use in R. This is the initial setup required before using Trafilatura functions.

```R
# getting started
> install.packages("reticulate")
> library(reticulate)

# loading the Trafilatura module
> trafilatura <- import("trafilatura")
```

--------------------------------

### Moment.js Initialization and Utilities

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Demonstrates the basic setup and utility functions for the Moment.js library. It includes helper functions for checking data types and creating moment objects.

```javascript
var e,t;function n(){return e.apply(null,arguments)}
function s(e){return e instanceof Array||"["object Array]".===Object.prototype.toString.call(e)}
function i(e){return null!=e&&"["object Object]".===Object.prototype.toString.call(e)}
function r(e){return void 0===e}
function a(e){return"number"==typeof e||"["object Number]".===Object.prototype.toString.call(e)}
function o(e){return e instanceof Date||"["object Date]".===Object.prototype.toString.call(e)}
function u(e,t){var n,s=[];for(n=0;n<e.length;++n)s.push(t(e[n],n));return s}
function l(e,t){return Object.prototype.hasOwnProperty.call(e,t)}
function h(e,t){for(var n in t)l(t,n)&&(e[n]=t[n]);return l(t,"toString")&&(e.toString=t.toString),l(t,"valueOf")&&(e.valueOf=t.valueOf),e}
function d(e,t,n,s){return bt(e,t,n,s,!0).utc()}
function c(e){return null==e._pf&&(e._pf={empty:!1,unusedTokens:[],unusedInput:[],overflow:-2,charsLeftOver:0,nullInput:!1,invalidMonth:null,invalidFormat:!1,userInvalidated:!1,iso:!1,parsedDateParts:[],meridiem:null,rfc2822:!1,weekdayMismatch:!1}),e._pf}
function f(e){if(null==e._isValid){var n=c(e),s=t.call(n.parsedDateParts,function(e){return null!=e}),i=!isNaN(e._d.getTime())&&n.overflow<0&&!n.empty&&!n.invalidMonth&&!n.invalidWeekday&&!n.weekdayMismatch&&!n.nullInput&&!n.invalidFormat&&!n.userInvalidated&&(!n.meridiem||n.meridiem&&s);if(e._strict&&(i=i&&0===n.charsLeftOver&&0===n.unusedTokens.length&&void 0===n.bigHour),null!=Object.isFrozen&&Object.isFrozen(e))return i;e._isValid=i}return e._isValid}
function m(e){var t=d(NaN);return null!=e?h(c(t),e):c(t).userInvalidated=!0,t}
t=Array.prototype.some?Array.prototype.some:function(e){for(var t=Object(this),n=t.length>>>0,s=0;s<n;s++)if(s in t&&e.call(this,t[s],s,t))return!0;return!1};var _=n.momentProperties=[]
function y(e,t){var n,s,i;r(t._isAMomentObject)||(e._isAMomentObject=t._isAMomentObject),r(t._i)||(e._i=t._i),r(t._f)||(e._f=t._f),r(t._l)||(e._l=t._l),r(t._strict)||(e._strict=t._strict),r(t._tzm)||(e._tzm=t._tzm),r(t._isUTC)||(e._isUTC=t._isUTC),r(t._offset)||(e._offset=t._offset),r(t._pf)||(e._pf=c(t)),r(t._locale)||(e._locale=t._locale),_.length>0)for(n=0;n<_.length;n++)r(i=t[s=_[_][n]])||(e[s]=i);return e}
var g=!1;function v(e){y(this,e),this._d=new Date(null!=e._d?e._d.getTime():NaN),this.isValid()||(this._d=new Date(NaN)),!1===g&&(g=!0,n.updateOffset(this),g=!1)}
function p(e){return e instanceof v||null!=e&&null!=e._isAMomentObject}
function w(e){return e<0?Math.ceil(e)||0:Math.floor(e)}
function M(e){var t=+e,n=0;return 0!==t&&isFinite(t)&&(n=w(t)),n}
function k(e,t,n){var 
```

--------------------------------

### Get Browser Plugins

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Retrieves a list of installed browser plugins. This can be used as part of a fingerprint.

```javascript
navigator.plugins&&(e=e.concat(this.getRegularPlugins())),e}
```

--------------------------------

### Adding Docstring with Sample Usage

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/reddit.com.init.html

Include a comprehensive __doc__ string in __init__.py to provide clear documentation and sample usage for the package.

```python
__doc__ = """
This is the main package.

Example Usage:
    >>> from mypackage import my_function
    >>> my_function()
"""
```

--------------------------------

### Passwordless Start

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Initiates the passwordless authentication flow.

```APIDOC
## POST /passwordless/start

### Description
Initiates the passwordless authentication flow, sending a code or link to the user.

### Method
POST

### Endpoint
/passwordless/start

### Parameters
#### Request Body
- **clientID** (string) - Required - The client ID of your Auth0 application.
- **responseType** (string) - Required - The type of response expected (e.g., 'code', 'token', 'id_token').
- **redirectUri** (string) - Required - The URI to redirect to after authentication.
- **scope** (string) - Optional - The scope of the request.
- **authParams** (object) - Optional - Additional authentication parameters.
  - **scope** (string) - Optional - The scope of the request.
  - **redirect_uri** (string) - Optional - The URI to redirect to after authentication.
  - **response_type** (string) - Optional - The type of response expected.

### Request Example
```json
{
  "clientID": "YOUR_CLIENT_ID",
  "responseType": "token",
  "redirectUri": "YOUR_REDIRECT_URI",
  "scope": "openid profile email",
  "authParams": {
    "scope": "openid profile email",
    "redirect_uri": "YOUR_REDIRECT_URI",
    "response_type": "token"
  }
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.

#### Response Example
```json
{
  "success": true
}
```
```

--------------------------------

### Werkzeug's Flexible __init__.py Approach

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/reddit.com.init.html

This example demonstrates a flexible approach, similar to Werkzeug, that supports both explicit import styles and convenience imports. It allows users to choose their preferred method of accessing package components.

```python
from werkzeug import ...
```

--------------------------------

### jQuery Animation Prefilters

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/diariolibre.com-republica.html

Functions that run before an animation starts, used for setup and handling special cases like 'toggle', 'show', and 'hide'.

```javascript
i.Animation.prefilters = [function(n, t, u) {
    var f, y, w, c, b, h, o, l, k = "width" in t || "height" in t,
        v = this,
        p = {},
        s = n.style,
        a = dt(n),
        e = r.get(n, "fxshow");
    for (f in u.queue || (null == (c = i._queueHooks(n, "fx")).unqueued && (c.unqueued = 0, b = c.empty.fire, c.empty.fire = function() {
            c.unqueued || b()
        }), c.unqueued++), v.always(function() {
        v.always(function() {
            c.unqueued--,
            i.queue(n, "fx").length || c.empty.fire()
        })
    })), t)
        if (y = t[f], sf.test(y)) {
            if (delete t[f], w = w || "toggle" === y, y === (a ? "hide" : "show")) {
                if ("show" !== y || !e || void 0 === e[f]) continue;
                a = !0
            }
            p[f] = e && e[f] || i.style(n, f)
        }
    if ((h = !i.isEmptyObject(t)) || !i.isEmptyObject(p))
        for (f in k && 1 === n.nodeType && (u.overflow = [s.overflow, s.overflowX, s.overflowY], null == (o = e && e.display) && (o = r.get(n, "display")), "none" == (l = i.css(n, "display")) && (o ? l = o : (ht([n], !0), o = n.style.display || o, l = i.css(n, "display"), ht([n]))), ("inline" === l || "inline-block" === l && null != o) && "none" === i.css(n, "float") && (h || (v.done(function() {
            s.display = o
        }), null == o && (l = s.display, o = "none" === l ? "" : l))),
            s.display = "inline-block"));
    u.overflow && (s.overflow = "hidden", v.always(function() {
        s.overflow = u.overflow[0],
        s.overflowX = u.overflow[1],
        s.overflowY = u.overflow[2]
    })),
        h = !1,
        p)
        h || (e ? "hidden" in e && (a = e.hidden) : e = r.access(n, "fxshow", {
            display: o
        }), w && (e.hidden = a), a && ht([n], !0)), v.done(function() {
        for (f in a || ht([n]), r.remove(n, "fxshow"), p) i.style(n, f, p[f])
    })),
        h = lf(a ? e[f] : 0, f, v), f in e || (e[f] = h.start, a && (h.end = h.start, h.start = 0))
}];
i.Animation.prefilter = function(n, t) {
    t ? v.prefilters.unshift(n) : v.prefilters.push(n)
};
```

--------------------------------

### TD Theme Initialization

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/ukbdnews.com.23646.html

This JavaScript code snippet is the start of the theme's initialization script. It likely contains further setup and event listeners for theme functionality.

```javascript
( function () {
 "use
```

--------------------------------

### Initialize Focused Crawler

Source: https://github.com/adbar/trafilatura/blob/master/docs/crawls.md

Import and call the focused_crawler function with a homepage URL to start a crawl. This example performs the first iteration and limits the number of seen URLs.

```python
from trafilatura.spider import focused_crawler

# perform the first iteration (will not work with this website, there are no internal links)
>>> to_visit, known_links = focused_crawler("https://example.org", max_seen_urls=1)
```

--------------------------------

### Example Package Structure

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/reddit.com.init.html

This illustrates a typical file structure where __init__.py is used to control the visibility of other modules and their contents. It shows how to organize public and private modules within a package.

```Text
stuff/
  __init__.py
  bigstuff.py
  privateStuff.py
```

--------------------------------

### Get Current Timestamp

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/thepointsguy.com.alaska.html

Provides a high-resolution timestamp, typically from `performance.now()`, or falls back to `Date.now()` if unavailable. Includes an offset for accurate timing relative to the agent's start.

```javascript
function r(){return c.exists&&performance.now?Math.round(performance.now()):(o=Math.max((new Date).getTime(),o))-a}function i(){return o}
```

--------------------------------

### Initialize Auth0 Client with Options

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Initializes the Auth0 client with base options including domain and token. Sets up request handlers and client instances for various services.

```javascript
function Se(t){Q.check(t,{type:"object",message:"options parameter is not valid"},{domain:{type:"string",message:"domain option is required"},token:{type:"string",message:"token option is required"},_sendTelemetry:{optional:!0,type:"boolean",message:"_sendTelemetry option is not valid"},_telemetryInfo:{optional:!0,type:"object",message:"_telemetryInfo option is not valid"}}),this.baseOptions=t,this.baseOptions.headers={Authorization:"Bearer "+this.baseOptions.token},this.request=new ot(this.baseOptions),this.baseOptions.rootUrl=r("https://"+this.baseOptions.domain,"api","v2")}
```

--------------------------------

### Get Cookie Value by Offset

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/observer.com-LVMH.html

A helper function to extract a cookie's value from the document.cookie string, given a starting offset. It handles finding the end of the cookie value.

```javascript
function getCookieVal(offset) {
  var endstr = document.cookie.indexOf (";", offset);
  if (endstr == -1) {
    endstr = document.cookie.length;
  }
  return unescape(document.cookie.substring(offset, endstr));
}
```

--------------------------------

### Create a Store with Plugins

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Creates a store instance using specified plugins and middleware.

```javascript
const engine=require(1337);module.exports=engine.createStore([require(1339),require(1340)],[require(1336)]);
```

--------------------------------

### Media.net Outstream Video Player Setup

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/wsvn.com-crash.html

Configures the Media.net outstream video player based on context and video object parameters. Requires the media.net video bundle to be installed.

```javascript
t,n=(0,i.Z)(e,"context")||"",o=(0,i.Z)(e,"vto");"outstream"==n&&o&&(e.renderer=((t=c.Th.install({url:"https://prebid.media.net/video/bundle.js"})).setRender((function(e){window.mnet.queue.push((function(){var t={width:e.width,height:e.height,vastTimeout:e.vto,maxAllowedVastTagRedirects:e.mavtr,allowVpaid:e.avp,autoPlay:e.ap,preload:e.pl,mute:e.mt},n=e.dfp_id,i=(0,r.getGptSlotInfoForAdUnitCode)(n).divId||n;window.mnet.mediaNetoutstreamPlayer(e,i,t)}))})),t)))
window.mnet=window.mnet||"{}",window.mnet.queue=window.mnet.queue||"[]",p.urlData={domain:f.domain,page:f.page,isTop:f.reachedTop},pbjs.medianetGlobals=pbjs.medianetGlobals||"{}"
```

--------------------------------

### Signup and Login with Credentials

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Handles user signup followed by login. It first signs up the user and then logs them in with the provided credentials.

```javascript
be.prototype.signupAndLogin=function(t,e){var n=this;return this.client.dbConnection.signup(t,function(r){if(r)return e(r);n.loginWithCredentials(t,e)})}
```

--------------------------------

### Promise-based Initialization and API Setup

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/agrarheute.com.abfallholz.html

Initializes the agent using a Promise, handling configuration and setting up API methods for error handling, tracing, and custom attributes.

```javascript
new Promise((function(e,n){
if(P)e(P);
else{
var r=(0,g.gG)();
try{
(0,m.L)(t.Z,r.info),
(0,O.Dg)(t.Z,r.init),
function(t,e){
if(!t)throw new Error("All loader-config objects require an agent identifier!");
_["t.Z"]=new E.I(e,j),
(0,g.Qy)(t,_["t.Z"],"loader_config")
}(t.Z,r.loader_config),
(0,y.s)(t.Z,{}),
function(t){
var e=(0,g.fP)(),
n=s.ee.get(t),
r=n.get("tracer"),
o="api-",
i="api-ixn-";
function a(){}
(0,w.D)(["setErrorHandler","finished","addToTrace","inlineHit","addRelease"],(function(t,n){
e[n]=u(o,n,!0,"api")
})),
e.addPageAction=u(o,"addPageAction",!0),
e.setCurrentRouteName=u(o,"routeName",!0),
e.setPageViewName=function(e,n){
if("string"==typeof e)return"/"!==e.charAt(0)&&(e="/"+e),
(0,y.O)(t).customTransaction=(n||"http://custom.transaction")+
e,
u(o,"setPageViewName",!0,"api")
},
e.setCustomAttribute=function(e,n){
var r,i=(0,m.C)(t);
return(0,m.L)(t,Object.assign({},i,{jsAttributes:Object.assign({},i.jsAttributes,(r={},r[e]=n,r))})),
u(o,"setCustomAttribute",!0,"api")
},
e.interaction=function(){
return(new a).get()
};
var c=a.prototype={
createTracer:function(t,e){
var o={},i=this,a="function"==typeof e;
return(0,b.p)("api-ixn-tracer",[(0,x.zO)(),t,o]),
function(){
if(r.emit((a?"":"no-")+"fn-start",[(0,x.zO)(),i,a]),a)try{
return e.apply(this,arguments)
}catch(t){
throw r.emit("fn-err",[arguments,this,"string"==typeof t?new Error(t):t]),t
}finally{
r.emit("fn-end",[(0,x.zO)()]),o)
}
}
}
};
function u(t,e,r,o){
return function(){
return(0,b.p)("record-supportability",["API/"+e+"/called"]),
(0,b.p)(t+e,[(0,x.zO)()]).concat(h()(arguments)),r?void 0:this
}
}(0,w.D)(["actionText,setName,setAttribute,save,ignore,onEnd,getContext,end,get"],(function(t,e){
c[e]=u(i,e)
})),
e.noticeError=function(t,e){
"string"==typeof t&&(t=new Error(t)),
(0,b.p)("record-supportability",["API/noticeError/called"]),
(0,b.p)("err",[t,(0,x.zO)(),!1,e])
}
}(t)
}catch(t){n(t)}
}
})).then((function(){
var e,n,r,o=(0,tt.K)(t.Z);
o.page_view_event&&new R(t.Z),
o.page_view_timing&&new D(t.Z),
o.metrics&&new $(t.Z),
e="lite",
n?setTimeout((function(){return v(e)}),r||1e3):
c((function(){return v(e)}))
}))
```

--------------------------------

### Get Browser Plugins (Regular and IE)

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

This function retrieves a list of installed browser plugins. It handles both standard navigator.plugins and ActiveXObject for IE. Plugins are mapped to a string format for easier processing.

```javascript
getRegularPlugins:function(){var e=[]
if(navigator.plugins)for(var t=0,n=navigator.plugins.length;t<n;t++)navigator.plugins[t]&&e.push(navigator.plugins[t]);return this.pluginsShouldBeSorted()&&(e=e.sort(function(e,t){return e.name>t.name?1:e.name<t.name?-1:0})),this.map(e,function(e){var t=this.map(e,function(e){return[e.type,e.suffixes].join("~")}).join(",");return[e.name,e.description,t].join("::")},this)},getIEPlugins:function(){var e=[]
return(Object.getOwnPropertyDescriptor&&Object.getOwnPropertyDescriptor(window,"ActiveXObject")||"ActiveXObject"in window)&&(e=this.map(["AcroPDF.PDF","Adodb.Stream","AgControl.AgControl","DevalVRXCtrl.DevalVRXCtrl.1","MacromediaFlashPaper.MacromediaFlashPaper","Msxml2.DOMDocument","Msxml2.XMLHTTP","PDF.PdfCtrl","QuickTime.QuickTime","QuickTimeCheckObject.QuickTimeCheck.1","RealPlayer","RealPlayer.RealPlayer(tm) ActiveX Control (32-bit)","RealVideo.RealVideo(tm) ActiveX Control (32-bit)","Scripting.Dictionary","SWCtl.SWCtl","Shell.UIHelper","ShockwaveFlash.ShockwaveFlash","Skype.Detection","TDCCtl.TDCCtl","WMPlayer.OCX","rmocx.RealPlayer G2 Control","rmocx.RealPlayer G2 Control.1"],function(e){try{return new window.ActiveXObject(e),e}catch(e){return null}})),navigator.plugins&&(e=e.concat(this.getRegularPlugins())),e}
```

--------------------------------

### Install Evaluation Dependencies and Run Script

Source: https://github.com/adbar/trafilatura/blob/master/docs/tests.md

Installs the necessary dependencies for evaluation and displays help for the evaluation script. Use --small for Trafilatura-only components or --all for all algorithms.

```bash
$ pip install -e ".[eval]"
$ python3 tests/evaluate.py --help
```

--------------------------------

### String.padStart() Example

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/chefkoch.de.mandarinen.html

Pads the current string with another string (multiple times, if needed) until the resulting string reaches the given length. The padding is applied from the start of the current string. This implementation includes a Safari-specific workaround.

```javascript
e(285),t.exports=e(7).String.padStart
```

```javascript
"use strict";var r=e(0),i=e(124),o=e(61),c=/Version\/10\.\d+(\.\d+)?( Mobile\/\w+)? Safari\//.test(o);r(r.P+r.F*c,"String",{padStart:function(t){return i(this,t,arguments.length>1?arguments[1]:void 0,!0)}})
```

--------------------------------

### Dockerfile with poetry install

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/pythonspeed.com.docker.html

A Dockerfile that installs poetry, copies project files, installs dependencies using poetry install --no-root, and then installs the application code. This approach separates dependency installation from application code installation.

```dockerfile
FROM python:3.8-slim
RUN pip install poetry
WORKDIR /tmp/myapp
COPY pyproject.toml poetry.lock .
RUN cd /tmp && poetry install --no-root
# Copy in everything else:
COPY . .
RUN poetry install

CMD flask run exampleapp:app
```

--------------------------------

### Starter JS Parameters

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/publishedreporter.com-Hottest.html

Parameters for a starter JavaScript module, controlling opt-in consent, session tracking, user login status, and AJAX URL.

```javascript
var ig_es_starter_main_js_params = {"show_opt_in_consent":"yes","session_tracking_enabled":"yes","user_logged_in":"no","consent_opt_in_message":"","consent_opt_out_message":"No Thanks","conset_after_opt_out_message":"You won't receive further emails from us, thank you!","ajax_url":"https:\/\/www.publishedreporter.com\/wp-admin\/admin-ajax.php","security":"dd2c8e2212"}
```

--------------------------------

### Initialize a New Poetry Project

Source: https://github.com/adbar/trafilatura/blob/master/tests/cache/pluralsight.com.python.html

Use `poetry init` to create a new project interactively, generating a `pyproject.toml` file. This is the starting point for defining your project's metadata and dependencies.

```bash
$ poetry init
```

--------------------------------

### Dockerfile with pipenv install during build

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/pythonspeed.com.docker.html

A Dockerfile that installs pipenv, copies Pipfile and Pipfile.lock, generates requirements.txt using pipenv lock --requirements, installs dependencies, copies application code, and installs the application.

```dockerfile
FROM python:3.7
RUN pip install pipenv
COPY Pipfile* /tmp
RUN cd /tmp && pipenv lock --requirements > requirements.txt
RUN pip install -r /tmp/requirements.txt
COPY . /tmp/myapp
RUN pip install /tmp/myapp
CMD flask run exampleapp:app
```

--------------------------------

### Connect to Epsilla and Setup Database

Source: https://github.com/adbar/trafilatura/blob/master/docs/tutorial-epsilla.md

Connect to a local Epsilla server, load or create a database named 'TrafilaturaDB', and set up a table named 'Trafilatura' with appropriate fields for document ID, content, and vector embeddings.

```python
from pyepsilla import vectordb
client = vectordb.Client(
    # replace with a production server if not running a local docker container
    host='localhost',
    port='8888'
)

status_code, response = client.load_db(
    db_name="TrafilaturaDB",
    db_path="/tmp/trafilatura_store"
)
print(response)

client.use_db(db_name="TrafilaturaDB")

# creates a table called Trafilatura
client.drop_table('Trafilatura')
client.create_table(
  table_name="Trafilatura",
  table_fields=[
    {"name": "ID", "dataType": "INT"},
    {"name": "Doc", "dataType": "STRING"},
    {"name": "Embedding", "dataType": "VECTOR_FLOAT", "dimensions": 384}
  ]
)
```

--------------------------------

### Install Packages in a Conda Environment

Source: https://github.com/adbar/trafilatura/blob/master/tests/cache/pluralsight.com.python.html

Installs a package from the conda repository into the active environment. Use $PACKAGE_NAME for the package you wish to install.

```bash
$ conda install $PACKAGE_NAME
```

--------------------------------

### Build Documentation with Sphinx

Source: https://github.com/adbar/trafilatura/blob/master/docs/index.md

Instructions for building the project's documentation using Sphinx. Ensure you have installed the required packages first.

```bash
pip install -r requirements.txt
sphinx-build -b html . _build/
```

--------------------------------

### Signup and Login Flow

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Handles user signup and then proceeds to log the user in using the provided credentials.

```javascript
var n=this;return this.webAuth.client.dbConnection.signup(t,function(r){return r?(e(r)):(t.realm=t.realm||t.connection,delete t.connection,n.webAuth.login(t,e))})

```

--------------------------------

### Get Attribute

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/fitforfun.de.sozialeaengste.html

Gets an attribute from an HTML element.

```javascript
function ie(e,t){return e[k("Z2V0QXR0cmlidXRl")]("getAttribute",t)}
```

--------------------------------

### Install Python Version with pyenv

Source: https://github.com/adbar/trafilatura/blob/master/tests/cache/pluralsight.com.python.html

Install a specific Python version using pyenv. This command allows for easy management of multiple Python installations.

```bash
$ pyenv install $PYTHON_VERSION_OR_DEFINITION_FILE
```

--------------------------------

### Install and Load Reticulate Package in R

Source: https://github.com/adbar/trafilatura/blob/master/docs/usage-r.md

Installs the reticulate package from CRAN and loads it into the R session. Ensure R and Python are installed on your system.

```R
> install.packages("reticulate")
> library(reticulate)
```

--------------------------------

### Sample URLs with Command-Line Utility

Source: https://github.com/adbar/trafilatura/blob/master/docs/url-management.md

Perform URL sampling on a list of URLs using the command-line utility. Specify input and output files, enable sampling, and set the desired sample size.

```bash
courlan --inputfile urls.txt --outputfile samples-urls.txt --sample --samplesize 50
```

--------------------------------

### Verify Poetry installation

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/github.com.yapapi.html

Check if Poetry is installed and accessible in your environment.

```bash
poetry --version
```

--------------------------------

### Load and Use Custom Configuration File in Python

Source: https://github.com/adbar/trafilatura/blob/master/docs/settings.md

This example demonstrates how to load custom settings from a configuration file using `use_config` and then apply these settings to the `extract` function. It also shows an alternative method of providing the file name directly to the `extract` function.

```python
from trafilatura import extract
from trafilatura.settings import use_config

newconfig = use_config("myfile.cfg")

extract(downloaded, config=newconfig)

extract(downloaded, settingsfile="myfile.cfg")
```

--------------------------------

### Initialize Auth0 Client with Minimal Options

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/nymag.com.polarization.html

Initializes the Auth0 client with only the required domain and clientID. Other options are set to defaults.

```javascript
function xe(t,e){2===arguments.length?this.auth0=t:e=t,Q.check(e,{type:"object",message:"options parameter is not valid"},{domain:{type:"string",message:"domain option is required"},clientID:{type:"string",message:"clientID option is required"},responseType:{optional:!0,type:"string",message:"responseType is not valid"},responseMode:{optional:!0,type:"string",message:"responseMode is not valid"}
```

--------------------------------

### Russian Text Example

Source: https://github.com/adbar/trafilatura/blob/master/tests/resources/utf8.html

An example of Russian text, showcasing the Cyrillic script in Unicode.

```Russian
Зарегистрируйтесь сейчас на Десятую Международную Конференцию по
Unicode, которая состоится 10-12 марта 1997 года в Майнце в Германии.
Конференция соберет широкий круг экспертов по  вопросам глобального
Интернета и Unicode, локализации и интернационализации, воплощению и
применению Unicode в различных операционных системах и программных
приложениях, шрифтах, верстке и многоязычных компьютерных системах.
```

--------------------------------

### Initialize Project Flags and Settings

Source: https://github.com/adbar/trafilatura/blob/master/tests/eval/tvtoday.de-willsmith.html

Sets up various configuration flags and initial states for the project. This includes random number generation, domain settings, and boolean flags that control features like safe detection, ad rendering, and recovery mechanisms.

```javascript
function _(){
if(n.mnxyg200Rnd=n.mnxyg200Rnd||I(),n.mnxyg494sdl&&!n.mnxyg494sdln&&(
D(4),
n.mnxyg494sdln=!0),
!n.mnxyg494sdl){
n.mnxyg494sdl=!0,
n.mnxyg494sdln=!0,
n.mnxygd463genat="10162124",
n.mnxygd463sthash="f7cb4",
n.mnxygd463Domain=Xn("zzz1wywrgd|1gh",-3),
n.mnxyg200Rnd=n.mnxyg200Rnd||I(),
n.mnxyg073forceimp=!1,
n.mnxyg463Flags.newSafDet=!!n.mnxyg463Flags.newSafDet&&n.mnxyg463Flags.newSafDet,
n.mnxyg463Flags.gnrcfll=void 0===n.mnxyg463Flags.gnrcfll||n.mnxyg463Flags.gnrcfll,
n.mnxyg463Flags.forceImp=!!n.mnxyg463Flags.forceImp&&n.mnxyg463Flags.forceImp,
n.mnxyg463Flags.autoRecov=!!n.mnxyg463Flags.autoRecov&&n.mnxyg463Flags.autoRecov,
n.mnxyg463Flags.fp=!!n.mnxyg463Flags.fp&&n.mnxyg463Flags.fp,
n.mnxyg463Flags.npm=!!n.mnxyg463Flags.npm&&n.mnxyg463Flags.npm,
n.mnxyg463Flags.rsp=void 0===n.mnxyg463Flags.rsp||n.mnxyg463Flags.rsp,
n.mnxyg463Flags.nw05elst=!!n.mnxyg463Flags.nw05elst&&n.mnxyg463Flags.nw05elst,
n.mnxyg463Flags.bl966Up=!!n.mnxyg463Flags.bl966Up&&n.mnxyg463Flags.bl966Up,
n.mnxygsc={},
n.mnxygsc.tsm={};
var e=!1,
t=R(function(){
if(e=!0,
null!=(In=fn("body"))){
if(n.mnxygebeBody)return;
n.mnxygebeBody=!0,
j(t),
var d=Q("style");
d.type="text/css",
$(Ln,d),
M()
}
},10),
d=R(function(){
j(d),
e||D(4)
},100)
}
}
function B(){
if(!_n){
if(_n=!0,
n.mnxyg3adInject=!1,
n.mnxyga01Track=!1,
n.mnxyg798Detect=!1,
n.mnxygebeBody=!1,
n.mnxyg494sdl=!1,
n.mnxyg494sdln=!1,
n.mnxyg200Rnd=!1,
L(),
n.mnxyg463Flags.nopmcache=!1,
n.mnxyg463Flags.strictNCNA=!1,
!H())
return void(_n=!1);
_(),
n[Xn("XDESYuhordghg",-3)]instanceof n.Function&&
n[Xn("XDESYuhordghg",-3)](n.mnxyg200Rnd)
}
}
function L(){
for(var e=0;e<n.mnxygCln.length;e++)
gn(n.mnxygCln[e]);
for(var e=0;e<n.mnxygClne.length;e++)
sn(n.mnxygClne[e][0],n.mnxygClne[e][1],n.mnxygClne[e][2],n.mnxygClne[e][3]);
for(var e=0;e<n.mnxygClni.length;e++)
j(n.mnxygClni[e]);
n.mnxygCln.length=0,
n.mnxygClne.length=0,
n.mnxygClni.length=0
}
function I(){
return Xn(K(),3)
}
function K(){
for(var 
```