### Quick Verification Loop Source: https://github.com/contentful/contentful-export/blob/main/AGENTS.md Run this command to perform a quick verification of the project setup, including installation, building, unit testing, and linting. ```bash npm install && npm run build && npm run test:unit && npm run lint ``` -------------------------------- ### Install Dependencies Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Clone the repository and install project dependencies using npm. ```bash git clone git@github.com:contentful/contentful-export.git cd contentful-export npm install ``` -------------------------------- ### Complete Contentful Export Configuration Example Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md A comprehensive example demonstrating all available configuration options for the contentful-export tool, including required, authentication, output, filtering, and performance settings. ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ // Required spaceId: 'my-space-id', managementToken: 'CFPAT-...', // Authentication & Connectivity deliveryToken: 'token-...', host: 'api.contentful.com', hostDelivery: 'cdn.contentful.com', // Environment & Scope environmentId: 'master', // Output & Files exportDir: './exported-data', saveFile: true, contentFile: 'space-export.json', errorLogFile: './logs/errors.json', // Content Filtering includeDrafts: false, includeArchived: false, queryEntries: ['content_type=page'], // Content Skipping skipContentModel: false, skipContent: false, skipRoles: false, skipWebhooks: false, skipTags: false, stripTags: false, // Asset Downloads downloadAssets: true, // Performance maxAllowedLimit: 1000, // Display useVerboseRenderer: true, // Headers & Tracking headers: { 'X-Custom-Header': 'value' } }) ``` -------------------------------- ### Asset Download Output Examples Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md Illustrates the real-time status updates during asset downloads, showing successful downloads, warnings for missing files, and errors for failed downloads. ```text ✓ downloaded image-file.jpg (https://images.contentful.com/2021/12/image.jpg) ✓ downloaded document.pdf (https://assets.contentful.com/2021/12/doc.pdf) ⚠ asset my-media has no file(s) ✗ error downloading https://broken-cdn.example.com/file.jpg: error response status: 404 ``` -------------------------------- ### Basic Configuration with Contentful Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md Demonstrates the basic setup for using the contentful-export library by passing required options to the runContentfulExport function. ```javascript import contentfulExport from 'contentful-export' const options = { spaceId: 'my-space', managementToken: 'my-token', // ... additional options } await contentfulExport(options) ``` -------------------------------- ### Install contentful-export CLI Source: https://github.com/contentful/contentful-export/blob/main/README.md Install the contentful-export package using npm. Ensure you have Node.js LTS installed. ```bash npm install contentful-export ``` -------------------------------- ### Get Full Source Space Configuration Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/get-space-data.md Demonstrates the configuration object required for the `getFullSourceSpace` function. It includes parameters for client initialization, space and environment IDs, and various options to control data fetching. ```javascript const config = { client: contentfulManagement.createClient({ accessToken: "YOUR_CMA_TOKEN", }), spaceId: "YOUR_SPACE_ID", environmentId: "master", // or "development", etc. skipContentModel: false, skipContent: false, skipWebhooks: false, skipRoles: false, skipEditorInterfaces: false, skipTags: false, stripTags: false, includeDrafts: false, includeArchived: false, maxAllowedLimit: 1000, }; const taskRunner = getFullSourceSpace(config); taskRunner.run().then((context) => { // Process the fetched data console.log("Fetched content types:", context.data.contentTypes.length); console.log("Fetched entries:", context.data.entries.length); console.log("Fetched assets:", context.data.assets.length); // ... and so on for other data types }).catch((error) => { console.error("Error fetching space data:", error); }); ``` -------------------------------- ### Proxy Configuration Parsing Example Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/parse-options.md Illustrates the parsing of a proxy configuration string into a structured object. This object includes protocol, hostname, port, and authentication credentials if provided. The function supports both basic host:port and user:password@host:port formats. ```javascript // Input: 'user:pass@proxy.example.com:8080' // Becomes: { protocol: 'http', hostname: 'proxy.example.com', port: 8080, username: 'user', password: 'pass' } ``` -------------------------------- ### Conventional Commits Examples Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Examples of valid commit messages following the Conventional Commits specification. These are used for automated versioning and changelog generation. ```bash feat: add support for exporting taxonomies fix: handle embargoed asset download timeout build(deps): bump contentful-management to v12 chore(ci): update Node version in workflow ``` -------------------------------- ### Example JSON Response Snippet Source: https://github.com/contentful/contentful-export/blob/main/README.md Illustrates the structure of a JSON response, highlighting fields like createdAt, updatedAt, publishedVersion, publishedAt, publishedCounter, and version. This is useful for understanding the data returned by the Contentful Management API. ```json "createdAt": "2020-01-06T12:00:00.000Z", "updatedAt": "2020-01-06T12:00:00.000Z", "publishedVersion": 23, "publishedAt": "2020-04-05T14:00:00.000Z", "publishedCounter": 1, "version": 50, ``` -------------------------------- ### Initialize Client with Custom HTTPS Agent Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/init-client.md Configure a Contentful client to use a custom HTTPS agent, which is useful for setting up proxy or TLS configurations. This example demonstrates setting a proxy host and port. ```javascript const https = require('https') const agent = new https.Agent({ hostname: 'proxy.example.com', port: 8080 }) const options = { spaceId: 'my-space', managementToken: 'my-token', httpsAgent: agent } const client = initClient(options) ``` -------------------------------- ### Query Parameter Processing Example Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/parse-options.md Demonstrates how query parameters provided as arrays are processed into a single object for use in API requests. This is useful for filtering entries or assets based on specific criteria. ```javascript // Input: ['content_type=blog', 'fields.slug=my-slug'] // Becomes: { 'content_type': 'blog', 'fields.slug': 'my-slug' } ``` -------------------------------- ### Utility Function to Get Headers Configuration Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Parses header strings into a configuration object. Useful for setting up request headers. ```javascript getHeadersConfig(value: string | string[]): object ``` -------------------------------- ### Initialize Contentful API Clients Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/module-overview.md Creates Contentful API client instances (CMA or CDA). Configures clients with logging handlers, default timeouts, and proxy settings. Use this to get clients for interacting with Contentful. ```javascript import initClient from 'lib/tasks/init-client' const cmaClient = initClient(options) const cdaClient = initClient(options, true) ``` -------------------------------- ### Example Bug Report Data Structure Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/errors.md When reporting bugs, include detailed information about the environment, export options, and the error itself. This structure provides a template for essential bug report data. ```javascript { nodeVersion: process.version, spaceId: 'my-space', environmentId: 'master', options: { skipRoles: false, skipContent: false, downloadAssets: false, maxAllowedLimit: 1000 }, error: { name: err.name, message: err.message, stack: err.stack } } ``` -------------------------------- ### Build Project Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Build the project, which includes cleaning, type-checking, and compiling with Babel. ```bash npm run build ``` -------------------------------- ### Recommended CLI Usage Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Illustrates the current recommended method for exporting content using the `contentful-cli`. This is the preferred approach for CLI-based exports. ```bash # New way (recommended) npx contentful-cli space export --space-id my-space --management-token abc123 ``` -------------------------------- ### Return Value Structure Example Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/types.md A record object where keys are from ContentfulExportField and values are arrays of corresponding exported entities. Only keys for which data was exported are present. ```typescript Record ``` ```typescript { contentTypes?: unknown[], // Content type definitions entries?: unknown[], // Content entries assets?: unknown[], // Asset definitions locales?: unknown[], // Locale configurations tags?: unknown[], // Tag assignments webhooks?: unknown[], // Webhook definitions roles?: unknown[], // Role and permission definitions editorInterfaces?: unknown[] // Editor interface metadata } ``` -------------------------------- ### Run All Tests Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Execute the full test suite, including linting, building, unit tests, and integration tests. Integration tests require specific environment variables. ```bash npm test ``` -------------------------------- ### Configure Proxy Correctly for contentful-export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/errors.md Demonstrates valid formats for the `proxy` configuration option. Incorrect formats can lead to configuration validation errors. ```javascript { proxy: 'proxy.example.com' } // Missing port { proxy: '8080' } // Missing host { proxy: 'user:pass@proxy.example.com' } // Missing port ``` ```javascript { proxy: 'proxy.example.com:8080' } ``` ```javascript { proxy: 'user:pass@proxy.example.com:8080' } ``` -------------------------------- ### Export Output Structure Example Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/README.md The resolved promise from the export function returns an object containing various Contentful entity types. Each key is present only if that entity type was successfully exported. ```typescript { contentTypes?: Array, entries?: Array, assets?: Array, locales?: Array, tags?: Array, webhooks?: Array, roles?: Array, editorInterfaces?: Array } ``` -------------------------------- ### Load Configuration from File Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md Options can be loaded from a JSON configuration file and merged with command-line or programmatic options. The config file has lower priority than programmatic options. ```javascript import contentfulExport from 'contentful-export' // config.json: // { // "spaceId": "my-space", // "managementToken": "token", // "includeDrafts": true, // "maxAllowedLimit": 500 // } const result = await contentfulExport({ config: './config.json', downloadAssets: true // Override specific option }) ``` -------------------------------- ### Using Partial Options with Spread Syntax Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/types.md Shows how to create a configuration object using partial types and the spread syntax. This is useful for defining a base set of required options and then extending them with additional optional properties. ```typescript import type { Options } from 'contentful-export' const baseOptions: Pick = { spaceId: 'my-space', managementToken: 'token' } const fullOptions: Options = { ...baseOptions, environmentId: 'staging', maxAllowedLimit: 500 } ``` -------------------------------- ### Merge Configuration Options Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Demonstrates how configuration options are merged, with user parameters taking the highest priority. Use this to understand how to override default or config file settings. ```javascript const result = await contentfulExport({ config: './config.json', // Loaded first downloadAssets: true // Overrides config }) ``` -------------------------------- ### Enable Verbose Renderer Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md Use `useVerboseRenderer: true` to display progress output line-by-line instead of a spinner. This is useful for CI/CD environments. ```javascript { useVerboseRenderer: true } ``` -------------------------------- ### Load Configuration from File Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Loads export configuration options from a JSON file. Individual options can still be overridden by passing them directly to the function. ```javascript // config.json { "spaceId": "my-space", "managementToken": "token", "includeDrafts": true, "maxAllowedLimit": 500 } ``` ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ config: './config.json', downloadAssets: true // Can override }) ``` -------------------------------- ### Import and Initialize Contentful Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/README.md Import the library and initialize the export process with your space ID and management token. Additional options can be provided for more specific export configurations. ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ spaceId: 'my-space-id', managementToken: 'CFPAT-...', // additional options... }) ``` -------------------------------- ### Run Contentful Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/module-overview.md The main entry point for the library. Orchestrates option parsing, client initialization, data fetching, asset downloads, and file writing. Use this for programmatic exports. ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ spaceId: 'my-space', managementToken: 'token' }) ``` -------------------------------- ### Optimize for Many Assets Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Configure these options when dealing with a large number of assets to ensure successful downloads. Enabling `downloadAssets` and increasing the `timeout` are key for this scenario. ```javascript { downloadAssets: true, timeout: 30000 // Longer timeout } ``` -------------------------------- ### Basic Asset Download Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md Initiates an export process that also downloads all associated assets to a specified directory. Requires space ID, management token, and a download directory. ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ spaceId: 'my-space', managementToken: 'token', downloadAssets: true, exportDir: './exported-space' }) console.log('Downloaded assets:', result.assetDownloads.successCount) ``` -------------------------------- ### Enable Asset Downloads Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md Set `downloadAssets` to `true` to download asset binary files to the local filesystem. This is required for embargoed assets. ```javascript { downloadAssets: true } ``` -------------------------------- ### Axios HTTP Client Configuration Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md This snippet shows how an axios instance is created with provided options for making download requests. It configures headers, timeout, and HTTP/HTTPS agents. ```javascript const httpClient = axios.create({ headers: options.headers, timeout: options.timeout, httpAgent: options.httpAgent, httpsAgent: options.httpsAgent, proxy: options.proxy }) ``` -------------------------------- ### initClient Function Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/init-client.md Initializes a Contentful API client. It can be configured for either the Content Management API (CMA) or the Content Delivery API (CDA) based on the `useCda` flag and provided options. ```APIDOC ## initClient(opts, useCda) ### Description Creates and configures a Contentful API client instance (either Content Management API or Content Delivery API). ### Signature ```javascript function initClient(opts: Options, useCda?: boolean): ContentfulClient ``` ### Parameters #### `opts` (Options) - Required Configuration object with API credentials and connection details. #### `useCda` (boolean) - Optional If true, creates a CDA client for public/published content access. If false, creates a CMA client for full content access. Defaults to `false`. ### Options (Required Properties) - **`spaceId`** (string) - Required - Contentful space ID - **`managementToken`** (string) - Required - CMA authentication token - **`deliveryToken`** (string) - Required - CDA authentication token (required if `useCda` is true) - **`environmentId`** (string) - Optional - Environment ID (defaults to 'master' in caller) - **`host`** (string) - Optional - CMA API host (defaults to 'api.contentful.com') - **`hostDelivery`** (string) - Optional - CDA API host (defaults to 'cdn.contentful.com') - **`logHandler`** (function) - Optional - Logging function for SDK output - **`timeout`** (number) - Optional - Request timeout in milliseconds - **`httpsAgent`** (HttpsAgent) - Optional - Custom HTTPS agent for proxy/TLS configuration - **`httpAgent`** (HttpAgent) - Optional - Custom HTTP agent for proxy configuration ### Client Type Selection #### CMA Client (`useCda = false`) Returns a Contentful Management API client with access to all content versions, content model definitions, roles, permissions, webhooks, tags, and full entity metadata. Created using `createCmaClient` from `contentful-management` SDK with legacy API compatibility mode. #### CDA Client (`useCda = true`) Returns a Contentful Delivery API client with access to published/released content only, offering faster response times. Configured with space alias, delivery token, delivery API host, and `.withoutLinkResolution` modifier. ### Default Configuration The function applies default configuration including a `timeout` of 10000ms and a `logHandler` that emits SDK log events to `contentful-batch-libs`. ### Return Type - **CMA Mode**: `ContentfulClient` - Management API client instance (legacy type) - **CDA Mode**: `ContentfulClient` - Delivery API client instance with `.withoutLinkResolution` modifier ### Usage Examples #### Initialize CMA client for complete export ```javascript import initClient from './lib/tasks/init-client' const options = { spaceId: 'my-space', managementToken: 'management-token-xyz', host: 'api.contentful.com', timeout: 10000 } const cmaClient = initClient(options) const space = await cmaClient.getSpace(options.spaceId) ``` #### Initialize CDA client for published-only export ```javascript const options = { spaceId: 'my-space', managementToken: 'management-token-xyz', deliveryToken: 'delivery-token-abc', environmentId: 'master', hostDelivery: 'cdn.contentful.com' } const cdaClient = initClient(options, true) const entries = await cdaClient.withAllLocales.getEntries() ``` #### With custom HTTPS agent for proxy ```javascript const https = require('https') const agent = new https.Agent({ hostname: 'proxy.example.com', port: 8080 }) const options = { spaceId: 'my-space', managementToken: 'my-token', httpsAgent: agent } const client = initClient(options) ``` ### Error Handling Errors during client initialization (invalid tokens, network issues) are propagated up through the task wrapper in the export pipeline. SDK configuration errors throw immediately; API authentication errors occur during the first API call (getSpace). ``` -------------------------------- ### Manage Rate Limiting with Reduced Limit and Verbose Output Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/errors.md To mitigate consistent rate limiting on large spaces, reduce `maxAllowedLimit` and enable `useVerboseRenderer` to monitor progress. ```javascript { maxAllowedLimit: 50, useVerboseRenderer: true // Monitor progress } ``` -------------------------------- ### Watch Mode for Builds Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Enable watch mode for incremental builds during development. ```bash npm run build:watch ``` -------------------------------- ### Run Unit Tests Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Execute only the unit tests for the project. ```bash npm run test:unit ``` -------------------------------- ### Basic Export to File Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/main-export-function.md Performs a basic export of all content and saves it to a JSON file. Requires space ID and management token. ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ spaceId: 'my-space-id', managementToken: 'my-management-token' }) console.log('Exported entries:', result.entries.length) console.log('Exported assets:', result.assets.length) ``` -------------------------------- ### Project File Structure Overview Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/README.md This snippet outlines the directory and file structure of the contentful-export project. It highlights the main entry points, task-specific directories, utility modules, and CLI scripts. ```text lib/ ├── index.js Main entry point ├── parseOptions.js Option parsing ├── tasks/ │ ├── init-client.js Client initialization │ ├── get-space-data.js Data fetching │ └── download-assets.js Asset downloading └── utils/ ├── embargoedAssets.js URL signing └── headers.js Header parsing bin/ └── contentful-export CLI entry (deprecated) types.d.ts TypeScript definitions dist/ Compiled output (not in git) ``` -------------------------------- ### Build Process for Contentful Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/module-overview.md This command compiles the library into distributable files using Babel. It generates CommonJS output suitable for Node.js environments. ```bash npm run build # Compiles lib/ → dist/ using Babel # Generates CJS output compatible with Node.js ``` -------------------------------- ### downloadAssets(options) Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md Downloads asset binary files from a Contentful space to the local filesystem. This function handles embargoed (signed URL) assets and manages download concurrency, returning a task executor compatible with the Listr task runner. ```APIDOC ## `downloadAssets(options)` ### Description Downloads asset binary files from a Contentful space to the local filesystem. Handles embargoed (signed URL) assets and manages download concurrency. ### Signature ```javascript function downloadAssets(options: DownloadOptions): (ctx: Context, task: Task) => Promise ``` ### Parameters #### Path Parameters * None #### Query Parameters * None #### Request Body * None ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | options.exportDir | string | Yes | Directory where asset files are saved | | options.headers | object | No | Custom HTTP headers for download requests | | options.timeout | number | No | Request timeout in milliseconds | | options.httpAgent | HttpAgent | No | Custom HTTP agent for proxy/connection configuration | | options.httpsAgent | HttpsAgent | No | Custom HTTPS agent for proxy/connection configuration | | options.proxy | object | No | Proxy configuration (if rawProxy is true) | | options.host | string | No | CMA API host (required for embargoed assets) | | options.accessToken | string | No | Management token (required for embargoed asset signing) | | options.spaceId | string | No | Space ID (required for embargoed asset signing) | | options.environmentId | string | No | Environment ID (required for embargoed asset signing) | ### Return Type ```javascript Promise ``` The returned promise resolves when all asset downloads complete. Updates are available via `task.output` property during download, and results are stored in `ctx.assetDownloads`: ```javascript ctx.assetDownloads = { successCount: number, // Successfully downloaded assets warningCount: number, // Assets skipped (no file data) errorCount: number // Download failures } ``` ### Download Process #### 1. HTTP Client Configuration Creates an axios instance with provided options: ```javascript const httpClient = axios.create({ headers: options.headers, timeout: options.timeout, httpAgent: options.httpAgent, httpsAgent: options.httpsAgent, proxy: options.proxy }) ``` #### 2. Asset Iteration Iterates through `ctx.data.assets` array with concurrency of 6: ```javascript Promise.map(ctx.data.assets, (asset) => { ... }, { concurrency: 6 }) ``` Each asset can have multiple locales with different file URLs. #### 3. File Validation For each asset: - Checks if `asset.fields.file` exists - If missing: emits warning, increments `warningCount`, continues - Extracts all locales from `asset.fields.file` object #### 4. URL Processing For each locale: - Extracts URL from `asset.fields.file[locale].url` - If URL missing: emits error, increments `errorCount`, continues - If URL starts with `//`: prepends `https:` protocol #### 5. Embargoed Asset Handling If URL matches embargoed pattern (`*.secure.*` domains): - Calculates JWT signature expiry timestamp (current + 6 hours) - Calls `signUrl()` to generate signed URL via asset_keys API - Replaces original URL with signed URL before download Non-embargoed URLs are downloaded as-is. #### 6. File Download ```javascript await downloadAsset({ url: fileUrl, directory: options.exportDir, httpClient: httpClient }) ``` The `downloadAsset` helper function: 1. Parses URL to extract host and pathname 2. Creates local file path: `exportDir//` 3. Creates directory structure recursively 4. Streams HTTP response to file using `streamPipeline` 5. Returns local file path on success 6. Throws error on HTTP failure (non-2xx status) #### 7. Result Tracking - Success: increments `successCount`, emits tick (✓) with URL - Error: increments `errorCount`, emits cross (✗) with error message ### Directory Structure Downloaded assets are organized by host: ``` exportDir/ ├── images.contentful.com/ │ ├── 2021/12/01/ │ │ └── asset-file.jpg │ └── 2022/01/15/ │ └── image-001.png ├── assets.contentful.com/ │ └── ... └── images.secure.contentful.com/ └── ... ``` Paths are URL-decoded to handle special characters in filenames. ``` -------------------------------- ### Download with Proxy Configuration Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md Configures the export process to use a proxy server for downloading assets. Supports both raw proxy string and direct passing to axios. ```javascript const result = await contentfulExport({ spaceId: 'my-space', managementToken: 'token', downloadAssets: true, exportDir: './assets', proxy: 'user:password@proxy.example.com:8080', rawProxy: true // Pass proxy to axios directly }) ``` -------------------------------- ### TypeScript Usage with Types Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Demonstrates how to use the contentful-export library in TypeScript projects, including importing types for configuration options. ```typescript import contentfulExport from 'contentful-export' import type { Options } from 'contentful-export' const config: Options = { spaceId: 'my-space', managementToken: 'token', includeDrafts: true } const result = await contentfulExport(config) ``` -------------------------------- ### Run Integration Tests Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Execute only the integration tests. Ensure necessary environment variables are set. ```bash npm run test:integration ``` -------------------------------- ### Initialize CMA Client for Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/init-client.md Use this snippet to initialize a Contentful Management API client for complete data export. Ensure you provide the correct `spaceId` and `managementToken`. ```javascript import initClient from './lib/tasks/init-client' const options = { spaceId: 'my-space', managementToken: 'management-token-xyz', host: 'api.contentful.com', timeout: 10000 } const cmaClient = initClient(options) const space = await cmaClient.getSpace(options.spaceId) ``` -------------------------------- ### Optimize for Large Spaces Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Use these options to improve performance when exporting from large Contentful spaces. Adjusting `maxAllowedLimit` and enabling `useVerboseRenderer` can help manage the export process. ```javascript { maxAllowedLimit: 100, // Smaller pages useVerboseRenderer: true // See progress } ``` -------------------------------- ### runContentfulExport Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/main-export-function.md The primary entry point for the contentful-export library. It initializes the client, fetches space data, optionally downloads assets, and writes the data to a JSON file. ```APIDOC ## `runContentfulExport(params)` ### Description The primary entry point for the contentful-export library. Orchestrates the complete export pipeline: client initialization, space data fetching, optional asset downloads, and JSON file writing. ### Method Signature ```javascript function runContentfulExport(params: Options): Promise> ``` ### Parameters #### `params` (Options) - Required Configuration object containing space ID, API tokens, and export preferences. ### Options Interface The `Options` object controls all aspects of the export operation: #### Path Parameters * **`spaceId`** (string) - Required - ID of the Contentful space to export * **`managementToken`** (string) - Required - CMA (Content Management API) authentication token #### Query Parameters * **`environmentId`** (string) - Optional - Environment ID within the space; defaults to master * **`deliveryToken`** (string) - Optional - CDA (Content Delivery API) token; if provided and `includeDrafts` is false, entries and assets are fetched from the public API instead of management API * **`exportDir`** (string) - Optional - Directory path where the JSON export file and downloaded assets will be saved; defaults to `process.cwd()` * **`saveFile`** (boolean) - Optional - Whether to write the export data to a JSON file on disk; defaults to `true` * **`contentFile`** (string) - Optional - Custom filename for the export JSON file; defaults to `contentful-export-{spaceId}-{environmentId}-{timestamp}.json` * **`includeDrafts`** (boolean) - Optional - Include draft (unpublished) entries and assets in the export. If true, `deliveryToken` is ignored; defaults to `false` * **`includeArchived`** (boolean) - Optional - Include archived entries and assets in the export; defaults to `false` * **`skipContentModel`** (boolean) - Optional - Skip exporting content types, locales, and editor interfaces; defaults to `false` * **`skipEditorInterfaces`** (boolean) - Optional - Skip exporting editor interfaces (content model metadata); defaults to `false` * **`skipContent`** (boolean) - Optional - Skip exporting entries and assets (only export content model and configuration); defaults to `false` * **`skipRoles`** (boolean) - Optional - Skip exporting space roles and permissions. Only available from master environment; defaults to `false` * **`skipWebhooks`** (boolean) - Optional - Skip exporting webhooks. Only available from master environment; defaults to `false` * **`skipTags`** (boolean) - Optional - Skip exporting tags (CMA-only feature, not available via CDA); defaults to `false` * **`stripTags`** (boolean) - Optional - Remove all tags from exported entries and assets; defaults to `false` * **`contentOnly`** (boolean) - Optional - Shorthand to set `skipRoles`, `skipContentModel`, and `skipWebhooks` to true. Exports only entries and assets; defaults to `false` * **`queryEntries`** (string[]) - Optional - Array of Contentful API query parameters to filter entries. Example: `['content_type=myType', 'sys.id=myId']` * **`queryAssets`** (string[]) - Optional - Array of Contentful API query parameters to filter assets. Same syntax as `queryEntries` * **`downloadAssets`** (boolean) - Optional - Download asset binary files to the local filesystem under `exportDir//`. Required for embargoed (signed URL) assets; defaults to `false` * **`maxAllowedLimit`** (number) - Optional - Maximum items per API page request. Reduce if receiving "Response size too big" errors from the API; defaults to `1000` * **`host`** (string) - Optional - CMA API host; defaults to `'api.contentful.com'` * **`hostDelivery`** (string) - Optional - CDA API host; defaults to `'cdn.contentful.com'` * **`proxy`** (string) - Optional - HTTP proxy configuration in format `host:port` or `user:password@host:port` * **`rawProxy`** (boolean) - Optional - If true, pass proxy config directly to Axios instead of creating a custom httpsAgent. Useful when custom proxy handling is needed; defaults to `false` * **`headers`** (object) - Optional - Additional HTTP headers to attach to all API requests * **`errorLogFile`** (string) - Optional - Path where errors encountered during export are logged; defaults to `contentful-export-error-log-{spaceId}-{environmentId}-{timestamp}.json` * **`useVerboseRenderer`** (boolean) - Optional - Display progress line-by-line instead of a spinner. Useful for CI/CD environments; defaults to `false` * **`config`** (string) - Optional - Path to a JSON config file containing any of the above options. Command-line options override file config ### Response * **`Promise>`** - A promise that resolves with an object containing the exported data, categorized by `ContentfulExportField`. ``` -------------------------------- ### downloadAssets Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/module-overview.md Downloads asset binary files to the local filesystem with support for concurrent downloads and embargoed asset signing. ```APIDOC ## downloadAssets ### Description Downloads asset binary files to the local filesystem. This function supports concurrent downloads, embargoed asset signing using JWT, directory organization by host, and detailed progress reporting. ### Signature ```javascript export default function downloadAssets(options: DownloadOptions): TaskExecutor ``` ### Returns A task executor function compatible with Listr. ### Parameters #### Request Body - **options** (DownloadOptions) - Required - Configuration object for downloading assets. ``` -------------------------------- ### Raw Proxy Configuration Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md Enables passing proxy configuration directly to axios by setting rawProxy to true. Use this for custom proxy handling or when the default agent approach fails. ```javascript { proxy: 'proxy.example.com:8080', rawProxy: true } ``` -------------------------------- ### Download Assets with Listr Task Executor Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/module-overview.md Returns a task executor function compatible with Listr for downloading asset binary files. Supports concurrent downloads, embargoed asset signing, and detailed progress reporting. Use this for downloading assets locally. ```javascript const task = downloadAssets(options) task(ctx, { output: 'status message' }) ``` -------------------------------- ### Fast Export Options Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Utilize these flags to speed up the export process by skipping non-essential data. Options like `skipContentModel`, `skipWebhooks`, `skipRoles`, and `contentOnly` reduce the amount of data exported. ```javascript { skipContentModel: true, // Skip non-essential data skipWebhooks: true, skipRoles: true, contentOnly: true // Use shorthand } ``` -------------------------------- ### Run Contentful Export Function Signature Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/main-export-function.md This is the main function signature for initiating a Contentful export. It takes an `Options` object and returns a Promise that resolves with the exported data. ```javascript function runContentfulExport(params: Options): Promise> ``` -------------------------------- ### Internal Client Initialization in Export Task Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/init-client.md This snippet shows how the `initClient` function is used internally within the export task to initialize both CMA and CDA clients based on provided options. ```javascript // lib/index.js: Initialize client task ctx.client = initClient(options) // CMA client if (options.deliveryToken && !options.includeDrafts) { ctx.cdaClient = initClient(options, true) // CDA client for public content } ``` -------------------------------- ### Configure Proxy with rawProxy Option Source: https://github.com/contentful/contentful-export/blob/main/README.md Use the `rawProxy` option set to `true` when encountering issues connecting to Contentful through a proxy. ```javascript contentfulExport({ proxy: 'https://cat:dog@example.com:1234', rawProxy: true, ... }) ``` -------------------------------- ### Importing Types and Contentful Export Function Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/types.md Import the `Options` type for configuration and the `contentfulExport` function. This snippet shows how to define a configuration object and use the export function, with a note on the expected result type. ```typescript import type { Options } from 'contentful-export' import contentfulExport from 'contentful-export' async function exportSpace(config: Options) { const result = await contentfulExport(config) // result type: Record if (result.entries) { console.log(`Exported ${result.entries.length} entries`) } } ``` -------------------------------- ### Handle Null, Undefined, or Empty Input Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/headers.md Returns an empty object when the input is null, undefined, or an empty string. This ensures predictable behavior for missing header configurations. ```javascript getHeadersConfig(null) // => {} getHeadersConfig(undefined) // => {} getHeadersConfig('') // => {} ``` -------------------------------- ### Integration with Main Export Task Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md Shows how the asset download task is conditionally included in the main export pipeline. It skips if the downloadAssets option is false or if no assets were fetched. ```javascript { title: 'Download assets', task: wrapTask(downloadAssets(options)), skip: (ctx) => !options.downloadAssets || !Object.prototype.hasOwnProperty.call(ctx.data, 'assets') } ``` -------------------------------- ### Run Single Unit Test with Jest Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Use this command to run a specific unit test file. Replace the path with the target test file. ```bash npx jest --testPathPattern=test/unit/tasks/init-client ``` -------------------------------- ### Internal Function to Fetch Full Source Space Signature Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md This internal function uses Listr to manage the fetching of all space data, including handling pagination. ```javascript getFullSourceSpace(config: FetchConfig): Listr ``` -------------------------------- ### Download with Custom Timeout Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md Sets a custom timeout for asset downloads to handle slow network connections. The timeout is specified in milliseconds. ```javascript const result = await contentfulExport({ spaceId: 'my-space', managementToken: 'token', downloadAssets: true, exportDir: './assets', timeout: 30000 // 30 second timeout for slow connections }) ``` -------------------------------- ### Integration with Export Options Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/headers.md Shows how getHeadersConfig is used internally to parse CLI header arguments into a format suitable for API requests, merging with other options. ```javascript // In lib/parseOptions.js const options = { ...defaultOptions, ...configFile, ...params, headers: addSequenceHeader(params.headers || getHeadersConfig(params.header)) } ``` -------------------------------- ### Internal Client Initialization Function Signature Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Initializes a Contentful API client, either for Content Delivery API (CDA) or Content Management API (CMA), based on the provided options. ```javascript initClient(opts: Options, useCda?: boolean): ContentfulClient ``` -------------------------------- ### Debug Environment Not Found Error Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/errors.md Shows how to debug an 'Environment not found' error by listing available environments and specifying the correct `environmentId`. ```javascript // Check available environments const space = await client.getSpace(spaceId) const environments = await space.getEnvironments() console.log('Available environments:', environments.items.map(e => e.sys.id)) // Use correct environment ID { environmentId: 'staging' } ``` -------------------------------- ### Programmatic Import Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Shows how to import the contentful-export library for programmatic use within your JavaScript projects. This is the recommended way to use the tool. ```javascript import contentfulExport from 'contentful-export' ``` -------------------------------- ### Debug Space Not Found Error Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/errors.md Illustrates how to debug a 'Space not found' error by verifying the `spaceId` and attempting to access the space directly using the Contentful client. ```javascript // Verify space ID console.log('Using spaceId:', options.spaceId) // Try accessing space directly const client = createClient({ accessToken: token }) const space = await client.getSpace(spaceId) ``` -------------------------------- ### Run Unit Tests in Watch Mode Source: https://github.com/contentful/contentful-export/blob/main/CONTRIBUTING.md Run unit tests continuously in watch mode, automatically re-running tests on file changes. ```bash npm run test:unit:watch ``` -------------------------------- ### Download Assets Function Signature Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md This is the signature for the `downloadAssets` function. It returns a task executor compatible with the Listr task runner. ```javascript function downloadAssets(options: DownloadOptions): (ctx: Context, task: Task) => Promise ``` -------------------------------- ### Downloading a Single Asset Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/download-assets.md This is the core function call for downloading an individual asset. It utilizes a helper function `downloadAsset` which handles URL parsing, file path creation, directory creation, and streaming the download. ```javascript await downloadAsset({ url: fileUrl, directory: options.exportDir, httpClient: httpClient }) ``` -------------------------------- ### Contentful Export Task Pipeline Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/module-overview.md Illustrates the sequence of operations performed by the Contentful Export library. This pipeline includes client initialization, data fetching, asset downloading, and log file writing. ```text 1. Initialize client ├── Create CMA client └── Create CDA client (if deliveryToken provided) 2. Fetching data from space ├── Connect to space and environment ├── Fetch content types ├── Fetch tags ├── Fetch editor interfaces ├── Fetch entries ├── Fetch assets ├── Fetch locales ├── Fetch webhooks └── Fetch roles 3. Download assets (optional) └── Stream files to exportDir 4. Write export log file (if saveFile: true) ├── Check directory exists ├── Create directory └── Write JSON to disk ``` -------------------------------- ### Export from Specific Environment Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Exports content from a specified Contentful environment, such as 'staging' or 'development'. ```javascript const result = await contentfulExport({ spaceId: 'my-space-id', managementToken: 'token', environmentId: 'staging' }) ``` -------------------------------- ### Deprecated CLI Usage Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/quick-reference.md Shows the old, deprecated command for exporting content using the CLI. It is recommended to use the new `contentful-cli` command instead. ```bash # Old way (deprecated) npx contentful-export --space-id my-space --management-token abc123 ``` -------------------------------- ### Initialize CDA Client for Published Content Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/init-client.md Initialize a Contentful Delivery API client to access only published content. This is suitable for read-only exports and requires `spaceId`, `deliveryToken`, and optionally `environmentId` and `hostDelivery`. ```javascript const options = { spaceId: 'my-space', managementToken: 'management-token-xyz', deliveryToken: 'delivery-token-abc', environmentId: 'master', hostDelivery: 'cdn.contentful.com' } const cdaClient = initClient(options, true) const entries = await cdaClient.withAllLocales.getEntries() ``` -------------------------------- ### HTTP Proxy Configuration (Authenticated) Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/configuration.md Configures an authenticated HTTP proxy for API requests using the 'user:password@host:port' format. ```javascript { proxy: 'proxyuser:proxypass@proxy.company.com:3128' } ``` -------------------------------- ### Test Directory Write Permissions Before Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/errors.md Before initiating an export, verify if the specified export directory is writable to prevent file system errors. This check uses Node.js 'fs' module to test access. ```javascript const fs = require('fs') // Test directory before export try { fs.accessSync(options.exportDir, fs.constants.W_OK) console.log('Directory is writable') } catch (err) { console.error('Cannot write to directory:', err.message) } ``` -------------------------------- ### Basic Contentful Export Source: https://github.com/contentful/contentful-export/blob/main/_autodocs/api-reference/get-space-data.md Perform a basic export of all content from a Contentful space. This internally uses `getFullSourceSpace` to fetch all data. ```javascript import contentfulExport from 'contentful-export' const result = await contentfulExport({ spaceId: 'my-space', managementToken: 'token' }) // Internally uses getFullSourceSpace to fetch all data ```