### Develop Static Page Locally Source: https://github.com/hitmasu/opencnpj/blob/main/README.md Install dependencies and start a local development server for the static page using npm. ```bash npm install ``` ```bash npm run dev ``` -------------------------------- ### Run Pipeline Command Examples Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/configuration.md Demonstrates various ways to run the pipeline command with different options. These examples show how to specify a month, enable cleanup, use a custom release ID, or skip ZIP generation. ```bash # Processa mês atual ou mais recente disponível dotnet run pipeline # Processa março de 2024 especificamente dotnet run pipeline -m 2024-03 # Processa com limpeza automática de artefatos dotnet run pipeline --cleanup-on-success # Processa com release ID customizado dotnet run pipeline --release-id v20240604-custom # Pula geração de ZIP para teste rápido dotnet run pipeline --skip-zip ``` -------------------------------- ### API Request Examples for Datasets Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/response-schema.md These examples demonstrate how to request specific datasets from the API using query parameters. You can specify single or multiple datasets. ```http GET /{cnpj}?datasets=receita,cno,rntrc ``` ```http GET /{cnpj}?dataset=receita&dataset=cno ``` ```http GET /{cnpj}?datasets=receita%20cno%20rntrc ``` -------------------------------- ### API Info Response Example Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md This JSON object provides metadata about the active release of the API, including storage release IDs and details about the ZIP archives for each dataset. ```json { "storage_release_id": "abc123def456", "datasets": { "cno": { "json_property_name": "cno", "storage_release_id": "abc123def456", "zip_available": true, "zip_size": 25000000, "zip_url": "https://r2.opencnpj.org/files/releases/cno/data.zip", "zip_md5checksum": "5d41402abc4b2a76b9719d911017c592" }, "rntrc": { "json_property_name": "rntrc", "storage_release_id": "abc123def456", "zip_available": true, "zip_size": 5000000, "zip_url": "https://r2.opencnpj.org/files/releases/rntrc/data.zip", "zip_md5checksum": "6d41402abc4b2a76b9719d911017c593" } } } ``` -------------------------------- ### GET /info Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md Returns metadata about the active release, including storage_release_id and ZIP information for each dataset. ```APIDOC ## GET /info ### Description Returns metadata about the active release, including storage_release_id and ZIP information for each dataset. The response is cached with `no-store` to ensure it's always fresh. ### Method GET ### Endpoint `/info` ### Response #### Success Response (200 OK) **Content-Type:** `application/json; charset=utf-8` **Cache:** `no-store` ```json { "storage_release_id": "abc123def456", "datasets": { "cno": { "json_property_name": "cno", "storage_release_id": "abc123def456", "zip_available": true, "zip_size": 25000000, "zip_url": "https://r2.opencnpj.org/files/releases/cno/data.zip", "zip_md5checksum": "5d41402abc4b2a76b9719d911017c592" }, "rntrc": { "json_property_name": "rntrc", "storage_release_id": "abc123def456", "zip_available": true, "zip_size": 5000000, "zip_url": "https://r2.opencnpj.org/files/releases/rntrc/data.zip", "zip_md5checksum": "6d41402abc4b2a76b9719d911017c593" } } } ``` ### Error Handling - **502 "info load failed"**: Failed to load info.json from R2 or assets. ``` -------------------------------- ### Example 2: Valid Explicit Datasets Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Demonstrates requesting valid datasets like 'cno'. The flow shows how the system parses the request, identifies available datasets, and returns the selected module keys. ```http GET /41132876000179?datasets=cno RuntimeInfo: { datasets: { "cno": {...}, "rntrc": {...} } } Fluxo: 1. parseRequestedDatasetKeys(params) → ["cno"] 2. availableModuleKeys = ["cno", "rntrc"] 3. available = Set(["receita", "cno", "rntrc"]) 4. unknown = [] (nenhum desconhecido) 5. includeReceita = "cno" === "receita"? Não → false 6. moduleKeys = ["cno"].filter(k => k !== "receita") → ["cno"] 7. Retornar { includeReceita: false, moduleKeys: ["cno"], cacheKey: "cno" } ``` -------------------------------- ### Example 4: Multiple Parameters with Natural Connectors Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Demonstrates requesting multiple datasets using different query parameters like 'datasets' and 'dataset'. The flow details how the system parses, deduplicates, and resolves these requests. ```http GET /41132876000179?datasets=receita,cno&dataset=rntrc Fluxo: 1. parseRequestedDatasetKeys(params) ├─ getAll("datasets") → ["receita,cno"] ├─ getAll("dataset") → ["rntrc"] ├─ Dividir por /[, ]/ ├─ Deduplicar → ["receita", "cno", "rntrc"] └─ Retorna ["receita", "cno", "rntrc"] 2. available = Set(["receita", "cno", "rntrc"]) 3. unknown = [] (todos válidos) 4. includeReceita = "receita" in requested? Sim → true 5. moduleKeys = ["cno", "rntrc"] (ordena: cno, rntrc) 6. Retornar { includeReceita: true, moduleKeys: ["cno", "rntrc"], cacheKey: "receita,cno,rntrc" } ``` -------------------------------- ### Worker Data Contract Example Source: https://github.com/hitmasu/opencnpj/blob/main/src/Worker/README.md An example of the expected JSON data format for a shard record. This format is assumed by the worker's runtime contract. ```json { "cnpj":"12345678000195","...":"..." } ``` -------------------------------- ### fetch Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-main-entry.md Função assíncrona que processa requisições HTTP GET e OPTIONS. Ela lida com CORS, retorna metadados de release, esquemas JSON, e dados de CNPJ. ```APIDOC ## fetch ### Description Função assíncrona que processa requisições HTTP GET e OPTIONS. Ela lida com CORS, retorna metadados de release, esquemas JSON, e dados de CNPJ. ### Method GET, OPTIONS ### Endpoint / ### Parameters #### Path Parameters - **cnpj** (string) - Opcional - Número do CNPJ a ser consultado. #### Query Parameters Nenhum #### Request Body Nenhum ### Request Example ```http OPTIONS / GET /info GET /schema GET /41132876000179 GET /41.132.876/0001-79 POST / ``` ### Response #### Success Response (200) - **body** (JSON) - Dados do CNPJ, metadados de release ou esquema JSON. #### Success Response (204) - **body** (None) - Resposta para requisições OPTIONS bem-sucedidas. #### Error Response (400) - **body** (JSON) - `{ "error": "invalid cnpj" }` - Retornado quando o CNPJ ou dataset é inválido. #### Error Response (404) - **body** (JSON) - Retornado quando o CNPJ não é encontrado. #### Error Response (405) - **body** (JSON) - `{ "error": "method not allowed" }` - Retornado quando o método HTTP não é permitido. #### Error Response (502) - **body** (JSON) - Retornado quando há um erro ao processar o shard. ``` -------------------------------- ### Example of using createStageError Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/errors.md Demonstrates how to use the `createStageError` function within a try-catch block to wrap parsing errors with contextual information. ```typescript try { const parsed = JSON.parse(text); } catch (error) { throw createStageError("info.json.parse", error); } ``` -------------------------------- ### Example 3: Invalid Dataset Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Shows how the API handles requests for invalid datasets, such as 'foo'. The flow illustrates the detection of unknown datasets and the error response returned. ```http GET /41132876000179?datasets=foo RuntimeInfo: { datasets: { "cno": {...}, "rntrc": {...} } } Fluxo: 1. parseRequestedDatasetKeys(params) → ["foo"] 2. available = Set(["receita", "cno", "rntrc"]) 3. unknown = ["foo"].filter(k => !available.has(k)) → ["foo"] 4. unknown.length > 0? Sim 5. Retornar { ok: false, error: "invalid dataset: foo" } ``` -------------------------------- ### CNPJ Data Response Example Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md This JSON object represents a successful response when querying a specific CNPJ. It includes details from the Receita Federal and other available datasets. ```json { "cnpj": "41132876000179", "razao_social": "EMPRESA LTDA", "nome_fantasia": "EMPRESA", "situacao_cadastral": "Ativa", "data_situacao_cadastral": "2024-05-09", "matriz_filial": "Matriz", "data_inicio_atividade": "2021-03-08", "cnae_principal": "2330301", "cnaes_secundarios": ["2330302"], "natureza_juridica": "Sociedade Limitada", "tipo_logradouro": "RUA", "logradouro": "PRINCIPAL", "numero": "1000", "complemento": "SALA 100", "bairro": "CENTRO", "cep": "01310100", "uf": "SP", "municipio": "SAO PAULO", "email": "contato@empresa.com.br", "telefones": [ { "ddd": "11", "numero": "30001234", "is_fax": false } ], "capital_social": "100000,00", "porte_empresa": "Empresa de Pequeno Porte (EPP)", "opcao_simples": "S", "data_opcao_simples": "2019-06-15", "opcao_mei": "N", "data_opcao_mei": "", "QSA": [ { "nome_socio": "JOAO SILVA", "cnpj_cpf_socio": "***339234**", "qualificacao_socio": "Administrador", "data_entrada_sociedade": "2022-12-29", "identificador_socio": "Pessoa Física", "faixa_etaria": "31 a 40 anos" } ], "cno": null, "rntrc": null } ``` -------------------------------- ### GET /{cnpj} Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md Queries specific CNPJ data, returning information from the Receita Federal and optional modules. Accepts CNPJ with or without mask. ```APIDOC ## GET /{cnpj} ### Description Queries specific CNPJ data, returning information from the Receita Federal and optional modules. Accepts CNPJ with or without mask. ### Method GET ### Endpoint `/{cnpj}` ### Parameters #### Path Parameters - **cnpj** (string) - Required - CNPJ with 14 characters. Accepts masked format (`XX.XXX.XXX/XXXX-XX`), internally normalized to alphanumeric. Pattern: `[A-Z0-9]{12}\d{2}` #### Query Parameters - **datasets** (string) - Optional - Defaults to "receita,cno,rntrc" - List of datasets to include, separated by comma or space. Valid values: `receita`, `cno`, `rntrc` - **dataset** (string) - Optional - Alias for `datasets` (both accumulate) ### Request Example ```bash # Consult with Receita data (default) curl "https://api.opencnpj.org/41132876000179" # With CNPJ mask curl "https://api.opencnpj.org/41.132.876/0001-79" # Selecting specific datasets curl "https://api.opencnpj.org/41132876000179?datasets=receita,cno" # Only CNO module curl "https://api.opencnpj.org/41132876000179?datasets=cno" # Multiple dataset parameters curl "https://api.opencnpj.org/41132876000179?datasets=receita&dataset=cno&dataset=rntrc" ``` ### Response #### Success Response (200 OK) - **cnpj** (string) - CNPJ number. - **razao_social** (string) - Company name. - **nome_fantasia** (string) - Trade name. - **situacao_cadastral** (string) - Registration status. - **data_situacao_cadastral** (string) - Date of registration status change. - **matriz_filial** (string) - Indicates if it's a headquarters or branch. - **data_inicio_atividade** (string) - Date of commencement of activities. - **cnae_principal** (string) - Primary CNAE code. - **cnaes_secundarios** (array) - Array of secondary CNAE codes. - **natureza_juridica** (string) - Legal nature of the company. - **tipo_logradouro** (string) - Type of street. - **logradouro** (string) - Street name. - **numero** (string) - Street number. - **complemento** (string) - Complementary address information. - **bairro** (string) - Neighborhood. - **cep** (string) - Postal code. - **uf** (string) - State (Federative Unit). - **municipio** (string) - City. - **email** (string) - Contact email. - **telefones** (array) - Array of phone numbers. - **capital_social** (string) - Stated capital. - **porte_empresa** (string) - Company size. - **opcao_simples** (string) - Indicates if the company opted for the simplified tax regime ('S' or 'N'). - **data_opcao_simples** (string) - Date of opting for the simplified regime. - **opcao_mei** (string) - Indicates if the company opted for MEI ('S' or 'N'). - **data_opcao_mei** (string) - Date of opting for MEI. - **QSA** (array) - List of partners or administrators. - **cno** (any) - CNO data, if requested and available. - **rntrc** (any) - RNTRC data, if requested and available. #### Response Example ```json { "cnpj": "41132876000179", "razao_social": "EMPRESA LTDA", "nome_fantasia": "EMPRESA", "situacao_cadastral": "Ativa", "data_situacao_cadastral": "2024-05-09", "matriz_filial": "Matriz", "data_inicio_atividade": "2021-03-08", "cnae_principal": "2330301", "cnaes_secundarios": ["2330302"], "natureza_juridica": "Sociedade Limitada", "tipo_logradouro": "RUA", "logradouro": "PRINCIPAL", "numero": "1000", "complemento": "SALA 100", "bairro": "CENTRO", "cep": "01310100", "uf": "SP", "municipio": "SAO PAULO", "email": "contato@empresa.com.br", "telefones": [ { "ddd": "11", "numero": "30001234", "is_fax": false } ], "capital_social": "100000,00", "porte_empresa": "Empresa de Pequeno Porte (EPP)", "opcao_simples": "S", "data_opcao_simples": "2019-06-15", "opcao_mei": "N", "data_opcao_mei": "", "QSA": [ { "nome_socio": "JOAO SILVA", "cnpj_cpf_socio": "***339234**", "qualificacao_socio": "Administrador", "data_entrada_sociedade": "2022-12-29", "identificador_socio": "Pessoa Física", "faixa_etaria": "31 a 40 anos" } ], "cno": null, "rntrc": null } ``` ### Error Handling - **400 "invalid cnpj"**: Invalid or unnormalized CNPJ format. - **400 "invalid dataset: {dataset}"**: One or more requested datasets do not exist. - **404 "not found"**: CNPJ not found in published shards. - **502 "invalid shard payload"**: Error processing shard data (corruption or invalid format). ``` -------------------------------- ### Query CNPJ Data with cURL Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md Examples of how to query CNPJ data using cURL. You can query with a standard CNPJ, a masked CNPJ, or specify which datasets to include in the response. ```bash # Consultar com dados da Receita (padrão) curl "https://api.opencnpj.org/41132876000179" ``` ```bash # Com máscara de CNPJ curl "https://api.opencnpj.org/41.132.876/0001-79" ``` ```bash # Selecionando datasets específicos curl "https://api.opencnpj.org/41132876000179?datasets=receita,cno" ``` ```bash # Apenas módulo CNO curl "https://api.opencnpj.org/41132876000179?datasets=cno" ``` ```bash # Múltiplos parâmetros dataset curl "https://api.opencnpj.org/41132876000179?datasets=receita&dataset=cno&dataset=rntrc" ``` -------------------------------- ### Handler Principal do Worker Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-main-entry.md Função assíncrona que processa requisições HTTP GET e OPTIONS. Retorna CORS pré-voo para OPTIONS, 405 para outros métodos, e processa caminhos específicos como /info, /schema, ou dados de CNPJ. ```typescript export default { async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise { // Implementação... } }; ``` -------------------------------- ### GET /schema Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md Returns the complete JSON Schema for a CNPJ response, matching the structure of the `/{cnpj}` endpoint. ```APIDOC ## GET /schema ### Description Returns the complete JSON Schema for a CNPJ response, matching the structure of the `/{cnpj}` endpoint. ### Method GET ### Endpoint `/schema` ``` -------------------------------- ### Build Static Page Source: https://github.com/hitmasu/opencnpj/blob/main/README.md Generate the static version of the page for deployment. ```bash npm run build ``` -------------------------------- ### Deploy Worker and Assets Source: https://github.com/hitmasu/opencnpj/blob/main/README.md Orchestrate the release process, including running the ETL, copying worker assets, testing the worker, deploying with Wrangler, and validating the deployment. ```bash src/script/deploy.sh ``` -------------------------------- ### Best Practice: Use Cache Key for HTTP Caching Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Demonstrates how to generate a cache key using `buildRecordCacheKey` that incorporates dataset selection details. This is crucial for effective HTTP caching, especially when dataset selections vary. ```typescript const selection = resolveDatasetSelection(params, runtimeInfo); if (selection.ok) { const cacheKey = buildRecordCacheKey(cnpj, prefix, runtimeInfo, selection.value); // Cache key inclui informações da seleção para variações } ``` -------------------------------- ### Layered CNPJ Validation Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-cnpj.md Demonstrates a multi-layered approach to CNPJ validation, starting with path extraction, followed by dataset selection, and finally data loading. ```typescript // Layer 1: Path extraction const cnpj = extractCnpjFromPath(pathname); if (!cnpj) return 400; // Layer 2: Dataset validation const selection = resolveDatasetSelection(searchParams, runtimeInfo); if (!selection.ok) return 400; // Layer 3: Data loading const record = await loadDatasetsFromShard(...); if (!record) return 404; ``` -------------------------------- ### Tipo para Informações de Runtime Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md Contém metadados de um release ativo publicado no R2 ou assets, incluindo o ID do release de armazenamento e informações sobre datasets. ```typescript export type RuntimeInfo = { storage_release_id?: string; datasets?: Record; }; ``` -------------------------------- ### Format Metadata: Date, Size, and Count Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/page-utilities.md Demonstrates formatting various metadata fields including dates, file sizes in bytes, and record counts using utility functions. ```typescript const dataset = { updated_at: "2026-04-14T12:30:00Z", zip_size: 25500000, record_count: 1234567 }; console.log(`Atualizado: ${formatDate(dataset.updated_at)}`); // "Atualizado: 14/04/2026 12:30" console.log(`Tamanho: ${formatBytes(dataset.zip_size)}`); // "Tamanho: 24,3 MB" console.log(`Registros: ${formatCount(dataset.record_count)}`); // "Registros: 1.234.567" ``` -------------------------------- ### loadRuntimeInfo Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Loads RuntimeInfo (info.json) as an object with in-memory caching. It checks embedded sources, then cache, then R2, then assets, storing the result in cache with a 60-second TTL. ```APIDOC ## loadRuntimeInfo ### Description Loads RuntimeInfo (info.json) as an object, utilizing in-memory caching. It first checks embedded sources, then a hot cache, then R2, and finally assets. The loaded information is stored in the hot cache with a Time-To-Live (TTL) of 60 seconds. ### Method (Not explicitly defined, but implies a data retrieval operation) ### Endpoint (Not explicitly defined) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```typescript const info = await loadRuntimeInfo(env); if (info) { console.log(info.storage_release_id); // "abc123" } ``` ### Response #### Success Response - **RuntimeInfo** (object) - An object containing runtime metadata, including `storage_release_id` and `datasets`. #### Null Response - **null** - Returned if RuntimeInfo cannot be loaded from any source. #### Response Example ```json { "storage_release_id": "abc123", "datasets": {} } ``` ``` -------------------------------- ### Get Company Information Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/response-schema.md Retrieves comprehensive company information, optionally including data from CNO (Cadastro Nacional de Obras) and RNTRC (Registro Nacional de Transportadores Rodoviários de Cargas). ```APIDOC ## GET /{cnpj} ### Description Retrieves company information based on the provided CNPJ. You can specify which datasets to include in the response using the `datasets` query parameter. ### Method GET ### Endpoint `/{cnpj}` ### Parameters #### Query Parameters - **datasets** (string) - Optional - A comma-separated list of datasets to include. Valid datasets are `receita`, `cno`, and `rntrc`. If not provided, all available datasets are returned by default. ### Request Example ``` GET /{cnpj}?datasets=receita,cno,rntrc ``` ### Response #### Success Response (200) Returns a JSON object containing company data. The structure of the response varies based on the requested datasets. See examples for detailed response structures. #### Response Example (receita, cno, rntrc) ```json { "cnpj": "41132876000179", "razao_social": "EMPRESA LTDA", "nome_fantasia": "EMPRESA", "situacao_cadastral": "Ativa", "data_situacao_cadastral": "2024-05-09", "matriz_filial": "Matriz", "data_inicio_atividade": "2021-03-08", "cnae_principal": "2330301", "cnaes_secundarios": ["2330302", "2330303"], "natureza_juridica": "Sociedade Limitada", "tipo_logradouro": "RUA", "logradouro": "PRINCIPAL", "numero": "1000", "complemento": "SALA 100", "bairro": "CENTRO", "cep": "01310100", "uf": "SP", "municipio": "SAO PAULO", "email": "contato@empresa.com.br", "telefones": [ {"ddd": "11", "numero": "30001234", "is_fax": false}, {"ddd": "11", "numero": "30005678", "is_fax": true} ], "capital_social": "100000,00", "porte_empresa": "Empresa de Pequeno Porte (EPP)", "opcao_simples": "S", "data_opcao_simples": "2019-06-15", "opcao_mei": "N", "data_opcao_mei": "", "QSA": [ { "nome_socio": "JOAO SILVA", "cnpj_cpf_socio": "***339234**", "qualificacao_socio": "Administrador", "data_entrada_sociedade": "2022-12-29", "identificador_socio": "Pessoa Física", "faixa_etaria": "31 a 40 anos" } ], "cno": { "updated_at": "2026-04-14T12:00:00Z", "obras": [ { "cno": "2024000001", "nome": "Obra Comercial", "situacao": {"codigo": "01", "descricao": "Ativa"}, "data_inicio": "2024-01-15", "municipio": "SAO PAULO", "uf": "SP", "area_total": "5000.50" } ] }, "rntrc": { "updated_at": "2026-04-14T12:00:00Z", "numero_rntrc": "123456789", "nome": "TRANSPORTADORA X", "categoria": "ETC", "situacao": "ATIVO", "municipio": "SAO PAULO", "uf": "SP", "equiparado": false } } ``` #### Response Example (receita only) ```json { "cnpj": "12345678000190", "razao_social": "EMPRESA Y LTDA", "nome_fantasia": "", "situacao_cadastral": "Ativa", "data_situacao_cadastral": "2023-10-20", "matriz_filial": "Filial", "data_inicio_atividade": "2020-06-01", "cnae_principal": "4711302", "cnaes_secundarios": [], "natureza_juridica": "Sociedade Limitada", "tipo_logradouro": "AV", "logradouro": "PAULISTA", "numero": "1000", "complemento": "", "bairro": "BELA VISTA", "cep": "01311100", "uf": "SP", "municipio": "SAO PAULO", "email": "vendas@empresay.com.br", "telefones": [ {"ddd": "11", "numero": "30005555", "is_fax": false} ], "capital_social": "50000,00", "porte_empresa": "Microempresa (ME)", "opcao_simples": "N", "data_opcao_simples": "", "opcao_mei": "S", "data_opcao_mei": "2023-01-01", "QSA": [], "cno": null, "rntrc": null } ``` ``` -------------------------------- ### Best Practice: Document Available Datasets Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Explains that the info endpoint should reflect the actual available datasets. This information is vital for callers to construct valid dataset queries. ```typescript // Info endpoint deve refletir datasets reais const info = await loadInfo(env); // Contém todos os datasets em info.datasets // Chamador pode usar isso para construir queries const available = Object.keys(info.datasets || {}); // ["cno", "rntrc"] (não inclui "receita" base) ``` -------------------------------- ### Load Binary Index from Assets Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Loads a binary index from static assets with in-memory caching. The cache has a TTL of 30 minutes and a maximum of 256 entries. ```typescript async function loadBinaryIndexFromAssets( env: Env, assetPath: string, ): Promise ``` -------------------------------- ### Tipo para Seleção de Dataset Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md Especifica quais datasets incluir na resposta de uma consulta de CNPJ, incluindo a opção de incluir dados da Receita e uma chave de cache. ```typescript export type DatasetSelection = { includeReceita: boolean; moduleKeys: string[]; cacheKey: string; }; ``` -------------------------------- ### Load Application Configuration Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/configuration.md Loads the application configuration from a JSON file. Supports loading from the current directory or a custom path. The globally loaded configuration can be accessed via AppConfig.Current. ```csharp // Carrega config.json do diretório atual var config = AppConfig.Load(); // Carrega de caminho personalizado var config = AppConfig.Load("/path/to/custom/config.json"); // Acessa configuração carregada globalmente var currentConfig = AppConfig.Current; ``` -------------------------------- ### Run ETL Pipeline Source: https://github.com/hitmasu/opencnpj/blob/main/README.md Execute the ETL pipeline to download, process, and publish CNPJ data. The pipeline can target a specific month or release ID, and optionally clean up local artifacts upon success. ```bash dotnet run pipeline ``` ```bash dotnet run pipeline -m YYYY-MM ``` ```bash dotnet run pipeline --release-id abc123... ``` ```bash dotnet run pipeline --cleanup-on-success ``` -------------------------------- ### loadInfo Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Retrieves the `info.json` (RuntimeInfo) via an HTTP response. It attempts to load this information from embedded sources, R2, or assets, returning a 404 if unavailable. ```APIDOC ## loadInfo ### Description Returns `info.json` (RuntimeInfo) via HTTP response. It prioritizes embedded sources for testing, then R2, then assets. Returns a 404 status if the information cannot be found. ### Method (Implicitly HTTP GET, as it returns a Response) ### Endpoint (Not explicitly defined, but implies an endpoint to fetch `info.json`) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```typescript const response = await loadInfo(env); // Status 200: { "storage_release_id": "abc123", "datasets": {...} } ``` ### Response #### Success Response (200) - **storage_release_id** (string) - The release ID of the storage. - **datasets** (object) - Contains dataset information. #### Error Response (404) Returned if `info.json` cannot be found from any source. #### Response Example ```json { "storage_release_id": "abc123", "datasets": {} } ``` ``` -------------------------------- ### DuckDbSettings C# Class Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md Settings for configuring the DuckDB database, including in-memory usage, thread pragmas, memory limits, and write behavior. ```csharp public class DuckDbSettings { public bool UseInMemory { get; set; } // Usar banco em memória public int ThreadsPragma { get; set; } // Pragmas de threads public string MemoryLimit { get; set; } // Limite de memória (ex.: "4GB") public int EngineThreads { get; set; } // Threads de engine public bool PreserveInsertionOrder { get; set; } // Preservar ordem de inserção public int PartitionedWriteMaxOpenFiles { get; set; } // Máximo de arquivos abertos na escrita particionada } ``` -------------------------------- ### Parse Requested Dataset Keys - Accumulating Parameters Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Shows how dataset keys from both 'datasets' and 'dataset' query parameters are combined. ```typescript // Parâmetro: "datasets=cno&dataset=rntrc" parseRequestedDatasetKeys(new URLSearchParams("datasets=cno&dataset=rntrc")) // Retorna: ["cno", "rntrc"] (acumula ambos) ``` -------------------------------- ### Load Binary Index with Fallback Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Loads a binary shard index, prioritizing assets and falling back to R2 storage. It supports optional release IDs and a preference for asset-based indexes. ```typescript async function loadBinaryIndex( env: Env, prefix: string, releaseId?: string, preferAssetIndex = false, ): Promise ``` -------------------------------- ### Use Index Caching Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-binary-index.md Illustrates how to implement a hot index cache to avoid re-parsing index files. Cached indices are returned directly, improving performance for repeated lookups. ```typescript // Hot index cache evita reparser índices const cached = getHotIndex(key); if (cached) { return cached; } const index = parseBinaryShardIndex(buffer, path); rememberHotIndex(key, index); // Cachear para próximas buscas ``` -------------------------------- ### Run Worker Tests Source: https://github.com/hitmasu/opencnpj/blob/main/src/Worker/README.md Execute the test suite for the OpenCNPJ Worker. Ensure you are in the 'src/Worker' directory before running. ```bash npm test ``` -------------------------------- ### Load Published Info Response Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Fetches the 'info.json' (RuntimeInfo) as an HTTP response. It attempts to load from embedded info, R2, or assets, returning a 404 if unavailable. ```typescript const response = await loadInfo(env); // Status 200: { "storage_release_id": "abc123", "datasets": {...} } ``` -------------------------------- ### Resolve Dataset Selection - No Parameters Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Demonstrates the outcome when no dataset parameters are provided in the query, resulting in all available datasets being selected. ```typescript // GET /41132876000179 (sem query) resolveDatasetSelection(new URLSearchParams(), runtimeInfo) // Retorna: { ok: true, value: { includeReceita: true, moduleKeys: ["cno", "rntrc"], // Todos disponíveis cacheKey: "receita,cno,rntrc" } } ``` -------------------------------- ### Tipo para Chave de Dataset (Frontend) Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md Define os tipos de datasets disponíveis na interface do frontend como uma união de strings literais. ```typescript export type DatasetKey = 'receita' | 'cno' | 'rntrc'; ``` -------------------------------- ### Parse Requested Dataset Keys - With Ignored Tokens Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Demonstrates how natural language connectors like 'e' and 'and' are ignored during dataset key parsing. ```typescript // Parâmetro: "receita e cno" parseRequestedDatasetKeys(new URLSearchParams("datasets=receita e cno")) // Retorna: ["receita", "cno"] ("e" é ignorado) ``` -------------------------------- ### Resolve Dataset Selection - Valid Parameters Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Shows how valid dataset parameters in the query string are processed to select specific modules and control recipe inclusion. ```typescript // GET /41132876000179?datasets=cno,rntrc const params = new URLSearchParams("datasets=cno,rntrc"); resolveDatasetSelection(params, runtimeInfo) // Retorna: { ok: true, value: { includeReceita: false, moduleKeys: ["cno", "rntrc"], cacheKey: "cno,rntrc" } } ``` -------------------------------- ### Company Data with Only Basic Registration Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/response-schema.md This snippet illustrates a response containing only the basic company registration data. Use this when requesting only the 'receita' dataset. ```json { "cnpj": "12345678000190", "razao_social": "EMPRESA Y LTDA", "nome_fantasia": "", "situacao_cadastral": "Ativa", "data_situacao_cadastral": "2023-10-20", "matriz_filial": "Filial", "data_inicio_atividade": "2020-06-01", "cnae_principal": "4711302", "cnaes_secundarios": [], "natureza_juridica": "Sociedade Limitada", "tipo_logradouro": "AV", "logradouro": "PAULISTA", "numero": "1000", "complemento": "", "bairro": "BELA VISTA", "cep": "01311100", "uf": "SP", "municipio": "SAO PAULO", "email": "vendas@empresay.com.br", "telefones": [ {"ddd": "11", "numero": "30005555", "is_fax": false} ], "capital_social": "50000,00", "porte_empresa": "Microempresa (ME)", "opcao_simples": "N", "data_opcao_simples": "", "opcao_mei": "S", "data_opcao_mei": "2023-01-01", "QSA": [], "cno": null, "rntrc": null } ``` -------------------------------- ### AppConfig C# Class Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md The root configuration object for the ETL application. It aggregates settings for various components like paths, rclone, DuckDB, shards, and integrations. ```csharp public class AppConfig { public PathsConfig Paths { get; set; } = new(); public RcloneSettings Rclone { get; set; } = new(); public DuckDbSettings DuckDb { get; set; } = new(); public ShardSettings Shards { get; set; } = new(); public DownloaderSettings Downloader { get; set; } = new(); public CnoIntegrationSettings CnoIntegration { get; set; } = new(); public RntrcIntegrationSettings RntrcIntegration { get; set; } = new(); } ``` -------------------------------- ### loadRecordFromShard Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Loads all available data for a given CNPJ from a specified R2 bucket shard. It retrieves both the main Receita data and any available module data. ```APIDOC ## loadRecordFromShard ### Description Loads all available data for a specific CNPJ from a given R2 bucket shard. This includes data from Receita and all other available modules associated with the CNPJ. ### Method (Not explicitly defined, but implies a data retrieval operation) ### Endpoint (Not explicitly defined) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None #### Parameters - **env** (Env) - Required - The environment configuration. - **bucket** (R2Bucket) - Required - The R2 bucket instance. - **prefix** (string) - Required - The shard prefix, typically the first 3 characters of the CNPJ. - **cnpj** (string) - Required - The normalized 14-character CNPJ. - **runtimeInfo** (RuntimeInfo | null) - Optional - Runtime metadata; will be loaded if omitted. ### Request Example ```typescript const record = await loadRecordFromShard(env, bucket, "411", "41132876000179"); // { // "cnpj": "41132876000179", // "razao_social": "...", // "cno": null, // "rntrc": {...} // } ``` ### Response #### Success Response - **Record** (object) - An object containing the CNPJ's data, including Receita and module information. #### Null Response - **null** - Returned if the CNPJ record cannot be found or loaded. #### Response Example ```json { "cnpj": "41132876000179", "razao_social": "Example Company Name", "cno": null, "rntrc": { "some_field": "some_value" } } ``` ``` -------------------------------- ### Handle Info Load Failures Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/errors.md Handles potential failures when loading the 'info.json' file, which is critical for runtime information. If loading fails from R2 or assets, it logs the error and returns a 502 response. ```typescript if (pathname === "/info") { try { return await loadInfo(env); } catch (error) { console.error("info load failed", error); return jsonError(502, "info load failed"); } } ``` -------------------------------- ### Best Practice: Pass RuntimeInfo for Updated Datasets Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-datasets.md Highlights the benefit of providing `runtimeInfo` to `resolveDatasetSelection` to ensure access to the most current list of available datasets. Omitting it may result in an empty list. ```typescript // ❌ Sem RuntimeInfo, usa lista vazia const selection = resolveDatasetSelection(params, null); // ✅ Com RuntimeInfo, conhece datasets disponíveis const runtimeInfo = await loadRuntimeInfo(env); const selection = resolveDatasetSelection(params, runtimeInfo); ``` -------------------------------- ### Load Binary Index from R2 Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Loads a binary index directly from R2 storage with in-memory caching. This function requires an R2Bucket instance and a specific key. ```typescript async function loadBinaryIndexFromR2( bucket: R2Bucket, key: string, ): Promise ``` -------------------------------- ### PathsConfig C# Class Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md Defines configuration for various directory paths used by the ETL process, including data, parquet, output, and download directories. ```csharp public class PathsConfig { public string DataDir { get; set; } // Diretório de dados extraídos public string ParquetDir { get; set; } // Diretório de Parquets gerados public string OutputDir { get; set; } // Diretório de shards de saída public string DownloadDir { get; set; } // Diretório de downloads do CNPJ } ``` -------------------------------- ### Load Runtime Info with In-Memory Cache Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-storage.md Loads RuntimeInfo (info.json) into an object, utilizing an in-memory cache with a 60-second TTL. It prioritizes embedded info, cache, then R2 or assets. ```typescript const info = await loadRuntimeInfo(env); if (info) { console.log(info.storage_release_id); // "abc123" } ``` -------------------------------- ### Tipo para Informações de Dataset Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/types.md Armazena metadados de um módulo de dados, como nome da propriedade JSON, ID do release de armazenamento, disponibilidade e detalhes do ZIP. ```typescript export type DatasetInfo = { json_property_name?: string; storage_release_id?: string; zip_available?: boolean; zip_size?: number; zip_url?: string; zip_md5checksum?: string; }; ``` -------------------------------- ### corsPreflight Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/api-reference/worker-http.md Constructs a response for OPTIONS requests (CORS preflight). This function returns a 204 No Content response with the necessary CORS headers to allow cross-origin requests. ```APIDOC ## corsPreflight ### Description Builds a response to OPTIONS requests (CORS preflight). It returns a `204 No Content` response with appropriate CORS headers. ### Function Signature ```typescript function corsPreflight(): Response ``` ### Return Value `Response` - 204 No Content response with CORS headers ### Headers ``` Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, OPTIONS Access-Control-Allow-Headers: Content-Type, Accept Access-Control-Max-Age: 86400 ``` ### Status 204 (No Content) ### Example ```typescript if (request.method === "OPTIONS") { return corsPreflight(); } ``` ``` -------------------------------- ### CORS and OPTIONS Response Headers Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md These headers are returned for CORS preflight requests and indicate allowed origins, methods, and headers. ```http Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, OPTIONS Access-Control-Allow-Headers: Content-Type, Accept Access-Control-Max-Age: 86400 ``` -------------------------------- ### OPTIONS Endpoint Source: https://github.com/hitmasu/opencnpj/blob/main/_autodocs/endpoints.md This endpoint supports CORS preflight requests for all other endpoints. It returns a 204 No Content status with appropriate CORS headers. ```APIDOC ## OPTIONS / ### Description Supports CORS preflight requests for all endpoints. Returns a 204 No Content status with CORS headers. ### Method OPTIONS ### Endpoint / ### Response #### Success Response (204) ### Headers ``` Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, OPTIONS Access-Control-Allow-Headers: Content-Type, Accept Access-Control-Max-Age: 86400 ``` ```