### Install and Start Colima for Docker on macOS Source: https://github.com/atlanhq/atlas-metastore/blob/master/LOCAL_SETUP.md Install Colima, Docker, and Docker Compose using Homebrew, then start Colima with specified disk and memory limits. Colima provides a container runtime for Docker on macOS. ```bash brew install colima docker docker-compose colima start --disk 10 --memory 4 ``` -------------------------------- ### Downstream Lineage Traversal Example Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/zero-graph-arb-discussion.md Demonstrates a step-by-step downstream traversal of lineage data starting from 'raw_orders', showing how each hop queries the 'edges_out' table. ```sql HOP 1: edges_out WHERE out_vertex_id = raw_orders AND edge_label = '__Process.inputs' → ETL Job HOP 2: edges_out WHERE out_vertex_id = ETL Job AND edge_label = '__Process.outputs' → clean_orders HOP 3: edges_out WHERE out_vertex_id = clean_orders AND edge_label = '__Process.inputs' → Agg Job HOP 4: edges_out WHERE out_vertex_id = Agg Job AND edge_label = '__Process.outputs' → order_summary ``` -------------------------------- ### Set up Python Virtual Environment and Install Dependencies Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/README.md Use these commands to create a virtual environment and install the necessary Python packages for local development. ```bash python3 -m venv .venv source .venv/bin/activate pip install mkdocs-material ``` -------------------------------- ### Start Infrastructure Services Locally Source: https://github.com/atlanhq/atlas-metastore/blob/master/README.md Use Docker Compose to start the necessary infrastructure services for local development. Ensure Docker is running. ```bash docker-compose -f local-dev/docker-compose.yaml up -d ``` -------------------------------- ### Install Elasticsearch Chart from Repository Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Install the Elasticsearch Helm chart using the previously added Elastic repository. ```bash helm install --name elasticsearch elastic/elasticsearch ``` -------------------------------- ### Bulk Entity Creation Example Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/api/entity-bulk.md This example demonstrates creating a 'Table' entity with basic attributes using the bulk API. Ensure proper Content-Type and Authorization headers are included. ```bash curl -X POST 'http://localhost:21000/api/atlas/v2/entity/bulk' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ' \ -d '{ "entities": [ { "typeName": "Table", "attributes": { "qualifiedName": "mydb.public.users@snowflake", "name": "users", "description": "User accounts table", "comment": "Contains all registered user accounts", "ownerUsers": ["data-team@company.com"], "ownerGroups": ["analytics"], "displayName": "Users Table" } } ]' ``` -------------------------------- ### Install Cassandra Chart Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/cassandra/README.md Installs the Cassandra chart into the 'cassandra' namespace with the release name 'cassandra'. Ensure a storage class is configured if persistence is enabled. ```bash helm install --namespace "cassandra" -n "cassandra" incubator/cassandra ``` -------------------------------- ### Install Java 17 with Homebrew Source: https://github.com/atlanhq/atlas-metastore/blob/master/LOCAL_SETUP.md Use Homebrew to install OpenJDK 17, a required version for the project. ```bash brew install openjdk@17 ``` -------------------------------- ### Install Elasticsearch Chart from Master Branch Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Install the Elasticsearch Helm chart by providing the local path after cloning the repository. ```bash helm install --name elasticsearch ./helm-charts/elasticsearch ``` -------------------------------- ### Install Dependencies Source: https://github.com/atlanhq/atlas-metastore/blob/master/dev-support/test-harness/README.md Install the necessary Python packages for the test harness. Requires Python 3.8+. ```bash pip install requests pyyaml ``` -------------------------------- ### Install Apache Atlas Python Client Source: https://github.com/atlanhq/atlas-metastore/blob/master/intg/src/main/python/README.md Install the Apache Atlas Python client using pip. This command fetches and installs the latest stable version of the library. ```bash pip install apache-atlas ``` -------------------------------- ### Start Atlas Locally Source: https://github.com/atlanhq/atlas-metastore/blob/master/CLAUDE.md Command to start an Atlas instance locally. Ensure all dependencies like Redis and Cassandra are running. Access the UI at http://localhost:21000. ```bash java -Datlas.home=deploy/ -Datlas.conf=deploy/conf -Datlas.data=deploy/data \ -Datlas.log.dir=deploy/logs -Dlogback.configurationFile=file:./deploy/conf/atlas-logback.xml \ --add-opens java.base/java.lang=ALL-UNNAMED -Xms512m \ org.apache.atlas.Atlas ``` -------------------------------- ### Install and Configure Jenv for Java Version Management Source: https://github.com/atlanhq/atlas-metastore/blob/master/LOCAL_SETUP.md Install jenv, a tool for managing multiple Java versions, and enable its export and Maven plugins for seamless version switching. ```bash brew install jenv jenv enable-plugin export jenv enable-plugin maven ``` -------------------------------- ### Install Elasticsearch Exporter Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas-read/charts/elasticsearch-exporter-read/README.md Installs the Elasticsearch Exporter using Helm with the default configuration. Ensure you have Helm and a Kubernetes cluster set up. ```bash $ helm install stable/elasticsearch-exporter ``` -------------------------------- ### Install Development Logstash Helm Chart Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/logstash/README.md Install a development version of the Logstash chart from a local clone of the repository, specifying an image tag. ```bash helm install logstash ./helm-charts/logstash --set imageTag=8.5.1 ``` -------------------------------- ### AtlasObjectId Reference Examples Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/api/entity-bulk.md Examples demonstrating how to reference other entities using AtlasObjectId. This includes referencing by temporary GUID for new entities, real GUID for existing entities, or by unique attributes. ```json { "typeName": "Column", "guid": "-2" } ``` ```json { "typeName": "Column", "guid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" } ``` ```json { "typeName": "Column", "uniqueAttributes": { "qualifiedName": "mydb.public.users.id@snowflake" } } ``` -------------------------------- ### Serve Documentation Locally Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/README.md Run this command to start a local web server for previewing the API documentation. The site will auto-reload on file changes. ```bash mkdocs serve ``` -------------------------------- ### Build Elasticsearch Chart for KIND Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Navigate to the KIND examples directory and build the Elasticsearch chart. ```bash cd examples/kubernetes-kind make ``` -------------------------------- ### ES Aggregation Strategy for Purpose GUIDs Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/ms-546/VALIDATION_SUMMARY.md Illustrates the two-step Elasticsearch query strategy: first aggregating AuthPolicies to get unique Purpose GUIDs, then fetching Purpose details by these GUIDs. ```text Step 1: Aggregate AuthPolicies → Get unique Purpose GUIDs Step 2: Multi-Get → Fetch Purpose details by GUIDs ``` -------------------------------- ### Run Goss Integration Tests Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Navigate to the default example directory and run the goss integration tests against the Elasticsearch deployment. ```bash cd examples/default make goss ``` -------------------------------- ### Get type definition headers Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Retrieves a lightweight listing of all type definition names and GUIDs. ```APIDOC ## GET /api/atlas/v2/types/typedefs/headers — Get type definition headers Lightweight listing of all type definition names and GUIDs. ### Method GET ### Endpoint /api/atlas/v2/types/typedefs/headers ### Response #### Success Response (200) - (array of objects) - A list of type definition headers. - **guid** (string) - The unique identifier of the type definition. - **name** (string) - The name of the type definition. - **category** (string) - The category of the type definition (e.g., ENTITY, RELATIONSHIP, CLASSIFICATION). ``` -------------------------------- ### Get lineage with composite request Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Accepts an `AtlasLineageRequest` with GUID, direction, depth, and optional filters in a single POST body. ```APIDOC ## POST /api/atlas/v2/lineage/getlineage — Get lineage with composite request ### Description Accepts an `AtlasLineageRequest` with GUID, direction, depth, and optional filters in a single POST body. ### Method POST ### Endpoint /api/atlas/v2/lineage/getlineage ### Request Body - **guid** (string) - Required - The unique identifier of the entity. - **direction** (string) - Optional - The direction of lineage to retrieve (`INPUT`, `OUTPUT`, `BOTH`). - **depth** (integer) - Optional - The depth of the lineage graph to retrieve. - **offset** (integer) - Optional - The offset for pagination. - **limit** (integer) - Optional - The maximum number of results to return. ``` -------------------------------- ### AtlasClientV2 Java Client Examples Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Demonstrates initialization and common operations using the AtlasClientV2 Java library, including fetching, creating, and deleting entities, and retrieving type definitions. ```java import org.apache.atlas.AtlasClientV2; import org.apache.atlas.model.instance.AtlasEntity; import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo; import org.apache.atlas.model.instance.EntityMutationResponse; import org.apache.atlas.model.typedef.AtlasTypesDef; import org.apache.atlas.model.SearchFilter; // --- Initialization --- AtlasClientV2 client = new AtlasClientV2( new String[]{"http://localhost:21000"}, new String[]{"admin", "admin"} ); // --- Fetch entity by GUID --- try { AtlasEntityWithExtInfo entity = client.getEntityByGuid("3b4a5c6d-1234-5678-abcd-ef0123456789"); System.out.println("Found: " + entity.getEntity().getAttribute("qualifiedName")); } catch (AtlasServiceException e) { System.err.println("Error fetching entity: " + e.getMessage()); } // --- Create an entity --- AtlasEntity tableEntity = new AtlasEntity("hive_table"); tableEntity.setAttribute("qualifiedName", "default.new_table@cluster1"); tableEntity.setAttribute("name", "new_table"); tableEntity.setAttribute("description", "A new Hive table"); AtlasEntity.AtlasEntityWithExtInfo entityInfo = new AtlasEntity.AtlasEntityWithExtInfo(tableEntity); try { EntityMutationResponse response = client.createEntity(entityInfo); Map guids = response.getGuidAssignments(); System.out.println("Created GUID: " + guids.values().iterator().next()); } catch (AtlasServiceException e) { System.err.println("Create failed: " + e.getMessage()); } // --- Get all type definitions --- try { AtlasTypesDef allTypes = client.getAllTypeDefs(new SearchFilter()); System.out.println("Entity types: " + allTypes.getEntityDefs().size()); System.out.println("Classification types: " + allTypes.getClassificationDefs().size()); } catch (AtlasServiceException e) { System.err.println("Failed to fetch types: " + e.getMessage()); } // --- Delete entity by GUID --- try { EntityMutationResponse deleteResponse = client.deleteEntityByGuid("3b4a5c6d-1234-5678-abcd-ef0123456789"); System.out.println("Deleted: " + deleteResponse.getDeletedEntities().size() + " entities"); } catch (AtlasServiceException e) { System.err.println("Delete failed: " + e.getMessage()); } ``` -------------------------------- ### Deploy Atlas and Dependent Services Source: https://github.com/atlanhq/atlas-metastore/blob/master/dev-support/atlas-docker/README.md Installs and starts Apache Atlas along with its dependent services (Solr, HBase, Kafka, Hive) in detached Docker containers. ```bash docker-compose -f docker-compose.atlas-base.yml -f docker-compose.atlas.yml -f docker-compose.atlas-hadoop.yml -f docker-compose.atlas-hbase.yml -f docker-compose.atlas-kafka.yml -f docker-compose.atlas-hive.yml up -d ``` -------------------------------- ### Get Type Definition Headers Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Obtain a lightweight list of all type definition names and their GUIDs. This is useful for quick lookups without fetching full definitions. ```bash curl -u admin:admin \ "http://localhost:21000/api/atlas/v2/types/typedefs/headers" ``` -------------------------------- ### Build Static Documentation Site Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/README.md Execute this command to build the static version of the API documentation. The output will be placed in the 'site/' directory. ```bash mkdocs build ``` -------------------------------- ### Create a Table with Columns Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/api/entity-bulk.md Use negative GUIDs to create parent-child relationships in a single request. This example demonstrates creating a table and its associated columns, establishing relationships between them. ```APIDOC ## POST /api/atlas/v2/entity/bulk ### Description Creates a table with its associated columns, establishing parent-child relationships using temporary GUIDs. ### Method POST ### Endpoint /api/atlas/v2/entity/bulk ### Request Body - **entities** (array) - Required - A list of entities to create. - **typeName** (string) - Required - The type of the entity (e.g., "Table", "Column"). - **guid** (string) - Required - A temporary GUID (e.g., "-1") for referencing within the request. - **attributes** (object) - Required - Key-value pairs for entity attributes. - **qualifiedName** (string) - Required - The unique qualified name for the entity. - **name** (string) - Required - The name of the entity. - **data_type** (string) - Optional - The data type for column entities. - **relationshipAttributes** (object) - Optional - Attributes defining relationships to other entities. - **columns** (array) - Optional - A list of column entities associated with a table. - **typeName** (string) - Required - The type of the related entity (e.g., "Column"). - **guid** (string) - Required - The temporary GUID of the related entity. - **table** (object) - Optional - The table entity associated with a column. - **typeName** (string) - Required - The type of the related entity (e.g., "Table"). - **guid** (string) - Required - The temporary GUID of the related entity. ### Request Example ```json { "entities": [ { "typeName": "Table", "guid": "-1", "attributes": { "qualifiedName": "mydb.public.orders@snowflake", "name": "orders" }, "relationshipAttributes": { "columns": [ { "typeName": "Column", "guid": "-2" }, { "typeName": "Column", "guid": "-3" }, { "typeName": "Column", "guid": "-4" } ] } }, { "typeName": "Column", "guid": "-2", "attributes": { "qualifiedName": "mydb.public.orders.order_id@snowflake", "name": "order_id", "data_type": "INTEGER" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } }, { "typeName": "Column", "guid": "-3", "attributes": { "qualifiedName": "mydb.public.orders.customer_id@snowflake", "name": "customer_id", "data_type": "INTEGER" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } }, { "typeName": "Column", "guid": "-4", "attributes": { "qualifiedName": "mydb.public.orders.total_amount@snowflake", "name": "total_amount", "data_type": "DECIMAL" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } } ] } ``` ### Response #### Success Response (200) - **mutatedEntities** (object) - Contains details of created entities. - **CREATE** (array) - List of created entities with their new GUIDs. - **typeName** (string) - The type of the entity. - **guid** (string) - The newly assigned GUID. - **status** (string) - The status of the entity (e.g., "ACTIVE"). - **attributes** (object) - Attributes of the created entity. - **guidAssignments** (object) - Maps temporary GUIDs to newly generated permanent GUIDs. #### Response Example ```json { "mutatedEntities": { "CREATE": [ { "typeName": "Table", "guid": "a1b2c3d4-...", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders@snowflake" } }, { "typeName": "Column", "guid": "b2c3d4e5-", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders.order_id@snowflake" } }, { "typeName": "Column", "guid": "c3d4e5f6-", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders.customer_id@snowflake" } }, { "typeName": "Column", "guid": "d4e5f6a7-", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders.total_amount@snowflake" } } ] }, "guidAssignments": { "-1": "a1b2c3d4-", "-2": "b2c3d4e5-", "-3": "c3d4e5f6-", "-4": "d4e5f6a7-" } } ``` ``` -------------------------------- ### Enable Minikube Addons and Build Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Enable necessary Minikube addons and build the Elasticsearch chart for local development. ```bash minikube addons enable default-storageclass minikube addons enable storage-provisioner cd examples/minikube make ``` -------------------------------- ### Custom Groovy Script Example Source: https://github.com/atlanhq/atlas-metastore/blob/master/tools/gremlin-atlas/SKILL.md This example demonstrates how to open a JanusGraph instance using a properties file and query for a vertex by its GUID. Remember to close the graph instance after use. Use bare variable assignments (e.g., `x = value`) instead of `def x = value` due to Groovy scoping. ```groovy // my-query.groovy import org.janusgraph.core.JanusGraphFactory graph = JanusGraphFactory.open('/tmp/gremlin-100-gremlin-1/conf/janusgraph-cql-es.properties') g = graph.traversal() result = g.V().has('__guid', 'YOUR-GUID-HERE').valueMap().next() println result graph.close() ``` -------------------------------- ### Enable MicroK8s Addons and Build Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Enable MicroK8s addons, including DNS, Helm, and storage, then build the Elasticsearch chart. ```bash microk8s.enable dns microk8s.enable helm microk8s.enable storage cd examples/microk8s make ``` -------------------------------- ### Download Archives for Atlas Setup Source: https://github.com/atlanhq/atlas-metastore/blob/master/dev-support/atlas-docker/README.md Execute this script to download necessary archives for setting up Atlas, HDFS, HBase, and Kafka services. ```bash ./download-archives.sh ``` -------------------------------- ### Get Roles, Groups, and Users for Entity Actions Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Identify which roles, groups, and users have permissions to perform a specific action on an entity. Requires entity GUID, type name, and the action. ```bash curl -u admin:admin -X POST \ -H "Content-Type: application/json" \ -d '[ { "guid": "3b4a5c6d-1234-5678-abcd-ef0123456789", "typeName": "hive_table", "action": "ENTITY_UPDATE" } ]' \ "http://localhost:21000/api/atlas/v2/entity/accessors" ``` -------------------------------- ### POST /api/meta/purposes/user API Endpoint Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/ms-546/VALIDATION_SUMMARY.md This endpoint retrieves all accessible Purposes for a user by utilizing Elasticsearch aggregation. It first aggregates AuthPolicies to get unique Purpose GUIDs and then performs a multi-get operation to fetch the Purpose details. ```APIDOC ## POST /api/meta/purposes/user ### Description Retrieves all accessible Purposes for a user by leveraging Elasticsearch aggregation. This approach involves two steps: first, aggregating AuthPolicies to obtain unique Purpose GUIDs, and second, performing a multi-get operation to fetch the detailed Purpose information. ### Method POST ### Endpoint /api/meta/purposes/user ### Request Body - **username** (string) - Required - The username for whom to fetch purposes. - **groups** (array of strings) - Required - A list of groups the user belongs to. - **limit** (integer) - Optional - The maximum number of purposes to return. - **offset** (integer) - Optional - The offset for pagination. ### Request Example ```json { "username": "john.doe", "groups": ["data-team", "analysts"], "limit": 100, "offset": 0 } ``` ### Response #### Success Response (200) - **purposes** (array of objects) - A list of purposes accessible to the user. Each purpose object contains 'guid', 'name', and 'displayName'. - **count** (integer) - The number of purposes returned in the current response. - **totalCount** (integer) - The total number of purposes accessible to the user. - **hasMore** (boolean) - Indicates if there are more purposes available beyond the current response. #### Response Example ```json { "purposes": [ { "guid": "...", "name": "...", "displayName": "..." } ], "count": 10, "totalCount": 50, "hasMore": true } ``` ``` -------------------------------- ### Verify Apache Atlas Python Client Installation Source: https://github.com/atlanhq/atlas-metastore/blob/master/intg/src/main/python/README.md Verify that the apache-atlas client has been successfully installed by listing installed packages. This command shows all installed Python packages and their versions. ```bash pip list Package Version ------------ --------- apache-atlas 0.0.11 ``` -------------------------------- ### Example: Deploy to 2 Specific Customers Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/cohort-release/DEVELOPER-RUNBOOK.md Create a custom cohort file for deploying to specific tenants like Affirm and Sheetz. Ensure you use the correct cluster names for the `name` field. ```bash # 1. Find cluster names # affirm.atlan.com → affirm-mt # sheetz.atlan.com → sheetz01 ``` ```json # 2. Create cohort file: atlan-releases/cohorts/atlas-custom-dq-validation.json { "description": "DQ feature validation on Affirm and Sheetz", "ring": "custom", "generatedAt": "2026-03-10", "tenantCount": 2, "tenants": [ {"name": "affirm-mt", "domain": "affirm.atlan.com", "assetCount": 0, "cloudProvider": "aws", "deploymentType": "Production"}, {"name": "sheetz01", "domain": "sheetz.atlan.com", "assetCount": 0, "cloudProvider": "aws", "deploymentType": "Trial"} ] } ``` -------------------------------- ### Cache GUID per (typeName + qualifiedName) in RequestContext Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/cassandra-partial-failure-strategy.md Cache GUIDs in RequestContext to ensure retries reuse the same GUID for a given typeName and qualifiedName. This is part of deterministic GUID generation for retries. ```java String cacheKey = typeName + ":" + qualifiedName; String guid = RequestContext.get().getCachedGuid(cacheKey); if (guid == null) { guid = UUID.randomUUID().toString(); RequestContext.get().cacheGeneratedGuid(cacheKey, guid); } ``` -------------------------------- ### Troubleshoot: Pod Won't Start After Backend Switch Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/tenant-migration-runbook.md Check Atlas logs for errors after switching the backend. Common issues include missing Cassandra properties or an non-existent keyspace. ```bash kubectl logs atlas-0 -n atlas -c atlas-main --previous ``` -------------------------------- ### Install Specific Elasticsearch Version Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Install a specific version of Elasticsearch by overriding the `imageTag` value during Helm installation. ```bash helm install --name elasticsearch elastic/elasticsearch --set imageTag=7.6.1 ``` -------------------------------- ### Fetch Multiple Entities by GUIDs Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Retrieves multiple entities in one call by providing a list of GUIDs. Use repeated `guid` query parameters. ```bash curl -u admin:admin \ "http://localhost:21000/api/atlas/v2/entity/bulk?guid=guid1&guid=guid2&guid=guid3&minExtInfo=false" ``` -------------------------------- ### Gremlin Atlas Setup and Execution Commands Source: https://github.com/atlanhq/atlas-metastore/blob/master/tools/gremlin-atlas/README.md These commands are used for the initial setup of the gremlin-100 distribution on the atlas pod and for running built-in scripts. Ensure you are connected to the tenant's vcluster before execution. ```bash # 1. Connect to the tenant's vcluster vcluster platform connect vcluster # 2. One-time setup: download gremlin-100 onto the atlas pod (~379MB, takes ~10s) tools/gremlin-atlas/scripts/gremlin-setup.sh [kubectl-context] # 3. Run a built-in script tools/gremlin-atlas/scripts/gremlin-run.sh index-status [kubectl-context] tools/gremlin-atlas/scripts/gremlin-run.sh sanity-check [kubectl-context] tools/gremlin-atlas/scripts/gremlin-run.sh schema-info [kubectl-context] ``` -------------------------------- ### Fetch multiple entities by GUIDs Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Retrieves multiple entities in a single API call by providing a list of their GUIDs. Use repeated `guid` query parameters. ```APIDOC ## GET /api/atlas/v2/entity/bulk — Fetch multiple entities by GUIDs ### Description Retrieves multiple entities in one call given a list of GUIDs. Use repeated `guid` query parameters. ### Method GET ### Endpoint /api/atlas/v2/entity/bulk ### Query Parameters - **guid** (string) - Required - The GUID of the entity to retrieve. Can be specified multiple times. - **minExtInfo** (boolean) - Optional - If true, returns minimal extended information. ### Response #### Success Response (200) - **entities** (array) - An array of entity objects. - **typeName** (string) - The type of the entity. - **guid** (string) - The GUID of the entity. - **referredEntities** (object) - A map of referred entities. ``` -------------------------------- ### Build Elasticsearch Chart for Docker for Mac Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/elasticsearch/README.md Navigate to the Docker for Mac examples directory and build the Elasticsearch chart. ```bash cd examples/docker-for-mac make ``` -------------------------------- ### Dockerfile for Logstash Plugin Installation Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/logstash/README.md Use this Dockerfile to install Logstash plugins by building a custom Docker image. This approach ensures immutability and avoids runtime plugin installations. ```dockerfile ARG logstash_version FROM docker.elastic.co/logstash/logstash:${logstash_version} RUN bin/logstash-plugin install logstash-output-kafka ``` -------------------------------- ### Initialize System Properties for Indexing Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/cassandra-es-index-architecture.md This phase is executed once during startup. It involves creating various mixed indexes for system properties, including GUID, typeName, state, timestamp, and superTypeNames. It also handles the creation of Elasticsearch indexes and mappings. ```java GraphBackedSearchIndexer.initialize() │ ├── management = graph.getManagementSystem() │ └── returns new CassandraGraphManagement(this) │ ├── createVertexMixedIndex("vertex_index", "search", []) │ └── CassandraGraphManagement: │ ├── new CassandraGraphIndex("vertex_index", mixed=true, vertex=true) │ ├── graphIndexes.put("vertex_index", index) │ └── ensureESIndexExists("atlas_graph_vertex_index") │ ├── HEAD /atlas_graph_vertex_index → 200? preloadESMappingCache() │ └── 404? PUT /atlas_graph_vertex_index (create index) │ ├── createEdgeMixedIndex("edge_index", "search", []) │ └── (same pattern) │ ├── createFullTextMixedIndex("fulltext_index", "search", []) │ └── (same pattern) │ └── createCommonVertexIndex() × ~50 system properties │ │ Examples: │ ├── __guid (String, SINGLE, unique=GLOBAL_UNIQUE) │ ├── __typeName (String, SINGLE, unique=GLOBAL_UNIQUE) │ ├── __state (String, SINGLE) │ ├── __timestamp (Long, SINGLE, multifields={date: epoch_millis}) │ ├── __superTypeNames (String, SET, multifields={keyword: normalizer}) │ ├── __traitNames (String, SET) │ └── ... ~45 more │ └── createVertexIndex(management, propName, uniqueKind, propClass, ...) │ ├── management.makePropertyKey(propName, propClass, cardinality) │ └── CassandraPropertyKey stored in graph.propertyKeys map │ ├── isIndexApplicable(propClass, cardinality)? │ └── true unless BigDecimal/BigInteger │ ├── !management.getGraphIndex("vertex_index").getFieldKeys().contains(key)? │ └── true (first time → not in fieldKeys yet) │ └── management.addMixedIndex("vertex_index", propertyKey, isStringField, esConfig, esFields) ├── index.addFieldKey(propertyKey) ← REGISTERS IN fieldKeys ├── addESFieldMapping() or addESFieldMappingWithSubFields() │ ├── Check esMappingCache → skip if already exists │ ├── Map Java class → ES type (String→keyword/text, Long→long, etc.) │ └── PUT /atlas_graph_vertex_index/_mapping {"properties":{"field":{...}}} └── return propertyKey.getName() (no JanusGraph field encoding) ``` -------------------------------- ### Install Cassandra Chart with Specific Cluster Size Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/cassandra/README.md Installs the Cassandra chart with a cluster size of 5 nodes. This can be set during installation using the --set argument or by modifying values.yaml. ```bash helm install --namespace "cassandra" -n "cassandra" --set config.cluster_size=5 incubator/cassandra/ ``` -------------------------------- ### JSON GUID Assignment Mapping Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/cassandra-partial-failure-strategy.md Illustrates the mapping of negative GUIDs to assigned GUIDs within a JSON response. This mapping can become inconsistent on retries if entities re-enter the CREATE path. ```json { "-1234": "a1b2c3d4-..." } ``` -------------------------------- ### Fetch Relationship by GUID Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Fetches a specific relationship by its unique identifier (GUID). ```APIDOC ## GET /api/atlas/v2/relationship/guid/{guid} — Fetch relationship by GUID ### Description Fetches a specific relationship by its unique identifier (GUID). ### Method GET ### Endpoint /api/atlas/v2/relationship/guid/{guid} ### Parameters #### Path Parameters - **guid** (string) - Required - The GUID of the relationship to fetch. #### Query Parameters - **extendedInfo** (boolean) - Optional - If true, includes extended information about the relationship and referred entities. ### Response #### Success Response (200) - **relationship** (object) - Details of the fetched relationship. - **typeName** (string) - The type of the relationship. - **guid** (string) - The GUID of the relationship. - **status** (string) - The status of the relationship (e.g., ACTIVE). - ... (other relationship details) - **referredEntities** (object) - Information about entities referred to in the relationship. ``` -------------------------------- ### Bulk Entity Creation with Referred Entities Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/api/entity-bulk.md This example demonstrates how to create entities, including a Table and its associated Columns, using the `entities` and `referredEntities` fields. `referredEntities` is useful for defining supporting entities separately from the primary ones. ```APIDOC ## POST /api/atlas/v2/entity/bulk ### Description Creates or updates entities in bulk. Supports defining primary entities and their associated referred entities. ### Method POST ### Endpoint /api/atlas/v2/entity/bulk ### Request Body - **entities** (array) - Required - The primary list of entities to create or update. - **referredEntities** (object) - Optional - A map of temporary GUIDs to entity definitions for supporting entities. ### Request Example ```json { "entities": [ { "typeName": "Table", "guid": "-1", "attributes": { "qualifiedName": "mydb.public.products@snowflake", "name": "products" }, "relationshipAttributes": { "columns": [ { "typeName": "Column", "guid": "-100" }, { "typeName": "Column", "guid": "-101" } ] } } ], "referredEntities": { "-100": { "typeName": "Column", "guid": "-100", "attributes": { "qualifiedName": "mydb.public.products.id@snowflake", "name": "id", "data_type": "INTEGER" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } }, "-101": { "typeName": "Column", "guid": "-101", "attributes": { "qualifiedName": "mydb.public.products.name@snowflake", "name": "name", "data_type": "VARCHAR" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } } } } ``` ### Response #### Success Response (200) - **guidAssignments** (object) - Maps temporary GUIDs to actual assigned GUIDs. - **mutatedEntities** (object) - Details of entities that were created or updated. #### Response Example ```json { "guidAssignments": { "-1": "a1b2c3d4-e5f6-7890-1234-567890abcdef", "-100": "b2c3d4e5-f6a7-8901-2345-67890abcdef1", "-101": "c3d4e5f6-a7b8-9012-3456-7890abcdef12" }, "mutatedEntities": { "Table": [ { "guid": "a1b2c3d4-e5f6-7890-1234-567890abcdef", "typeName": "Table", "attributes": { "qualifiedName": "mydb.public.products@snowflake", "name": "products" } } ], "Column": [ { "guid": "b2c3d4e5-f6a7-8901-2345-67890abcdef1", "typeName": "Column", "attributes": { "qualifiedName": "mydb.public.products.id@snowflake", "name": "id", "data_type": "INTEGER" } }, { "guid": "c3d4e5f6-a7b8-9012-3456-7890abcdef12", "typeName": "Column", "attributes": { "qualifiedName": "mydb.public.products.name@snowflake", "name": "name", "data_type": "VARCHAR" } } ] } } ``` ``` -------------------------------- ### Create Table with Columns using Temporary GUIDs Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/api/entity-bulk.md Use negative GUIDs to create parent-child relationships between entities like tables and columns in a single request. The response includes assigned GUIDs for the created entities. ```bash curl -X POST 'http://localhost:21000/api/atlas/v2/entity/bulk' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ' \ -d '{ "entities": [ { "typeName": "Table", "guid": "-1", "attributes": { "qualifiedName": "mydb.public.orders@snowflake", "name": "orders" }, "relationshipAttributes": { "columns": [ { "typeName": "Column", "guid": "-2" }, { "typeName": "Column", "guid": "-3" }, { "typeName": "Column", "guid": "-4" } ] } }, { "typeName": "Column", "guid": "-2", "attributes": { "qualifiedName": "mydb.public.orders.order_id@snowflake", "name": "order_id", "data_type": "INTEGER" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } }, { "typeName": "Column", "guid": "-3", "attributes": { "qualifiedName": "mydb.public.orders.customer_id@snowflake", "name": "customer_id", "data_type": "INTEGER" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } }, { "typeName": "Column", "guid": "-4", "attributes": { "qualifiedName": "mydb.public.orders.total_amount@snowflake", "name": "total_amount", "data_type": "DECIMAL" }, "relationshipAttributes": { "table": { "typeName": "Table", "guid": "-1" } } } ] }' ``` ```json { "mutatedEntities": { "CREATE": [ { "typeName": "Table", "guid": "a1b2c3d4- மரு", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders@snowflake" } }, { "typeName": "Column", "guid": "b2c3d4e5- மரு", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders.order_id@snowflake" } }, { "typeName": "Column", "guid": "c3d4e5f6- மரு", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders.customer_id@snowflake" } }, { "typeName": "Column", "guid": "d4e5f6a7- மரு", "status": "ACTIVE", "attributes": { "qualifiedName": "mydb.public.orders.total_amount@snowflake" } } ] }, "guidAssignments": { "-1": "a1b2c3d4- மரு", "-2": "b2c3d4e5- மரு", "-3": "c3d4e5f6- மரு", "-4": "d4e5f6a7- மரு" } } ``` -------------------------------- ### Build Project for Intel Mac Source: https://github.com/atlanhq/atlas-metastore/blob/master/README.md Build the project using Maven. This command is for Intel Macs and skips tests and linting. ```bash mvn clean -Dmaven.test.skip -DskipTests -Drat.skip=true -DskipOverlay -DskipEnunciate=true install -Pdist ``` -------------------------------- ### Bulk Delete Entities by GUIDs Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Deletes multiple entities in a single request by specifying their GUIDs. ```bash curl -u admin:admin -X DELETE \ "http://localhost:21000/api/atlas/v2/entity/bulk?guid=guid1&guid=guid2" ``` -------------------------------- ### Setup and Run Gremlin Scripts Source: https://github.com/atlanhq/atlas-metastore/blob/master/tools/gremlin-atlas/SKILL.md Use these commands to set up the gremlin-100 distribution on the atlas pod and then run either built-in or custom Groovy scripts. Ensure kubectl is configured for the target tenant. ```bash # 1. One-time setup per pod restart: download gremlin-100 onto the atlas pod tools/gremlin-atlas/scripts/gremlin-setup.sh [kubectl-context] # 2. Run a built-in script tools/gremlin-atlas/scripts/gremlin-run.sh [kubectl-context] # 3. Run a custom local groovy script tools/gremlin-atlas/scripts/gremlin-run.sh /path/to/my-script.groovy [kubectl-context] ``` ```bash tools/gremlin-atlas/scripts/gremlin-run.sh $ARGUMENTS ``` -------------------------------- ### Check Startup Logs for Cassandra Backend Source: https://github.com/atlanhq/atlas-metastore/blob/master/docs/tenant-migration-runbook.md Inspects the Atlas pod logs to confirm that the Cassandra backend was loaded and JanusGraph initialization was skipped. This check is crucial for verifying a faster startup time. ```bash kubectl logs atlas-0 -n atlas -c atlas-main | grep -i "graphdb\|cassandra\|backend" ``` -------------------------------- ### Install Elasticsearch Exporter with --set Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas-read/charts/elasticsearch-exporter-read/README.md Install the Elasticsearch Exporter Helm chart using the `--set` argument to override default configuration values. This is useful for quick adjustments or setting specific parameters during installation. ```bash helm install --name my-release \ --set key_1=value_1,key_2=value_2 \ stable/elasticsearch-exporter ``` -------------------------------- ### Build Project for Apple Silicon Mac Source: https://github.com/atlanhq/atlas-metastore/blob/master/README.md Build the project using Maven. This command is specifically for Apple Silicon Macs and skips tests and linting. ```bash mvn clean -Dos.detected.classifier=osx-x86_64 -Dmaven.test.skip -DskipTests -Drat.skip=true -DskipOverlay -DskipEnunciate=true install -Pdist ``` -------------------------------- ### Fetch entity header by GUID Source: https://context7.com/atlanhq/atlas-metastore/llms.txt Retrieves a lightweight header for an entity using its GUID. This is more efficient than a full entity fetch and includes only essential information like GUID, typeName, status, display text, and classification names. ```APIDOC ## GET /api/atlas/v2/entity/guid/{guid}/header ### Description Returns a lightweight `AtlasEntityHeader` with only the GUID, typeName, status, display text, and classification names — much cheaper than a full entity fetch. ### Method GET ### Endpoint /api/atlas/v2/entity/guid/{guid}/header ### Response #### Success Response (200) - **typeName** (string) - The type of the entity. - **guid** (string) - The unique identifier of the entity. - **status** (string) - The status of the entity (e.g., ACTIVE). - **displayText** (string) - The display name of the entity. - **classificationNames** (array) - A list of classification names applied to the entity. #### Response Example ```json { "typeName": "hive_table", "guid": "3b4a5c6d-1234-5678-abcd-ef0123456789", "status": "ACTIVE", "displayText": "orders", "classificationNames": ["PII", "Confidential"] } ``` ``` -------------------------------- ### Install Released Logstash Helm Chart Source: https://github.com/atlanhq/atlas-metastore/blob/master/helm/atlas/charts/logstash/README.md Install a released version of the Logstash chart using Helm. ```bash helm install logstash elastic/logstash ``` -------------------------------- ### Run Tests with Full Trace Log on Staging Source: https://github.com/atlanhq/atlas-metastore/blob/master/dev-support/test-harness/README.md Execute tests on the 'staging' tenant and log the full request/response details to a JSONL file. ```bash python3 run.py --tenant staging --trace-log ```