### Install Dependencies Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-streams-utils/provisioning/readme.md Install the necessary Python packages for the provisioning script. ```bash pip install boto3 pip install fire pip install tabulate ``` -------------------------------- ### Install Required Libraries Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-serverless-evaluator/README.md Install the necessary Python libraries for the script. Ensure you have Python 3.7+ and pip3 installed. ```bash pip3 install requirements.txt ``` -------------------------------- ### YAML Bulk Load Configuration Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/bulk-load-config.md Define required and optional parameters for bulk loading in this YAML format. Ensure camelCase is used for keys. ```yaml # Required parameters bucketName: my-neptune-data-bucket neptuneEndpoint: my-cluster.cluster-abc123.us-east-1.neptune.amazonaws.com iamRoleArn: arn:aws:iam::123456789012:role/NeptuneLoadFromS3Role # Optional parameters s3Prefix: neptune parallelism: OVERSUBSCRIBE monitor: true ``` -------------------------------- ### Install Python Handler with Provisioning Script Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-streams-utils/readme.md Use this script to install a Python handler into the Neptune Streams polling framework. Ensure the handler is built and uploaded to S3 first. ```bash python provision_neptune_streams_handler.py \ --cluster_id=neptunedbcluster-abcdefghijkl \ --handler_s3_bucket=my-bucket \ --handler_s3_key=streams/stream_handler.zip \ --region=us-east-1 ``` -------------------------------- ### Build the neo4j-to-neptune tool Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/readme.md Use this Maven command to clean and install the neo4j-to-neptune project, compiling the code and packaging it. ```bash mvn clean install ``` -------------------------------- ### Install Java Handler with Provisioning Script Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-streams-utils/readme.md Use this script to install a Java handler into the Neptune Streams polling framework. Ensure the handler is built and uploaded to S3 first. Specify the Lambda runtime if it's not the default. ```bash python provision_neptune_streams_handler.py \ --cluster_id=neptunedbcluster-abcdefghijkl \ --handler_s3_bucket=my-bucket \ --handler_s3_key=streams/stream_handler.jar \ --lambda_runtime=java8 \ --region=us-east-1 ``` -------------------------------- ### Bulk Load Process Output Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/bulk-load-config.md This output demonstrates the typical flow of the bulk load process, including file upload to S3, Neptune connectivity, and load status monitoring. ```text Vertices: 171 Edges : 253 Output : output/1751656971039 output/1751656971039 Completed in x second(s) S3 Bucket: my-bucket S3 Prefix: neptune AWS Region: us-east-2 IAM Role ARN: arn:aws:iam::123456789000:role/NeptunePolicy Neptune Endpoint: my-neptune-db.cluster-xxxxxxxxxxxx.us-east-2.neptune.amazonaws.com Bulk Load Parallelism: MEDIUM Uploading Gremlin load data to S3... Starting async upload of files from /tmp/output/1751656971039 to s3://my-bucket/neptune/1751656971039 Starting async upload of /tmp/output/1751656971039/vertices.csv to s3://my-bucket/neptune/1751656971039/vertices.csv Starting async upload of /tmp/output/1751656971039/edges.csv to s3://my-bucket/neptune/1751656971039/edges.csv Successfully uploaded vertices.csv - ETag: "abc123..." Successfully uploaded edges.csv - ETag: "def456..." Successfully uploaded 2 files from /tmp/output/1751656971039 Files uploaded successfully to S3. Files available at: s3://my-bucket/neptune/1751656971039/ Starting Neptune bulk load... Testing connectivity to Neptune endpoint... Successful connected to Neptune. Status: 200 healthy Neptune bulk load started successfully! Load ID: 12345678-1234-1234-1234-123456789012 Monitoring load progress for job: 12345678-1234-1234-1234-123456789012 Neptune bulk load status: LOAD_IN_PROGRESS Neptune bulk load status: LOAD_IN_PROGRESS Neptune bulk load completed with status: LOAD_COMPLETED ``` -------------------------------- ### Install Neptune Serverless Evaluator Dependencies Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt Installs the necessary Python packages for the Neptune Serverless Evaluator script. ```bash pip3 install -r neptune-serverless-evaluator/requirements.txt ``` -------------------------------- ### Clone Neptune Tools Repository Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Clone the amazon-neptune-tools repository to get started. ```bash git clone https://github.com/awslabs/amazon-neptune-tools.git ``` -------------------------------- ### StartsWith Operator Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Filter instances based on attribute values that begin with a specified prefix. ```json {"instanceId": {"startsWith": "neptune-db-"} } ``` -------------------------------- ### Install neptune-gremlin Package Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-gremlin-js/README.md Install the neptune-gremlin package using npm. This command is typically run in your project's root directory. ```sh npm install neptune-gremlin ``` -------------------------------- ### Neptune Bulk Load Process Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/readme.md This output demonstrates the steps involved in Neptune's bulk load process after data has been converted to CSV. It includes uploading data to S3, initiating the bulk load job, and monitoring its progress. ```text SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Sep. 04, 2025 12:14:34 P.M. com.amazonaws.services.neptune.io.Neo4jStreamWriter INFO: Successfully connected to Neo4j database at: bolt://localhost:7687 Sep. 04, 2025 12:14:34 P.M. com.amazonaws.services.neptune.io.Neo4jStreamWriter streamToFile INFO: Starting data export to file: /tmp/output/1757013274850/neo4j-stream-data-temp.csv Sep. 04, 2025 12:14:35 P.M. com.amazonaws.services.neptune.io.Neo4jStreamWriter streamToFile INFO: Successfully exported 1 records (425 lines) to file: /tmp/output/1757013274850/neo4j-stream-data-temp.csv Sep. 04, 2025 12:14:35 P.M. com.amazonaws.services.neptune.io.Neo4jStreamWriter close INFO: Neo4j driver closed successfully Vertices: 171 Edges : 253 Output : output/1757013274850 output/1757013274850 Completed in 0 second(s) S3 Bucket: my-bucket S3 Prefix: neptune AWS Region: us-west-2 IAM Role ARN: arn:aws:iam::123456789100:role/NeptuneReadS3 Neptune Endpoint: db-neptune-1.cluster-xxxxxxxxxxxx.us-west-2.neptune.amazonaws.com Neptune Port: 8182 Bulk Load Parallelism: OVERSUBSCRIBE Bulk Load Monitor: true Uploading Gremlin load data to S3... Starting sequential upload of files from /tmp/output/1757013274850 to s3://my-bucket/neptune/1757013274850 Uploading file 1 of 2: vertices.csv Starting upload with compression of /tmp/output/1757013274850/vertices.csv to s3://my-bucket/neptune/1757013274850/vertices.csv.gz File size: 7.2 KB Initiating Transfer Manager upload... Upload with compression completed for /tmp/output/1757013274850/vertices.csv Successfully uploaded vertices.csv (1/2) Uploading file 2 of 2: edges.csv Starting upload with compression of /tmp/output/1757013274850/edges.csv to s3://my-bucket/neptune/1757013274850/edges.csv.gz File size: 17.6 KB Initiating Transfer Manager upload... Upload with compression completed for /tmp/output/1757013274850/edges.csv Successfully uploaded edges.csv (2/2) Successfully uploaded all 2 files from /tmp/output/1757013274850 Files uploaded successfully to S3. Files available at: s3://my-bucket/neptune/1757013274850/ Starting Neptune bulk load... Testing connectivity to Neptune endpoint... Successfully connected to Neptune. Status: 200 healthy Neptune bulk load started successfully with load ID: a478c673-xxxx-xxxx-xxxx-77a70a430be8 Monitoring load progress for job: a478c673-xxxx-xxxx-xxxx-77a70a430be8 Neptune bulk load status: LOAD_IN_PROGRESS Neptune bulk load status: LOAD_IN_PROGRESS Neptune bulk load status: LOAD_IN_PROGRESS Neptune bulk load status: LOAD_IN_PROGRESS Neptune bulk load completed with status: LOAD_COMPLETED ``` -------------------------------- ### Validate Queries and Save to File Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/opencypher-compatability-checker/README.md Example of validating queries from 'queries.json' and saving the results to 'results.json'. ```bash java -jar NeptuneNeo4jMigrationHelper-1.0.jar --input queries.json --output results.json ``` -------------------------------- ### Example Output from openCypher Compatibility Checker Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt JSON structure representing the results of the openCypher compatibility check. Indicates whether each query is supported and provides details on unsupported functions or clauses. ```json // output (stdout or results.json) { "results": [ { "id": 1, "supported": true, "errorDefinitions": [] }, { "id": 2, "supported": false, "errorDefinitions": [ { "position": "line 1, column 8 (offset: 7)", "name": "apoc.coll.intersection", "replacement": "collintersection", "description": "apoc.coll.intersection is not supported in this release but try replacing with collintersection" } ] } ] } ``` -------------------------------- ### Validate Queries and Print to Console Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/opencypher-compatability-checker/README.md Example of validating queries from 'queries.json' and printing the results directly to the console. ```bash java -jar NeptuneNeo4jMigrationHelper-1.0.jar --input queries.json ``` -------------------------------- ### Example Input for openCypher Compatibility Checker Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt JSON structure for providing openCypher queries to the compatibility checker. Specifies the target system (NA for Neptune Analytics) and a list of queries with IDs. ```json // input.json { "targetSystem": "NA", "queries": [ { "id": 1, "query": "MATCH (n:Person) RETURN n LIMIT 10" }, { "id": 2, "query": "RETURN apoc.coll.intersection([1,2,3], [2,3,4])" } ] } ``` -------------------------------- ### Equals Operator Example (Explicit) Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Explicitly use the 'equals' operator to match an instance attribute to a specific value. ```json {"role": {"equals": "reader"} } ``` -------------------------------- ### Add Batches of Edges with Custom Token Mapping Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md This example shows how to add batches of edges when the input data is not formatted according to the Gremlin load format. A `TokenMapping` object is used to specify how to interpret the custom fields. ```python from neptune_python_utils.endpoints import Endpoints from neptune_python_utils.batch_utils import BatchUtils from neptune_python_utils.mappings import TokenMappings rows = [ {'ID': 'e-1', 'Label': 'FOLLOWS', 'FromVertex': 'person-3', 'ToVertex': 'person-7', 'strength': 10}, {'ID': 'e-2', 'Label': 'FOLLOWS', 'FromVertex': 'person-7', 'ToVertex': 'person-1', 'strength': 5}, {'ID': 'e-3', 'Label': 'FOLLOWS', 'FromVertex': 'person-7', 'ToVertex': 'person-2', 'strength': 7}, {'ID': 'e-4', 'Label': 'FOLLOWS', 'FromVertex': 'person-1', 'ToVertex': 'person-5', 'strength': 1}, {'ID': 'e-5', 'Label': 'FOLLOWS', 'FromVertex': 'person-5', 'ToVertex': 'person-1', 'strength': 12}, {'ID': 'e-6', 'Label': 'FOLLOWS', 'FromVertex': 'person-5', 'ToVertex': 'person-6', 'strength': 8}, {'ID': 'e-7', 'Label': 'FOLLOWS', 'FromVertex': 'person-4', 'ToVertex': 'person-6', 'strength': 8} ] tokens = TokenMappings(id_token='ID', label_token='Label', from_token='FromVertex', to_token='ToVertex') batch = BatchUtils(Endpoints()) batch.add_edges(batch_size=3, rows=rows, token_mappings=tokens) batch.close() ``` -------------------------------- ### OR Operator Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Use the 'or' operator to select instances that satisfy at least one of the specified conditions. ```json { "or": [ {"status": "available"}, {"availabilityZone": "eu-west-2"} ] } ``` -------------------------------- ### Provision Handler with Default Name Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-streams-utils/provisioning/readme.md Example of provisioning a Neptune Streams handler using default settings. Ensure the handler zip file is uploaded to the specified S3 bucket and key. ```python python provision_neptune_streams_handler.py --cluster_id=neptunedbcluster-xyz0a0a0abc \ --handler_s3_bucket=my-bucket \ --handler_s3_key=handlers/example_handler.zip \ --region=us-east-1 ``` -------------------------------- ### Initiate Data Load via Bulk Loader API Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Prepare-Data-Spotify.ipynb Starts a data load job using the Neptune bulk loader API. Ensure to replace placeholders with your specific S3 bucket, IAM role, and region details. ```bash DB_HOST= awscurl --service neptune-db \ -X POST \ -H 'Content-Type: application/json' \ https://$DB_HOST:8182/loader -d '{ "source": "s3:///", "format": "csv", "iamRoleArn": "arn:aws:iam:::role/", "mode": "AUTO", "region": "", "failOnError": "FALSE", "parallelism": "OVERSUBSCRIBE", "updateSingleCardinalityProperties": "TRUE" }' ``` -------------------------------- ### Bulk Load Data from S3 (Asynchronous with Status Check) Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Starts an asynchronous bulk data load from S3 and returns a BulkLoadStatus object to monitor progress. Allows fetching detailed status, errors, and waiting for completion. ```python from neptune_python_utils.bulkload import BulkLoad bulkload = BulkLoad( source='s3://ianrob-neptune-lhr/mysql-to-neptune/', update_single_cardinality_properties=True) load_status = bulkload.load_async() status, json = load_status.status(details=True, errors=True) print(json) load_status.wait() ``` -------------------------------- ### Successful Custom Endpoint Creation Log Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Example CloudWatch log output demonstrating the successful creation of a custom endpoint, including cluster ID, configuration, and instance details. ```log [INFO] cluster_id: ianrob-target [INFO] resource_prefix: ianrob- [INFO] config: {'customEndpoints': [{'customEndpoint': 'ianrob-all-instances', 'specification': {'endpointIsAvailable': True, 'instanceType': 'db.serverless'}}]} [INFO] cluster_metadata: {'clusterEndpoint': 'ianrob-target.cluster-abcdefghijkl.us-east-1.neptune.amazonaws.com', 'readerEndpoint': 'ianrob-target.cluster-ro-abcdefghijkl.us-east-1.neptune.amazonaws.com', 'instances': [{'instanceId': 'ianrob-target-instance-0', 'role': 'writer', 'endpoint': 'ianrob-target-instance-0.abcdefghijkl.us-east-1.neptune.amazonaws.com', 'status': 'available', 'endpointIsAvailable': True, 'availabilityZone': 'us-east-1c', 'instanceType': 'db.serverless', 'promotionTier': 1, 'tags': [{'key': 'Application', 'value': 'MyApp'}]}, {'instanceId': 'ianrob-target-instance-1', 'role': 'reader', 'endpoint': 'ianrob-target-instance-1.abcdefghijkl.us-east-1.neptune.amazonaws.com', 'status': 'available', 'endpointIsAvailable': True, 'availabilityZone': 'us-east-1b', 'instanceType': 'db.serverless', 'promotionTier': 1, 'tags': [{'key': 'Application', 'value': 'MyApp'}]}]} [INFO] Starting applying config... [INFO] Starting applying config for endpoint 'ianrob-all-instances' [INFO] specification: {'endpointIsAvailable': True, 'instanceType': 'db.serverless'} [INFO] conditions: [field 'endpointIsAvailable' == 'True'] AND [field 'instanceType' == 'db.serverless'] [INFO] instance_ids: ['ianrob-target-instance-0', 'ianrob-target-instance-1'] [INFO] excluded_ids: [] [INFO] Creating custom endpoint... [INFO] endpoint_id: ianrob-all-instances [INFO] member_instance_ids: ['ianrob-target-instance-0', 'ianrob-target-instance-1'] [INFO] excluded_ids: [] [INFO] Endpoint 'ianrob-all-instances' created [INFO] Finished applying config for endpoint 'ianrob-all-instances' [INFO] Finished applying config ``` -------------------------------- ### AND Operator Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Combine multiple conditions using the 'and' operator to select instances that meet all specified criteria. ```json { "and": [ {"role": "writer"}, {"status": "available"} ] } ``` -------------------------------- ### Spotify CSV to Neptune Graph Configuration Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt Example JSON configuration file for the csv_converter.py script, mapping Spotify CSV data to Neptune graph nodes and edges. Defines source files, node/edge properties, and relationships. ```json // data-config-spotify.json – full example mapping Spotify CSV to graph nodes/edges { "source_folder": ".source", "data_folder": ".data", "fileNames": ["spotify_songs.csv"], "nodes": [ { "csvFileName": "Track.csv", "select": "'track_name' in row and row['track_name'] != ''", "id": "uuid()", "label": "'Track'", "uniqueKey": "row['track_artist'] + '-' + row['track_name']", "properties": [ { "property": "track_name", "key": "track_name" }, { "property": "track_popularity", "key": "track_popularity" }, { "property": "danceability", "dataType": "Double", "key": "danceability" } ] }, { "csvFileName": "Artist.csv", "select": "'track_artist' in row and row['track_artist'] != ''", "id": "uuid()", "label": "'Artist'", "uniqueKey": "row['track_artist']", "properties": [ { "property": "name", "value": "row['track_artist']" } ] } ], "edges": [ { "csvFileName": "Track_Artist_Edges.csv", "select": "'track_name' in row and 'track_artist' in row", "id": "uuid()", "label": "'BY_ARTIST'", "from": "row['track_artist'] + '-' + row['track_name']", "to": "row['track_artist']", "fromLabel": "'Track'", "toLabel": "'Artist'", "properties": [] } ] } ``` -------------------------------- ### Configure Neptune Connection via Proxy Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Connect to Neptune through a proxy (e.g., bastion host, load balancer) by specifying the proxy's DNS and port. This example uses default proxy settings. ```python from neptune_python_utils.endpoints import Endpoints endpoints = Endpoints( neptune_endpoint='demo.cluster-111222333.eu-west-2.neptune.amazonaws.com', region_name='us-west-2', proxy_dns='localhost', proxy_port=8182) ``` -------------------------------- ### Batch Insert Vertices with Multi-Value String Properties Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md This example demonstrates inserting vertices with multi-value string properties using the ':String[]' syntax. Neptune supports set cardinality for vertex properties, so duplicate values will not be stored. ```python from neptune_python_utils.endpoints import Endpoints from neptune_python_utils.batch_utils import BatchUtils rows = [ {'~id': 'person-1', '~label': 'Person', 'name:String[]': 'test-1-name;test-1-name-x;test-1-name-x-2'}, {'~id': 'person-2', '~label': 'Person', 'name:String[]': 'test-2-name'}, {'~id': 'person-3', '~label': 'Person', 'name:String[]': 'test-3-name'}, {'~id': 'person-4', '~label': 'Person', 'name:String[]': 'test-4-name'}, {'~id': 'person-5', '~label': 'Person', 'name:String[]': 'test-5-name'}, {'~id': 'person-6', '~label': 'Person', 'name:String[]': 'test-6-name'}, {'~id': 'person-7', '~label': 'Person', 'name:String[]': 'test-7-name'} ] batch = BatchUtils(Endpoints()) batch.add_vertices(batch_size=3, rows=rows) batch.close() ``` -------------------------------- ### Start Neptune Export using AWS CLI Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/export-neptune-to-elasticsearch/readme.md Invoke the Lambda function using the AWS CLI to initiate the export process from Neptune. The function returns the AWS Batch job details for the export. ```bash aws lambda invoke \ --function-name arn:aws:lambda:eu-west-1:000000000000:function:export-neptune-to-kinesis-xxxx \ --region eu-west-1 \ /dev/stdout ``` -------------------------------- ### Complex specification for dynamic endpoint management Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt An example JSON configuration for the dynamic custom endpoints Lambda function, demonstrating complex 'and'/'or' logic and tag operators for specifying endpoint membership criteria. ```json // Example: complex specification with and/or/tag operators { "customEndpoints": [ { "customEndpoint": "my-cluster-analytics", "specification": { "and": [ { "or": [ { "role": "writer", "tags": { "all": [{"key":"App","value":"Sales"}] } }, { "role": "reader", "tags": { "all": [{"key":"App","value":"Sales"},{"key":"Group","value":"Reporting"}] } } ] }, { "endpointIsAvailable": true } ] } }, { "customEndpoint": "my-cluster-online", "specification": { "role": "reader", "instanceType": "db.r6g.xlarge", "tags": { "and": [ { "any": [{"key":"Group","value":"online"}] }, { "none": [{"key":"Cache","value":"cold"}] } ] } } } ] } ``` -------------------------------- ### Configure OpenSearch Serverless Data Access Policy Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/export-neptune-to-elasticsearch/readme.md When using with Amazon OpenSearch Serverless, add the LambdaExecutionRole from the CloudFormation stack to your Data Access Policy. This example shows the structure of the policy rules. ```json [ { "Rules": [ { "Resource": ["index/*/*"], "Permission": ["aoss:*"], "ResourceType": "index" } ], "Principal": [ "arn:aws:iam::000000000000:role/xxxx-LambdaExecutionRole-xxxx" ] } ] ``` -------------------------------- ### Display Help Message Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-gremlin/README.md Use the -h or --help flags to display the full help message, which lists all available command-line arguments and their descriptions. ```bash $ python3 csv-gremlin.py -h ``` -------------------------------- ### Basic Command Usage Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/opencypher-compatability-checker/README.md Use this command to run the compatibility checker. Replace with the actual JAR version. ```bash java -jar NeptuneNeo4jMigrationHelper-.jar --input [--output ] ``` -------------------------------- ### LessThan Operator Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Select instances where a numeric attribute is less than a specified value. ```json {"promotionTier": {"lessThan": 2} } ``` -------------------------------- ### GreaterThan Operator Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Select instances where a numeric attribute is greater than a specified value. ```json {"promotionTier": {"greaterThan": 1} } ``` -------------------------------- ### Run CSV Converter Help Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Prepare-Data-Spotify.ipynb Displays help information for the csv_converter.py script. ```python run -i csv_converter.py -h ``` -------------------------------- ### Establish Neptune Connection and Query Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md This snippet demonstrates how to establish a connection to Neptune using the Neptune Python Utils library and execute a Gremlin query. It assumes you have already obtained AWS credentials. ```python from neptune_python_utils.gremlin_utils import GremlinUtils from neptune_python_utils.endpoints import Endpoints import boto3 session = boto3.session.Session() credentials = session.get_credentials() endpoints = Endpoints(credentials=credentials) gremlin_utils = GremlinUtils(endpoints) conn = gremlin_utils.remote_connection() g = gremlin_utils.traversal_source(connection=conn) print(g.V().limit(10).valueMap().toList()) conn.close() ``` -------------------------------- ### Check On-Demand Provisioned Instance Cost vs. Neptune Serverless Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-serverless-evaluator/README.md Run the pricing script to compare the cost of Neptune Serverless with equivalent On-Demand provisioned instances. The script analyzes data from CloudWatch logs over a specified period. ```bash python pricing.py -n database-2-instance-1 -p 1 -r us-east-1 ``` -------------------------------- ### NotEquals Operator Example Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Use the 'notEquals' operator to exclude instances that match a specific attribute value. ```json {"role": {"notEquals": "reader"} } ``` -------------------------------- ### Initialize Glue Job and Gremlin Client Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Sets up the AWS Glue job context and initializes the GlueGremlinClient using connection information retrieved from AWS Glue connections. Ensure your Glue connection is of type 'JDBC' or 'NETWORK' and contains the Neptune endpoint. ```python import sys, boto3, os from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job from awsglue.transforms import ApplyMapping from awsglue.transforms import RenameField from awsglue.transforms import SelectFields from awsglue.dynamicframe import DynamicFrame from pyspark.sql.functions import lit from pyspark.sql.functions import format_string from gremlin_python import statics from gremlin_python.structure.graph import Graph from gremlin_python.process.graph_traversal import __ from gremlin_python.process.strategies import * from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection from gremlin_python.process.traversal import * from neptune_python_utils.glue_neptune_connection_info import GlueNeptuneConnectionInfo from neptune_python_utils.glue_gremlin_client import GlueGremlinClient from neptune_python_utils.glue_gremlin_csv_transforms import GlueGremlinCsvTransforms from neptune_python_utils.endpoints import Endpoints from neptune_python_utils.gremlin_utils import GremlinUtils args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATABASE_NAME', 'NEPTUNE_CONNECTION_NAME', 'AWS_REGION', 'CONNECT_TO_NEPTUNE_ROLE_ARN']) sc = SparkContext() glueContext = GlueContext(sc) job = Job(glueContext) job.init(args['JOB_NAME'], args) database = args['DATABASE_NAME'] product_table = 'salesdb_product' product_category_table = 'salesdb_product_category' supplier_table = 'salesdb_supplier' # Create Gremlin client gremlin_endpoints = GlueNeptuneConnectionInfo(args['AWS_REGION'], args['CONNECT_TO_NEPTUNE_ROLE_ARN']).neptune_endpoints(args['NEPTUNE_CONNECTION_NAME']) gremlin_client = GlueGremlinClient(gremlin_endpoints) # Create Product vertices print("Creating Product vertices...") # 1. Get data from source SQL database datasource0 = glueContext.create_dynamic_frame.from_catalog(database = database, table_name = product_table, transformation_ctx = "datasource0") datasource1 = glueContext.create_dynamic_frame.from_catalog(database = database, table_name = product_category_table, transformation_ctx = "datasource1") datasource2 = datasource0.join( ["CATEGORY_ID"],["CATEGORY_ID"], datasource1, transformation_ctx = "join") # 2. Map fields to bulk load CSV column headings format applymapping1 = ApplyMapping.apply(frame = datasource2, mappings = [("NAME", "string", "name:String", "string"), ("UNIT_PRICE", "decimal(10,2)", "unitPrice", "string"), ("PRODUCT_ID", "int", "productId", "int"), ("QUANTITY_PER_UNIT", "int", "quantityPerUnit:Int", "int"), ("CATEGORY_ID", "int", "category_id", "int"), ("SUPPLIER_ID", "int", "supplierId", "int"), ("CATEGORY_NAME", "string", "category:String", "string"), ("DESCRIPTION", "string", "description:String", "string"), ("IMAGE_URL", "string", "imageUrl:String", "string")], transformation_ctx = "applymapping1") # 3. Append prefixes to values in ID columns (ensures vertices for different types have unique IDs across graph) applymapping1 = GlueGremlinCsvTransforms.create_prefixed_columns(applymapping1, [('~id', 'productId', 'p'),('~to', 'supplierId', 's')]) # 4. Select fields for upsert selectfields1 = SelectFields.apply(frame = applymapping1, paths = ["~id", "name:String", "category:String", "description:String", "unitPrice", "quantityPerUnit:Int", "imageUrl:String"], transformation_ctx = "selectfields1") # 5. Upsert batches of vertices selectfields1.toDF().foreachPartition(gremlin_client.upsert_vertices('Product', batch_size=100)) ``` -------------------------------- ### Compare Neptune Serverless Costs Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt Compares Neptune Serverless costs against On-Demand provisioned instances for a specified period. Requires the instance name, period in days, and region. ```python python neptune-serverless-evaluator/pricing.py \ -n database-2-instance-1 \ -p 1 \ -r us-east-1 ``` ```python python neptune-serverless-evaluator/pricing.py \ -n my-neptune-instance \ -r eu-west-1 ``` -------------------------------- ### Build the Project Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-gremlin-js/README.md Build the neptune-gremlin project. This command runs ESLint, copies the JavaScript file to the CDK app's Lambda folder, and synthesizes the CDK application. ```sh npm run build ``` -------------------------------- ### Run CSV Conversion with Configuration Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/conversion-config.md Specify a YAML file for advanced conversion rules using the --conversion-config parameter. ```bash java -jar neo4j-to-neptune.jar convert-csv \ -i /path/to/neo4j-export.csv \ -d /path/to/output \ --conversion-config conversion-config.yaml \ [other options] ``` -------------------------------- ### Build Lambda Code Package Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/export-neptune-to-elasticsearch/lambda/README.md Execute this script to create the ZIP file for the Lambda functions. The output is placed in the 'target' directory. ```shell sh build.sh ``` -------------------------------- ### Get All Nodes and Edges Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-gremlin-js/README.md Retrieve all nodes and edges present in the Neptune graph. This is useful for a full graph export or initial inspection. ```javascript const searchResult = await connection.search({}) ``` -------------------------------- ### Equals Operator Example (Implicit) Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md The 'equals' operator is implicit for most attributes. This is a simpler way to express the same condition as the explicit 'equals' operator. ```json {"role": "reader" } ``` -------------------------------- ### Convert, Upload to S3, and Load to Neptune Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt Perform a full conversion, upload to S3, and trigger a Neptune bulk load in a single command. Requires input CSV, output directory, Neptune endpoint, S3 bucket, IAM role, and type inference. ```bash java -jar neo4j-to-neptune.jar convert-csv \ --input /tmp/neo4j-export.csv \ --dir output \ --neptune-endpoint my-cluster.cluster-abc123.us-east-2.neptune.amazonaws.com \ --bucket-name my-neptune-bucket \ --iam-role-arn arn:aws:iam::123456789012:role/NeptuneLoadFromS3 \ --infer-types \ --parallelism OVERSUBSCRIBE \ --monitor ``` -------------------------------- ### Get Bulk Load Job Status and Errors Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Prepare-Data-Spotify.ipynb Retrieves detailed status and error information for a specific bulk load job using its ID. ```bash LOAD_ID=ff90fd71-66c5-427b-be1f-f95ded45ae70 DB_HOST= awscurl --service neptune-db \ -X GET \ https://$DB_HOST:8182/loader/$LOAD_ID?details=true&errors=true ``` -------------------------------- ### Verify Directory Contents Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Prepare-Data-Spotify.ipynb Prints the current working directory and lists the contents of the .source and .data directories. ```bash pwd ls echo '.source' ls -l .source echo '.data' ls -l .data ``` -------------------------------- ### Convert CSV and Bulk Load using Configuration File Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/readme.md Initiate the CSV conversion and Neptune bulk load process by referencing a YAML configuration file containing all necessary settings. ```bash java -jar neo4j-to-neptune.jar convert-csv \ --input /path/to/neo4j-export.csv \ --dir /path/to/output \ --bulk-load-config bulk-load-config.yaml \ [additional options] ``` -------------------------------- ### Download Tinkerpop Modern GraphML Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/graphml2csv/README.md Fetches the Tinkerpop modern graph data in GraphML format using curl. This file is used as an example input for the conversion script. ```bash $ curl https://raw.githubusercontent.com/apache/tinkerpop/master/data/tinkerpop-modern.xml -o tinkerpop-modern.xml ``` -------------------------------- ### Display Graph Notebook Configuration Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Spotify-Data-Query.ipynb Execute this magic command to display the current configuration of the graph notebook. ```python %graph_notebook_config ``` -------------------------------- ### Deploy CDK Test App Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-gremlin-js/README.md Deploy the associated AWS CDK test application. This involves building the project, installing dependencies, bootstrapping CDK, synthesizing, and deploying the stack. ```sh npm run build cd cdk-test-app npm install cd lambda npm install cd .. npx cdk bootstrap npx cdk synth npx cdk diff npx cdk deploy ``` -------------------------------- ### Convert and bulk load using configuration file (short flags) Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/readme.md This command converts Neo4j CSV data and initiates a Neptune bulk load using shorthand flags for input and output directory, and a configuration file for bulk load parameters. ```bash java -jar neo4j-to-neptune.jar convert-csv \ -i /tmp/neo4j-export.csv \ -d output \ --bulk-load-config bulk-load-config.yaml \ --infer-types ``` -------------------------------- ### Convert CSV and Bulk Load using CLI Parameters Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/readme.md Use this command to convert Neo4j CSV files and initiate a bulk load into Neptune by providing all necessary configurations via command-line arguments. ```bash java -jar neo4j-to-neptune.jar convert-csv \ --input /path/to/neo4j-export.csv \ --dir /path/to/output \ --neptune-endpoint your-cluster.cluster-abc123.region.neptune.amazonaws.com \ --bucket-name your-s3-bucket \ --iam-role-arn arn:aws:iam::account:role/YourNeptuneRole \ [additional options] ``` -------------------------------- ### Access Denied Error Message Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Example CloudWatch log message indicating an 'AccessDenied' error when attempting to create a DB cluster endpoint, likely due to insufficient IAM permissions. ```log An error occurred (AccessDenied) when calling the CreateDBClusterEndpoint operation: User: arn:aws:sts::... is not authorized to perform: rds:CreateDBClusterEndpoint on resource: arn:aws:rds:::cluster-endpoint:... because no identity-based policy allows the rds:CreateDBClusterEndpoint action ``` -------------------------------- ### Basic Bulk Load Configuration Command Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/bulk-load-config.md Use this command to initiate the CSV conversion and bulk load process into Neptune, specifying a YAML configuration file. ```bash java -jar neo4j-to-neptune.jar convert-csv \ -i /tmp/neo4j-export.csv \ -d output \ --bulk-load-config bulk-load.yaml ``` -------------------------------- ### Export all data from Neo4j to CSV Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/readme.md This Cypher command uses the APOC library to export all nodes and relationships from Neo4j to a single CSV file. Ensure APOC is installed and exports are enabled in the configuration. ```cypher CALL apoc.export.csv.all( "neo4j-export.csv", {d:','} ) ``` -------------------------------- ### Edge ID Transformation using Neo4j Format References Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/conversion-config.md Transform edge IDs using a template that references original Neo4j edge properties like type, start, and end IDs. ```yaml edgeIdTransformation: ~id: "{_type}_{_start}_{_end}" ``` -------------------------------- ### Find Playlists by Genre Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Spotify-Data-Query.ipynb This Gremlin query finds playlists belonging to a specific genre. It starts by finding the genre by name, then traverses the 'HAS_GENRE' edge to find the associated playlists, and returns their names. ```gremlin %%gremlin // find playlists for Genre g.V().hasLabel('Genre').has('name', 'pop') .in('HAS_GENRE') .valueMap('name') ``` -------------------------------- ### Run CSV Converter with Configuration Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Prepare-Data-Spotify.ipynb Executes the csv_converter.py script with a specified data configuration file and enables S3 integration. ```python %run -i csv_converter.py -v ./data-config-spotify.json --s3 ``` -------------------------------- ### Find Tracks by Artist Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/notebooks/Spotify-Data-Query.ipynb This Gremlin query finds all tracks associated with a specific artist. It starts by finding the artist by name and label, then traverses the 'BY_ARTIST' edge to find the tracks, and finally retrieves the 'track_name' property. ```gremlin %%gremlin // given an artist, find tracks g.V() .has('name', 'Martin Garrix') .hasLabel('Artist').as('art') .in('BY_ARTIST').as('trk') .valueMap('track_name') ``` -------------------------------- ### Connect to Neptune Database Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-gremlin-js/README.md Establish a connection to your Neptune database instance. Ensure NEPTUNE_ENDPOINT, NEPTUNE_PORT, and USE_IAM environment variables are set correctly. ```javascript const gremlin = require("neptune-gremlin") // Get configuration values from the environment const host = process.env.NEPTUNE_ENDPOINT const port = process.env.NEPTUNE_PORT const useIam = process.env.USE_IAM === "true" // Create a new connection to the Neptune database const connection = new gremlin.Connection(host, port, {useIam}) await connection.connect() ``` -------------------------------- ### Alternative YAML Format for List Input Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/conversion-config.md Use a compact format for lists within the configuration file to avoid long bullet lists. ```yaml skipVertices: byId: ["vertex_123","vertex_456","user_999"] ``` -------------------------------- ### CSV Converter Command Line Help Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/csv-to-neptune-bulk-format/README.md Displays the help message for the csv_converter.py script, outlining available arguments and their usage. ```bash py .\csv_converter.py -h System File Encoding: cp1252 usage: csv_converter.py [-h] [-e LOCAL_ENC] [--s3] [--dup] [-v] DATA_CONF_FILE [DATA_CONF_FILE ...] Copyright 2018 Amazon.com, Inc. or its affiliates. Licensed under the Apache License 2.0 http://aws.amazon.com/apache2.0/ positional arguments: DATA_CONF_FILE Space separated, one or more Data Configuration File(s) (json) Use separate files if Node/Edges are different. optional arguments: -h, --help show this help message and exit -e LOCAL_ENC, --enc LOCAL_ENC Optional: encoding for the source files 'utf-8' or 'cp1252' --s3 Use S3 as source and destination of files --dup Generate file for duplicates -v, --verbose Emit Verbose logging A utility python script to convert CSV data file into the Amazon Neptune CSV format for bulk ingestion. See https://docs.aws.amazon.com/neptune/latest/userguide/bulk-load-tutorial- format-gremlin.html. ``` -------------------------------- ### Get Neptune Connection Info from Glue Data Catalog Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Use this snippet to retrieve Neptune connection information directly from the AWS Glue Data Catalog. This is particularly useful within AWS Glue jobs where connection details are managed centrally. ```python import sys from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job from neptune_python_utils.glue_neptune_connection_info import GlueNeptuneConnectionInfo from neptune_python_utils.endpoints import Endpoints args = getResolvedOptions(sys.argv, ['JOB_NAME', 'AWS_REGION', 'CONNECT_TO_NEPTUNE_ROLE_ARN']) sc = SparkContext() glueContext = GlueContext(sc) job = Job(glueContext) job.init(args['JOB_NAME'], args) region = args['AWS_REGION'] role_arn = args['CONNECT_TO_NEPTUNE_ROLE_ARN'] endpoints = GlueNeptuneConnectionInfo(region, role_arn).neptune_endpoints('neptune-db') ``` -------------------------------- ### Connect and Query using Environment Variables Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Connects to the Gremlin endpoint using NEPTUNE_CLUSTER_ENDPOINT and NEPTUNE_CLUSTER_PORT environment variables. Automatically uses IAM DB Auth if enabled. ```python from neptune_python_utils.gremlin_utils import GremlinUtils GremlinUtils.init_statics(globals()) gremlin_utils = GremlinUtils() conn = gremlin_utils.remote_connection() g = gremlin_utils.traversal_source(connection=conn) print(g.V().limit(10).valueMap().toList()) conn.close() ``` -------------------------------- ### Build neo4j-to-neptune CLI Tool Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt Build the neo4j-to-neptune Java CLI tool using Maven. This tool converts Neo4j APOC CSV exports or live streams into Neptune Gremlin bulk-load CSV format. ```bash cd neo4j-to-neptune && mvn clean install ``` -------------------------------- ### Trust Relationship for Neptune Access Role (Glue Job) Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Example trust relationship policy for a Neptune access role, allowing a specific AWS Glue job's IAM role to assume it. This enables the Glue job to connect to Neptune. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111111111111:role/MyGlueIAMRole" }, "Action": "sts:AssumeRole" } ] } ``` -------------------------------- ### Basic Conversion Configuration Structure Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neo4j-to-neptune/docs/conversion-config.md Define label mappings, ID transformations, and filtering rules in a YAML file. ```yaml # Label Mappings vertexLabels: OldVertexLabel: NewVertexLabel Person: Individual Company: Organization Product: Item edgeLabels: OLD_RELATIONSHIP: NEW_RELATIONSHIP WORKS_FOR: EMPLOYED_BY LIVES_IN: RESIDES_IN KNOWS: CONNECTED_TO # ID Transformations vertexIdTransformation: ~id: "{template}" edgeIdTransformation: ~id: "{template}" # Filtering Rules skipVertices: byId: - "vertex_id" byLabel: - "LabelName" skipEdges: byLabel: - "RELATIONSHIP_TYPE" ``` -------------------------------- ### Neptune Access Role Policy for IAM DB Auth Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Example IAM policy statement for a Neptune access role, granting permission to connect to a Neptune database using IAM DB authentication. Restrict the resource ARN for enhanced security. ```json { "Version": "2012-10-17", "Statement": [ { "Action": "neptune-db:connect", "Resource": "arn:aws:neptune-db:eu-west-1:111111111111:*/*", "Effect": "Allow" } ] } ``` -------------------------------- ### Glue Job IAM Role Policy to Assume Neptune Access Role Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/neptune-python-utils/readme.md Example inline policy to attach to an AWS Glue job's IAM role, granting permission to assume the Neptune access role. This is necessary for the Glue job to establish a connection. ```json { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::111111111111:role/GlueConnectToNeptuneRole", "Effect": "Allow" } ] } ``` -------------------------------- ### Configure Neptune Bulk Load with YAML Source: https://context7.com/awslabs/amazon-neptune-tools/llms.txt Use a YAML configuration file to specify settings for Neptune bulk loading, including S3 bucket, Neptune endpoint, IAM role, parallelism, and monitoring. This simplifies complex command-line arguments. ```yaml # bulk-load-config.yaml: # bucket-name: "my-neptune-bulk-load-bucket" # neptune-endpoint: "my-cluster.cluster-abc123.us-east-1.neptune.amazonaws.com" # iam-role-arn: "arn:aws:iam::123456789012:role/NeptuneLoadFromS3Role" # parallelism: "OVERSUBSCRIBE" # monitor: true ``` ```bash java -jar neo4j-to-neptune.jar convert-csv \ --input /tmp/neo4j-export.csv \ --dir output \ --bulk-load-config bulk-load-config.yaml \ --infer-types ``` -------------------------------- ### Build Lambda Function Source: https://github.com/awslabs/amazon-neptune-tools/blob/master/dynamic-custom-endpoints/readme.md Build the Lambda function using the provided build script. Ensure you are using Python 3.9. ```bash cd amazon-neptune-tools/dynamic-custom-endpoints/lambda sh build.sh ```