Megamind

https://github.com/z5labs/megamind
Admin
Megamind is an eventually consistent Knowledge Graph construction service that provides a modern,...
Tokens:5,848
Snippets:43
Trust Score:7.4
License:Apache-2.0
Update:6 months ago
Show doc for...
Tokens:
Raw
### gRPC Client Usage (grpcurl)

Source: https://context7.com/z5labs/megamind/llms.txt

Instructions and examples for using the `grpcurl` command-line tool to interact with the Megamind gRPC ingest service.

```APIDOC
## gRPC Client Usage (grpcurl)

This section details how to use `grpcurl` to interact with the Megamind gRPC server.

### Prerequisites
- Ensure the Megamind gRPC server is running.
- Have `grpcurl` installed.

### Commands

1.  **Start the gRPC server:**
    ```bash
    ingest serve grpc --addr 0.0.0.0:8080
    ```

2.  **Ingest a single subgraph using `grpcurl`:**
    ```bash
    grpcurl -plaintext -d '{ "triples": [ { "subject": {"type": "Person", "tuid": "1"}, "predicate": {"name": "name"}, "object": {"string": "Alice"} } ] }' localhost:8080 proto.SubgraphIngest/IngestSubgraph
    ```

    **Expected output:**
    ```
    {} 
    ```
    (An empty `IngestResponse` indicates success.)
```

--------------------------------

### gRPC Client Usage (Go)

Source: https://context7.com/z5labs/megamind/llms.txt

Example Go code demonstrating how to connect to the Megamind gRPC server and ingest subgraphs using both unary and client-streaming calls.

```APIDOC
## gRPC Client Usage (Go)

This Go program demonstrates how to interact with the Megamind gRPC service for subgraph ingestion.

### Usage
1. Ensure the Megamind gRPC server is running.
2. Compile and run the Go program.

### Code Example
```go
package main

import (
    "context"
    "log"

    "github.com/z5labs/megamind/services/ingest/proto"
    "github.com/z5labs/megamind/subgraph"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func main() {
    // Connect to Megamind gRPC server
    conn, err := grpc.Dial("localhost:8080", grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("failed to connect: %v", err)
    }
    defer conn.Close()

    client := proto.NewSubgraphIngestClient(conn)
    ctx := context.Background()

    // Example 1: Ingest a single subgraph (unary call)
    sg := &subgraph.Subgraph{
        Triples: []*subgraph.Triple{
            {
                Subject:   &subgraph.Subject{Type: "Person", Tuid: "1"},
                Predicate: &subgraph.Predicate{Name: "name"},
                Object:    &subgraph.Object{Value: &subgraph.Object_String_{String_: "Alice"}},
            },
            {
                Subject:   &subgraph.Subject{Type: "Person", Tuid: "1"},
                Predicate: &subgraph.Predicate{Name: "age"},
                Object:    &subgraph.Object{Value: &subgraph.Object_Int64{Int64: 30}},
            },
        },
    }

    _, err = client.IngestSubgraph(ctx, sg)
    if err != nil {
        log.Fatalf("IngestSubgraph failed: %v", err)
    }
    log.Println("Single subgraph ingested successfully")

    // Example 2: Stream multiple subgraphs (client streaming)
    stream, err := client.Ingest(ctx)
    if err != nil {
        log.Fatalf("Ingest stream failed: %v", err)
    }

    subgraphs := []*subgraph.Subgraph{
        {Triples: []*subgraph.Triple{{Subject:   &subgraph.Subject{Type: "Person", Tuid: "2"}, Predicate: &subgraph.Predicate{Name: "name"}, Object:    &subgraph.Object{Value: &subgraph.Object_String_{String_: "Bob"}}}}}
        {Triples: []*subgraph.Triple{{Subject:   &subgraph.Subject{Type: "Person", Tuid: "3"}, Predicate: &subgraph.Predicate{Name: "name"}, Object:    &subgraph.Object{Value: &subgraph.Object_String_{String_: "Charlie"}}}}}
    }

    for _, sg := range subgraphs {
        if err := stream.Send(sg); err != nil {
            log.Fatalf("stream.Send failed: %v", err)
        }
    }

    _, err = stream.CloseAndRecv()
    if err != nil {
        log.Fatalf("stream.CloseAndRecv failed: %v", err)
    }
    log.Println("Batch subgraphs ingested successfully")
}
```
```

--------------------------------

### CLI Commands - Ingest Service

Source: https://context7.com/z5labs/megamind/llms.txt

Commands for starting the ingest service as either a gRPC or HTTP server.

```APIDOC
## CLI Commands

### Ingest Service Commands

The ingest service binary provides commands to start gRPC or HTTP servers for receiving subgraph data.

#### Start gRPC server

- **Default port (8080):**
  ```bash
  ingest serve grpc
  ```

- **Custom address:**
  ```bash
  ingest serve grpc --addr 0.0.0.0:9090
  ```

#### Start HTTP server

- **Default port (8080):**
  ```bash
  ingest serve http
  ```

- **Custom address:**
  ```bash
  ingest serve http --addr 0.0.0.0:9090
  ```
```

--------------------------------

### Example Subgraph in JSON Format

Source: https://context7.com/z5labs/megamind/llms.txt

Illustrates how a knowledge graph subgraph can be represented in JSON format, adhering to the defined Protobuf structure.

```json
{
  "triples": [
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "name"},
      "object": {"string": "Alice"}
    },
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "age"},
      "object": {"int64": 30}
    },
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "works_at"},
      "object": {"subject": {"type": "Company", "tuid": "100"}}
    }
  ]
}
```

--------------------------------

### Configure Terraform Deployment

Source: https://context7.com/z5labs/megamind/llms.txt

Example usage of the Megamind Terraform module for Kubernetes and Knative deployment.

```hcl
# terraform/main.tf example usage
module "megamind" {
  source = "./modules/megamind"

  # Kubernetes configuration is handled by the module
  # Knative Serving and Eventing are set up automatically
}

# The deployment includes:
# - Knative Operator (v1.7.0)
# - Knative Serving for auto-scaling
# - Knative Eventing for event-driven architecture
# - Kubernetes Dashboard (v2.6.1)
```

--------------------------------

### Ingest Service CLI Commands

Source: https://context7.com/z5labs/megamind/llms.txt

Commands to start the ingest service as either a gRPC or HTTP server. Allows specifying custom addresses and ports.

```bash
# Start gRPC server on default port (8080)
ingest serve grpc

# Start gRPC server on custom address
ingest serve grpc --addr 0.0.0.0:9090

# Start HTTP server on default port (8080)
ingest serve http

# Start HTTP server on custom address
ingest serve http --addr 0.0.0.0:9090
```

--------------------------------

### Define Subgraph Data Structure

Source: https://context7.com/z5labs/megamind/llms.txt

Example JSON format for representing triples in subgraphs.

```json
{"triples":[{"subject":{"type":"Person","tuid":"1"},"predicate":{"name":"name"},"object":{"string":"Alice"}}]}
{"triples":[{"subject":{"type":"Person","tuid":"2"},"predicate":{"name":"name"},"object":{"string":"Bob"}}]}
{"triples":[{"subject":{"type":"Person","tuid":"3"},"predicate":{"name":"name"},"object":{"string":"Charlie"}}]}
```

--------------------------------

### Implement SubgraphIngester Server

Source: https://context7.com/z5labs/megamind/llms.txt

Initializes the SubgraphIngester with both gRPC and HTTP handlers, including graceful shutdown logic.

```go
package main

import (
    "context"
    "net"
    "os"
    "os/signal"
    "syscall"

    "github.com/z5labs/megamind/services/ingest/grpc"
    "github.com/z5labs/megamind/services/ingest/http"
    "github.com/z5labs/megamind/services/ingest/ingest"
    "go.uber.org/zap"
)

func main() {
    logger, _ := zap.NewProduction()
    defer logger.Sync()

    // Create the core SubgraphIngester
    ingester := ingest.NewSubgraphIngester(logger)

    // Setup graceful shutdown
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    go func() {
        <-sigCh
        logger.Info("shutting down...")
        cancel()
    }()

    // Option 1: Start gRPC server
    grpcListener, _ := net.Listen("tcp", ":8080")
    go func() {
        if err := grpc.Serve(ctx, grpcListener, ingester); err != nil {
            logger.Error("gRPC server error", zap.Error(err))
        }
    }()

    // Option 2: Start HTTP server
    httpIngester := http.NewSubgraphIngester(logger, ingester)
    httpListener, _ := net.Listen("tcp", ":8081")
    if err := httpIngester.Serve(ctx, httpListener); err != nil {
        logger.Error("HTTP server error", zap.Error(err))
    }
}
```

--------------------------------

### Build All Megamind Components

Source: https://github.com/z5labs/megamind/blob/main/CONTRIBUTING.md

Use Bazel to build all components of the Megamind project. This command is used after cloning and setting up the repository.

```bash
bazel build //...
```

--------------------------------

### Go gRPC Client for Subgraph Ingestion

Source: https://context7.com/z5labs/megamind/llms.txt

Demonstrates how to connect to the Megamind gRPC server and ingest subgraphs using both unary and client-streaming calls. Ensure the gRPC server is running before executing.

```go
package main

import (
    "context"
    "log"

    "github.com/z5labs/megamind/services/ingest/proto"
    "github.com/z5labs/megamind/subgraph"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func main() {
    // Connect to Megamind gRPC server
    conn, err := grpc.Dial("localhost:8080", grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("failed to connect: %v", err)
    }
    defer conn.Close()

    client := proto.NewSubgraphIngestClient(conn)
    ctx := context.Background()

    // Example 1: Ingest a single subgraph (unary call)
    sg := &subgraph.Subgraph{
        Triples: []*subgraph.Triple{
            {
                Subject:   &subgraph.Subject{Type: "Person", Tuid: "1"},
                Predicate: &subgraph.Predicate{Name: "name"},
                Object:    &subgraph.Object{Value: &subgraph.Object_String_{String_: "Alice"}},
            },
            {
                Subject:   &subgraph.Subject{Type: "Person", Tuid: "1"},
                Predicate: &subgraph.Predicate{Name: "age"},
                Object:    &subgraph.Object{Value: &subgraph.Object_Int64{Int64: 30}},
            },
        },
    }

    _, err = client.IngestSubgraph(ctx, sg)
    if err != nil {
        log.Fatalf("IngestSubgraph failed: %v", err)
    }
    log.Println("Single subgraph ingested successfully")

    // Example 2: Stream multiple subgraphs (client streaming)
    stream, err := client.Ingest(ctx)
    if err != nil {
        log.Fatalf("Ingest stream failed: %v", err)
    }

    subgraphs := []*subgraph.Subgraph{
        {Triples: []*subgraph.Triple{{
            Subject:   &subgraph.Subject{Type: "Person", Tuid: "2"},
            Predicate: &subgraph.Predicate{Name: "name"},
            Object:    &subgraph.Object{Value: &subgraph.Object_String_{String_: "Bob"}},
        }}}, 
        {Triples: []*subgraph.Triple{{
            Subject:   &subgraph.Subject{Type: "Person", Tuid: "3"},
            Predicate: &subgraph.Predicate{Name: "name"},
            Object:    &subgraph.Object{Value: &subgraph.Object_String_{String_: "Charlie"}},
        }}},
    }

    for _, sg := range subgraphs {
        if err := stream.Send(sg); err != nil {
            log.Fatalf("stream.Send failed: %v", err)
        }
    }

    _, err = stream.CloseAndRecv()
    if err != nil {
        log.Fatalf("stream.CloseAndRecv failed: %v", err)
    }
    log.Println("Batch subgraphs ingested successfully")
}
```

--------------------------------

### Clone Megamind Repository

Source: https://github.com/z5labs/megamind/blob/main/CONTRIBUTING.md

Clone the Megamind repository to your local development machine. This is the first step in setting up the development environment.

```bash
git clone https://github.com/z5labs/megamind.git
```

--------------------------------

### HTTP REST API for Subgraph Ingestion

Source: https://context7.com/z5labs/megamind/llms.txt

Demonstrates ingesting a subgraph using an HTTP POST request. The request body must be JSON-formatted protobuf. Ensure the HTTP server is running.

```bash
# Start the HTTP server
ingest serve http --addr 0.0.0.0:8080

# Ingest a subgraph via curl
curl -X POST http://localhost:8080/subgraph/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "triples": [
      {
        "subject": {"type": "Person", "tuid": "1"},
        "predicate": {"name": "name"},
        "object": {"string": "Alice"}
      },
      {
        "subject": {"type": "Person", "tuid": "1"},
        "predicate": {"name": "email"},
        "object": {"string": "alice@example.com"}
      },
      {
        "subject": {"type": "Person", "tuid": "1"},
        "predicate": {"name": "follows"},
        "object": {"subject": {"type": "Person", "tuid": "2"}}
      }
    ]
  }'

# Expected: HTTP 200 on success, HTTP 500 on error
```

--------------------------------

### Run All Megamind Unit Tests

Source: https://github.com/z5labs/megamind/blob/main/CONTRIBUTING.md

Execute all unit tests within the Megamind project using Bazel. This command is essential for verifying code changes.

```bash
bazel test //...
```

--------------------------------

### Megamind CLI Tool for Dgraph Ingestion

Source: https://context7.com/z5labs/megamind/llms.txt

Utilities for direct ingestion to Dgraph using the megamind CLI tool. Supports ingesting from JSON files, stdin, and with protobuf encoding.

```bash
# Ingest subgraphs from a JSON file (one subgraph per line)
megamind dgraph ingest subgraphs ./data/subgraphs.json

# Ingest subgraphs from stdin
cat ./data/subgraphs.json | megamind dgraph ingest subgraphs -

# Ingest using protobuf encoding
megamind dgraph ingest subgraphs --encoding proto ./data/subgraphs.pb

# Example input file format (JSON Lines - one subgraph per line)
```

--------------------------------

### Subgraph Ingest gRPC Service Definition (Protobuf)

Source: https://context7.com/z5labs/megamind/llms.txt

Defines the gRPC service for ingesting subgraphs. Includes methods for single subgraph ingestion (unary) and batch ingestion (client streaming).

```protobuf
service SubgraphIngest {
  // Ingest a single subgraph (unary)
  rpc IngestSubgraph (subgraph.Subgraph) returns (IngestResponse);

  // Ingest multiple subgraphs via client streaming
  rpc Ingest (stream subgraph.Subgraph) returns (IngestResponse);
}

message IngestResponse {}
```

--------------------------------

### CLI Commands - Megamind CLI Tool

Source: https://context7.com/z5labs/megamind/llms.txt

Commands for using the Megamind CLI tool to ingest data directly into Dgraph.

```APIDOC
### Megamind CLI Tool

The megamind CLI tool provides utilities for direct ingestion to Dgraph from files or stdin.

#### Ingest subgraphs from a JSON file

- **JSON Lines format (one subgraph per line):**
  ```bash
  megamind dgraph ingest subgraphs ./data/subgraphs.json
  ```

#### Ingest subgraphs from stdin

```bash
cat ./data/subgraphs.json | megamind dgraph ingest subgraphs -
```

#### Ingest using protobuf encoding

```bash
megamind dgraph ingest subgraphs --encoding proto ./data/subgraphs.pb
```

#### Example input file format (JSON Lines)

```json
{
  "triples": [
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "name"},
      "object": {"string": "Alice"}
    }
  ]
}
{
  "triples": [
    {
      "subject": {"type": "Person", "tuid": "2"},
      "predicate": {"name": "name"},
      "object": {"string": "Bob"}
    }
  ]
}
```
```

--------------------------------

### grpcurl for gRPC Subgraph Ingestion

Source: https://context7.com/z5labs/megamind/llms.txt

Utilizes grpcurl to interact with the gRPC ingest service. This command-line tool allows for testing gRPC endpoints without writing client code. Ensure the gRPC server is running.

```bash
# Start the gRPC server
ingest serve grpc --addr 0.0.0.0:8080

# Ingest a single subgraph using grpcurl
grpcurl -plaintext -d '{
  "triples": [
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "name"},
      "object": {"string": "Alice"}
    }
  ]
}' localhost:8080 proto.SubgraphIngest/IngestSubgraph

# Expected output: {} (empty IngestResponse on success)
```

--------------------------------

### HTTP REST API - POST /subgraph/ingest

Source: https://context7.com/z5labs/megamind/llms.txt

Documentation for the HTTP REST API endpoint used to ingest a single subgraph.

```APIDOC
## HTTP REST API

### POST /subgraph/ingest

Ingests a single subgraph via HTTP POST. The request body must be JSON-formatted protobuf matching the Subgraph message structure.

#### Method
POST

#### Endpoint
`/subgraph/ingest`

#### Request Body
- **triples** (array[Subgraph.Triple]) - Required - An array of triples representing the subgraph.
  - **subject** (Subgraph.Subject) - Required - The subject of the triple.
    - **type** (string) - Required - The type of the subject.
    - **tuid** (string) - Required - The unique identifier for the subject.
  - **predicate** (Subgraph.Predicate) - Required - The predicate of the triple.
    - **name** (string) - Required - The name of the predicate.
  - **object** (Subgraph.Object) - Required - The object of the triple.
    - **string** (string) - Optional - String value for the object.
    - **int64** (integer) - Optional - Int64 value for the object.
    - **subject** (Subgraph.Subject) - Optional - Nested subject value for the object.

#### Request Example
```json
{
  "triples": [
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "name"},
      "object": {"string": "Alice"}
    },
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "email"},
      "object": {"string": "alice@example.com"}
    },
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "follows"},
      "object": {"subject": {"type": "Person", "tuid": "2"}}
    }
  ]
}
```

#### Response
- **Success Response (200 OK)**: Returns an empty JSON object `{}` on successful ingestion.
- **Error Response (500 Internal Server Error)**: Returned if an error occurs during ingestion.

#### Example Usage (curl)
```bash
curl -X POST http://localhost:8080/subgraph/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "triples": [
      {
        "subject": {"type": "Person", "tuid": "1"},
        "predicate": {"name": "name"},
        "object": {"string": "Alice"}
      },
      {
        "subject": {"type": "Person", "tuid": "1"},
        "predicate": {"name": "email"},
        "object": {"string": "alice@example.com"}
      },
      {
        "subject": {"type": "Person", "tuid": "1"},
        "predicate": {"name": "follows"},
        "object": {"subject": {"type": "Person", "tuid": "2"}}
      }
    ]
  }'
```
```

--------------------------------

### gRPC SubgraphIngest Service

Source: https://context7.com/z5labs/megamind/llms.txt

The SubgraphIngest service allows for the ingestion of knowledge graph data through unary or streaming gRPC calls.

```APIDOC
## gRPC SubgraphIngest

### Description
The SubgraphIngest service provides methods to ingest subgraphs into the Knowledge Graph. It supports both single subgraph ingestion and batch ingestion via client streaming.

### Methods
- **IngestSubgraph**: Unary call to ingest a single Subgraph.
- **Ingest**: Client-streaming call to ingest multiple Subgraphs.

### Request Body (Subgraph)
- **triples** (repeated Triple) - A collection of Triple messages representing knowledge relationships.

### Triple Structure
- **subject** (Subject) - The entity (type and tuid).
- **predicate** (Predicate) - The relationship name.
- **object** (Object) - The value or reference (Subject, string, int64, or float64).

### Request Example
{
  "triples": [
    {
      "subject": {"type": "Person", "tuid": "1"},
      "predicate": {"name": "name"},
      "object": {"string": "Alice"}
    }
  ]
}

### Response
#### Success Response
- **IngestResponse** (message) - Empty response indicating successful ingestion.
```

--------------------------------

### Subgraph Data Model Definition (Protobuf)

Source: https://context7.com/z5labs/megamind/llms.txt

Defines the structure for Subgraphs, Triples, Subjects, Predicates, and Objects used in Megamind. This schema is used for data serialization and communication.

```protobuf
message Subgraph {
  repeated Triple triples = 1;
}

message Triple {
  Subject subject = 1;
  Predicate predicate = 2;
  Object object = 3;
}

message Subject {
  string type = 1;  // Entity type (e.g., "Person", "Company")
  string tuid = 2;  // Type-unique identifier
}

message Predicate {
  string name = 1;  // Relationship name (e.g., "name", "works_at")
}

message Object {
  oneof value {
    Subject subject = 1;  // Reference to another entity
    string string = 2;    // String value
    int64 int64 = 3;      // Integer value
    double float64 = 4;   // Floating-point value
  }
}
```
Tokens:
Raw