Nakadi Event Broker

https://github.com/zalando/nakadi
Admin
Nakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka...
Tokens:62,776
Snippets:462
Trust Score:7.8
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Running a Server

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/getting-started.md

Installs and starts a Nakadi container via the gradlew command.

```shell
./gradlew startNakadi

```

--------------------------------

### Stopping a Server

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/getting-started.md

Stops the running Nakadi service.

```shell
./gradlew stopNakadi

```

--------------------------------

### Consuming from the Beginning of a Partition

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-lola.md

Example of using the 'begin' offset in the 'X-Nakadi-Cursors' header to start consuming from the beginning of a partition.

```shell
curl -v http://localhost:8080/event-types/order_received/events \
  -H 'X-Nakadi-Cursors:[{"partition": "0", "offset":"begin"}]'
```

--------------------------------

### Install Ruby and Jekyll

Source: https://github.com/zalando/nakadi/blob/master/docs/README.md

Commands to install Ruby and Jekyll, and bundle dependencies for local documentation build.

```bash
sudo apt-get install ruby rube-dev 
gem install jekyll
bundle install
```

--------------------------------

### Consuming from Specific Partitions and Offsets

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-lola.md

Example of using the 'X-Nakadi-Cursors' header to specify partitions and starting offsets for consumption.

```shell
curl -v http://localhost:8080/event-types/order_received/events \
  -H 'X-Nakadi-Cursors: [{"partition": "0", "offset":"12"}]'
```

--------------------------------

### IDE Setup

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/developing.md

Gradle task for setting up the project for Eclipse IDE.

```bash
./gradlew eclipse
```

--------------------------------

### Consuming Events via GET Request

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-hila.md

Example of how to consume events by sending a GET request to the subscription's event resource.

```sh
curl -v -XGET "http://localhost:8080/subscriptions/038fc871-1d2c-4e2e-aa29-1579e8f2e71f/events"
```

--------------------------------

### Consuming from the beginning of a partition

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_concepts.md

Example of how to consume events from partition '0' starting from the very beginning of the stream using the 'BEGIN' offset value.

```shell
curl -v http://localhost:8080/event-types/order.ORDER_RECEIVED/events \
  -H 'X-Nakadi-Cursors:[{"partition": "0", "offset":"BEGIN"}]'
```

--------------------------------

### Start storage services

Source: https://github.com/zalando/nakadi/blob/master/README.md

Starts Docker Compose services for PostgreSQL, ZooKeeper, and Kafka, useful for development.

```sh
./gradlew startStorages
```

--------------------------------

### Consuming from specific partitions and offsets

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_concepts.md

Example of how to consume events from a specific partition ('0') starting at a particular offset ('12') using the X-Nakadi-Cursors header.

```shell
curl -v http://localhost:8080/event-types/order.ORDER_RECEIVED/events \
  -H 'X-Nakadi-Cursors: [{"partition": "0", "offset":"12"}]'
```

--------------------------------

### Create a Subscription

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-hila.md

Example of creating a subscription by posting to the /subscriptions collection resource.

```sh
curl -v -XPOST "http://localhost:8080/subscriptions" -H "Content-type: application/json" -d '{
    "owning_application": "order-service",
    "event_types": ["order.ORDER_RECEIVED"]
  }'
```

--------------------------------

### Start Nakadi Docker Compose

Source: https://github.com/zalando/nakadi/blob/master/docker-arm/README.MD

Start the Nakadi services using Docker Compose in the docker-arm directory. Allow time for services to initialize.

```bash
docker-compose up -d --build
```

--------------------------------

### Subscription Creation Response

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-hila.md

Example response when a subscription is created.

```json
HTTP/1.1 201 Created
Content-Type: application/json;charset=UTF-8

{
  "owning_application": "order-service",
  "event_types": [
    "order.ORDER_RECEIVED"
  ],
  "consumer_group": "default",
  "read_from": "end",
  "id": "038fc871-1d2c-4e2e-aa29-1579e8f2e71f",
  "created_at": "2016-09-23T16:35:13.273Z"
}
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/zalando/nakadi/blob/master/docs/README.md

Command to build the documentation and start the local web server.

```bash
bundle exec jekyll serve
```

--------------------------------

### Getting a Subscription Response

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-hila.md

Example JSON response for retrieving a single subscription.

```json
HTTP/1.1 200 OK
{
  "owning_application": "order-service",
  "event_types": [
    "order.ORDER_RECEIVED"
  ],
  "consumer_group": "default",
  "read_from": "end",
  "id": "038fc871-1d2c-4e2e-aa29-1579e8f2e71f",
  "created_at": "2016-09-23T16:35:13.273Z"
}
```

--------------------------------

### Checking Current Position

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-hila.md

Example of how to check the current position of a subscription by sending a GET request to the cursors resource.

```sh
curl -v -XGET "http://localhost:8080/subscriptions/038fc871-1d2c-4e2e-aa29-1579e8f2e71f/cursors"
```

--------------------------------

### Connecting to an Event Stream

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-lola.md

Example of how to connect to the stream of an Event Type using a GET request.

```shell
curl -v http://localhost:8080/event-types/order_received/events 
```

--------------------------------

### Docker environment setup command

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/faq.md

Command to evaluate the Docker machine environment variables, often needed when encountering Docker daemon issues.

```shell
eval "$(docker-machine env default)"
```

--------------------------------

### Running Tests

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/developing.md

Gradle commands for running unit and acceptance tests.

```bash
./gradlew build
./gradlew startNakadiForTest
./gradlew acceptance-test:test
```

--------------------------------

### Running Containers

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/developing.md

Gradle commands for managing Docker containers for Nakadi and its storage.

```bash
./gradlew startNakadi
./gradlew stopNakadi
./gradlew startStorages
```

--------------------------------

### Example Event Data

Source: https://github.com/zalando/nakadi/blob/master/core-services/src/test/resources/org/zalando/nakadi/partitioning/events.10000.A.txt

This is an example of event data in JSON format.

```json
{"color":"VbMBnIrjym","price":999,"name":"T","sku":"N","categoryId":5}
{"color":"nfseiRQRRc","price":326,"name":"g","sku":"u","categoryId":7}
{"color":"XuuWBXqAHu","price":88,"name":"n","sku":"j","categoryId":9}
{"color":"RKAvdYwXBI","price":608,"name":"A","sku":"a","categoryId":4}
{"color":"nlcdDfHfRR","price":559,"name":"s","sku":"r","categoryId":2}
{"color":"PmhdUrGJuS","price":734,"name":"a","sku":"C","categoryId":9}
{"color":"GqpljrWMJL","price":59,"name":"d","sku":"W","categoryId":4}
{"color":"sAlaUTedpo","price":217,"name":"B","sku":"q","categoryId":2}
{"color":"TJWDDzVVDz","price":42,"name":"b","sku":"p","categoryId":2}
{"color":"QrerYFpacI","price":904,"name":"f","sku":"T","categoryId":8}
{"color":"VxUETrbdyy","price":547,"name":"T","sku":"K","categoryId":5}
{"color":"muidgqanFt","price":514,"name":"O","sku":"v","categoryId":9}
{"color":"GgBjbfFxSd","price":70,"name":"A","sku":"A","categoryId":9}
{"color":"OOFinfKwDm","price":548,"name":"M","sku":"z","categoryId":3}
{"color":"DNXQzXXMel","price":137,"name":"R","sku":"x","categoryId":3}
{"color":"xWzDpsEEqm","price":790,"name":"y","sku":"r","categoryId":6}
{"color":"QbkLkvXmWM","price":319,"name":"K","sku":"T","categoryId":4}
{"color":"micxtTZnQd","price":348,"name":"H","sku":"t","categoryId":4}
{"color":"ShcssxopzE","price":999,"name":"u","sku":"M","categoryId":6}
{"color":"ZUsokrtIZY","price":902,"name":"C","sku":"F","categoryId":6}
{"color":"NeUvIAipMh","price":599,"name":"C","sku":"K","categoryId":7}
{"color":"QTiyHfEJhw","price":508,"name":"E","sku":"C","categoryId":2}
{"color":"zFLApvOAog","price":658,"name":"q","sku":"s","categoryId":2}
{"color":"XohkJeuAAl","price":26,"name":"s","sku":"J","categoryId":2}
{"color":"XVGPHWARZH","price":125,"name":"G","sku":"W","categoryId":5}
{"color":"DueJIFEGyC","price":436,"name":"e","sku":"R","categoryId":0}
{"color":"MXoXUCygnY","price":18,"name":"N","sku":"N","categoryId":0}
{"color":"MZuOGbXCAO","price":670,"name":"D","sku":"J","categoryId":6}
{"color":"rtyWMjiCmD","price":63,"name":"D","sku":"J","categoryId":5}
{"color":"bARxaKWDDf","price":3,"name":"z","sku":"E","categoryId":1}
{"color":"cwnTPcrTlz","price":103,"name":"q","sku":"n","categoryId":3}
{"color":"hXQDGHrWoN","price":448,"name":"U","sku":"J","categoryId":5}
{"color":"cOuiMuTLuu","price":450,"name":"D","sku":"P","categoryId":2}
{"color":"fJvXyuWpkc","price":275,"name":"j","sku":"s","categoryId":6}
{"color":"TJwPQZVaiR","price":907,"name":"R","sku":"g","categoryId":7}
{"color":"xkiqCBPYly","price":133,"name":"E","sku":"t","categoryId":4}
{"color":"nfREGIuyxZ","price":542,"name":"A","sku":"t","categoryId":6}
{"color":"ydItfeBzDA","price":458,"name":"U","sku":"W","categoryId":2}
{"color":"IzidjBHowj","price":628,"name":"Q","sku":"W","categoryId":1}
{"color":"desAABLOph","price":17,"name":"M","sku":"V","categoryId":4}
{"color":"lkbyjZFGAt","price":828,"name":"x","sku":"Y","categoryId":4}
{"color":"yMkJzxNNmG","price":364,"name":"B","sku":"g","categoryId":2}
{"color":"IJFWCSLRsI","price":824,"name":"N","sku":"w","categoryId":4}
{"color":"sVuPQJDllW","price":870,"name":"S","sku":"P","categoryId":7}
{"color":"MnZKjPrJnI","price":521,"name":"G","sku":"U","categoryId":1}
{"color":"CSIoVxLlVG","price":563,"name":"Y","sku":"s","categoryId":1}
{"color":"haDDAuponW","price":552,"name":"n","sku":"u","categoryId":9}
{"color":"FRjDnUZNIO","price":417,"name":"j","sku":"c","categoryId":3}
{"color":"SteWaBpciF","price":226,"name":"g","sku":"Y","categoryId":8}
{"color":"ihzlwTVjHG","price":385,"name":"z","sku":"n","categoryId":3}
{"color":"sPDlBdtcjC","price":327,"name":"f","sku":"N","categoryId":6}
{"color":"GOLdkxABfD","price":234,"name":"r","sku":"f","categoryId":9}
{"color":"kOqoJXwFXQ","price":560,"name":"R","sku":"g","categoryId":6}
{"color":"uKqPuLiLTo","price":866,"name":"M","sku":"Z","categoryId":5}
{"color":"oGAmnZurUc","price":154,"name":"P","sku":"c","categoryId":2}
{"color":"IKziWCOoSb","price":777,"name":"Y","sku":"Q","categoryId":9}
{"color":"RQXvFlLFaU","price":419,"name":"x","sku":"j","categoryId":4}
{"color":"cJLFPHqwyD","price":781,"name":"K","sku":"E","categoryId":4}
{"color":"JacRfFdWAM","price":952,"name":"c","sku":"m","categoryId":2}
{"color":"KynhVjNoFO","price":468,"name":"I","sku":"U","categoryId":3}
{"color":"woaEeNTIHP","price":929,"name":"R","sku":"b","categoryId":8}
{"color":"aeoQNmwCIo","price":745,"name":"Q","sku":"V","categoryId":4}
{"color":"VwAXNRyYKi","price":929,"name":"M","sku":"A","categoryId":7}
{"color":"UVEfCCXcVK","price":198,"name":"L","sku":"m","categoryId":1}
{"color":"rZOSHUwlPz","price":933,"name":"V","sku":"d","categoryId":3}
{"color":"tEsfkMVhxs","price":986,"name":"S","sku":"d","categoryId":0}
{"color":"CYTVfCyphc","price":137,"name":"x","sku":"y","categoryId":2}
{"color":"CmucQsdQlw","price":191,"name":"M","sku":"h","categoryId":9}
{"color":"JtMkWBbzqT","price":641,"name":"p","sku":"w","categoryId":3}
{"color":"YBidvRRlSa","price":170,"name":"c","sku":"r","categoryId":6}
```

--------------------------------

### Main Development Tasks

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/developing.md

Standard Gradle tasks for building and cleaning the project.

```bash
./gradlew build
./gradlew clean
```

--------------------------------

### Committing Cursors

Source: https://github.com/zalando/nakadi/blob/master/docs/_documentation/using_consuming-events-hila.md

Example of how to commit cursors by posting to the Subscription's cursor resource.

```sh
curl -v -XPOST "http://localhost:8080/subscriptions/038fc871-1d2c-4e2e-aa29-1579e8f2e71f/cursors"\
  -H "X-Nakadi-StreamId: ae1e39c3-219d-49a9-b444-777b4b03e84c" \
  -H "Content-type: application/json" \
  -d '{
    "items": [
      {
        "partition": "0",
        "offset": "543",
        "event_type": "order.ORDER_RECEIVED",
        "cursor_token": "b75c3102-98a4-4385-a5fd-b96f1d7872f2"
      },
      {
        "partition": "1",
        "offset": "923",
        "event_type": "order.ORDER_RECEIVED",
        "cursor_token": "a28568a9-1ca0-4d9f-b519-dd6dd4b7a610"
      }
    ]'
}
```
Tokens:
Raw