Benthos is a messaging bridge service that supports a wide and growing list of input and output protocols.

A range of internal buffer strategies are available, allowing you to select a balance between latency, protection against back pressure and file based persistence, or nothing at all (direct bridge).

Benthos has inputs, an optional buffer, and outputs, which are all set in a single config file.

+--------------------------------------+
|            Input Stream              |
| ( ZMQ, NSQ, AMQP, Kafka, HTTP, ... ) |--+
+--------------------------------------+  |
                                          v
             +--------------------------------------------+
             |                   Buffer                   |
             | ( Memory-Mapped Files, Memory, None, etc ) |
             +--------------------------------------------+
                             |
                             |  +--------------------------------------+
                             +->|             Output Stream            |
                                | ( ZMQ, NSQ, AMQP, Kafka, HTTP, ... ) |
                                +--------------------------------------+

Currently supported input/output targets:

• ZMQ4 (PUSH, PULL, SUB, PUB)
• Nanomsg/Scalability Protocols (PUSH, PULL, SUB, PUB)
• RabbitMQ (AMQP)
• NSQ
• NATS
• Kafka
• HTTP 1.1 POST/GET
• STDIN/STDOUT
• File

You can also have multiple outputs or inputs by choosing a routing strategy (fan in, fan out, round robin, etc.)

For a full and up to date list you can print them from the binary:

# Print inputs, buffers and output options
benthos --print-inputs --print-buffers --print-outputs | less

Build with Go:

go get github.com/jeffail/benthos/cmd/...

Or, download from here.

benthos -c ./config.yaml

Check out the samples in ./config, or create a fully populated default configuration file:

benthos --print-yaml > config.yaml
benthos --print-json > config.json

The configuration file should contain a section for an input, output, and a buffer. For example, if we wanted to output to a ZMQ4 push socket our output section in a YAML config might look like this:

output:
  type: zmq4
  zmq4:
    addresses:
      - tcp://*:1234
    socket_type: PUSH

There are also configuration sections for logging and metrics, if you print an example config you will see the available options.

Benthos isn't doing much, so it's reasonable to expect low latencies and high throughput. Here's a table of benchmarks from an 4-core (2.4ghz) machine, bridging messages of 5000 bytes through a 500MB memory buffer via various protocols:

Take these results with a pinch of salt. I've added some benchmarking utilities in ./cmd/test, hopefully a third party can cook us up some more meaningful figures from a better set up than I have.

Benthos supports ZMQ4 for both data input and output. To add this you need to install libzmq4 and use the compile time flag when building benthos:

go install -tags "ZMQ4" ./cmd/...

Versions of go above 1.6 should automatically go get all vendored libraries. Otherwise, while cloning use --recursive:

git clone https://github.com/jeffail/benthos --recursive

Or, if the repo is already cloned, get the latest libraries with:

git submodule update --init

To add new libraries simply run:

PACKAGE=github.com/jeffail/util
git submodule add https://$PACKAGE vendor/$PACKAGE"

It might be handy to set yourself a function for this in your .bashrc:

function go-add-vendor {
	git submodule add https://$1 vendor/$1
}

There's a Dockerfile for creating a benthos docker image. This is built from scratch and so you'll need to build without CGO (CGO_ENABLED=0) for your benthos build to run within it. Create it like this:

CGO_ENABLED=0 make docker
docker run --rm benthos

Then use the image:

docker run --rm -v ~/benthos.yaml:/config.yaml -v /tmp/data:/data -p 8080:8080 \
	benthos -c /config.yaml