concepts

a stream of data is called a topic

a topic has one or more channels. Each channel receives a copy of all messages. In practice a channel maps to a downstream service consuming a topic.

a channel can have multiple clients connected which maps to the load balancing concept of a classic queue.

topics and channels are created at first use by publishing or subscribing.

a client is connected to all nsqd instaces that provide the given topic.

distributed

high-availability and no single point of failure

durability

by default messages are kept in memory but if --mem-queue-size is set to 0 all messages are persisted to disk

delivery

messages are delivered at least once. in failure scenarios messages can be delivered multiple times and client needs to take care.

order

you cannot rely on the order of messages

components

nsqd

the daemon that receives, queues, and delivers messages to clients

It can be run standalone but is normally configured in a cluster with nsqlookupd instance(s)

It listens on two TCP ports, one for clients and another for the HTTP API. It can optionally listen on a third port for HTTPS.

nsqlookupd

clients query nsqlookupd to discover which nsqd instance produces the given topic

nsqd broadcasts discovery information to nsqlookupd

nsqadmin

a Web UI to view aggregated cluster stats in realtime and perform various administrative tasks.

utilities

nsql_tail Consumes the specified topic/channel and writes to stdout

nsq_to_http Consumes the specified topic/channel and performs HTTP requests (GET/POST) to the specified endpoints

to_nsq takes a std stream, splits on newlines (or other) and publishes the strings to nsq as messages

starting a local cluster with docker

pull the image: docker pull nsqio/nsq

in the following commands replace the given IP address with your OWN IP ADDRESS

start three docker containers:

docker run -p 4160:4160 -p 4161:4161 nsqio/nsq /nsqlookupd
docker run -p 4150:4150 -p 4151:4151 nsqio/nsq /nsqd --lookupd-tcp-address=192.168.178.22:4160 --broadcast-address=192.168.178.22
docker run -p 4171:4171 nsqio/nsq /nsqadmin --lookupd-http-address=192.168.178.22:4161

now you can open the admin console: http://127.0.0.1:4171/

using the system

HTTP API

http can be used for publishing messages but not for consuming messages

publish a message:

curl -d "content of message" 127.0.0.1:4151/pub?topic=checkins

consuming messages from shell

install nsq\_tail

download: https://github.com/nsqio/nsq/releases

and sudo cp ~/Downloads/nsq-1.2.0.linux-amd64.go1.12.9/bin/nsq_tail /usr/local/bin

then:

nsq_tail --lookupd-http-address=127.0.0.1:4161 --topic=checkins

optionaly add 2> /dev/null to see message text only

client libraries

are listed here: https://nsq.io/clients/client_libraries.html

of course I prefer Go but here is consumer written in Java:

NSQLookup lookup = new DefaultNSQLookup();
lookup.addLookupAddress("localhost", 4161);
NSQConsumer consumer = new NSQConsumer(lookup, "speedtest", "dustin", (message) -> {
        System.out.println("received: " + message);            
        //now mark the message as finished.
        message.finished();
        
        //or you could requeue it, which indicates a failure and puts it back on the queue.
        //message.requeue();
});
        
consumer.start();