Consul - Micro Services Discovery

Consul Service Discovery

Consul is a HashiCorp tool for service discovery, service registry, and health checks.
Service Discovery and Configuration made easy, distributed, highly available and data center aware.

1. Service Discovery:Consul makes it simple for services to register themselves and to discover other services via a DNS or HTTP interface. Register external services such as SaaS providers as well.
1. DNS Query Interface: Look up services using Consul's built-int DNS server. Support existing infrastructure without any code change.
1. $ dig web-frontend.service.consul. ANY
2. Failure Detection: Pairing service discovery with health checking prevents routing requests to unhealthy hosts and enables services to easily provide circuit breakers.
3. Multi Datacenter: Consul scales to multiple datacenters out of the box with no compilcated configuration. lookup services in other datacenters, or keep the request local.
4. Key-value storage: Flexible key/value store for dynamic configuration, feature flagging, coordination, leader election and more, Long poll for near-instant notification of configuration changes.
1. Consul provide a hierarchy Key/Value store with a simple http API. Managing configuration has never been simpler.
1. $ consul kv put foo bar
2. $ consul kv get foo
3. $ consul kv get -detailed foo
4. $ consul kv delete foo

Consul is designed to be friendly to both the DevOps community and application developers, making it perfect for modern, elastic infrastructure.

Architecture

Every node that provides services to Consul runs a Consul agent. running an agent is not required for discovering other services or getting / setting key/value data. The agent is responsible for health checking the services on the node as well as the node itself.

The agent talk to one or more Consul Servers. The Consul server are where data is stored and replicated. The servers themselves elect a leader. While Consul can function with one server., 3 or 5  (2n +1) is recommended to avoid failure scenarios leading to data loss. A cluster of Consul servers is recommended for each data-center.

Components of your infrastructure that need to discover other services or nodes can query any of the Consul servers or any of the Consul agents. The agents forward queries to the servers automatically.

Each data-center runs a cluster of Consul servers. When a cross-data-center service discovery or configuration request is made, the local Consul servers forward the request to the remote data-center and return the result.

Glossary

• Agent: An agent is the long running daemon on every member of the Consul cluster. It is started by running $ consul agent. The agent is able to run in either client or server mode. Since all nodes must be running an agent., it is simpler to refer to the node as being either client or server, but there are other instances of the agent. All agents can run the DNS or HTTP interfaces and are responsible for running checks and keeping services in sync.
• Client: A client is an agent that forwards all RPCs to a server. The client is relatively stateless. The only background activity a client performs is taking part in the LAN gossip pool. This has a minimal resource overhead and consumes only a small amount of network bandwidth.
• Server: A server is an agent with an expanded set of responsibility including participating in the Raft quorum. maintaining cluster state, responding to RPC queries, exchanging WAN gossip with other data-centers and forwarding queries to leaders or remote data-centers.
• Data Center: While the definition of a data-center seems obvious, there are subtle details that must be considered. For example, in EC2, are multiple availability zones considered to comprise a single data-center? We define a data-center to be a networking environment that is private, low latency, and high bandwidth. This excludes communication that would traverse the public internet, but for our purposes multiple availability zones within a single EC2 region would be considered part of a single data-center.
• Consensus: Agreement upon the elected leader as well as agreement on the ordering of transactions. Since these transactions are applied to a finite-state machine, our defintion of consensus implies the consistency of a replicated state machine.
• Gossip: Gossip is built on top of Serf which provides a full gossip protocol that is used for multiple purposes. Serf provides membership, failure detection, and event broadcasting. It's enough to know that gossip involves random node-to-node communication, primarily over UDP.
• LAN Gossip: Refer to the LAN gossip pool which contains nodes that are all located on the same local area network or data-center.
• WAN Gossip: Refer to WAN gossip pool which contains only servers. These servers are primarily located in different data-centers and typically communicate over the internet or wide area network.
• RPC: Remote procedure call. This is a request / response mechanism allowing a client to make a request to server.

High Level Architecture Diagram

High Level Architecture Diagram

Let's break this image and describes each piece. First of all, we can see that there are 2 datacenters, labeled "one" and "two". Consul has first class support for multiple data-centers and expects this to be the common case.

Within each data-center, we have mixture of clients and servers. It is expected that there be between 3 to 5 servers. This strikes a balance between availability in the case of failure and performance, as consensus gets progressively slower as more machines are added. However, there is no limit to the number of clients, and they can easily scale into the thousands or tens of thousands.

All the nodes that are in a data-center participate in a gossip protocol. This means there is a gossip pool that contains all the nodes for a given datacenter. This servers a few purposes:

1.  there is no need to configure clients with the addresses of servers, discovery is done automatically.
2. The work of detecting node failures is not placed on the servers but is distributed, this makes failure detection much more scalable that naive heartbeating schemes.
3. It is used as a messaging layer to notify when important events such as leader election take place.

The servers in each datacenter are all part of a single Raft peer set. This means that they work together to elect a single leader, a selected server which has extra duties. This leader is responsible for processing all queries and transactions. Transactions must also be replicated to all peers as part of consensus protocol. Because of this requirement, when a non-leader server receives an RPC request, it forwards it to the cluster leader.

The server nodes also operates as part of a WAN gossip pool. This pool is different from the LAN pool as it is optimized for the higher latency of the internet and is expected to contain only other Consul server nodes. The purpose of this pool is to allow data-centers to discover each other in a low-touch manner. Bringing a new data-center online is as easy as joining the existing WAN gossip. Because the servers are all operating in this pool, it also enables cross-datacenter requests. When a server receives a request for a different data-center, it forwards it to a random server in the correct data-center. That server may then forward to the local leader.

"This result in a very low coupling between data-centers, but because of failure detection, connection caching and multiplexing , cross-data-center requests are relatively fast and reliable."

Consul Internals

CONSUL Commands (CLI)

Consul is controlled via  a very easy to use command-line interface (CLI). Consul is only a single command-line application: consul. This application then takes a subcommand such as "agent" or "members". It also responds to -h and --help, to get help for any specific command, pass the -h flag to the relevant subcommand, for example to see help about the join subcommand:

    $ consul join -h

Consul Setup

Consul Service Discovery tool example:

   # Run these on all consul servers.

   $sudo su

   $ apt-get update

   $ apt-get install git unzip -y

   $ cd /root

   $ wget https://dl.bintray.com/mitchellh/consul/0.5.2_linux_amd64.zip

   $ unzip 0.5.2_linux_amd64.zip

   $ rm -f 0.5.2_linux_amd64.zip

   $ mv consul /usr/bin/

   $ git clone https://github.com/anupkumardixit/consul_demo.git

   # Bootstrap / Web UI Server

   ---------------------------

   $ wget https://dl.bintray.com/mitchellh/consul/0.5.2_web_ui.zip

   $ unzip 0.5.2_web_ui.zip

   $ rm -f 0.5.2_web_ui.zip

   $ cd /root/consul_demo

   $ cp bootstrap.json config.json

   # Save this keygen! Note, if your key has a slash in it you need to escape them for setup.sh. Or just regenerate one until it doesn't have a slash :)

consul keygen

   # Non Boostrap Consul Server

   ---------------------------

   $ cd /root/consul_demo

   $ cp server.json config.json

  # Consul Agent Server

   ---------------------------

   $ apt-get install apache2 -y

   $ cd /root/consul_demo

   $ cp agent.json config.json

   $ ./setup.sh HOSTNAME ENCRYPT_KEY IP_OF_BOOTSTRAP IP_NON_BOOTSTRAP

nohup consul agent -config-dir /root/consul_demo/config.json &

# Now lets test on our agent server.

   $ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1

   $ curl http://localhost:8500/v1/kv/web/key1

   $ curl http://localhost:8500/v1/kv/web/key1?raw