infra:opentsdb_grafana_metrics

OpenTSDB / Grafana metricsDB

This setup describes a single instance metric collection and graphing system based on opentsdb. It is capable of storing metrics with up to millisecond time stamp granularity. Withouth metric degeneration over time as you often see with classical rrdtool based systems.

The stack will consist of 3 major components.

Apache HBase™ is the Hadoop database. A non-relational, distributed, scalable, big data store. Capable of hosting very large tables – billions of rows X millions of columns – atop clusters of commodity hardware. A additional nice feature is, that it also allows to scale down to a single instance. Removing the need for setting up an entire hadoop ecosystem (HDFS & zookeeper).

A 'Time Series DataBase'. Consisting of a stateless TSD (Time Series Daemon) and associated commandline tools. Running on top of a Hbase database. As the TSD itself is stateless, you can run as many instances in parallel as required. So it is able to scale nicely with Hbase. The TSD provides a simple telnet-style protocol, HTTP API and built-in GUI.

A web based analysing tool capable of querying various data soures. Focusing on delivering 'beautifull' graps & dashboards'

Both Hbase & OpenTSDB are java based. So start with downloading and installing java8 or higher. We will use the standard openjdk that comes with Debian (as that is what I'm running). But should be available on any Linux distro.

  1. Download & install openjdk8
    sudo apt-get install openjdk8
  2. Download latest stable hbase release from http://ftp.nluug.nl/internet/apache/hbase/stable and untar to /opt. It should create a versioned hbase directory e.g. '/opt/hbase-1.2.5'. I like to create a symbolic link to it like '/opt/hbase', as this allows easy updates. But you could also move it to /opt/hbase or just leave it as is.
  3. Configure hbase tcp-ports & directories by editing '/opt/hbase/conf/hbase-site.xml' and '/opt/hbase/conf/hbase-env.sh'. See below for the settings I use. These are not strictly required. But then hbase would listen on non-standard ports and your data would get stored in /tmp. For several distro's this would mean that your data would not survive a reboot. So at least set the 'hbase.rootdir' and 'hbase.zookeeper.property.dataDir' properties and create both matching directories on your file system.
    hbase-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>hbase.rootdir</name>
        <value>file:///opt/hbase-data/hbase</value>
      </property>
      <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/opt/hbase-data/zookeeper</value>
      </property>
      <property> 
        <name>hbase.master.port</name>
        <value>60000</value>
      </property>
      <property>
        <name>hbase.master.info.port</name>
        <value>60010</value>
      </property>
      <property>
        <name>hbase.regionserver.port</name>
        <value>60020</value>
      </property>
      <property>
        <name>hbase.regionserver.info.port</name>
        <value>60030</value>
      </property>
    </configuration>
  4. Configure hbase to use the installed version of openjdk. Add or uncomment following line in file:
    hbase-env.sh
    export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  5. Start hbase
    /opt/hbase/bin/start-hbase.sh

    To make sure hbase has started and is running correctly you can connect with a webbrowser to it's web ui at port 60010.

  6. Download & install opentsdb from https://github.com/OpenTSDB/opentsdb/releases. Either as rpm or deb.
  7. Configure opentsdb to automatically create new metrics. Add following line to:
    /etc/opentsdb/opentsdb.conf
    tsd.core.auto_create_metrics=true
  8. Create the required tables for opentsdb in hbase
    COMPRESSION=SNAPPY HBASE_HOME=/opt/hbase /usr/share/opentsdb/tools/create_tables.sh
  9. Start opentsdb via it's sysv init script.
    service opentsdb start

    . To make sure OpenTSDB has started connect to it's web ui on port 4242. You should now be able to push data to it and generate basic graphs via opentsdb's built in web-ui. A good way to test is setting up opentsdb self monitoring by running following script.

    tsdmon.sh
    #!/bin/bash
    INTERVAL=15
    while :; do
      echo stats || exit
      sleep $INTERVAL
    done | nc -w 30 localhost 4242 \
        | sed 's/^/put /' \
        | nc -w 30 localhost 4242

    . When run this script will read all tsd stats and store them into opentsdb every 15 seconds.

  10. Download and install grafana (.rpm or .deb)
  11. Start the grafana service
  12. Login to the web console & define opentsdb (localhost:4242) as a datasource

Dowload & install grafana….?

  • infra/opentsdb_grafana_metrics.txt
  • Last modified: 10/10/2019 11:38
  • by harm