Awesome Dashing dashboards with Icinga 2

vagrant_dashingWe at NETWAYS are using Dashing on our office dashboards already. This blog post solely targets integrating yet another new API providing data – the Icinga 2 REST API introduced in v2.4.

The following instructions were taken from the existing Vagrant boxes and their puppet manifests to allow faster installation. Doing it manually shouldn’t be an issue though ;-)

Requirements

Ensure that the following packages are installed, example for RHEL 7 with EPEL enabled:

package { [ 'rubygems', 'rubygem-bundler', 'ruby-devel', 'openssl', 'gcc-c++', 'make', 'nodejs' ]:
  ensure => 'installed',
  require => Class['epel']
}

Furthermore put a specific /etc/gemrc file which disables installing the documentation for gems – this can take fairly long and is not required by default. Especially not when provisioning a Vagrant box or a Docker container.

dashing-icinga2

I’ve created that project as demo for Icinga Camp Portland with the help of the existing Icinga 1.x dashing scripts from Markus, and a new job for fetching the Icinga 2 status data from its REST API.

Clone the git repository somewhere applicable. You don’t need any webserver for it, Dashing uses Thin to run a simple webserver on its own.

vcsrepo { '/usr/share/dashing-icinga2':
  ensure   => 'present',
  path     => '/usr/share/dashing-icinga2',
  provider => 'git',
  revision => 'master',
  source   => 'https://github.com/Icinga/dashing-icinga2.git',
  force    => true,
  require  => Package['git']
}

Install the dashing gem

The installation might take pretty long when it tries to install the gem’s documentation files. Therefore the flags “–no-rdoc” and “–no-ri” ensure that this isn’t done and only the dashing gem and its dependencies are installed into the system.

exec { 'dashing-install':
  path => '/bin:/usr/bin:/sbin:/usr/sbin',
  command => "gem install --no-rdoc --no-ri dashing",
  timeout => 1800
}

Install the gems for dashing-icinga2

Next to the dashing application itself the project requires additional gems, such as a rest client for communicating with the Icinga 2 REST API (check the Gemfile for details). Additionally the bundled gems are not installed into the system’s library but locally into the dashing-icinga2 git clone underneath the “binpaths” directory (this is to prevent conflicts with rubygem packages in the first place).

exec { 'dashing-bundle-install':
  path => '/bin:/usr/bin:/sbin:/usr/sbin',
  command => "cd /usr/share/dashing-icinga2 && bundle install --path binpaths", # use binpaths to prevent 'ruby bundler: command not found: thin'
  timeout => 1800
}

Dashing startup script

Put a small startup script somewhere executable to (re)start the Dashing application.

file { 'restart-dashing':
  name => '/usr/local/bin/restart-dashing',
  owner => root,
  group => root,
  mode => '0755',
  source => "puppet:////vagrant/files/usr/local/bin/restart-dashing",
}

Dashing runs as Thin process which puts its pid into the local tree. It is merely all about killing the process, removing the pid and then starting dashing again. “-d” puts the process into daemonize mode (not foreground) as well as “-p 8005” tells the application where to listen for browsers connecting to. Adjust that for your needs :)

#!/bin/bash

cd /usr/share/dashing-icinga2
kill -9 $(cat tmp/pids/thin.pid)
rm -f tmp/pids/thin.pid
/usr/local/bin/dashing start -d -p 8005

Now run Dashing.

exec { 'dashing-start':
  path => '/bin:/usr/bin:/sbin:/usr/sbin',
  command => "/usr/local/bin/restart-dashing",
  require => Service['icinga2'],
}

Configure the Icinga 2 API

The dashing job script just requires read-only access to the /v1/status endpoint. Being lazy I’ve just enabled everything but you should consider limited access :)

object ApiUser "dashing" {
  password = "icinga2ondashingr0xx"
  client_cn = NodeName
  permissions = [ "*" ]
}

Configure the Dashing job

There’s a bug in Dashing where job scripts ignore the settings from the config.ru file so there is no other way than to put the Icinga 2 REST API credentials and PKI paths directly into the jobs/icinga2.rb file.

$node_name = Socket.gethostbyname(Socket.gethostname).first
if defined? settings.icinga2_api_nodename
  node_name = settings.icinga2_api_nodename
end
#$api_url_base = "https://192.168.99.100:4665"
$api_url_base = "https://localhost:5665"
if defined? settings.icinga2_api_url
  api_url_base = settings.icinga2_api_url
end
$api_username = "dashing"
if defined? settings.icinga2_api_username
  api_username = settings.icinga2_api_username
end
$api_password = "icinga2ondashingr0xx"
if defined? settings.icinga2_api_password
  api_password = settings.icinga2_api_password
end

Modifications?

You really should know your HTML and Ruby foo before starting to modify the dashboards. The main widget used inside the dashboards/icinga2.erb file is “Simplemon” defined as data-view attribute. It is already provided inside the dashing-icinga2 repository. data-row and data-col define the location on the dashboard matrix.

    <li data-row="2" data-col="2" data-sizex="1" data-sizey="1">
      <div data-id="icinga-host-down" data-view="Simplemon" data-title="Hosts Down"></div>
    </li>

The important part is the data-id attribute – that’s the value coming from the icinga2 job defined in jobs/icinga2.erb.

The job update interval is set to 1 second in jobs/icinga2.erb:

SCHEDULER.every '1s' do

Connecting to the Icinga 2 REST API, fetching the status data as JSON and then iterating over these dictionaries is pretty straight forward. Additional programming examples can be found inside the Icinga 2 documentation.

Take the “hosts down” example from above:

hosts_down = status["num_hosts_down"].to_int

Now send the event to dashing by calling the send_event function providing the previosuly extracted value and the demanded color.

  send_event('icinga-host-down', {
   value: hosts_down.to_s,
   color: 'red' })

In case you’re wondering which values are fetched, let dashing run in foreground and print the “status” dictionary to get an idea about possible keys and values. Or query the Icinga 2 REST API with your own client first.

More?

You can play around with an already pre-installed environment inside the icinga2x Vagrant box and if you’re interested in an automated setup, check the puppet provisioner manifest.

Merry Xmas and a Happy New Year

After a pretty awesome but also exhausting year full of Icinga 2 & Icinga Web 2 we’re looking forward to some of course silent Christmas holidays :-)

icinga_org_wishes_for_2015Last year we asked you what you want to see in Icinga in 2015 – and we have to admit, announcing Icinga Director as config tool for Icinga 2 at OSMC, we’re pretty close in providing the most demanded one soon. You can follow the progress on the Git repository and the Redmine project. And of course, test it and provide feedback. We’re planning to have the first release with us at Icinga Camp Berlin on March 1st 2016, so keep your fingers crossed!

A Graphite Module for Icinga Web 2 is still in the early phasis, first things come first – releasing Icinga Web 2 in its first stable release, and a few months later, v2.1.0 already paving the way for the module api. There are also other Icinga Web 2 modules in development:

  • PNP for graph integration (this has been renamed from the previous one called “PNP4Nagios”)
  • Business Process
  • Generic TTS for ticket system integration
  • NagVis for map integration

You can test-drive their integration inside the Vagrant box, ready for your Christmas holidays. We plan to release after the holiday season, fixing the last bugs and also providing documentation.

While the Icinga 2 API was not the top-voted wish for 2015, we still just did it. A comprehensive core API with object creation/modification at runtime, access to all object attributes, advanced filters and secured access with SSL only. The Vagrant box also provides Dashing fully integrated using the Icinga 2 API. Go for your own projects during the holiday season and show it to the community in January, making the new year even more awesome :-)

2015 was not only releases and features – we announced “Icinga Partners & Services” offering professional services to many of you asking for it, and we also have our own Icinga shop offering Icinga swag – if you like it you should put a sticker on it!

Icinga received a Bossie Award, Bernd was interviewed by FLOSS Weekly and we created our icinga2 Docker container. We were also travelling a lot in 2015 – Icinga Camp Barcelona, Antwerp, Kuala Lumpur & Portland and other events. We are looking forward to meet you in 2016 again, perhaps San Francisco or Berlin :-) Last but not least – Icinga turned 6 years in May and we welcome lots of new team members.

Icinga Web 2.1.1, which was released today, has a little present for you inside the git repository, packages and Vagrant boxes: A Christmas theme :-) (thanks Eric & Tom!)

We wish you a merry Christmas and a happy new year! Now everyone:

I-cin-ga, I-cin-ga, Icinga all the way,
Oh! What fun it is to fork on all my cores at once.

Icinga at OSMC 2015

OSMC_Logo_Date_1024x432The Open Source Monitoring Conference celebrated their 10th year of community glory. We as team Icinga have a special relationship to this conference. In 2009 it was the first location where we announced the first stable Icinga version after forking from Nagios. I for myself was the first guy from Austria joining the team in Nuremberg in real life (“Oh, you are that tall?”). And we learned to love the conference and the community – heavily working on new releases to present and show-case on the first day, right before the evening event :-)

This year was even more special – the program was filled with many Icinga 2 related talks, and also Icinga team members holding a presentation: Bernd on Icinga, Tom Gelf on the Icinga Director, Tom De Vylder with Puppet magic and Bernd Ahlers talking Graylog.

We also had the fancy Vagrant boxes at our Icinga booth and during Bernds presentation. Coffee breaks, the evening event & the hackathon ensured to chat about Icinga 2, the API, Icinga Web 2 in friends & family atmosphere.

 

Icinga 2 & Icinga Web 2

Focusing on Icinga 2 and Icinga Web 2, Bernd took the challenge to present everything we released right before the conference. Starting with Icinga 2 v2.4, explaining the new Graphite schema and show-casing it using Grafana dashboards.

The API was certainly one of those “oh” – “ah” – “wow” moments in the audience: Creating, modifying and deleting hosts and service at runtime without a daemon reload. Bernd also prepared some slick REST API client commands and also triggered some actions (reschedule a check, or send a check result).

Next up: Icinga Web 2 v2.1.0 in a live demo with integrated PNP graphs, Business Processes and NagVis. Bernd answered many questions from the audience, be it “export to Excel”, adding new dashboards or using a config tool even.

 

Icinga Director

On the second day Tom talked about the Icinga Director – the upcoming configuration tool for Icinga 2. While it will solve the “problem” of having a configuration UI, it also enables you to automatically import external sources (PuppetDB, Foreman, CMDB, etc.). Diving directly into a magic live demo and users on Twitter already cloning the git repository. Test it while it is hot shaping it for its final release in 2016.

 

Lazy road to monitoring using Icinga 2 & Puppet

Tom De Vylder being one of the Icinga 2 Puppet module maintainers provided insights into the current state of development and gave a clear vision how to automate your monitoring environments.

 

OSMC Hackathon

From live hacking Icinga Web 2 modules to collecting ideas on generating configuration objects using Foreman and Icinga 2 API this was just one part of the hackathon on thursday. We’ve also discussed a Logstash output using the Icinga 2 API, and cluster enhancements for checking nodes. There was certainly space for more integration stuff, such as Graylog, Prometheus or fancy metric dashboards in Grafana.

 

See you soon …

More presentations about Monitoring at Spotify, Using Grafana, etc. can be found in the event archive.

After leaving an awesome conference we certainly look forward to 2016 when those ideas will hopefully result into community open source releases. We cannot wait for next years OSMC (29.11.-2.12.2016) including the 2nd hackathon :-)

PS: Icinga Camp Berlin 2016 is coming soon – pack your bags and join us for a day full of #monitoringlove.

 

Vagrant box playtime

vagrant_icingaweb2_dashboardWhile preparing for our OSMC booth and talk, we thought about enhancing the existing Vagrant boxes and include more demo cases. While the icinga2x-cluster boxes illustrate the cluster in a master-checker setup, the standalone box icinga2x focuses on a single Icinga 2 instance with Icinga Web 2 and the Icinga 2 API.

Alongside the Icinga 2 API and Icinga Web 2 there are numerous additions to the icinga2x Vagrant box:

 

PNP

vagrant_icinaweb2_detail_graphs_ttsPNP4Nagios is installed from the EPEL repository. The Icinga 2 Perfdata feature ensures that performance data files are written and the NPCD daemon updates the RRD files. Navigate to the host or service detail in Icinga Web 2 and watch the beautiful graphs. There’s also a menu entry in Icinga Web 2 providing an iframe to the PNP web frontend on its own.

 

GenericTTS

There are demo comments including a ticket id inside the Vagrant box. A simple script feeds them into the Icinga 2 API and the Icinga Web 2 module takes care of parsing the regex and adding a URL for demo purposes.

 

Business Process

vagrant_icingaweb2_business_processThe box provides 2 use cases for a business process demo: web services and mysql services. In order to check the MySQL database serving DB IDO and Icinga Web 2, we’ve added the check_mysql_health plugin (Icinga 2 v2.4 also provides a CheckCommand inside the ITL <plugins-contrib> already, so integration is a breeze).

These Icinga 2 checks come configured as Business Processes in the Icinga Web 2 module which also allows you to change and simulate certain failure scenarios. You’ll also recognise a dashboard item for the Top Level View allowing you to easily navigate into the BP tree and the host and service details. Pretty cool, eh?

 

NagVis

vagrant_icingaweb2_nagvisThe puppet module installs the latest stable NagVis release and configures the DB IDO as backend. The integration into Icinga Web 2 uses a newly developed module providing a more complete style and integrated authentication for the NagVis backend. Though there are no custom dashboards yet – send in a patch if you have some cool ones :)

 

 

Graphite

vagrant_graphite_webThe Graphite backend installation is helped with Puppet modules, the main difference is that Graphite Web VHost is listening on port 8003 by default (80 is reserved for Icinga Web 2). The carbon cache daemon is listening on 2003 where the Icinga 2 Graphite feature is writing the metrics to.

 

 

Grafana

vagrant_grafanaGrafana 2 uses Graphite Web as datasource. It comes preconfigured with the Icinga 2 dashboard providing an overview on load, http, mysql metrics and allows you to easily modify or add new graphs to your dashboard(s).

 

 

Dashing

vagrant_dashingWe’ve had a Dashing demo using the Icinga 2 API with us at Icinga Camp Portland though it required some manual installation steps. Since the Vagrant box already enabled the Icinga 2 API, the provisioner now also installs Dashing and the demo files. Note: Installing the Ruby gems required for Dashing might take a while depending on your internet connection. If Dashing is not running, call `restart-dashing`.

 

 

Playtime!

The icinga2x box requires a little more resources so make sure to have 2 cpu cores and 2 GB RAM available. You’ll need Vagrant and Virtualbox or Parallels installed prior to provision the box.

git clone https://github.com/Icinga/icinga-vagrant.git
cd icinga-vagrant/icinga2x
vagrant up

The initial provisioning takes a while depending on your internet connection.

Each web frontend is available on its own using the host-only network address 192.168.33.5:

Icinga Web 2 http://192.168.33.5/icingaweb2 icingaadmin/icinga
PNP4Nagios http://192.168.33.5/pnp4nagios
Graphite Web http://192.168.33.5:8003
Grafana 2 http://192.168.33.5:8004 admin/admin
Dashing http://192.168.33.5:8005

 

In the future we’ll add more Icinga Web 2 modules or other addons, just let us know what you want to play with or send a patch even :-)

Icinga 2 v2.4.1 bugfix release

Icinga StickerThis release fixes a problem when using recurring downtimes (“ScheduledDowntime”) causing Icinga 2 to crash on startup. There are further fixes for old compilers on Debian Squeeze, Ubuntu Precise and RHEL 6. The API setup wizard does not overwrite existing certificates anymore. The node setup wizards also incorporate the NodeName and ZoneName constant by default, not the previously used FQDN.

The ITL CheckCommand ‘running_kernel’ now allows you to optionally use the ‘running_kernel_use_sudo’ attribute. One further addition are global constants to fetch PlatformName. PlatformVersion, PlatformKernel and PlatformKernelVersion.

A common problem which we’ve analysed in our community support channels is the usage of existing SSL certificates with the Icinga 2 API. In case you are encountering the SSL error “SSL3_READ_BYTES:sslv3 alert unsupported certificate” when querying the API using curl or a modern browser, please ensure that the host’s SSL certificate version is 3, not 1. More details on the mailing lists.

Icinga 2 v2.4.1 packages should be available soon, meanwhile make sure to check the Changelog below.

 

Changelog

 

Changes

  • ITL
    • Add running_kernel_use_sudo option for the running_kernel check
  • Configuration
    • Add global constants: `PlatformName`. `PlatformVersion`, `PlatformKernel` and `PlatformKernelVersion`
  • CLI
    • Use NodeName and ZoneName constants for ‘node setup’ and ‘node wizard’

Feature

  • Feature 10622: Add by_ssh_options argument for the check_by_ssh plugin
  • Feature 10693: Add running_kernel_use_sudo option for the running_kernel check
  • Feature 10716: Use NodeName and ZoneName constants for ‘node setup’ and ‘node wizard’

Bugfixes

  • Bug 10528: Documentation example in “Access Object Attributes at Runtime” doesn’t work correctly
  • Bug 10615: Build fails on SLES 11 SP3 with GCC 4.8
  • Bug 10632: “node wizard” does not ask user to verify SSL certificate
  • Bug 10641: API setup command incorrectly overwrites existing certificates
  • Bug 10643: Icinga 2 crashes when ScheduledDowntime objects are used
  • Bug 10645: Documentation for schedule-downtime is missing required paremeters
  • Bug 10648: lib/base/process.cpp SIGSEGV on Debian squeeze / RHEL 6
  • Bug 10661: Incorrect web inject URL in documentation
  • Bug 10663: Incorrect redirect for stderr in /usr/lib/icinga2/prepare-dirs
  • Bug 10667: Indentation in command-plugins.conf
  • Bug 10677: node wizard checks for /var/lib/icinga2/ca directory but not the files
  • Bug 10690: CLI command ‘repository add’ doesn’t work
  • Bug 10692: Fix typos in the documentation
  • Bug 10708: Windows setup wizard crashes when InstallDir registry key is not set
  • Bug 10710: Incorrect path for icinga2 binary in development documentation
  • Bug 10720: Remove –master_zone from –help because it is currently not implemented