Archive for the 'Technical' Category

Page 3 of 4

Playing with IDOUtils, Oracle and ocilib

Published by Michael Friedrich on 6. October 2009 in Core, Development, Misc and Technical. 0 Comments Tags: Database, Development, Icinga, IDOUtils, libdbi.

Since I’m very busy at work this time I just wanted to give you some feedback on my current attempt – IDOUtils with Oracle based on ocilib.

Regarding that libdbi currently doesn’t provide a stable Oracle driver and furthermore, the main interface doesn’t allow prepared statements and parameter bindings, ocilib should be used and enabled during configure.

Currently, I am preparing a version which works as an addon to IDOUtils with libdbi, meaning that the libdbi still has to be installed and optional ocilib. Oracle support with ocilib will be enabled using –enable-oracle during configure.

Configure modifications

I have chosen to let ido2db link against ocilib at runtime which gives the benefit of only loading the ocilib and based on that the Oracle libs. The configure flag enables the include directory /usr/local/include where ocilib.h resides and the library directory /usr/local/lib with ocilib. In the first initial version, the path will be hard coded, later I will implement configure flags just as –with-ocilib-lib/inc for more customizations.

The code within configure also enables the include statement for ocilib.h and the #define USE_ORACLE which will be used in the Code to point to ocilib DB handling instead of libdbi – just as NDOUtils Oracle does.

OCILIB_INC="/usr/local/include" OCILIB_LIB="/usr/local/lib" ... AC_DEFINE(HAVE_OCILIB_H,"1") ... AC_DEFINE_UNQUOTED(USE_ORACLE)

Initscript modifications

Meanwhile because of the linking at runtime it is required to hand the library path over to ido2db since it’s not in systems default library path. If not exported before running ido2db within the init script, error will look like this:

# /etc/init.d/ido2db start Starting ido2db:/opt/icinga/bin/ido2db: error while loading shared libraries: libocilib.so.3: cannot open shared object file: No such file or directory done.

And start of ido2db simply fails because of the missing LD_LIBRARY_PATH. This path is defined running configure, so I have modified the rc.ido2db.in to export this environment variable with the concurrent ocilib library path.

#add ocilib lib path to link at runtime if enabled LD_LIBRARY_PATH=@ido2db_runtime@ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH

@ido2db_runtime@ will be defined running configure, the basic setting is done in configure.in (and running autoconf 2.61 afterwards to generate configure script).

dnl create ido2db_runtime for rc.ido2db.in ido2db_runtime="$OCILIB_LIB" AC_SUBST(ido2db_runtime)

RHEL’s autoconf is quite old (2.59) so I compiled my own static version using this article.
The reason why I am working on RHEL is quite simple – it’s in a VM and already configured with NDOUtils Oracle so the working environment (Oracle DB connection, etc) is already there :-)

Future plans

So for the first part, ocilib initialization during build is enabled and working. My next steps are

* adapt DB init, connection, passing queries to ocilib
* add Oracle based MERGE queries for the ON DUPLICATE KEY MySQL queries
* rewrite heavy queries to prepared statements/parameter bindings
* improve housekeeping DELETE sequences (since ido2db dies regularly when deleting more thant 500k queries at startup…)

Stay tuned! =)

Icinga 0.8.4 is out – bugfixes and updates for IDOUtils and Docu

Published by Michael Friedrich on 16. September 2009 in Core, Development, Misc, Releases and Technical. 2 Comments Tags: 0.8.4, Database, Development, Docbook, Download, Icinga, IDOUtils, libdbi, PostgreSQL, release, Sourceforge.

Since we decided to give you the opportunity to test the new IDOUtils with re-engineered code for more RDBMs and Postgres support too we were aware of the fact that there would have been bugfixes to apply ;-)

There were several issues to fix, not only Postgres but also typos and rewritten queries. We also added some improvements to the code, to mention configure support for manual libdbi detection and also the new version handling for IDOUtils. It’s now the same als Icinga Core, the old fashioned 1.4b8 has been dropped since we implemented a lot of new stuff to the original code.

Following the Changelog what’s been done so far:

idoutils: fix failure_prediction_options in tables hosts, services for postgres
idoutils: add configure support for manual libdbi detection
idoutils: fix typo laste_state (wrong column)
idoutils: fix buffer freeing with NAGIOS_SIZEOF_ARRAY
idoutils: delete old mysql files coming from ndoutils
idoutils: add new version handling, idomod/ido2db now shows version of Icinga core
idoutils: rename existing mysql/pgsql files, added upgrade/fix files
idoutils: fix wrong typecast in servicestatusdata_add, no more segfault while query preparing
idoutils: add experimental db trimming interval option in ido2db.cfg
core: fix wrong copyright url for Ethan
docu: fix quickinstall guides

Those changes and improvements to the code are worth a new release!

Thanks for testing and reporting bugs and improvements to our mailinglists, dev tracker and here in the blog! =)

PS: Concerning Oracle, I’ve started preparing the code in another branch (edit Makefile for ocilib, rewrite db connection, add parameter bindings for queries and so on). I hope this will be done within the next few weeks and maybe we’ll catch up with another Icinga release before 1.0 alpha :-)

A peek behind the Icinga web interface

Published by Marius Hein on 7. September 2009 in Misc and Webinterface. 7 Comments

You’ve probably already seen the paparazzi shots, but the real sneak preview is here. In the lead up to the Alpha release, we’ve put a lot of work in to the web interface – so we want your thoughts on our progress and plans. So, the Icinga web interface in a nutshell:

Ajax paradigm

We believe the major advantage of an Ajax driven web interface is that refreshes are only made to the relevant page areas as opposed to the entire page. Thus program load is significantly reduced while being also much more flexible. We expect this to also lead to faster search and configuration change times.

Widget architecture

We took some inspiration from iGoogle and Apple dashboards to design the web interface with flexibility and user customisation in mind. So Icinga’s interface is a loose collection of components we call Cronks which sit in a few containers (border, north, east and centre columns) on an open layout.

Cronks can be dragged and dropped around the screen, shown, hidden, minimised, resized and customised. Essentially widgets, these mini Javascript applications interact with the server to receive data and other information on user demand. They generally display specific and minimal information, which makes them also easily extensible. We like them because they break down check results into manageable chunks to be displayed when wanted, instead of overwhelming the user with a screen full of 20 different pieces of blinking information.

So these Cronks can be dragged and dropped into panels, tabs and containers much alike gadgets on an iGoogle page or Apple dashboard, offering the user flexible control over their interface.

Growl-like pop up notifications

Similar to the Apple notifier, balloons emerge on the browser margins to directly inform the user of changes. This kind of global notification system offers multiple notifications from multiple sources – which we believe is perfect for monitoring.

ExtJS, Json and CSS

Behind all the features on screen we have endeavoured to create a flexible and open layout from the outset which has been easily achieved with CSS and ExtJS Javascript library. In our opinion, ExtJS is one of the best libraries around, offering many features, while being easily extensible and independent of external libraries. Json was naturally our choice for data exchange between the server (PHP, Agavi), framework and Cronks, being itself native Javascript.

The thinking behind the Icinga web interface was flexibility, speed and customisability. We wanted to give the user the freedom to mould their interface to suit their needs. Let us know if you think we’re on the right track in the comments.

Playing with IDOUtils and Postgresql

Published by Michael Friedrich on 1. September 2009 in Core, Development and Technical. 1 Comment Tags: Database, Development, Icinga, IDOUtils, libdbi.

After a long time debugging, code re factoring, implementing and testing new stuff another success story for Icinga!

Let’s start with the details :-)

IDOUtils is a “simple” kind of INSERT, UPDATE and DELETE application so no worries about difficult SELECT and JOIN. Meanwhile the biggest problem are the queries which are not normalized and truly MySQL specific. Since Hendrik provided the normalization of the INSERT queries I stepped further into the INSERT OR UPDATE queries. Those are non-standard and i.e. Postgres or Oracle do not support them.

Oracle queries can be taken from NDOUtils Oracle mostly using a trick to MERGE statements (props to David Schmidt!), but the main goal is to rewrite all of them to fit parameter bindings. Check this article for more information.

Postgres is a bit more difficult because MERGE or UPSERT (another approach to INSERT OR UPDATE) is not yet supported. The documentation points to a function written in plpgsql.

The biggest problem – how to break up the code and create functions which are deciding upon RDBM which query to build and execute?

My first approach using on single function which gets different queries as string and decides what to do was a nice try but simply failed looking at the design – being generic and making the code more readable, too.

So I tried it the hard way and split the code for the SQL-queries from dbhandler.c into a new file called dbqueries.c – for the INSERT OR UPDATE queries for the first part.

One single function for each query has been implemented and performs then the RDBM specific query. For Postgres I decided to try an UPDATE in the first place and if there are now rows affected an INSERT will be issued instead. Unique constraints defined in table scheme are matching the UPDATE constraints.

Furthermore there were other issues to fix for Postgres – string escaping, Unix time conversion functions and the task of getting the last insert ID. MySQL doesn’t use defined sequence ids (dunno yet how last insert id works…) but Postgres does that for each table. After a hard night debugging I fixed that too – now the table relationship is working for Postgres too.

Concerning the fact that each value is stored in the data array it’s quite simple to implement more queries for more RDBMs – MySQL and Postgres will take care of giving you examples on how to build the queries :-)

You’re very welcome to test Postgresql in upcoming Icinga 0.8.3!

Please report bugs or feature request to our mailinglists and at our development section.

Looking forward to Oracle and ocilib =)

Paparazzi shots of Icinga´s Webinterface

Published by Julian Hein on 27. August 2009 in Misc and Webinterface. 16 Comments

Some Paparazzi shots have been discovered, of something that might or might not be Icinga’s next web interface. Though we cannot confirm their authenticity, we are restrained from withholding them.

Let us know what you make of it in the comments.

Ode to Agavi

Published by Marius Hein on 28. July 2009 in API and Misc. 4 Comments Tags: agavi, API, Icinga.

Closing into the 14 day rush to the next Icinga release 0.8.2, we had a moment to reflect on our coding madness. As a developer, the one thing that makes such work easier is the choice of a good framework. And that is why we take our hats off to our trusty Agavi framework.

Why Agavi? It was no hard choice. As an open source, PHP 5, object oriented framework, it met all our web interface needs.

Thanks to its very strict Model-view-controller architectural pattern, we have very clean, testable and easily extensible code. By separating user input from application actions and from the interface view, there is much less complexity in the design architecture which leads to greater flexibility and easier extension. Less complexity and not to mention PHP open Icinga to a much larger developer community.

Extensibility is also built in through its modular structure which is perfect for all the Icinga addons that will no doubt come in time. Addons can be easily integrated into the frontend, added on as modules.

Icinga Web Architecture – a home for an Addon

Unlike other frameworks which restrict which components can be used, Agavi is highly flexible and leaves most implementation choices the developer. So at the end of the day we have complete control over the code. It’s independence from database abstractions and multi-language support further enhances to Icinga’s flexibility, making it accessible to diverse users.

As Agavi is an active project, we trust it can only get better. Most recently, Agavi’s improvements to its build system and its integration of the Phing (Phing is not GNU make) makes coding that bit faster. And if you notice that ever ticking counter on the right, you would understand why.

Playing with IDOUtils, libdbi & Oracle

Published by Michael Friedrich on 14. July 2009 in Core, Development and Technical. 5 Comments Tags: contribute, Database, Development, Icinga, IDOUtils, libdbi.

First of all, since I became one of the maintainers for NDOUtils Oracle I’ve been looking forward to commit this project back to the upstream. It is a bit tricky because current code has been merged from NDOUtils 1.4.b8 (partly) and it is using a self written Library for OCI (Oracle Call Interface, liboci). Even though SQL-Queries and the DB layout are slightly modified to fit to Oracle.

Focusing resources on Icinga and IDOUtils with the best source base so far includes using the libdbi as a database abstraction layer.

libdbi

So far, libdbi is written in C and consists of two modules:

libdbi (API which is called from inside IDOUtils)
libdbi-drivers (driver modules for each RDBM desired, loaded by libdbi)

Currently, the libdbi contains stable drivers for

Firebird/Interbase
FreeTDS (provides access to MS SQL Server and Sybase)
MySQL
PostgreSQL
SQLite/SQLite3

Not completed drivers are

Testing the Oracle driver

As a matter of fact, I wanted to test the Oracle driver and it turned out to be really “not completed”. What I did (a complete description can be found here):

First step was to download and install the Oracle Instantclient and the SDK (remote Oracle DB). Next step: Install the libdbi. The libdbi needs to be installed first – if you try the drivers first, configure will fail because of missing dbi.h!

And the biggest part: The libdbi driver for Oracle. While the source in the tar-ball contains a typo in dbd_oracle.c the CVS snapshot had problems with automake on RHEL x64. Nice isn’t it? After having that fixed configure needs to be pointed to the Oracle lib and inc dir from Instant Client. Then I slightly modified the DB scheme (kicked the table prefix, renamed one table and all primary keys to fit the maximum of 30 characters (Oracle!)). Also hacked some C-Code into IDOUtils to be ready for the first test (only a “Hello World”-SQL-Query).

Starting IDOUtils with Oracle config, getting nice error messages and not that good memory address exceptions on the Oracle server. Meaning, the libdbi driver for Oracle doesn’t even work with one single test query :-(

Future Outlook

For that reason I have been looking forward towards another solution. Reading the libdbi mailinglists there has been a discussion about rewriting the Oracle driver with ocilib. Concerning the fact that libdbi API doesn’t even support parameter bindings (which are essential for performance) it would be very tricky to get a working solution as soon as possible.

ocilib looks very nice and is a very mature and stable project which is why we are planning to build IDOUtils with Oracle support on that base. This will be less work than a libdbi driver but also enough to implement, test, debug and hopefully release soon.

Meanwhile our focus will relay on implementing IDOUtils support for the stable libdbi drivers just as Postgres or SQLite.

If anyone reading this and having knowledge of normalizing the SQL queries feel free to contact us! It will be very much appreciated getting more DB knowledge into this project! :-)

Annoying ido2db init script

Published by Hendrik Bäcker on 1. July 2009 in Core and Development. 1 Comment

*grmpf* – is just a short word of my actual thinking.

Many people are missing a stable way to start or stop the database daemon, sometimes it doesn’t clean up the socket pipe or you have to handle different command lines to kill it of – quite annoying isn’t it?

I decided to solve this with a semi-good new init script, dealing with configure and Makefiles to create and install it in a easy to use way but… what the hell?

The IDO2DB Daemon cares about a kill signal (SIGTERM for default) but his childs, one for each client connection doesn’t. This results in waise childs reading from a deleted socket… not realy usefull.

The actual git master now has a child signal fix for ido2db and a brand new working init script.

I hope someone else is as happy as I am about this.

Vote for your favorite database

Published by Bernd Erk on 8. June 2009 in Core and Technical. 9 Comments Tags: Database, Icinga, IDOUtils.

We are planing to provide other databases than MySQL within the next Icinga Versions. Would you like to vote for your favourite database so that the Icinga Project fits your needs.

Loading ...

Using Icinga-API for fun and profit – IDO basics – part 1

Published by Michael Lübben on 4. June 2009 in API and Development. 5 Comments Tags: API.

Since Icinga-API is finally capable of supporting developers with
data we are going to blog a little to make coders out there feel
comfortable with this wonderful piece of work.

Here you can see the current progress of the API and related interfaces:

API overall: 50%
IDO interface: 50%
file interface: 25%

If you want to fetch data from the IDO you have to take care about the
following requirements:

PHP5
web-server module for PHP PDOs

Enough talking… let’s see some code!

1.) configuration

Configuration is simply done by using an associative array.

$idoConfig = array (
	‘type’ 		=> ‘<Type of database>’,
	‘host’ 		=> ‘<Database hostname>’,
	‘database’ 	=> ‘<Databasename>’,
	‘user’ 		=> ‘<Username>’,
	‘password’ 	=> ‘<password>’,
	‘persistent’ 	=> <true | false>,
	‘table_prefix’ 	=> ‘<table prefix>’,
);

Example:

$idoConfig = array (
	‘type’ 		=> ‘mysql’,
	‘host’ 		=> ‘localhost’,
	‘database’ 	=> ‘ido’,
	‘user’ 		=> ‘idouser’,
	‘password’ 	=> ‘idopassword’,
	‘persistent’ 	=> true,
	‘table_prefix’ 	=> ‘icinga_’,
);

2.) fetching data: hostnames and corresponding states
Create an instance of class IcingaApi:

$api = IcingaApi::getConnection(IcingaApi::CONNECTION_IDO, $idoConfig);

Create your search:

$apiRes = $api->createSearch()
->setSearchTarget(IcingaApi::TARGET_HOST)
->setResultColumns(array(’HOST_NAME’, ‘HOST_CURRENT_STATE’))
->fetch();

By using setSearchFilter() you can define filters to narrow down the result set:

$apiRes = $api->createSearch()
->setSearchTarget(IcingaApi::TARGET_HOST)
->setResultColumns(array(’HOST_NAME’, ‘HOST_CURRENT_STATE’))
->setSearchFilter(HOST_NAME, ‘Switch%’, IcingaApi::MATCH_LIKE)
->fetch();

3.) processing result

foreach($apiRes as $apiHandle){
	echo ‘Host ‘.$apiHandle->host_name.’ has state ‘.$apiHandle->host_current_state.’<br />’;
}

Output without filter:
Host localhost has state 0
Host MySql has state 0
Host router-01 has state 0
Host windows100 has state 0
Host Apache_01 has state 0

Output with filters:
Host switch70 has the current state 0
Host switch71 has the current state 0
Host switch72 has the current state 0
Host switch73 has the current state 0
Host switch74 has the current state 0
Host switch75 has the current state 0
Host switch76 has the current state 0
Host switch77 has the current state 0

4.) complete code without use of filters:

<?
// Path to icinga api file
$apiFile = ‘icinga-api/IcingaApi.php’;
 
// Database connection
$idoConfig = array (
	‘type’ 		=> ‘mysql’,
	‘host’ 		=> ‘localhost’,
	‘database’ 	=> ‘ido’,
	‘user’ 		=> ‘idouser’,
	‘password’ 	=> ‘idopass’,
	‘persistent’ 	=> true,
	‘table_prefix’ 	=> ‘icinga_’,
);
 
// Include required files
require_once($apiFile);
 
// Instance the class
$api = IcingaApi::getConnection(IcingaApi::CONNECTION_IDO, $idoConfig);
 
// Create search
$apiRes = $api->createSearch()
->setSearchTarget(IcingaApi::TARGET_HOST)
->setResultColumns(array(’HOST_NAME’, ‘HOST_CURRENT_STATE’))
->fetch();
 
// Create output
foreach($apiRes as $apiHandle){
	echo ‘Host ‘.$apiHandle->host_name.’ has the current state ‘.$apiHandle->host_current_state.’<br />’;
}
?>

That’s all folks but there’s more to follow!

Please have a look at the git repository for further information:
https://git.icinga.org/index?p=icinga-api.git;a=summary

Have fun!