Archive for the 'Core' Category

Icinga IDOUtils – More Improvements Part III

Published on 24. February 2010 by Michael Friedrich in Core, Development, IDOUtils and Technical. 0 Comments Tags: , , , , , , , , .

One last shot this time for upcoming Icinga 1.0.1 and IDOUtils:

After getting several core patches into the master and also fixing duplicated service/hoststatus updates being sent to the neb module (thanks to Matthieu Kermagoret) there will be more improvements for IDOUtils.

Since the threaded housekeeper is doing fine, it is possible to periodically clean more tables. By popular demand, the following options have been added to ido2db.cfg

They can be used for your likings, by default they are not set.

If you want to help us test for the upcoming release, you are very welcome to do so!

To help you with GIT, we now have a quite detailed tutorial how to use GIT based on Icinga in our Developer Wiki =)

  • Share/Bookmark

Icinga Core – More Enhancements

Published on 19. February 2010 by Michael Friedrich in Core, Development and Technical. 0 Comments Tags: , , , , , , , .

First of all – many thanks to Vitali Voroth and DECOIT GmbH and also Bill McGonigle for providing such great stuff and improving Icinga.

So what it’s all about?

As you might know, we are “monitoring” the Nagios world too and recently on the developer mailing list, an interesting patch popped up:

Currently the Icinga core sets state to CRITICAL if a service check times out. This is the default and can only be changed by recompiling the code. For several reasons you might want to define that yourself – and also, what does CRITICAL mean in this context? If the load on the monitoring box is too high, a service check may generate a timeout, not only a connection loss or similar.

We’ve been asking Bill McGonigle if we can take his patch for Icinga (it’s not applied in current Nagios CVS where it was built against), test it and in case apply it to give it back to the community. It’s a great idea to add the service_check_timeout_state to icinga.cfg and let the user decide upon his demands what state will be set in case of emergency. Bill suggested a new approach for Icinga too – changing the default state from CRITICAL to UNKNOWN. We think this is a great idea and so will it be in upcoming Icinga 1.0.1 :-)

That’s not all, folks …

Vitali Voroth on behalf of DECOIT GmbH sent a rather huge and exclusive improvement for Icinga core: escalation conditions.

Better to describe with an excerpt of the docs:

Using a patch it is now possible to define an escalation_condition (similar to escalation_options [w,u,c,r]). An escalation with a defined condition will only be escalated if the current state of a particular host/service fits the condition. One possible example of use for this could be the following scenario:

Think of two different escalations for the same service foo. One of them should only escalate when service bar is OK, the other should escalate if bar is CRITICAL or WARNING. Now think about foo being the main service offered by a company and the admin has to react immediately if it is down. bar could be a service indicating if the admin is in the office or at home and the escalation would react as following:

* If the admin is in the office, send an email first, after 5 minutes send an SMS
* If the admin is at home, send an SMS first and after 30 minutes a second SMS to the admin and the head of department

A really nice patch and Team Icinga is very happy about this core related enhancement! :-)

And as you will expect – Icinga Core provides the enhancements, while the documentation will be updated too for Icinga 1.0.1 =)

You want more?

If YOU ever wanted your ideas and patches within Nagios/Icinga, do not hesitate to contact us. And even if you want to contribute and develop Icinga, you are very welcome to do so!

Spread the word and show love for Icinga :-)

  • Share/Bookmark

Icinga IDOUtils – More Improvements Part II

Published on 17. February 2010 by Michael Friedrich in Core, Development, IDOUtils and Technical. 0 Comments Tags: , , , , , , , , , , .

As mentioned in the last post, there are other improvements for Icinga and IDOUtils.

This time, I want to give you a deeper look onto database performance and the housekeeping stuff.

As you might know, selecting, updating or even deleting a row from a table heavily depends on the row count. If table size grows bigger e.g. in the historical tables from IDOUtils, those queries will be slower and hold back the main process. Current approach of IDOUtils is one forked child of ido2db for one idomod connection – working sequentially on the gotten data.

So even one select taking longer will slow down the data processing and worst case the socket will get blocking and idomod complains about writing to data sink.

But how to resolve those issues?

First of all there were several approaches originally found in mysql-mods.sql – setting indexes on table columns which are being used within the WHERE clause. Regarding the fact that ido2db is not just an insert application, but also deletes historical data on demand (table trimming options), selects objects for caching and furthermore updates existing rows (service/hoststatus e.g.) we decided to apply most useful indexes on the table creation statements. It does slow down an insert a bit, but the overall benefit is much bigger than that :-)

Also the upcoming Icinga Web benefits from that – e.g. the logentries tables select performs a lot faster when using the API and a RDBMS.

But that’s not all – indexes are only one approach of improvement. In the last few months, Hendrik, Christoph and myself discussed a lot about the periodic housekeeping. The basic approach was to remove housekeeping function from the main data processing. Simply because historical deletes on large tables will take even longer and prevent new data being written to the database.

There have been discussions about a cronjob and seperated forked processes for housekeeping, but we wanted something within ido2db and simple to use. So Hendrik came up with the idea to create an own thread within each ido2db child which runs completely seperated from the main data processing flow – the so-called threaded housekeeper.

The thread just waits for the appropriate instance getting connected and then performs the periodic housekeeping – independant from the main flow. And it does not interfere with the normal data processing. So to speak it resolves a big performance issue within IDOUtils.

Basically, this is the way it performs:

  • sleep a while after creation and intialization
  • idle wait for database connection and connected instance from main process
  • perform periodic maintenance not interferring with main process
  • will be terminated when ido2db shuts down

Best thing so far – it has been implemented and tested and improved quite a while. Mostly done in our own git branches, but the final solution is within current git master and will be one of the outstanding new features for Icinga IDOUtils in the upcoming Icinga 1.0.1 release.

Stay tuned for more updates!

… and prepare for Icinga 1.0.1! =)

  • Share/Bookmark

Icinga IDOUtils – More improvements Part I

Published on 15. February 2010 by Michael Friedrich in Core, Development, IDOUtils, Misc and Technical. 0 Comments Tags: , , , , , , , .

It’s been a while since I made several changes to the initial Oracle implementation in Icinga IDOUtils. Code has been split, first start of using prepared statements and binded params with ocilib and some other changes to the code.

In the last few weeks I have been investigating a lot on how to implement more improvements and optimize the critical path of data input from Icinga Core.

I want to start with IDOUtils Oracle, more information on other improvements for Icinga and IDOUtils will follow :)

Oracle implementation splits up into several parts taken care of:

  • Rewrite all queries to prepared statements and bind params at runtime
  • Add dynamic procedures for DELETE statements
  • Drop autoincrement emulation by one sequence and insert triggers
  • Add sequences for each table and use INSERT INTO … (id, …) VALUES (seq_name.nextval, …)
  • Add RETURNING id INTO :id for INSERT statements to save one round trip
  • MERGE does not support returning INTO, added SELECT seq_name.currval query instead for fetching last inserted id
  • Rewrite selecting cached objects from DB

The rewritten queries are divided as follows:

  • 1x SELECT latest data time as is (called only at startup)
  • dynamic procedure for DELETE on table by instance_id called at startup for cleaning config/status
  • dynamic procedure for DELETE on tably by instance_id, field compared to time called during periodic cleanup
  • all other queries are prepared with their own statement handler
    • 4x DELETE
    • 52x MERGE
    • 9x INSERT
    • 9x UPDATE
    • 5x SELECT

This summarizes into about 8000 lines (+) and 2000 lines (-) of code modifications :-)

Furthermore I have been thinking on how to provide an upgrade path for all existing IDOUtils Oracle users. Importing data using the newly applied sequences might lead into errors regarding currval of each sequence. A basic upgrade procedure has been provided already – if you want to try, get the latest GIT master.

Stay tuned for more interesting stories to tell :)

… and watch out for Icinga 1.0.1 and fresh IDOUtils Oracle!

  • Share/Bookmark

Icinga development visualized by Gource

Published on 29. January 2010 by Michael Friedrich in API, Core, Development, Documentation, Technical and Webinterface. 1 Comment Tags: , , , , , , , , , , , , , , .

Hi there,

Icinga and the fork happened not that long ago but during this period of time a lot of nice things happened.

Providing Icinga Core with integrated IDOUtils supporting MySQL/Postgres/Oracle, fresh docbook format and therefore enhanced documentation, a completely new Icinga API based on IDOUtils and providing data for the new upcoming Icinga Web. Also lots of other improvements and enhancements.

Writing a historical overview would get boring soon. So we decided to catch up on another Idea: gource.

It’s a small program fetching all commits within our git repositories (core, doc, api, web) and presenting the timeline and changes using rendered pictures.

But that’s not all, it is possible to convert that to nice looking movies.

But there is so much to tell…

Not this time!

Just relax and watch :-)

Icinga Core

Icinga Doc

Icinga API

Icinga Web

  • Share/Bookmark

Icinga Database View Model

Published on 11. December 2009 by Bernd Erk in Core, Development and Technical. 0 Comments Tags: , , , .

I am sure there will always be much discussion around the IDO-/NDOUtils database model. In my opinion there are two major problems with the model at the end of the day:

  • Normalization
    The tables have a lot of redundant  information regarding their unique id’s. Different object types have a corresponding id and different object tables. To query a bunch of data you need to join the object and instance tables in most of the cases. This makes it hard to find a specific value without knowledge of the model.
  • Prefix and table names
    Icinga or Nagios as a prefix for every table makes no sense. The reason is that every supported database has schemas to store the tables and that is a better place to distinguish this. On the other hand we have a problem with Oracle to store tables with more than 30 characters.

In addition to that, some other problems like blocking, broker finetuning and loss of performance due to a lack of correct indices on the tables is hard work to do. At the moment, there is no time to change the whole model for every supported database and there is also a chance that the community has interesting suggestions worth first considering.

As a first step we want to introduce a new view layer based on the existing ndo model. A view is a “virtual” database object that queries the original data in the defined target table. I know this is not 100% true for every database, because we have various view types in oracle, but for now it is the only important thing. With this first early version we tried to solve these three issues:

  • Every object table (which has an own object_type in the object_table) includes the correct object_id and a join to the instance table
  • Every table is grouped into a configuration, historical and runtime area, which makes it easier to find a way through the model
  • Every table has a grouped tablename for example ic_hosts_escs (icinga, configuration, hosts, escalation)

In an early stage of modeling I recognized that this will not be the final step and I decided to develop a code generator based on Java. Because it is a drop away thing, there was no focus on performance or style, so please forgive me. You can download the generator here as it is without any warranty.

What do you think about this approach?

View Results

Loading ... Loading ...

You can download the first version of the view model here and run it against your database just check that your prefix is correct. We are looking forward to your feedback and also a rating on this idea.

  • Share/Bookmark

Fixes for Icinga IDOUtils MySQL

Published on 12. November 2009 by Michael Friedrich in Core. 1 Comment Tags: , , , , , .

Hi there,

just wanted to give you some updates regarding several fixes for Icinga IDOUtils. There were reports about doubled rows within several tables, where data only gets inserted and not updated. During my analysis it came up that there are several mistaken unique constraint definitions within the table creation for MySQL.

The unique constraint makes sure that if an INSERT will try to insert updated data, that this will create an internal exception which is caught within the ON DUPLICATE KEY clause. If caught, an UPDATE will be issued and everything is fine.

Regarding the table servicecheck, this was missing. So the start time of the servicecheck was inserted to the database, and when the servicecheck was complete, the end time also was inserted as own row into the database. Kind of useless 2 rows isn’t it? ;-)

Today i did some further investigations on that since this happened with systemcommands too. It came up that tables timedevents and timedeventqueue had that “feature” too.

This is really bad because there are lots of those queries issued and data will grow fast. Not this time because there has been another modification to idomod.cfg – the data processing options haven been modified to ignore timedevents by default. It will improve IDOUtils a bit, but your feedback is as always very welcome!

Back to topic – those missing unique constraints have been added to actual GIT Master (fixing #173 and #181 – check for analysis and comparison) so make sure you get the latest and the greatest! :-)

For those who are using Postgresql or Oracle – I have implemented and debugged them in deep. And they have own WHERE clauses for UPDATE – so no worries about that, everything is fine! =)

  • Share/Bookmark

Playing with Oracle, ocilib and parameter bindings

Published on 9. November 2009 by Michael Friedrich in Core. 0 Comments Tags: , , , , , , , , , , , .

Hi there,

IDOUtils queries differ quite a lot – some of the are just executed during startup, while others happen all the time. By analyzing the performance on our Oracle database with grid  it came to the top queries just like for

  • servicechecks, servicestatus
  • hostchecks, hoststatus
  • timedevents
  • programstatus

But how to improve the performance of those queries when they are called all the time?

Well, the query as is is always the same, only the values happen to change. So the basic idea is to prepare the statements with value place holders and if it comes to the query, just to bind the paramaters (values) to the prepared statement and execute that. This is a real performance boost compared to putting the query within the rdbm cache all the time.

Generally speaking the query statements are prepared after database connection and the statement handle is stored within the global dbinfo object (where the connection handler resides too).

dbinfo.oci_statement_programstatus = OCI_StatementCreate(dbinfo.oci_connection);
OCI_Prepare(dbinfo.oci_statement_programstatus, MT("MERGE INTO table USING DUAL ON (v1=:X1) WHEN MATCHED THEN UPDATE SET v2=:X2 WHEN NOT MATCHED THEN INSERT (v1, v2) VALUES (:X1, :X2)"))

When a query should be executed, all values will be binded (X1, X2) to the statement.

OCI_BindUnsignedBigInt(dbinfo.oci_statement_programstatus, MT(":X1"), (big_uint *) value1)
OCI_BindString(dbinfo.oci_statement_programstatus, MT(":X2"), (char *) value2)

Then the query gets executed.

OCI_Execute(dbinfo.oci_statement_programstatus);

Well it sounds quite simple but regarding the architecture of *DOUtils it was a hard nut to crack. The most common problem was the query buffer building – each unixtimestamp conversion is done before query building and sending the query. That does not fit for prepared statements where the whole query is pushed into the database cache.

Within the code, there is an char* array which gets the SQL-code from ndo2db_db_timet_to_sql and this is then printed to the whole statements. Not very useful since you may paste that right within each query. For the prepared statements, I’ve added all plain unixtimestamps to the data[] array and then binding the values directly.

(SELECT unixts2date(:X3) FROM DUAL)

So the bind param task has been done for the initial steps, improved delete statements and other improvements need to be implemented.

Another thing which was quite nasty is that Oracle support was dependant on libdbi, but it was not even used. So I decided to split the code completely and change configure. If you use –enable-oracle it will only require ocilib to work, it does not complain about a missing libdbi. The other way around it also works fine just like it was.

Conclusion to that – you won’t need libdbi to get Oracle support for Icinga IDOUtils – just ocilib.

Those improvements have been pushed to actual GIT master und you are very welcome to test and report bugs! =)

  • Share/Bookmark

Icinga IDOUtils will support Oracle RDBM in 1.0 RC

Published on 20. October 2009 by Michael Friedrich in Core and Releases. 0 Comments Tags: , , , , , , , , , , , .

Hi there,

work was not getting better and getting Oracle to work was on hold. So I decided to push a night of coding after I had prepared the MERGE queries.

And yes, it was successful – initial support for Oracle is done!

Oracle driver requirements

You”ll catch the problems with libdbi and Oracle in older blog posts. The new driver proposed by myself was ocilib, developed by Vincent Rogier. Currently it’s kind of a break up, so you need to have the libdbi installed and then install ocilib as an add-on (and Oracle libs and includes i.e. the Oracle Instant Client). As far as I know ocilib is not in the repositories right now so get the latest version and compile them yourself (documentation is really good!).

Modifying Configure for Oracle

If you enable IDOUtils during configure you now have the opportunity to use the flag –enable-oracle – if you didn’t install ocilib to default path /usr/local/ [lib/include] you can use

--with-ocilib-lib=/path/to/ocilib/lib
--with-ocilib-inc=/path/to/ocilib/include

to point configure to ocilib. It will be linked at runtime so you do not need to tell configure where $ORACLE_HOME and $LD_LIBRARY_PATH relays. Configure will output the following

  • export LD_LIBRARY_PATH in ido2db Initscript where ocilib resides (OCI_IMPORT_RUNTIME)
  • enable #define USE_ORACLE and ocilib.h for the compiler
  • create ido2db.cfg-sample with Oracle support

After that you can perform a normal install.

Whencompiling the code  it heavily depends on #define USE_ORACLE – if you plan to change back to another RDBM using libdbi you’ll have to issue
# make distclean
# make clean
# ./configure --enable-idoutils

Oracle Database Setup

In module/idoutils/db/ you will find oracle.sql and oracle-drop.sql. The first one creates the table definitions, a time conversion function and all the triggers and sequences needed for commonly used insert ids. oracle-drop.sql is just for testing purposes and cleans the database scheme.
Make sure you setup the Oracle DB with an appropriate scheme with username/password. Then copy oracle.sql to your $ORACLE_HOME at the db server and import it e.g. by using sqlplus
# su - oracle
$ sqlplus dbuser/dbpass
SQL> @oracle.sql

Then edit your ido2db.cfg for using Oracle. Please note that Oracle ignores the db_host, instead point db_name to //DBSERVER/DBNAME

db_servertype=oracle
db_port=1521
db_user=icinga
db_pass=icinga

That should do the trick. If you are experiencing problems turn the debug_level=-1 and debug_verbosity=2 and make sure the max_debug_file_size is set to at least 100 MB – the improved debug output will put a lot of output into that file.

Changes to the code

The biggest part has been done already getting Postgres to work – rewrite the INSERT OR UPDATE queries and extract the unique constraints for UPDATE conditions. Those queries have been adapted to use the MERGE trick. Some queries tried to issue an UPDATE where the unique constraint contained a value to be updated too. That failed heavily but introduces a really nice feature of ocilib.

By using OCI_Initialize it is possible to register an error handler function. This function simply gets the last OCI error and writes that to syslog and debug output. So it is really easy to find out why queries will fail and the nice thing is – the function doesn’t need to be called everywhere, just register it to ocilib.

Another heavy task was getting the insert id – MySQL supports last_insert_id but Postgres and Oracle don’t. In Postgres it’s rather easy defining the PK as SERIAL and getting the sequence id in order to get the last insert id. For Oracle, there are several ON INSERT TRIGGERs defined in oracle.sql which auto increment the id (primary key). Given that a specified function reads that values from the opened session.

Simply said, you do the following for an Oracle DB query in IDOUtils

  • OCI_Initialize(ido2db_ocilib_err_handler, NULL, OCI_ENV_DEFAULT)
  • oci_connection = OCI_ConnectionCreate(dbname,username, password, OCI_SESSION_DEFAULT);
  • oci_statement = OCI_StatementCreate(oci_connection);
  • OCI_ExecuteStmt(oci_statement, MT(“SELECT * FROM ….”));
  • OCI_Commit(oci_connection);
  • oci_resultset = OCI_GetResultset(oci_statement);
  • instance_id = OCI_GetUnsignedInt(idi->dbinfo.oci_resultset, 1);
  • OCI_ConnectionFree(oci_connection);
  • OCI_Cleanup();

There were some other minor and major changes to the code…

  • changed NOW() to SYSDATE
  • modify table serviceescalation_contactgroups to serviceescalationcontactgroups (30 chars max in Oracle)
  • primary key only is id anstead of [tablename]_id (30 chars max in Oracle)
  • dropped table_prefix – ido2db.cfg setting will be ignored
  • long_output uses CLOB (Character Large Object) since varchar2 supports 4000 bytes at maximum

I’ve also added a runtime version check for ocilib – if the library contains errors and does not export symbols correctly ido2db will quit correctly.

Conclusion

This is the initial version of Oracle support for Icinga IDOUtils. It was a bunch of work but there are many things to follow:

  • rewrite heavily used queries (host/service/check/status, timedevents) to prepared statements and parameter bindings
  • improve housekeeping DELETE queries with partitioned COMMITs
  • improve getting the insert ids
  • do not depend on libdbi if –enable-oracle is used

Oracle support for IDOUtils will be in Icinga 1.0 RC – watch out for the upcoming release and have fun testing! Please report any bugs or feature requests to the mailinglists and/or our dev tracker!

Many thanks to David Schmidt for implementing the first version of NDOUtils Oracle, many ideas have been improved within here. And also many many many thanks to Vincent Rogier for implementing such a great Oracle driver within the project ocilib. It is a pleasure coding based on ocilib, reading the documentation meanwhile and getting instant support for free! :-)

Since this is the third RDBM to be maintained by only 2 Core team members, please contact us if you like to participate and/or help us improving more RDBM support! :)

  • Share/Bookmark

Playing with IDOUtils, Oracle and ocilib

Published on 6. October 2009 by Michael Friedrich in Core, Development, Misc and Technical. 0 Comments Tags: , , , , .

Since I’m very busy at work this time I just wanted to give you some feedback on my current attempt – IDOUtils with Oracle based on ocilib.

Regarding that libdbi currently doesn’t provide a stable Oracle driver and furthermore, the main interface doesn’t allow prepared statements and parameter bindings, ocilib should be used and enabled during configure.

Currently, I  am preparing a version which works as an addon to IDOUtils with libdbi, meaning that the libdbi still has to be installed and optional ocilib. Oracle support with ocilib will be enabled using –enable-oracle during configure.

Configure modifications

I have chosen to let ido2db link against ocilib at runtime which gives the benefit of only loading the ocilib and based on that the Oracle libs. The configure flag enables the include directory /usr/local/include where ocilib.h resides and the library directory /usr/local/lib with ocilib. In the first initial version, the path will be hard coded, later I will implement configure flags just as –with-ocilib-lib/inc for more customizations.

The code within configure also enables the include statement for ocilib.h and the #define USE_ORACLE which will be used in the Code to point to ocilib DB handling instead of libdbi – just as NDOUtils Oracle does.

OCILIB_INC="/usr/local/include"
OCILIB_LIB="/usr/local/lib"
...
AC_DEFINE(HAVE_OCILIB_H,"1")
...
AC_DEFINE_UNQUOTED(USE_ORACLE)

Initscript modifications

Meanwhile because of the linking at runtime it is required to hand the library path over to ido2db since it’s not in systems default library path. If not exported before running ido2db within the init script, error will look like this:

# /etc/init.d/ido2db start
Starting ido2db:/opt/icinga/bin/ido2db: error while loading shared libraries: libocilib.so.3: cannot open shared object file: No such file or directory
done.

And start of ido2db simply fails because of the missing LD_LIBRARY_PATH. This path is defined running configure, so I have modified the rc.ido2db.in to export this environment variable with the concurrent ocilib library path.

#add ocilib lib path to link at runtime if enabled
LD_LIBRARY_PATH=@ido2db_runtime@
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH

@ido2db_runtime@ will be defined running configure, the basic setting is done in configure.in (and running autoconf 2.61 afterwards to generate configure script).

dnl create ido2db_runtime for rc.ido2db.in
ido2db_runtime="$OCILIB_LIB"
AC_SUBST(ido2db_runtime)

RHEL’s autoconf is quite old (2.59) so I compiled my own static version using this article.
The reason why I am working on RHEL is quite simple – it’s in a VM and already configured with NDOUtils Oracle so the working environment (Oracle DB connection, etc) is already there :-)

Future plans

So for the first part, ocilib initialization during build is enabled and working. My next steps are

* adapt DB init, connection, passing queries to ocilib
* add Oracle based MERGE queries for the ON DUPLICATE KEY MySQL queries
* rewrite heavy queries to prepared statements/parameter bindings
* improve housekeeping DELETE sequences (since ido2db dies regularly when deleting more thant 500k queries at startup…)

Stay tuned! =)

  • Share/Bookmark