idb-project/the-idb

The infrastructure database core application

the idb

Once upon a time we used spreadsheets and wiki pages to document our server landscape. Then the age of working together, more people joined the team and it became tedious to maintain all those wiki pages.

And we thought, why not keep to the principle of:

Automate all boring things!

And the idb - the infrastructure database core application - was created.

What this really is

This is a rails based web application known as 'idb-core', that is fed by various 'adapters' to corelate information for machines, inventory-items, networks and such. You can find various idb adapters within the github organization:

https://github.com/idb-project

It all began with puppet and puppetdb

The primary adapter for gathering informations has been PuppetDB. That's the reason why - unlike other adapters - the PuppetDB adapter is bundled directly within the idb. Don't fear: the idb can be used without puppet just fine.

Install packages

You might find several but probably untested packages for various operating systems at https://packager.io/gh/idb-project/the-idb/refs/develop

Contact to the team

Either contact us through github or use team AT the-idb.org.

Commercial support

While the idb is an open source project, there is commercial support available from the fine folks at bytemine. They offer the idb packaged for various platforms and these come with support and services. (Disclaimer: currently all people working on the idb are employed by bytemine)

the-idb's Issues

API should return NIC information

Redmine#4040
currently the NICs are not returned at querying the API.

IP-addresses aren't sorted

Redmine#4662
IP-addresses aren't sorted like one would expect, see attached screenshot.

per module mail recipient

Redmine#3977
It would be cool if we could start breaking up the e-mailing from idb into several pieces.
Initial idea would be to have a per module recipient list (eg. one or many)

A module is:

• inventory
• network
• machines
• locations

Create an interface to plan a maintenance

Redmine#2548
There should be some kind of button in the machine details to plan a maintenance.

• Take a date
• Take a user who is responsible
• Maybe notes about the update. (to add ticket references etc.)

Resolve bridge interface to parent interface

Redmine#2647
Try to resolve the parent interface of bridge interfaces and show them in parents. behing the bridge.

Subject of version-diff-mails contains fixed app title

Redmine#4529
Within the version presenter the app title displayed in the email subject is set to "IDB". The config option "IDB.config.design.title" should be used instead.

Add a JSON column to machines reflecting software versions

Redmine#4651
machine has a active softwareconfiguration.

softwareconfiguration would consist of a field holding a json structure and for the papertrail a date and active/inactive marker, thus previous software lists can be kept around for a machine.

{
"software": [{
"name": "foo",
"version": "1.2.3.4",
"license_key": "abcde",
"description": "a piece of installed software"
}, {
"name": "bar"
}]
}

Logical depencencies when adding new items

Redmine#4496
Example: When adding an inventory item, the install date should be after the purchase date.

adapter name field

Redmine#4355
i think it would be useful to have a field that contains the name of the adapter which has updated/created a machine.

Implement tags on all objects

Redmine#4666
Every object (machine, inventory, owner, etc.) can have 0..n tags. By a special view one can filter/search for these tags over all objects.

allow more fine-grained informations on service / maintenance

Redmine#3085
The field should allow that one can store:

• serviced by whom
• part of a service contract
• SLA for the host
• maintainence window

implement first version of idbicinga1 adapter

Redmine#4096
Please write an icinga1 config parser for 'host' definitions. The parser should extract info like network address and hostname and submit these to the idb api.

Allow grouping of related machines

Redmine#2533
To allow grouping of machines by i.e. customers, we need to be able to assign machines to groups. One machine can be in multiple groups.

To avoid building an interface for that, we just create a configuration file that contains the group names and an unique identifier.

Machine facts should be used like tags so one can filter by operating system and location for example.

Batch delete

Redmine#2981
I would like something that allows me to batch delete from the machines listing.

Example: I filter for a certain hostname pattern within the fqdn field and want to delete the whole result OR part of the result within another click.

rename startup script of idb-public-api

Redmine#4314
currently: bytemine-idb-public-api
should be: idb-public-api

per module mail recipient

Redmine#3977
It would be cool if we could start breaking up the e-mailing from idb into several pieces.
Initial idea would be to have a per module recipient list (eg. one or many)

A module is:

• inventory
• network
• machines
• locations

Improve client-side performance with large tables

Redmine#4515
With the current amount of machines it is barely unusable on older workstations. Have a look at the custom JS that e.g. creates the summary, and at tablesorter in general.

allow more fine-grained informations on service / maintenance

Redmine#3085
The field should allow that one can store:

• serviced by whom
• part of a service contract
• SLA for the host
• maintainence window

define a hostname mangling method

Redmine#4399
it is frequently required to mangle the hostnames received from other management tools down to a fqdn.
define a method for doing this and implement it in idbclient.

Report duplicate MAC addresses

Redmine#3965
If by an automatic update, e.g. by puppetDB, a MAC address is used that already exists in the IDB an email should be sent out that there is a conflicting MAC address.

icinga2 Integration via event stream api

Redmine#2676
It would be nice to have some icinga information on the machine's detail page.

Some ideas (tbd):

• Link to the machine in icinga.
• Overview of the status.
• Links to the alert tickets in RT
• Buttons to ack, disable notifications, re-check
  ** ack one service
  ** ack all critical services

RequestTracker integration

Redmine#2667
There should be a tab pane on the machine details that could show related request tracker tickets for a machine.

Idea on how to implement this:

RT will get a custom field where the RT-API or a regular user can add the FQDN or Icinga name for machine. The IDB gets a background job that collects all tickets that have the custom machine name field set to a value on a regular basis and links them to the corresponding machines.

• The machine name field in RT can contain the FQDN or the Nagios/Icinga name.
  The IDB can resolve the name because it can read the Icinga config.
• Check which fields should be stored in the IDB for a ticket.
  ** Ticket id
  ** status?
  ** owner?
• Storing above data requires updating the fields on a regular basis to get ticket updates. (owner change, resolved, ...) tbd

Power feed->machine connection currently not implemented

Redmine#4518
ATM it is not possible to see which machines are connected to a feed, but that is why we created this feature.

copyable inventory items

Redmine#4396
Inventory items should be copyable to allow time saving when adding multiple similar items.

icinga2 Integration via event stream api

Redmine#2676
It would be nice to have some icinga information on the machine's detail page.

Some ideas (tbd):

• Link to the machine in icinga.
• Overview of the status.
• Links to the alert tickets in RT
• Buttons to ack, disable notifications, re-check
  ** ack one service
  ** ack all critical services

Report duplicate MAC addresses

Redmine#3965
If by an automatic update, e.g. by puppetDB, a MAC address is used that already exists in the IDB an email should be sent out that there is a conflicting MAC address.

RequestTracker integration

Redmine#2667
There should be a tab pane on the machine details that could show related request tracker tickets for a machine.

Idea on how to implement this:

RT will get a custom field where the RT-API or a regular user can add the FQDN or Icinga name for machine. The IDB gets a background job that collects all tickets that have the custom machine name field set to a value on a regular basis and links them to the corresponding machines.

• The machine name field in RT can contain the FQDN or the Nagios/Icinga name.
  The IDB can resolve the name because it can read the Icinga config.
• Check which fields should be stored in the IDB for a ticket.
  ** Ticket id
  ** status?
  ** owner?
• Storing above data requires updating the fields on a regular basis to get ticket updates. (owner change, resolved, ...) tbd

Add some internal statistics to visualize errors and allow monitoring

Redmine#2661
Background jobs may fail due to the unavailability of external services. Those errors should be catched and somehow recorded (not via exceptions) so we can alert if there are too many errors or errors over a log period of time. Having jobs fail from time to time is normal, but we want to know if the jobs stop working at all.

technical informations won't be summarised in owners view

Redmine#4428
Click on "Owners" -> Choose owner -> Machines

Summary will always show 0C and 0GB for CPU and RAM in the machines table

Create an interface to plan a maintenance

Redmine#2548
There should be some kind of button in the machine details to plan a maintenance.

• Take a date
• Take a user who is responsible
• Maybe notes about the update. (to add ticket references etc.)

Allow re-labeling of general purpose fields

Redmine#4380
The fields:

config instructions
business purpose
business criticality
Software characteristics
Business notification

should be allowed to be re-labeled via config file

Allow grouping of related machines

Redmine#2533
To allow grouping of machines by i.e. customers, we need to be able to assign machines to groups. One machine can be in multiple groups.

To avoid building an interface for that, we just create a configuration file that contains the group names and an unique identifier.

Machine facts should be used like tags so one can filter by operating system and location for example.

integrate with oxidized to document switches, routers and pfsenses

Redmine#4569
as well as other network devices.

https://github.com/ytti/oxidized

There is general bytemine task #4568 to evaluate oxidized within our infrastructure - so once that is done and oxidized looks promising we can tackle this idea.

Extend the 'untracked' feature to handle e.g. API calls

Redmine#4309
We would like to track an API call that does not explicitely set the "create_machine" option as an untracked entry, like the automatically discovered puppetDB hosts not added to IDB.

Therefore we have to create a structure with a special Redis key able to hold the FQDNs and the payload; than we hook the store mechanism into the API if machine does not exist and "create_machine" is not set.
Then the list handling and "Add to IDB" feature has to be implemented.

display services that need a restart

Redmine#4410
might be useful for e.g. libc updates as an indicator that even a reboot should be considered.

for yum:
yum-utils provides needs-restarting

for apt:
debian-goodies provides checkrestart

puppetize bootstrapping of vagrant boxes

Redmine#3084
There is an initial bootstrap.pp present. this needs to be extended as follows:

• create database 'idb' and 'idb_test' within the mysql

• grant access to these databases to these with the credentials found in config/databases.yml

• run 'gem install bundler' within the shared directory to the vagrantbox (/vagrant/)

• run 'bundle install --path vendor/bundle' within the shared directory to the vagrantbox (/vagrant/)

• run 'bundle exec rake db:schema:load' within the shared directory to the vagrantbox (/vagrant/)

puppetize bootstrapping of vagrant boxes

Redmine#3084
There is an initial bootstrap.pp present. this needs to be extended as follows:

• create database 'idb' and 'idb_test' within the mysql

• grant access to these databases to these with the credentials found in config/databases.yml

• run 'gem install bundler' within the shared directory to the vagrantbox (/vagrant/)

• run 'bundle install --path vendor/bundle' within the shared directory to the vagrantbox (/vagrant/)

• run 'bundle exec rake db:schema:load' within the shared directory to the vagrantbox (/vagrant/)

Supplier contact information

Redmine#4376
The supplier is backed with more information than a name in other peoples database, including address/telephone/mail. Maybe this would also make sense for the IDB.

Show Name/FQDN/etc. instead of Id in History

Redmine#4661
In the history of an item the relevant name should be shown instead of an internal id, see attached screenshot.

Resolve bridge interface to parent interface

Redmine#2647
Try to resolve the parent interface of bridge interfaces and show them in parents. behing the bridge.

display every ip address of a machine in the machines overview

Redmine#4354
i think it would be useful to show every ip (eg. when one has an ip from a log file and wants to know the machine which owns it)

Add some internal statistics to visualize errors and allow monitoring

Redmine#2661
Background jobs may fail due to the unavailability of external services. Those errors should be catched and somehow recorded (not via exceptions) so we can alert if there are too many errors or errors over a log period of time. Having jobs fail from time to time is normal, but we want to know if the jobs stop working at all.

OS release list: Should be user configurable

Redmine#3954
especially for UCS 4.0 and 4.1 are missing. Perhaps a good idea to change this hardcoded way of versioning.

pending updates inserted via api not displayed as pending updates

Redmine#4316

define a hostname mangling method

Redmine#4399
it is frequently required to mangle the hostnames received from other management tools down to a fqdn.
define a method for doing this and implement it in idbclient.

Batch delete

Redmine#2981
I would like something that allows me to batch delete from the machines listing.

Example: I filter for a certain hostname pattern within the fqdn field and want to delete the whole result OR part of the result within another click.

fixed column width when filtering on results

Redmine#4459
right now, if you filter the results, the column width is not fixed.
The attached screenshots show what I mean.

OS release list: Should be user configurable

Redmine#3954
especially for UCS 4.0 and 4.1 are missing. Perhaps a good idea to change this hardcoded way of versioning.

adjust columns in summary broken; tablesorter handling needs refactoring

Redmine#4426
If a summary for a customer should be created with only specific columns, some of them are not working, if others are already hided.

In my case I wanted to hide all inventory columns and all unimportant columns from the machines:

• click on Columns
• find Inventory No
• unmark all columns from Inventory No to Comment
• unmark BN

Only columns Owner, CI, SC, BP, BC left.
When showing all columns again, then unmark and mark a random column from the machine, all informations are back.

Seems like there are dependencies.

Another example:

• unmark User and Owner from Inventory
• try to toggle VM Host from Machines

Every location type can be assigned to a power supply

Redmine#4519
But it should be limited to power feeds.