rafal-szypulka/itm_exporter

ITM Exporter for Prometheus

itm_exporter's Issues

KFWITM454E Request failed due to offline managed system(s).'."} source="main.go:126"

Please have a look and see if you can tell what up with this. Its a new installtion and for the most part its working...

ITM v6

DE Version CJ063007_tms630fp7

config.yaml
itm_server_url: "http://jgapsr000000008:15200"
itm_server_user: "sysadmin"
itm_server_password: "xxxxxxxx"
connection_timeout: 15
collection_timeout: 45
groups:

• name: "KLZCPU"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets"
  labels: ["CPUID", "ORIGINNODE"]
  metrics: ["BUSYCPU", "IDLECPU", "SYSCPU", "USRCPU", "WAITCPU"]
  managed_system_group: "*LINUX_SYSTEM"
• name: "KLZVM"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets"
  labels: ["ORIGINNODE"]
  metrics: ["MEMUSEDPCT", "VSFREEPCT"]
  managed_system_group: "*LINUX_SYSTEM"
• name: "KLZDISK"
  datasets_uri: '/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets'
  labels: ["ORIGINNODE", "DSKNAME", "MOUNTPT"]
  metrics: ["DSKFREEPCT", "DSKUSEDPCT", "DSKFREE", "DSKUSED", "INDFREEPCT"]
  managed_system_group: "*LINUX_SYSTEM"
• name: "QSG_QUEUES"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets"
  label: ["QSGNAME"," QNAME"]
  metrics: ["CURDEPTH"]
  managed_system_group: "*QUEUE_SHARING_DETAILS"
• name: "QSG_CFSTR"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets"
  label: ["STRNAME"]
  metrics: ["QSGNAME","USEDLSTE","MAXLSTE"]
  managed_system_group: "*QUEUE_SHARING_DETAILS"
• name: "SYSCPUUTIL"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets"
  labels: ["ORIGINNODE"]
  metrics: ["RMFMVSCPU", "RMFLPARCPU"]
  managed_system_group: "*MVS_SYSTEM"

This produce the following
http://jgapsr000000008:8000/metrics

HELP itm_scrape_duration_seconds Time ITM attribute group scrape took.

TYPE itm_scrape_duration_seconds gauge

itm_scrape_duration_seconds{group="KLZCPU"} 0.145134457
itm_scrape_duration_seconds{group="KLZDISK"} 0.065749022
itm_scrape_duration_seconds{group="KLZVM"} 0.065682694
itm_scrape_duration_seconds{group="QSG_CFSTR"} 0.185712258
itm_scrape_duration_seconds{group="QSG_QUEUES"} 0.145967956
itm_scrape_duration_seconds{group="SYSCPUUTIL"} 0.064833224

HELP itm_scrape_duration_seconds_total Time ITM scrape took.

TYPE itm_scrape_duration_seconds_total gauge

itm_scrape_duration_seconds_total 0.185715429

HELP itm_up itm_exporter successfully connected to the TEP data provider

TYPE itm_up gauge

itm_up 1

And the error is

ERRO[2388] {"msgId":"ATKRST100E","stackTrace":"com.ibm.usmi.console.navigator.model.NavException: (ATKRST100E) ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'.: nested exception is: \n\tcom.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.rest.RestURI.getExceptionResponse(RestURI.java:580)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2304)\n\tat sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)\n\tat java.lang.reflect.Method.invoke(Method.java:611)\n\tat org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:54)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:34)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.OptionsMethodHandler.handleRequest(OptionsMethodHandler.java:46)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:52)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)\n\tat com.ibm.tivoli.rest.RestRequestProcessor.handleRequest(RestRequestProcessor.java:381)\n\tat org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:908)\n\tat com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)\n\tat com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:277)\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)\n\tat com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:166)\n\tat com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)\n\tat com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)\n\tat com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1691)\nCaused by: com.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:838)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getLargeTableModelNodes(ITMNavReportCollection.java:317)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getAllNodes(ITMNavReportCollection.java:343)\n\tat com.ibm.tivoli.rest.navmodel.NavNodeFilter.getNodes(NavNodeFilter.java:152)\n\tat com.ibm.tivoli.rest.navmodel.NavCollection.getNodes(NavCollection.java:1191)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2287)\n\t... 73 more\nCaused by: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.TEPS.CTQuery.JVMRequest.createException(JVMRequest.java:426)\n\tat com.ibm.TEPS.CTQuery.JVMRequest.execute(JVMRequest.java:198)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:833)\n\t... 78 more\n","msgSeverity":"error","msgText":"ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'."} source="main.go:126"

I can even list the following which is telling me I have a good connection to the TEPS.
tivuser@jgapsr000000008 PROD $ ./main listAttributes --attributeGroup=QSG_CFSTR --dataset=/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets
+--------------------------+------------+
| DESCRIPTION | ATTRIBUTES |
+--------------------------+------------+
| Origin Node | ORIGINNODE |
| QSG Name | QSGNAME |
| CF Struct Name | STRNAME |
| Alter Date & Time | ALTERDTTM |
| Struct Level | STRLEVEL |
| Description (Deprecated) | STRDESC |
| Recovery Supported | RECOVERY |
| CF Struct Type | STRTYPE |
| Struct Status | STRSTATUS |
| Max Stor | MAXSIZE |
| % Stor Used | USEDPCT |
| Max Entries | MAXLSTE |
| Used Entries | USEDLSTE |
| % Entries Used | LSTEPCT |
| Failure Date & Time | FAILDTTM |
| Description | USTRDESC |
| Offload Usage | OFFLDUSE |
| Recording Time | WRITETIME |
+--------------------------+------------+

I already have grafana and prometheus working with other products

Ps: I have installed the APM Datasource and that works as long as you dont load it to much.

What am I missing

Thanks
Leon

Scrapes have super long runtimes

At some point the ITM API seems to significantly slow down, the scraping and the request to /metrics take moer than a minute with the default config and around 3 managed systems. Eventually the portal server freezes as well. Any idea?

/metrics sample

itm_scrape_duration_seconds{group="NTMEMORY"} 59.202864944
itm_scrape_duration_seconds{group="WTSYSTEM"} 59.197552453

- name: "WTSYSTEM"
  datasets_uri: "/providers/itm.TEMS/datasources/TMSAgent.%25IBM.STATIC021/datasets"
  labels: ["ORIGINNODE"]
  metrics: ["PCTTLPCSRT","PCTTLPRIVT","PCTTLUSERT","SYSUPDAYS"]
  managed_system_group: "*NT_SYSTEM"
- name: "NTMEMORY"
  datasets_uri: "/providers/itm.TEMS/datasources/TMSAgent.%25IBM.STATIC021/datasets"
  labels: ["ORIGINNODE"]
  metrics: ["AVAILBTMEM","COMMBYTE", "TOTMEMBYTE","CACHEBTS","MEMUPCT","AVAILPCT","CACHEPCT"]
  managed_system_group: "*NT_SYSTEM"

[UNIXPS] Is it possible filter the atribute unixps before ?

Hi, there is a way to filter what i want to get. ? for example UNIXPS, i just want the cpuid=-1, this atribute there is a lot of information.

Thanks!

Issue to get data from customSQL Oracle RDB

Hi i need some help here, I did create a custom sql to get metrics from Oracle in ITM, but when i try get this data from itm_exporter, the return is null.

here is my config:
- name: "KRZRDBCUSQ" datasets_uri: "/providers/itm.TEMS/datasources/TMSAgent.%25IBM.OracleAgentRDB/datasets" labels: ["ORIGINNODE", "STRVALUE1"] metrics: ["NUMVALUE1"] managed_system_group: "ger_prom_rdb"

itm_scrape_duration_seconds{group="KRZRDBCUSQ"} 2.117378146

Problem with loading metrics of some servers

Hello!

We have got combunations of Prometheus + ITM exporter to create dashboards of server usage.
But we have problem with attribute group KLZ Disk, LNX Disk. It doesnt get all list of servers.
There is no Redhat 8.0 and some old. We could find the solution for this. We have this metrics in tivoli monitoring.
Could you give any advice, how to solve this problem.

We use tivoli agent 06.30.07
ITM_exporter 0.6
exporter config

• name: "KLZDISK"
  dataset: xxxxxx/dataset
  labels: ["ORIGINNODE","DSKNAME"]
  METRICS: ["DSKFREEPCT"]

itm_exporter timeout error

Hi,

I'm facing this error when I try to run itm_exporter with many servers tied to the manage system group. Is there a timeout limit? I changed the timeout parameter in the scrap config to 300s.

I ran the curl and i got the result in a few seconds.

There is any config in itm_exporter ?

curl -u sysadmin:xxxxxxx "http://localhost:15200/ibm/tivoli/rest/providers/itm.itm.brhocam01ir00uj/datasources/TMSAgent.%25IBM.STATIC134/datasets/MetricGroup.KLZNET/items?param_SourceToken=teste_prometheus&optimize=true&param_refId=br61qrufosfv8knt1p0g&properties=all"

`[root@brhodsh01srsprj itm_exporter]# ./itm_exporter export
time="2020-05-25T16:32:40-03:00" level=info msg="Starting itm_exporter in export mode..."
time="2020-05-25T16:32:40-03:00" level=info msg="Author: Rafal Szypulka"
time="2020-05-25T16:32:40-03:00" level=info msg="itm_exporter listening on port::8000"
time="2020-05-25T16:33:43-03:00" level=error msg="Get http://129.39.186.133:15200/ibm/tivoli/rest/providers/itm.itm.brhocam01ir00uj/datasources/TMSAgent.%25IBM.STATIC134/datasets/MetricGroup.KLZNET/items?param_SourceToken=teste_prometheus&optimize=true&param_refId=br61qrufosfv8knt1p0g&properties=all: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x884a3f]

goroutine 29 [running]:
main.MakeAsyncRequest(0xc0000dc700, 0xf6, 0xc0002aa800, 0x6, 0xc000084900)
/Users/rafalszypulka/goprojects/src/itm_exporter/main.go:119 +0x2bf
created by main.ITMCollector.Collect
/Users/rafalszypulka/goprojects/src/itm_exporter/main.go:262 +0x19d3
You have mail in /var/spool/mail/root`

itm_exporter cannot execute on windows

Hi

I had extracted the itm_exporter on windows TEPS machine, but i am unable to execute the itm_exporter command as its not a windows batch file. I download from the following link https://github.com/rafal-szypulka/itm_exporter/releases,
please advise if i need to download the file from a different source

Change relative path to absolute

Hello, why when I change the line

itm_exporter/main.go

Line 331 in 74c9b84

yamlFile, err := ioutil.ReadFile("config.yaml")

to

yamlFile, err := ioutil.ReadFile("/etc/itm_exporter/config.yaml")

Is the exporter unable to display all the parameters that exist in the .YAML file?