prometheus-community/json_exporter

A prometheus exporter which scrapes remote JSON by JSONPath

json_exporter's Issues

Reading json sub level values with JSONPath filter expressions

I've been trying to use json_exporter to read an API with the following structure (simplified and no properties available for 2nd element)):

[	
    {
        "machineID": "SG_8284215608661916610",
        "online": true,
        "name": "Oxford Store",
        "uptimeMS": 56725789,
        "properties": [{
                "key": "siteNumber",
                "value": "321"
            }, {
                "key": "customer",
                "value": "Greenchurch"
            }
        ]
    }, {
        "machineID": "SG_3997078225428050463",
        "online": true,
        "name": "Liverpool Store",
        "uptimeMS": 56723784,
        "properties": []
    }
]

The following config allows me to extract all of the data except for the properties array:

---
metrics:
- name: simplehelp
  type: object
  help: Simplehelp machine Id
  path: $[*]
  labels:
    environment: simplehelp             # static label
    machineId: $.machineID              # dynamic label
    name: $.name
    state: $.state
    uptime: $.uptimeMS
    customer: $.properties[?(@.key=="customer")].value
    siteNumber: $.properties[?(@.key=="siteNumber")].value
  values:
    online: $.online

Prometheus output (simplified to reduce space, instance removed etc):

simplehelp_online{environment="simplehelp",machineId="SG_8284215608661916610",name="Oxford Store",uptime="56725789"} 1
simplehelp_online{environment="simplehelp",machineId="SG_3997078225428050463",name="Liverpool Store",uptime="56723784"} 1

I get an error for the 2 property values:

level=error ts=2020-11-10T16:54:21.423Z caller=collector.go:156 msg="Failed to compile path for label" path="$.properties[?(@.key==\"siteNumber\")].value" err="Expected number, key, or *, but got none"
level=error ts=2020-11-10T16:54:21.423Z caller=collector.go:156 msg="Failed to compile path for label" path="$.properties[?(@.key==\"customer\")].value" err="Expected number, key, or *, but got none"

if for the customer and siteNumber I do something like:

    customer: $.properties[1].value
    siteNumber: $.properties[0].value

then it extracts, but I can't guarantee the position in the array of either value.

I have asked the API maintainer if they can modify the output so the values are mapped like the following which I know I can extract:

"properties": {
    "siteNumber": "321",
    "customer": "greenchurch"
}

but it sounds like that might take a while to get done (if ever).

According to some online JSONPath emulators, the config I'm using with filters should work. Can you tell me if I'm doing something wrong or do filters not work in your implementation of JSONPath

ALSO,

It would be a good idea to include an example of configuring Prometheus for the probe stuff. It took me ages to track down how to do it.

Some refference for example/config.yml

Hello,

I would like to use your "prometheus json exporter", but I'm not a GO programmer.
Could you add some documentation about reference on what to write in a "config.yml" (example/config.yml)
How to create different types of metrics ...
Also, would it be possible to "tail -F /var/log/app.json" ??
Thanks,
Radu Popa

Metrics: path definition fails

I try to read out a Home Automation API. It has about 300 devices connected. Direct access of the api brings back something like:

[
  {
    "id": 1,
    "name": "zwave",
    "roomID": 0,
    "type": "com.fibaro.zwavePrimaryController",
    "baseType": "",
 ...
      "setSipUserID": 1,
      "setSipUserPassword": 1
    },
    "created": 1609413176,
    "modified": 1609413176,
    "sortOrder": 2
  },
  {
    "id": 3,
    "name": "Wetter",
    "roomID": 7,
...

With a defined metric

metrics:
- name: temperature_devices
  type: object
  help: Devices of HC2 with temperature mesurement
  path: '{$[0:310]}'
  labels:
    id: '{@.id}'
    type: '{@.type}'
    name: '{@.name}'
  values:
    id: '{@.id}'

the exporter brings back all devices:

# HELP temperature_devices_id Devices of HC2 with temperature mesurement
# TYPE temperature_devices_id untyped
temperature_devices_id{id="1",name="zwave",type="com.fibaro.zwavePrimaryController"} 1
temperature_devices_id{id="100",name="UG Az - Rollanden",type="com.fibaro.FGRM222"} 100
temperature_devices_id{id="101",name="UG Az - Steuerung",type="com.fibaro.remoteController"} 101
temperature_devices_id{id="102",name="UG Az - Heizung",type="com.fibaro.zwaveDevice"} 102
temperature_devices_id{id="103",name="UG Az - Heizung",type="com.fibaro.setPoint"} 103
temperature_devices_id{id="104",name="UG Az - Termometer",type="com.fibaro.temperatureSensor"```

If I change the path to
path: '{$[]}' the exporter isn't working
path: '{$[0:300][?(@.id == "107")]}' the exporter won't do it too.

So the filtering and unlimited requests seems not to work.
I attached the test data.json to the case.

Not able to build the project

I'm trying to build the project. It says build/src folder not found. Kindly help

Scrape JSON from a (HTTPS) REST API with bearer token authentication

I am a newbie at prometheus and exporters so beforehand, apologies for any stupid questions, assumptions, etc.

I have an application that exposes a REST API that I want to scrape so I can use the metrics in Prometheus/Grafana. The json-exporter looks like a good fit for this but after reading the documentation I concluded that I can only use the json_exporter to scrape REST API's that use plain HTTP.

The REST API I am dealing with only supports secured (SSL/HTTPS) API calls. More specifically, I have to get an OAUTH 2.0 token (POST with username, password and grant-type set to ' password') and next, with this (bearer) token I query all the metrics.

Is there any way to make this work with the json_exporter? Can I configure the json_exporter to scrape a HTTPS REST API endpoint? And does this work with bearer tokens? (the REST API endpoint does not support basic authentication)

jsonpath library unmaintained?

The jsonpath library in use (github.com/kawamuray/jsonpath) has not been updated in the past 5 years and thus, looks unmaintained. Maybe we should consider moving to k8s.io/client-go/util/jsonpath (or any other up to date library), which is expected to be well maintained and has nice documentation too - https://kubernetes.io/docs/reference/kubectl/jsonpath.

Question: nested json structure

Hi,

thank you very much for this neat handy tool.

I try to monitor a project where you can host your own online radio station: https://github.com/AzuraCast/AzuraCast

The most interesting statistics come from here: curl https://demo.azuracast.com/api/nowplaying

[
   {
      "station":{
         "id":1,
         "name":"AzuraTest Radio",
         "shortcode":"azuratest_radio",
         "description":"A test radio station.",
         "frontend":"icecast",
         "backend":"liquidsoap",
         "listen_url":"https://demo.azuracast.com/radio/8000/radio.mp3",
         "url":null,
         "playlist_pls_url":"https://demo.azuracast.com/public/azuratest_radio/playlist.pls",
         "playlist_m3u_url":"https://demo.azuracast.com/public/azuratest_radio/playlist.m3u",
         "is_public":true,
         "mounts":[
            {
               "path":"/radio.mp3",
               "is_default":true,
               "id":1,
               "name":"128kbps MP3",
               "url":"https://demo.azuracast.com/radio/8000/radio.mp3",
               "bitrate":128,
               "format":"mp3",
               "listeners":{
                  "current":0,
                  "unique":0,
                  "total":0
               }
            },
            {
               "path":"/mobile.mp3",
               "is_default":false,
               "id":2,
               "name":"64kbps MP3",
               "url":"https://demo.azuracast.com/radio/8000/mobile.mp3",
               "bitrate":64,
               "format":"mp3",
               "listeners":{
                  "current":0,
                  "unique":0,
                  "total":0
               }
            }
         ],
         "remotes":[
            
         ]
      },
      "listeners":{
         "current":0,
         "unique":0,
         "total":0
      },
      "live":{
         "is_live":false,
         "streamer_name":"",
         "broadcast_start":null
      },
      "now_playing":{
         "elapsed":21,
         "remaining":291,
         "sh_id":12,
         "played_at":1614584862,
         "duration":312,
         "playlist":"default",
         "streamer":"",
         "is_request":false,
         "song":{
            "id":"7fa040c6121bdea61f79c3d1afaf243b",
            "text":"Nitro Fun & Desso - Believe (feat. Brenton Mattheus)",
            "artist":"Nitro Fun & Desso",
            "title":"Believe (feat. Brenton Mattheus)",
            "album":"",
            "genre":"",
            "lyrics":"I searched the earth for you\r\nwade into the furthest oceans\r\nI made my way through\r\njust going through the motions\r\nI left to chase my dreams\r\nI'll find my way to you\r\nohhh I can see the homeward beams\r\nlike stars that shine for you\r\noh I believe in everything you said x2\r\nI searched the world for you \r\nchsing after both our future\r\nbelieved in everything, that you said\r\nso I had left to chase my dreams to \r\nfind a place in life\r\nyou showed me everything I need my\r\nhome is by your side\r\noh I believe in everything you said x2",
            "art":"https://demo.azuracast.com/api/station/1/art/5543a117d2bf08f65ac16dbb-1614582041.jpg",
            "custom_fields":[
               
            ]
         }
      },
      "playing_next":{
         "cued_at":1614584857,
         "duration":261,
         "playlist":"default",
         "is_request":false,
         "song":{
            "id":"b2dc32f124d9d0f37b517d7d2fd2e52a",
            "text":"Desh - Halla",
            "artist":"Desh",
            "title":"Halla",
            "album":"",
            "genre":"",
            "lyrics":"",
            "art":"https://demo.azuracast.com/api/station/1/art/e55d13660107291661c1651d-1614582043.jpg",
            "custom_fields":[
               
            ]
         }
      },
      "song_history":[
         {
            "sh_id":11,
            "played_at":1614584656,
            "duration":209,
            "playlist":"default",
            "streamer":"",
            "is_request":false,
            "song":{
               "id":"ae791aacc23c9322b64bcb56ec15881d",
               "text":"Luke Swanson - Amnesia",
               "artist":"Luke Swanson",
               "title":"Amnesia",
               "album":"",
               "genre":"",
               "lyrics":"",
               "art":"https://demo.azuracast.com/api/station/1/art/3f939c7fd05df64b800456f2-1614582041.jpg",
               "custom_fields":[
                  
               ]
            }
         },
         {
            "sh_id":10,
            "played_at":1614584377,
            "duration":280,
            "playlist":"default",
            "streamer":"",
            "is_request":false,
            "song":{
               "id":"f049ed1d7f7b65d0aa1a1d3dbb240fa2",
               "text":"Internet Empire - Imagine",
               "artist":"Internet Empire",
               "title":"Imagine",
               "album":"",
               "genre":"",
               "lyrics":"",
               "art":"https://demo.azuracast.com/api/station/1/art/ead642b456401c9ec7be3c2a-1614582042.jpg",
               "custom_fields":[
                  
               ]
            }
         },
         {
            "sh_id":9,
            "played_at":1614584212,
            "duration":170,
            "playlist":"default",
            "streamer":"",
            "is_request":false,
            "song":{
               "id":"064cc6e26a2cca5d0a6100e3ee30668c",
               "text":"RVDEKID - Lowkey",
               "artist":"RVDEKID",
               "title":"Lowkey",
               "album":"LO-FI FEELS",
               "genre":"",
               "lyrics":"",
               "art":"https://demo.azuracast.com/api/station/1/art/aba2a061e1f8b04e3e8ed085-1614582042.jpg",
               "custom_fields":[
                  
               ]
            }
         },
         {
            "sh_id":8,
            "played_at":1614583986,
            "duration":228,
            "playlist":"default",
            "streamer":"",
            "is_request":false,
            "song":{
               "id":"0299ae3ceb62eb822a5b2ffd92f31625",
               "text":"Avionics - Flipside",
               "artist":"Avionics",
               "title":"Flipside",
               "album":"",
               "genre":"",
               "lyrics":"",
               "art":"https://demo.azuracast.com/api/station/1/art/f7b6f58b66e825c474883d0f-1614582043.jpg",
               "custom_fields":[
                  
               ]
            }
         },
         {
            "sh_id":7,
            "played_at":1614583639,
            "duration":348,
            "playlist":"default",
            "streamer":"",
            "is_request":false,
            "song":{
               "id":"75dd3eb1f149634eb09df4a6e9cb249d",
               "text":"Lokan - Ex Nihilo",
               "artist":"Lokan",
               "title":"Ex Nihilo",
               "album":"",
               "genre":"",
               "lyrics":"",
               "art":"https://demo.azuracast.com/api/station/1/art/efff738c3f645f11ae1f8f59-1614582044.jpg",
               "custom_fields":[
                  
               ]
            }
         }
      ],
      "is_online":true,
      "cache":"redis"
   }
]

I tried following config and failed:

metrics:
- name: total_current_listeners
  path: "{ [*].listeners.current }"
  help: Example of a top-level global value scrape in the json
  labels:
    station: "{[*].station.name}"

- name: per_mount
  type: object
  path: "{ [*].station.mounts[*] }"
  help: num of current mount listeners
  labels:
    station: "{..name}"
    name: "{ .name }"
    bitrate: "{ .bitrate }"
    format: "{ .format }"
  values:
    listeners: "{ .listeners.current }"

Result:

# HELP per_mount_listeners num of current mount listeners
# TYPE per_mount_listeners untyped
per_mount_listeners{bitrate="128",format="mp3",name="128kbps MP3",station="128kbps MP3"} 0
per_mount_listeners{bitrate="64",format="mp3",name="64kbps MP3",station="64kbps MP3"} 0
# HELP total_current_listeners Example of a top-level global value scrape in the json
# TYPE total_current_listeners untyped
total_current_listeners{station="AzuraTest Radio"} 0

I can't get the station label per mount to work.

Also tried other options according to this page: https://kubernetes.io/docs/reference/kubectl/jsonpath/

Can anyone point me in the right direction please?

"skipping not structual" warning

I've got a flat json like so:

{
  "account_type": "standard",
  "cash_credits": "22.20300",
  "city": "",
  "timezone": "America/Chicago"
}

My config.yml is this:

- name: plivo
  type: object
  path: $.cash_credits
  values:
    credits: $.cash_credits

when it gets parsed by the json_exporter, I get this warning and no metrics...

WARN[2017-07-05T15:17:40-05:00] skipping not structual result;path:<&{$.cash_credits+ [%!s(*jsonpath.operator=&{3 0 0 false map[cash_credits:{}] [] [] []})] %!s(bool=true)}>,value:<"21.81800">

I'm guessing I've got something wrong in my config.yml, but not sure yet what I need to do.

Bug: cannot use two backslashes at the end of label string

If I use two backslashes in JSON-file at the end of label string, then JSON exporter exposes only metrics before this metric.

{
  "metrics": [
    {
        "path": "dpc\\root\\GRs\\PK\\gr047_mirror",
        "value": "0"
    },
    {
        "path": "dpc\\root\\GR\\PK\\",
        "value": "1"
    },
    {
        "path": "dpc\\root\\GRs\\PK\\gr048_mirror",
        "value": "1"
    }
  ]
}

Expected:

# HELP zabbix_metrics_value zabbix_metrics
# TYPE zabbix_metrics_value untyped
zabbix_metrics_value{path="dpc\\root\\GRs\\PK\\gr047_mirror"} 0
zabbix_metrics_value{path="dpc\\root\\GR\\PK\\"} 1
zabbix_metrics_value{path="dpc\\root\\GRs\\PK\\gr048_mirror"} 1

Result:

# HELP zabbix_metrics_value zabbix_metrics
# TYPE zabbix_metrics_value untyped
zabbix_metrics_value{path="dpc\\root\\GRs\\PK\\gr047_mirror"} 0

Enable CI to publish `latest` as well as `released` images

Currently there is only a single image with tag master in the docker hub repo - https://hub.docker.com/r/prometheuscommunity/json-exporter

Enable the CI to publish a latest tag at each git push event (to master branch) and a versioned tag at each git tag event.

only id data parsing

If I try to get data from below json format from my .net json api
{
"values": [
{
"id": "2021-01-19T17:36:59.627",
"count": "15.00.16.02.01",
"some_boolean": "15.00.16.02.011-01-01",
"module": "Batch",
"state": "ACTIVE"
},
{
"id": "2021-01-19T17:36:59.628",
"count": "15.00.16.02.02",
"some_boolean": "15.00.16.02.011-01-02",
"state": "INACTIVE"
},
{
"id": "2021-01-19T17:36:59.629",
"count": "15.00.16.02.03",
"some_boolean": "15.00.16.02.011-01-03",
"state": "ACTIVE"
}
]
}

Its only parsing "id"
rest all giving error
Kindly help me to get data from diff format given below
{
"values": [
{
"id": "2021-01-19T17:36:59.627",
"count": "15.00.16.02.01",
"some_boolean": "15.00.16.02.011-01-01",
"module": "Batch",
"state": "ACTIVE"
},
{
"id": "2021-01-19T17:36:59.628",
"count": "15.00.16.02.02",
"some_boolean": "15.00.16.02.011-01-02",
"module": "Batch1",
"state": "INACTIVE"
},
{
"id": "2021-01-19T17:36:59.629",
"count": "15.00.16.02.03",
"some_boolean": "15.00.16.02.011-01-03",
"module": "Batch2",
"state": "ACTIVE"
}
]
}

I tried to modify using given json but could get only "id" data into prom 2021-01-19T17:36:59.629, rest all giving error

waiting for your reply

Add [unit] tests

Allow specifying bind address

Currently there is only a --port option

Option to customize the interface to bind

It'd be nice to have an option to customize the interface(s) to be exposed.
For instance to only listen to localhost.

netstat -tupan | grep 7979
tcp6 0 0 :::7979 :::* LISTEN 20908/./json_export

number of metrics

Hi,
Why when I curl I get many metrics of http... can I get only the metrics in the config file?

jsonpath for length of an array

Is there a way to get length of an array?

Example:

- name: active_values_count
  type: object
  path: $.values[*]?(@.state == "ACTIVE").length()

Missing target URL variable, string helpers and one metrics for each array items

Hello, I tried to parse the following JSON with json_exporter

{
  "status": "disconnected",
  "topics": [
    {
      "topic": "production",
      "count": 606037,
      "lag": 485172
    },
   {
      "topic": "production",
      "count": 23403,
      "lag": 2
    },
    {
      "topic": "staging",
      "count": 121,
      "lag": 0
    }
  ]
}

But I failed to:

• access to the URL path to keep some part for a label. I want to extract the main value from https://example.org/main/topics
• create 2 metrics for both production topic
• create a metric based on the status with something like $.status == "connected"
• Do a simple math operation: we want the lag total for all topics

Here the most advanced config I got:

metrics:
    - name: json_topics
      type: object
      path: "{ .topics[*] }"
      help: Topic info
      labels:
        topic: "{ .topic }"
      values:
        lag: "{ .lag }"
        count: "{ .count }"

Thanks in advance

Extracting substrings from a key

More of a jsonpath question I guess, if I have a json like the following

{
    "route/api_1_test/POST/time/4XX.p9999": 1234
}

I'm trying to transform it into
finagle_http_time{status="4xx",route="api_test",method="POST",metric_type="p9999"}

- name: finagle_http_time
  path: $.route/(.+)/(.+)/time/(.+)\.(.+)
  labels:
    status: "$4"
    route: "$1"
    method: "$2"
    metric_type: "$5"

How do I extract the status value from the key? Is this possible with the json_exporter?

[FR] Add authorization with an API KEY token

Use case:
I am trying to scrape an endpoint with an API KEY аuthorization using Prometheus with json-exporter ( prom/prometheus:main and prometheuscommunity/json-exporter:v0.3.0 docker images):

docker run -d --network=host --name=prometheus -v /path/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:main --config.file=/etc/prometheus/prometheus.yml --web.listen-address=:9095

docker run -d --name=json-exporter --network=host -v /path/jsonexporter/config.yml:/config.yml prometheuscommunity/json-exporter:v0.3.0 --config.file=/config.yml

with the following Prometheus prometheus.yml configuration file:

global:
  evaluation_interval: 15s
  scrape_interval: 15s
  scrape_timeout: 10s
  external_labels:
    environment: prometheus_in_docker
	
scrape_configs:

  - job_name: json_exporter_self
    static_configs:
      - targets:
        - <host>:7979 ## Location of the json exporter's real <hostname>:<port>

  ## gather the metrics from third party json sources, via the json exporter
  - job_name: json_exporter_data
    metrics_path: /probe
    authorization:
      credentials: '{my_secret_token}'
      type: apikey	
    static_configs:
      - targets:
        - http://<endpoint_host>:<port>/api/v1/data
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <host>:7979 ## Location of the json exporter's real <hostname>:<port>

and the following json-exporter config.yml configuration file:

metrics:
- name: example_metric
  path: "{ .data }"
  help: Example 
  labels:
    environment: testdata # static label

prom/prometheus:main and prometheuscommunity/json-exporter:v0.3.0 are up and running, self scraping works fine (http://:7979/metrics), but a json scraping doesn't work (http://:7979/probe), the error is the following:

Failed to fetch JSON response. TARGET: http://<endpoint_host>:/api/v1/data, ERROR: 401 Unauthorized

The endpoint itself is available:
curl -X GET 'http://<endpoint_host>:/api/v1/data' -H 'Authorization: {my_secret_token}'

Feature request:
Could you please implement authorization with API KEY token?

(please see link with additional discussion in the Prometheus repository)

panic: runtime error: slice bounds out of range

Operating systems reproduced in:
OSX 10.14.1 (18B75)
ubuntu-18.04 (vagrant box)

Steps to Reproduce:

1. Clone the repo git clone https://github.com/kawamuray/prometheus-json-exporter.git
2. Follow the readme instructions to build.

./gow build -o json_exporter .```
3. Spin up the python simplehttpserver
`$ python -m SimpleHTTPServer 8000`
4. Execute `json_exporter`
`./json_exporter http://localhost:8000/example/data.json example/config.yml`

Expected Results:
```$ ./json_exporter http://localhost:8000/example/data.json example/config.yml &
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_global_value>
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_active>
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_count>
127.0.0.1 - - [08/Feb/2016 22:44:38] "GET /example/data.json HTTP/1.1" 200 -```

Add rustycl0ck as a maintainer

I would like to invite @rustycl0ck as a maintainer if they're interested.

Support http config

• Pass basic auth parameters
• Pass TLS client parameters

Update README

Add notice about the current development phase of the project and disclaimer about possible backward incompatibility in future versions

panic: runtime error: slice bounds out of range

Operating systems reproduced in:
OSX 10.14.1 (18B75)
ubuntu-18.04 (vagrant box)

Go Version:
go version go1.11.4 darwin/amd64

Steps to Reproduce:

1. Clone the repo git clone https://github.com/kawamuray/prometheus-json-exporter.git
2. Follow the readme instructions to build.

./gow get .
./gow build -o json_exporter .

3. Spin up the python simplehttpserver
   $ python -m SimpleHTTPServer 8000
4. Execute json_exporter
./json_exporter http://localhost:8000/example/data.json example/config.yml

Expected Results:

$ ./json_exporter http://localhost:8000/example/data.json example/config.yml &
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_global_value>
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_active>
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_count>
127.0.0.1 - - [08/Feb/2016 22:44:38] "GET /example/data.json HTTP/1.1" 200 -

Actual Result:

vagrant@vagrant:~/goworkspace/src/github.com/kawamuray/prometheus-json-exporter$ ./json_exporter http://localhost:8000/example/data.json example/config.yml
INFO[2019-01-31T22:10:00Z] metric registered;name:<example_global_value>
INFO[2019-01-31T22:10:00Z] metric registered;name:<example_value_active>
INFO[2019-01-31T22:10:00Z] metric registered;name:<example_value_count>
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
github.com/kawamuray/jsonpath.(*stack).pop(...)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/stack.go:126
github.com/kawamuray/jsonpath.evaluatePostFix(0xc00008d360, 0x3, 0x4, 0xc000137508, 0xc0001375c0, 0x0, 0x0, 0x0)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/expression.go:348 +0x3c5f
github.com/kawamuray/jsonpath.(*exprBucket).evaluate(0xc0001377b8, 0xc000137938, 0x7b0543, 0xaaaaaaaaaaaaaaaa)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/eval.go:305 +0x43f
github.com/kawamuray/jsonpath.(*query).trySpillOver(0xc0000a1680)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/eval.go:162 +0x159
github.com/kawamuray/jsonpath.pathEndValue(0xc0000a1680, 0xc00012c5a0, 0xc0000f4d10, 0x8f4048)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/eval.go:282 +0x271
github.com/kawamuray/jsonpath.(*Eval).Iterate(0xc00012c5a0, 0xc000138e70, 0x1)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/eval.go:102 +0x375
github.com/kawamuray/jsonpath.(*Eval).Next(0xc00012c5a0, 0x15f, 0x200)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/jsonpath/eval.go:140 +0x3e
github.com/kawamuray/prometheus-json-exporter/jsonexporter.(*ValueScraper).forTargetValue(0xc000059ac0, 0xc00012a200, 0x15f, 0x200, 0xc000137bb0, 0xc000138390, 0x7e6df0)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/prometheus-json-exporter/jsonexporter/scraper.go:51 +0xc3
github.com/kawamuray/prometheus-json-exporter/jsonexporter.(*ObjectScraper).Scrape(0xc00000dc00, 0xc00012a200, 0x15f, 0x200, 0xc00000e0c0, 0x0, 0x0)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/prometheus-json-exporter/jsonexporter/scraper.go:149 +0x89
github.com/kawamuray/prometheus-json-exporter/jsonexporter.(*Collector).Collect(0xc000138390, 0xc00000e0c0)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/prometheus-json-exporter/jsonexporter/collector.go:77 +0xe5
github.com/kawamuray/prometheus-exporter-harness/harness.(*exporter).main(0xc00000e0b0, 0xc0000a4c60)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/prometheus-exporter-harness/harness/exporter.go:71 +0x2ed
github.com/kawamuray/prometheus-exporter-harness/harness.(*exporter).main-fm(0xc0000a4c60)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/prometheus-exporter-harness/harness/main.go:37 +0x34
github.com/urfave/cli.HandleAction(0x830380, 0xc0000594a0, 0xc0000a4c60, 0x0, 0x0)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/urfave/cli/app.go:503 +0x7c
github.com/urfave/cli.(*App).Run(0xc0000ae540, 0xc00001e090, 0x3, 0x3, 0x0, 0x0)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/urfave/cli/app.go:268 +0x5b7
github.com/kawamuray/prometheus-exporter-harness/harness.Main(0xc00010f880)
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/build/src/github.com/kawamuray/prometheus-exporter-harness/harness/main.go:66 +0x5d
main.main()
	/home/vagrant/goworkspace/src/github.com/kawamuray/prometheus-json-exporter/json_exporter.go:12 +0xc7

NOTE: The extremely strange thing about this issue is, if I debug this in VSCode with the launch.json below the app works perfectly fine.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceRoot}/json_exporter.go",
            "env": {},
            "args": ["http://localhost:8000/example/data.json", "example/config.yml"]
        }
    ]
}

Failed to execute jsonpath.

I know this is not the best looking json but that's all i get to work with from our developers. Any chance to make this work?
jsonpath.com is showing that it should work.

Json:
{ "status": "ok", "info": { "EVENT_GRID": { "status": "up", "system": "Event Grid", "details": "Event Grid is online." } } }

Query:
`

• name: dependencies_down
  type: object
  help: API Check
  path: '{.info[?(@.status == "up")]}'
  labels:
  dep_name: '{.system}'
  values:
  on: 0
  `

Error:
son_exporter | level=error ts=2021-03-04T23:52:03.713Z caller=collector.go:128 msg="Failed to execute jsonpath" err="map[EVENT_GRID:map[details:Event Grid is online. status:up system:Event Grid]] is not array or slice and cannot be filtered" path="{.info[?(@.status==\"down\")]}" data="{\"status\":\"ok\",\"info\":{\"EVENT_GRID\":{\"status\":\"up\",\"system\":\"Event Grid\",\"details\":\"Event Grid is online.\"}},\"error\":{},\"details\":{\"EVENT_GRID\":{\"status\":\"up\",\"system\":\"Event Grid\",\"details\":\"Event Grid is online.\"}}}" json_exporter | level=error ts=2021-03-04T23:52:03.713Z caller=collector.go:69 msg="Failed to extract json objects for metric" err="map[EVENT_GRID:map[details:Event Grid is online. status:up system:Event Grid]] is not array or slice and cannot be filtered" metric="Desc{fqName: \"dependencies_up_on\", help: \"API Check\", constLabels: {}, variableLabels: [dep_name]}"

Thanks in advance.

publish binary releases

We should publish binary releases

Migrate to Prometheus Community?

Is there any interest in maintaining this exporter? Maybe we could move it to the Prometheus Community repos?

https://github.com/prometheus-community

Using timestamp from api as timestamp in promehteous metric

Is there a way to use timestamp from the api data as timestamp in output result? in promehteous documentation in instrumentation part there is :

metric_name [
  "{" label_name "=" `"` label_value `"` { "," label_name "=" `"` label_value `"` } [ "," ] "}"
] value [ timestamp ]

I have this data from my api:

{
  "data": {
    "object1": {
      "label": "test1",
      "value": 25551.671688,
      "timestamp": 1613115501
    },
    "object2": {
      "label": "test1"
      "value": 20630.985927,
      "timestamp": 1613115603
    }
}

is it possible to get output like this?

example_metrics{label="test1"}   20630.985927   1613115603
example_metrics{label="test1"}   25551.671688   1613115501

Prometheus Exporters Hub by this repository! Thanks! :)

Removed

"Failed to parse value as float" error when the config file from the example is used

Steps to reproduce:

• Start python server

python3 -m http.server

data.json file with content from example

• Start docker container

docker run -d --network=host -v /opt/jsonexporter/config.yml:/config.yml prometheuscommunity/json-exporter --config.file=/config.yml

config.yml file with content from example

• Add target info to prometheus.yml configuration file

  - job_name: json_exporter
    static_configs:
      - targets:
        - <host_ip>:7979 ## Location of the json exporter's real <hostname>:<port>

  ## gather the metrics from third party json sources, via the json exporter
  - job_name: jsonexporter
    metrics_path: /probe
    static_configs:
      - targets:
        - http://<host_ip>:8080/data.json
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: <host_ip>:7979 ## Location of the json exporter's real <hostname>:<port>

Current behavior:
prometheus-community/json_exporter is up and running, self scraping is working (http://<host_ip>:7979/metrics), but metrics scraping from the endpoint (data.json) file is failed with a error:

level=error ts=2021-02-10T11:06:50.866Z caller=collector.go:55 msg="Failed to extract float value for metric" path="{ .counter }" err="failed to parse value as float; value: \"{ .counter }\"; err: strconv.ParseFloat: parsing \"{ .counter }\": invalid syntax" metric="Desc{fqName: \"example_global_value\", help: \"Example of a top-level global value scrape in the json\", constLabels: {environment=\"beta\",location=\"planet-{.location}\"}, variableLabels: []}"
level=error ts=2021-02-10T11:06:50.866Z caller=collector.go:68 msg="Failed to compile path" path="{.values[?(@.state == \"ACTIVE\")]}" err="Expected $ or @ at start of path instead of  U+007B '{'"

I have tried to change the content of the data.json file, like the syntax in the config.yml file, but that did not work. Also I have tried to use public endpoints instead of my python test server - the result is not comforting

Question:
Is this a bug or misconfiguration?
How to use json-exporter in docker container with the configuration from the example?

Add support for `POST` method

Can we add support to POST data to the remote endpoint.

Problems with "null" values: possibly return 0 or a default value?

I am trying to scrape the JSON data feed off a hardware appliance with json_exporter, but the problem is that it sometimes returns a number, but other times returns null. This causes non-continuous metrics in Prometheus, which causes a cascade of difficulties.

The hardware appliance in question is an SMA Sunny Boy solar inverter, and I'm trying to avoid writing my own shim to ingest the data into Prometheus. The JSON it outputs looks roughly like (snipped for clarity):

{"result": { "0199-B3549FA1": {"6100_40263F00": {"1": [{"val": 1200}]}}}}

Using this config:

metrics:
- name: pv_power
  path: "{ .result..['6100_40263F00']..val }"

I get the sane value of:

pv_power 1200

However, when the sun sets, the solar inverter for some bizarre reason outputs null:

{"result": { "0199-B3549FA1": {"6100_40263F00": {"1": [{"val": null}]}}}}

This results in the metric becoming NaN:

pv_power NaN

And this makes it impossible to do certain formulas in Prometheus (e.g. calculating my house power (pv_power - grid_power_supplied) + grid_power_absorbed because when one of the metrics is missing, the entire calculation fails).

To avoid writing my own shim to parse the input JSON, I would need either NaN to be interpreted as 0, or have the ability to specify a default value.

Multiple end points?

I use this exporter for a bespoke system spitting out some json content. Works great however I have another 100+ application servers I would like to pull into Prometheus.

How does this product scale? From my limited understanding the read me suggests only a single url and config can be passed into the exporter.

Can you please advise how I can use this on scale for multiple instances ?

Switch the cli flags to kingpin

To make this more like the other exporters

Failed to convert extracted value to float64" path="{ .results }"

Hello,
I just deployed recently json_exporter.
We have some endpoints where we expose some metrics in json format, but apparently the json_exporter is complaining about it.

Here is the log from the exporter:
level=error ts=2021-03-02T15:08:36.542Z caller=collector.go:63 msg="Failed to convert extracted value to float64" path="{ .results }" value="{\"cron-is-running\":{\"description\":\"\",\"duration\":0.0002779960632324219,\"name\":\"Cron is running\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"f-c-m-notifications\":{\"description\":\"There are successful sent notifications\",\"duration\":0.0017249584197998047,\"name\":\"FCM Notifications\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"guestbook-retention-was-processed\":{\"description\":\"\",\"duration\":0.0002741813659667969,\"name\":\"Guestbook retention was processed\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"news-synchronize\":{\"description\":\"\",\"duration\":0.0002570152282714844,\"name\":\"news-synchronize\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"queue-fail-jobs\":{\"description\":\"Checks if queue contains failed jobs\",\"duration\":0.0013568401336669922,\"name\":\"Queue fail jobs\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"queue-works\":{\"description\":\"Checks if queue job is processed last 3 minutes\",\"duration\":0.00030303001403808594,\"name\":\"Queue works\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"reports-retention-was-processed\":{\"description\":\"\",\"duration\":0.0002760887145996094,\"name\":\"Reports retention was processed\",\"reason\":\"\",\"status\":\"success\",\"tags\":[\"basic\"]},\"reservations-retention-was-processed\":{\"description\":\"\",\"duration\":0.0002760887145996094,\"name\":\"Reservations retention was processed\

Our json looks like this:

{"results":{"throttling":{"name":"Throttling","description":"Throttling limit","status":"success","reason":"","duration":0.0076601505279541016,"tags":["basic"]},"queue-fail-jobs":{"name":"Queue fail jobs","description":"Checks if queue contains failed jobs","status":"failed","reason":"There are 99 failed jobs. Please fix or forget them.","duration":0.016350984573364258,"tags":["basic"]},"f-c-m-notifications":{"name":"FCM Notifications","description":"There are successful sent notifications","status":"success","reason":"","duration":0.0019271373748779297,"tags":["basic"]},"scheduled-notifications":{"name":"Scheduled Notifications","description":"There are no waiting notifications","status":"success","reason":"","duration":0.0015687942504882812,"tags":["basic"]},"signage-a-p-i":{"name":"Signage API","description":"Signage API responses with code 2xx","status":"success","reason":"","duration":0.4234778881072998,"tags":["basic"]},"synchronizing-events":{"name":"Synchronizing events","description":"","status":"success","reason":"","duration":0.0007641315460205078,"tags":["basic"]},"synchronizing-transports":{"name":"Synchronizing transports","description":"","status":"success","reason":"","duration":0.0005240440368652344,"tags":["basic"]},"cron-is-running":{"name":"Cron is running","description":"","status":"success","reason":"","duration":0.0005209445953369141,"tags":["basic"]},"news-synchronize":{"name":"news-synchronize","description":"","status":"success","reason":"","duration":0.0004658699035644531,"tags":["basic"]},"guestbook-retention-was-processed":{"name":"Guestbook retention was processed","description":"","status":"success","reason":"","duration":0.0004639625549316406,"tags":["basic"]},"reservations-retention-was-processed":{"name":"Reservations retention was processed","description":"","status":"success","reason":"","duration":0.00037384033203125,"tags":["basic"]},"reports-retention-was-processed":{"name":"Reports retention was processed","description":"","status":"success","reason":"","duration":0.0002779960632324219,"tags":["basic"]},"queue-works":{"name":"Queue works","description":"Checks if queue job is processed last 3 minutes","status":"success","reason":"","duration":0.0003039836883544922,"tags":["basic"]}},"stats":{"time":0.4546797275543213,"time_per_test":0.034975363658024713,"number_of_tests":13},"succes

the config of the exporter is pretty simple:
`---
metrics:

• name: lifecheck
  path: "{ .* }"
  help: scrapes the entire json
  labels:
  environment: lifecheck-inside-server`

What can I do to avoid this error?
Thanks

Can this read a local json file ?

by f.ex. giving it a url like: file:///my/path/file.json ?

Unable to define "$" (read: the entire document} as the path

In the following example, data1 and config1 do not behave as expected, but data2 and config2 do.

Example Usage

$ cat example/data1.json
{
     "id": "id-A",
    "count": 1,
    "state": "ACTIVE"
}

$ cat example/data2.json
{
    "thing":
    {
         "id": "id-A",
        "count": 1,
        "state": "ACTIVE"
     }
}

$ cat example/config1.yml
- name: example_value1
  type: object
  path: $
  labels:
    environment: beta # static label
    id: $.id          # dynamic label
  values:
    active: 1      # static value
    count: $.count # dynamic value

$ cat example/config2.yml
- name: example_value2
  type: object
  path: $.thing
  labels:
    environment: beta # static label
    id: $.id          # dynamic label
  values:
    active: 1      # static value
    count: $.count # dynamic value


$ python -m SimpleHTTPServer 8000 &
Serving HTTP on 0.0.0.0 port 8000 ...

$ ./json_exporter http://localhost:8000/example/data1.json example/config1.yml &
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_active>
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_count>
127.0.0.1 - - [08/Feb/2016 22:44:38] "GET /example/data.json HTTP/1.1" 200 -

$ curl http://localhost:7979/metrics | grep ^example

$ kill xxxxx

$ ./json_exporter http://localhost:8000/example/data2.json example/config2.yml &
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_active>
INFO[2016-02-08T22:44:38+09:00] metric registered;name:<example_value_count>
127.0.0.1 - - [08/Feb/2016 22:44:38] "GET /example/data.json HTTP/1.1" 200 -

$ curl http://localhost:7979/metrics | grep ^example
example_value2_active{environment="beta",id="id-A"} 1
example_value2_count{environment="beta",id="id-A"} 1

Update logging

Update the logging library to use prometheus/common/promlog.

Fails to start

failed to parse yaml;err:<yaml: unmarshal errors:\n  line 2: cannot unmarshal !!map into []*jsonexporter.Config>

Don't even know what that's supposed to mean.

Terminal output on ubuntu 20.04 LTS

Hi,
at the moment I try to learn more about Prometheus and tried to get the example data into the system.
It's working, but I got some annoying messages, on every scrape.
Do you have an idea how to disable it?

Thank you for your help.
Ingmar

Enable CI for PRs

Enable CI to run build and tests on PRs

Copy blackbox_exporter's support of basic_auth for targets

Blackbox_exporter supports basic_auth block for targets.

  # The HTTP basic authentication credentials for the targets.
  basic_auth:
    [ username: <string> ]
    [ password: <secret> ]
    [ password_file: <filename> ]

Json_exporter should too.

Reference array index

Probably I miss the correct JSONPath syntax. I have a target giving output like:
"meters": [ { "power": 0, "is_valid": true, "total": 999 }, { "power": 0, "is_valid": true, "total": 808 } ]
How do I refer in a label to the array index, if no unique ID is present as in your example?

Create a Helm chart for deployment to k8s

Currently I could not find any charts repo in prometheus or prometheus-community org. Is there a place where prometheus charts are stored, or is there any helm charts repo from where prometheus related charts can be consumed?

This is required because the helm/charts repo has been deprecated and the charts present there are neither accepting new features nor new charts.

@SuperQ @roidelapluie Any idea if having a separate charts repo for prometheus is in a discussion phase or not?

Unable to customise the exposed 7979 port

If I want to run two json-exporter processes on the same machine

This could become somewhat less important if #4 is merged 👍

Does not appear to work atm?

I would love to use this, but does not appear to build atm? (Running on Linux Mint 17.3, 64 bit intel)

# github.com/Sirupsen/logrus
build/src/github.com/Sirupsen/logrus/logger.go:32: undefined: sync.Pool
# github.com/prometheus/client_golang/prometheus
build/src/github.com/prometheus/client_golang/prometheus/registry.go:450: syntax error: unexpected range, expecting {
build/src/github.com/prometheus/client_golang/prometheus/registry.go:564: non-declaration statement outside function body
build/src/github.com/prometheus/client_golang/prometheus/registry.go:565: non-declaration statement outside function body
build/src/github.com/prometheus/client_golang/prometheus/registry.go:566: syntax error: unexpected }

Parsing, splitting etc... strings

Hello, dear team

Trying to utilize your handy exporter to parse some monitoring data from some app.
Having an a issue while trying to parse this sample json output, e.g:
{ "Timestamp": "2020-Nov-10 04:15:38", "ProcessInfo": { "started": "2020-Oct-05 21:57:42", "uptime": "846:17:56", "cpu_usage": "1%" } }

with this sample config:
- name: ProcessInfo type: object help: Process CPU usage path: $.ProcessInfo labels: environment: beta # static label values: cpu_usage: $.cpu_usage

I end up getting a reasonable error:
level=error ts=2020-11-10T11:13:15.653Z caller=collector.go:81 msg="Failed to extract value" path=$.cpu_usage err="failed to parse value as float; value: \"0%\"; err: strconv.ParseFloat: parsing \"0%\": invalid syntax"

since "%" character can`t be parsed as float....

Unfortunately can`t find a way to transform the "$.cpu_usage".
Is is possible to somehow split the string and to filter out unwanted characters?
Is there a way to mutate the e.g. DATE and UPTIME values to ones accepted by this exporter and further prometheus scrapper job?

Thank you in advance.

Multi probe target with different username/password

Hi, I'm a beginner of json_exporter and wondering is it possible to probe multiple target with different auth info

What i want:
one json_exporter container that can probe exampleA.com/dataA.json with basic auth userName1/passwd1,
and probe exampleB.com/dataB.json with userName2/passwd2 in the same time.

looking forward to you kindly replys.

Set the listening port manually

Is there a way to manually specify the default port number 7979? I would like to run two or more instances of this exporter on the same box, thanks.

deep path's don't work

$.key.object

or any variation

$.key.object+

does not work on simple json :

{"key": {"object:"value"} }