Authentication 401 Error on River

[searchisko]

Authentication 401 Error on River

[velias]

1. Are you sure you configured username and password correctly? It have to be in remote part of configuration, not index as you wrote. You can look for INFO log message Configured GET JSON remote client. S... where you can see configured username.

2. River supports HTTP Basic authentication for now only. I see some sights of NTLM in your log. Are you sure your curl used Basic auth not NTLM (as it seems curl supports NTLM)

[Chadwiki]

I have this set in the remote part of the config, Right beneath the URL.
The target URL accepts basic authentication.
I will check the Curl command in the morning.

Any other suggestions,
I started to looking CA cert issue. I had tested another URL with curl and was forced to use the -k or insecure switch. Not on the URL I'm using for this config.

I can paste my config later.

Btw thanks for the help on the previous issues.
I should have known the fields issue. Uses the same style of config in the jira river. ;-) basically same code base.

[velias]

Try curl with --basic which should force it to use Basic authentication only.
Other thing is that river used Basic authentication in Pre-emptive mode, which means it automatically sends necessary auth header in first request, without waiting for server to challenge it over 401. Most REST API's need this mode because they return anonymously available data in preemptive mode is not used. I'm not sure if it is possible to simulate this mode over curl to check you server support it.

Regarding CA. If you use https server which certificate is not signed by common trusted authority, then you have to import https server certificate into cacerts store of JVM used to run river. Eg. http://codebistro.com/2010/03/25/adding-cacert-to-the-java-trusted-store/

Yep, this river is generalization of a bit older jira river, shares base ideas and some codebase definitely.

[Chadwiki]

CURL with forced --basic works

[Chadwiki]

This issue is on hold until API URL target is back online

[Chadwiki]

Here is an update on curl, output, river config - I am still receiving an 401. I'm not sure if its the username field or something else. I did remove the data Dir on test system and rebuilt the river.
I have checked the Cert Chain, I'm at a loss...

CURL:
curl --user username@companyid:password "https://companyid.saas.appdynamics.com/controller/rest/applications/ServiceMgmnt/metric-data?metric-path=Overall%20Application%20Performance%7CErrors%20per%20Minute&time-range-type=BEFORE_NOW&duration-in-mins=15&output=json" -k --basic

OUTPUT:

[{
"frequency": "ONE_MIN",
"metricName": "BTM|Application Summary|Errors per Minute",
"metricPath": "Overall Application Performance|Errors per Minute",
"metricValues": [ {
"count": 1356,
"current": 8,
"max": 0,
"min": 0,
"occurrences": 0,
"standardDeviation": 0,
"startTimeInMillis": 1400007600000,
"sum": 388,
"useRange": false,
"value": 26
}]
}]

RIVER CONFIG:

{
"type" : "remote",
"remote" : {
"urlGetDocuments" : "https://companyid.saas.appdynamics.com/controller/rest/applications/ServiceMgmnt/metric-data?metric-path=Overall%20Application%20Performance%7CErrors%20per%20Minute&time-range-type=BEFORE_NOW&duration-in-mins=15&output=json",
"getDocsResFieldDocuments" : "metricValues",
"username" : "username@companyid",
"password" : "password",
"timeout" : "30s",
"spacesIndexed" : "MAIN",
"spaceKeysExcluded" : "",
"indexUpdatePeriod" : "5m",
"indexFullUpdatePeriod" : "4h",
"simpleGetDocuments" : "true",
"maxIndexingThreads" : 1
},
"index" : {
"index" : "remote_river_index",
"type" : "appdynamics",
"remote_field_document_id" : "metricPath",
"fields" : {
"frequency" : {"remote_field" : "frequency"},
"metricPath" : {"remote_field" : "metricPath"},
"count" : {"remote_field" : "count"},
"current" : {"remote_field" : "current"},
"max" : {"remote_field" : "max"},
"min" : {"remote_field" : "min"},
"occurances" : {"remote_field" : "occurances"},
"standardDiviation" : {"remote_field" : "standardDiviation"},
"startTimeInMillis" : {"remote_field" : "startTimeInMillis"},
"sum" : {"remote_field" : "sum"},
"useRange" : {"remote_field" : "useRange"},
"value" : {"remote_field" : "value"}
}

},
"activity_log": {
    "index" : "remote_river_activity",
    "type"  : "remote_river_indexupdate"
}

}

ERROR in River:
error_message: "Failed remote system HTTP GET request to the url 'https://companyid.saas.appdynamics.com/controller/rest/applications/ServiceMgmnt/metric-data?metric-path=Overall%20Application%20Performance%7CErrors%20per%20Minute&time-range-type=BEFORE_NOW&duration-in-mins=15&output=json'. HTTP error code: 401 Response body: <title>GlassFish Server Open Source Edition 3.1.2.2 - Error report</title><style type="text/css"></style>

HTTP Status 401 -

---

type Status report

message

descriptionThis request requires HTTP authentication ().

---

GlassFish Server Open Source Edition 3.1.2.2

"

[Chadwiki]

Maybe this will help. it was a successful test using Java...

URL url = new URL("https://company.saas.appdynamics.com/controller/rest/applications/");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
String userpass = " username@companyid:password ";
String basicAuth = "Basic " + javax.xml.bind.DatatypeConverter.printBase64Binary(userpass.getBytes());
conn.setRequestProperty ("Authorization", basicAuth);

            if (conn.getResponseCode() != 200) {
                    throw new RuntimeException("Failed : HTTP error code : "
                                    + conn.getResponseCode());
            }

            BufferedReader br = new BufferedReader(new InputStreamReader(
                    (conn.getInputStream())));

            String output;
            while ((output = br.readLine()) != null) {
                    System.out.println(output);
            }

            conn.disconnect();

[velias]

River uses HttpClient for calls. Exact code can be found at https://github.com/searchisko/elasticsearch-river-remote/blob/master/src/main/java/org/jboss/elasticsearch/river/remote/HttpRemoteSystemClientBase.java
Exact version of HttpClient is:

 <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.2.6</version>
  </dependency>

If you should test/debug exactly this code and httpclient version with your server and credentials it should help probably.

[Chadwiki]

I'm back to this issue.
ES - 1.3.2 + Remote_plugin 1.5.0
I have tested forcing the -basic auth in CURL, which works correctly.

FYI - I am working on a clean no old _river system.
I was able to connect to the open API test endpoint. This one does not require authentication. This proved the plugin works... {{ http://docs.appdynamics.com/download/attachments/20187207/REST_WildCardBT_metric-dataJSON.txt?version=1&modificationDate=1394226069000&api=v2 }}

The error has not changed and no other details are available in the logs...

[Chadwiki]

error_message: "Failed remote system HTTP GET request to the url 'https://company.saas.appdynamics.com/controller/rest/applications/SEAP%20-%20CE%20-%20Production%20-%201/metric-data?metric-path=Backends%7CDefault%20Web%20Site/ClaqServices%7CAverage%20Response%20Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=15&output=json'. HTTP error code: 401 Response body: <html xmlns="http://www.w3.org/1999/xhtml\"><title>GlassFish Server Open Source Edition 3.1.2.2 - Error report</title><style type="text/css"></style>

HTTP Status 401 -

---

type Status report

message

descriptionThis request requires HTTP authentication ().

---

GlassFish Server Open Source Edition 3.1.2.2

"

[velias]

I just realized that you probably store password into incorrect field of river configuration. pwd should be used instead of password

[Chadwiki]

{
"type" : "remote",
"remote" : {
"urlGetDocuments" : "https://company.saas.appdynamics.com/controller/rest/applications/SEAP%20-%20CE%20-%20Production%20-%201/metric-data?metric-path=Backends%7CDefault%20Web%20Site/ClaqServices%7CAverage%20Response%20Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=15&output=json",
"username" : "company@servicemgmt",
"pwd" : "passw0rd",
"timeout" : "5s",
"spacesIndexed" : "MAIN",
"spaceKeysExcluded" : "",
"indexUpdatePeriod" : "1m",
"indexFullUpdatePeriod" : "0",
"simpleGetDocuments" : "true",
"maxIndexingThreads" : 2
}, ......