Bug: No response returned if ldap server is unreachable

[travisghansen]

Bug: No response returned if ldap server is unreachable

[travisghansen]

Interesting, ldap may have a timeout setting too...lemme check.

Does it resolve and the service is simply not available? Or can it not even resolve?

[leojonathanoh]

@travisghansen it's not resolvable, because even the kubernetes service is not available at that point

[travisghansen]

Look at the doc here: https://github.com/vesse/node-ldapauth-fork/blob/master/README.md

Try setting the connectTimeout parameter.

More details here: https://github.com/ldapjs/node-ldapjs/blob/v1.0.1/docs/client.md

We depend on the first which in turn depends on the second. You should be able to pass the options in though via the connection property as we just pass them through.

[leojonathanoh]

@travisghansen ok I will try the connectTimeout parameter.

I will also try extending the pipeline to:

• htpasswd
• ldap
• htpasswd

to see how eas logs show. If it gets to the third auth step, I suppose, ldap timeout might not be an issue. brb

[travisghansen]

Actually, I misunderstood too, we should be returning a 503 for sure (or some response) regardless. Did you not see a 503 returned in the logs?

[leojonathanoh]

@travisghansen there was no response. I got a 500 after 240s.

[leojonathanoh]

@travisghansen No 503 in the eas logs

[travisghansen]

Actually that might make sense. Drop the timeout below 240s and let me know what you get..we have a high level try/catch which should return a 503 for anything not being sanely watched regardless.

Anyway, let me know..

[leojonathanoh]

@travisghansen

Tried ConnectTimeout, did not help. I tried this with a 3 auth plugin pipeline, it gets stuck on the ldap, not proceeding to the last auth plugin.

On the first visit, where no is no Authorization request header, the auth goes through the entire pipeline, with each failing with 401, and the pipeline fails and returns a 401. The user is presented with login form, and enters the basic auth creds. However, same thing happens, and the pipeline gets stuck at ldap (plugin 1), not proceeding to plugin 2.

Pipeline:

• htpasswd
• ldap
• htpasswd

ingress.yaml annotation:

  annotations:
    # Enable authentication - Forward auth to external-auth-server
    ingress.kubernetes.io/auth-type: forward
    ingress.kubernetes.io/auth-url: http://external-auth-server.external-auth-server.svc.k8s:8080/verify?fallback_plugin=0&config_token=NDen459%2FIQnT%2B9oCNAdyaELqmD3TSv5PGsqyXp5YnbCPn2zTOmlYlvHRSrWbXOwFiS%2BUyNsU67tt%2FfSp5yshlg%2B7WUpglwvvdr%2FoAyttdtWRnkdqRQ70tC6YB%2BgM%2FMBytuADlW9bvcBhe1pxETP6Ij4EztON3otwnlNkjrMIhJLzSsmtyztO2NPzArKRGTr%2FxS%2Fpm5VvalzyhD8LYl3T%2BQLb1AvpilzBwnOIA7nKCbJe2qdBfgiQe5H6AqCtb5Q7MxMdStxH1uepSmwhKd7kcBZOVwLnKiKDu%2FwskwLcFCU6bhnV3kT9nR8%2BdR%2Fdz%2FsdpFqLbVkH9RcxxOwRnsPg91gdYWeb8vUvlal6IcAQ%2BFPxEqf74uOD9PJNrFlb95R8kOKmtqBpSVzTpjqwkqhUaFloVaDv7wANh2x4D%2FdG1%2BapjSEcA8AM4jkF3xQUomiyPNZUGzMZfGZDBWn7VqMxlmyXpcdOL18mbS3f40Eav27KNfl0dDyNKfrf38zvto5wv16igOhBJenqhc2TE916URyZD%2FDqg%2FkK9piGWtoWUQyqsjwzM14C6okY7fOxiRYN3yN9%2FwtHlKUFECvvCDaNhgLohC%2BiB2mUwS1K4Wcerq1tMtLps0DsmGOwUIAPthL0cU%2FvwSa8wTjTnvZxLW6OBCJcGVoztvt821r%2F4GTzfPC80y6USn%2BCmSJiFC93k5sTt9Ia11haBVp16Wf7pt6LZoMd8b3kUGC3V9j%2B%2BzXffe6L6rOHc0tu91jHdDHDGvNJKI1Sn1M%2Ba4y66AW93HatuaofesVhgQu90e6y%2FiWeRXpcUwCgcw6cEXhaPxXFKecEb1YTgyQnfJ75m0BbtLOFMgEHnC%2FjsHPFdzRMxCd%2FbXxvKxkBUSpcaCfEaiU4h5nRKvzO8j%2FVkanZFepOltFK6oFBUm42eFZvyzEcNzHOsSqPAjSGa4IKi1XxmnTbQ1ipcxis%2F6m23jtKR0fWsbBLyw8vUped%2BZztKGVM8zoymMdLJIfKj%2B%2F2yqBDCfLIlmUOJA0JqkwbqUFQmISRhYkFAAwDvVacc00zi5UgG0AhvjM%3D
    ingress.kubernetes.io/auth-response-headers: X-Userinfo, X-Id-Token, X-Access-Token, Authorization

config_token

let config_token = {
  /**
   * future feature: allow blocking certain token IDs
   */
  //jti: <some known value>

  /**
   * using the same aud for multiple tokens allows sso for all services sharing the aud
   */
  //aud: "some application id", //should be unique to prevent cookie/session hijacking, defaults to a hash unique to the whole config
  eas: {
    plugins: [
      {
        type: "htpasswd",
        htpasswd: "foo:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\nbar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\n" 
      }
      ,
      {
        type: "ldap",
        session_cache_ttl: 0, // seconds to cache successful logins
        connection: {
          url: "ldap://ldap.ldap.svc.k8s:389",
          bindDN : "cn=foo,dc=example,dc=com",
          bindCredentials : 'password',
          searchBase : 'ou=people,dc=example,dc=com',
          searchFilter : '(uid={{username}})',
          connectTimeout: 1000
        }
      }
      ,
      {
        type: "htpasswd",
        htpasswd: "bar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\nbaz:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\n" 
      }
    ]
  }
};

eas logs:

"service":"external-auth-server","level":"debug","message":"cache opts: {\"ttl\":0}"}
{"service":"external-auth-server","level":"info","message":"revoked JTIs: []"}
{"service":"external-auth-server","level":"info","message":"starting server on port 8080"}

// First visit, no Authorization request header
{"message":"starting verify pipeline","level":"info","service":"external-auth-server"}
{"service":"external-auth-server","level":"debug","message":"config token: {\"eas\":{\"plugins\":[{\"type\":\"htpasswd\",\"htpasswd\":\"foo:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\"},{\"type\":\"ldap\",\"session_cache_ttl\":0,\"connection\":{\"url\":\"ldap://ldap.ldap.svc.k8s:389\",\"bindDN\":\"cn=foo,dc=example,dc=com\",\"bindCredentials\":\"password\",\"searchBase\":\"ou=people,dc=example,dc=com\",\"searchFilter\":\"(uid={{username}})\",\"connectTimeout\":1000}},{\"type\":\"htpasswd\",\"htpasswd\":\"bar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbaz:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\"}]},\"iat\":1568057349,\"audMD5\":\"d5eafdfc89fc27e9c180e4b31d609327\"}"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: htpasswd"}
{"service":"external-auth-server","level":"debug","message":"plugin response {\"statusCode\":401,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[],\"clearCookies\":[],\"headers\":{\"WWW-Authenticate\":\"Basic realm=\\\"external authentication server\\\"\"},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"htpasswd\",\"htpasswd\":\"foo:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\",\"pcb\":{}}}}"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: ldap"}
{"service":"external-auth-server","level":"verbose","message":"parent request info: {\"uri\":\"https://foo.example.com/dashboard/\",\"parsedUri\":{\"scheme\":\"https\",\"host\":\"foo.example.com\",\"path\":\"/dashboard/\",\"reference\":\"absolute\"},\"parsedQuery\":{},\"method\":\"GET\"}"}
{"service":"external-auth-server","level":"debug","message":"plugin response {\"statusCode\":401,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[],\"clearCookies\":[],\"headers\":{\"WWW-Authenticate\":\"Basic realm=\\\"external authentication server\\\"\"},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"ldap\",\"session_cache_ttl\":0,\"connection\":{\"url\":\"ldap://ldap.ldap.svc.k8s:389\",\"bindDN\":\"cn=foo,dc=example,dc=com\",\"bindCredentials\":\"password\",\"searchBase\":\"ou=people,dc=example,dc=com\",\"searchFilter\":\"(uid={{username}})\",\"connectTimeout\":1000,\"cache\":true,\"reconnect\":true,\"timeout\":3000,\"idleTimeout\":10000},\"pcb\":{},\"assertions\":{}}}}"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: htpasswd"}
{"service":"external-auth-server","level":"debug","message":"plugin response {\"statusCode\":401,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[],\"clearCookies\":[],\"headers\":{\"WWW-Authenticate\":\"Basic realm=\\\"external authentication server\\\"\"},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"htpasswd\",\"htpasswd\":\"bar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbaz:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\",\"pcb\":{}}}}"}
{"service":"external-auth-server","level":"info","message":"end verify pipeline with status: 401"}

// Now with the Authorization request header
{"message":"starting verify pipeline","level":"info","service":"external-auth-server"}
{"service":"external-auth-server","level":"debug","message":"config token: {\"eas\":{\"plugins\":[{\"type\":\"htpasswd\",\"htpasswd\":\"foo:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\"},{\"type\":\"ldap\",\"session_cache_ttl\":0,\"connection\":{\"url\":\"ldap://ldap.ldap.svc.k8s:389\",\"bindDN\":\"cn=foo,dc=example,dc=com\",\"bindCredentials\":\"password\",\"searchBase\":\"ou=people,dc=example,dc=com\",\"searchFilter\":\"(uid={{username}})\",\"connectTimeout\":1000}},{\"type\":\"htpasswd\",\"htpasswd\":\"bar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbaz:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\"}]},\"iat\":1568057349,\"audMD5\":\"d5eafdfc89fc27e9c180e4b31d609327\"}"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: htpasswd"}
{"service":"external-auth-server","level":"debug","message":"plugin response {\"statusCode\":401,\"statusMessage\":\"\",\"body\":\"\",\"cookies\":[],\"clearCookies\":[],\"headers\":{\"WWW-Authenticate\":\"Basic realm=\\\"external authentication server\\\"\"},\"authenticationData\":{},\"plugin\":{\"server\":{},\"config\":{\"type\":\"htpasswd\",\"htpasswd\":\"foo:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\nbar:$apr1$qHDFfhPC$nITSVHgYbDAK1Y0acGRnY0\\n\",\"pcb\":{}}}}"}
{"service":"external-auth-server","level":"info","message":"starting verify for plugin: ldap"}
{"service":"external-auth-server","level":"verbose","message":"parent request info: {\"uri\":\"https://foo.example.com/dashboard/\",\"parsedUri\":{\"scheme\":\"https\",\"host\":\"foo.example.com\",\"path\":\"/dashboard/\",\"reference\":\"absolute\"},\"parsedQuery\":{},\"method\":\"GET\"}"}
{"service":"external-auth-server","level":"verbose","message":"LdapAuth connection closed: undefined"

[travisghansen]

Ok let me get a sec env setup for it and give it a whirl again.

Does it take 1 second or 240 to hit that last log line?

[leojonathanoh]

@travisghansen definitely far longer than 1 second. I'm letting it run now, to verify it really takes 240s.

[leojonathanoh]

@travisghansen oops i think i misunderstod. To hit the last log line, it's immediate.

[travisghansen]

But then it still sits for the traefik timeout?

[leojonathanoh]

it appears so. I just got a 500 response after 240033 ms. I assume this is traefik's timeout

[travisghansen]

Sounds right yeah. I'll have a look at how best to handle that situation. Probably get back to you sometime in the next 24h or so..

[leojonathanoh]

sure thing

[travisghansen]

I've got some code setup to work around this. I'm still unclear if it's related to: ldapjs/node-ldapjs#551 or just generally missing code/handling in the ldapauth library.

I've got a few other little fixes and tweaks I'm queuing up for another patch release so I'll include my changes for this as well. I'll close after it's been released.

[travisghansen]

Released to latest and v0.5.5.

[leojonathanoh]

just to confirm that v0.5.5 fixed the issue.

[travisghansen]

Thanks for the report!