Error using LDAP

[saltstack]

Error using LDAP

[whiteinge]

That last curl command may be failing due to the empty -d arg param. You can just omit it if the function doesn't take arguments.

Use the -H flag in Pepper to see the full HTTP exchange which will help narrow down the problem.

The SALT_* env variables in your paste are not respected by Pepper. Is that a typo? They should all be SALTAPI_* instead. And there is no SALT_ENV or SALTAPI_ENV variable at all.

[PierreR]

Thanks for your quick reply.

Yep there are typos (the file is actually generated). I have just updated the original description.

I have also tried to remove -d arg or use -d arg='', but without success. It is just the same (strange) error. I am a bit puzzle by the fact I am not able to reuse the auth token to make a proper salt-api curl request.

[whiteinge]

There's a bunch of things going on here so let's back up and go through them one-by-one.

Problem:

Pepper had been working with the pam eauth backend but is not working now that you've switched it to ldap.

Questions:

1. Since you edited the Salt Master config both the master and salt-api need to be restarted. Both daemons share in-memory objects (the master's config) and sometimes they both need to be fully stopped (not restarted) before they are started again. After stopping both daemons do a quick pgrep -l salt to double-check they are fully stopped. Then start the salt master then salt-api.
2. If the problem persists after that, please run Pepper with the -H flag to make sure that Pepper is sending the expected credentials. You'll need to dig them out of the raw HTTP debug output. If you want to sanitize the data and paste it here I can also take a look.

[PierreR]

I have done point 1 but without success.

I should note that when I login (salt-api) there is already an error even if I do get a token that seems to work fine:

curl -sS -i -k https://localhost:8000/login -H 'Accept: application/x-yaml' -d username='name' -d password='pass' -d eauth='ldap'
HTTP/1.1 500 Internal Server Error
Content-Length: 583
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/3.2.2
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Tue, 17 Nov 2015 19:57:58 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: f34e1c1f6916aed84a5097ffcfca22d60ba96f68
Content-Type: application/json
Set-Cookie: session_id=f34e1c1f6916aed84a5097ffcfca22d60ba96f68; expires=Wed, 18 Nov 2015 05:57:58 GMT; Path=/

{"status": 500, "return": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py\", line 435, in hypermedia_handler\n    ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py\", line 34, in __call__\n    return self.callable(*self.args, **self.kwargs)\n  File \"/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py\", line 1440, in POST\n    user_groups = set(token['groups'])\nTypeError: 'NoneType' object is not iterable\n"}

For 2, I have to admit I am actually using pepper with this PR as I need https without cert verification. Unfortunately and I guess this is as a consequence H does not make any change ... So I guess I would need to revert to http if I want to test it with -H.

[PierreR]

PS all these salt-api errors look strange to me. Do I do something wrong ? I know the token is OK because whenever I modify it, I will have an unauthorized error as expected instead of the unpack(b) received extra data.

[PierreR]

This is the pastie of the full error for

curl -ks https://localhost:8000 -H "X-Auth-Token: f34e1c1f6916aed84a5097ffcfca22d60ba96f68" -d client='local' -d tgt='*' -d fun='test.ping' 

If I can make it works on the server with just salt-api (using curl) I would be sure the problem is on the pepper side. But right now I have the feeling it is on salt-api side ...

[whiteinge]

Ah. I can't help with Pepper since you're using a tweaked version. Pull req #62 added support for not verifying SSL certs. Hopefully that will work for you too. I'll cut a new release with it soon.

So moving on to troubleshooting salt-api:

1. Standard troubleshooting question: What version of Salt are you running?
2. The two tracebacks you're seeing, TypeError: 'NoneType' object is not iterable and SaltClientError: unpack(b) received extra data. are coming from Salt. I know for sure the first one has been fixed. So let's start with your Salt version and take it from there.

[PierreR]

As wrote in my first post ;-), I am using 2015.5.5-1.el7 from EPEL CentOS 7.1.

Would you advice to try a newest version using the saltstack-repo as described here: https://docs.saltstack.com/en/latest/topics/installation/rhel.html.

[whiteinge]

Apologies! I missed that.

The first traceback was fixed in saltstack/salt#26975. I'm not sure about the second. It doesn't look familiar. Update to the current point-release (.6) and try again. Please open a new issue in the Salt repository if the other one persists.

[PierreR]

I have been able to trace the problem. There is an error returned by the LDAP backend when searching for groups.

The problem has been solved after fixing the salt master configuration.

That said the handling of such exception by the salt master is ... well catastrophic. It leads to inconsistencies: the command line checks for None group while the salt-api does not. My colleague will propose a fix to the Salt repository for this.

Thanks for your support. It has been really helpful.

[whiteinge]

Oh, good! I'm very glad you were able to dig that out of a silent catch statement. Yikes. :-P I'm under the vague impression the auth code in Salt does that to avoid leaking auth configuration into log files. Keep that in mind while working on a fix. I'll keep an eye out for what you and your colleague come up with.