Error 500 when trying to get a secret

[1password]

Error 500 when trying to get a secret

[ag-adampike]

Hey @ankycooper! Thanks for filing this issue.

We encountered something similar with another customer recently, and in that case it turned out to be an Advanced Protection firewall rule for their 1Password account which was blocking Cloud Providers.

Is your Kubernetes cluster hosted on a cloud provider (e.g. AWS, Azure, Google Cloud, etc.)? If so, can you check your firewall settings to see if that is the case? It could also be another firewall rule depending on what is set there.

To confirm this is the error you're seeing, please check the logs from the connect-sync container of the Pod for 1Password Connect. You should see something like:

…
(Forbidden (Firewall Rule)), Your request was blocked by an Advanced Protection firewall rule.
…

[ankycooper]

It's a self hosted dev k3s cluster. Teraform and ansible integrations work. Op cli works as well. I assume my network firewall isn't blocking anything. I don't have a business account but a family so there is no 1password firewall.

…

On Tue, 22 Nov 2022, 11:40 pm Adam Pike, ***@***.***> wrote: Hey @ankycooper <https://github.com/ankycooper>! Thanks for filing this issue. We encountered something similar with another customer recently, and in that case it turned out to be an Advanced Protection firewall rule <https://support.1password.com/firewall-rules/> for their 1Password account which was blocking Cloud Providers <https://support.1password.com/firewall-rules/#about-ip-addresses-and-anonymous-ips> . Is your Kubernetes cluster hosted on a cloud provider (e.g. AWS, Azure, Google Cloud, etc.)? If so, can you check your firewall settings to see if that is the case? — Reply to this email directly, view it on GitHub <#124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE5KNHG43Q3WXASG3ZLXFY3WJS5LRANCNFSM6AAAAAASHMOOIY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ankycooper]

works with minikube, but fails with k3s with the error above.

[ag-adampike]

Thanks for the updates. It initially smelled a lot like the issue we recently encountered with another customer, which is why I jumped on it.

Thanks for the extra detail; the more information the better! It's particularly interesting that it's inconsistent behaviour across platforms, and this seems like an operator specific issue rather than 1Password Connect (i.e. the plugin, not the server itself).

All that said, I'll defer to our engineering team for a deeper investigation.

[ankycooper]

Thanks please let me know if you happen to get a fix.

…

On Thu, 24 Nov 2022, 8:54 am Adam Pike, ***@***.***> wrote: Thanks for the updates. It initially smelled a lot like the issue we recently encountered with another customer, which is why I jumped on it. Thanks for the extra detail; the more information the better! It's particularly interesting that it's inconsistent behaviour across platforms, and this seems like an operator specific issue rather than 1Password Connect (i.e. the plugin, not the server itself). All that said, I'll defer to our engineering team for a deeper investigation. — Reply to this email directly, view it on GitHub <#124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE5KNHCZ4GR4SEDPBROZH7DWJ2HCVANCNFSM6AAAAAASHMOOIY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mdnfiras]

i think there is already a fix that got merged: #108

also we got this problem and updating to chart version 1.9.0 fixed it.

[ankycooper]

I'm already on v1.9.0 tried v1.8.1 got a different error message (still code is 500)
{"level":"info","ts":1669318119.2693477,"logger":"controller_onepassworditem","msg":"Reconciling OnePasswordItem","Request.Namespace":"default","Request.Name":"upsteam-apikey"} {"level":"error","ts":1669318127.3086646,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"onepassworditem-controller","request":"default/upsteam-apikey","error":"Failed to retrieve item: status 500: failed to initiate, review service logs for details","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/workspace/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"} {"level":"info","ts":1669318128.3101885,"logger":"controller_onepassworditem","msg":"Reconciling OnePasswordItem","Request.Namespace":"default","Request.Name":"upsteam-apikey"}

[jpcoenen]

Apologies for the late reply here. @ankycooper, are you still running into issues? If so, could you please check the logs of the API container of the Connect Pod? It's a bit hidden, but the following line suggests Connect is not able to initialize correctly:

Failed to retrieve item: status 500: failed to initiate, review service logs for details