permitio / opal-fetcher-postgres Goto Github PK
View Code? Open in Web Editor NEWAn OPAL custom data fetcher to bring authorization state from Postgres
Home Page: https://opal.ac
License: Apache License 2.0
An OPAL custom data fetcher to bring authorization state from Postgres
Home Page: https://opal.ac
License: Apache License 2.0
when I update the databases data it is not correct the values of OPA to the latest values
Hi OPAL team, hmm, I started to get this error today for this repo. Any idea? Thanks!
➜ docker-compose up
[+] Building 13.2s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 31B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/authorizon/opal-client:latest 3.2s
=> [1/5] FROM docker.io/authorizon/opal-client:latest@sha256:1da4cc56616d777cef36d26de4d82916fc392afeba2aae5519afea19c7dccbcf 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 1.30MB 0.1s
=> CACHED [2/5] WORKDIR /app/ 0.0s
=> [3/5] COPY . ./ 0.1s
=> [4/5] RUN python setup.py install 1.5s
=> [5/5] RUN pip install wait-for-it 8.1s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:4b5659aa6f2b54c312ba3b93c55bc0d3c509181ef872aff0847ab07c2d7ba52b 0.0s
=> => naming to docker.io/library/opal-fetcher-postgres_opal_client 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 4/4
⠿ Container opal-fetcher-postgres_broadcast_channel_1 Created 0.0s
⠿ Container opal-fetcher-postgres_example_db_1 Created 0.0s
⠿ Container opal-fetcher-postgres_opal_server_1 Created 0.0s
⠿ Container opal-fetcher-postgres_opal_client_1 Recreated 0.1s
Attaching to broadcast_channel_1, example_db_1, opal_client_1, opal_server_1
broadcast_channel_1 |
broadcast_channel_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
broadcast_channel_1 |
broadcast_channel_1 | 2021-09-09 17:30:19.918 UTC [1] LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
broadcast_channel_1 | 2021-09-09 17:30:19.918 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
broadcast_channel_1 | 2021-09-09 17:30:19.918 UTC [1] LOG: listening on IPv6 address "::", port 5432
broadcast_channel_1 | 2021-09-09 17:30:19.921 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
broadcast_channel_1 | 2021-09-09 17:30:19.924 UTC [21] LOG: database system was shut down at 2021-09-09 17:29:57 UTC
broadcast_channel_1 | 2021-09-09 17:30:19.937 UTC [1] LOG: database system is ready to accept connections
example_db_1 |
example_db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
example_db_1 |
example_db_1 | 2021-09-09 17:30:20.076 UTC [1] LOG: starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
example_db_1 | 2021-09-09 17:30:20.076 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
example_db_1 | 2021-09-09 17:30:20.076 UTC [1] LOG: listening on IPv6 address "::", port 5432
example_db_1 | 2021-09-09 17:30:20.079 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
example_db_1 | 2021-09-09 17:30:20.084 UTC [27] LOG: database system was shut down at 2021-09-09 17:29:57 UTC
example_db_1 | 2021-09-09 17:30:20.090 UTC [1] LOG: database system is ready to accept connections
opal_server_1 | [2021-09-09 17:30:21 +0000] [1] [INFO] Starting gunicorn 20.1.0
opal_server_1 | [2021-09-09 17:30:21 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
opal_server_1 | [2021-09-09 17:30:21 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_server_1 | [2021-09-09 17:30:21 +0000] [9] [INFO] Booting worker with pid: 9
opal_server_1 | [2021-09-09 17:30:21 +0000] [10] [INFO] Booting worker with pid: 10
opal_server_1 | [2021-09-09 17:30:21 +0000] [11] [INFO] Booting worker with pid: 11
opal_server_1 | [2021-09-09 17:30:21 +0000] [12] [INFO] Booting worker with pid: 12
opal_server_1 | 2021-09-09T17:30:21.657897+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-09T17:30:21.709369+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-09T17:30:21.736570+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-09T17:30:21.736989+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_server_1 | 2021-09-09T17:30:21.737376+0000 |opal_server.server | INFO | leadership lock acquired, leader pid: 9
opal_server_1 | 2021-09-09T17:30:21.737518+0000 |opal_server.server | INFO | listening on webhook topic: 'webhook'
opal_server_1 | 2021-09-09T17:30:21.738816+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '41deef66f4714d4da2edec6ef1078b0b', 'subscriber_id': '9080fb2e6a82434095246fa9cda352b7', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fc532fb0310>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fc532fb3640>), 'notifier_id': None}
opal_server_1 | 2021-09-09T17:30:21.739108+0000 |opal_server.policy.watcher.task | INFO | Launching repo watcher
opal_server_1 | 2021-09-09T17:30:21.739778+0000 |opal_common.git.repo_cloner | INFO | Cloning repo from 'https://github.com/authorizon/opal-example-policy-repo' to '/regoclone'
opal_server_1 | 2021-09-09T17:30:21.739956+0000 |opal_common.git.repo_cloner | INFO | Repo already exists in '/regoclone'
opal_server_1 | 2021-09-09T17:30:21.800354+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-09T17:30:21.800808+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_server_1 | 2021-09-09T17:30:21.854116+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-09T17:30:21.906423+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-09T17:30:21.906625+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_server_1 | 2021-09-09T17:30:21.956982+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-09T17:30:22.013798+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-09T17:30:22.014042+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_client_1 | [2021-09-09 17:30:22 +0000] [9] [INFO] Starting gunicorn 20.1.0
opal_client_1 | [2021-09-09 17:30:22 +0000] [9] [INFO] Listening at: http://0.0.0.0:7000 (9)
opal_client_1 | [2021-09-09 17:30:22 +0000] [9] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_client_1 | [2021-09-09 17:30:22 +0000] [11] [INFO] Booting worker with pid: 11
opal_client_1 | 2021-09-09T17:30:22.776529+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
opal_client_1 | 2021-09-09T17:30:22.776886+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
opal_client_1 | 2021-09-09T17:30:22.802421+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
opal_client_1 | 2021-09-09T17:30:22.802631+0000 |opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
opal_client_1 | 2021-09-09T17:30:22.825239+0000 |opal_client.opa.runner | INFO | Launching opa runner
opal_client_1 | 2021-09-09T17:30:22.826799+0000 |opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info
opal_client_1 | 2021-09-09T17:30:22.842910+0000 |opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-09-09T17:30:22Z'}
opal_server_1 | 2021-09-09T17:30:23.451689+0000 |opal_common.git.repo_watcher | INFO | Launching polling task, interval: 30 seconds
opal_client_1 | 2021-09-09T17:30:23.834381+0000 |opal_client.opa.runner | INFO | Running OPA initial start callbacks
opal_client_1 | 2021-09-09T17:30:23.834992+0000 |opal_client.data.updater | INFO | Launching data updater
opal_client_1 | 2021-09-09T17:30:23.835351+0000 |opal_client.policy.updater | INFO | Launching policy updater
opal_client_1 | 2021-09-09T17:30:23.835596+0000 |opal_client.data.updater | INFO | Subscribing to topics: ['policy_data']
opal_client_1 | 2021-09-09T17:30:23.836373+0000 |opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
opal_client_1 | 2021-09-09T17:30:23.837027+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
opal_client_1 | 2021-09-09T17:30:23.837917+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
opal_client_1 | 2021-09-09T17:30:23.843181+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
opal_client_1 | 2021-09-09T17:30:23.843570+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
opal_server_1 | 2021-09-09T17:30:23.849333+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Listening for incoming events from broadcast channel (first listener started)
opal_server_1 | 2021-09-09T17:30:23.849735+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Spawning broadcast listen task
opal_server_1 | 2021-09-09T17:30:23.874777+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
opal_server_1 | 2021-09-09T17:30:23.875580+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '1d521aff17ce49cba19c32833ff51f7c', 'subscriber_id': '95cfad20c966449c93fd01fe5eae0fae', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fc532fb7580>>, 'notifier_id': None}
opal_server_1 | 2021-09-09T17:30:23.876083+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected
opal_server_1 | 2021-09-09T17:30:23.876830+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Starting broadcaster listener
opal_server_1 | 2021-09-09T17:30:23.894454+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected
opal_server_1 | 2021-09-09T17:30:23.904230+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'c7e4128addde41f9b41e78d6e21bb9c7', 'subscriber_id': '96aa370defc74f14ab2ace2ee874faa3', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fc532a8fe50>, 'notifier_id': None}
opal_client_1 | 2021-09-09T17:30:23.908319+0000 |opal_client.data.updater | INFO | Connected to server
opal_client_1 | 2021-09-09T17:30:23.908431+0000 |opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
opal_server_1 | 2021-09-09T17:30:23.908219+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '17bb79917a5e4d478fbd707a1fd535e5', 'subscriber_id': '3b5a87e12519497ebce614d10932d49b', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fc532a8fdc0>, 'notifier_id': None}
opal_client_1 | 2021-09-09T17:30:23.908553+0000 |opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal_server:7002/data/config'
opal_client_1 | 2021-09-09T17:30:23.911477+0000 |opal_client.policy.updater | INFO | Connected to server
opal_client_1 | 2021-09-09T17:30:23.911853+0000 |opal_client.policy.updater | INFO | Refetching policy code (full bundle)
opal_server_1 | 2021-09-09T17:30:23.914931+0000 |opal_server.data.api | INFO | Serving source configuration
opal_server_1 | 2021-09-09T17:30:23.915966+0000 |uvicorn.protocols.http.httptools_impl | INFO | 172.25.0.5:60762 - "GET /data/config HTTP/1.1" 200
opal_client_1 | 2021-09-09T17:30:23.917804+0000 |opal_client.data.updater | INFO | Triggering data update with id: ca7697c68c3f4c489bd4bc663a0eb82d
opal_client_1 | 2021-09-09T17:30:23.918044+0000 |opal_client.data.updater | INFO | Fetching policy data
opal_client_1 | 2021-09-09T17:30:23.918305+0000 |opal_client.data.fetcher | INFO | Fetching data from url: postgresql://postgres@example_db:5432/postgres
opal_server_1 | 2021-09-09T17:30:23.936952+0000 |uvicorn.protocols.http.httptools_impl | INFO | 172.25.0.5:60764 - "GET /policy?path=. HTTP/1.1" 200
opal_client_1 | 2021-09-09T17:30:23.936638+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws
opal_client_1 | 2021-09-09T17:30:23.939081+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: '6d849b1ce92bae05c31146470e66e07c3c41b164'
opal_client_1 | 2021-09-09T17:30:23.942061+0000 |opal_client.opa.logger | INFO | Received request. GET /v1/policies
opal_client_1 | 2021-09-09T17:30:23.942817+0000 |opal_client.opa.logger | INFO | Sent response. GET /v1/policies -> 200
opal_client_1 | 2021-09-09T17:30:23.944976+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/data
opal_client_1 | 2021-09-09T17:30:23.945914+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/data -> 204
example_db_1 | 2021-09-09 17:30:23.948 UTC [35] ERROR: relation "city" does not exist at character 15
example_db_1 | 2021-09-09 17:30:23.948 UTC [35] STATEMENT: SELECT * from city;
opal_client_1 | 2021-09-09T17:30:23.948788+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/policies/rbac.rego
opal_client_1 | 2021-09-09T17:30:23.956322+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/rbac.rego -> 200
opal_client_1 | 2021-09-09T17:30:23.958569+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/policies/utils.rego
opal_client_1 | 2021-09-09T17:30:23.961442+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/utils.rego -> 200
opal_client_1 | 2021-09-09T17:30:23.962971+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws
example_db_1 | 2021-09-09 17:30:24.421 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:30:24.421 UTC [35] STATEMENT: SELECT * from city;
example_db_1 | 2021-09-09 17:30:25.058 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:30:25.058 UTC [35] STATEMENT: SELECT * from city;
example_db_1 | 2021-09-09 17:30:27.559 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:30:27.559 UTC [35] STATEMENT: SELECT * from city;
opal_client_1 | 2021-09-09T17:30:33.938164+0000 |opal_client.data.fetcher |ERROR | Timeout while fetching url: postgresql://postgres@example_db:5432/postgres
opal_client_1 | Traceback (most recent call last):
opal_client_1 |
opal_client_1 | File "/root/.local/bin/gunicorn", line 8, in <module>
opal_client_1 | sys.exit(run())
opal_client_1 | │ │ └ <function run at 0x7f4ef9412940>
opal_client_1 | │ └ <built-in function exit>
opal_client_1 | └ <module 'sys' (built-in)>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
opal_client_1 | WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
opal_client_1 | └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
opal_client_1 | super().run()
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
opal_client_1 | Arbiter(self).run()
opal_client_1 | │ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f4ef93ed7f0>
opal_client_1 | └ <class 'gunicorn.arbiter.Arbiter'>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
opal_client_1 | self.manage_workers()
opal_client_1 | │ └ <function Arbiter.manage_workers at 0x7f4ef8d22160>
opal_client_1 | └ <gunicorn.arbiter.Arbiter object at 0x7f4ef93ed910>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
opal_client_1 | self.spawn_workers()
opal_client_1 | │ └ <function Arbiter.spawn_workers at 0x7f4ef8d22280>
opal_client_1 | └ <gunicorn.arbiter.Arbiter object at 0x7f4ef93ed910>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
opal_client_1 | self.spawn_worker()
opal_client_1 | │ └ <function Arbiter.spawn_worker at 0x7f4ef8d221f0>
opal_client_1 | └ <gunicorn.arbiter.Arbiter object at 0x7f4ef93ed910>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
opal_client_1 | worker.init_process()
opal_client_1 | │ └ <function UvicornWorker.init_process at 0x7f4ef83fdca0>
opal_client_1 | └ <uvicorn.workers.UvicornWorker object at 0x7f4ef83ff130>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
opal_client_1 | super(UvicornWorker, self).init_process()
opal_client_1 | │ └ <uvicorn.workers.UvicornWorker object at 0x7f4ef83ff130>
opal_client_1 | └ <class 'uvicorn.workers.UvicornWorker'>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
opal_client_1 | self.run()
opal_client_1 | │ └ <function UvicornWorker.run at 0x7f4ef83fddc0>
opal_client_1 | └ <uvicorn.workers.UvicornWorker object at 0x7f4ef83ff130>
opal_client_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
opal_client_1 | loop.run_until_complete(server.serve(sockets=self.sockets))
opal_client_1 | │ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7f4ef83eef40>]
opal_client_1 | │ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f4ef83ff130>
opal_client_1 | │ │ │ └ <function Server.serve at 0x7f4ef8631d30>
opal_client_1 | │ │ └ <uvicorn.server.Server object at 0x7f4ef88f94c0>
opal_client_1 | │ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
opal_client_1 | └ <uvloop.Loop running=True closed=False debug=False>
opal_client_1 | > File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.13-py3.8.egg/opal_client/data/fetcher.py", line 63, in handle_url
opal_client_1 | response = await self._engine.handle_url(url, config=config)
opal_client_1 | │ │ │ │ └ {'fetcher': 'PostgresFetchProvider', 'query': 'SELECT * from city;', 'connection_params': {'password': 'postgres'}}
opal_client_1 | │ │ │ └ 'postgresql://postgres@example_db:5432/postgres'
opal_client_1 | │ │ └ <function FetchingEngine.handle_url at 0x7f4ef736f310>
opal_client_1 | │ └ <opal_common.fetcher.engine.fetching_engine.FetchingEngine object at 0x7f4ef88ed8e0>
opal_client_1 | └ <opal_client.data.fetcher.DataFetcher object at 0x7f4ef72fce20>
opal_client_1 | File "/usr/local/lib/python3.8/site-packages/opal_common-0.1.13-py3.8.egg/opal_common/fetcher/engine/fetching_engine.py", line 104, in handle_url
opal_client_1 | await asyncio.wait_for(wait_event.wait(), timeout)
opal_client_1 | │ │ │ │ └ 10
opal_client_1 | │ │ │ └ <function Event.wait at 0x7f4ef8760040>
opal_client_1 | │ │ └ <asyncio.locks.Event object at 0x7f4ef6f09460 [unset]>
opal_client_1 | │ └ <function wait_for at 0x7f4ef8769940>
opal_client_1 | └ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
opal_client_1 | File "/usr/local/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
opal_client_1 | raise exceptions.TimeoutError()
opal_client_1 | │ └ <class 'asyncio.exceptions.TimeoutError'>
opal_client_1 | └ <module 'asyncio.exceptions' from '/usr/local/lib/python3.8/asyncio/exceptions.py'>
opal_client_1 |
opal_client_1 | asyncio.exceptions.TimeoutError
opal_client_1 | 2021-09-09T17:30:33.949107+0000 |opal_client.data.updater |ERROR | Failed to fetch url postgresql://postgres@example_db:5432/postgres, got exception:
example_db_1 | 2021-09-09 17:30:35.103 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:30:35.103 UTC [35] STATEMENT: SELECT * from city;
example_db_1 | 2021-09-09 17:30:37.384 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:30:37.384 UTC [35] STATEMENT: SELECT * from city;
opal_server_1 | 2021-09-09T17:30:53.428806+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
opal_server_1 | 2021-09-09T17:30:55.551686+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at '6d849b1ce92bae05c31146470e66e07c3c41b164'
example_db_1 | 2021-09-09 17:31:08.056 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:31:08.056 UTC [35] STATEMENT: SELECT * from city;
opal_server_1 | 2021-09-09T17:31:25.531347+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
opal_server_1 | 2021-09-09T17:31:27.180727+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at '6d849b1ce92bae05c31146470e66e07c3c41b164'
opal_client_1 | 2021-09-09T17:31:32.875584+0000 |opal_client.opa.logger | INFO | Received request. GET /v1/data
opal_client_1 | 2021-09-09T17:31:32.876693+0000 |opal_client.opa.logger | INFO | Sent response. GET /v1/data -> 200
example_db_1 | 2021-09-09 17:31:45.503 UTC [35] ERROR: current transaction is aborted, commands ignored until end of transaction block
example_db_1 | 2021-09-09 17:31:45.503 UTC [35] STATEMENT: SELECT * from city;
opal_server_1 | 2021-09-09T17:31:57.159742+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
opal_server_1 | 2021-09-09T17:31:58.816467+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at '6d849b1ce92bae05c31146470e66e07c3c41b164'
Requests to a postgres data source can fail if a data source config entry's url uses the format postgresql://<user>:<password>@<host>/<db>
, specifically the password field may fail to be parsed by the underlying library asyncpg. There's an issue in the asyncpg repo which helped me figure out that resolving this (if not using the connection_params
field) requires changing the format to postgresql://<host>/<db>?user=<user>&password=<password>
as referenced in this comment.
No code changes necessary but I think noting that bug in the docs could potentially save other folks some debugging time
When I run docker compose up
, I get error:
[2021-08-23 17:43:31 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-08-23 17:43:31 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-08-23 17:43:31 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-08-23 17:43:31 +0000] [9] [INFO] Booting worker with pid: 9
2021-08-23T17:43:31.756294+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-08-23T17:43:31.756561+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-08-23T17:43:31.776586+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-08-23T17:43:31.776740+0000 |opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
2021-08-23T17:43:31.797014+0000 |opal_client.opa.runner | INFO | Launching opa runner
2021-08-23T17:43:31.798399+0000 |opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info
2021-08-23T17:43:31.813046+0000 |opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-08-23T17:43:31Z'}
2021-08-23T17:43:32.804447+0000 |opal_client.opa.runner | INFO | Running OPA initial start callbacks
2021-08-23T17:43:32.805396+0000 |opal_client.policy.updater | INFO | Launching policy updater
2021-08-23T17:43:32.805677+0000 |opal_client.data.updater | INFO | Launching data updater
2021-08-23T17:43:32.805880+0000 |opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
2021-08-23T17:43:32.806317+0000 |opal_client.data.updater | INFO | Subscribing to topics: ['policy_data']
2021-08-23T17:43:32.806967+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
2021-08-23T17:43:32.807783+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
2021-08-23T17:43:32.811801+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
2021-08-23T17:43:32.812388+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
2021-08-23T17:43:32.871213+0000 |opal_client.policy.updater | INFO | Connected to server
2021-08-23T17:43:32.871684+0000 |opal_client.policy.updater | INFO | Refetching policy code (full bundle)
2021-08-23T17:43:32.873803+0000 |opal_client.data.updater | INFO | Connected to server
2021-08-23T17:43:32.873948+0000 |opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-08-23T17:43:32.874025+0000 |opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal_server:7002/data/config'
2021-08-23T17:43:32.879804+0000 |opal_client.data.updater | INFO | Triggering data update with id: bda17568be5c4c3180f8d62f387e6f0e
2021-08-23T17:43:32.879997+0000 |opal_client.data.updater | INFO | Fetching policy data
2021-08-23T17:43:32.880289+0000 |opal_client.data.fetcher | INFO | Fetching data from url: postgresql://postgres@example_db:5432/postgres
2021-08-23T17:43:32.896534+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws
2021-08-23T17:43:32.898193+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: 'c60470ef8046152a8498d81df0eff66ddd1632e2'
2021-08-23T17:43:32.900306+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41390', 'req_id': 1, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:32.901079+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41390', 'req_id': 1, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 13, 'resp_duration': 0.568914, 'resp_status': 200, 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:32.903906+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41392', 'req_id': 2, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:32.906962+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41392', 'req_id': 2, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 315, 'resp_duration': 3.040543, 'resp_status': 400, 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:32.913913+0000 |opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='postgresql://postgres@example_db:5432/postgres', destination path='/cities'
2021-08-23T17:43:32.915890+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41394', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/data/cities', 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:32.916704+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41394', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/data/cities', 'resp_bytes': 0, 'resp_duration': 0.889005, 'resp_status': 204, 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:32.992279+0000 |opal_client.opa.logger | INFO | OPA is out of date. {'current_version': '0.30.2', 'download_opa': 'https://openpolicyagent.org/downloads/v0.31.0/opa_linux_amd64', 'latest_version': '0.31.0', 'release_notes': 'https://github.com/open-policy-agent/opa/releases/tag/v0.31.0', 'time': '2021-08-23T17:43:32Z'}
2021-08-23T17:43:34.911369+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41396', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-08-23T17:43:34Z'}
2021-08-23T17:43:34.912880+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41396', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 315, 'resp_duration': 1.120678, 'resp_status': 400, 'time': '2021-08-23T17:43:34Z'}
2021-08-23T17:43:34.913772+0000 |opal_client...base_policy_store_client |WARNING | OPA transaction failed, transaction id=c60470ef8046152a8498d81df0eff66ddd1632e2, actions=['set_policies'], error=RetryError(<Future at 0x7f7fa0672f70 state=finished raised ValueError>)
2021-08-23T17:43:34.914065+0000 |fastapi_websocket_rpc.websocket_rpc_c...|ERROR | RPC Error
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 45, in __call__
result = await fn(*args, **kwargs)
│ │ └ {'policy_id': 'rbac.rego', 'policy_code': '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This ...
│ └ (<opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>,)
└ <function OpaClient.set_policy at 0x7f7fa0b1d940>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 173, in set_policy
return await proxy_response_unless_invalid(opa_response, accepted_status_codes=[status.HTTP_200_OK])
│ │ │ └ 200
│ │ └ <module 'starlette.status' from '/root/.local/lib/python3.8/site-packages/starlette/status.py'>
│ └ <ClientResponse(http://localhost:8181/v1/policies/rbac.rego) [400 Bad Request]>
│ <CIMultiDictProxy('Content-Type': 'applicatio...
└ <function proxy_response_unless_invalid at 0x7f7fa0b0cca0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 54, in proxy_response_unless_invalid
raise ValueError("OPA Client: unexpected status code: {}, error: {}".format(response.status_code, error))
│ │ └ {'code': 'invalid_parameter', 'message': 'error(s) occurred while compiling module(s)', 'errors': [{'code': 'rego_type_error'...
│ └ 400
└ <starlette.responses.Response object at 0x7f7fa0672eb0>
ValueError: OPA Client: unexpected status code: 400, error: {'code': 'invalid_parameter', 'message': 'error(s) occurred while compiling module(s)', 'errors': [{'code': 'rego_type_error', 'message': 'undefined function data.utils.hasPermission', 'location': {'file': 'rbac.rego', 'row': 37, 'col': 2}}]}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f7fa2b85940>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f7fa2b5f820>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f7fa2494160>
└ <gunicorn.arbiter.Arbiter object at 0x7f7fa2b5f940>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f7fa2494280>
└ <gunicorn.arbiter.Arbiter object at 0x7f7fa2b5f940>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f7fa24941f0>
└ <gunicorn.arbiter.Arbiter object at 0x7f7fa2b5f940>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f7fa1b44b80>
└ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f7fa1b44ca0>
└ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
loop.run_until_complete(server.serve(sockets=self.sockets))
│ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7f7fa1b30f10>]
│ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
│ │ │ └ <function Server.serve at 0x7f7fa1e1cc10>
│ │ └ <uvicorn.server.Server object at 0x7f7fa206b4f0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
> File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/websocket_rpc_client.py", line 127, in __connect__
await self.channel.on_connect()
│ │ └ <function RpcChannel.on_connect at 0x7f7fa0b46ee0>
│ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa06ab820>
└ <fastapi_websocket_rpc.websocket_rpc_client.WebSocketRpcClient object at 0x7f7fa0690d60>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 237, in on_connect
await self.on_handler_event(self._connect_handlers, self)
│ │ │ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa06ab820>
│ │ │ └ [<bound method PubSubClient._primary_on_connect of <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa069...
│ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa06ab820>
│ └ <function RpcChannel.on_handler_event at 0x7f7fa0b46e50>
└ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa06ab820>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 234, in on_handler_event
await asyncio.gather(*(callback(*args, **kwargs) for callback in handlers))
│ │ │ │ └ [<bound method PubSubClient._primary_on_connect of <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa069...
│ │ │ └ {}
│ │ └ (<fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa06ab820>,)
│ └ <function gather at 0x7f7fa1eb5ca0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_pubsub/pub_sub_client.py", line 217, in _primary_on_connect
await asyncio.gather(*(callback(self, channel) for callback in self._on_connect))
│ │ │ │ │ └ [<bound method PolicyUpdater._on_connect of <opal_client.policy.updater.PolicyUpdater object at 0x7f7fa205f910>>]
│ │ │ │ └ <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa06907c0>
│ │ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa06ab820>
│ │ └ <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa06907c0>
│ └ <function gather at 0x7f7fa1eb5ca0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy/updater.py", line 108, in _on_connect
await self.update_policy()
│ └ <function PolicyUpdater.update_policy at 0x7f7fa09f63a0>
└ <opal_client.policy.updater.PolicyUpdater object at 0x7f7fa205f910>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy/updater.py", line 214, in update_policy
await store_transaction.set_policies(bundle)
│ │ └ PolicyBundle(manifest=['data.json', 'rbac.rego', 'utils.rego'], hash='c60470ef8046152a8498d81df0eff66ddd1632e2', old_hash=Non...
│ └ <function AbstractPolicyStore.set_policies at 0x7f7fa0b02550>
└ <opal_client.policy_store.base_policy_store_client.PolicyStoreTransactionContextManager object at 0x7f7fa0672760>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 256, in set_policies
return await self._set_policies_from_complete_bundle(bundle)
│ │ └ PolicyBundle(manifest=['data.json', 'rbac.rego', 'utils.rego'], hash='c60470ef8046152a8498d81df0eff66ddd1632e2', old_hash=Non...
│ └ <function OpaClient._set_policies_from_complete_bundle at 0x7f7fa0b22430>
└ <opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 268, in _set_policies_from_complete_bundle
await self.set_policy(policy_id=module.path, policy_code=module.rego)
│ │ │ │ │ └ '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This example defines an RBAC model for a Pet St...
│ │ │ │ └ RegoModule(path='rbac.rego', package_name='app.rbac', rego='# Role-based Access Control (RBAC)\n# ---------------------------...
│ │ │ └ 'rbac.rego'
│ │ └ RegoModule(path='rbac.rego', package_name='app.rbac', rego='# Role-based Access Control (RBAC)\n# ---------------------------...
│ └ <function AsyncRetrying.wraps.<locals>.async_wrapped at 0x7f7fa0b1daf0>
└ <opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 79, in async_wrapped
return await fn(*args, **kwargs)
│ │ └ {'policy_id': 'rbac.rego', 'policy_code': '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This ...
│ └ (<opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>,)
└ <function OpaClient.set_policy at 0x7f7fa0b1d9d0>
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 42, in __call__
do = self.iter(retry_state=retry_state)
│ │ └ <tenacity.RetryCallState object at 0x7f7fa0672bb0>
│ └ <function BaseRetrying.iter at 0x7f7fa0ac5c10>
└ <AsyncRetrying object at 0x7f7fa0b0ee80 (stop=<tenacity.stop.stop_after_attempt object at 0x7f7fa0b0ed30>, wait=<tenacity.wai...
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 373, in iter
six.raise_from(retry_exc, fut.exception())
│ │ │ │ └ <function Future.exception at 0x7f7fa1fba550>
│ │ │ └ <Future at 0x7f7fa0672f70 state=finished raised ValueError>
│ │ └ RetryError(<Future at 0x7f7fa0672f70 state=finished raised ValueError>)
│ └ <function raise_from at 0x7f7fa0bd2790>
└ <module 'six' from '/root/.local/lib/python3.8/site-packages/six.py'>
File "<string>", line 3, in raise_from
tenacity.RetryError: RetryError[<Future at 0x7f7fa0672f70 state=finished raised ValueError>]
2021-08-23T17:43:35.858602+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
2021-08-23T17:43:35.869748+0000 |opal_client.policy.updater | INFO | Connected to server
2021-08-23T17:43:35.869938+0000 |opal_client.policy.updater | INFO | Refetching policy code (full bundle)
2021-08-23T17:43:35.892245+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: 'c60470ef8046152a8498d81df0eff66ddd1632e2'
2021-08-23T17:43:35.893954+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41402', 'req_id': 5, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-08-23T17:43:35Z'}
2021-08-23T17:43:35.894336+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41402', 'req_id': 5, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 13, 'resp_duration': 0.153132, 'resp_status': 200, 'time': '2021-08-23T17:43:35Z'}
2021-08-23T17:43:35.896196+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41404', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-08-23T17:43:35Z'}
2021-08-23T17:43:35.897293+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41404', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 315, 'resp_duration': 1.059368, 'resp_status': 400, 'time': '2021-08-23T17:43:35Z'}
2021-08-23T17:43:37.901788+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41406', 'req_id': 7, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-08-23T17:43:37Z'}
2021-08-23T17:43:37.902706+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41406', 'req_id': 7, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 315, 'resp_duration': 0.967893, 'resp_status': 400, 'time': '2021-08-23T17:43:37Z'}
2021-08-23T17:43:37.903923+0000 |opal_client...base_policy_store_client |WARNING | OPA transaction failed, transaction id=c60470ef8046152a8498d81df0eff66ddd1632e2, actions=['set_policies'], error=RetryError(<Future at 0x7f7f9e60d9a0 state=finished raised ValueError>)
2021-08-23T17:43:37.904287+0000 |fastapi_websocket_rpc.websocket_rpc_c...|ERROR | RPC Error
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 45, in __call__
result = await fn(*args, **kwargs)
│ │ └ {'policy_id': 'rbac.rego', 'policy_code': '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This ...
│ └ (<opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>,)
└ <function OpaClient.set_policy at 0x7f7fa0b1d940>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 173, in set_policy
return await proxy_response_unless_invalid(opa_response, accepted_status_codes=[status.HTTP_200_OK])
│ │ │ └ 200
│ │ └ <module 'starlette.status' from '/root/.local/lib/python3.8/site-packages/starlette/status.py'>
│ └ <ClientResponse(http://localhost:8181/v1/policies/rbac.rego) [400 Bad Request]>
│ <CIMultiDictProxy('Content-Type': 'applicatio...
└ <function proxy_response_unless_invalid at 0x7f7fa0b0cca0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 54, in proxy_response_unless_invalid
raise ValueError("OPA Client: unexpected status code: {}, error: {}".format(response.status_code, error))
│ │ └ {'code': 'invalid_parameter', 'message': 'error(s) occurred while compiling module(s)', 'errors': [{'code': 'rego_type_error'...
│ └ 400
└ <starlette.responses.Response object at 0x7f7f9e60dd60>
ValueError: OPA Client: unexpected status code: 400, error: {'code': 'invalid_parameter', 'message': 'error(s) occurred while compiling module(s)', 'errors': [{'code': 'rego_type_error', 'message': 'undefined function data.utils.hasPermission', 'location': {'file': 'rbac.rego', 'row': 37, 'col': 2}}]}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f7fa2b85940>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f7fa2b5f820>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f7fa2494160>
└ <gunicorn.arbiter.Arbiter object at 0x7f7fa2b5f940>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f7fa2494280>
└ <gunicorn.arbiter.Arbiter object at 0x7f7fa2b5f940>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f7fa24941f0>
└ <gunicorn.arbiter.Arbiter object at 0x7f7fa2b5f940>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f7fa1b44b80>
└ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f7fa1b44ca0>
└ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
loop.run_until_complete(server.serve(sockets=self.sockets))
│ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7f7fa1b30f10>]
│ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f7fa1b34160>
│ │ │ └ <function Server.serve at 0x7f7fa1e1cc10>
│ │ └ <uvicorn.server.Server object at 0x7f7fa206b4f0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
> File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/websocket_rpc_client.py", line 127, in __connect__
await self.channel.on_connect()
│ │ └ <function RpcChannel.on_connect at 0x7f7fa0b46ee0>
│ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa063e4f0>
└ <fastapi_websocket_rpc.websocket_rpc_client.WebSocketRpcClient object at 0x7f7fa0690d60>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 237, in on_connect
await self.on_handler_event(self._connect_handlers, self)
│ │ │ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa063e4f0>
│ │ │ └ [<bound method PubSubClient._primary_on_connect of <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa069...
│ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa063e4f0>
│ └ <function RpcChannel.on_handler_event at 0x7f7fa0b46e50>
└ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa063e4f0>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 234, in on_handler_event
await asyncio.gather(*(callback(*args, **kwargs) for callback in handlers))
│ │ │ │ └ [<bound method PubSubClient._primary_on_connect of <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa069...
│ │ │ └ {}
│ │ └ (<fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa063e4f0>,)
│ └ <function gather at 0x7f7fa1eb5ca0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_pubsub/pub_sub_client.py", line 217, in _primary_on_connect
await asyncio.gather(*(callback(self, channel) for callback in self._on_connect))
│ │ │ │ │ └ [<bound method PolicyUpdater._on_connect of <opal_client.policy.updater.PolicyUpdater object at 0x7f7fa205f910>>]
│ │ │ │ └ <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa06907c0>
│ │ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f7fa063e4f0>
│ │ └ <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f7fa06907c0>
│ └ <function gather at 0x7f7fa1eb5ca0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy/updater.py", line 108, in _on_connect
await self.update_policy()
│ └ <function PolicyUpdater.update_policy at 0x7f7fa09f63a0>
└ <opal_client.policy.updater.PolicyUpdater object at 0x7f7fa205f910>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy/updater.py", line 214, in update_policy
await store_transaction.set_policies(bundle)
│ │ └ PolicyBundle(manifest=['data.json', 'rbac.rego', 'utils.rego'], hash='c60470ef8046152a8498d81df0eff66ddd1632e2', old_hash=Non...
│ └ <function AbstractPolicyStore.set_policies at 0x7f7fa0b02550>
└ <opal_client.policy_store.base_policy_store_client.PolicyStoreTransactionContextManager object at 0x7f7f9e60dbe0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 256, in set_policies
return await self._set_policies_from_complete_bundle(bundle)
│ │ └ PolicyBundle(manifest=['data.json', 'rbac.rego', 'utils.rego'], hash='c60470ef8046152a8498d81df0eff66ddd1632e2', old_hash=Non...
│ └ <function OpaClient._set_policies_from_complete_bundle at 0x7f7fa0b22430>
└ <opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 268, in _set_policies_from_complete_bundle
await self.set_policy(policy_id=module.path, policy_code=module.rego)
│ │ │ │ │ └ '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This example defines an RBAC model for a Pet St...
│ │ │ │ └ RegoModule(path='rbac.rego', package_name='app.rbac', rego='# Role-based Access Control (RBAC)\n# ---------------------------...
│ │ │ └ 'rbac.rego'
│ │ └ RegoModule(path='rbac.rego', package_name='app.rbac', rego='# Role-based Access Control (RBAC)\n# ---------------------------...
│ └ <function AsyncRetrying.wraps.<locals>.async_wrapped at 0x7f7fa0b1daf0>
└ <opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 79, in async_wrapped
return await fn(*args, **kwargs)
│ │ └ {'policy_id': 'rbac.rego', 'policy_code': '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This ...
│ └ (<opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>,)
└ <function OpaClient.set_policy at 0x7f7fa0b1d9d0>
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 42, in __call__
do = self.iter(retry_state=retry_state)
│ │ └ <tenacity.RetryCallState object at 0x7f7f9e60de80>
│ └ <function BaseRetrying.iter at 0x7f7fa0ac5c10>
└ <AsyncRetrying object at 0x7f7fa0b0ee80 (stop=<tenacity.stop.stop_after_attempt object at 0x7f7fa0b0ed30>, wait=<tenacity.wai...
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 373, in iter
six.raise_from(retry_exc, fut.exception())
│ │ │ │ └ <function Future.exception at 0x7f7fa1fba550>
│ │ │ └ <Future at 0x7f7f9e60d9a0 state=finished raised ValueError>
│ │ └ RetryError(<Future at 0x7f7f9e60d9a0 state=finished raised ValueError>)
│ └ <function raise_from at 0x7f7fa0bd2790>
└ <module 'six' from '/root/.local/lib/python3.8/site-packages/six.py'>
File "<string>", line 3, in raise_from
tenacity.RetryError: RetryError[<Future at 0x7f7f9e60d9a0 state=finished raised ValueError>]
2021-08-23T17:43:39.286038+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
2021-08-23T17:43:39.297423+0000 |opal_client.policy.updater | INFO | Connected to server
2021-08-23T17:43:39.297608+0000 |opal_client.policy.updater | INFO | Refetching policy code (full bundle)
2021-08-23T17:43:39.328751+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: 'c60470ef8046152a8498d81df0eff66ddd1632e2'
2021-08-23T17:43:39.331631+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41412', 'req_id': 8, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-08-23T17:43:39Z'}
2021-08-23T17:43:39.332070+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41412', 'req_id': 8, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 13, 'resp_duration': 0.132113, 'resp_status': 200, 'time': '2021-08-23T17:43:39Z'}
2021-08-23T17:43:39.334898+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41414', 'req_id': 9, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-08-23T17:43:39Z'}
2021-08-23T17:43:39.336371+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41414', 'req_id': 9, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 315, 'resp_duration': 0.946037, 'resp_status': 400, 'time': '2021-08-23T17:43:39Z'}
2021-08-23T17:43:41.342537+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:41416', 'req_id': 10, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-08-23T17:43:41Z'}
2021-08-23T17:43:41.344037+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:41416', 'req_id': 10, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 315, 'resp_duration': 1.443397, 'resp_status': 400, 'time': '2021-08-23T17:43:41Z'}
2021-08-23T17:43:41.345722+0000 |opal_client...base_policy_store_client |WARNING | OPA transaction failed, transaction id=c60470ef8046152a8498d81df0eff66ddd1632e2, actions=['set_policies'], error=RetryError(<Future at 0x7f7f9e58b640 state=finished raised ValueError>)
2021-08-23T17:43:41.346221+0000 |fastapi_websocket_rpc.websocket_rpc_c...|ERROR | RPC Error
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/tenacity/_asyncio.py", line 45, in __call__
result = await fn(*args, **kwargs)
│ │ └ {'policy_id': 'rbac.rego', 'policy_code': '# Role-based Access Control (RBAC)\n# --------------------------------\n#\n# This ...
│ └ (<opal_client.policy_store.opa_client.OpaClient object at 0x7f7fa0deb4f0>,)
└ <function OpaClient.set_policy at 0x7f7fa0b1d940>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 173, in set_policy
return await proxy_response_unless_invalid(opa_response, accepted_status_codes=[status.HTTP_200_OK])
│ │ │ └ 200
│ │ └ <module 'starlette.status' from '/root/.local/lib/python3.8/site-packages/starlette/status.py'>
│ └ <ClientResponse(http://localhost:8181/v1/policies/rbac.rego) [400 Bad Request]>
│ <CIMultiDictProxy('Content-Type': 'applicatio...
└ <function proxy_response_unless_invalid at 0x7f7fa0b0cca0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.11-py3.8.egg/opal_client/policy_store/opa_client.py", line 54, in proxy_response_unless_invalid
raise ValueError("OPA Client: unexpected status code: {}, error: {}".format(response.status_code, error))
│ │ └ {'code': 'invalid_parameter', 'message': 'error(s) occurred while compiling module(s)', 'errors': [{'code': 'rego_type_error'...
│ └ 400
└ <starlette.responses.Response object at 0x7f7f9e58bac0>
I tried to have a clean setup by
docker system prune
docker rm -f $(docker ps -a -q) # Delete all containers
docker volume rm $(docker volume ls -q) # Delete all volumes
and run again, but still same. Any idea? thanks!
is there a way to update data in the postgres db using the opal server api?
I saw some code in the provider
file.
from opal_common.logger import logger
logger.debug('...')
I also tried to add some
print('...')
However, they never got printed after I docker compose up --build
.
How can I turn on the logs? Thanks!
It would be great to update authorizon/opal-client
to permitio/opal-client
! 😃
https://github.com/permitio/opal-fetcher-postgres/blob/master/Dockerfile#L1
Also note once switch to permitio/opal-client
, you will see
SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.