sixcodes/brandenburg

Paused for now.

brandenburg's Issues

json.decoder.JSONDecodeError error on api startup

Converter data type from SQL Server to BigQuery

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

https://www.connectionstrings.com/sql-server-2012-data-types-reference/

Configure badges

[ ] - Codecov
[ ] - Typing Hint
[ ] - Black

Create connection with SNS(AWS) as sms broker

aioredis.errors.MaxClientsError: ERR max number of clients reached on Heroku platform

Checklist

• The bug is reproducible against the latest release and/or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

The number of redis connections is over the limit (256) during the receipt of POSTS of new tables through the batch of endpoints (https://brandenburg.orbia.ag/v1/import/batch/) and push (https://brandenburg.orbia.ag/v1/import/push/)

To reproduce

The problem occurs after a sequence of calls to the batch endpoint, in which the following exceptions begin to appear:

Mar 9 02:00:55 brandenburg-prod app[web] info   File "./brandenburg/toolbox/_backends/redis.py", line 58, in set_cache
Mar 9 02:00:55 brandenburg-prod app[web] info     with await cls.__instance.conn as cache:
Mar 9 02:00:55 brandenburg-prod app[web] info TypeError: object NoneType can't be used in 'await' expression

Mar 9 02:12:30 brandenburg-prod app[web] info   File "./brandenburg/toolbox/_backends/redis.py", line 49, in __disconnect
Mar 9 02:12:30 brandenburg-prod app[web] info     cls.__instance.conn.close()
Mar 9 02:12:30 brandenburg-prod app[web] info AttributeError: 'NoneType' object has no attribute 'close' 

These exceptions end up throwing ERROR: Exception in ASGI application in LogDNA and a while after errors in set_cache and __disconnect start to appear
exceptions aioredis.errors.MaxClientsError: ERR max number of clients reached.

Expected behavior

The batch endpoint should receive multiple requests without errors in the _set_last_run function

Actual behavior

The batch endboint throws exceptions to the _set_last_run function related to methods
set_cache and __disconnect of the RedisBackend class.

Debugging material

Mar 9 02:00:55 brandenburg-prod app[web] INFO INFO:     34.236.239.193:0 - "POST /v1/import/batch/ HTTP/1.1" 500 Internal Server Error
Mar 9 02:00:55 brandenburg-prod app[web] ERROR ERROR:    Exception in ASGI application
Mar 9 02:00:55 brandenburg-prod app[web] info Traceback (most recent call last):
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
Mar 9 02:00:55 brandenburg-prod app[web] info     result = await app(self.scope, self.receive, self.send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     return await self.app(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/fastapi/applications.py", line 190, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await super().__call__(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/applications.py", line 111, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await self.middleware_stack(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     raise exc from None
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await self.app(scope, receive, _send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/httpsredirect.py", line 19, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await self.app(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await self.app(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     raise exc from None
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await self.app(scope, receive, sender)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/routing.py", line 566, in __call__
Mar 9 02:00:55 brandenburg-prod app[web] info     await route.handle(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/routing.py", line 227, in handle
Mar 9 02:00:55 brandenburg-prod app[web] info     await self.app(scope, receive, send)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/routing.py", line 41, in app
Mar 9 02:00:55 brandenburg-prod app[web] info     response = await func(request)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/fastapi/routing.py", line 189, in app
Mar 9 02:00:55 brandenburg-prod app[web] info     dependant=dependant, values=values, is_coroutine=is_coroutine
Mar 9 02:00:55 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
Mar 9 02:00:55 brandenburg-prod app[web] info     return await dependant.call(**values)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "./brandenburg/routers/imports.py", line 43, in import_batch
Mar 9 02:00:55 brandenburg-prod app[web] info     result, processed = await BatchService.execute(batch, batch.service_id, "batch")
Mar 9 02:00:55 brandenburg-prod app[web] info   File "./brandenburg/services/batch.py", line 22, in execute
Mar 9 02:00:55 brandenburg-prod app[web] info     await cls._set_last_ran(batch)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "./brandenburg/services/batch.py", line 45, in _set_last_ran
Mar 9 02:00:55 brandenburg-prod app[web] info     await cache.set_cache(key=table, value=last_updated_at, ttl=-1)
Mar 9 02:00:55 brandenburg-prod app[web] info   File "./brandenburg/toolbox/_backends/redis.py", line 58, in set_cache
Mar 9 02:00:55 brandenburg-prod app[web] info     with await cls.__instance.conn as cache:
Mar 9 02:00:55 brandenburg-prod app[web] info TypeError: object NoneType can't be used in 'await' expression

Mar 9 02:12:30 brandenburg-prod app[web] INFO INFO:     34.236.239.193:0 - "POST /v1/import/batch/ HTTP/1.1" 500 Internal Server Error
Mar 9 02:12:30 brandenburg-prod app[web] ERROR ERROR:    Exception in ASGI application
Mar 9 02:12:30 brandenburg-prod app[web] info Traceback (most recent call last):
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
Mar 9 02:12:30 brandenburg-prod app[web] info     result = await app(self.scope, self.receive, self.send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     return await self.app(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/fastapi/applications.py", line 190, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await super().__call__(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/applications.py", line 111, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await self.middleware_stack(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     raise exc from None
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await self.app(scope, receive, _send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/httpsredirect.py", line 19, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await self.app(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/middleware/cors.py", line 78, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await self.app(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     raise exc from None
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await self.app(scope, receive, sender)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/routing.py", line 566, in __call__
Mar 9 02:12:30 brandenburg-prod app[web] info     await route.handle(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/routing.py", line 227, in handle
Mar 9 02:12:30 brandenburg-prod app[web] info     await self.app(scope, receive, send)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/starlette/routing.py", line 41, in app
Mar 9 02:12:30 brandenburg-prod app[web] info     response = await func(request)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/fastapi/routing.py", line 189, in app
Mar 9 02:12:30 brandenburg-prod app[web] info     dependant=dependant, values=values, is_coroutine=is_coroutine
Mar 9 02:12:30 brandenburg-prod app[web] info   File "/app/.heroku/python/lib/python3.7/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
Mar 9 02:12:30 brandenburg-prod app[web] info     return await dependant.call(**values)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "./brandenburg/routers/imports.py", line 43, in import_batch
Mar 9 02:12:30 brandenburg-prod app[web] info     result, processed = await BatchService.execute(batch, batch.service_id, "batch")
Mar 9 02:12:30 brandenburg-prod app[web] info   File "./brandenburg/services/batch.py", line 22, in execute
Mar 9 02:12:30 brandenburg-prod app[web] info     await cls._set_last_ran(batch)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "./brandenburg/services/batch.py", line 45, in _set_last_ran
Mar 9 02:12:30 brandenburg-prod app[web] info     await cache.set_cache(key=table, value=last_updated_at, ttl=-1)
Mar 9 02:12:30 brandenburg-prod app[web] info   File "./brandenburg/toolbox/_backends/redis.py", line 61, in set_cache
Mar 9 02:12:30 brandenburg-prod app[web] info     await cls.__disconnect()
Mar 9 02:12:30 brandenburg-prod app[web] info   File "./brandenburg/toolbox/_backends/redis.py", line 49, in __disconnect
Mar 9 02:12:30 brandenburg-prod app[web] info     cls.__instance.conn.close()
Mar 9 02:12:30 brandenburg-prod app[web] info AttributeError: 'NoneType' object has no attribute 'close'

Environment

Heroku Dyno (standard 2X) x2
Heroku Addon Redis Enterprise Cloud 100MB

Create connection with SES(aws) as email broker

Create connection with Twilio as whatsapp broker

Create connection with Zenvia as whatsapp broker

Create connection with Zenvia as sms broker

Logger revision

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Migrate to use poetry instead of pipenv

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Create a plug and play approach for new broker/pipe

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

[Jinja2] template engine do get files from GCS

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Create connection with Sendgrid as email broker

Create connection with Twilio as sms broker

Create script to setup local environment

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Create the salesforce broker

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Create endpoint to import a file to be inserted on GCS or S3

Review README docs

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Create connection with Mailgun as email broker

Create notifier endpoint

/notify/

Request Body

"contact": "",
"data": {},
"by": "",// [sms, whatsapp, email] 
"template_name": "",
"locale": ""

Handling with multiples events and send for kafka, pubsub or rabbitmq

Checklist

• There are no similar issues or pull requests for this yet.

Is your feature related to a problem? Please describe.

Describe the solution you would like.

Describe alternatives you considered

Additional context

Ensure CI integration with Github Actions is working

Improve the basic documentation