opensuse / defrag-api Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU General Public License v3.0
License: GNU General Public License v3.0
Hi guys, do you want to add a code formatter like black
to have a more readable code? Have in mind that this will format all the code to a unique one
Steps to reproduce
Expected behavior
Information about the bug is displayed.
Actual behavior
The following returns:
{
"query": {
"service": "bugs",
"item_key": 1175535
},
"results_count": null,
"results": null,
"error": "A likely programming error occurred: This container type does not support __getitem__: <class 'pottery.dict.RedisDict'>"
}
Detailed information
In order to get more detailed information about what exactly goes wrong, we need to the query
function in the Run
class in
defrag/modules/helpers/services_manager.py so that it looks like this:
@staticmethod
async def query(query: CacheQuery, fallback: Optional[partial] = None) -> QueryResponse:
""" Tries to run the given query, doing all the caching work along the way. A 'finally' clause might
be in order. """
if not ServicesManager.services:
raise QueryException(
"Services need to be initialized before running a query.")
try:
async with Run.Cache(query, fallback) as results:
return QueryResponse(query=query, results=results, results_count=len(results))
except QueryException as err:
return QueryResponse(query=query, error=f"Unable to satisfy this query for this reason: {err}")
except Exception as err:
raise err # This line here changed
return QueryResponse(query=query, error=f"A likely programming error occurred: {err}")
Doing so will give us the following stacktrace:
INFO: 127.0.0.1:56334 - "GET /bugs/bug/1175535 HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
return await self.app(scope, receive, send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
await super().__call__(scope, receive, send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc from None
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc from None
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/routing.py", line 580, in __call__
await route.handle(scope, receive, send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/routing.py", line 241, in handle
await self.app(scope, receive, send)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/routing.py", line 52, in app
response = await func(request)
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/fastapi/routing.py", line 216, in app
raw_response = await run_endpoint_function(
File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
return await dependant.call(**values)
File "/home/jens/src/defrag-api/defrag/modules/bugs.py", line 166, in get_bug
return await Run.query(cache_query, fallback)
File "/home/jens/src/defrag-api/defrag/modules/helpers/services_manager.py", line 217, in query
raise err
File "/home/jens/src/defrag-api/defrag/modules/helpers/services_manager.py", line 212, in query
async with Run.Cache(query, fallback) as results:
File "/home/jens/src/defrag-api/defrag/modules/helpers/services_manager.py", line 189, in __aenter__
if items_from_cache := await self.cache.search_items(item_key=self.query.item_key):
File "/home/jens/src/defrag-api/defrag/modules/helpers/sync_utils.py", line 42, in inner
return await loop.run_in_executor(None, f_saturated)
File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/jens/src/defrag-api/defrag/modules/helpers/cache_stores.py", line 54, in search_items
raise Exception(
Exception: This container type does not support __getitem__: <class 'pottery.dict.RedisDict'>
For this, https://github.com/pontaoski/QueryKit might be useful.
We should probably first focus on the milestone -- if you guys accept it.
But right after it, it would be nice to start using it for defrag
. Structural pattern matching and better type annotations would help. It would take about 5 minutes to refactor to become 3.10-compliant.
Currently we have:
├── defrag
│ ├── __init__.py (1)
│ ├── __main__.py (2)
│ ├── modules
│ │ ├── helpers (3)
│ ├── profiling
│ └── tests
This comes with a number of caveats.
gunicorn
requires the package.module:object
syntax. This means we have to the run the application with something like gunicorn -w 4 -k uvicorn.workers.UvicornWorker defrag.main:app
. __main__.py
is not expressible in this syntax as far as I know.gunicorn
, as gunicorn needs to latch on to the actually application
object exposed by fastAPI. There is simple solution (tested here) but it's not the most beautiful one.Recommendations:
__main__.py
to main.py
defrag/
The built-in python logger is I/O blocking. This means that using the built-in logging module will interfere with your asynchronous application performance (Adrien: which means that it interferes with profiling). aiologger aims to be the standard Asynchronous non-blocking logging for python and asyncio. Source: https://github.com/async-worker/aiologger
@cboltz
Hey Christian,
sorry for pinning you.
could you please add @thunderbirdtr to this repository so that he can commit and merge PRs to it?
Also, can you give me permissions to change this repository's settings?
Thank you very much!
We do need a dep manager. There is the built-in virtualenv, Pipenv and others. I prefer Pipenv because it's very readable and intuitive. Are you Ok using it? If not, what alternative do you favor?
Do we agree to 100% test coverage on functions (including classes constructors if any)?
Do we agree to annotating all functions with documentation comments? It can be very short, but it should make clear why certain design decisions where made when being explicit helps using the function.
I will update my PRs and already merged code in the upcoming days to meet these requirements. My PR for Reddit / Twitter already tries to satisfy these conditions. See #23
There is one last issue about this: How to be able to run tests involving the use of personal credentials? Ideally no personal credential would be used, but realistically, working as volunteers with limited time and money, it's not easy. For example, I created a dev account on Twitter to be able to fetch Tweets, and unless I pay money the dev account includes 1-2 credentials from my personal Twitter account. :(
... should be total for all endpoints handlers + all functions defined in helpers
, which of course implies that app-level initialization with credentials works.
search team members by name or roles as docs, packagers, Heroes, Board members, Members, etc) and detect if they have common channels with the user [bot feature] (so that their privacy is respected while also being identifiable)
Hi I am currently a CS student in Ireland (I am from from Belgium :) ) and I would be interested to contribute to this project!!
I have got in touch with @why-not-try-calmer on Matrix and he told me the conversation was on Telegram for the most part. Is there also somewhere on Matrix I can ask a few questions and, if you guys agree, get to contribute to this repository? I want to make sure I get the right ideas about the API before doing anything stupid 👯♀️
This telegram command function bring "searching bugs from openSUSE"
With the incoming organizer.py
module it is time to think about which data we should provide to whom.
Module | Data offered to pushees | Data offered to pollers | Permissions |
---|---|---|---|
organizer | events | events | * creators should be alowed to read+write everything about what they created; * consumers should have read access to all public events; * only whitelisted consumers should have access to restricted events |
dispatcher | events notifications | events notifications | * notifications should be cancellable only by their creators; * a single consumer that we control should be authorized to poll pending notifications |
suggestions (formerly "voting") | list of votes + suggestions | list of votes + suggestions | *creators should be able to edit description and/or title; * participants (voters) should be able to get the identifier of any ongoing vote / suggestion |
update: I am trying to learn things here and there.
The FastAPI docs caught my attention with this sentence:
So I peeked under the hood and I found: https://github.com/tiangolo/fastapi/blob/717a1ec40992d00e34635f0b4c9b68ab5a6daef7/fastapi/concurrency.py#L4
This means that starlette -- the asgi server -- already resolves sync function calls with threads, and thus that we don't need to ad our own threading with to_async
. However, I also think that starlette operates with an all-or-nothing strategy: every sequences of function calls that start with an async endpoint, like:
app.get(...)
async def ....
are evaluated with the event loop, and all sequences of function calls that start without an async endpoint with threads + the event loop. This means that we don't have to use our own threading on top of that, but also that we should not mix async with sync things along a single route.
This seems to leave us with two options:
async
function calls (including from routes/paths) except where these calls can be scheduled without awaiting them (i.e. when we make a request with aiohttp); orpottery
, the main source of blocking sync calls, and make everything async.pottery
is sparing me a lot of time because I basically don't have to write any code to talk directly to redis, which is amazing. So I am all in favour of solution (1).
The aim in resolving this issue is to write a module for @karatekbot in telegram.
I am using Pylance to guide/enforce type annotations (as per Python 3.8.10) and autopep8 as formatter.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.