opensuse/defrag-api

Goal

The goal of this project is to combat the fragmentation between teams, groups and services in openSUSE.

It seems to us that openSUSE has many niche places that don't talk to each other that much, let alone report their activities to hubs accessible to outsiders or potentially interested contributors.

Also, events and activities are not always visible and more often than not, people would find out about a community event or an outage when it's too late.

We suspect that this kind of fragmentation and opacity makes it more difficult than necessary for new users to onboard the openSUSE Project and for users to enjoy it as much as it deserves.

Our hope is that in so doing we will also help strengthen the bounds between people, teams and communities.

Features

• • => not implemented
• [...] => work in progress
• • => implemented

Middlewares

Cross-platform (Matrix, Telegram) moderation (depends on: https://github.com/openSUSE/defrag-api/projects/2#card-64817715)

• • implement CAPTCHA (e.g. with emojis)
• • (partially implemented in openGM) implement moderation tools (globally)
• • implement roles & badges for users (globally)

Broadcast (push-based)

• • send service alerts
• • send community announcements
• • probe and list external services for status (unknown issue | undergoing maintenance | ups and downs | all good

On-demand (poll-based)

• • probe and list external services for status (unknown issue | undergoing maintenance | ups and downs | all good
• list recent news / events / contribution opportunities:
  • Reddit
  • Twitter
• • search forum posts
• • search wiki
• • search bugs on bugzilla
• • search openSUSE documentation
• • search Factory + Pre-Factory packages with zypper + opi
• • search for Progress / Pagure
• • search for activities/events

How to deploy?

Please see deploying.md for information on how to deploy the project.

How to contribute?

1. Set up your environment.
2. Think about what you want to change or what feature you want to add. Optionnally, talk to us about it at https://t.me/openSUSE_defrag. Ask as many questions as you need.
3. Fork & clone the repository.
4. Open an Issue where where you:
   • describe the goal of the change / new feature
   • provide an example of a typical use-case
   • request specific changes to the current code to support your feature, if you think that your feature justify making these changes and if you need such changes

Even though this is not required, in my experience working in peers (2 people working together on different parts of the same thing) works well. This can spare you a lot of time if the other person is more familiar with the code base. The better your Issue, the more likely someone will be willing to work with you.

Important to keep in mind:

1. We are building an async server-side application/service. Make sure you don't write any code performing I/O bound computation that is not async. We provide the typical loop.run_in_executor(None, ...) trick to support sync libraries.
2. If your Pull Request introduces a new function, it must feature a new unittest using pytest for it, unless your function is consumed by another function which is introduced in the same Pull Request and which does have a corresponding unittest. (Later we will need to have 1 unittest for each function though.)
3. Try to squash your commits on your Pull Requests to avoid noise.

Set up your environment

• a virtual environment, such as pipenv, virtualenv, venv, etc. See this page for ideas.
• a Python 3.8 interpreter run from the CPython runtime (the default one)
• a static type checker such as pyright (We are using it through Pylance and coc-pyright).
• configure the project as described in configuration.md

Helpers to get started with GitHub

• https://www.dataschool.io/how-to-contribute-on-github/

Design of this application

Architecture

Handling logic

defrag-api's Issues

Feature: Package search

For this, https://github.com/pontaoski/QueryKit might be useful.

querying information about a bug gives internal server error

Steps to reproduce

1. Start up defrag
2. Point your browser to http://localhost:8000/bugs/bug/1175535

Expected behavior
Information about the bug is displayed.

Actual behavior
The following returns:

{
  "query": {
    "service": "bugs",
    "item_key": 1175535
  },
  "results_count": null,
  "results": null,
  "error": "A likely programming error occurred: This container type does not support __getitem__: <class 'pottery.dict.RedisDict'>"
}

Detailed information
In order to get more detailed information about what exactly goes wrong, we need to the query function in the Run class in
defrag/modules/helpers/services_manager.py so that it looks like this:

    @staticmethod
    async def query(query: CacheQuery, fallback: Optional[partial] = None) -> QueryResponse:
        """ Tries to run the given query, doing all the caching work along the way. A 'finally' clause might
        be in order. """
        if not ServicesManager.services:
            raise QueryException(
                "Services need to be initialized before running a query.")
        try:
            async with Run.Cache(query, fallback) as results:
                return QueryResponse(query=query, results=results, results_count=len(results))
        except QueryException as err:
            return QueryResponse(query=query, error=f"Unable to satisfy this query for this reason: {err}")
        except Exception as err:
            raise err # This line here changed
            return QueryResponse(query=query, error=f"A likely programming error occurred: {err}")

Doing so will give us the following stacktrace:

INFO:     127.0.0.1:56334 - "GET /bugs/bug/1175535 HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/fastapi/routing.py", line 216, in app
    raw_response = await run_endpoint_function(
  File "/home/jens/src/defrag-api/ven/lib64/python3.8/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/jens/src/defrag-api/defrag/modules/bugs.py", line 166, in get_bug
    return await Run.query(cache_query, fallback)
  File "/home/jens/src/defrag-api/defrag/modules/helpers/services_manager.py", line 217, in query
    raise err
  File "/home/jens/src/defrag-api/defrag/modules/helpers/services_manager.py", line 212, in query
    async with Run.Cache(query, fallback) as results:
  File "/home/jens/src/defrag-api/defrag/modules/helpers/services_manager.py", line 189, in __aenter__
    if items_from_cache := await self.cache.search_items(item_key=self.query.item_key):
  File "/home/jens/src/defrag-api/defrag/modules/helpers/sync_utils.py", line 42, in inner
    return await loop.run_in_executor(None, f_saturated)
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/jens/src/defrag-api/defrag/modules/helpers/cache_stores.py", line 54, in search_items
    raise Exception(
Exception: This container type does not support __getitem__: <class 'pottery.dict.RedisDict'>

telegram feature /bugs for searching openSUSE bugzilla

This telegram command function bring "searching bugs from openSUSE"

Replace logger with aiologger

The built-in python logger is I/O blocking. This means that using the built-in logging module will interfere with your asynchronous application performance (Adrien: which means that it interferes with profiling). aiologger aims to be the standard Asynchronous non-blocking logging for python and asyncio. Source: https://github.com/async-worker/aiologger

Restructuring top-level main & init

Currently we have:

├── defrag
│   ├── __init__.py  (1)
│   ├── __main__.py  (2)
│   ├── modules
│   │   ├── helpers  (3)
│   ├── profiling
│   └── tests

This comes with a number of caveats.

• gunicorn requires the package.module:object syntax. This means we have to the run the application with something like gunicorn -w 4 -k uvicorn.workers.UvicornWorker defrag.main:app. __main__.py is not expressible in this syntax as far as I know.
• as we close in to deployment the distinction between (1) and (2) is not clear. (1) does the job of enumerating enabled modules and exposing them to the entire package, while (2) does the job of registering modules-as-services and to expose special handlers ('startup', 'shutdown'). But the way this is written currently does not work if we run the app as mentioned in the previous bullet point, because the main function defined in (2) is not accessible to gunicorn, as gunicorn needs to latch on to the actually application object exposed by fastAPI. There is simple solution (tested here) but it's not the most beautiful one.
• helpers (3) are considered modules, but there is no need to since they are pure functions and don't depend on any state for their functionality. Also, some portions of code across several modules depend on helpers. Making them modules suggest that nothing depends on them and/or that they are stateful, when neither is true.

Recommendations:

• let's rename __main__.py to main.py
• let's move all the business logic dealing with importing modules and registering modules as services to (1), and call the registration unconditionnally there; let's have (2) only expose he special handlers as it already does.
• let's move (3) directly under defrag/

Repository permissions

@cboltz
Hey Christian,
sorry for pinning you.
could you please add @thunderbirdtr to this repository so that he can commit and merge PRs to it?
Also, can you give me permissions to change this repository's settings?
Thank you very much!

Captchas

The aim in resolving this issue is to write a module for @karatekbot in telegram.

Add a code formatter like `black`

Hi guys, do you want to add a code formatter like black to have a more readable code? Have in mind that this will format all the code to a unique one

Unit test coverage...

... should be total for all endpoints handlers + all functions defined in helpers, which of course implies that app-level initialization with credentials works.

async/sync: offloading entire function call trees to starlette-managed threads (fully sync) VS thin sync-to-async wrappers (fully async)

update: I am trying to learn things here and there.

The FastAPI docs caught my attention with this sentence:

So I peeked under the hood and I found: https://github.com/tiangolo/fastapi/blob/717a1ec40992d00e34635f0b4c9b68ab5a6daef7/fastapi/concurrency.py#L4

This means that starlette -- the asgi server -- already resolves sync function calls with threads, and thus that we don't need to ad our own threading with to_async. However, I also think that starlette operates with an all-or-nothing strategy: every sequences of function calls that start with an async endpoint, like:

app.get(...)
async def ....

are evaluated with the event loop, and all sequences of function calls that start without an async endpoint with threads + the event loop. This means that we don't have to use our own threading on top of that, but also that we should not mix async with sync things along a single route.

This seems to leave us with two options:

1. Remove all async function calls (including from routes/paths) except where these calls can be scheduled without awaiting them (i.e. when we make a request with aiohttp); or
2. Stop using pottery, the main source of blocking sync calls, and make everything async.

pottery is sparing me a lot of time because I basically don't have to write any code to talk directly to redis, which is amazing. So I am all in favour of solution (1).

Python 3.10

We should probably first focus on the milestone -- if you guys accept it.

But right after it, it would be nice to start using it for defrag. Structural pattern matching and better type annotations would help. It would take about 5 minutes to refactor to become 3.10-compliant.

Profiling: some tools

• https://github.com/pyroscope-io/pyroscope trash
• https://github.com/benfred/py-spy

Feature: Search Forums

Review of the data flow in view of a sound authentication / authorization scheme

With the incoming organizer.py module it is time to think about which data we should provide to whom.

No security issue

• bugs
• docs
• reddit
• search (for now)
• twitter
• wikis

Potential security issue

• search

Security issues

Module	Data offered to pushees	Data offered to pollers	Permissions
organizer	events	events	* creators should be alowed to read+write everything about what they created; * consumers should have read access to all public events; * only whitelisted consumers should have access to restricted events
dispatcher	events notifications	events notifications	* notifications should be cancellable only by their creators; * a single consumer that we control should be authorized to poll pending notifications
suggestions (formerly "voting")	list of votes + suggestions	list of votes + suggestions	*creators should be able to edit description and/or title; * participants (voters) should be able to get the identifier of any ongoing vote / suggestion

Feature: Caching

Feature: Search the wiki

https://www.mediawiki.org/wiki/API:Search

Feature: Search Bugzilla

Feature: Team phonebook

search team members by name or roles as docs, packagers, Heroes, Board members, Members, etc) and detect if they have common channels with the user [bot feature] (so that their privacy is respected while also being identifiable)

Tests

Python Dependency Manager: Pipenv?

We do need a dep manager. There is the built-in virtualenv, Pipenv and others. I prefer Pipenv because it's very readable and intuitive. Are you Ok using it? If not, what alternative do you favor?

Other platforms than Telegram?

Hi I am currently a CS student in Ireland (I am from from Belgium :) ) and I would be interested to contribute to this project!!

I have got in touch with @why-not-try-calmer on Matrix and he told me the conversation was on Telegram for the most part. Is there also somewhere on Matrix I can ask a few questions and, if you guys agree, get to contribute to this repository? I want to make sure I get the right ideas about the API before doing anything stupid 👯‍♀️

Comment and add tests for all functions

Do we agree to 100% test coverage on functions (including classes constructors if any)?

Do we agree to annotating all functions with documentation comments? It can be very short, but it should make clear why certain design decisions where made when being explicit helps using the function.

I will update my PRs and already merged code in the upcoming days to meet these requirements. My PR for Reddit / Twitter already tries to satisfy these conditions. See #23

There is one last issue about this: How to be able to run tests involving the use of personal credentials? Ideally no personal credential would be used, but realistically, working as volunteers with limited time and money, it's not easy. For example, I created a dev account on Twitter to be able to fetch Tweets, and unless I pay money the dev account includes 1-2 credentials from my personal Twitter account. :(

Type annotations and formatter

I am using Pylance to guide/enforce type annotations (as per Python 3.8.10) and autopep8 as formatter.

1. Do you agree to enforce type annotations at least on function signatures?
2. Do you agree to use Pylance for it?
3. Which formatter do you prefer?