fengsp/django-grpc-framework

gRPC for Django.

Django gRPC Framework

Django gRPC framework is a toolkit for building gRPC services, inspired by djangorestframework.

Requirements

• Python (3.6, 3.7, 3.8)
• Django (2.2, 3.0), Django REST Framework (3.10.x, 3.11.x)
• gRPC, gRPC tools, proto3

Installation

$ pip install djangogrpcframework

Add django_grpc_framework to INSTALLED_APPS setting:

INSTALLED_APPS = [
    ...
    'django_grpc_framework',
]

Demo

Here is a quick example of using gRPC framework to build a simple model-backed service for accessing users, startup a new project:

$ django-admin startproject demo
$ python manage.py migrate

Generate .proto file demo.proto:

python manage.py generateproto --model django.contrib.auth.models.User --fields id,username,email --file demo.proto

Generate gRPC code:

python -m grpc_tools.protoc --proto_path=./ --python_out=./ --grpc_python_out=./ ./demo.proto

Now edit the demo/urls.py module:

from django.contrib.auth.models import User
from django_grpc_framework import generics, proto_serializers
import demo_pb2
import demo_pb2_grpc


class UserProtoSerializer(proto_serializers.ModelProtoSerializer):
    class Meta:
        model = User
        proto_class = demo_pb2.User
        fields = ['id', 'username', 'email']


class UserService(generics.ModelService):
    queryset = User.objects.all()
    serializer_class = UserProtoSerializer


urlpatterns = []
def grpc_handlers(server):
    demo_pb2_grpc.add_UserControllerServicer_to_server(UserService.as_servicer(), server)

That's it, we're done!

$ python manage.py grpcrunserver --dev

You can now run a gRPC client to access the service:

with grpc.insecure_channel('localhost:50051') as channel:
    stub = demo_pb2_grpc.UserControllerStub(channel)
    for user in stub.List(demo_pb2.UserListRequest()):
        print(user, end='')

django-grpc-framework's Issues

failed to connect to all addresses

https://djangogrpcframework.readthedocs.io/en/latest/quickstart.html

I did this tutorial but when executing this:

with grpc.insecure_channel('localhost:50051') as channel:
stub = account_pb2_grpc.UserControllerStub(channel)
for user in stub.List(account_pb2.UserListRequest()):
print(user, end='')

always return

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1605744210.127297754","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4165,"referenced_errors":[{"created":"@1605744210.127291858","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

i have a docker for this project

version: '3'

services:
db:
image: postgres
env_file:
- ./backend/.env
ports:
- "6543:5432"
networks:
- webappnetwork
backend:
build: ./backend
volumes:
- ./backend/:/code
env_file:
- ./backend/.env
ports:
- "8000:8000"
command: python manage.py runserver 0.0.0.0:8000
depends_on:
- db
networks:
- webappnetwork
grpc:
build: ./backend
volumes:
- ./backend/:/code
env_file:
- ./backend/.env
ports:
- "50051:50051"
command: python manage.py grpcrunserver --dev
depends_on:
- db
- backend
networks:
- webappnetwork

networks:
webappnetwork:
driver: bridge

volumes:
database-data:
pgadmin-data:

django-grpc-framework doesn't work with Django 4.1.1

Following quick start tutorial:
https://djangogrpcframework.readthedocs.io/en/latest/quickstart.html
with Django 4.1.1

command
python manage.py grpcrunserver --dev

return error:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()
  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/franco/.virtualenvs/proto/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/home/franco/.virtualenvs/proto/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/franco/.virtualenvs/proto/lib/python3.8/site-packages/django/core/management/__init__.py", line 279, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/franco/.virtualenvs/proto/lib/python3.8/site-packages/django/core/management/__init__.py", line 49, in load_command_class
    return module.Command()
  File "/home/franco/.virtualenvs/proto/lib/python3.8/site-packages/django/core/management/base.py", line 274, in __init__
    raise TypeError("requires_system_checks must be a list or tuple.")
TypeError: requires_system_checks must be a list or tuple.

attribute adjustment requires_system_checks

FIX: fix for requires_system_checks attribute as in Django version 4.1 there was a change to this attribute in the file

django/core/management/base.py

at the link django/django@1cb4950

the change is described,

the attribute previously received a boolean value, but has been changed to receive a list or tuple

"Using a boolean value for requires_system_checks is "
                "deprecated. Use '__all__' instead of True, and [] (an empty "
                "list) instead of False.",

the change was performed and the requires_system_checks attribute now receives an empty list [] as False

How to use paging queries？

class ListModelMixin:
def List(self, request, context):
"""
List a queryset. This sends a sequence of messages of
serializer.Meta.proto_class to the client.

    .. note::

        This is a server streaming RPC.
    """
    queryset = self.filter_queryset(self.get_queryset())
    serializer = self.get_serializer(queryset, many=True)
    for message in serializer.message:
        yield message

combine grpcrunserver and certificate

Good afternoon, how could I start the server in tls or ssl and that it works on port 443?

This project still alive?

Hi @fengsp

I found your library and it is quite interesting. I would like to know if you have plans to keep the project.
I would like to send some pull requests to the repository but I'm not sure if the project will be continued.
Thank you very much in advance.

Join forces with django-socio-grpc ?

Hi django-grpc-framework developers, @fengsp, @sobolevn and @Montagu,

there is a nicely developed fork of this repo at django-socio-grpc.

Wouldn't it be great to join forces and develop the django-grpc-framework together in one repo,
joining all the resources ?

django-socio-grpc has some very nice convenience features, django-grpc-framework is lacking - but documentation could be improved.

What do think about this idea ?

How to enable support of the reflection API

Hi, thank you for the awesome library!

I have used grpcurl cli to inspect my dev grpc server (run by python manage.py grpcrunserver --dev) and then trying to inspect available methods using grpcurl -plaintext localhost:50051 list:

$ grpcurl -plaintext localhost:50051 list
Failed to list services: server does not support the reflection API

I would like to know is there a way to enable reflection api? I think it would be useful to inspect new apis for the documentation purpose like openapi or something

Discussion on "How apt is gRPC for Django?"

Hey!
I have been using Django for some time in past but am completely new to gRPC.
I have found some really cool things in gRPC but some serious comments saying gRPC is not meant for / not good for Django.

Could someone post some helpful resources commenting on this issue? Your personal views will also be helpful.

grpc_settings: get user_stettings does not seem to work properly ?

I cannot access GRPC_FRAMEWORK user settings properly (might be a bug in line 97 of your setting.py module):

I specified some user settings in my django settings file:

settings.py
GRPC_FRAMEWORK = { # 'ROOT_HANDLERS_HOOK': 'handlers.grpc_handlers', 'GRPC_HANDLERS': [] }
but when I try to retrieve the user settings:
from django_grpc_framework.settings import grpc_settings print(grpc_settings.GRPC_HANDLERS)

I get the expected error:
print(grpc_settings.GRPC_HANDLERS) File "/home/mark/py3venv/lara10/lib/python3.9/site-packages/django_grpc_framework/settings.py", line 93, in __getattr__ raise AttributeError("Invalid gRPC setting: '%s'" % attr) AttributeError: Invalid gRPC setting: 'GRPC_HANDLERS'

but it will not search in the self._user_settings

I think, there is an underscore missing in line 97 of django_grpc_framework/settings.py:
It should probably read:
try: # Check if present in user settings **val = self._user_settings[attr]**

I would be happy, if you could resolve the bug soon - or update the documentation on the correct way of adding user settings.
Thanks a lot !

howto register multiple grpc handlers from multiple apps in one django project ?

I have a django project with > 10 different apps, each having a gRPC interface.
What is the cleanest way to register all gRPC handlers in your framework ?
It would be nice, if I could use the GRPC_FRAMEWORK settings for that, but it currently has only a GRPC_ROOT_HANDLERS hook, which points to one module.

What about adding there a list of GRPC_HANDLERS ?
Example:

GRPC_FRAMEWORK = { 'GRPC_HANDLERS': ['app1.grpc_interface.handlers', 'app2.grpc_interface.handlers', ...] }
This would allow to add new handlers even with environment variables, e.g. for configuring a complex django project.

interceptor error

my test interceptor code:

import grpc
from loguru import logger`



class ValidateInterceptor(grpc.ServerInterceptor):

    def intercept_service(self, continuation, handler_call_details):
        logger.info('this is my interceptor')
        return continuation(handler_call_details)

i think there is something wrong in djangogrpcframework settings while handling interceptor path, it needs the () to instantiate the class which imports from the settings with string. Could you help me take a look?

How to set the options of default grpc server?

It seems there is no api to set the options of grpc server. The options can be set only when initialize the grpc server.

I encountered an issue says "Received message larger than max (4750306 vs. 4194304)". I know how to solve it by setting the options of customized grpc server.

But I'd like to change the options of the default grpc server and i didn't found the entry of the default server. Does anyone can help or any suggestions?

django apps as grpc clients

This is quite an interesting project, but I have an orthogonal question, sorry for asking on the wrong forum.

We have grpc services written in Java and a Django-based portal with vueJS front end. To communicate with grpc services, we end up adding a django rest framework layer to do the marshaling/unmarshalling of grpc to REST so VueJS can consume or add an Envoy Proxy on the server-side to expose REST api's. Do you happen to have pointers for us to directly consume grpc messages in a Django app with Vue.js front-end? -- Thank you

details = "Exception calling application: 'BlogProtoSerializer' object has no attribute 'proto_class'"

I checked everything and followed every step of the document. Got as much as help from AIs too, no progress. I'll share any code if necessary.

problem in test

hi
when I use RPCTestCase and call the grpc function
I saw error that postgres connection close
but before call the function I can query and every thing is ok

Instructions on how to run this in production?

Do we use Nginx or guincorn?

Using both REST and gRPC

I'm new to django. Can we use django with both REST and gRPC framework? I tried add paths to urlpatterns in settings.py to use REST but gRPC handler hook seems to ignore it.

how to support secured channel

without this, it is fairly not usable :(

attribute adjustment requires_system_checks

FIX: fix for requires_system_checks attribute as in Django version 4.1 there was a change to this attribute in the file

django/core/management/base.py

at the link django/django@1cb4950

the change is described,

the attribute previously received a boolean value, but has been changed to receive a list or tuple

"Using a boolean value for requires_system_checks is "
"deprecated. Use 'all' instead of True, and [] (an empty "
"list) instead of False.",

the change was performed and the requires_system_checks attribute now receives an empty list[] as False

How to implement authorization with django build auth module

This extension is pretty good. But how to implement authorization. Thanks.

Logging support

Hi, thanks for this library, really impressed by how similar everything is to the Rest framework and hopefully this can become as popular some day.

One large worry that I noticed immediately on working with this framework was the lack of logging, I had to work with logs on the client side to know things are working or not. However, in a production scenario there are a large number of cases where we rely more on server logging and monitoring since it is easier to manage.

Suggestions:

• Access log for all requests made to the server
• Application log which logs any failures encountered in processing a request
• Preferably create an exception handler since they can be handled with frameworks for realtime monitoring like Cloudwatch or Sentry.

Unnecessary invoking db.close_old_connections() causes test case failed

I hit problem with test case failed and complaining - exception is connection already closed.

I fix this issue by removing invoking db.close_old_connections() in Servicer's handler() before the try block. Django manages database connection by itself(https://docs.djangoproject.com/en/3.1/ref/databases/#connection-management) and to me it's not necessary to invoke that manually.

attribute adjustment requires_system_checks

FIX: fix for requires_system_checks attribute as in Django version 4.1 there was a change to this attribute in the file

django/core/management/base.py

at the link django/django@1cb4950

the change is described,

the attribute previously received a boolean value, but has been changed to receive a list or tuple

"Using a boolean value for requires_system_checks is "
"deprecated. Use 'all' instead of True, and [] (an empty "
"list) instead of False.",

the change was performed and the requires_system_checks attribute now receives an empty list[] as False

Customize serializer

I am looking at how to customize serializer

How do I have data from two join tables?

In the restframework, I can do like :

class BookSerializer(serializers.ModelSerializer):
    category_name = serializers.CharField(source='category.name', read_only=True)
    publisher_name = serializers.CharField(source='publisher.name', read_only=True)
    author_last_name = serializers.CharField(source='author.last_name', read_only=True)
    author_first_name = serializers.CharField(source='author.first_name', read_only=True)

    class Meta:
        model = Book
        fields = (
            'id', 
            'title', 
            'category', 
            'category_name', 
            'publisher', 
            'publisher_name', 
            'author', 
            'author_last_name', 
            'author_first_name'
        )

How should I do in grpc?

I almost got my django-grpc + react CRUD working.

https://github.com/ccapeng/bookstore_grpc
https://github.com/ccapeng/bookstore-redux-grpc

Bug: serializing/deserializing of JSONfields fails

Dear developers,

JSON de-serialization fails with the following Error:

python data_grpc_client.py
Traceback (most recent call last):
File "/my_data/data_grpc_client.py", line 8, in
for datum in stub.List(my_data.lara_data_pb2.DataListRequest()):
File ".../python3.9/site-packages/grpc/_channel.py", line 426, in next
return self._next()
File .../lib/python3.9/site-packages/grpc/_channel.py", line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Exception iterating responses: Failed to parse data_JSON field: expected string or bytes-like object."
debug_error_string = "{"created":"@1648155177.535827174","description":"Error received from peer ipv6:[::1]:50051","file":"src/core/lib/surface/call.cc","file_line":905,"grpc_message":"Exception iterating responses: Failed to parse data_JSON field: expected string or bytes-like object.","grpc_status":2}"

What is missing to support JSONField serialization / deserialization ?

To reproduce the Error:

I have a django (4.0.3) model with a JSON field:

# models.py

class Data(models.Model):
  data_id = models.AutoField(primary_key=True)
  data_JSON = models.JSONField(blank=True, null=True, help_text="JSON representation of the data")

My serializer.py looks like this:

from  my_data.models import Data
from django_grpc_framework import proto_serializers
from rest_framework import serializers
import my_data.my_data_pb2


class DataProtoSerializer(proto_serializers.ModelProtoSerializer):
    class Meta:
        model = Data
        proto_class = my_data.lara_data_pb2.Data
        fields = ['data_id', 'data_JSON']

The auto generated protofile file (I used the django framework proto generator) my_data.proto:

syntax = "proto3";

package my_data;

import "google/protobuf/empty.proto";

service DataController {
    rpc List(DataListRequest) returns (stream Data) {}
    rpc Create(Data) returns (Data) {}
    rpc Retrieve(DataRetrieveRequest) returns (Data) {}
    rpc Update(Data) returns (Data) {}
    rpc Destroy(Data) returns (google.protobuf.Empty) {}
}

message Data {
    int32 data_id = 1;
    string data_JSON = 2;
}

message DataListRequest {
}

message DataRetrieveRequest {
    int32 data_id = 1;
}

My service description - services.py :

from my_data.models import Data
from django_grpc_framework import generics
from my_data.serializers import DataProtoSerializer


class DataService(generics.ModelService):
    """
    gRPC service that allows users to be retrieved or updated.
    """
    queryset = Data.objects.all().order_by('data_id') 
    serializer_class = DataProtoSerializer

And finally the client: data_grpc_client.py:

import grpc
import my_data.my_data_pb2
import my_data.my_data_pb2_grpc


with grpc.insecure_channel('localhost:50051') as channel:
    stub = my_data.my_data_pb2_grpc.DataControllerStub(channel)
    for datum in stub.List(my_data.my_data_pb2.DataRequest()):
        print("datum:",datum.name, end='')