Encoding with application/x-www-form-urlencoded

[grpc-ecosystem]

Encoding with application/x-www-form-urlencoded

[theRealWardo]

we ended up rewriting application/x-www-form-urlencoded requests as JSON prior to the grpc-gateway receiving them like this:

        formWrapper := func(h http.Handler) http.Handler {
                return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                        log.Printf("Got request: %#v\n", r)
                        if strings.ToLower(strings.Split(r.Header.Get("Content-Type"), ";")[0]) == "application/x-www-form-urlencoded" {
                                log.Println("Rewriting form data as json")
                                if err := r.ParseForm(); err != nil {
                                        http.Error(w, err.Error(), http.StatusBadRequest)
                                        log.Println("Bad form request", err.Error())
                                        return
                                }
                                jsonMap := make(map[string]interface{}, len(r.Form))
                                for k, v := range r.Form {
                                        if len(v) > 0 {
                                                jsonMap[k] = v[0]
                                        }
                                }
                                jsonBody, err := json.Marshal(jsonMap)
                                if err != nil {
                                        http.Error(w, err.Error(), http.StatusBadRequest)
                                }

                                r.Body = ioutil.NopCloser(bytes.NewReader(jsonBody))
                                r.ContentLength = int64(len(jsonBody))
                                r.Header.Set("Content-Type", "application/json")
                        }
                        mux.ServeHTTP(w, r)
                })
        }
        if err := http.ListenAndServe(":8080", formWrapper(mux)); err != nil {
                log.Fatalf("failed to start gateway server on 8080: %v", err)
        }

[buckhx]

Is this supported or still open?

I'm implementing an OAuth2 server and the RFC requires the request Content-Type to be application/x-www-form-urlencoded.

[jhowliu]

Below is my custom marshaler for decoding the request in x-www-form-urleencoded format and returning the response in JSON. Hope this can help someone who suffer the same problem.

package marshaler

import (
	"encoding/json"
	"fmt"
	"io"
	"io/ioutil"
	"net/url"

	"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
	"github.com/grpc-ecosystem/grpc-gateway/v2/utilities"
	"google.golang.org/protobuf/proto"
)

type UrlEncodeMarshal struct {
	runtime.Marshaler
}

// ContentType means the content type of the response
func (u UrlEncodeMarshal) ContentType(_ interface{}) string {
	return "application/json"
}

func (u UrlEncodeMarshal) Marshal(v interface{}) ([]byte, error) {
	return json.Marshal(v)
}

// NewDecoder indicates how to decode the request
func (u UrlEncodeMarshal) NewDecoder(r io.Reader) runtime.Decoder {
	return runtime.DecoderFunc(func(p interface{}) error {
		msg, ok := p.(proto.Message)
		if !ok {
			return fmt.Errorf("not proto message")
		}

		formData, err := ioutil.ReadAll(r)
		if err != nil {
			return err
		}

		values, err := url.ParseQuery(string(formData))
		if err != nil {
			return err
		}

		filter := &utilities.DoubleArray{}

		err = runtime.PopulateQueryParameters(msg, values, filter)

		if err != nil {
			return err
		}

		return nil
	})
}

[mickael-kerjean]

@jhowliu do you prefer to make a PR yourself so I add the tests to your PR or should I create the PR with your code myself?

[rinatio]

+1 It would be useful to have it built-in!

[jinleileiking]

After one afternoon reading the f**king source code.

This solution is not good.

1. It cannot check the input whether a int or string by pb.
2. ...

The good solution is:

1. make post && form use get logic.
2. change the autogen code:

- func request_Example_GetForm_0(ctx context.Context, marshaler runtime.Marshaler, client ExampleClient, req *http.Request, pathPar
|     var protoReq FormRequest
|     var metadata runtime.ServerMetadata
|      
|        ---------------------------from form data to genrate : req.URL.Query() ------------------------------
|     if  err := runtime.PopulateQueryParameters(&protoReq, req.URL.Query(), filter_Example_GetForm_0); err != nil {
|         return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
|     }
|
|     msg, err := client.GetForm(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
|     return msg, metadata, err
|
| }

2. change the form data to url.Values then call:

PopulateQueryParameters(msg proto.Message, values url.Values, filter *utilities.DoubleArray)

[tuhao1020]

any update? @yugui

[johanbrandhorst]

I don't know anyone working on this at the moment. Are you interested in helping add support for this?

[tuhao1020]

@johanbrandhorst I'll try it after reading the relevant code

[rmilejcz]

what are the protocols for this in 2022? We are trying to support receiving data from third party APIs (so we cannot control the format of the post request) and we ran into this issue. What are people doing to properly serve x-www-form-urlencoded via gprc-gateway? Is there any option besides writing our own custom marshaler?

I see this pull request but it was not merged, and it references another PR that was merged but it doesn't seem to affect whether or not we can add support for x-www-form-urlencoded requests

[johanbrandhorst]

I think your only recourse today is a custom marshaler.

[mickael-kerjean]

This works fine. Any chance this could get merge in here? I'm happy to contribute in unit testing as it seems that was missing part in https://github.com/grpc-ecosystem/grpc-gateway/pull/548/files

[johanbrandhorst]

Yes, we'd be happy to accept a new contribution for this, thank you!

[jhowliu]

Hi @mickael-kerjean , sorry about waiting for a while. I have made a PR. Feel free to add the tests.

https://github.com/jhowliu/grpc-gateway/tree/feat/support-url-form-encoded-marshaler