freeconf / restconf Goto Github PK
View Code? Open in Web Editor NEWImplementation of RESTCONF Management protocol - IETF RFC8040
License: Apache License 2.0
Implementation of RESTCONF Management protocol - IETF RFC8040
License: Apache License 2.0
json object lack the "prefix" from the modules they are imported from to avoid name collisions.
note: beyond the technical challenge, this breaks compatibility fairly significantly. might consider a mode where this be be opt-out
The server is returning a 200 OK even if set an error. The error handling in https://github.dev/freeconf/restconf/util.go calls http.Error(w, msg, code) and I clearly see while debugging that the code is set to 401:
func handleErr(compliance ComplianceOptions, err error, r *http.Request, w http.ResponseWriter) bool {
if err == nil {
return false
}
fc.Debug.Printf("web request error [%s] %s %s", r.Method, r.URL, err.Error())
msg := err.Error()
code := fc.HttpStatusCode(err)
if !compliance.SimpleErrorResponse {
errResp := errResponse{
Type: "protocol",
Tag: decodeErrorTag(code, err),
Path: decodeErrorPath(r.RequestURI),
Message: msg,
}
var buff bytes.Buffer
fmt.Fprintf(&buff, `{"ietf-restconf:errors":{"error":[`)
json.NewEncoder(&buff).Encode(&errResp)
fmt.Fprintf(&buff, `]}}`)
msg = buff.String()
}
http.Error(w, msg, code)
return true
}
http.Error(w, msg, code)
generates the error message http: superfluous response.WriteHeader call from github.com/freeconf/restconf.handleErr (util.go:91)
This is the http.Error() function:
func Error(w ResponseWriter, error string, code int) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(code)
fmt.Fprintln(w, error)
}
The error occurs when calling w.WriteHeader(code)
. The variable wroteHeader
in the response is already set and cannot be changed.
func (w *response) WriteHeader(code int) {
if w.conn.hijacked() {
caller := relevantCaller()
w.conn.server.logf("http: response.WriteHeader on hijacked connection from %s (%s:%d)", caller.Function, path.Base(caller.File), caller.Line)
return
}
if w.wroteHeader {
caller := relevantCaller()
w.conn.server.logf("http: superfluous response.WriteHeader call from %s (%s:%d)", caller.Function, path.Base(caller.File), caller.Line)
return
}
.
.
.
This is caused by the sendOutput
done in freeconf/restconf/browser_handler.go at line 171 before the handleErr and line 172.
sendOutput
calls fmt.Fprintf
at line 198. This is calling WriteHeader
eventually.
Comply with media types outlines in RESTCONF RFC
https://datatracker.ietf.org/doc/html/rfc8040#section-11.3
Currently freeconf mostly ignores media types except in a few rare instances
notification should be wrapped with
{
"ietf-restconf:notification" : {
"eventTime": "2008-07-08T00:01:00Z",
"event": {
payload here
}
}
}
see https://datatracker.ietf.org/doc/html/rfc7950#section-7.16.3
If I understand correctly, by adding CAs in fc-restconf.web.tls.ca, a client will have its cert validated against any of the CAs according to
Line 31 in 141e395
Does this mean that if a new CA needs to be added, then the server needs to be restarted?
it would be nice if we could set this to RequireAnyClientCert or RequestClientCert and then be able to manually validate the cert using a Filter. This way, the application using restconf would be able to handle requests and validate the certs using a dynamic list that is handled outside of restconf
Or any other suggestions to achieve the same goal?
There is
https://freeconf.org/docs/gettingstarted/
which describes a "Go" way and a "python" way.
When choosing the "Go" way, the repo is downloaded from github.com/freeconf/restconf
Webserver/RESTCONF is then in "Go"?
When going the python way, a Python code is downloaded from same repo?
So the repo contains 2 separated projects?
Or is the webserver (or some parts) always in "Go" and "just" used from "Python" side?
Or is there perhaps some automatic conversion from "Go" to "Python"?
One of the backgrounds for my questions:
Server is running on
http://localhost:8080/
Try the following commands:
curl -X GET
http://localhost:8080/restconf/data/hello:
Starting event listener
Event listener started
Event listener waiting for event
ERRO 2024/07/22 15:10:35 browser_handler.go:150: rpc error: code = Unknown desc = Exception iterating responses: exceptions must derive from BaseException
Event listener connection closed
We have perhaps done a wrong request (or something else).
But I was wondering why "browser_handler.go" was mentioned, even we used the Python way...
In the case where content-type is set to yang-data+xml in a request, we would need restconf to be able to parse the XML data into the yang model.
This is irrelevant from the Accept header.
I don't think we need to force the response to XML. The response can still be sent with yang-data+json, as long as the Accept header contains it. I will address this in a different issue.
Support verifying that a leaf
of type empty
is present or not in ActionRequest.Input
.
Currently, calling req.Input.GetValue(...)
will always return nil
whatever the leaf is there or not.
Having something like req.Input.HasLeaf(name)
would be nice.
Content-Type set in util::handleErr() is overwritten by function Error() in net/http/server.go
Line 105 in 7e8989a
Here is the code in server.go caussing the problem:
func Error(w ResponseWriter, error string, code int) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(code)
fmt.Fprintln(w, error)
}
What I was doing in util::handleErr() was to not call http.Error(). I was doing this:
w.Header().Set("Content-Type", string(mime))
w.WriteHeader(code)
w.Header().Set("X-Content-Type-Options", "nosniff")
fmt.Fprintln(w, msg)
ApplyStartupConfigFile from local.go file currently panics when something goes wrong opening the config file (for instance when we provide an incorrect startup file path). This prevents applications from properly handling errors. It is also not in line with the function signature, which suggests it should return errors when something goes wrong
Current implementation:
https://github.com/freeconf/restconf/blob/master/device/local.go#L112
func (self *Local) ApplyStartupConfigFile(fname string) error {
cfgRdr, err := os.Open(fname)
defer cfgRdr.Close()
if err != nil {
panic(err)
}
return self.ApplyStartupConfig(cfgRdr)
}
Restconf does not set the body in the right format (XML) for errors with a body when the accept in the request is XML. The response is always in JSON.
The merge request for XML were including the required code in util.go but it has not been taken.
The ServeHTTP() function in server.go does create its own context based on context.Background(). Is there a special reason not to use r.Context() ? Using the http.Request context (which is normally context.Background()) should make it easier to handle situations with http based context interruptions.
I noticed that when restconf panics due to the use of a wrong type for reflection it does not print the name of the field that caused the problem but only the type. I think it could simplify debugging to have access to that information
when adding a leaf, in the car example, with "type empty", I get the error: typedef empty not found
At the moment its not possible to use freeconf/restconf without it opening a HTTP server in the background, which makes it difficult/impossible to integrate into existing services.
#36 should fix this
I am no expert on reflection in go but I was wondering if you had any plan to support reflection for base types such as the example that follows. I was surprised to have to use a custom parser (albeit a simple one) to parse a list of string and integers.
list bar {
key foo;
leaf foo {
type int32 {
range "0..120";
}
}
leaf baz {
type string;
}
}
You reference should be updated to latest version to pull in the new union fixes.
For example,
POST /restconf/operations/example-ops:reboot HTTP/1.1
Host: example.com
Content-Type: application/yang-data+json
{
"example-ops:input" : {
"delay" : 600,
"message" : "Going down for system maintenance",
"language" : "en-US"
}
}
does not work. We get an error saying "Expected format: http://server/restconf[=device]/operation/module:path"
But the RFC does not mention anything about "=device". Also, looking at server.go#L146, it seems that "operations" is simply not implemented and the error message is misleading
When returning an error with
// Option #2 - enhanced but still an 401 error return fmt.Errorf("Bad ACL %w", fc.UnauthorizedError)
It would be useful if restconf could create a payload according to the model of ietf-restconf:errors
{
"ietf-restconf:errors": {
"error": [
{
"error-type": "protocol",
"error-tag": "lock-denied",
"error-message": "Lock failed; lock already held"
}
]
}
}
In server_test.go the test in line 54 seems to be wrong... the return type suggest this was meant to do a "GET" operation on the "car" data, but what the test does is to retrieve the schema of "car", exactly as the test beginning in line 51.
This can also be seen in the file testdata/gold/car.json, which doesn't contain JSON data but a YANG file.
I guess the test should be a line like:
req, _ := http.NewRequest("GET", fmt.Sprintf("%s/restconf/data/car", addr), nil)
Restconf go.mod does not refer to the latest freeconf/yang with commit:
With this change freecong/nodeutil/json_rdr.go is returning two values and the current code in browser_handler.go is also expecting two values but restconf/go.mod is pointing to an older version of freeconf/yang.
I am using go version go1.17.6 windows/amd64
When I try to go get freeconf/restconf I get the following error:
go get github.com/freeconf/restconf
# github.com/freeconf/yang/val
..\..\..\..\go\pkg\mod\github.com\freeconf\[email protected]\val\conv.go:596:36: rv.CanInt undefined (type reflect.Value has no field or method CanInt)
..\..\..\..\go\pkg\mod\github.com\freeconf\[email protected]\val\conv.go:689:36: rv.CanUint undefined (type reflect.Value has no field or method CanUint)
The current content-type in a POST response is set to application/json.
According to RFC8040 it must be application/yang-data+xml or application/yang-data+json. In our case it is application/yang-data+json.
When doing a POST with content-type "application/yang-data+json", the restconf server responds with a correctly formated response that conforms to our yang model.
But if the request is made with application/yang-data+xml. then the response is send as application/yang-data+json, but with the top level attribute stripped out.
Response content type not correctly set after XML pull request merge.
The current code on master is calling browser_handler::setContentType(). See
Line 206 in 1ccf812
It must calls util::setResponseContentType() .
It is meant to be directed to web app, which have have any endpoint
Here is an example of an errors body in xml:
<errResponseWrapper>
<errors>
<error>
<error-type>protocol</error-type>
<error-tag>access-denied</error-tag>
<error-path>ietf-sztp-bootstrap-server:report-progress</error-path>
<error-message>Bad ACL not authorized</error-message>
</error>
</errors>
</errResponseWrapper>
errResponseWrapper should not be there and xmlns must be set to "urn:ietf:params:xml:ns:yang:ietf-restconf"
Line 87 in 9848fef
I'm not 100% sure if this is RFC compliant but I think the logic would be valid:
Restconf could default to always return data with yang-data+json.
Unless the client sends "Accept: yang-data+xml" and nothing else. In this case, restconf would send the reply in xml.
bascically:
Client Accept Header | What restconf returns |
---|---|
yang-data+json | yang-data+json |
yang-data+json and yang-data+xml | yang-data+json |
yang-data+xml | yang-data+xml |
something else | 415 ? |
So the logic is only based on the Accept header and not on the Content-Type.
Hi!
I'm still tweaking the "Next step" and "RESTCONF client" example. This time I want to start the car through the RPC and then see that status reflected in the car:running
leaf:
import (
"fmt"
"strings"
"testing"
"github.com/freeconf/examples/car"
"github.com/freeconf/restconf"
"github.com/freeconf/restconf/client"
"github.com/freeconf/restconf/device"
"github.com/freeconf/yang/fc"
"github.com/freeconf/yang/node"
"github.com/freeconf/yang/nodeutil"
"github.com/freeconf/yang/source"
)
func connectClient() {
// YANG: just need YANG file ietf-yang-library.yang, not the yang of remote system as that will
// be downloaded as needed
ypath := restconf.InternalYPath
// Connect
proto := client.ProtocolHandler(ypath)
dev, err := proto("http://localhost:9998/restconf")
if err != nil {
panic(err)
}
// Get a browser to walk server's management API for car
car, err := dev.Browser("car")
if err != nil {
panic(err)
}
root := car.Root()
defer root.Release()
// Subscribe to notifications
sel, err = root.Find("update")
if err != nil {
panic(err)
}
defer sel.Release()
unsub, err := sel.Notifications(func(n node.Notification) {
msg, err := nodeutil.WriteJSON(n.Event)
if err != nil {
panic(err)
}
fmt.Println(msg)
})
if err != nil {
panic(err)
}
defer unsub()
// Wait a bit just to ensure we receive the car started notification
time.Sleep(time.Second)
// Start car
sel, err = root.Find("start")
if err != nil {
panic(err)
}
if _, err = sel.Action(nil); err != nil {
panic(err)
}
println("Waiting for some notifications...")
for i := 0; i < 10; i++ {
sel, err = root.Find("running")
if err != nil {
panic(err)
}
running, err := sel.Get()
if err != nil {
panic(err)
}
fmt.Printf("Is the car running? %v\n", running)
time.Sleep(time.Second)
}
}
func main() {
connectClient()
}
Output:
Waiting for some notifications...
Is the car running? false
{"event":"carStarted"}
Is the car running? false
Is the car running? false
Is the car running? false
Is the car running? false
Is the car running? false
{"event":"flatTire"}
{"event":"carStopped"}
Is the car running? false
Is the car running? false
Is the car running? false
Is the car running? false
I understand that the configuration is retrieved from the server and cached locally. I just wonder how I can forcibly re-retrieve the configuration? I appreciate your time!
Hello, I try to use the following yang file to test config-false
:
module hello {
leaf message {
type string;
}
leaf config-false-message {
type string;
config false;
}
}
And then I POST to http://localhost:8080/restconf/data/hello:
with the body:
{"config-false-message":"test"}
I observed that the POST method successfully edited the "config-false-message".
However, based my understanding from dicussion and RFC 6020-YANG, the leaf marked with config false
cannot be manipulated by POST or other method.
Could you please confirm if this behavior is expected?
Thank you very much.
The RFC shows that the data being posted as inputs to a RPC call should be embeded in an object called "input" as shown below
{
"example-ops:input" : {
"delay" : 600,
"message" : "Going down for system maintenance",
"language" : "en-US"
}
}
It seems that freeconf only supports posting the payload directly such as
{
"delay" : 600,
"message" : "Going down for system maintenance",
"language" : "en-US"
}
Hi!
If I interpret the supported RFC 8525 correctly, the container modules-state
from restconf/data/ietf-yang-library
is deprecated in favor of a container yang-library
. However, when I call the endpoint on the Car example, the reply contains modules-state
but not yang-library
. Is this a bug?
the ApplyOptions() function in stock/web.go creates a new http server every times new parameters are set via restconf.
I think it should use Shutdown()/Close() on the old server before opening the new one.
Maybe with something like the following change (this one has a hardcoded 10 second wait for graceful shutdown):
diff --git a/stock/web.go b/stock/web.go
index 6cbdf4b..192f38d 100644
--- a/stock/web.go
+++ b/stock/web.go
@@ -42,6 +42,14 @@ func (service *HttpServer) ApplyOptions(options HttpServerOptions) {
if options == service.options {
return
}
+ if service.Server != nil {
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ go func() {
+ _ = service.Server.Shutdown(ctx)
+ cancel()
+ }()
+ _ = service.Server.Close()
+ }
service.options = options
service.Server = &http.Server{
Addr: options.Port,
I get this error when I have
Device (uses choose) <- Client Gateway <- REST GET
I assume the client doesn't properly support Choose
Hi!
I'm trying to set up a RESTCONF client through FREECONF. I'm working through the Car example and aim to present the user with configuration, update the configuration, and display the new configuration. Unfortunately, after a configuration update I fail to display the full configuration again.
My client code:
// Courtesy to https://freeconf.org/docs/examples/restconf-client/
package main
import (
"github.com/freeconf/restconf"
"github.com/freeconf/restconf/client"
"github.com/freeconf/yang/nodeutil"
)
func connectClient() {
// YANG: just need YANG file ietf-yang-library.yang, not the yang of remote system as that will
// be downloaded as needed
ypath := restconf.InternalYPath
// Connect
proto := client.ProtocolHandler(ypath)
dev, err := proto("http://localhost:8080/restconf")
if err != nil {
panic(err)
}
// Get a browser to walk server's management API for car
car, err := dev.Browser("car")
if err != nil {
panic(err)
}
root := car.Root()
defer root.Release()
actual, err := nodeutil.WritePrettyJSON(car.Root())
if err != nil {
panic(err)
}
println("========== initial config ==========")
println(actual)
n, err := nodeutil.ReadJSON(`{"speed":50}`)
if err != nil {
panic(err)
}
err = root.UpsertFrom(n)
if err != nil {
panic(err)
}
actual, err = nodeutil.WritePrettyJSON(car.Root())
if err != nil {
panic(err)
}
println("========== config after setting speed ==========")
println(actual)
n, err = nodeutil.ReadJSON(`{"pollInterval":50}`)
if err != nil {
panic(err)
}
err = root.UpsertFrom(n)
if err != nil {
panic(err)
}
actual, err = nodeutil.WritePrettyJSON(car.Root())
if err != nil {
panic(err)
}
println("========== config after setting pollInterval ==========")
println(actual)
}
func main() {
connectClient()
}
The output:
========== initial config ==========
{
"speed":50,
"pollInterval":50,
"running":false,
"miles":0,
"lastRotation":0,
"tire":[
{
"pos":0,
"size":"H15",
"worn":false,
"wear":100,
"flat":false},
{
"pos":1,
"size":"H15",
"worn":false,
"wear":100,
"flat":false},
{
"pos":2,
"size":"H15",
"worn":false,
"wear":100,
"flat":false},
{
"pos":3,
"size":"H15",
"worn":false,
"wear":100,
"flat":false}]}
========== config after setting speed ==========
{
"speed":50,
"tire":[
{
"size":"H15"},
{
"size":"H15"},
{
"size":"H15"},
{
"size":"H15"}]}
========== config after setting pollInterval ==========
{
"pollInterval":50,
"tire":[
{
"size":"H15"},
{
"size":"H15"},
{
"size":"H15"},
{
"size":"H15"}]}
I had expected that the nodeutil.WritePrettyJSON(car.Root())
calls would return the full config, not just the updated part + all configuration further dow the hierarchy. I'd be happy to learn how to obtain the full config (and why the above output is sensible).
Could you help me out? Looking forward to your (usually impressively swift) reply!
Hey!
I've been playing around with FreeCONF for some days. In general I like it, and it really helps me to get a better understanding of YANG and RESTCONF. Today I tried to patch/put/post data to a binary leaf, without success. Perhaps you could point me to a solution?
Approach: I follow the "Getting started" tutorial and change the type of message
into type binary;
. If I start the server, I get the following error:
Starting server...
Traceback (most recent call last):
File ".../freeconf/server.py", line 62, in <module>
browser = main()
File ".../freeconf/server.py", line 55, in main
dev.apply_startup_config_file("./startup.json")
File ".../freeconf/.venv/lib/python3.10/site-packages/freeconf/device.py", line 26, in apply_startup_config_file
self.__apply_startup_config_stream(self.driver.fs.new_rdr_file(configFile))
File ".../freeconf/.venv/lib/python3.10/site-packages/freeconf/device.py", line 36, in __apply_startup_config_stream
self.driver.g_device.ApplyStartupConfig(req)
File ".../freeconf/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1176, in __call__
return _end_unary_response_blocking(state, call, False, None)
File ".../freeconf/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "cannot coerse 'string' to binary value"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-04-09T14:41:51.964747905+02:00", grpc_status:2, grpc_message:"cannot coerse \'string\' to binary value"}"
>
So that didn't go very well. For now, I can remove the "message"
key from ./startup.json
(so "hello: {}"
) and start the server again without issues. Then, I try to POST a base64-encoded string as described in RFC7950 as follows:
# base64 encoding of "Hello, world!"
curl -X POST -d '{"message": "SGVsbG8sIHdvcmxkIQ=="}' http://localhost:8080/restconf/data/hello
which returns
cannot coerse 'string' to binary value
Is base64 decoding of JSON strings for binary types not implemented? If it is, what am I doing wrong?
Looking forward to your help.
Hi, I was trying to figure out how to add a list to my restconf interface and I eventually found your more comprehensive car example. I think adding a couple of examples for lists and maps in the README would improve adoption. I would also clarify that reflection doesn't work in that case and highlight the parts where it does. I would also show how to update and query list/maps using curl. I would provide you with my example but you already have a better one in your car.yaml.
Let me know if you would like me to contribute with an example.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.