tufanbarisyildirim/gonginx

Nginx configuration parser helps you to parse, edit, regenerate your nginx config in your go applications

Gonginx

TBH, I would like to rewrite the parser next time I need it again :)) but it still does its job.

Gonginx is an Nginx configuration parser helps you to parse, edit, regenerate your nginx config files in your go applications. It makes managing your balancer configurations easier.

Basic grammar of an nginx config file

%token Keyword Variable BlockStart BlockEnd Semicolon Regex

%%

config      :  /* empty */ 
            | config directives
            ;
block       : BlockStart directives BlockEnd
            ;
directives  : directives directive
            ;
directive   : Keyword [parameters] (semicolon|block)
            ;
parameters  : parameters keyword
            ;
keyword     : Keyword 
            | Variable 
            | Regex
            ;

API

Core Components

• Parser

  Parser is the main package that analyzes and turns nginx structred files into objects. It basically has 3 libraries, lexer explodes it into tokens and parser turns tokens into config objects which are in their own package,

• Config

  Config package is representation of any context, directive or their parameters in golang. So basically they are models and also AST

• Dumper

  Dumper is the package that holds styling configuration only.

Examples

• Formatting
• Adding a Server to upstream block
• add-custom-directive
• dump-nginx-config
• update-directive
• update-server-listen-port

Examples and Library Reference

TODO

• associate comments with config objects to print them on config generation and make it configurable with dumper.Style
• move any context wrapper into their own file (remove from parser)
• Parse included files recusively, keep relative path on load, save all in a related structure and make that optional in dumper.Style
• Implement specific searches, like finding servers by server_name (domain) or any upstream by target etc.
• add more examples
• link the parent directive to any directive for easier manipulation

Limitations

There is no known limitations yet. PRs are more than welcome if you want to implement a specific directive / block, please read Contributing before your first PR.

License

MIT License

gonginx's Issues

Dumper does not preserve the formatting in log_format

I have a log_format like this:

    log_format main '{"@timestamp":"$time_iso8601",'
                    '"host":"$hostname",'
                    '"server_ip":"$server_addr",'
                    '"client_ip":"$remote_addr",'
                    '"remote_user":"$remote_user",'
                    '"xff":"$http_x_forwarded_for",'
                    '"domain":"$host",'
                    '"url":"$uri",'
                    '"referer":"$http_referer",'
                    '"args":"$args",'
                    '"upstream_response_time":"$upstream_response_time",'
                    '"request_time":"$request_time",'
                    '"request_method":"$request_method",'
                    '"status":"$status",'
                    '"size":"$body_bytes_sent",'
                    '"content_length":"$content_length",'
                    '"request_length":"$request_length",'
                    '"protocol":"$server_protocol",'
                    '"upstreamhost":"$upstream_addr",'
                    '"file_dir":"$request_filename",'
                    '"http_user_agent":"$http_user_agent",'
                    '"upstream_cache_status":"$upstream_cache_status",'
                    '"file_name":"$file_name"'
                    '}';

After the dump code ,the result is:

log_format main {@timestamp:, host:, server_ip:, client_ip:, remote_user:, xff:, domain:, url:, referer:, args:, upstream_response_time:, request_time:, request_method:, status:, size:, content_length:, request_length:, protocol:, upstreamhost:, file_dir:, http_user_agent:, upstream_cache_status:, file_name: };

panic: unexpected token BlockEnd (})

Part of my nginx conf:

 64     init_by_lua_block {
 65         local xxx= require 'xxx'
 66
 67         local config_file = "./conf/xxx.conf"
 68         local env_path = "/x/env.properties"
 69         local config = xxx.init({
 70             config = config_file,
 71             env = env_path
 72         })
 73
 74         context = {
 75             xxx= xxx,
 76             config = config
 77         }
 78     }

I already used the latest version:

require (
	github.com/tufanbarisyildirim/gonginx v0.0.0-20230325082000-26dcb15a9df4
)

the error:

panic: unexpected token BlockEnd (}) on line 72, column 9

goroutine 1 [running]:
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:212 +0x93b
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:164 +0xbc
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:205 +0x7a5
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:164 +0xbc
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:205 +0x7a5
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:164 +0xbc
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:205 +0x7a5
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0xc000194000)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:164 +0xbc
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).Parse(...)
        github.com/tufanbarisyildirim/gonginx/parser/parser.go:147

Adding a new location block

I see there are methods to add a server and upstream but not a location directive.
Am I missing it?

Thanks

parser.parseStatement does not follow grammar for parsing parameters

The grammar outlined in the documentation suggests that parsing directives follows these rules:

directive   : Keyword [parameters] (semicolon|block)
            ;
parameters  : parameters keyword
            ;
keyword     : Keyword 
            | Variable 
            | Regex
            ;

This is implemented in parseStatement and the code iterates over tokens and checks for quoted string or keyword tokens, but not for variables even though the keyword production above clearly indicates Variables are allowed.

This can be easily checked by trying to parse the following fragment:

...
http {
   ...
    map $host $clientname {
        default -;
    }
  ...
}

In this case the parameters are variables and cause panic during parse

mime.types are not recognized properly

package main

import (
"fmt"
"log"

"github.com/tufanbarisyildirim/gonginx/parser"

)

func main() {
p, err := parser.NewParser("/etc/nginx/nginx.conf", parser.WithIncludeParsing())
if err != nil {
log.Fatalln("parser", err)
}

c, err := p.Parse()
if err != nil {
	log.Fatalln("parser2", err)
}

directives := c.FindDirectives("server")

for _, block := range directives {
	for _, directive := range block.GetBlock().GetDirectives() {
		if directive.GetName() == "server_name" {
			fmt.Println(directive.GetName(), directive.GetParameters())
		}
	}
}

}

unknown directive 'text/html' on line 3, column 5

nginx.conf
#...
include /etc/nginx/mime.types;
#...

mime.types

types {
text/html html htm shtml;
....

can't use custom directives

Nginx ecosystem has a huge community developed modules with lots of custom directives. We need to add an option to provide our own custom directives
thanks for reporting and the idea @jwriteclub

Painc casued by parsing config which contained lua block

Part of my nginx config:

http {
    lua_package_path "/usr/local/nginx/test/lib/?.lua;;";
    lua_shared_dict tracing_buffer 100m;
    init_by_lua_block {
        require "resty.core"
        collectgarbage("collect")
    }

   server {
      listen       80;
      #  some route rules .... 
  }
}

My code:

p, err := parser.NewParser("/Users/caiweicheng/Downloads/conf/nginx.conf")
if err != nil {
    return err
}
return p.Parse()

Full error:

panic: unexpected token BlockEnd (}) on line 27, column 5

goroutine 1 [running]:
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0x14000132000, 0x3, 0x14000124150)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:227 +0xaac
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0x14000132000, 0x2)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:165 +0xc4
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0x14000132000, 0x5, 0x14000124138)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:220 +0x6c0
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0x14000132000, 0x2)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:165 +0xc4
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseStatement(0x14000132000, 0x5, 0x140001240e0)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:220 +0x6c0
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).parseBlock(0x14000132000, 0x2c)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:165 +0xc4
github.com/tufanbarisyildirim/gonginx/parser.(*Parser).Parse(...)
        /Users/username/go/pkg/mod/github.com/tufanbarisyildirim/[email protected]/parser/parser.go:147
main.parseNginx(0x14000102058, 0x14000102000)
        /Users/username/go-project/awesomeProject/nginxparse/main.go:17 +0x74
main.main()

I found out that this error is due to the fact that each line of the lua block does not need to end with a semicolon. But when NewParser judges the end of the block, it must end with a semicolon.

What did you expect to see?

At least parse nginx config not painc.
I have two thoughts:

• First,NewParser handles lua modules specially, and supports no semicolon ending;
• Second, all block parsing can support not ending with a semicolon.

unknown directive 'check_status'

not support check_status directive

DumpConfig not print lua block content

package main

import (
	"fmt"

	"github.com/tufanbarisyildirim/gonginx"
	"github.com/tufanbarisyildirim/gonginx/parser"
)

func main() {
	p := parser.NewStringParser(`
server {
location = /foo {
rewrite_by_lua_block {
res = ngx.location.capture("/memc",
{ args = { cmd = "incr", key = ngx.var.uri } } # comment contained unexpect '{'
# comment contained unexpect '}' 
)
t = { key="foo", val="bar" }
}
}
}
`)

	c := p.Parse()
	fmt.Println(gonginx.DumpConfig(c, gonginx.IndentedStyle))

}

result

server {
    location = /foo {
        rewrite_by_lua_block {

        }
    }
}

gonginx does not seem to catch syntax errors in `nginx.conf`

We were hoping to use gonginx in the config-file-validator tool to provide syntax validation for nginx.conf files. During some testing we noticed that gonginx will not throw an error if the syntax of the nginx.conf file is invalid. For example, in this test we intentionally did not close a server block and it parsed without error.

Is this expected behavior, a bug, or are we parsing incorrectly?

NewParser use os.Open and how to close it

NewParser use os.Open and how to close it ?

got "unexpected token" with the follow nginx config file at line 45 ("text/css")

worker_processes 1;

events {

    worker_connections 1024;
}

## rtmp {
## 
##     server {
## 
##         listen 10935;
## 
##         application live {
## 
##             live on;
##         }
## 
##         application hls {
## 
##             live on;
##             hls on;
##             hls_path temp/hls;
##             hls_fragment 8s;
##         }
##     }
## }

http {

    include mime.types;
    default_type application/octet-stream;

    sendfile on;
    keepalive_timeout 65;
    # Enable Gzip
    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_min_length 1100;
    gzip_buffers 4 8k;
    gzip_proxied any;
    gzip_types
    # text/html is always compressed by HttpGzipModule
    text/css
    text/javascript
    text/xml
    text/plain
    text/x-component
    application/javascript
    application/json
    application/xml
    application/rss+xml
    font/truetype
    font/opentype
    application/vnd.ms-fontobject
    image/svg+xml;

    gzip_static on;

    gzip_proxied expired no-cache no-store private auth;
    gzip_disable "MSIE [1-6]\.";
    gzip_vary on;

	log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
					  
    access_log  logs/access.log  main;
    access_log off;

    client_max_body_size 20m;
    server {

        listen 4600;

        server_name platform_style;
		
		proxy_buffer_size 1024k;
		proxy_buffers 16 1024k;
		proxy_busy_buffers_size 2048k;
		proxy_temp_file_write_size 2048k;
		
      
        location /platform/ {

            proxy_pass http://127.0.0.1:5000/platform/;
        }
		
		## location /stat {
		## 
        ##     rtmp_stat all;
        ##     rtmp_stat_stylesheet stat.xsl;
        ## }

        location /stat.xsl {

            root html;
        }

        location /hls {

            #server hls fragments
            types {

                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
            }
            alias temp/hls;
            expires -1;
        }
		
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {

            root html;
        }
        	
    }
	

}

unquote parameters

for example load_module "modules/ngx_stream_module.so";

we will get a quoted string now

{
        "Block": null,
        "Name": "load_module",
        "Parameters": [
            "\"modules/ngx_stream_module.so\""
        ]
}

Dumper does not preserve the formatting in Lua blocks.

          if i have a block like this：

         set_by_lua_block $file_name {
                local t = ngx.var.uri
                local query = string.find(t, "?", 1)
                if query ~= nil then
                   t = string.sub(t, 1, query-1)
                end
                return t;
            }

then the code gonginx.DumpBlock(conf.Block, gonginx.IndentStyle) will dump this result：

   set_by_lua_block {


                            local t = ngx.var.uri local query = string.find(t, "?" , 1) if query ~= nil then t = string.sub(t, 1, query-1) end return t;


            }

it's worse when dump it several times @tufanbarisyildirim

Originally posted by @FeiYing9 in #20 (comment)

is this project given up?

[feature request]enhanced server object capabilities for easier access and modification of nginx configuration

propose to enhance the server object by adding new attributes and methods that will simplify the access and modification of its listening configuration, location details, and other pertinent settings. The following are potential enhancements and features that could be incorporated：

// get server by servername
server := conf.GetBlock(*.config.HTTP).GetServerByServerName("example.com")

// get all listening ports for the server
listenPorts := server.GetListenPorts()

// change the first listen port configuration to 8080
server.Listens[0].SetListenPort(8080)

// retrieve all location configurations for the server
locations := server.GetLocations()

Please consider implementing these functionalities to streamline working with complex server configurations and encourage more robust and dynamic management of Nginx setup through this library.

Let's discuss the feasibility and potential implementation details for these enhancements under this issue thread.

panic: unknown directive '''' on line 32, column 2

https://github.com/tufanbarisyildirim/gonginx/blob/master/examples/formatting/main.go
运行报错了
panic: unknown directive '''' on line 32, column 2

类似报错的还有
案例
location /aaa/ {
limit_rate 10M;
check_status;
}
报错unknown directive 'check_status' on line 256, column 17

案例
map $http_x_forwarded_for $clientRealIp
{
"" $remote_addr;
~^(?P[0-9.]+),?.*$ $firstAddr;
}
报错unknown directive '""' on line 2047, column 5

案例upstream www.abc.com{
server 1.2.3.4:8080;
check interval=5000 rise=1 fall=2 timeout=1000;
}

报错unknown directive 'check' on line 481, column 2

> ${var} breaks the parser.

Looks like ${var} can break the parser.

         > Hi @pksml85 can I have your example config file to improve [gonginx](https://github.com/tufanbarisyildirim/gonginx)?

Hi @tufanbarisyildirim. Sorry for the delay... but here is the example. GoNginx destroys my location blocks.

Here is my original config code and here is the mangled code from GoNginx

Originally posted by @pksml85 in 0xJacky/nginx-ui#73 (comment)

example request

Hi tufanbarisyildirim
Would you give an example? How to read and write http or server nodes. I am a beginner.

unknown directive 'application/octet-stream'

Hello,

When I add this into the nginx configuration in server block, the parser gives me unknown directive error:

    types {
        application/octet-stream mp4;
        application/octet-stream srt;
        application/octet-stream ass;
    }

exact error message: unknown directive 'application/octet-stream' on line 10, column 9

Is there anyway to skip this or even better make it parse this?

Thank you in advance.

set_by_lua_block $wocao {} not support

panic now