mrbaseman/parse_yaml

a simple yaml parser implemented in bash

parse_yaml's Issues

wrong results if variables are set

parse_yaml produces wrong results if a variable i is set to a value in current bash session.

Reproduce:

1. set value to variable i: $ i="ZZZ test string ZZZ"
2. echo $(parse_yaml path/to/valid/yaml.yaml)

The output contains strings with of the initial contents of the variable $i of the current bash session.

An approach for solving this issue could be inside the parse_yaml function to back up the current value of $i in a temporary variable, clear the $i variable, run the function, and then write back the backup to the variable.
Or, just don't care about the contents of $i (since its use case is 99% a temporary loop variable) and clear it at the beginning of the function.

Bug: sed: 2: "s|^[[:space:]]*#.*||;s| ...": undefined label '1;t;:1;s|^[[:space:]]*$||;t2;p;:2;d'

I found a bug :)

But I have no idea how to fix it :/

This fails on the following input:

$ ./parse_yaml.sh user-data
sed: 2: "s|^[[:space:]]*#.*||;s| ...": undefined label '1;t;:1;s|^[[:space:]]*$||;t2;p;:2;d'

$ cat user-data
hostname: ulinux-test
manage_etc_hosts: true
user: root
disable_root: False
password: $5$SXUfZAMG$IPWDcNbmgl1A91OybwYZQcDhg2V4e0noOxojBt5z/78
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo9UKNJ6X+LUxY9E2Ne3IxluaBTQWE0nCgyVfPHgSJhrI1HDiKcwc2S10kzMOnWco47sOb0Te4/tOHh+jlBbSTiW2CqQpIkZ8Bz4d1VXIPpUCUreP8gSuWxWNrdiU3TpfaeLSn0BkQhssvUhooAVPcRlgpmLETeGVlY1G2Qc1YcrQusz5E/5vLp4wkycZnGvHJ9+R5M2gn12Wnj+Fg8xMdqsvKmtBasPTX9YcJxVj/CBJZUB5impNg7ml9mESGDOpU11H9UWLUiDjIOmiDu3baNUiUmUnrUG44HBnstvbglEEOjsywML9xB81iZkA8RPluFpMU/+BWPI3nIKyRZn2p [email protected]
chpasswd:
  expire: False
package_upgrade: true

parse_yaml script is not working or identifying if the yamlfile has any issues

We are using Linux machine of Red Hat Enterprise Linux Server release 7.9 (Maipo) and Ubuntu 18. We have noticed some strange issue when using the parse-yaml.sh script is that, when we preform source parse-yaml.sh , and after that when executing some parse command such as parse-yaml test.yaml, the command will be hung for long time without giving any output.

By enabling the debug, I could understand that its getting stuck in + cat dev-policy.yaml, i

-bash-4.2$ parse_yaml dev-policy.yaml policy
+ parse_yaml dev-policy.yaml policy
+ unset i
+ unset fs
+ local prefix=policy
+ local separator=_
+ local indexfix=-1
+ grep --color=auto -q 'GNU Awk'
+ awk --version
+ indexfix=-1
++ echo @
++ tr @ '\034'
+ local 's=[[:space:]]*' 'sm=[ \t]*' 'w=[a-zA-Z0-9_.]*' $'fs=\034' 'i=  '
+ awk $'-F\034' '{multi=0;
        if(match($0,/[ \t]*\|[ \t]*$/)){multi=1; sub(/[ \t]*\|[ \t]*$/,"");}
        if(match($0,/[ \t]*>[ \t]*$/)){multi=2; sub(/[ \t]*>[ \t]*$/,"");}
        while(multi>0){
            str=$0; gsub(/^[ \t]*/,"", str);
            indent=index($0,str);
            indentstr=substr($0, 0, indent+-1) "  ";
            obuf=$0;
            getline;
            while(index($0,indentstr)){
                obuf=obuf substr($0, length(indentstr)+1);
                if (multi==1){obuf=obuf "\\n";}
                if (multi==2){
                    if(match($0,/^[ \t]*$/))
                        obuf=obuf "\\n";
                        else obuf=obuf " ";
                }
                getline;
            }
            sub(/[ \t]*$/,"",obuf);
            print obuf;
            multi=0;
            if(match($0,/[ \t]*\|[ \t]*$/)){multi=1; sub(/[ \t]*\|[ \t]*$/,"");}
            if(match($0,/[ \t]*>[ \t]*$/)){multi=2; sub(/[ \t]*>[ \t]*$/,"");}
        }
    print}'
+ sed -ne 's|,[[:space:]]*\]|]|g' -e :1 -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(&[a-zA-Z0-9_.]*\)[[:space:]]*\[[[:space:]]*\(.*\)[[:space:]]*,[[:space:]]*\(.*\)[[:space:]]*\]|\1\2: \3[\4]\n\1  - \5|;t 1' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(&[a-zA-Z0-9_.]*\)[[:space:]]*\[[[:space:]]*\(.*\)[[:space:]]*\]|\1\2: \3\n\1  - \4|;' -e :2 -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\[[[:space:]]*\(.*\)[[:space:]]*,[[:space:]]*\(.*\)[[:space:]]*\]|\1\2: [\3]\n\1  - \4|;t 2' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\[[[:space:]]*\(.*\)[[:space:]]*\]|\1\2:\n\1  - \3|;' -e :3 -e 's|^\([[:space:]]*\)-[[:space:]]*\[[[:space:]]*\(.*\)[[:space:]]*,[[:space:]]*\(.*\)[[:space:]]*\]|\1- [\2]\n\1  - \3|;t 3' -e 's|^\([[:space:]]*\)-[[:space:]]*\[[[:space:]]*\(.*\)[[:space:]]*\]|\1-\n\1  - \2|;p'
+ sed -ne 's|^\([[:space:]]*\):|\1|' -e 's|^\([[:space:]]*\)\(---\)\([[:space:]]*\)||' -e 's|^\([[:space:]]*\)\(\.\.\.\)\([[:space:]]*\)||' -e 's|^\([[:space:]]*\)-[[:space:]]*["'\'']\(.*\)["'\''][[:space:]]*$|\1\2|p;t' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*["'\'']\(.*\)["'\''][[:space:]]*$|\1\2\3|p;t' -e 's|^\([[:space:]]*\)-[[:space:]]*\(.*\)[[:space:]]*$|\1\2|' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*["'\'']\?\(.*\)[[:space:]]*$|\1\2\3|' -e 's|^\([[:space:]]*\)["'\'']\?\([^&][^]\+\)["'\''][[:space:]]*$|\1\2|' -e 's|^\([[:space:]]*\)["'\'']\?\([^&][^]\+\)[[:space:]]*$|\1\2|' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*["'\'']\(.*\)[[:space:]]*$|\1\2\3|' -e 's|^\([[:space:]]*\)["'\'']\([^&][^]*\)["'\''][[:space:]]*$|\1\2|' -e 's|^\([[:space:]]*\)["'\'']\([^&][^]*\)[[:space:]]*$|\1\2|' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(.*\)[[:space:]]*$|\1\2\3|' -e 's|^\([[:space:]]*\)\([^&][^]*\)["'\''][[:space:]]*$|\1\2|' -e 's|^\([[:space:]]*\)\([^&][^]*\)[[:space:]]*$|\1\2|' -e 's|[[:space:]]*$||p'
+ awk $'-F\034' '{
        gsub(/\t/,"        ",$1);
        if(NF>3){if(value!=""){value = value " ";}value = value  $4;}
        else {
            if(match($1,/^&/)){anchor[substr($1,2)]=full_vn;getline};
            indent = length($1)/length("  ");
            vname[indent] = $2;
            value= $3;
            for (i in vname) {if (i > indent) {delete vname[i]; idx[i]=0}}
            if(length($2)== 0){  vname[indent]= ++idx[indent] };
            vn=""; for (i=0; i<indent; i++) { vn=(vn)(vname[i])("_")}
            vn="policy" vn;
            full_vn=vn vname[indent];
            if(vn=="policy")vn="policy_";
            if(vn=="_")vn="__";
        }
        gsub(/\./,"_",full_vn);
        gsub(/\\"/,"\"",value);
        gsub(/'\''/,"'\''\"'\''\"'\''",value);
        assignment[full_vn]=value;
        if(!match(assignment[vn], full_vn))assignment[vn]=assignment[vn] " " full_vn;
        if(match(value,/^\*/)){
            ref=anchor[substr(value,2)];
            if(length(ref)==0){
                printf("%s='\''%s'\''\n", full_vn, value);
            } else {
                for(val in assignment){
                    if((length(ref)>0)&&index(val, ref)==1){
                        tmpval=assignment[val];
                        sub(ref,full_vn,val);
                        if(match(val,"_$")){
                            gsub(ref,full_vn,tmpval);
                        } else if (length(tmpval) > 0) {
                            printf("%s='\''%s'\''\n", val, tmpval);
                        }
                        assignment[val]=tmpval;
                    }
                }
            }
        } else if (length(value) > 0) {
            printf("%s='\''%s'\''\n", full_vn, value);
        }
    }END{
        for(val in assignment){
            if(match(val,"_$"))
                printf("%s='\''%s'\''\n", val, assignment[val]);
        }
    }'
+ sed -ne 's|,[[:space:]]*}|}|g' -e :1 -e 's|^\([[:space:]]*\)-[[:space:]]*{[[:space:]]*\(.*\)[[:space:]]*,[[:space:]]*\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(.*\)[[:space:]]*}|\1- {\2}\n\1  \3: \4|;t 1' -e 's|^\([[:space:]]*\)-[[:space:]]*{[[:space:]]*\(.*\)[[:space:]]*}|\1-\n\1  \2|;' -e :2 -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(&[a-zA-Z0-9_.]*\)[[:space:]]*{[[:space:]]*\(.*\)[[:space:]]*,[[:space:]]*\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(.*\)[[:space:]]*}|\1\2: \3 {\4}\n\1  \5: \6|;t 2' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(&[a-zA-Z0-9_.]*\)[[:space:]]*{[[:space:]]*\(.*\)[[:space:]]*}|\1\2: \3\n\1  \4|;' -e :3 -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*{[[:space:]]*\(.*\)[[:space:]]*,[[:space:]]*\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(.*\)[[:space:]]*}|\1\2: {\3}\n\1  \4: \5|;t 3' -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*{[[:space:]]*\(.*\)[[:space:]]*}|\1\2:\n\1  \3|;p'
+ sed -e 's|^\([[:space:]]*\)?|\1-|' -ne 's|^\([[:space:]]*\)-[[:space:]]*\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(.*\)|\1-\n\1 \2: \3|' -ne 's|^[[:space:]]*#.*||;s|[[:space:]]*#[^"'\'']*$||;s|^\([^"'\''#]*\)#.*|\1|;t 1' -ne t -ne :1 -ne 's|^[[:space:]]*$||;t 2' -ne p -ne :2 -ne d
+ sed -e 's|^\([[:space:]]*\)\([a-zA-Z0-9_.]*\)[[:space:]]*:[[:space:]]*\(&[a-zA-Z0-9_.]*\)\(.*\)|\1\2:\4\n\3|' -e 's|^\([[:space:]]*\)-[[:space:]]*\(&[a-zA-Z0-9_.]*\)\(.*\)|\1- \3\n\2|'
+ cat dev-policy.yaml

Here the dev-policy.yaml file is as below.

---
- apiname: myapi
  scope: api
  inboundsession: >
    <base />

    <choose>
        <when condition="@(context.Request.Url.Path.Contains("xxxxxxx))">
            <rate-limit-by-key calls="xxx" renewal-period="xx" counter-key="@(Regex.Match(context.Request.Headers.GetValueOrDefault("X-xxxxxxx-For",""), @"^[.0-9]*")?.Value)" />
        </when>
        <otherwise>
            <check-header name="X-AssignedxxxxID" failed-check-httpcode="400" failed-check-error-message="Bad request" ignore-case="true" />
            <rate-limit-by-key calls="xxxx" renewal-period="xx" counter-key="@(Regex.Match(context.Request.Headers.GetValueOrDefault("X-xxxxx-For",""), @"^[.0-9]*")?.Value)" />
            <rate-limit-by-key calls="xxx" renewal-period="xx" counter-key="@(context.Request.Headers.GetValueOrDefault("X-AssignedxxxxxxID",""))" />
        </otherwise>
    </choose>
  outboundsession: |
    <base />
  backendsession: |
    <base />
  onerrorsession: >
    <base />

    <choose>
        <when condition="@(context.LastError.Reason == "RateLimitExceeded")">
            <emit-metric name="policy_error_rate_limit_exceeded" value="1">
                <dimension name="ClientIP" value="@(context.Request.Headers.GetValueOrDefault("xxxxxx","empty-ip"))" />
                <dimension name="AssignedDeviceID" value="@(context.Request.Headers.GetValueOrDefault("X-AssignedxxxxxID","empty-xxxxx-id"))" />
                <dimension name="xxxx ID" />
            </emit-metric>
        </when>
    </choose>
- operationname: myapi
  operationapi: myapi
  scope: operation
  inboundsession: |
    <base />
  outboundsession: |
    <base />
  backendsession: |
    <base />

but when i create a fresh dev-policy.yaml file from scratch within the server itself, its parsing properly. Not sure how to fix this issue. Is this because of invalid input yaml file? if yes, how we can make correct validation before processing it ?

Parseing YAML issue

Hi, Thanks for your wonderful function! I have the following yaml

---
tools:
  - name: fly
    version_command: "fly --version"
    minimum_version: "5.6"
  - name: docker
    version_command: "docker --version | awk -F'[, ]' '{print $3}' | awk -F'[-]' '{print $1}'"
    minimum_version: "24.0"

When I parse the docker version command the first awk command loses some info. In particular the comma and the following space. Any way to work around this?

docker --version | awk -F'[]' '{print $3}' | awk -F'[-]' '{print $1}'

Escaped ampersand is not a known regexp operator

Error:

awk: cmd. line:5: warning: regexp escape sequence `\&' is not a known regexp operator

System details:

$ uname -a
Linux host 5.6.11-arch1-1 #1 SMP PREEMPT Wed, 06 May 2020 17:32:37 +0000 x86_64 GNU/Linux
$ awk --version
GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)

Fix is to remove the \ from \& on line 5 of the awk command:

        if(match(\$1,/^&/)){anchor[substr(\$1,2)]=full_vn;getline};

explicit about BSD macOS support?

This explicitly detects GNU awk and mawk, but it doesn't say anything about BSD style awk and sed.

Are these believed to work out of the box?

Perhaps something on the readme.md mentioning the position on these.

The code could include a warning and a recommendation to install gawk (GNU awk on macOS) and gsed, and use those if available.

I may be willing to contribute improvements to do this.

double separator

Hello,

Your script is very useful !

I don't know why but sometimes your script generates a double separator. It's very diffuclt for me to understand why because the same file can has single separator ou double.

Do you know this problem ?

yaml file :

table:
    stripes: even
    caption_prefix: Table
code:
    linenums: false
    theme: monokai_sublime
toc:
    display: true
    levels: 1
    title: Table OF Contents
    align: right
image:
    caption_prefix: Figure
heading:
    numbered: false
    chapter_prefix: Chapter
output:
    prefix: ~

the call :

parse_yaml config.yml "dac_" "_"

the result :

dac_table__stripes="even"
dac_table__caption_prefix="Table"
dac_code__linenums="false"
dac_code__theme="monokai_sublime"
dac_toc__display="true"
dac_toc__levels="1"
dac_toc__title="Table OF Contents"
dac_toc__align="right"
dac_image__caption_prefix="Figure"
dac_heading__numbered="false"
dac_heading__chapter_prefix="Chapter"
dac_output__prefix="~"
dac__=" dac_table dac_code dac_toc dac_image dac_heading dac_output"
dac_output__=" dac_output__prefix"
dac_toc__=" dac_toc__display dac_toc__levels dac_toc__title dac_toc__align"
dac_heading__=" dac_heading__numbered dac_heading__chapter_prefix"
dac_image__=" dac_image__caption_prefix"
dac_code__=" dac_code__linenums dac_code__theme"
dac_table__=" dac_table__stripes dac_table__caption_prefix"

Array parsing isn't working as expected

@mrbaseman

I'm using exactly the same example showed in README.md for this test.

This is the file:

---
global:
  input:
    - "main.c"
    - "main.h"
  flags: [ "-O3", "-fpic" ]
  sample_input:
    -  { property1: value1, property2: value2 }
    -  { property1: "value 3", property2: 'value 4' }
  licence: |
    this is published under
    open source license
    in the hope that it would 
    be useful
...

This is the result:

global__input__1="main.c"
global__input__2="main.h"
global__flags="1- "-O3"
global__flags="1- "-O3 1- "-fpic"
global__sample_input__1property1="value1" 
global__sample_input__1property2="value2"
global__sample_input__1property1="value 3"
global__sample_input__1property2="value 4"
global__licence="this is published under"
global__licence="this is published under open source license"
global__licence="this is published under open source license in the hope that it would"
global__licence="this is published under open source license in the hope that it would be useful"
__=" global"
global__=" global__input global__flags global__sample_input global__licence"
global__input__=" global__input__1 global__input__2"
global__sample_input__=" global__sample_input__1 global__sample_input__1property1 global__sample_input__1property2 global__sample_input__2"

These lines:

global__sample_input__1property1="value1" 
global__sample_input__1property2="value2"
global__sample_input__1property1="value 3"
global__sample_input__1property2="value 4"

Should be like bellow, not like above

global_sample_input_1_property1="value1"
global_sample_input_1_property2="value2"
global_sample_input_2_property1="value 3"
global_sample_input_2_property2="value 4"

Tags with dots are not parsed correctly

Parsing the sample:

key.withdots: blue
another.key: yellow
key: red

results to:

="key.withdots: blue"
="key.withdots: blue another.key: yellow"
key="red"
__=" key"

no output

I apologize if this is the wrong place for this. I cloned the repo and was just trying to use the sample sample.yml but I'm not getting any output when I run parse_yaml.sh. I am running CentOS Linux release 8.4.2105. Are there any package dependencies or anything special I need to do? Thanks!

improve handling of nested quotes

multiple single quotes aren't processed properly 'something ''inside'' a string' becomes something ''inside'' a string instead of something 'inside' a string

see #6 (comment)

Duplicate BASH script to POSIX shell.

bash is not a portable shell, unlike sh, so it'd be nice if the script could run on just the bourne shell.

Incompatible with mawk 1.3.3

mawk 1.3.3 doesn't support [[:space:]] character class: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=65617 and there are also subtle differences in index() function newlline character handling.

Incorrect parsing of some lists

I've experienced an incorrect parsing of list items.

This yaml produces the variables below, in particular:

• api_plan_keyless_resource_filtering_whitelist_1='path: /swagger'
  whereas I expected
• api_plan_keyless_resource_filtering_whitelist_1_path=/swagger

api:
  name: be-dotnet-01
  add_env_prefix: false
  version: v1
  enabled: true
  plan:
    keyless:
      enabled: true
      resource_filtering:
        enable: true
        whitelist:
          - path: "/swagger"
            methods: ["GET", "PUT"]
    jwt:
      enabled: false
    apikey:
      enabled: false

...
api_plan_=' api_plan_keyless api_plan_jwt api_plan_apikey'
api_plan_keyless_=' api_plan_keyless_enabled api_plan_keyless_resource_filtering'
api_plan_keyless_enabled=true
api_plan_keyless_resource_filtering_=' api_plan_keyless_resource_filtering_enable api_plan_keyless_resource_filtering_whitelist'
api_plan_keyless_resource_filtering_enable=true
api_plan_keyless_resource_filtering_whitelist_=' api_plan_keyless_resource_filtering_whitelist_1'
api_plan_keyless_resource_filtering_whitelist_1='path: /swagger'
api_plan_keyless_resource_filtering_whitelist_1_=' api_plan_keyless_resource_filtering_whitelist_1_methods'
api_plan_keyless_resource_filtering_whitelist_1_methods_=' api_plan_keyless_resource_filtering_whitelist_1_methods_1 api_plan_keyless_resource_filtering_whitelist_1_methods_2'
api_plan_keyless_resource_filtering_whitelist_1_methods_1=GET
api_plan_keyless_resource_filtering_whitelist_1_methods_2=DELETE
...

The following yaml, instead, seems to be parsed fine:

api:
  name: be-dotnet-01
  add_env_prefix: false
  version: v1
  enabled: true
  plan:
    keyless:
      enabled: true
      resource_filtering:
        enable: true
        whitelist:
          -
            path: "/swagger"
            methods: ["GET", "DELETE"]
    jwt:
      enabled: false
    apikey:
      enabled: false

...
api_plan_=' api_plan_keyless api_plan_jwt api_plan_apikey'
api_plan_keyless_=' api_plan_keyless_enabled api_plan_keyless_resource_filtering'
api_plan_keyless_enabled=true
api_plan_keyless_resource_filtering_=' api_plan_keyless_resource_filtering_enable api_plan_keyless_resource_filtering_whitelist'
api_plan_keyless_resource_filtering_enable=true
api_plan_keyless_resource_filtering_whitelist_=' api_plan_keyless_resource_filtering_whitelist_1'
api_plan_keyless_resource_filtering_whitelist_1_=' api_plan_keyless_resource_filtering_whitelist_1_path api_plan_keyless_resource_filtering_whitelist_1_methods'
api_plan_keyless_resource_filtering_whitelist_1_methods_=' api_plan_keyless_resource_filtering_whitelist_1_methods_1 api_plan_keyless_resource_filtering_whitelist_1_methods_2'
api_plan_keyless_resource_filtering_whitelist_1_methods_1=GET
api_plan_keyless_resource_filtering_whitelist_1_methods_2=DELETE
api_plan_keyless_resource_filtering_whitelist_1_path=/swagger
...

How I ran the script (in the bash shell):

• source parse_yaml.sh
• eval $(parse_yaml api.yaml)
• set -o posix ; set | grep keyless

convert parsed yaml file outputs to azurepipeline variables

We tried to use this script to parse our Azurepolicy yaml inputs and need to process these parsed output further in our azurepipeline to have various policy file creation depend on these variable generated. We need to have many conditions in our pipeline task further to create a custom policy file based on the inputs given for (polcyname, type, session etc..). But by using the script, we are not getting a way to have conditions applied dynamically based on the variables generated.

So in brief , our issue is how we can enable the conditions for these generated outputs ( there might be one more policy for each apps, and this conditions need to be applied dynamically) or use all the generated outputs to pipeline variable to use them in further in the task

################ Policy ################

- name: plicyA
  session: 
     - inbound
	 - backend
  scope: api
  apiname:
  customvalue1: xxxxx
  customvalue2: xxxxx
     

- name: policyB
  scope: operation
  operation:
      - operation1
	  - operation2
  session: 
     - inbound
	 - backend
  customvalue3: xxxxx
  customvalue4: xxxxx
  
etc.....................................

pipeline file is

stages:
- stage: check
  displayName: 'check'
  variables:
    subscription: 'xxxxxxxxxxxxxxx'
  pool:
      name: xxxxx
  jobs:
  - job: api
    displayName: 'api policy'
    variables:
    - group: check_api_policy
    workspace:
      clean: all
    pool:
      name: xxxxxxx
    steps:
    - bash: |      
         source $(System.DefaultWorkingDirectory)/scripts/parse_yaml.sh
         parse_yaml $(System.DefaultWorkingDirectory)/api-ops.yaml policy
         echo "*****************************************"
         for f in $policy_ ; do eval echo \$f \$${f} ; done
         echo "*****************************************"
		 
		 #######################################
		 convert all the above generated variables to azure pipeline variables
		 can we dynamically apply conditions based on the variables generated from above
		 #######################################
      name: Parse_yaml

Requirement

• convert all the above generated variables to azure pipeline variables

• can we dynamically apply conditions based on the variables generated from above

   eg: if policy is A is "x" , create a xml file for ipfilter
       if policy_session is inbound - create a xml file for :inbound"
       so like above  depending on the various policies generated outputs.. different 
       conditions..
  

.

Fail the parsing if the input yaml have syntax error

@mrbaseman Can we have a feature or way to fail the script parsing if the input file parsing didt work or yaml file input has wrong syntax.

awk: cmd. line:37: Call to undefined function

When cloning this repo, and running
source parse_yaml.sh
then
parse_yaml sample.yml

I get awk: cmd. line:37: Call to undefined function

Still seem to get all the values printing out correctly, but also getting this error.

possible bug in anchor parsing?

Test example:

---
MainSourceFile:  '/test/4.9.c'
Diagnostics:
  - DiagnosticName:  bf-4.9
    DiagnosticMessage:
      Message:         '''\0'' is assigned to a pointer. Probably meant: *ptr = ''\0''.'
      FilePath:        '/test/4.9.c'
      FileOffset:      429
      Replacements:
        - FilePath:        '/test/4.9.c'
          Offset:          421
          Length:          7
          ReplacementText: '*(tmp.ptr)'
  - DiagnosticName:  bf-4.9
    DiagnosticMessage:
      Message:         '''\0'' is assigned to a pointer. Probably meant: *ptr = ''\0''.'
      FilePath:        '/test/4.9.c'
      FileOffset:      499
      Replacements:
        - FilePath:        '/test/4.9.c'
          Offset:          488
          Length:          10
          ReplacementText: '*(param->ptr)'
...

The last sed would strip off quotes before passing into awk for anchor parsing, so ReplacementText's value becomes *(param->ptr) (no quote). I think the unquoting should be
done in the anchor parsing stage, right?

Style question - 3 spaces indent?

I noticed that the whole parse_yaml file uses 3 spaces to indent. Is this a deliberate style choice? It seems unusual. Why not 4?

I ran in it thru https://github.com/lovesegfault/beautysh which set it to 4 spaces indentation as its default.