np-guard/netpol-analyzer

A Golang library for analyzing k8s connectivity-configuration resources (a.k.a. network policies)

netpol-analyzer

About netpol-analyzer

This repo contains a Golang library and CLI for analyzing k8s connectivity-configuration resources (a.k.a. network policies).

CLI usage

Evaluate command

Evaluate if a specific connection allowed

Usage:
  k8snetpolicy evaluate [flags]

Aliases:
  evaluate, eval, check, allow

Examples:
  # Evaluate if a specific connection is allowed on given resources from dir path
  k8snetpolicy eval --dirpath ./resources_dir/ -s pod-1 -d pod-2 -p 80

  # Evaluate if a specific connection is allowed on a live k8s cluster
  k8snetpolicy eval -k ./kube/config -s pod-1 -d pod-2 -p 80

Flags:
      --destination-ip string          Destination (external) IP address
      --destination-namespace string   Destination pod namespace (default "default")
  -d, --destination-pod string         Destination pod name
  -p, --destination-port string        Destination port (name or number)
  -h, --help                           help for evaluate
      --protocol string                Protocol in use (tcp, udp, sctp) (default "tcp")
      --source-ip string               Source (external) IP address
  -n, --source-namespace string        Source pod namespace (default "default")
  -s, --source-pod string              Source pod name, required

Global Flags:
  -c, --context string      Kubernetes context to use when evaluating connections in a live cluster
      --dirpath string      Resources dir path when evaluating connections from a dir
      --fail                fail on the first encountered error
      --include-json        consider JSON manifests (in addition to YAML) when analyzing from dir
  -k, --kubeconfig string   Path and file to use for kubeconfig when evaluating connections in a live cluster
  -q, --quiet               Runs quietly, reports only severe errors and results
  -v, --verbose             Runs with more informative messages printed to log

List command

Lists all allowed connections based on the workloads, network policies, and Ingress/Route resources defined.

Connections between workload to itself are excluded from the output.

Usage:
  k8snetpolicy list [flags]

Examples:
  # Get list of allowed connections from resources dir path
  k8snetpolicy list --dirpath ./resources_dir/

  # Get list of allowed connections from live k8s cluster
  k8snetpolicy list -k ./kube/config

Flags:
  -f, --file string            Write output to specified file
      --focusworkload       Focus connections of specified workload in the output (supported formats: <workload-name>, <workload-namespace>/<workload-name>)
                            (to focus connections from Ingress/Route only, use `ingress-controller` as <workload-name>)
  -o, --output string       Required output format (txt, json, dot, csv, md) (default "txt")
  -h, --help   help for list

Global Flags:
  -c, --context string      Kubernetes context to use when evaluating connections in a live cluster
      --dirpath string      Resources dir path when evaluating connections from a dir
      --fail                fail on the first encountered error
      --include-json        consider JSON manifests (in addition to YAML) when analyzing from dir
  -k, --kubeconfig string   Path and file to use for kubeconfig when evaluating connections in a live cluster
  -q, --quiet               Runs quietly, reports only severe errors and results
  -v, --verbose             Runs with more informative messages printed to log

Diff command

Reports all differences in allowed connections between two different directories of YAML manifests.

Usage:
  k8snetpolicy diff [flags]

Examples:
  # Get list of different allowed connections between two resources dir paths
  k8snetpolicy diff --dir1 ./resources_dir/ --dir2 ./other_resources_dir/

Flags:
      --dir1  string  First resources dir path
      --dir2  string  Second resources dir path to be compared with the first dir path
  -f, --file string            Write output to specified file
  -o, --output string Required output format (txt, csv, md, dot) (default "txt")  
  -h, --help   help for diff

Global Flags:
  -c, --context string      Kubernetes context to use when evaluating connections in a live cluster
      --dirpath string      Resources dir path when evaluating connections from a dir
      --fail                fail on the first encountered error
      --include-json        consider JSON manifests (in addition to YAML) when analyzing from dir
  -k, --kubeconfig string   Path and file to use for kubeconfig when evaluating connections in a live cluster
  -q, --quiet               Runs quietly, reports only severe errors and results
  -v, --verbose             Runs with more informative messages printed to log  

Example outputs:

$ k8snetpolicy eval --dirpath tests/onlineboutique -s adservice-77d5cd745d-t8mx4 -d emailservice-54c7c5d9d-vp27n -p 80

default/adservice-77d5cd745d-t8mx4 => default/emailservice-54c7c5d9d-vp27n over tcp/80: false



$ k8snetpolicy list --dirpath tests/onlineboutique_workloads

0.0.0.0-255.255.255.255 => default/redis-cart[Deployment] : All Connections
default/checkoutservice[Deployment] => default/cartservice[Deployment] : TCP 7070
default/checkoutservice[Deployment] => default/currencyservice[Deployment] : TCP 7000
default/checkoutservice[Deployment] => default/emailservice[Deployment] : TCP 8080
default/checkoutservice[Deployment] => default/paymentservice[Deployment] : TCP 50051
default/checkoutservice[Deployment] => default/productcatalogservice[Deployment] : TCP 3550
default/checkoutservice[Deployment] => default/shippingservice[Deployment] : TCP 50051
default/frontend[Deployment] => default/adservice[Deployment] : TCP 9555
default/frontend[Deployment] => default/cartservice[Deployment] : TCP 7070
default/frontend[Deployment] => default/checkoutservice[Deployment] : TCP 5050
default/frontend[Deployment] => default/currencyservice[Deployment] : TCP 7000
default/frontend[Deployment] => default/productcatalogservice[Deployment] : TCP 3550
default/frontend[Deployment] => default/recommendationservice[Deployment] : TCP 8080
default/frontend[Deployment] => default/shippingservice[Deployment] : TCP 50051
default/loadgenerator[Deployment] => default/frontend[Deployment] : TCP 8080
default/recommendationservice[Deployment] => default/productcatalogservice[Deployment] : TCP 3550
default/redis-cart[Deployment] => 0.0.0.0-255.255.255.255 : All Connections



$ ./bin/k8snetpolicy diff --dir1 tests/onlineboutique_workloads --dir2 tests/onlineboutique_workloads_changed_netpols
Connectivity diff:
source: default/checkoutservice[Deployment], destination: default/cartservice[Deployment], dir1:  TCP 7070, dir2: TCP 8000, diff-type: changed
source: default/checkoutservice[Deployment], destination: default/emailservice[Deployment], dir1:  TCP 8080, dir2: TCP 8080,9555, diff-type: changed
source: default/cartservice[Deployment], destination: default/emailservice[Deployment], dir1:  No Connections, dir2: TCP 9555, diff-type: added
source: default/checkoutservice[Deployment], destination: default/adservice[Deployment], dir1:  No Connections, dir2: TCP 9555, diff-type: added
source: 128.0.0.0-255.255.255.255, destination: default/redis-cart[Deployment], dir1:  All Connections, dir2: No Connections, diff-type: removed
source: default/checkoutservice[Deployment], destination: default/currencyservice[Deployment], dir1:  TCP 7000, dir2: No Connections, diff-type: removed
source: default/frontend[Deployment], destination: default/adservice[Deployment], dir1:  TCP 9555, dir2: No Connections, diff-type: removed
source: default/redis-cart[Deployment], destination: 0.0.0.0-255.255.255.255, dir1:  All Connections, dir2: No Connections, diff-type: removed

Additional details about the connectivity analysis and its output is specified here.

Additional details about the connectivity diff command and its output is specified here.

Build the project

Make sure you have golang 1.21+ on your platform

git clone [email protected]:np-guard/netpol-analyzer.git
cd netpol-analyzer
make mod 
make build

Test your build by running ./bin/k8snetpolicy -h.

netpol-analyzer's Issues

add semantic-diff cmd

allow connectivity-diff between two directories containing workloads and netpols manifests.

extend computation of allowed connections for ip-blocks

Currently a src/dst peer can be an ip address, but not an ip-block.
Need to extend support for ip-block (cidr), assuming it is a disjoint ip-block.

list: create a package with exported function for connections map output

The list cli command can use this function.

share CLI options

error handling updates

change to warning instead of error the case where there are no network policies

error handling improvements

add to ConnlistAnalyzer errors the errors produced from other packages besides scanner which are already added, even if they are fatal errors.
Define the relevant type of errors to be added.

add workflows

• dependabot
• codeql-analysis
• make release
• tests
• consider linter settings

GetDisjointIPBlocks should be exported?!(TBD)

Should GetDisjointIPBlocks func in eval/resources.go be exported?

tests improvements

• use assertion package consistently for all unit tests
• convert to table-driven tests
• add function to allow overriding/generate expected output
• avoid issues of line endings \n\r vs \n for output comparison
• on require/assert calls, add message with details about test name
• avoid hard-coded expected output as string vars in the code, instead use files with expected output, so that it is easier to update expected output. (see 3rd item)

packages to handle (by priority):

• connlist
• diff
• cmd

add list command

• support for connections map between pairs of pods, from dir path or live cluster
• extend to other resource types (deployments, replicasets, etc..)
• extend to include disjoint ip blocks

Add connlist output visualization as connectivity graph output

Support the dot format output

Introduce sub-commands for CLI

For example: full connectivity map or specific check

Enable caching of policy evaluation results

Coalesce Pods to their "owner" object

Resolving Pods to the owning objects (e.g., ReplicaSet, Deployment, StatefulSet, Job) reduces the number of objects stored and can also provide more meaningful results to users (e.g., traffic between Deployments and not Pods).
Note that there's a possibility for Pods to deviate from their owner (e.g., users or controllers changing labels on running pods) so we should be prepared to handle that.

Possible implementation changes:

• Define a new Workload struct that subsumes the current role played by Pods. The struct may also have a variant attribute to handle changes applied to Pod labels after creation from an "owner". The variant can be "encoded" via sha(pod.labels).
• Add map of Pod name to Workload (direct pointer or via a key: owner-ns/owner-name/variant)
• Evaluate policies against workloads, not Pods

This seems to be (partially?) already implemented by pkg/eval.

Introduce support for (cluster scoped) AdminNetworkPolicy API.

k8s is adding support for cluster scoped NetworkPolicy objects see here and here. Note that there are both Admin and Baseline objects to be accounted for.

Explore how to enable these in the policy engine.

caching by pod owner workload - consider possible change in pod's named port w.r.t its owner

refactor cmd package files to eliminate use of init func

Improve README.md

• Better explain the capabilities this code offers
• Include Usage section, showing how to run from command-line
• Include examples of outputs
• Include Build section with details how to build the project
• Provide some details how to consume as a Go library

Remove global policy engine state to an object

Change also applies to global functions becoming receiver based

default vs unspecified namespaces

Kubernetes YAML can leave the namespace unspecified. In this case, kubectl applies the namespace from the kubectl config.

k8snetpolicy list sets the namespace to "default", (metav1.NamespaceDefault), if the namespace is blank.

This means k8snetpolicy list gives the same output when the input contains namespace: "default" or leaves the namespace off. I found this confusing. Some other tools I use preserved the blank namespace.

It might be worthwhile to treat "" like a valid real namespace, and preserve it, even though we don't know what it will be.
(Alternately it might be good to pick up up from the kubectl context. I'd prefer a blank namespace in the output, though.)

IPBlock: support ipv6

• consider representing ip-block peers based on net/netip . Only use ranges for computation of disjoint ip-blocks.
• support computation of disjoint ip-blocks for ipv6 as well.

add more tests

change connlist.FromDir returned type

Instead of returning a string of the connectivity map, return a slice of connection objects.

change Peer type to an interface

Pod and IPBlock are mutually exclusive - that is, only one or the other can be set, not both
change to Peer interface, implemented for either mode

add verbosity level support

add option for verbosity level, trough flags -q (quiet) and -v (verbose)

usage appears after executing errors

on tests/bad_netpols , the output is:

> Error: network policy default/shippingservice-netpol named port error: cannot convert named port for an IP destination
> Usage:
>   k8snetpolicy list [flags]

> Examples:
>   # Get list of allowed connections from resources dir path
>   k8snetpolicy list --dirpath ./resources_dir/
> 
>   # Get list of allowed connections from live k8s cluster
>   k8snetpolicy list -k ./kube/config
> 
> Flags:
>       --focusworkload string   Focus connections of specified workload name in the output
>   -h, --help                   help for list
>   -o, --output string          Required output format (txt,json,dot,csv,md) (default "txt")
> 
> Global Flags:
>   -c, --context string      Kubernetes context to use when evaluating connections in a live cluster
>       --dirpath string      Resources dir path when evaluating connections from a dir
>   -k, --kubeconfig string   Path and file to use for kubeconfig when evaluating connections in a live cluster 

why do we get usage info when the command was valid? (used ./bin/k8snetpolicy list --dirpath tests/bad_netpols )?

Support md, CSV and dot output formats

• md
• csv
• dot

netpolErr : "named port is not defined in a selected workload"

consider changing this netpol-err (raised by : ConvertPodNamedPort) into warning

It is legal case that the selected workload may not contain the required named-port specified in the netpol.

connlist: add option to get resources from multiple dirs

idea: 'k8snetpolicy list --output json'

I would like to get the list of allowed connections in a JSON format.

Do not warn about OpenShift resources as not being a k8s resource

example warning:
WARN: Yaml document is not a K8s resource: no kind "Route" is registered for version "route.openshift.io/v1" in scheme "pkg/runtime/scheme.go:100" in file: ..., document: 0

connlist required changes

• Peer2PeerConnection: change into an interface, and do not expose the internal fields.
  Src/Dst: create another interface called Peer, with 3 methods: getName, getNamesmace, getIP, so it is clear for the consumer if it is a name/namespace combination or an ip address.
  Ports: change to PortRange, with startPort and endPort. When they are equal it represents a single port. When they are both equal to 0 it means that the port range is empty.

• Library input: currently it is FromDir and FromK8sCluster. should expect a list of YAML manifests. (extracted from dir or cluster). Separate to have one function that extracts the YAML, and another function that calls the PolicyEngine, with the YAML inputs (slice of strings, where each string is a YAML document).
  The function that should accept a list of YAML files should also be exported. (FromYAMLManifests).
  FromDir: add the option to add a custom walker. The consumer will provide a walker object (that can prevent scanning certain dirs for example).

Smaller comments:

• Add package documentation.
• context.TODO() : consider replacing with context.WithTimeout().
• ConnectionSet.GetProtocolsAndPortsMap() : return explicit type of port range instead of list of ports (which currently is always expected to have exactly two ports values).
• consistent slice/map initialization.
• new line separator '\n' : should be tested on all systems. Can instead use a constant newline separator of a standard library.
• Output printing : do not use %v for printing numbers, be more specific with %d.
• separator const var: use a more specific name, such as portRangeSeparator

change netpols map

For netpolsMap - use a map[name]*NetworkPolicy instead of []*NetworkPolicy for easier management and lookup

Allow incremental policy engine state updates

Currently the engine's world view is set once, in its entirety, and does not change afterwards.
In an active cluster, we should be able to update state (i.e., insert, update and delete objects) at runtime, tracking the cluster's state.
The suggestion is to replace eval.SetResources of all resources with eval.Upsert and eval.Delete of individual resources.

simplify connlist output: remove trivial connectivity lines when possible

Connections from pod to itself can be removed from the report.
Connection from deployment to itself is considered trivial only if the deployment has one replica.

allow querying policies against "higher level" constructs such as Deployments, not only Pods

Add option to subset connlist output

Reorganize internal packages

• For example, IPBlock should not be within k8s package

• consider moving some of the k8s.Peer-related code in eval/check.go to internal/k8s/peer.go, e.g., isPodToItself() and isPeerNodeIP(). Ideally, eval should work on a more abstract level.

• some of the files (e.g., ipBlock.go) are missing the license header.

Consider if/how to handle Service object as destination

Selected Pods may have arbitrary labels in addition to the Service selector labels, so need to find if this can be done without scanning all matching Pods and evaluating all policies against them.

Support NetworkPolicyList resources (and maybe other list resources)

Document exported symbols

add support for Route and Ingress resources

connlist command should consider connectivity inferred from Route and Ingress resources as well, combined with network policy connectivity analysis.
For example, consider the Route below:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: hello-world
  namespace: helloworld
spec:
  port:
    targetPort: 8000
  to:
    kind: Service
    name: hello-world
    weight: 100

The report of connlist may contain a line as follows:
{ingress-controller} => hello-world : TCP 8080

But, if network policy analysis identifies that such connectivity is not permitted, this line will not be added to the report.

Release as a kubectl plugin

Follow Krew's instruction to make a multi-platform distribution.
See this example.

Run cmd/netpolicy executable against an active cluster

Load the relevant state from the active cluster before calling for policy evaluation

connlist output: explain ip-range in connectivity output

IP-blocks in connlist output are considered as having allowed connectivity only in one direction (ingress or egress).
A connection line between two workloads does mean the both ingress and egress directions are allowed.
Explain it in the documentation, about this interpretation of the output.

Move directory scanning functionality to its own package

Retain the eval package as decoupled from the sources of object definitions (e.g., directory with YAML files, active cluster, mocked up testing code, ...)

A flag to write output to file

probably only required for the list subcommand.

PodFromCoreObject required fix

PodFromCoreObject should not always issue an error / ignore pods not scheduled or not assigned IP addresses.
Only on live cluster mode this is relevant, as opposed to handling pod resources from dir.

consider scan pkg changes

Compare with scanning functionality from cluster-topology-analyzer, consider the requirements for resource scanning in both repos, and consume the scanning functions from one package to avoid duplication.

handle cache with concurrent access

The cache is not currently go-routine safe from concurrent access. We'll need it for live policy checks on connect,