appgate/sdp-operator

Kubernetes Operator for Appgate SDP

Welcome to our GitHub

Product specific repos

Appgate SDP

The Appgate SDP related repositories have a repository name which starts with sdp-. Please find the landing page for Appgate SDP here.

Appgate FP

The Appgate FP related repositories have a repository name which starts with fp-. Please find the landing page for AppGate FP here.

sdp-operator's Issues

Support k8s > 1.16

There are some changes when creating CRD:

1. API version v1beta1 has been deprecated
2. a json schema is required now for each CRD (we should be able to derive this automatically from the open spec API)

Support setting provider

Right now it is hardcoded to local.
We should add support for setting the provider, while keeping the default value of local.

Seems we dont log the errors produced when applying plan

This problem was referenced here #56 (comment)

Add support for controller certificate in appgate-operator-conf configmap

Right now we don't verify the server certificate which is obviously not OK for v1.

Add support for partial updates

Use case:

One team wants to manage a subset of the entities (for example to manage k8s entitlements) from github without affecting other entities (via a tag for example).

Right now we can not update partial entities because the dependency checker complains about missing entities that depend on the ones we want to manage.

Wrong order when applying plan

It tried to apply Policy before Site.

2020-09-24 07:41:27,382 [INFO] [appgate-operator/appgate-test] AppgatePlan Summary:
2020-09-24 07:41:27,383 [INFO] [appgate-operator/appgate-test] + <class 'appgate.openapi.parser.Policy'> simple_setup_POL [427d30cf-b4a9-4973-bd74-08df84d585e8]
2020-09-24 07:41:27,509 [ERROR] [aggpate-client] https://ec2-3-83-104-221.compute-1.amazonaws.com:444//admin//policies :: 422: {"id":"unprocessable entity","message":"Invalid reference to another entity"}
2020-09-24 07:41:27,509 [ERROR] [appgate-client] payload :: {'name': 'simple_setup_POL', 'expression': "if ( claims.user.username == 'bob' ) {\n return true;\n} \n\nreturn false;", 'tags': [], 'entitlements': ['9eb29d80-0bd2-47c9-b849-816962a81b19', 'd8f09f37-9401-4136-b9
69-95ebbb175c1c'], 'entitlementLinks': [], 'ringfenceRules': [], 'ringfenceRuleLinks': [], 'administrativeRoles': [], 'id': '427d30cf-b4a9-4973-bd74-08df84d585e8'}
2020-09-24 07:41:27,509 [ERROR] [aggpate-client] POST /admin//policies :: Expecting a response but we got empty data
2020-09-24 07:41:27,509 [INFO] [appgate-operator/appgate-test] + <class 'appgate.openapi.parser.Entitlement'> simple_setup_ENT_HTTP [d8f09f37-9401-4136-b969-95ebbb175c1c]
2020-09-24 07:41:27,632 [ERROR] [aggpate-client] https://ec2-3-83-104-221.compute-1.amazonaws.com:444//admin//entitlements :: 422: {"id":"unprocessable entity","message":"Validation failed.","errors":[{"field":"site","message":"does not exist"}]}
2020-09-24 07:41:27,633 [ERROR] [appgate-client] payload :: {'name': 'simple_setup_ENT_HTTP', 'site': '3d844467-8d82-42d2-946a-53b3d8ff11ec', 'tags': [], 'conditions': ['ee7b7e6f-e904-4b4f-a5ec-b3bef040643e'], 'actions': [{'subtype': 'tcp_up', 'action': 'allow', 'hosts':
['172.17.115.2'], 'ports': ['80', '443'], 'types': [], 'monitor': {}}], 'appShortcuts': [], 'appShortcutScripts': [], 'id': 'd8f09f37-9401-4136-b969-95ebbb175c1c'}
2020-09-24 07:41:27,633 [ERROR] [aggpate-client] POST /admin//entitlements :: Expecting a response but we got empty data
2020-09-24 07:41:27,633 [INFO] [appgate-operator/appgate-test] + <class 'appgate.openapi.parser.Entitlement'> simple_setup_ENT_PING [9eb29d80-0bd2-47c9-b849-816962a81b19]
2020-09-24 07:41:27,756 [ERROR] [aggpate-client] https://ec2-3-83-104-221.compute-1.amazonaws.com:444//admin//entitlements :: 422: {"id":"unprocessable entity","message":"Validation failed.","errors":[{"field":"site","message":"does not exist"}]}
2020-09-24 07:41:27,756 [ERROR] [appgate-client] payload :: {'name': 'simple_setup_ENT_PING', 'site': '3d844467-8d82-42d2-946a-53b3d8ff11ec', 'tags': ['was', 'here', 'operator', 'sdp'], 'conditions': ['ee7b7e6f-e904-4b4f-a5ec-b3bef040643e'], 'actions': [{'subtype': 'icmp_
up', 'action': 'allow', 'hosts': ['172.17.115.6', '172.17.115.2'], 'ports': [], 'types': ['0-255']}], 'appShortcuts': [], 'appShortcutScripts': [], 'id': '9eb29d80-0bd2-47c9-b849-816962a81b19'}
2020-09-24 07:41:27,757 [ERROR] [aggpate-client] POST /admin//entitlements :: Expecting a response but we got empty data
2020-09-24 07:41:27,757 [INFO] [appgate-operator/appgate-test] + <class 'appgate.openapi.parser.Appliance'> gateway-8b61286b-caf5-47df-8702-c1506a4afe3c-site1 [9f8cde7c-c231-4a86-b150-10a05c6e3648]
2020-09-24 07:41:27,887 [ERROR] [aggpate-client] https://ec2-3-83-104-221.compute-1.amazonaws.com:444//admin//appliances :: 422: {"id":"unprocessable entity","message":"Validation failed.","errors":[{"field":"site","message":"does not exist"}]}
2020-09-24 07:41:27,887 [ERROR] [appgate-client] payload :: {'name': 'gateway-8b61286b-caf5-47df-8702-c1506a4afe3c-site1', 'hostname': 'ec2-54-89-225-112.compute-1.amazonaws.com', 'clientInterface': {'hostname': 'ec2-54-89-225-112.compute-1.amazonaws.com', 'allowSources':
 [{'address': '0.0.0.0', 'netmask': 0}, {'address': '::', 'netmask': 0}]}, 'peerInterface': {'hostname': 'ec2-54-89-225-112.compute-1.amazonaws.com', 'allowSources': [{'address': '::', 'netmask': 0}, {'address': '0.0.0.0', 'netmask': 0}]}, 'networking': {'hosts': [], 'nic
s': [{'name': 'eth0', 'enabled': True, 'ipv4': {'dhcp': {'enabled': True, 'dns': True, 'routers': True, 'ntp': False, 'mtu': False}, 'static': []}, 'ipv6': {'dhcp': {'enabled': False, 'dns': True, 'ntp': False, 'mtu': False}, 'static': []}}], 'dnsServers': [], 'dnsDomains
': [], 'routes': []}, 'tags': [], 'site': '3d844467-8d82-42d2-946a-53b3d8ff11ec', 'ntp': {'servers': [{'hostname': '3.ubuntu.pool.ntp.org'}, {'hostname': '1.ubuntu.pool.ntp.org'}, {'hostname': '0.ubuntu.pool.ntp.org'}, {'hostname': '2.ubuntu.pool.ntp.org'}]}, 'sshServer':
 {'enabled': True, 'allowSources': [{'address': '0.0.0.0', 'netmask': 0}, {'address': '::', 'netmask': 0}]}, 'snmpServer': {'allowSources': []}, 'healthcheckServer': {'allowSources': [{'address': '::', 'netmask': 0}, {'address': '0.0.0.0', 'netmask': 0}]}, 'prometheusExpo
rter': {'allowSources': []}, 'ping': {'allowSources': [{'address': '::', 'netmask': 0}, {'address': '0.0.0.0', 'netmask': 0}]}, 'logServer': {}, 'controller': {}, 'gateway': {'enabled': True, 'vpn': {'allowDestinations': [{'nic': 'eth0'}]}}, 'logForwarder': {'tcpClients':
 [], 'awsKineses': [], 'sites': []}, 'connector': {'expressClients': [], 'advancedClients': []}, 'rsyslogDestinations': [], 'hostnameAliases': [], 'id': '9f8cde7c-c231-4a86-b150-10a05c6e3648'}
2020-09-24 07:41:27,887 [ERROR] [aggpate-client] POST /admin//appliances :: Expecting a response but we got empty data
2020-09-24 07:41:27,887 [INFO] [appgate-operator/appgate-test] + <class 'appgate.openapi.parser.Site'> simple_setup Site [3d844467-8d82-42d2-946a-53b3d8ff11ec

The latest version of the base image we use for running the linter and the tests does no work anymore

python:rc-slim has been updated and now the build seems to be broken.

set eq is not deterministic

Order in the EntityWrapper is important since we need to called it in the right order and right now sometimes we get the expected entity being self and sometimes being other.

Still seems we have problems when updating data into the metadata configmap

We keep treating this entity like modified because the generation is not updated in the metadata configmap.

2020-09-24 08:00:45,225 [INFO] [appgate-state] Validating expected state entities
2020-09-24 08:00:45,805 [INFO] [appgate-operator/appgate-test] Updating current state from controller
2020-09-24 08:00:48,183 [INFO] [appgate-operator/appgate-test] No more events for a while, creating a plan
2020-09-24 08:00:48,728 [INFO] [appgate-operator/appgate-test] AppgatePlan Summary:
2020-09-24 08:00:48,728 [INFO] [appgate-operator/appgate-test] * <class 'appgate.openapi.parser.GlobalSettings'> GlobalSettings [GlobalSettings]
2020-09-24 08:00:48,728 [INFO] [appgate-operator/appgate-test]    DIFF for GlobalSettings:
2020-09-24 08:00:48,728 [INFO] ---
2020-09-24 08:00:48,728 [INFO] +++
2020-09-24 08:00:48,728 [INFO] @@ -10,3 +10,3 @@
2020-09-24 08:00:48,728 [INFO]      "appDiscoveryDomains": [],
2020-09-24 08:00:48,729 [INFO] -    "generation": 1
2020-09-24 08:00:48,729 [INFO] +    "generation": 2
2020-09-24 08:00:48,729 [INFO]  }

Current metadata confuses argocd

The problem is that we store "our" metadata in the same place as k8s metadata and argocd uses it to compare entities. Since our values never end up in k8s argocd keeps saying the entities are out of symc.

There are 2 possible solutions:

1. Store the metadata in annotations, for "reasons" this is just a list of key-value pairs so that means saving the metadata as a json encoded in a string (json inside json). This obviously looks really bad.
2. Inject a new optional attribute on every instance and add there the metadata, if present we merge it with k8s metadata and keep doing what we do now. This attribute is only dumped on k8s but never on appgate side.

Support setting device id

Right now a new one is generated on every run, creating a new device entry every time on the controller.

The operator should persist the generated device id into the config map, see #71.
The operator should also allow setting the device id directly.

Add support for health checks that can be used by argocd

Add support for multiple CRD versions

We should support different CRD versions for different appgate versions simultaneously in the same cluster

Persist device id in configmap

Else the operator creates a lot of onboarding cookie entries in the appgate controller.

metadata configmap is not updated on PUT for singletons

When patching we get an empty response (None) so we never update the metadata config map, as a result the generation is always setting the object as modified.

Add support for API version 16

See https://github.com/appgate/sdp-api-specification/tree/version-16

Logging errors is broken :O

Call stack:
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/appgate/__main__.py", line 173, in <module>
    main()
  File "/root/appgate/__main__.py", line 151, in main
    main_run(OperatorArguments(
  File "/root/appgate/__main__.py", line 43, in main_run
    asyncio.run(run_k8s(args))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
    handle._run()
  File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/root/appgate/appgate.py", line 315, in main_loop
    log.error('[appgate-operator/%s] Error %s:', e)
Message: '[appgate-operator/%s] Error %s:'
Arguments: ('8c9adc05-8207-47eb-b76b-c5cfd6564077',)

Some fields are saved as sets but compared as lists

This results as diffs even though nothing changed, for example the ntp servers.

Problem when deleting all the entities from github

2020-09-25 10:15:16,233 [ERROR] [appgate-operator/aitor-demo1] Found errors in expected state and plan can not be applied.
2020-09-25 10:15:16,233 [ERROR] [appgate-operator/aitor-demo1] Entity: gateway-ad1ce7c3-1a5d-4131-8dfc-ca6d20de3195-site1 references: [8a4add9e-0e99-4bb1-949c-c9faf9a49ad4] (field site), but it is not defined in the system.
2020-09-25 10:15:16,233 [INFO] [appgate-operator/aitor-demo1] Waiting for more events that can fix the state.

Create a new cli option to validate entities.

Something like appgate-operator validate --directory entities for instance.

Then that command could be run as part of the CI in a PR so the entities could be validated before merging.

how to more easily reference other resources other than by UUID?

When trying to use the operator I run into a lot of cases where I have to reference another resource by UUID, like in the identity-provider resource below referencing ip pools

apiVersion: beta.appgate.com/v1
kind: IdentityProvider-v15
metadata:
  name: local
spec:
... etc ...
  ipPoolV4: XXXXXX
  ipPoolV6: XXXXXX
  name: local
  notes: Built-in Identity Provider on local database.
  tags:
  - builtin
  type: LocalDatabase

is there a better way that I can reference other resources, such as by resource name?

The examples included in the repo are not working

The exception is this:

Path: .appgate_metadata.passwords

Exceptions:

  'list' object has no attribute 'items'

  Path: .appgate_metadata.passwords

  Not None

  Path: .appgate_metadata.passwords

2021-03-03 20:29:39,380 [INFO] [appgate-operator/appgate-test-1] Nothing changed! Keeping watching!

2021-03-03 20:29:39,461 [INFO] [k8s-configmap-client/appgate-test-1-configmap/appgate-test-1] Reading ClientConnection-singleton: not found

2021-03-03 20:29:39,462 [ERROR] [appgate-operator/appgate-test-1] Unhandled error for clientconnections

Traceback (most recent call last):

  File "/root/appgate/appgate.py", line 235, in run_entity_loop

    entity = load(ev.object.spec, ev.object.metadata, entity_type)

  File "/root/appgate/attrs.py", line 197, in load

    return loader.load(data, entity)

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 294, in load

    raise e

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 290, in load

    return cast(T, func(self, value, type_))

  File "/root/appgate/attrs.py", line 179, in _attrload

    return _namedtupleload_wrapper(orig_values, l, value, t)

  File "/root/appgate/attrs.py", line 102, in _namedtupleload_wrapper

    entity = dataloader._namedtupleload(l, value, t)

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 508, in _namedtupleload

    params[k] = l.load(

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 294, in load

    raise e

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 290, in load

    return cast(T, func(self, value, type_))

  File "/root/appgate/attrs.py", line 179, in _attrload

    return _namedtupleload_wrapper(orig_values, l, value, t)

  File "/root/appgate/attrs.py", line 102, in _namedtupleload_wrapper

    entity = dataloader._namedtupleload(l, value, t)

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 508, in _namedtupleload

    params[k] = l.load(

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 294, in load

    raise e

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 290, in load

    return cast(T, func(self, value, type_))

  File "/root/venv/lib/python3.9/site-packages/typedload/dataloader.py", line 553, in _unionload

    raise TypedloadValueError(

typedload.exceptions.TypedloadValueError: Value of list could not be loaded into typing.Optional[typing.Dict[str, typing.Union[str, typing.Dict[str, str]]]]

Path: .appgate_metadata.passwords

Exceptions...```