`failed to run apt update` on simultaneous builds

[DnPlas]

`failed to run apt update` on simultaneous builds

[DnPlas]

@mr-cal thanks for trying to reproduce the issue, maybe you haven't bootstrapped a juju controller, this message suggests that is the case ERROR tests/integration/test_kfp_functional_v2.py::test_build_and_deploy - FileNotFoundError: [Errno 2] No such file or directory: '/home/developer/.local/share/juju/controllers.yaml'.

I tried reproducing this again today, and perhaps the issue was in fact the destructive mode setting, I must've mixed the logs. In other tests I did during the day, I saw that this and other issues related to simultaneous builds are not present anymore, but if anything I will let you know. Thanks for the follow up!

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-2832.

This message was autogenerated

[mr-cal]

Thanks for the report, it does look like the build-packages is what exposed this issue.

I think the problem is building multiple charms in destructive mode. We could patch craft-parts to wait for apt locks around here, but then the more complications will follow. For example, different builds may install different versions of packages, require conflicting packages, or add packages from other architectures.

The right solution for building multiple charms in parallel is to use build environments. All the issues here have been resolved. If it is not working, can you please create new issues in github?

[DnPlas]

hey @mr-cal thanks for the follow up!

I observe this behaviour w/o destructive mode, in fact using destructive mode yields a different issue. I have provided steps to reproduce in the description of this issue w/o destructive mode, so you should be able to reproduce, even outside of a Github/CI environment.

The right solution for building multiple charms in parallel is to use build environments. All the issues here have been resolved. If it is not working, can you please create new issues in github?

Please ignore that file as the behaviour I am describing is not using this file at all, and that comment is outdated.

[mr-cal]

Thanks for those extra details. I'll try to reproduce it locally and see what I can find.

[mr-cal]

Do you have a way to reproduce this locally?

I've taken this tox -e bundle-integration-${{ matrix.sdk }} -- --model kubeflow --bundle=./tests/integration/bundles/kfp_latest_edge.yaml.j2, substituted ${{ matrix.sdk }} for v2, installed juju from 3.1, microk8s from 1.25-strict/stable, created a kube/config file, and am stuck on the next set of errors.

  File "/home/developer/dev/charms/kfp-operators/.tox/bundle-integration-v2/lib/python3.11/site-packages/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/home/developer/dev/charms/kfp-operators/.tox/bundle-integration-v2/lib/python3.11/site-packages/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
  File "/home/developer/dev/charms/kfp-operators/.tox/bundle-integration-v2/lib/python3.11/site-packages/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/home/developer/dev/charms/kfp-operators/.tox/bundle-integration-v2/lib/python3.11/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://192.168.1.221:16443/apis/kubeflow.org/v1beta1/namespaces/kubeflow/viewers/viewer-mnist?fieldManager=kfp-operators'
For more information check: https://httpstatuses.com/404
------------------------------------------------ Captured log call -------------------------------------------------
INFO     httpx:_client.py:1013 HTTP Request: PATCH https://192.168.1.221:16443/apis/kubeflow.org/v1beta1/namespaces/kubeflow/viewers/viewer-mnist?fieldManager=kfp-operators "HTTP/1.1 404 Not Found"
============================================= short test summary info ==============================================
FAILED tests/integration/test_kfp_functional_v2.py::test_apply_sample_viewer - httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://192.168.1.221:16443/apis/kubeflow.org/v1be...
ERROR tests/integration/test_kfp_functional_v2.py::test_build_and_deploy - FileNotFoundError: [Errno 2] No such file or directory: '/home/developer/.local/share/juju/controllers.yaml'
ERROR tests/integration/test_kfp_functional_v2.py::test_upload_pipeline - httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://192.168.1.221:16443/apis/kubeflow.org/v1/p...
ERROR tests/integration/test_kfp_functional_v2.py::test_create_and_monitor_run - httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://192.168.1.221:16443/apis/kubeflow.org/v1/p...
ERROR tests/integration/test_kfp_functional_v2.py::test_create_and_monitor_recurring_run - httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://192.168.1.221:16443/apis/kubeflow.org/v1/p...
ERROR tests/integration/test_kfp_functional_v2.py::test_viz_server_healthcheck - FileNotFoundError: [Errno 2] No such file or directory: '/home/developer/.local/share/juju/controllers.yaml'
=========================================== 1 failed, 5 errors in 0.06s ============================================
bundle-integration-v2: exit 1 (1.13 seconds) /home/developer/dev/charms/kfp-operators> pytest -vv --tb=native -s --model kubeflow --bundle=./tests/integration/bundles/kfp_latest_edge.yaml.j2 /home/developer/dev/charms/kfp-operators/tests/integration/test_kfp_functional_v2.py pid=110018
  bundle-integration-v2: FAIL code 1 (1.17=setup[0.04]+cmd[1.13] seconds)
  evaluation failed :( (1.22 seconds)

[mr-cal]

I wonder if it would be simpler to reproduce directly with pytest-operator

[mr-cal]

Great, glad to hear it's working!