Comments (3)
alexlatchford
commented on June 8, 2024
1
Yeah we are just concluding an RCA after we had a cohort of workflows "stuck" in Running state as the upgrade of the workflow-controller was rolled out from v3.4.16 to v3.5.6. We had the same conclusion that likely #11947 was the root cause, but #12402 seems related too! In either case looks like it's isolated to a transient issue on our end due to incompatibility between argoexec:v3.4.16 sidecars inside the active pods of workflows running at the time of the upgrade and the new workflow-controller:v3.5.6 control plane! Thanks for the original report and the maintainers for all the effort here!
from argo-workflows.
jswxstw
commented on June 8, 2024
It seems unreliable to determine whether the task result has been successfully reported through the AnnotationKeyReportOutputsCompleted.
from argo-workflows.
shuangkun
commented on June 8, 2024
After reproducing it, there are indeed some problems and need to think about how to fix it.
from argo-workflows.
Related Issues (20)
- Archived workflows deletion with GC stopped working after move to namespace install HOT 6
- Devcontainer `Permission denied` error during `chown`
- Unable to run Controller locally on Arm mac M1 HOT 1
- UI: Open custom `links` in new tab or with router navigation depending on address HOT 1
- `MetricsError`: custom metrics at Workflow-level unable to resolve output variables HOT 1
- support `retryPolicy` combination HOT 3
- `podSpecPatch` not applied when configured in `workflowDefaults` and workflow is submitted from template HOT 1
- `podSpecPatch` doesn't evaluate `retries` when defined at workflow spec HOT 5
- feat: support fish completion
- Use trusted distribution for Python SDK publishing
- 3.5.7 Server pods are crashing after upgrade HOT 10
- use label variables in metrics HOT 6
- `make docs` reorders `.spelling` as case-insensitive on Mac HOT 1
- 3.5.7 No more result from API when `workflows.argoproj.io/phase=Running` HOT 7
- Custom Metrics are continuously updated for failed Cron Workflows HOT 2
- Links from workflows' containers in UI stopped being clickable after update HOT 2
- 3.5.6 Logs flooded with `Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io`
- UI: "Finished before" field only affects visible Workflows HOT 1
- Workflow stuck in `Running` when Karpenter `pod deleted`
- 3.5.7 Server keepa restarting, panicking HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from argo-workflows.