Comments (6)
Hi, thanks for the information provided and I am starting the investigations. I will update here once I found anything useful.
from azure-functions-durable-js.
Hi, @shyamal890 I would like to confirm this activity function expected behavior. What is the failed activity function durableActScrapeWebsite is working for? For how long should this activity function return to the orchestrator? I queried the failed instanceId, and for this activity function, it seems that it never returns to the orchestrator. And the function failed due to the time out. Thanks!
from azure-functions-durable-js.
The failed activity durableActScrapeWebsite is scraping a particular website through puppeteer. You are right that it timed out but the activity function hanged. we are logging each scrape activity and hence it should have returned a log but for some reason it doesn't.
Also, recently started noticing that logs are getting mixed up between functions, that is a http function's log is showing up in durable function and vice-versa. I think this shouldn't happen right?
Here's the setting in host.json
"functionTimeout": "03:00:00"
from azure-functions-durable-js.
So, the activity function should return in 3 hours but it didn't, right? I just want to confirm this part. Thanks!
And I will query the instnace Id you provided in another issue page too to check. Will update once I found any.
from azure-functions-durable-js.
Hi, @shyamal890. I queried the instanced you provided 983c33044cfa4df6b0669502b56bfea9
and fc13abeb0d1d466bb10b29f265789151
. The two are from the same reason: the message that scheduled activity function in the work-item queue cannot be processed successfully.
The messageId is 10286b8f-dab6-40a2-88ec-4400a6e9baa5
for this issue and 4257ccf6-d903-4158-a1e0-ea4f079c5343
for issue #545 .
As you can see, there were many workers trying to process it but couldn't succeed. And finally, the functions failed due to time out. I checked the age of messages of all the queues. From the picture below we can see that the blue one which is the work-item queue, has much larger Age than the other control queues.
Since you mentioned that the activity function should return to the orchestration in 3 hours, I would assume the problem should because of the work-item queue being unhealthy. There are several options we can do:
- Move to another Taskhub. This is the easiest way, and we will have completely new created queues. Also, since it's all new, we will lose the history, which is stored in the old task hub, and thus all the functions cannot resume from where it's left but start from the beginning.
- Create a new work-item queue. We can manually rename the current work-item queue as work-item-backup. Then restart the app, there will be a new work-item queue created. It's a little more complicated but we can have all the history and resume the function.
For the timeout problem, specifically, the timeout setting is designed to ensure that the function runs on a single VM for a maximum of 3 hours. This is to avoid permanent function since nothing can run forever. And the work-item queue can be read and wrote by any active worker. From the screenshot we can also see there would be different worker trying to process the message when it became visible. So, as long as it doesn't keep running at one worker for 3 hours, it won't time out.
I would suggest trying either of the options above to see if we can mitigate the issue. And please feel free to reach out if there is any further questions. Thanks!
from azure-functions-durable-js.
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.
from azure-functions-durable-js.
Related Issues (20)
- Intermittent Error Running Suborchestrator
- weak typing for yield results on `callActivity` HOT 1
- ❓Survey: Your experience with Durable Functions
- [Please ignore] Testing GH triage automation HOT 1
- Failed: node exited with code 1 HOT 2
- Function hangs and doesn't even timeout unless terminated HOT 2
- This repo is missing important files HOT 1
- Durable entity creation? (and lifecycle management) HOT 3
- waitForExternalEvent not completing in reasonable time when event arrives early
- Update axios package version HOT 10
- Receiving error "Cannot read properties of undefined" at method getClient HOT 7
- Orchestrations are failing to find the orchestration but its listed in the known orchestrators HOT 1
- context.df.isReplaying is always undefined HOT 1
- Update axios dependency in both v2 and v3 HOT 2
- Hello world works locally but fails when deployed to Azure HOT 1
- `callHttp` long polling returning error: `Reason: Message: Converting circular structure to JSON`
- `context.df.Task.any` launches tasks even if the task was already launched and sometimes return result from a different `Task` HOT 6
- context.df.Task.any with createTimer and waitForExternalEvent leaves orchestrator in invalid state
- Known regression: Activity input fails to deserialize when the input is an array HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-functions-durable-js.