API calls to AWX read operation timed out

[sylvain-de-fuster]

API calls to AWX read operation timed out

[fosterseth]

any trends in the API calls you are making? what endpoints, how busy is the system when this happens (does it coincide with high traffic overall)?

are the uswgi processes staying health when you see timeouts?

[sylvain-de-fuster]

Hello,

It doesn't seem to be a specific/unique endpoint impacted.
Usually, we use endpoints matching these purposes :

• Create job templates
• Create schedules
• Create inventories
• Launch jobs

The system doesn't have to be heavy loaded to experience theses errors.

It's very "precise" :
For example, a playbook with a task that use awx.awx.schedule with a loop can :

• run perfectly
• fail on one unique iteration of the list parsed

All that regardless of system usage.

For the uswgi part, do you have specific logs of metrics that could help me answering you ?

I can notice some messages like that in the web container but not sure it always coincide to timeouts and are related generally speaking :

• Lots ok WSCONNECT/WSDISCONNECT sometimes

127.0.0.1:39582 - - [24/Jan/2024:15:15:11] "WSCONNECTING /websocket/" - -
127.0.0.1:39582 - - [24/Jan/2024:15:15:11] "WSCONNECT /websocket/" - -
127.0.0.1:39596 - - [24/Jan/2024:15:15:15] "WSCONNECTING /websocket/" - -
127.0.0.1:39596 - - [24/Jan/2024:15:15:15] "WSCONNECT /websocket/" - -
127.0.0.1:39816 - - [24/Jan/2024:15:15:23] "WSCONNECTING /websocket/" - -
127.0.0.1:39816 - - [24/Jan/2024:15:15:23] "WSCONNECT /websocket/" - -
127.0.0.1:39826 - - [24/Jan/2024:15:15:23] "WSCONNECTING /websocket/" - -
127.0.0.1:39826 - - [24/Jan/2024:15:15:23] "WSCONNECT /websocket/" - -
127.0.0.1:39826 - - [24/Jan/2024:15:15:30] "WSDISCONNECT /websocket/" - -
127.0.0.1:53770 - - [24/Jan/2024:15:15:34] "WSDISCONNECT /websocket/" - -
127.0.0.1:44636 - - [24/Jan/2024:15:15:42] "WSCONNECTING /websocket/" - -
127.0.0.1:44636 - - [24/Jan/2024:15:15:43] "WSCONNECT /websocket/" - -
127.0.0.1:44644 - - [24/Jan/2024:15:15:45] "WSCONNECTING /websocket/" - -
127.0.0.1:44644 - - [24/Jan/2024:15:15:45] "WSCONNECT /websocket/" - -
127.0.0.1:44654 - - [24/Jan/2024:15:15:45] "WSCONNECTING /websocket/" - -
127.0.0.1:44654 - - [24/Jan/2024:15:15:45] "WSCONNECT /websocket/" - -
127.0.0.1:44654 - - [24/Jan/2024:15:15:57] "WSDISCONNECT /websocket/" - -
127.0.0.1:44336 - - [24/Jan/2024:15:16:19] "WSDISCONNECT /websocket/" - -
127.0.0.1:33904 - - [24/Jan/2024:15:16:21] "WSCONNECTING /websocket/" - -
127.0.0.1:33904 - - [24/Jan/2024:15:16:21] "WSCONNECT /websocket/" - -
127.0.0.1:33912 - - [24/Jan/2024:15:16:21] "WSCONNECTING /websocket/" - -
127.0.0.1:33912 - - [24/Jan/2024:15:16:21] "WSCONNECT /websocket/" - -
127.0.0.1:33912 - - [24/Jan/2024:15:16:23] "WSDISCONNECT /websocket/" - -
127.0.0.1:33926 - - [24/Jan/2024:15:16:23] "WSCONNECTING /websocket/" - -

• Some uWSGI kills/respawned

worker 2 killed successfully (pid: 1476)
Respawned uWSGI worker 2 (new pid: 1501)
mounting awx.wsgi:application on /

If I check all the web logs (2024-01-24 15:17:17,684 to 2024-01-25 13:25:34,331 for this case)

# kubectl logs pod/awx-faky-web-XXXXXX -c awx-faky-web | grep 'Respawned uWSGI' | wc -l
48

If you need more informations, don't hesitate.