Comments (3)
It is taking almost three minutes to process one chunk file:
2024-06-25T21:28:17Z [Information] Request URL: 'https://edavdeviasadoc.blob.core.windows.net/content/AzureBatch/EDAV%20Azure%20Batch%20-%20Customer%20Manual%20Guide.docx/EDAV%20Azure%20Batch%20-%20Customer%20Manual%20Guide-23.json'
Request method: 'PUT'
Request headers:
'Content-Length': '1098'
'x-ms-blob-type': 'REDACTED'
'x-ms-version': 'REDACTED'
'Content-Type': 'application/octet-stream'
'Accept': 'application/xml'
'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.11.9 (Linux-5.15.153.1-2.cm2-x86_64-with-glibc2.31)'
'x-ms-date': 'REDACTED'
'x-ms-client-request-id': 'd5f92382-3339-11ef-8688-c21a2cb949bc'
'Authorization': 'REDACTED'
A body is sent with the request
2024-06-25T21:28:17Z [Information] Response status: 201
Response headers:
'Content-Length': '0'
'Content-MD5': 'REDACTED'
'Last-Modified': 'Tue, 25 Jun 2024 21:28:16 GMT'
'ETag': '"0x8DC955DBA2E8555"'
'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
'x-ms-request-id': 'ccd90fc0-c01e-003e-6c46-c760c1000000'
'x-ms-client-request-id': 'd5f92382-3339-11ef-8688-c21a2cb949bc'
'x-ms-version': 'REDACTED'
'x-ms-content-crc64': 'REDACTED'
'x-ms-request-server-encrypted': 'REDACTED'
'Date': 'Tue, 25 Jun 2024 21:28:16 GMT'
2024-06-25T21:28:17Z [Information] Path and SAS token for file in azure storage are now generated
2024-06-25T21:28:18Z [Information] Request URL: 'https://edavdeviasadoc.blob.core.windows.net/content/AzureBatch/EDAV%20Access%20Chart%20and%20Personas%20Customer%20Guide.docx/EDAV%20Access%20Chart%20and%20Personas%20Customer%20Guide-19.json'
Request method: 'PUT'
Request headers:
'Content-Length': '6592'
'x-ms-blob-type': 'REDACTED'
'x-ms-version': 'REDACTED'
'Content-Type': 'application/octet-stream'
'Accept': 'application/xml'
'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.11.9 (Linux-5.15.153.1-2.cm2-x86_64-with-glibc2.31)'
'x-ms-date': 'REDACTED'
'x-ms-client-request-id': 'd6fd5a14-3339-11ef-9cdb-c21a2cb949bc'
'Authorization': 'REDACTED'
A body is sent with the request
2024-06-25T21:28:19Z [Information] Path and SAS token for file in azure storage are now generated
2024-06-25T21:28:19Z [Information] Response status: 201
Response headers:
'Content-Length': '0'
'Content-MD5': 'REDACTED'
'Last-Modified': 'Tue, 25 Jun 2024 21:28:18 GMT'
'ETag': '"0x8DC955DBB3B3AF5"'
'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
'x-ms-request-id': '075cf635-401e-001f-0346-c744ba000000'
'x-ms-client-request-id': 'd6fd5a14-3339-11ef-9cdb-c21a2cb949bc'
'x-ms-version': 'REDACTED'
'x-ms-content-crc64': 'REDACTED'
'x-ms-request-server-encrypted': 'REDACTED'
'Date': 'Tue, 25 Jun 2024 21:28:18 GMT'
from pubsec-info-assistant.
A significant portion of the time appears to be sitting in queues...
{
"status": "FileLayoutParsingOther - message sent to enrichment queue",
"status_timestamp": "2024-06-25 19:22:36",
"status_classification": "Debug"
},
{
"status": "TextEnrichment - Received message from text-enrichment-queue ",
"status_timestamp": "2024-06-25 19:37:38",
"status_classification": "Debug"
},
The system is configured with a minimum number of instances in the App Service Plan with a scale out configuration. The time to scale can vary so if you have uploaded many files at the same time the queues will grow in depth and take time to clear out.
You are welcome to increase the SKUs, increase the number of instances, etc. But you will need to performance test and validate scaling as part of these updates.
from pubsec-info-assistant.
Hello team,
We resolved the issue. The issues was related to our private endpoints on Multi AI Service Account. We had two DNS entries pointing to same privatelink and when a call made from text enrichment function app, it was a hit and miss. Whenever retry pinged correct endpoint, it was going thru and other times it was getting rejected. After removing invalid DNS entry, it resolved the issue and started working as expected.
Thanks for your response.
from pubsec-info-assistant.
Related Issues (20)
- Governance Infused Ingestion, Embedding and RAG HOT 1
- Release v1.1.1 streamed responses: only the first line is displayed HOT 3
- SharePoint feature issue HOT 3
- How to upgrade from gpt-35-turbo-16k to gpt-4o HOT 5
- Azure DevOps Pipeline or GitHub Action for deploying the solution as IaC HOT 2
- Instructure for setting up sandbox environment is outdate HOT 1
- Enrichment Web App Deployment fails. HOT 5
- Error: 500: Failed to embed HOT 3
- Missing Parameter in GOVCloud environment file
- WebApp fails to start due to tenacity version upgrading to 8.4.0 - 8.4.1 ( previously working version was tenacity 8.3.0 )
- Few-shot examples with energy conservation are used in prompts when chatting ungrounded with the models HOT 1
- Sharepoint ingest missing some files under subfolder HOT 3
- Segregate UI and backend
- how do we add a folder along with query to search in the specific folder, instead of doing an everywhere search? Is there way we can add this to the prompt HOT 1
- Upon uploading the new document, the website reveals the blob URL during the session HOT 4
- Function test intermittent failed, files were uploaded to "upload" container, but not processing to "content" container
- Unused/duplicate function purge_soft_deleted_blob HOT 1
- Documents stuck in embeddings-queue HOT 4
- Inconsistency in answers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pubsec-info-assistant.