Comments (3)
This is the github repository for the Spark 2 connector (which doesn't even support patch).
That said, "/", and "" are disallowed characters for "id" - it has always been documented in our resource model documentation here: https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.resource.id?view=azure-dotnet
It only works for upserts if it effectively creates new documents - any point operations on an existing document (upsert, replace, delete, read) with id containing "/" or "" will fail - only create (or upsert resulting in Create) and query are possible.
I agree that it is confusing that we even allow these documents to be created - reason for not preventing them from even getting created is that this would result in a breaking change (API-version with breaking behavior across backend and all SDKs) - which is not worth doing.
Bottom-line - the only fix is to not use "/" and "" characters in "id" column. I assume you are seeing "/" because you use some base64 encoded value there. If so masking it is needed - see this code for example where we do the masking in our encryption library (also using base64 encrypted id with masking) https://github.com/Azure/azure-cosmos-dotnet-v3/blob/a34bac772e11642b8855251467d3f5a36a6c1fb8/Microsoft.Azure.Cosmos.Encryption/src/EncryptionProcessor.cs#L475-L489
from azure-cosmosdb-spark.
The limitation is also documented here: https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-item-limits
If you can tell me in which part of documentation you would have realized this limitation better, please let me know and I will work in the background with the right people to improve the documentation.
from azure-cosmosdb-spark.
@FabianMeiswinkel - I think we can both agree this is really in the weeds. It should be documented in the Java SDK and listed as a limitation in the Spark Cosmos connector. The fact that we have been using the connector since August should be an indication. If it was properly documented we would have seen it.
from azure-cosmosdb-spark.
Related Issues (20)
- is group by supported ?
- Documentation around OCC (optimistic concurrency control) for this library
- Read Data after enabling TTL on container level HOT 5
- Cosmos db OLTP connector custom query with TOP or Limit clause return wrong number of document HOT 1
- Using changefeedmaxpagesperbatch = 1 on streaming
- CosmosDBSpark.load(ss,readConfig) failing with SSL exception.
- Snapshot Isolation Guarantees HOT 1
- Unable to read edges from Cosmos DB
- Cosmos Changefeed Spark Streaming stops suddenly
- Performance issues observed after updating from azure-cosmosdb-spark_2.4.0_2.11-2.0.1 to azure-cosmosdb-spark_2.4.0_2.11-3.70 HOT 2
- Can't write CosmosDB HOT 3
- Can't write CosmosDB with call stack HOT 1
- Question - dose this connector work with java SpringBoot? HOT 1
- Help - Cosmos Spark OLTP 3 Connecter With SpringBoot Issue HOT 1
- Unable to Write or Read (un-encrypted) data from Cosmos DB Containers created via Cosmos DB SDK Encryption HOT 1
- Help - How to get Azure Cosmos Db's total Ru's for one year
- Cannot install on Databricks 11.3 LTS (Spark 3.3.0 HOT 2
- This repo is missing important files
- spark.cosmos.changeFeed.itemCountPerTriggerHint not honoured HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-cosmosdb-spark.