mgholam/raptordb-document

NoSql, JSON based, Document store database with compiled .net map functions and automatic hybrid bitmap indexing and LINQ query filters (now with standalone Server mode, Backup and Active Restore, Transactions, Server side queries, MonoDroid support, HQ-Branch Replication, .net core v2.0, .net standard v2.0)

raptordb-document's Issues

UWP support (PCL)

Is it possible to run this on Windows Phone or UWP? Perhaps a PCL?

JSON Schema Validation?

I use https://github.com/RicoSuter/NJsonSchema/ and it understand annotation attrbitues(like required, range, etc) from ComponentModel and from Newtonsoft.Json . Does raptor support validation by these?

Thanks

Do you have a plan for DotNetCore ?

:)

Question: Hierarcical data in view

Is it possible to make hierarchcal data in a view?
Or if not can you make a hierarchy of views on one document?

Typically the idea of document databases is to keep the structure of the document but you have a suggested flattening of data in the views. If keeping the document structure is not possible why not just use a relational database as you still need to think normalization but just are not having the standard sql interface for querying.

I am trying to find a document database that is schemabound like yours (In my world input or output schemas still makes it schemabound). I know I need to generate the views from my json schemas. If the above hierarchy is supported I might modify so schemas are used directly.

Cannot Save DB on VisulaStudio Solution

Hi, I have a problem when I go to save the db. VisualStudio does not generate errors but I cannot find the DB files in the path. I gave full control to everyone on the destination folder so the problem should not be there

WAHBitarray is expanding to bitarray state on any operation

I have tested some count queries on views and I have found that WAHBitarray is expanding from Indexes to Bitarray in AndNot method when removing deleted rows from result. But expansion takes little bit longer than index query and then CountOnes method consumes 15x that time.
Do you have any special reason to decompress WAHBitarray before any operation? Or is something wrong in my db?

operation issue

"keyword1 keyword2" means keyword1 AND keyword2
"keyword1 keyword2 -keyword3" should means include both keyword1 and keyword2, but could not have keyword3, right? when I used this, I got the result only includes keywords3, just like I search with "keyword3" only.

System.Exception: 'Unrecognized token at index = 1'

I receive the error in Title when I try to execute a full text search (ex. Db.FullTextSearch(filter);).
The Db is a singleton property of type RaptorDB.RaptorDB:

exactly the error occurs when I try to open db. (in the db there are about 1000 objects)

this is the stack trace.

at fastBinaryJSON.BJsonParser.ParseValue(Boolean& breakparse)
at fastBinaryJSON.deserializer.ToObject(Byte[] json, Type type)
at fastBinaryJSON.deserializer.ToObject[T](Byte[] json)
at fastBinaryJSON.BJSON.ToObject[T](Byte[] json)
at RaptorDB.StorageFileHF.InitializeFreeList()
at RaptorDB.StorageFileHF.Initialize(String filename, UInt16 blocksize)
at RaptorDB.StorageFileHF..ctor(String filename, UInt16 blocksize)
at RaptorDB.KeyStoreHF..ctor(String folder)
at RaptorDB.RaptorDB.Initialize()
at RaptorDB.RaptorDB..ctor(String FolderPath, ITokenizer tokenizer)
at RaptorDB.RaptorDB.Open(String FolderPath)

System.Exception : 'Unexpectedly reached end of string' and System.Exception: Expected colon at index

I get the error when transforming the JSON returned from GetObjectHF to Object with fastJSON, but I think the error is related to raptorDB.

Here's the stack trace :

   System.Exception: Unexpectedly reached end of string
   à fastJSON.JsonParser.ParseString(Char* p, Boolean val)
   à fastJSON.JsonParser.ParseValue(Char* p, Boolean val)
   à fastJSON.JsonParser.ParseObject(Char* p)
   à fastJSON.JsonParser.ParseValue(Char* p, Boolean val)
   à fastJSON.JsonParser.ParseArray(Char* p)
   à fastJSON.JsonParser.ParseValue(Char* p, Boolean val)
   à fastJSON.JsonParser.Decode()
   à fastJSON.deserializer.ToObject(String json, Type type)
   à fastJSON.deserializer.ToObject[T](String json)
   à fastJSON.JSON.ToObject[T](String json)

Here's my code:

    public class BaseRepository<T>
    {
        protected IRaptorDB DB;
        
        public BaseRepository(IRaptorDB DB)
        {
            this.DB = DB ?? throw new ArgumentNullException(nameof(DB));
        }

        protected virtual string Key { get; } 

        public T GetObjectFromKeyStore()
        {
            IKeyStoreHF keystore = this.DB.GetKVHF();
            object o;
            if (keystore == null)
                return default(T);
            
            o = keystore.GetObjectHF(Key);
            if (o == null)
                return default(T);

            return fastJSON.JSON.ToObject<T>(o.ToString());
        }

        public bool SetObjectToKeyStore(T obj) 
        {
            IKeyStoreHF keystore = this.DB.GetKVHF();

            return keystore.SetObjectHF(Key, fastJSON.JSON.ToJSON(obj));
        }
    }

It happens randomly and re-writing the document without changing the JSON or deleting the DB folder and starting fresh fixes the issue, but it's not a solution.

What makes me think the problem is with RaptorDB and not fastJSON is that to me it's seems that the string returned by GetObjectHF is badly formatted.
Here's an example of a string that gives this error:
"[{"$type":"domaine.Engin, domaine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"XXXFIELD":"XXXX\\483deeab-1ce8-4c79-b45f-588785e0e1e4","Codes":"S50","Nom":"XNON ","CodeDepot":null,"Marqueurs":"",
"Description":"S50 - MINIPELLE CATERPILLAR PAYEN ","Code":"S50","RessourceOid":1139,"PlanningXXXFIELD":null},{"$type":"domaine.Engin, domaine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nu潬祲itaux ","Adresse":""},
{"$type":"1","XXXFIELD":"XXXX\\f704e193-95b5-4bc2-b427-115dbf5affac","Code":"530007","Nom":"530007 Chantiers \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

There are many more \0\0 in the end, I excluded to make it more readable.

System.Exception: Expected colon at index 29340
   à fastJSON.JsonParser.ParseObject(Char* p)
   à fastJSON.JsonParser.ParseValue(Char* p, Boolean val)
   à fastJSON.JsonParser.ParseArray(Char* p)
   à fastJSON.JsonParser.ParseValue(Char* p, Boolean val)
   à fastJSON.JsonParser.Decode()
   à fastJSON.deserializer.ToObject(String json, Type type)
   à fastJSON.deserializer.ToObject[T](String json)
   à fastJSON.JSON.ToObject[T](String json)

This is a related error that happens randomly but again, the JSON returned seems to be badly formatted.

"[{"$type":"domaine.Engin, domaine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null","XXXField":"XXXXX\\483deeab-1ce8-4c79-b45f-588785e0e1e4","Codes":"S50","Nom":"MINIPELLE CATERPILLAR PublicKeyToken=null","XXXField":"XXXXX\\49b3598a-fca5-43df-b262-7bc4eb9fb120","C"

I'd love some help.
Thank you in advance.

How to reduce the size of .mgbmp file ?

How to reduse the size of .mgbmp file ? RemoveKey only remove key from idx and mark the storageitem in *.mgbmp as deleted.

But the file size has not decreased.
Is there anyway to remove storageitem from raptordb??

IDictionary in Schema

just getting things up and running, however I have a error, which I'm not sure how to handle.

'System.Collections.Generic.IDictionary2[System.String,System.String]', on 'RaptorDB.TypeIndexes1[T]' violates the constraint of type 'T'.

View

`[RegisterView]
public class RaptorDBLogEventView : View
{

    public RaptorDBLogEventView()
    {
        this.Name = "LiteDBLogEventView";
        this.Description = "A primary view for LogEvent";
        this.isPrimaryList = true;
        this.isActive = true;
        this.BackgroundIndexing = true;
        this.Version = 1;

        this.Schema = typeof(RaptorDBLogEventBSchema);

        this.AddFireOnTypes(typeof(RaptorDBLogEvent));

        this.Mapper = (api, docid, doc) =>
        {
            api.Emit(docid, doc);
        };
    }
}

`

Schema

`public class RaptorDBLogEventBSchema : RDBSchema
{
public Guid Id { get; set; }

    public string Message { get; set; }

    public DateTimeOffset Timestamp { get; set; }

    public LogEventLevel Level { get; set; }

    public string MessageTemplate { get; set; }

    public IDictionary<string, string> Props { get; set; }

    public Guid? CorrelationId { get; set; }

    public string Exception { get; set; }

}`

RaptorDB Translation

I did an analysis on the RaptorDB Project and I identified where I can put changes to various languages, English, Portuguese, French, and etc ...this is one of the changes I have identified that I can make to help. Can I upload the files for analysis?I want to contribute to the RaptorDB project.

Large Large files

I was testing RaptorDB using the sample code, codes are really easy to write (than couchbase) but after running test I found out that database stored about 200MB on the disk. I tried a compression with WinRar and it compressed data to 4MB(!). This means that a huge part of it is just being repeated.
I think something must be changed in saving mechanism.

Cryptography

1. MurMurHash is not cryptographically safe hash function so you shouldn't use it for passwords
2. What about to encrypt the network traffic simply by AES with the hashed users password

Inconsistency when consulting data

My APP logic is:

1. Get data from an API
2. Delete the current data in raptorDB
3. Insert the data from the API in the DB
4. use the data from the DB

Randomly I get different results from the DB when querying it.
I've added some logs in the APP to try to understand it, since it doesn't happen in my machine but occurs repeatedly in the production machine.

2020-02-24 09:42:38.9950|TRACE|application.App|Plannings API: 2 (Amount of data returned from DB)
2020-02-24 09:42:38.9950|TRACE|application.App|Plannings DB: 2 (Querying the data from the DB)
2020-02-24 09:42:38.9950|TRACE|application.App|Plannings deleted: 2 (Data deleted from the DB)
2020-02-24 09:42:38.9950|TRACE|application.App|Plannings inserted: 2 (Data inserted from the DB)
2020-02-24 09:42:38.9950|TRACE|application.App|Plannings DB: 2 (Querying the DB just after the insert to check if the amount is the same)
2020-02-24 09:42:41.7042|TRACE|application.App|LoadPlanningCollaborateur -> Planning.Count: 2 (Checking the amount again)
...
user interaction without changing DB
...
2020-02-24 09:43:51.2066|TRACE|application.App|LoadPlanningCollaborateur -> Planning.Count: 0 (Same query, same parameters, no data returned)

It can return sometimes one row.

here's my view

    /// <summary>
    /// View des plannings
    /// </summary>
    [RegisterView]
    public class PlanningView : View<Planning> 
    {
        public PlanningView()
        {
            this.isPrimaryList = true;
            this.Name = nameof(PlanningView);
            this.isActive = true;
            this.BackgroundIndexing = false;
            this.TransactionMode = true;
            this.Schema = typeof(PlanningViewRowSchema);
            this.Mapper = (IMapAPI api, Guid docid, Planning doc) => api.EmitObject(docid, doc);
            this.Version = 2;
        }
    }

Querying the DB :

return DB.Query<PlanningViewRowSchema>(filter);

Saving to DB :

        public Guid PutPlanning(Planning Planning)
        {
            if (Planning.DateCreation == DateTime.MinValue)
                Planning.DateCreation = DateTime.Now;

            if (DB.Save<PlanningViewRowSchema>(Planning.Id, PlanningViewRowSchema.FromPlanning(Planning)))
                return Planning.Id;
            else
                return Guid.Empty;
        }

Deleting from DB:

        public int DeletePlannings(List<Planning> Plannings)
        {
            int count = 0;
            foreach (var planning in Plannings)
            {
                if (DB.Delete(planning.Id))
                    count += 1;
            }
            return count;
        }

I've tried changing the view parameters like using this.TransactionMode = true; but it didn't change the results.
Any help is welcomed.

Deserialization crashing on Dictionary<string, byte[]>

fastJSON and fastBinaryJSON are crashing on bytearray deserialization in StringKeyDictionary.

threw an exception of type 'System.InvalidCastException'
    Data: {System.Collections.ListDictionaryInternal}
    HResult: -2147467262
    HelpLink: null
    InnerException: null
    Message: "Unable to cast object of type 'System.Byte[]' to type 'System.Collections.Generic.List`1[System.Object]'."
    Source: "RaptorDB.Common"
    StackTrace: "   at fastBinaryJSON.deserializer.CreateStringKeyDictionary(Dictionary`2 reader, Type pt, Type[] types, Dictionary`2 globalTypes)\r\n   at fastBinaryJSON.deserializer.ParseDictionary(Dictionary`2 d, Dictionary`2 globaltypes, Type type, Object input)\r\n   at fastBinaryJSON.deserializer.ToObject(Byte[] json, Type type)\r\n   at fastBinaryJSON.BJSON.ToObject(Byte[] json)"
    TargetSite: {System.Object CreateStringKeyDictionary(System.Collections.Generic.Dictionary`2[System.String,System.Object], System.Type, System.Type[], System.Collections.Generic.Dictionary`2[System.String,System.Object])}

I think the problem is that parser parsed it as byte array, but then it was expected to be of type List<object>