Longs become Integers after export/import

[sea-steve]

Longs become Integers after export/import

[sea-steve]

I don't know how you plan to approach a fix for this, since other data types (BigDecimal, BigInteger, etc.) may also be affected by this issue. For me it was just Long values, so in case this is helpful to you as a fix or anyone else trying to work around this issue, below is a diff (for 3.4.4 NitriteJsonImporter.java) that I've tested and resolves the issue for me. (I've minimally validated that this change fixes the data in my project: by exporting the corrupt db and then importing it with the patch applied. And, my POC - see original post - runs without throwing/logging an exception after this patch is applied.)

Patch/fix (for Long values only):

# This patch file was generated by NetBeans IDE
# It uses platform neutral UTF-8 encoding and \n newlines.
--- a/nitrite/src/main/java/org/dizitart/no2/tool/NitriteJsonImporter.java
+++ b/nitrite/src/main/java/org/dizitart/no2/tool/NitriteJsonImporter.java
@@ -24,13 +24,17 @@
 import org.dizitart.no2.objects.ObjectRepository;
 
 import java.io.IOException;
+import java.lang.reflect.Field;
 import java.util.Map;
+import lombok.extern.log4j.Log4j2;
+import org.apache.commons.lang3.reflect.FieldUtils;
 
 import static org.dizitart.no2.Constants.*;
 
 /**
  * @author Anindya Chatterjee.
  */
+@Log4j2
 class NitriteJsonImporter {
     private JsonParser parser;
     private Nitrite db;
@@ -82,7 +86,7 @@
                 }
 
                 if (TAG_DATA.equals(fieldName) && repository != null) {
-                    readCollectionData(repository.getDocumentCollection());
+                    readCollectionData(repository.getDocumentCollection(), repository);
                 }
             }
         }
@@ -112,7 +116,7 @@
                 }
 
                 if (TAG_DATA.equals(fieldName)) {
-                    readCollectionData(collection);
+                    readCollectionData(collection, null);
                 }
             }
         }
@@ -144,7 +148,7 @@
     }
 
     @SuppressWarnings("unchecked")
-    private void readCollectionData(NitriteCollection collection) throws IOException {
+    private void readCollectionData(NitriteCollection collection, ObjectRepository repository) throws IOException {
         // move to [
         parser.nextToken();
 
@@ -165,7 +169,10 @@
                     parser.nextToken();
                     objectMap = (Map<String, Object>) parser.readValueAs(Map.class);
                     objectMap.put(DOC_ID, id);
+                    if (repository != null) {
+                      fixLongs(objectMap, repository.getType());
                 }
+                }
 
                 if (objectMap != null) {
                     Document document = new Document(objectMap);
@@ -177,4 +184,28 @@
             }
         }
     }
+    
+    private void fixLongs(Map<String, Object> objectMap, Class repositoryType) {
+      log.debug("In fixLongs with repositoryType: " + repositoryType.getName());
+      Field[] fields = FieldUtils.getAllFields(repositoryType);
+      for (Field field : fields) {
+        log.debug("Checking field: " + field.getName());
+        log.debug("Field type: " + field.getType());
+        if (field.getType().equals(Long.class)) {
+          String fieldName = field.getName();
+          log.debug("Found Long field: " + fieldName);
+          Object importedFieldVal = objectMap.get(fieldName);
+          log.debug("Field val: " + importedFieldVal);
+          if (importedFieldVal != null) {
+            log.debug("Field val type: " + importedFieldVal.getClass());
+            log.debug("Field val instanceof Integer: " + (importedFieldVal instanceof Integer));
+            if (importedFieldVal instanceof Integer) {
+              Long longVal = new Long((Integer)importedFieldVal);
+              objectMap.put(fieldName, longVal);
+              log.debug("Field type after update: " + objectMap.get(fieldName).getClass());
 }
+          }
+        }
+      }
+    }
+}

[anidotnet]

Hi, thanks for the investigation. This bug has already been addressed in the upcoming version Nitrite 4.x. This version is not released yet, but will soon. Unfortunately, I won't be doing any further development on 3.x. For the temporary measure you can go ahead with your fix. Alternatively, you can take a look at the new import/export code in branch 4.x and change your solution for the time being.

[sea-steve]

Ok, I searched through the Issues and couldn't find one discussing this bug. Can you point me to the issue?

[anidotnet]

I didn't mean that there was any github issues regarding this. What I meant is this bug was identified and addressed in 4.x branch. Here is the code - https://github.com/nitrite/nitrite-java/blob/4.x/nitrite-support/src/main/java/org/dizitart/no2/support/NitriteJsonExporter.java

[sea-steve]

Ok, so it was an undocumented bug. That sucks, because had I known there was an bug with the 3.4.4 export/import, I wouldn't have used it -- and I wouldn't have wasted a bunch of time investigating the issue and creating repro code for you.

Frankly, I'm a little confused about the current state of this project. You've said that 4.x is not "released" yet, but 3.4.4 apparently has some pretty significant undocumented bugs that cause data corruption... What branch do you recommend as the current "production" version of this project?

[anidotnet]

You can keep using 3.4.4 except the import/export functionality. I do apologize for the confusion. In 4.x the library was rebuilt from the ground up to implement some new features which can't be built in 3.x at scale. There are breaking changes in the api and the documentation is not fully ready to be released yet.

For the import/export functionality, as I have said earlier, you can roll your own lib by looking at how it is done in 4.x. Just don't use the in-built import/export feature for the time being.

[sea-steve]

Ok, might be a good idea to either remove the 3.x Exporter and Importer. Or put a big warning comment in the code. Or mark those classes as deprecated.