Search results depend on the order of fields

[nextapps-de]

Search results depend on the order of fields

[lludol]

I have the same bug!

To reproduce create documents with multiple attributes.
In the search fields put every attributes.

Then put almost the same data in two differents attributes for 3-4 documents.

Exemple

{ id: 1, name: 'fooA', name2: 'fooA456' }
{ id: 1, name: 'fooA', name2: 'fooA456' }
{ id: 1, name: 'fooA', name2: 'fooA456' }

{ id: 1, name: 'fooE', name2: 'fooA123' }
{ id: 1, name: 'fooF', name2: 'fooA123' }
{ id: 1, name: 'fooG', name2: 'fooA123' }

Now you can type in your search something like this: fooA
You will only get the first block (the 3 elements in my example).

I have the same bug in production but with more data!
Now I am fucked...

[ts-thomas]

Please change over to the version >= 0.7.x which solves that issue, thanks a lot.

[stefandesu]

I also encountered other problems which occur when importing large datasets into an index (thus the original title until I discovered that it was unrelated to the size of the dataset), but splitting the index into many smaller indexes seemed to alleviate those problems (and it's still plenty fast!).

[ts-thomas]

@stefandesu The issue occurs by using "notation:0 in the field description. Actually array index notation is not supported. But this looks very easy to implement.

To solve this actually you need to serialize the array notation into a string when adding:

let index = new FlexSearch({
  doc: {
    id: "uri",
    field: [
      "prefLabel:de",
      "editorialNote:de",
      "scopeNote:de",
      "notation"
    ]
  },
  profile: "score"
});

Either:

simple array

concepts.forEach(function(item){ 
    item.notation = item.notation.join(" "); 
});

Or:

complex array

concepts.forEach(function(item){ 
    item.notation = JSON.stringify(item.notation); 
});

[stefandesu]

@ts-thomas Thanks for your reply, but I think you are wrong. I updated the example above to use a string instead of the array notation (just like you suggested) and it's still happening. Also, you said "array index notation is not supported", but it did work with notation:0 when it was last in the field array.

Please try the code out. I'm able to reproduce the issue on both my work computer and my personal computer. (I also used ./flexsearch/flexsearch for the import initially because I was using a fork, but I updated that as well. Sorry about that.)

Edit: I think the array notation (notation:0) works because in JavaScript, arrays are basically objects, so maybe the feature didn't need to be implemented because it is already inherently supported just like objects.

[stefandesu]

Note that this is still an issue as of FlexSearch 0.6.22 and it's preventing me from giving FlexSearch a serious go for our project. If needed, I can try to provide a PR with tests that demonstrate the problem using a simpler dataset than above.

[jer-sen]

In fact, only the results with the first matching field are returned. If you search "blue" with indexed fields ["a", "b", "c"] in [{ id: "1", a:"green", b:"blue", c:"yellow"}, {id: "2", a:"orange", b:"black", c:"blue"}] you will get only the first item since the first matching field is "b" and second item field "b" doesn't match the query.

result[count++] = tmp; should be added to this line

flexsearch/flexsearch.js

Line 3473 in a4e6e67

check[index] = z + 1;

[stefandesu]

@Jay1337 I could not reproduce the issue with your example, but I found out that adding another field to the index (even if that field does not exist in the data) makes it fail:

const FlexSearch = require("flexsearch")

const indexSuccess = new FlexSearch({
  doc: {
    id: "id",
    field: [
      "a",
      "b",
      "c",
    ]
  },
})
const indexFail = new FlexSearch({
  doc: {
    id: "id",
    field: [
      "a",
      "b",
      "c",
      "d",
    ]
  },
})

const data = [
  { id: "1", a: "green", b: "blue", c: "yellow" },
  { id: "2", a: "orange", b: "black", c: "blue" },
]

indexSuccess.add(data)
indexFail.add(data)

let results

results = indexSuccess.search("blue")
console.log(results)
// Prints out two results as expected.

results = indexFail.search("blue")
console.log(results)
// Prints only one result -> bug!

Unfortunately, adding your fix doesn't change this, so the bug must be somewhere else.

In addition to that, I found out that if you try to index a field that is not available on all items in the data, FlexSearch will throw an error. I created a new issue for this: #122

[JankoDedic]

Happens to me as well. When the first field matches, the following ones aren't considered at all.

[stefandesu]

I just updated my original issue so that it can be executed with version 0.6.32. Unfortunately, I wasn't able to get Flexsearch running at all with the upcoming version 0.7.0 in Node: "ReferenceError: window is not defined"