Indexing with AWS-S3 Bucket is failing

[ApsaraDhanasekar11]

Indexing with AWS-S3 Bucket is failing

[fergiemcdowall]

Thanks for the bug report @ApsaraDhanasekar11. I think I understand your problem, but there are too many variables to reproduce it accurately.

Could you include a standalone script/test that demonstrates the issue?

[ApsaraDhanasekar11]

Hi @fergiemcdowall , thanks for replying back. Please find the below example code and help us with details. Thanks!

const levelup = require('levelup');
const si = require('search-index');
const s3leveldown = require('s3leveldown');

const s3Store = await levelup(s3leveldown(bucketName, S3Client));

const idx = await si({
db: s3Store,
storeVectors: true
});

let data = [
{
_id: 'a',
description: 'Use template to list'
},
{
_id: 'b',
description: 'All versions and updates'
},
{
_id: 'c',
description: 'Final is the file name'
}
];

const result = await idx.PUT(data, {
storeVectors: true });

// results is :: [
{ _id: 'a', operation: 'PUT', status: 'CREATED' },
{ _id: 'b', operation: 'PUT', status: 'CREATED' },
{ _id: 'c', operation: 'PUT', status: 'CREATED' }
]

// ****** The above code creates the index as below::
{ key: 'description:file#1.00', value: [ 'c' ] }
{ key: 'description:final#1.00', value: [ 'c' ] }
{ key: 'description:list#1.00', value: [ 'a' ] }
{ key: 'description:name#1.00', value: [ 'c' ] }
{ key: 'description:template#1.00', value: [ 'a' ] }
{ key: 'description:updates#1.00', value: [ 'b' ] }
{ key: 'description:use#1.00', value: [ 'a' ] }
{ key: 'description:versions#1.00', value: [ 'b' ] }
{ key: '￮DOCUMENT_COUNT￮', value: 3 }
{
key: '￮DOC_RAW￮a￮',
value: { _id: 'a', description: 'Use template to list' }
}
{
key: '￮DOC_RAW￮b￮',
value: { _id: 'b', description: 'All versions and updates' }
}
{
key: '￮DOC_RAW￮c￮',
value: { _id: 'c', description: '"Final is the file name' }
}
{
key: '￮DOC￮a￮',
value: {
_id: 'a',
description: [ 'list#1.00', 'template#1.00', 'to#1.00', 'use#1.00' ]
}
}
{
key: '￮DOC￮b￮',
value: {
_id: 'b',
description: [ 'all#1.00', 'and#1.00', 'updates#1.00', 'versions#1.00' ]
}
}
{
key: '￮DOC￮c￮',
value: {
_id: 'c',
description: [ 'file#1.00', 'final#1.00', 'is#1.00', 'name#1.00', 'the#1.00' ]
}
}
{ key: '￮FIELD￮description￮', value: 'description' }
//// ************************* ///////

// ****** For the Search/ query : ******* //
const result = await indexedDb.QUERY( {
GET: {
FIELD: ['description'],
VALUE: {
GTE: 'versions',
LTE: 'versions'
},
}
}); ----> Tried other options like Query->(GET, SEARCH) , _SEARCH, _GET

But the result was ::
RESULT: [
{
_id: 'c',
_match: [
'description:file#1.00',
'description:final#1.00',
'description:name#1.00'
]
},
{
_id: 'a',
_match: [
'description:list#1.00',
'description:template#1.00',
'description:use#1.00'
]
},
{
_id: 'b',
_match: [ 'description:updates#1.00', 'description:versions#1.00' ]
}
],
RESULT_LENGTH: 3
} :: which is actually giving results of all previous alphabetical words from a-v (as "versions" begins with v)

While trying to identify the process flow, I noticed the GET function has internal implementation of db.createReadStream method which should actually filter the data according to the keywords passed in GTE & LTE. But looks like this is failing and instead bringing up the entire result set restricting upto the first character (alphabetic order)..