algolia / firestore-algolia-search Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
When creating a firestore doc it automatically creates an id for the document as known. later on, this object is indexed by algolia with it's own prop objectID. any idea to automatically transform this objectID to id because my app is based to work with the id prop and not objectID when getting search results from algolia? this would be very useful to avoid unnecessary duplicate code in my application. thank you
It would be wonderful to have a configuration option in the extension setup to allow passing the document reference path of the firestore document to the transform functions and algolia index. Perhaps as a key called firestore_path
This would allow users to use Algolia to search for documents across collections and then return the actual Firestore document on the front end.
Currently the only linking between Firestore Documents and Algolia Objects are the object ID (AKA Doc ID). This isn't enough on its own to link back to the original document. A full path of the document would allow for this to happen.
Hi,
I 'm trying to import existing Firestore documents using npx firestore-algolia-search
and following this guide , but I'm receiving this error:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
Cannot find module '@google-cloud/firestore'
NODE v10.23.3
NPM v6.14.11
In our Firestore collection, tags are listed as Arrays of Maps (objects), like the following document:
{...,
tags: [
TagId: "uYgvhNdciHcuK7ioGZEv",
Title: "SARS-CoV-2 (COVID-19)"
]}
How can I filter the searh results using these tags?
Hello devs,
First of all - thank you for such an amazing and time saving tool. It really helps us to move much faster with Fire + Algolia.
One thing I found missing is support for collectionGroups. It would extremely help as currently, we've to have global
collection
in order for it to sync with Algolia. However there are many instances as it's better to nest it under some collection
instead of keeping everything global
.
As a workaround, we have subcollection and collection of the same thing. Collection is used for algolia only, and subcollection for other things. It would be great to rely on only subcollection
in this case as it would take away this constant sync between main subcollection
and collection
which is only used in Algolia
.
Looking forward!
Cheers
When I deploy the algolia search in my app it works really well. But the other cloud functions which I have deployed works only when algolia search is not deployed. I get the following error in the logs
Unreachable hosts - your application id may be incorrect.
My application Id is correct and I am using the firebase blaze plan.
Version:
"algoliasearch": "^4.11.0",
"express": "^4.17.2",
"firebase-admin": "^9.12.0",
"firebase-functions": "^3.16.0",
Sharing my code below
algolia.js
const ALGOLIA_APP_ID = APP_ID;
const functions = require("firebase-functions");
const ALOGOLIA_ADMIN_KEY = API_KEY;
const ALGOLIA_INDEX_NAME = 'profile_card';
const client = algoliasearch(ALGOLIA_APP_ID, ALOGOLIA_ADMIN_KEY);
const index = client.initIndex(ALGOLIA_INDEX_NAME);
index.setSettings({ searchableAttributes: ['username', '_tags'], attributesForFaceting: ['searchable(_tags)'], removeStopWords: true });
exports.createPost = functions.firestore
.document('EUU/{euId}')
.onCreate(async (snap, context) => {
const newValue = snap.data();
newValue.objectID = snap.id;
return index.saveObject(newValue);
console.log('Finished');
});
exports.updatePost = functions.firestore
.document('EUU/{euId}')
.onUpdate(async (snap, context) => {
const afterUpdate = snap.after.data();
afterUpdate.objectID = snap.after.id;
return index.saveObject(afterUpdate);
});
exports.deletePost = functions.firestore
.document('EUU/{euId}')
.onDelete(async (snap, context) => {
const oldID = snap.id;
return index.deleteObject(oldID);
});
index.js
const functions = require("firebase-functions");
const admin = require('firebase-admin');
const algo = require('./algolia');
// const cashfree = require('./cashfree');
admin.initializeApp(functions.config().functions);
const options = {
priority: 'high'
}
exports.sendRequest = functions.firestore.document('EUU/{euId}/Requests/{id}').onCreate(async (snapshot, context) => {
const request = snapshot.data();
// console.log(context.params.euId);
admin.firestore().collection('mobileNotificationToken').doc(context.params.euId).get().then((doc) => {
const deviceTokens = doc.get('tokens');
// console.log(deviceTokens);
var payload = {
notification: {
title: 'Session request from ' + request.sender_name,
body: 'Time slot: ' + request.time_slot,
},
data: {
tag: 'request',
'reason': request.reason,
request_id: snapshot.id,
sender_id: context.params.euId,
},
};
return admin.messaging().sendToDevice(deviceTokens, payload, options).catch((error) => {
// console.log('Error sending message:', error);
console.error("Request Failed", error)
})
});
})
exports.paymentNotif = functions.firestore.document('EUU/{euId}/Requests/{id}').onUpdate(async (change, context) => {
const updatedStatus = change.after.data();
if (updatedStatus.status == 'Accepted') {
admin.firestore().collection('mobileNotificationToken').doc(updatedStatus.sender_id).get().then((doc) => {
const deviceTokens = doc.get('tokens');
console.log(deviceTokens);
var payload = {
notification: {
title: `Appointment Accepted by ${updatedStatus.receiver_name}`,
body: `Your appointment has been accepted`,
},
data: {
message: 'Your Session is booked',
tag: 'payment',
sender_id: context.params.euId,
'request_id': context.params.id,
},
};
const notificaiton = {
title: `Appointment Accepted by ${updatedStatus.receiver_name}`,
body: `Your appointment accepted`,
euId: context.params.euId,
payment_done: false,
reqId: context.params.id,
createdAt: admin.firestore.Timestamp.now(),
type: 'Type.Payment'
}
const docs = admin.firestore().collection('Seeker').doc(updatedStatus.sender_id);
const EUdocs = admin.firestore().collection('EUU').doc(updatedStatus.sender_id);
EUdocs.get().then((snap) => {
if (snap.exists) {
EUdocs.collection('notifications').add(notificaiton).then(() => console.log('notification stored to db'));
} else {
docs.collection('notifications').add(notificaiton).then(() => console.log('notification stored to db'));
}
});
return admin.messaging().sendToDevice(deviceTokens, payload, options)
.then((response) => {
console.log("Successfully sent message: ", response);
}).catch((error) => {
console.log('Error sending message:', error);
})
});
// const id = change.id;
// return change.after.ref.set({ id: { 'payment': 'pending' } });
} else {
console.log('DIDNT WORK');
}
})
exports.sendNotification = functions.firestore.document('EUU/{euId}/Notifications/{id}').onCreate(async (snapshot, context) => {
const request = snapshot.data();
console.log(context.params.euId);
admin.firestore().collection('mobileNotificationToken').doc(context.params.euId).get().then((doc) => {
const deviceTokens = doc.get('tokens');
console.log(deviceTokens);
var payload = {
notification: {
title: 'Push Notification',
body: 'Hello',
},
data: {
message: 'Your Session is booked',
tag: 'notification',
// receiver_id: context.params.euId,
},
};
admin.messaging().sendToDevice(deviceTokens, payload, options)
.then((response) => {
console.log("Successfully sent message: ", response);
}).catch((error) => {
console.log('Error sending message:', error);
})
});
})
exports.paymentSuccess = functions.firestore.document('Seeker/{seekerId}/Orders/{id}').onUpdate(async (change, context) => {
const updatedStatus = change.after.data();
if (updatedStatus.txStatus == 'SUCCESS') {
admin.firestore().collection('mobileNotificationToken').doc(updatedStatus.euId).get().then((doc) => {
const deviceTokens = doc.get('tokens');
console.log(deviceTokens);
var payload = {
notification: {
title: 'Session Confirmation',
body: 'Your session has been confirmed',
},
data: {
message: 'Your Session is booked',
},
};
return admin.messaging().sendToDevice(deviceTokens, payload, options)
.then((response) => {
console.log("Successfully sent message: ", response);
}).catch((error) => {
console.log('Error sending message:', error);
})
});
admin.firestore().collection('mobileNotificationToken').doc(updatedStatus.seekerId).get().then((doc) => {
const deviceTokens = doc.get('tokens');
console.log(deviceTokens);
var payload = {
notification: {
title: 'Session Confirmation',
body: 'Your session has been confirmed',
},
data: {
message: 'Your Session is booked',
},
};
return admin.messaging().sendToDevice(deviceTokens, payload, options)
.then((response) => {
console.log("Successfully sent message: ", response);
}).catch((error) => {
console.log('Error sending message:', error);
})
});
// const id = change.id;
// return change.after.ref.set({ id: { 'payment': 'pending' } });
} else {
console.log('PAYMENT SUCCESS DIDNT WORK');
}
});
exports.createPost = algo.createPost;
exports.deletePost = algo.deletePost;
exports.updatePost = algo.updatePost;
I used the "import existing documents" section from firebase extension and it gives the following error
{"severity":"ERROR","message":"Error when performing Algolia index Error: Value for argument \"collectionPath\" is not a valid resource path. Path must be a non-empty string.\n at Object.validateResourcePath (/Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/@google-cloud/firestore/build/src/path.js:414:15)\n at Firestore.collection (/Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/@google-cloud/firestore/build/src/index.js:561:16)\n at retrieveDataFromFirestore (/Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:45:42)\n at /Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:75:9\n at Interface._onLine (readline.js:335:5)\n at Interface._line (readline.js:666:8)\n at Interface._ttyWrite (readline.js:1010:14)\n at ReadStream.onkeypress (readline.js:213:10)\n at ReadStream.emit (events.js:315:20)\n at emitKeys (internal/readline/utils.js:345:14)"}
Hello, I'm having trouble with importing a firestore collection using the following script:
LOCATION=us-central1
PROJECT_ID=hidden
ALGOLIA_APP_ID=hidden
ALGOLIA_API_KEY=hidden
ALGOLIA_INDEX_NAME=test
COLLECTION_PATH=activities
FIELDS=name;title;description
GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>
npx firestore-algolia-search
Terminal output:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAPIKey":"********","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y
{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read properties of undefined (reading 'split')\n at retrieveDataFromFirestore (/Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:76:65)\n at /Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:93:9\n at Interface._onLine (node:readline:485:5)\n at Interface._line (node:readline:864:8)\n at Interface._ttyWrite (node:readline:1216:14)\n at ReadStream.onkeypress (node:readline:288:10)\n at ReadStream.emit (node:events:520:28)\n at emitKeys (node:internal/readline/utils:358:14)\n at emitKeys.next ()\n at ReadStream.onData (node:internal/readline/emitKeypressEvents:61:36)"}
npm/npx version is 8.3.1
Repeatable Steps:
Thanks in advance please help!!!!!!!
We currently sync multiple collections into different indices in Algolia. From reviewing the implementation here, it does not seem like that's currently supported. The Algolia website (https://www.algolia.com/developers/firebase-search-extension) does list that it's possible ("Set up or combine multiple collections, or sub-collections, into a single search index"), so am I missing something?
Error: HTTP Error: 400, Secret ID ***my-api-key***
is invalid, must be of form projects/*/secrets/*/versions/*
Created the key as recomended, and have previously installed working extension for a collection with this key, now trying to add another for another collection, I get this error.
Hello! We currently have some cloud functions that mutates our Algolia indices when relevant documents are created, updated, or deleted, and we are excited to discover that this Firebase extension can help remove some of this code. However, one of our indices change one field to a more search-friendly form, i.e. turning a serialized JSON of rich text into plaintext. We would like to move all of our function code to using this extension, but this is currently blocking us from doing so. Thanks!
Hello everyone,
I've been trying to run the restore script for firestore-algolia-search
. According to other issues, decided to run it from a bash script:
#!/bin/bash
LOCATION=us-central1\
PROJECT_ID=project-id\
ALGOLIA_APP_ID=111111\
ALGOLIA_API_KEY=111111\
ALGOLIA_INDEX_NAME=index\
COLLECTION_PATH=path\
FIELDS=fields\
GOOGLE_APPLICATION_CREDENTIALS=./key.json\
npx firestore-algolia-search
But then the output is firestore-algolia-search: command not found
. I'm writing this issue here because other npx script would work in bash scripts. npx is installed globally and the command is working outside of bash (but no with parameters declared beforehand)
Thanks a lot
I am able to successfully configure one collection with one indices. but I want to configure multiple collection and each collection would have corresponding indice. Is it possible in using this extension?
Here's what happening :
before deletion I tested by changing the field data and the changes are reflected on index. Just that the firestore field deletion part does not delete on algolia index.
Am I missing something or is it an issue??
Hi, I posted this on the Algolia forum: https://discourse.algolia.com/t/how-to-handle-firestore-reference-types-in-algolia-currently-they-store-sensitive-and-unnecessary-information-when-imported/12685
But after having reviewed the code of this tool, I think it's more appropriate to open an issue here.
Quick summary:
Reference
field value from Firestore is a JSON object with a bunch of extraneous data (rather than, e.g. just a string with the path or an array of the path segments)_firestore.settings.credentials
sub-field that includes a private SSH key!So after looking at the code, it looks like there's no processing at all happening on the extracted data, and it seems like it would be relatively straightforward to implement something. Like if(value.dataType == DocumentReference) payload[field] = value._path.segments;
- I don't know the exact syntax or field names off the top of my head but something like that should work I think.
I don't mind implementing this but maybe someone from Algolia should specify what the desired behaviour is (should it preserve any of the random non-private information? should it be an array like segments right now or just a string (i.e. segments.join('/')
)?
indexing fields : channelName,thumbnail,title,duration,id,date,description,category,isVisible,videoScore,views (from firebase ui)
fields from log : [ 'fields: channelName,thumbnail,title,duration,id,date,description,category,isVisible,videoScore,views,' ]
but it is showing error Error when performing Algolia index Error: Value for argument "field" is not a valid field path.
It was working fine before.
I've followed all the instructions and ran following script to insert Firestore data to Algolia indexes.
PROJECT_ID=<PROJECT_ID>\
ALGOLIA_APP_ID=<ALGOLIA_APP_ID>\
ALGOLIA_INDEX_NAME=<INDEX_NAME>\
COLLECTION_PATH=<COLLECTION_PATH>
TRANSFORM_FUNCTION=<FIREBASE_FUNCTION_NAME>\
GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>\
npx firestore-algolia-search
But I keep getting error as below.
{"severity":"ERROR","message":"Error when performing Algolia index FetchError: invalid json response body at
https://accounts.google.com/ServiceLogin?<URL_WITH_SOME_INFO> reason: Unexpected token < in JSON at
position 1\n at /Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/node-fetch/lib/index.js:273:32\n
at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Object.exports.default
(/Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/transform.js:29:26)\n
at async Object.extract [as default] (/Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-
search/lib/extract.js:42:21)\n at async processQuery
(/Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:54:29) {\n type:
'invalid-json'\n}"}
For "GOOGLE_APPLICATION_CREDENTIALS", I went to Firebase Console > Settings > Service Accounts and generated a new key for "Firebase Admin SDK" for this project and used the JSON file.
When I manually update a document on Firestore, it does add it on Algolia index.
So I think my json data could be invalid... but why?
Hi, I have a very similar issue to:
https://github.com/algolia/firestore-algolia-search/issues/43#event-4954001620
and
https://github.com/algolia/firestore-algolia-search/issues/42#issuecomment-869508635
Ignoring the error messages like for issue #42 and using the one liner command provided in #43 did not solve my problem.
Here is the command I ran:
LOCATION=us-central1 ALGOLIA_APP_ID=D********* ALGOLIA_API_KEY=b****************0 ALGOLIA_INDEX_NAME=dev_foodly COLLECTION_PATH=Restaurants/{restaurantId}/Locations FIELDS=restaurantName GOOGLE_APPLICATION_CREDENTIALS=foodlyServiceAccountKeyDev.json npx [email protected]
And this is my output:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAppId":"D7MMB2JA20","algoliaAPIKey":"********","algoliaIndexName":"dev_foodly","collectionPath":"Restaurants/{restaurantId}/Locations","fields":"restaurantName","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y
✨ Done in 41.50s.
LOCATION=us-central1
PROJECT_ID=nexiev
ALGOLIA_APP_ID=
ALGOLIA_API_KEY=
ALGOLIA_INDEX_NAME=ev
COLLECTION_PATH=ev
GOOGLE_APPLICATION_CREDENTIALS=nestra-firebase-adminsdk-jufxu-6f8cfa91bb.json
npx firestore-algolia-search
I ran the above command in Powershell. But I got the following error and existing records were not imported.
WARNING: The backfill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y
{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read property 'split' of undefined\n at retrieveDataFromFirestore (C:\Users\Afzal\AppData\Roaming\npm-cache\_npx\7620\node_modules\firestore-algolia-search\lib\import\index.js:72:65)\n at C:\Users\Afzal\AppData\Roaming\npm-cache\_npx\7620\node_modules\firestore-algolia-search\lib\import\index.js:80:9\n at Interface._onLine (readline.js:327:5)\n at Interface._line (readline.js:658:8)\n at Interface._ttyWrite (readline.js:999:14)\n at ReadStream.onkeypress (readline.js:205:10)\n at ReadStream.emit (events.js:315:20)\n at emitKeys (internal/readline/utils.js:335:14)\n at emitKeys.next ()\n at ReadStream.onData (readline.js:1133:36)"}
It might be a bit of a hassle for users to clone the repo and run the import script. One option is to make it runnable via an npx command like npx github:algolia/firestore-algolia-search/...
I only want to index records that are published. I have a boolean isPublished and I would only like records from my collection that are published to be indexed.
I've tried using the transform function but I get an invalid JSON error if I try to return anything other than the data.
Does the extension allow me to filter records before they are sent to Algolia?
Suggest using validationRegex
so that someone installing the extension can be notified early if an entered param value is incorrect.
COLLECTION_PATH
: here's an example validationRegex.
Everything works as expected and syncs perfectly except booleans. If a new document is created it gets loaded in. However if only the boolean field gets updated, it doesn't update it. The field is called 'hidden' so not sure if that's a problem.
The logs for my cloud functions do not show an error. However the bool does not show up inside the data being transmitted. The field 'hidden' is in my indexed fields settings for the plugin.
Would love to hear how to continue.
Version: 0.4.1
Edit: apparently the same thing goes for numbers. I tried to change the field type to string and the 'hidden' value syncs perfectly. So it really depends on the type and not on a configuration setting.
I've been using this extension for a few days now and so far it's working very well. However, although creating a new Firebase document correctly indexed it in Algolia, updating the document did not.
After many hours searching for a possible cause, I noticed there were spaces between each comma, the Firebase Function gets triggered, but nothing is updated in the Algolia counter part. Since indexed fields can't contain spaces (I think?) maybe the function should remove all spaces when parsing the Indexable Fields.
If there's a quirk I'm missing, at least I hope this helps someone like me with OCD who needs to put a space after each comma 😁
I'm trying to set this extension up with the emulator, and I did manage to download the extension, and subsequently launch it, but the problem is that the import script will obviously take the data from the online project I downloaded the extension from, and I'm looking for a way for algolia to use my local firestore db instead.
Starting with the first import (I'd create a separate Application in Algolia for it and use its details when configuring), and subsequently ensuring that when data is added to my local firestore db it triggers the onWrite
in the extension, I'd really appreciate some guidance.
Things I tried:
I replaced this:
functions.handler.firestore.document
...with this (as recommended by SO answers):
functions.firestore.document(`projects/${process.env.PROJECT_ID}/databases/(default)/documents/${process.env.COLLECTION_PATH}/{documentID}`)
(envs do exist and do not appear to report errors during init)
I also tried adding default exporting the executeIndexOperation
variable from index
to no avail.
What happens on record creation is that the log.init()
function does appear to kick in, but nothing inside onWrite
(not even log.start()
).
In the API key description, for first time users who just want to set up as quickly as possible, when I was asked for API key, I have two options in algolia api keys - Search-Only API key or Admin API key. In the description it says Do not use the admin API key, so naturally revert to the other available API key which is Search Only. Since it says "We recommend creating new key", - I assume it's best practice, but to play around my "Search-Only" key should work just fine. Wrong. I set up the index, and cannot get records no matter what i do. Until i go to "Search API Logs" and see all of my POST requests in the 400 code. So now i see that if i do not create a new api key it will never work.
Suggestion change the wording to the likes of:
Algolia API Key Description
We require API key with “addObject”, “deleteObject”, “listIndexes”, “deleteIndex”, “editSettings”, and “settings” permissions. Do not use the Admin API key or Search Only API Key.
Context: We have our own Algolia sync cloud function.
Reading through your sources I can see you will have a race condition issue due to hot and cold-start functions. It's not the easiest to detect, but can cause havoc in production.
I'm not at the office till tomorrow, but I will record a demonstration of the issue and simple solution.
Hi, Thank you for all your help!
I started to use this extension lately and realized the value of the timestamp will be changed to a number of milliseconds.
ref: https://github.com/algolia/firestore-algolia-search/blob/main/functions/src/processors.ts#L62
I need to do anything another when decoding the struct which includes another type like this.
Are there some reasons to have it as a number type?
Moreover, my suggestion is to use a dictionary type like GeoPoint
What do you think?
Sorry if it's my misunderstanding, thank you!
Is there a way to skip/ignore particular document updates, i.e. filtering. For example if i have a collection of documents, each document containing a 'type' property, and I'm only interested in document updates where type == 'sport', how could i achieve this?
I thought of using the transform function to check the 'type' and return an empty response if it's not a value im concerned with, e.g. 'sport' in this example, but looking at the code here, I'm not sure that would be the correct approach.
Any suggestions appreciated, thanks
As far as I understand, the extensions should only bridge new data to Algolia, which stores the received data in records and sends those back to the user whenever a query is performed.
However, I have noticed that whenever I perform a query on my client (iOS in this case) the read usage of my Cloud Firestore increases by 1 for every hit of the query (which results in an enormous quantity of reads).
Is this an intentional behavior?
Thanks.
I could not find such a feature in the docs. Can I configure the function to only create an index in Algolia, if for example a 'verified' property is 'true'?
We upgraded to:
Search with Algolia
algolia/[email protected]
We are seeing this error:
ext-firestore-algolia-search-executeIndexOperation
Error when performing Algolia index { FetchError: invalid json response body at https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://uc.appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://xxxxxx/extAlgoliaSearchCustomers reason: Unexpected token < in JSON at position 1 at /workspace/node_modules/node-fetch/lib/index.js:272:32 at process._tickCallback (internal/process/next_tick.js:68:7) message: 'invalid json response body at https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://uc.appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://xxxxxxx/extAlgoliaSearchCustomers reason: Unexpected token < in JSON at position 1', type: 'invalid-json' }
When trying to run an initial import of my data from Firestore, I get the error {"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
. I've seen other people have encountered this but the resolution isn't clear.
I'm running the command provided after installing the extension on Firebase and I believe my API key has the correct access.
Unlike the other reports, it doesn't start iterating over my data or importing it into the index.
Have I missed something?
I realise that the reason nothing was being imported is that I have subcollections within my collections. It looks like:
> [Users]
> {USER_ID}
> [Objects]
> {OBJECT_ID}
If I set my configuration to index Users/{parentId}/Objects
then I can see the records imported into my index. My question now is what's the best practice for indexing them? I need some of the User information on the record so I can ensure the right people see the right objects. Is there a recognised 'correct' way of doing this?
Hi,
Currently, I have deployed the algolia-firestore extension along with a transform function as explained in the README. With my current deployment, I get this error:
Error when performing Algolia index { FetchError: invalid json response body at https://accounts.google.com/ServiceLogin?passive=true&continue=https://el.appengine.google.com/_ah/conflogin%3Fstate%3D~< some long key >
reason: Unexpected token < in JSON at position 1
So looks like Algolia app is not properly authenticated before calling the firebase transform function. This results in the app skipping the transform and just writing the firebase doc as it is.
The README Mentions:
The Transform Firebase Function should be set up to unauthenticated users at this time
But there doesn't seem to be any option to set https callable functions to be unauthenticated in firebase. Can you please help understand what exactly I need to do for the transformation to work?
Currently the retrieveDataFromFirestore
function of the import script takes the last segment (the last subcollection) of the collection path and runs a collection group query on it.
This leads to documents that are not part of the config collection path being imported into Algolia.
For eg: if my firestore architecture is as follows:
And if my collection path stated in the config is users/{x}/records
. The import script would currently import records under both Users
as well as TestUsers
. This is not the expected behavior and could potentially lead to incurring unexpected costs.
Perform the collection group query as normal and then filter the results by matching the doc.ref.path
to the config.collectionPath
. A simple for
loop (incremented by 2) matching each collection segment would let you know if the document returned is part of the collection path stated in the config and can continue with the processQuery
flow.
Attempting to run the script so I can synchronize my Firestore collection with the Algolia index, I just can't get it running. I have tried multiple versions of node: latest, LTS, and a few other random versions to no success. Have also tried it on both macOS and Ubuntu 20.04 (on WSL2). The error message and trace is:
npx firestore-algolia-search
node:internal/modules/cjs/loader:927
throw err;
^
Error: Cannot find module './lib/import/index'
Require stack:
- /home/mbrown/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/cli.js
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:924:15)
at Function.Module._load (node:internal/modules/cjs/loader:769:27)
at Module.require (node:internal/modules/cjs/loader:996:19)
at require (node:internal/modules/cjs/helpers:92:18)
at Object.<anonymous> (/home/mbrown/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/cli.js:3:1)
at Module._compile (node:internal/modules/cjs/loader:1092:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
at Module.load (node:internal/modules/cjs/loader:972:32)
at Function.Module._load (node:internal/modules/cjs/loader:813:14)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/home/mbrown/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/cli.js'
]
}
New to Algolia and trying to get the Firebase extension up-and-running, but having some issues with the backfill script.
This is the script I am running:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAPIKey":"********","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y
{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read property 'split' of undefined\n at retrieveDataFromFirestore (/Users/simon/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:72:65)\n at /Users/simon/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:80:9\n at Interface._onLine (readline.js:340:5)\n at Interface._line (readline.js:671:8)\n at Interface._ttyWrite (readline.js:1015:14)\n at ReadStream.onkeypress (readline.js:213:10)\n at ReadStream.emit (events.js:376:20)\n at emitKeys (internal/readline/utils.js:345:14)\n at emitKeys.next (<anonymous>)\n at ReadStream.onData (readline.js:1149:36)"}
I'm using the following command:
LOCATION=europe-west2\
PROJECT_ID=my-project-name\
ALGOLIA_APP_ID=0000000000\
ALGOLIA_API_KEY=0000000000\
ALGOLIA_INDEX_NAME=my_index\
COLLECTION_PATH=my-collection\
FIELDS=field1,field2,field3\
GOOGLE_APPLICATION_CREDENTIALS=/...json\
npx firestore-algolia-search
I tried reinstalling the extension and still having zero joy.
Any idea where I might be going wrong?
It's not possible to have a subcollection tied to the function, it has to be on the root or it won't work.
I would like to use this extension, however my organization at work has put in restrictions around where secrets can be created in the Google Secret Manager so simply creating a secret using this extension fails:
Error: HTTP Error: 400, Constraint constraints/gcp.resourceLocations violated for [redacted] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.
A solution to this problem would be to allow myself to create the secret ahead of time and skip this step. Interested in hearing your thoughts.
When I update a field value within a Firestore document to null, the extension executes, but does not actually update the value in the Algolia index. I would assume the Algolia value would be deleted from the index, as null values are not transferred from Firestore to Algolia.
We recently ran a script to add a field to all pre-existing documents, and even though this field was not part of the indexable fields list, documents that were previously not in the index were added to it. We would like only for documents to be added to the index if they have all the indexable fields, and only if the indexable fields have changed.
Is it possible to transform the data before loading it to Algolia when using the script npm firestore-algolia search
?
Hi, am really new using algolia, i don't know but i have following all step in firebase extension doc. I install algolia through firebase extension.
I created new api key and divert it to proper indices(by creating new indices). But i got error in algolia functions when i inserted new record from my website, what logs said is:
Error when performing Algolia index {"transporterStackTrace":[{"triesLeft":3,"response":{"isTimedOut":false,"status":403,"content":"{"message":"Index not allowed with this API key","status":403}"},"host":{"accept":2,"url":"NZ3GVUOMSD.algolia.net","protocol":"https"},"request":{"url":"https://NZ3GVUOMSD.algolia.net/1/indexes/algoId/batch?x-algolia-agent=Algolia%20for%20JavaScript%20(4.8.6)%3B%20Node.js%20(10.24.1)%3B%20firestore_integration%20(0.5.1)","connectTimeout":2,"data":"{"requests":[{"action":"partialUpdateObject","body":{"objectID":"crSuskBJaQpfOAODUNey","title":"Test 5","category":"kesehatan","lastmodified":{"_operation":"IncrementSet","value":1626963102502}}}]}","method":"POST","headers":{"x-algolia-application-id":"NZ3GVUOMSD","x-algolia-api-key":"*****","content-type":"application/x-www-form-urlencoded"},"responseTimeout":30}}],"name":"ApiError","status":403}
Then, i checked algolia dashboard and there is nothing there, still empty. Can somebody explain to me what happened here? Thanks
I am getting the following message after setting up the extension through firebase
{ "message": "Not enough rights to update an object near line:1 column:1893", "status": 400 }
I assumed that all permission would be setup on install. App ID and API Key are correct. Did I miss a step?
After installing the "Search with Algolia" Firebase extension, the extension page has three tabs:
1- How this extension works
2- Extension configuration
3- APIs and resources
I followed the instructions in first tab "How this extension works". I've copied the instructions that I get on this page below for your reference.
After running npx firestore-algolia-search
, the first lines of console logs are as follows.
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAppId":<YOUR_ALGOLIA_APP_ID>,"algoliaAPIKey":"********","algoliaIndexName":"nodesIndex","collectionPath":"nodes","fields":"admin,content,createdAt,nodeType,tags,title","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N):
I answered "y" to this question. Then, it generated a lot of logs but there is no index record generated on Algolia.
You can test out this extension right away:
Go to the Cloud Firestore tab.
If it doesn’t exist already, create a collection called nodes.
Create, update, or delete a document in the nodes collection. Go to Algolia’s dashboard and verify in Algolia that a record is created, updated, or deleted in the nodesIndex index for application id ................
Using the extension
This extension listens to the Cloud Firestore collection nodes. If you create, update, or delete a document within that collection, this extension will:Indexes the document and send all the fields or configured fields defined in the extension.
or, removes the record from Algolia index if the document is deleted.
(Optional) Import existing documents or Reindex after configuration changes
This extension starts monitoring the nodes collection after a successful installation. Any existing documents created before the extension installation can be back-filled into your Algolia nodesIndex Index using the import script. Also, you will need to run the import script if you change the Indexable Fields, Index Name, Application Id, and/or API Key configuration.The import script will read all existing documents in the nodes collection and insert them into the Algolia nodesIndex index.
Important notes
You must run the import script over the entire collection after installing the Search with Algolia extension; otherwise you will have missing records in your Algolia nodesIndex index.
lastmodified attributed is added to all Algolia records to eliminate race condition issues related from the Cloud Function cold start.
Run the script
The script will use the extension configuration before the import process starts.Run the import process using npx.
Make sure that you’ve installed the required tools to run the import script:
To access the npx command tools, you need to install Node.js.
If you use npm v5.1 or earlier, you need to explicitly install npx. Run npm install --global npx.
The import script uses Application Default Credentials to communicate with Firebase.
Please follow the instructions to generate a key for your service account.Execute the below command:
Update the path to the Google Application credentials.
Clear out the FIELDS param if it contains { unspecified parameter } in the command below since it’s an invalid value.
LOCATION=us-east4
ALGOLIA_APP_ID=<YOUR_ALGOLIA_APP_ID>
ALGOLIA_API_KEY=<YOUR_API_KEY>
ALGOLIA_INDEX_NAME=nodesIndex
COLLECTION_PATH=nodes
FIELDS=admin,content,createdAt,nodeType,tags,title
GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>
npx firestore-algolia-search
Monitoring
As a best practice, you can monitor the activity of your installed extension, including checks on its health, usage, and logs.
Wondering if there is any plan to add the extension for Firebase realtime database. Thanks!
Current the transform function returns the json payload from the transform endpoint.
Currently it seems that in order to skip a record it'd be best to throw on the callable function, however this would create a false positive log. Instead, is it worth returning an empty object and checking this on the response? Happy to make a PR if so.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.