Giter Site home page Giter Site logo

firestore-algolia-search's Introduction

Search Firestore with Algolia

Author: Algolia (https://www.algolia.com)

Description: Enable full text search of your Cloud Firestore data with Algolia.

Details: Use this extension to index your Cloud Firestore data to Algolia and keep it synced.

You can then configure your relevance using the Algolia dashboard or API clients. From there, you can use Algolia's front-end libraries to incorporate search components into your Firebase app's UI.


We welcome bug reports and feature requests as well as pull requests in this GitHub repository.

Firebase CLI

firebase ext:install algolia/firestore-algolia-search --project=<your-project-id>

Learn more about installing extensions in the Firebase Extensions documentation: console, CLI


Details

Use this extension to index your Cloud Firestore data to Algolia and keep it synced. The extension is applied and configured on a Firestore collection or subcollection.

This extension listens for changes on the specified collection. If you add a document, the extension indexes it as a record in Algolia. The extension only indexes the fields defined in the extension configuration and uses the document Id as the Algolia object Id.

Anytime you update a document, the extension propagates the update to the corresponding Algolia record. If you delete a document, the extension removes the corresponding Algolia record.

Additional setup

Before installing this extension, make sure that you've set up:

Billing

This extension uses the following Firebase services which may have associated charges:

  • Cloud Firestore
  • Cloud Functions

This extension also uses the following third-party services:

You are responsible for any costs associated with your use of these services.

Note from Firebase

To install this extension, your Firebase project must be on the Blaze (pay-as-you-go) plan. You will only be charged for the resources you use. Most Firebase services offer a free tier for low-volume use. Learn more about Firebase billing.

You will be billed a small amount (typically less than $0.10) when you install or reconfigure this extension. See Cloud Functions under Firebase Pricing for a detailed explanation.

Configuration Parameters

Cloud Functions Location: Where do you want to deploy the functions created for this extension? You usually want a location close to your database. For help selecting a location, refer to the location selection guide.

  • Database Id: Enter the database id that contains the collection(s) that you want to monitor? Firebase supports multiple databases in a project. In order to use a non default database, enter the database Id.

  • Collection Path: What is the path to the Cloud Firestore collection where the extension should monitor for changes? For subcollection, the syntax is parent_collection/{parentId}/target_collection. (please note, there is not depublication process on subcollections).

  • Fields: What document fields should be indexed to provide the best search experience? For more information on which fields to index to Algolia, see the Algolia documentation on records. This can be a comma separated list or left blank to index all fields. For performance reasons, record size is limited. If you're receiving errors that your records are too large, refer to the reducing record size documentation.

  • Force Data Sync: Are you facing data inconsistent issues possibly due to race conditions? This setting will make sure the function is using the latest data from Firestore by making another read call before processing and sending to Algolia.

  • Algolia Application ID: What is the Algolia Application Id? This is the Algolia application you want to index your data to. You can find your credentials including application ID on your Algolia dashboard, under the API keys tab.

  • Algolia API Key: What is your Algolia API key? We recommend creating a new API key with "addObject", "deleteObject", "listIndexes", "deleteIndex", "editSettings", and "settings" permissions. Do not use the Admin API key.

  • Algolia Index Name: What is the Algolia index name? This is the name of the Algolia index where the records will be persisted. Refer to naming your index for more information.

  • Alternative Object Id: Are you looking to use another Firestore document property or document path as the Algolia ObjectID? Specify an alternative Firestore document property to use for the Algolia record Id. Use (path) if the document path is desired as the Algolia ObjectID. The default is the Firestore document Id. If you set this property, make sure to clear out the Algolia Index since the ObjectID will be different resulting in duplicate records.

  • Transform Function Name (experimental): What is the Firebase Cloud Function Name? This is the name of the Firestore Cloud Function for transforming the data before transmitting to Algolia for indexing. This function should be deployed to the same Firebase Project and Location as the Firestore/Algolia extension. Refer to Call functions for your app. Below is an example of a Transform function used for my testing:

      import * as functions from "firebase-functions";
    
      const doStuffToData = (payload: any) => {
        return {
        ...payload,
        "hello": "world",
        };
      };
    
      export const helloWorld = functions.https.onCall((payload) => {
        const transformedData = doStuffToData(payload);
        return transformedData;
      });

    Note: The Transform Firebase Function should be set up to unauthenticated users at this time.

Cloud Functions

  • executeIndexOperation: Firestore document-triggered function that creates, updates, or deletes data in Algolia.

firestore-algolia-search's People

Contributors

aseidma avatar cabljac avatar e-krebs avatar haroenv avatar i14h avatar jhuleatt avatar joehan avatar meschreiber avatar nikkothari22 avatar smomin avatar smomin-bd avatar thatfiredev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

firestore-algolia-search's Issues

Error: Cannot find module './lib/import/index'

Attempting to run the script so I can synchronize my Firestore collection with the Algolia index, I just can't get it running. I have tried multiple versions of node: latest, LTS, and a few other random versions to no success. Have also tried it on both macOS and Ubuntu 20.04 (on WSL2). The error message and trace is:

npx firestore-algolia-search
node:internal/modules/cjs/loader:927
  throw err;
  ^

Error: Cannot find module './lib/import/index'
Require stack:
- /home/mbrown/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/cli.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:924:15)
    at Function.Module._load (node:internal/modules/cjs/loader:769:27)
    at Module.require (node:internal/modules/cjs/loader:996:19)
    at require (node:internal/modules/cjs/helpers:92:18)
    at Object.<anonymous> (/home/mbrown/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/cli.js:3:1)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/mbrown/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/cli.js'
  ]
}

Best practice for indexing sub collections

When trying to run an initial import of my data from Firestore, I get the error {"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}. I've seen other people have encountered this but the resolution isn't clear.

I'm running the command provided after installing the extension on Firebase and I believe my API key has the correct access.

Unlike the other reports, it doesn't start iterating over my data or importing it into the index.

Have I missed something?

I realise that the reason nothing was being imported is that I have subcollections within my collections. It looks like:

  > [Users]
    > {USER_ID}
      > [Objects]
        > {OBJECT_ID}

If I set my configuration to index Users/{parentId}/Objects then I can see the records imported into my index. My question now is what's the best practice for indexing them? I need some of the User information on the record so I can ensure the right people see the right objects. Is there a recognised 'correct' way of doing this?

Error fetching records from Firestore to Algolia using Firebase Extensions

I've followed all the instructions and ran following script to insert Firestore data to Algolia indexes.

  PROJECT_ID=<PROJECT_ID>\
  ALGOLIA_APP_ID=<ALGOLIA_APP_ID>\
  ALGOLIA_INDEX_NAME=<INDEX_NAME>\
  COLLECTION_PATH=<COLLECTION_PATH>
  TRANSFORM_FUNCTION=<FIREBASE_FUNCTION_NAME>\
  GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>\
  npx firestore-algolia-search

But I keep getting error as below.

{"severity":"ERROR","message":"Error when performing Algolia index FetchError: invalid json response body at
 https://accounts.google.com/ServiceLogin?<URL_WITH_SOME_INFO> reason: Unexpected token < in JSON at
 position 1\n    at /Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/node-fetch/lib/index.js:273:32\n    
at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Object.exports.default 
(/Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/transform.js:29:26)\n    
at async Object.extract [as default] (/Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-
search/lib/extract.js:42:21)\n    at async processQuery 
(/Users/sichoi/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:54:29) {\n  type: 
'invalid-json'\n}"}

For "GOOGLE_APPLICATION_CREDENTIALS", I went to Firebase Console > Settings > Service Accounts and generated a new key for "Firebase Admin SDK" for this project and used the JSON file.

When I manually update a document on Firestore, it does add it on Algolia index.

So I think my json data could be invalid... but why?

Better descriptions for params in extension config

Descriptions for some params are too short and assume the user knows the details about Algolia which might not be the case. A better more descriptive message would help. A link to the related docs would also help (for all params)

image

Configuring, when to store a document

I could not find such a feature in the docs. Can I configure the function to only create an index in Algolia, if for example a 'verified' property is 'true'?

Unable to sync firestore record

I used the "import existing documents" section from firebase extension and it gives the following error

{"severity":"ERROR","message":"Error when performing Algolia index Error: Value for argument \"collectionPath\" is not a valid resource path. Path must be a non-empty string.\n    at Object.validateResourcePath (/Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/@google-cloud/firestore/build/src/path.js:414:15)\n    at Firestore.collection (/Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/@google-cloud/firestore/build/src/index.js:561:16)\n    at retrieveDataFromFirestore (/Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:45:42)\n    at /Users/physcocode/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:75:9\n    at Interface._onLine (readline.js:335:5)\n    at Interface._line (readline.js:666:8)\n    at Interface._ttyWrite (readline.js:1010:14)\n    at ReadStream.onkeypress (readline.js:213:10)\n    at ReadStream.emit (events.js:315:20)\n    at emitKeys (internal/readline/utils.js:345:14)"}

Algolia API Key: creation failure

I would like to use this extension, however my organization at work has put in restrictions around where secrets can be created in the Google Secret Manager so simply creating a secret using this extension fails:

Error: HTTP Error: 400, Constraint constraints/gcp.resourceLocations violated for [redacted] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.

A solution to this problem would be to allow myself to create the secret ahead of time and skip this step. Interested in hearing your thoughts.

Algolia API Key Description

In the API key description, for first time users who just want to set up as quickly as possible, when I was asked for API key, I have two options in algolia api keys - Search-Only API key or Admin API key. In the description it says Do not use the admin API key, so naturally revert to the other available API key which is Search Only. Since it says "We recommend creating new key", - I assume it's best practice, but to play around my "Search-Only" key should work just fine. Wrong. I set up the index, and cannot get records no matter what i do. Until i go to "Search API Logs" and see all of my POST requests in the 400 code. So now i see that if i do not create a new api key it will never work.

Suggestion change the wording to the likes of:

Algolia API Key Description
We require API key with “addObject”, “deleteObject”, “listIndexes”, “deleteIndex”, “editSettings”, and “settings” permissions. Do not use the Admin API key or Search Only API Key.

image

image

image

Firestore DocumentReference path in Algolia Object

It would be wonderful to have a configuration option in the extension setup to allow passing the document reference path of the firestore document to the transform functions and algolia index. Perhaps as a key called firestore_path

This would allow users to use Algolia to search for documents across collections and then return the actual Firestore document on the front end.

Currently the only linking between Firestore Documents and Algolia Objects are the object ID (AKA Doc ID). This isn't enough on its own to link back to the original document. A full path of the document would allow for this to happen.

Handling of 'reference' field type in Firestore - currently it exposes sensitive information and is unintuitively formatted

Hi, I posted this on the Algolia forum: https://discourse.algolia.com/t/how-to-handle-firestore-reference-types-in-algolia-currently-they-store-sensitive-and-unnecessary-information-when-imported/12685

But after having reviewed the code of this tool, I think it's more appropriate to open an issue here.

Quick summary:

  • It seems that the format returned when fetching a Reference field value from Firestore is a JSON object with a bunch of extraneous data (rather than, e.g. just a string with the path or an array of the path segments)
  • Included in that data is a _firestore.settings.credentials sub-field that includes a private SSH key!
  • Because we are retrieving the data with the admin SDK, this SSH key has admin privileges
  • Apart from the security issue, there's a load of useless information that inflates the size of records, and the format in general makes handling the field in Algolia pretty complicated

So after looking at the code, it looks like there's no processing at all happening on the extracted data, and it seems like it would be relatively straightforward to implement something. Like if(value.dataType == DocumentReference) payload[field] = value._path.segments; - I don't know the exact syntax or field names off the top of my head but something like that should work I think.

I don't mind implementing this but maybe someone from Algolia should specify what the desired behaviour is (should it preserve any of the random non-private information? should it be an array like segments right now or just a string (i.e. segments.join('/'))?

Error when performing Algolia index Error: Value for argument "field" is not a valid field path.

indexing fields : channelName,thumbnail,title,duration,id,date,description,category,isVisible,videoScore,views (from firebase ui)

fields from log : [ 'fields: channelName,thumbnail,title,duration,id,date,description,category,isVisible,videoScore,views,' ]

but it is showing error Error when performing Algolia index Error: Value for argument "field" is not a valid field path.

It was working fine before.

Algolia index not updating fields changed to null value.

When I update a field value within a Firestore document to null, the extension executes, but does not actually update the value in the Algolia index. I would assume the Algolia value would be deleted from the index, as null values are not transferred from Firestore to Algolia.

[Feature Request] Sync with collections OR collectionGroups

Hello devs,

First of all - thank you for such an amazing and time saving tool. It really helps us to move much faster with Fire + Algolia.

One thing I found missing is support for collectionGroups. It would extremely help as currently, we've to have global collection in order for it to sync with Algolia. However there are many instances as it's better to nest it under some collection instead of keeping everything global.

As a workaround, we have subcollection and collection of the same thing. Collection is used for algolia only, and subcollection for other things. It would be great to rely on only subcollection in this case as it would take away this constant sync between main subcollection and collection which is only used in Algolia.

Looking forward!

Cheers

Unable to Import Records Into Algolia

Hi, I have a very similar issue to:
https://github.com/algolia/firestore-algolia-search/issues/43#event-4954001620
and
https://github.com/algolia/firestore-algolia-search/issues/42#issuecomment-869508635

Ignoring the error messages like for issue #42 and using the one liner command provided in #43 did not solve my problem.

Here is the command I ran:
LOCATION=us-central1 ALGOLIA_APP_ID=D********* ALGOLIA_API_KEY=b****************0 ALGOLIA_INDEX_NAME=dev_foodly COLLECTION_PATH=Restaurants/{restaurantId}/Locations FIELDS=restaurantName GOOGLE_APPLICATION_CREDENTIALS=foodlyServiceAccountKeyDev.json npx [email protected]
And this is my output:

{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAppId":"D7MMB2JA20","algoliaAPIKey":"********","algoliaIndexName":"dev_foodly","collectionPath":"Restaurants/{restaurantId}/Locations","fields":"restaurantName","severity":"INFO","message":"Initializing extension with configuration"}

WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota.  Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details.  Do you want to continue? (y/N): y
✨  Done in 41.50s.

Unauthenticated firebase functions?

Hi,
Currently, I have deployed the algolia-firestore extension along with a transform function as explained in the README. With my current deployment, I get this error:

Error when performing Algolia index { FetchError: invalid json response body at https://accounts.google.com/ServiceLogin?passive=true&continue=https://el.appengine.google.com/_ah/conflogin%3Fstate%3D~< some long key >
reason: Unexpected token < in JSON at position 1

So looks like Algolia app is not properly authenticated before calling the firebase transform function. This results in the app skipping the transform and just writing the firebase doc as it is.

The README Mentions:
The Transform Firebase Function should be set up to unauthenticated users at this time

But there doesn't seem to be any option to set https callable functions to be unauthenticated in firebase. Can you please help understand what exactly I need to do for the transformation to work?

Import script runs collection group query that does not respect collection path in config.

Currently the retrieveDataFromFirestore function of the import script takes the last segment (the last subcollection) of the collection path and runs a collection group query on it.

This leads to documents that are not part of the config collection path being imported into Algolia.
For eg: if my firestore architecture is as follows:

  • Users
    • Records
  • TestUsers
    • Records
  • Admin

And if my collection path stated in the config is users/{x}/records. The import script would currently import records under both Users as well as TestUsers. This is not the expected behavior and could potentially lead to incurring unexpected costs.

Proposed Solution

Perform the collection group query as normal and then filter the results by matching the doc.ref.path to the config.collectionPath. A simple for loop (incremented by 2) matching each collection segment would let you know if the document returned is part of the collection path stated in the config and can continue with the processQuery flow.

Algolia index not updating deleted fields.

Here's what happening :

  1. I remove a field on firestore collection. [field 'type' in my case]
  2. The respective field is not deleted on algolia index. [I can still see 'type' on algolia index, I refreshed browser many times and even checked after few hours]

before deletion I tested by changing the field data and the changes are reflected on index. Just that the firestore field deletion part does not delete on algolia index.
Am I missing something or is it an issue??

Make import script runnable via npx

It might be a bit of a hassle for users to clone the repo and run the import script. One option is to make it runnable via an npx command like npx github:algolia/firestore-algolia-search/...

How to configure multiple collections?

I am able to successfully configure one collection with one indices. but I want to configure multiple collection and each collection would have corresponding indice. Is it possible in using this extension?

Use with emulator?

I'm trying to set this extension up with the emulator, and I did manage to download the extension, and subsequently launch it, but the problem is that the import script will obviously take the data from the online project I downloaded the extension from, and I'm looking for a way for algolia to use my local firestore db instead.

Starting with the first import (I'd create a separate Application in Algolia for it and use its details when configuring), and subsequently ensuring that when data is added to my local firestore db it triggers the onWrite in the extension, I'd really appreciate some guidance.

Things I tried:

I replaced this:
functions.handler.firestore.document

...with this (as recommended by SO answers):
functions.firestore.document(`projects/${process.env.PROJECT_ID}/databases/(default)/documents/${process.env.COLLECTION_PATH}/{documentID}`)

(envs do exist and do not appear to report errors during init)

I also tried adding default exporting the executeIndexOperation variable from index to no avail.

What happens on record creation is that the log.init() function does appear to kick in, but nothing inside onWrite (not even log.start()).

Type of timestamp value is better dictionary type than number

Hi, Thank you for all your help!

I started to use this extension lately and realized the value of the timestamp will be changed to a number of milliseconds.
ref: https://github.com/algolia/firestore-algolia-search/blob/main/functions/src/processors.ts#L62

I need to do anything another when decoding the struct which includes another type like this.
Are there some reasons to have it as a number type?
Moreover, my suggestion is to use a dictionary type like GeoPoint

What do you think?
Sorry if it's my misunderstanding, thank you!

Race condition

Context: We have our own Algolia sync cloud function.

Reading through your sources I can see you will have a race condition issue due to hot and cold-start functions. It's not the easiest to detect, but can cause havoc in production.

I'm not at the office till tomorrow, but I will record a demonstration of the issue and simple solution.

Unable to Import Existing Records Into Algolia

LOCATION=us-central1
PROJECT_ID=nexiev
ALGOLIA_APP_ID=
ALGOLIA_API_KEY=
ALGOLIA_INDEX_NAME=ev
COLLECTION_PATH=ev
GOOGLE_APPLICATION_CREDENTIALS=nestra-firebase-adminsdk-jufxu-6f8cfa91bb.json
npx firestore-algolia-search

I ran the above command in Powershell. But I got the following error and existing records were not imported.

WARNING: The backfill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y

{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read property 'split' of undefined\n at retrieveDataFromFirestore (C:\Users\Afzal\AppData\Roaming\npm-cache\_npx\7620\node_modules\firestore-algolia-search\lib\import\index.js:72:65)\n at C:\Users\Afzal\AppData\Roaming\npm-cache\_npx\7620\node_modules\firestore-algolia-search\lib\import\index.js:80:9\n at Interface._onLine (readline.js:327:5)\n at Interface._line (readline.js:658:8)\n at Interface._ttyWrite (readline.js:999:14)\n at ReadStream.onkeypress (readline.js:205:10)\n at ReadStream.emit (events.js:315:20)\n at emitKeys (internal/readline/utils.js:335:14)\n at emitKeys.next ()\n at ReadStream.onData (readline.js:1133:36)"}

New Documents Added When Adding Field To Preexisting Documents

We recently ran a script to add a field to all pre-existing documents, and even though this field was not part of the indexable fields list, documents that were previously not in the index were added to it. We would like only for documents to be added to the index if they have all the indexable fields, and only if the indexable fields have changed.

Error performing backfill with Firebase extensions

New to Algolia and trying to get the Firebase extension up-and-running, but having some issues with the backfill script.

This is the script I am running:

{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAPIKey":"********","severity":"INFO","message":"Initializing extension with configuration"}

WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota.  Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details.  Do you want to continue? (y/N): y

{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read property 'split' of undefined\n    at retrieveDataFromFirestore (/Users/simon/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:72:65)\n    at /Users/simon/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:80:9\n    at Interface._onLine (readline.js:340:5)\n    at Interface._line (readline.js:671:8)\n    at Interface._ttyWrite (readline.js:1015:14)\n    at ReadStream.onkeypress (readline.js:213:10)\n    at ReadStream.emit (events.js:376:20)\n    at emitKeys (internal/readline/utils.js:345:14)\n    at emitKeys.next (<anonymous>)\n    at ReadStream.onData (readline.js:1149:36)"}

I'm using the following command:

LOCATION=europe-west2\ 
PROJECT_ID=my-project-name\ 
ALGOLIA_APP_ID=0000000000\ 
ALGOLIA_API_KEY=0000000000\ 
ALGOLIA_INDEX_NAME=my_index\ 
COLLECTION_PATH=my-collection\ 
FIELDS=field1,field2,field3\ 
GOOGLE_APPLICATION_CREDENTIALS=/...json\ 
npx firestore-algolia-search

I tried reinstalling the extension and still having zero joy.

Any idea where I might be going wrong?

Can't run firestore-algolia-search in bash script

Hello everyone,

I've been trying to run the restore script for firestore-algolia-search. According to other issues, decided to run it from a bash script:

#!/bin/bash

LOCATION=us-central1\
PROJECT_ID=project-id\
ALGOLIA_APP_ID=111111\
ALGOLIA_API_KEY=111111\
ALGOLIA_INDEX_NAME=index\
COLLECTION_PATH=path\
FIELDS=fields\
GOOGLE_APPLICATION_CREDENTIALS=./key.json\
npx firestore-algolia-search

But then the output is firestore-algolia-search: command not found . I'm writing this issue here because other npx script would work in bash scripts. npx is installed globally and the command is working outside of bash (but no with parameters declared beforehand)

Thanks a lot

objectID to id

When creating a firestore doc it automatically creates an id for the document as known. later on, this object is indexed by algolia with it's own prop objectID. any idea to automatically transform this objectID to id because my app is based to work with the id prop and not objectID when getting search results from algolia? this would be very useful to avoid unnecessary duplicate code in my application. thank you

Skip certain documents from being indexed

Is there a way to skip/ignore particular document updates, i.e. filtering. For example if i have a collection of documents, each document containing a 'type' property, and I'm only interested in document updates where type == 'sport', how could i achieve this?

I thought of using the transform function to check the 'type' and return an empty response if it's not a value im concerned with, e.g. 'sport' in this example, but looking at the code here, I'm not sure that would be the correct approach.

Any suggestions appreciated, thanks

Import script

Hello, I'm having trouble with importing a firestore collection using the following script:

LOCATION=us-central1
PROJECT_ID=hidden
ALGOLIA_APP_ID=hidden
ALGOLIA_API_KEY=hidden
ALGOLIA_INDEX_NAME=test
COLLECTION_PATH=activities
FIELDS=name;title;description
GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>
npx firestore-algolia-search

Terminal output:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAPIKey":"********","severity":"INFO","message":"Initializing extension with configuration"}

WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N): y
{"severity":"ERROR","message":"Error when performing Algolia index TypeError: Cannot read properties of undefined (reading 'split')\n at retrieveDataFromFirestore (/Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:76:65)\n at /Users/marius/.npm/_npx/0ca4ecfa9810ae5b/node_modules/firestore-algolia-search/lib/import/index.js:93:9\n at Interface._onLine (node:readline:485:5)\n at Interface._line (node:readline:864:8)\n at Interface._ttyWrite (node:readline:1216:14)\n at ReadStream.onkeypress (node:readline:288:10)\n at ReadStream.emit (node:events:520:28)\n at emitKeys (node:internal/readline/utils:358:14)\n at emitKeys.next ()\n at ReadStream.onData (node:internal/readline/emitKeypressEvents:61:36)"}

npm/npx version is 8.3.1

Filtering Tags as Firestore Arrays of Maps (Objects)

In our Firestore collection, tags are listed as Arrays of Maps (objects), like the following document:

{...,
tags: [
TagId: "uYgvhNdciHcuK7ioGZEv",
Title: "SARS-CoV-2 (COVID-19)"
]}

How can I filter the searh results using these tags?

Filtering collection

I only want to index records that are published. I have a boolean isPublished and I would only like records from my collection that are published to be indexed.

I've tried using the transform function but I get an invalid JSON error if I try to return anything other than the data.

Does the extension allow me to filter records before they are sent to Algolia?

Firebase Cloud functions error 'Unreachable hosts - your application id may be incorrect. If the error persists, contact [email protected]'

When I deploy the algolia search in my app it works really well. But the other cloud functions which I have deployed works only when algolia search is not deployed. I get the following error in the logs
Unreachable hosts - your application id may be incorrect.

My application Id is correct and I am using the firebase blaze plan.

Version:
"algoliasearch": "^4.11.0",

    "express": "^4.17.2",

    "firebase-admin": "^9.12.0",

    "firebase-functions": "^3.16.0",
Sharing my code below

algolia.js

const ALGOLIA_APP_ID = APP_ID;
const functions = require("firebase-functions");
const ALOGOLIA_ADMIN_KEY = API_KEY;
const ALGOLIA_INDEX_NAME = 'profile_card';
const client = algoliasearch(ALGOLIA_APP_ID, ALOGOLIA_ADMIN_KEY);
const index = client.initIndex(ALGOLIA_INDEX_NAME);


index.setSettings({ searchableAttributes: ['username', '_tags'], attributesForFaceting: ['searchable(_tags)'], removeStopWords: true });
exports.createPost = functions.firestore
    .document('EUU/{euId}')
    .onCreate(async (snap, context) => {
        const newValue = snap.data();
        newValue.objectID = snap.id;
        return index.saveObject(newValue);
        console.log('Finished');
    });

exports.updatePost = functions.firestore
    .document('EUU/{euId}')
    .onUpdate(async (snap, context) => {
        const afterUpdate = snap.after.data();
        afterUpdate.objectID = snap.after.id;
        return index.saveObject(afterUpdate);
    });

exports.deletePost = functions.firestore
    .document('EUU/{euId}')
    .onDelete(async (snap, context) => {
        const oldID = snap.id;
        return index.deleteObject(oldID);
    });

index.js

const functions = require("firebase-functions");
const admin = require('firebase-admin');
const algo = require('./algolia');
// const cashfree = require('./cashfree');
admin.initializeApp(functions.config().functions);

const options = {
  priority: 'high'
}
exports.sendRequest = functions.firestore.document('EUU/{euId}/Requests/{id}').onCreate(async (snapshot, context) => {
  const request = snapshot.data();
  // console.log(context.params.euId);
  admin.firestore().collection('mobileNotificationToken').doc(context.params.euId).get().then((doc) => {
    const deviceTokens = doc.get('tokens');
    // console.log(deviceTokens);
    var payload = {
      notification: {
        title: 'Session request from ' + request.sender_name,
        body: 'Time slot: ' + request.time_slot,

      },
      data: {
        tag: 'request',
        'reason': request.reason,
        request_id: snapshot.id,
        sender_id: context.params.euId,
      },


    };
    return admin.messaging().sendToDevice(deviceTokens, payload, options).catch((error) => {
      // console.log('Error sending message:', error);
      console.error("Request Failed", error)
    })
  });
})
exports.paymentNotif = functions.firestore.document('EUU/{euId}/Requests/{id}').onUpdate(async (change, context) => {
  const updatedStatus = change.after.data();
  if (updatedStatus.status == 'Accepted') {
    admin.firestore().collection('mobileNotificationToken').doc(updatedStatus.sender_id).get().then((doc) => {
      const deviceTokens = doc.get('tokens');
      console.log(deviceTokens);
      var payload = {
        notification: {
          title: `Appointment Accepted by ${updatedStatus.receiver_name}`,
          body: `Your appointment has been accepted`,

        },
        data: {
          message: 'Your Session is booked',
          tag: 'payment',
          sender_id: context.params.euId,
          'request_id': context.params.id,
        },

      };
      const notificaiton = {
        title: `Appointment Accepted by ${updatedStatus.receiver_name}`,
        body: `Your appointment accepted`,
        euId: context.params.euId,
        payment_done: false,
        reqId: context.params.id,
        createdAt: admin.firestore.Timestamp.now(),
        type: 'Type.Payment'
      }
      const docs = admin.firestore().collection('Seeker').doc(updatedStatus.sender_id);
      const EUdocs = admin.firestore().collection('EUU').doc(updatedStatus.sender_id);

      EUdocs.get().then((snap) => {
        if (snap.exists) {
          EUdocs.collection('notifications').add(notificaiton).then(() => console.log('notification stored to db'));

        } else {
          docs.collection('notifications').add(notificaiton).then(() => console.log('notification stored to db'));

        }
      });
      return admin.messaging().sendToDevice(deviceTokens, payload, options)
        .then((response) => {
          console.log("Successfully sent message: ", response);
        }).catch((error) => {
          console.log('Error sending message:', error);
        })
    });
    // const id = change.id;
    // return change.after.ref.set({ id: { 'payment': 'pending' } });

  } else {
    console.log('DIDNT WORK');
  }

})

exports.sendNotification = functions.firestore.document('EUU/{euId}/Notifications/{id}').onCreate(async (snapshot, context) => {
  const request = snapshot.data();
  console.log(context.params.euId);
  admin.firestore().collection('mobileNotificationToken').doc(context.params.euId).get().then((doc) => {
    const deviceTokens = doc.get('tokens');
    console.log(deviceTokens);
    var payload = {
      notification: {
        title: 'Push Notification',
        body: 'Hello',

      },
      data: {
        message: 'Your Session is booked',
        tag: 'notification',
        // receiver_id: context.params.euId,
      },

    };
    admin.messaging().sendToDevice(deviceTokens, payload, options)
      .then((response) => {

        console.log("Successfully sent message: ", response);
      }).catch((error) => {
        console.log('Error sending message:', error);
      })
  });
})

exports.paymentSuccess = functions.firestore.document('Seeker/{seekerId}/Orders/{id}').onUpdate(async (change, context) => {
  const updatedStatus = change.after.data();
  if (updatedStatus.txStatus == 'SUCCESS') {
    admin.firestore().collection('mobileNotificationToken').doc(updatedStatus.euId).get().then((doc) => {
      const deviceTokens = doc.get('tokens');
      console.log(deviceTokens);
      var payload = {
        notification: {
          title: 'Session Confirmation',
          body: 'Your session has been confirmed',

        },
        data: {
          message: 'Your Session is booked',
        },
      };
      return admin.messaging().sendToDevice(deviceTokens, payload, options)
        .then((response) => {
          console.log("Successfully sent message: ", response);
        }).catch((error) => {
          console.log('Error sending message:', error);
        })
    });
    admin.firestore().collection('mobileNotificationToken').doc(updatedStatus.seekerId).get().then((doc) => {
      const deviceTokens = doc.get('tokens');
      console.log(deviceTokens);
      var payload = {
        notification: {
          title: 'Session Confirmation',
          body: 'Your session has been confirmed',

        },
        data: {
          message: 'Your Session is booked',
        },
      };
      return admin.messaging().sendToDevice(deviceTokens, payload, options)
        .then((response) => {
          console.log("Successfully sent message: ", response);
        }).catch((error) => {
          console.log('Error sending message:', error);
        })
    });
    // const id = change.id;
    // return change.after.ref.set({ id: { 'payment': 'pending' } });

  } else {
    console.log('PAYMENT SUCCESS DIDNT WORK');
  }

});
exports.createPost = algo.createPost;
exports.deletePost = algo.deletePost;
exports.updatePost = algo.updatePost;

Screenshot (17)

Permission error

I am getting the following message after setting up the extension through firebase
{ "message": "Not enough rights to update an object near line:1 column:1893", "status": 400 }

I assumed that all permission would be setup on install. App ID and API Key are correct. Did I miss a step?

Mutate and Create Fields When Sending to Algolia

Hello! We currently have some cloud functions that mutates our Algolia indices when relevant documents are created, updated, or deleted, and we are excited to discover that this Firebase extension can help remove some of this code. However, one of our indices change one field to a more search-friendly form, i.e. turning a serialized JSON of rich text into plaintext. We would like to move all of our function code to using this extension, but this is currently blocking us from doing so. Thanks!

Every Query performed on Algolia counts up towards the read usage count of Cloud Firestore

As far as I understand, the extensions should only bridge new data to Algolia, which stores the received data in records and sends those back to the user whenever a query is performed.
However, I have noticed that whenever I perform a query on my client (iOS in this case) the read usage of my Cloud Firestore increases by 1 for every hit of the query (which results in an enormous quantity of reads).
Is this an intentional behavior?
Thanks.

Subcollections support

It's not possible to have a subcollection tied to the function, it has to be on the root or it won't work.

Only updating a bool does not update record in index

Everything works as expected and syncs perfectly except booleans. If a new document is created it gets loaded in. However if only the boolean field gets updated, it doesn't update it. The field is called 'hidden' so not sure if that's a problem.

The logs for my cloud functions do not show an error. However the bool does not show up inside the data being transmitted. The field 'hidden' is in my indexed fields settings for the plugin.

Would love to hear how to continue.
Version: 0.4.1

Edit: apparently the same thing goes for numbers. I tried to change the field type to string and the 'hidden' value syncs perfectly. So it really depends on the type and not on a configuration setting.

FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail

After installing the "Search with Algolia" Firebase extension, the extension page has three tabs:
1- How this extension works
2- Extension configuration
3- APIs and resources

I followed the instructions in first tab "How this extension works". I've copied the instructions that I get on this page below for your reference.
After running npx firestore-algolia-search, the first lines of console logs are as follows.
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"algoliaAppId":<YOUR_ALGOLIA_APP_ID>,"algoliaAPIKey":"********","algoliaIndexName":"nodesIndex","collectionPath":"nodes","fields":"admin,content,createdAt,nodeType,tags,title","severity":"INFO","message":"Initializing extension with configuration"}
WARNING: The back fill process will index your entire collection which will impact your Search Operation Quota. Please visit https://www.algolia.com/doc/faq/accounts-billing/how-algolia-count-records-and-operation/ for more details. Do you want to continue? (y/N):
I answered "y" to this question. Then, it generated a lot of logs but there is no index record generated on Algolia.

You can test out this extension right away:

Go to the Cloud Firestore tab.

If it doesn’t exist already, create a collection called nodes.

Create, update, or delete a document in the nodes collection. Go to Algolia’s dashboard and verify in Algolia that a record is created, updated, or deleted in the nodesIndex index for application id ................

Using the extension
This extension listens to the Cloud Firestore collection nodes. If you create, update, or delete a document within that collection, this extension will:

Indexes the document and send all the fields or configured fields defined in the extension.
or, removes the record from Algolia index if the document is deleted.
(Optional) Import existing documents or Reindex after configuration changes
This extension starts monitoring the nodes collection after a successful installation. Any existing documents created before the extension installation can be back-filled into your Algolia nodesIndex Index using the import script. Also, you will need to run the import script if you change the Indexable Fields, Index Name, Application Id, and/or API Key configuration.

The import script will read all existing documents in the nodes collection and insert them into the Algolia nodesIndex index.

Important notes
You must run the import script over the entire collection after installing the Search with Algolia extension; otherwise you will have missing records in your Algolia nodesIndex index.
lastmodified attributed is added to all Algolia records to eliminate race condition issues related from the Cloud Function cold start.
Run the script
The script will use the extension configuration before the import process starts.

Run the import process using npx.

Make sure that you’ve installed the required tools to run the import script:

To access the npx command tools, you need to install Node.js.
If you use npm v5.1 or earlier, you need to explicitly install npx. Run npm install --global npx.
The import script uses Application Default Credentials to communicate with Firebase.
Please follow the instructions to generate a key for your service account.

Execute the below command:

Update the path to the Google Application credentials.
Clear out the FIELDS param if it contains { unspecified parameter } in the command below since it’s an invalid value.
LOCATION=us-east4
ALGOLIA_APP_ID=<YOUR_ALGOLIA_APP_ID>
ALGOLIA_API_KEY=<YOUR_API_KEY>
ALGOLIA_INDEX_NAME=nodesIndex
COLLECTION_PATH=nodes
FIELDS=admin,content,createdAt,nodeType,tags,title
GOOGLE_APPLICATION_CREDENTIALS=</path/to/service/account/key>
npx firestore-algolia-search
Monitoring
As a best practice, you can monitor the activity of your installed extension, including checks on its health, usage, and logs.

Allow transform to skip records

Current the transform function returns the json payload from the transform endpoint.

Currently it seems that in order to skip a record it'd be best to throw on the callable function, however this would create a false positive log. Instead, is it worth returning an empty object and checking this on the response? Happy to make a PR if so.

extension error

We upgraded to:

Search with Algolia
algolia/[email protected]

We are seeing this error:

ext-firestore-algolia-search-executeIndexOperation
Error when performing Algolia index { FetchError: invalid json response body at https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://uc.appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://xxxxxx/extAlgoliaSearchCustomers reason: Unexpected token < in JSON at position 1 at /workspace/node_modules/node-fetch/lib/index.js:272:32 at process._tickCallback (internal/process/next_tick.js:68:7) message: 'invalid json response body at https://accounts.google.com/ServiceLogin?service=ah&passive=true&continue=https://uc.appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://xxxxxxx/extAlgoliaSearchCustomers reason: Unexpected token < in JSON at position 1', type: 'invalid-json' }

Update from X.X.X to 0.5.9 Breaks Connection to Algolia

Repeatable Steps:

  1. Install firebase extension 0.5.4
  2. Connect Algolia via custom API keys
  3. Import collections via firestore-algolia-search npx package.
  4. change firestore entries and succussfully see them update in algolia.
  5. update firebase extension to 0.5.9.
  6. Try and change firestore data and see reflection but nothing changes unless i manually run the migrations via the cli.

Thanks in advance please help!!!!!!!

Invalid secret id

Error: HTTP Error: 400, Secret ID ***my-api-key*** is invalid, must be of form projects/*/secrets/*/versions/*

Created the key as recomended, and have previously installed working extension for a collection with this key, now trying to add another for another collection, I get this error.

Error when inserting data from Client

Hi, am really new using algolia, i don't know but i have following all step in firebase extension doc. I install algolia through firebase extension.

I created new api key and divert it to proper indices(by creating new indices). But i got error in algolia functions when i inserted new record from my website, what logs said is:

Error when performing Algolia index {"transporterStackTrace":[{"triesLeft":3,"response":{"isTimedOut":false,"status":403,"content":"{"message":"Index not allowed with this API key","status":403}"},"host":{"accept":2,"url":"NZ3GVUOMSD.algolia.net","protocol":"https"},"request":{"url":"https://NZ3GVUOMSD.algolia.net/1/indexes/algoId/batch?x-algolia-agent=Algolia%20for%20JavaScript%20(4.8.6)%3B%20Node.js%20(10.24.1)%3B%20firestore_integration%20(0.5.1)","connectTimeout":2,"data":"{"requests":[{"action":"partialUpdateObject","body":{"objectID":"crSuskBJaQpfOAODUNey","title":"Test 5","category":"kesehatan","lastmodified":{"_operation":"IncrementSet","value":1626963102502}}}]}","method":"POST","headers":{"x-algolia-application-id":"NZ3GVUOMSD","x-algolia-api-key":"*****","content-type":"application/x-www-form-urlencoded"},"responseTimeout":30}}],"name":"ApiError","status":403}

Then, i checked algolia dashboard and there is nothing there, still empty. Can somebody explain to me what happened here? Thanks

Algolia not updating Indexed Fields (proposal)

I've been using this extension for a few days now and so far it's working very well. However, although creating a new Firebase document correctly indexed it in Algolia, updating the document did not.

After many hours searching for a possible cause, I noticed there were spaces between each comma, the Firebase Function gets triggered, but nothing is updated in the Algolia counter part. Since indexed fields can't contain spaces (I think?) maybe the function should remove all spaces when parsing the Indexable Fields.

config

If there's a quirk I'm missing, at least I hope this helps someone like me with OCD who needs to put a space after each comma 😁

Missing module in script to import existing documents

Hi,

I 'm trying to import existing Firestore documents using npx firestore-algolia-search and following this guide , but I'm receiving this error:

{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
Cannot find module '@google-cloud/firestore'

NODE v10.23.3
NPM v6.14.11

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.