gatsby-source-mongodb not fetching Atlas data

[judsonlmoore]

gatsby-source-mongodb not fetching Atlas data

[judsonlmoore]

After sleeping on it, I thought to try again with a fresh mongodb cluster and using mongodb example data (sample_mflix dataset). With this, I was able to make a connection using these settings:

    {
      resolve: `gatsby-source-mongodb`,
      options: {
        connectionString: process.env.MONGODB_URI,
        dbName: process.env.MONGODB_DB,
        collection: process.env.MONGODB_COLLECTION,
        extraParams: {
          retryWrites: true,
          w: `majority`,
        },
      },
    },

So having established a connect to mongodb, I tried again with my data and again it failed. So then I got to thinking my data is too big (±300k items). Turns out, this is likely the problem. When I adjusted my settings to include a query to limit what data I got back, I started getting data:

    {
      resolve: `gatsby-source-mongodb`,
      options: {
        connectionString: process.env.MONGODB_URI,
        dbName: process.env.MONGODB_DB,
        collection: process.env.MONGODB_COLLECTION,
        extraParams: {
          retryWrites: true,
          w: `majority`,
        },
        query: {
          [process.env.MONGODB_COLLECTION]: { field: { $eq: value } }
        },
      },
    },

It appears that the plugin does not support mongodb aggregations, which would be better than the query method (if that's wrong, documentation about using aggregation with this plugin would be great).

Additionally, how can a site using this plugin scale to hundreds of thousands / millions of pages if indeed the plugin struggles to pull so much data? Is there another method? Further optimizations?

So at this point I'm not 100% sure if this is a bug, or a support request, but I'll leave this here for now.

[judsonlmoore]

Update: I reported this issue while traveling and working with my Apple MBA M2 laptop, now that I am home, I'm trying to continue development on an Apple iMac i7 computer (specs below from gatsby info --clipboard) and with the exact same settings and .env data, I am getting this error in the console:

ERROR  UNKNOWN
(node:6004) [DEP0170] DeprecationWarning: The URL mongodb://{username}:{password}@ac-lahiygk-shard-00-01
.0lxmxaq.mongodb.net:27017,ac-lahiygk-shard-00-00.0lxmxaq.mongodb.net:27017,ac-lahiygk-shard-00-02.0lxmxaq.mongodb.n
et:27017/test?authSource=admin&replicaSet=atlas-3ei0ma-shard-0&retryWrites=true&w=majority&ssl=true is invalid.
Future versions of Node.js will throw an error.
(Use `node --trace-deprecation ...` to show where the warning was created)

Which is really weird because a) I don't get it on both computers, and b) the URL in that error isn't constructed similarly to the one I am providing: mongodb+srv://{username}:{password}@{database}.0lxmxaq.mongodb.net

So I have no idea where that error is getting that URL from.

System:
    OS: macOS 13.6.3
    CPU: (8) x64 Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.21 - /usr/local/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm
  Browsers:
    Chrome: 121.0.6167.139
    Safari: 17.1.2
  npmPackages:
    gatsby: ^5.13.2 => 5.13.3
    gatsby-plugin-image: ^3.13.1 => 3.13.1
    gatsby-plugin-mdx: ^5.13.1 => 5.13.1
    gatsby-plugin-postcss: ^6.13.1 => 6.13.1
    gatsby-plugin-sharp: ^5.13.1 => 5.13.1
    gatsby-plugin-sitemap: ^6.13.1 => 6.13.1
    gatsby-source-filesystem: ^5.13.1 => 5.13.1
    gatsby-source-mongodb: ^5.13.1 => 5.13.1
    gatsby-transformer-sharp: ^5.13.1 => 5.13.1

[judsonlmoore]

Update: I noticed that on the first computer I was running node v18.18.0 and on the second computer I was running node v20.10.0. I changed the node version on the second computer to match the first and now I get a successful build.