Unable to add weaviate-ts-client to new React app

[weaviate]

Unable to add weaviate-ts-client to new React app

[Surfytom]

Managed to get it down just to this. This could be an issue with child_process just not working on a built chrome extension as I've run into a similar issue before just thought i'd add though.

ERROR in ./node_modules/weaviate-ts-client/dist/index.js 3:126365-126389 Module not found: Error: Can't resolve 'child_process' in 'node_modules\weaviate-ts-client\dist'

had to change my webpack.config.js file to add a bunch of false statements to get rid of the errors:

use strict;

const { merge } = require('webpack-merge');

const common = require('./webpack.common.js');
const PATHS = require('./paths');

// Merge webpack configuration files
const config = (env, argv) =>
merge(common, {
entry: {
popup: PATHS.src + '/popup.js',
contentScript: PATHS.src + '/contentScript.js',
background: PATHS.src + '/background.js',
},
devtool: argv.mode === 'production' ? false : 'source-map',
resolve: {
fallback: {
"fs": false,
"tls": false,
"net": false,
"path": false,
"zlib": false,
"http": false,
"https": false,
"stream": false,
"crypto": false,
"crypto-browserify": false,
"os": false,
"util": false,
"process": false,
"buffer": false,
"assert": false,
"string_decoder": false
},
},
});

module.exports = config;

[parkerduckworth]

I will shortly make a new release on npm which contains these changes. I'll update here when it is available

[parkerduckworth]

@lanegoolsby as promised, here is the new release which is compatible with the browser:
https://github.com/weaviate/typescript-client/releases/tag/v1.1.0

[lanegoolsby]

Tested and its working! Much appreciated @parkerduckworth!

[lanegoolsby]

I was able to take a wild guess at the packages to add but didn't get very far. Went from 15 to errors to 6.

Failed to compile.

Module not found: Error: Can't resolve 'fs' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/weaviate-ts-client/dist'
WARNING in [eslint] 
src/App.js
  Line 1:8:  'logo' is defined but never used  no-unused-vars

ERROR in ./node_modules/weaviate-ts-client/dist/index.js 1235:9-22
Module not found: Error: Can't resolve 'fs' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/weaviate-ts-client/dist'

ERROR in ./node_modules/weaviate-ts-client/dist/index.js 3398:9-26
Module not found: Error: Can't resolve 'crypto' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/weaviate-ts-client/dist'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
	- add a fallback 'resolve.fallback: { "crypto": require.resolve("crypto-browserify") }'
	- install 'crypto-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
	resolve.fallback: { "crypto": false }

ERROR in ./node_modules/weaviate-ts-client/dist/index.js 5955:7-23
Module not found: Error: Can't resolve 'https' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/weaviate-ts-client/dist'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
	- add a fallback 'resolve.fallback: { "https": require.resolve("https-browserify") }'
	- install 'https-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
	resolve.fallback: { "https": false }

ERROR in ./node_modules/weaviate-ts-client/dist/index.js 5957:7-31
Module not found: Error: Can't resolve 'child_process' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/weaviate-ts-client/dist'

ERROR in ./node_modules/weaviate-ts-client/dist/index.js 5958:7-28
Module not found: Error: Can't resolve 'path/posix' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/weaviate-ts-client/dist'

ERROR in ./node_modules/zlib/lib/zlib.js 1:0-43
Module not found: Error: Can't resolve './zlib_bindings' in '/Users/LGoolsby/dev/playground/searchv2/react-app/node_modules/zlib/lib'

webpack compiled with 6 errors and 1 warning

Here's my package.json. I'm sure some of these dependencies aren't the correct packages.

{
  "name": "react-app",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.16.5",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "assert": "^2.0.0",
    "buffer": "^6.0.3",
    "child_process": "^1.0.2",
    "crypto": "^1.0.1",
    "fs": "^0.0.1-security",
    "graphql": "^16.6.0",
    "graphql-request": "^5.2.0",
    "https": "^1.0.0",
    "net": "^1.0.2",
    "os": "^0.1.2",
    "path": "^0.12.7",
    "process": "^0.11.10",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-scripts": "5.0.1",
    "stream": "^0.0.2",
    "util": "^0.12.5",
    "weaviate-client": "^2.14.5",
    "weaviate-ts-client": "^1.0.0",
    "web-vitals": "^2.1.4",
    "zlib": "^1.0.5"
  },
  "scripts": {
    "start": "BROWSER=none react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "typescript": "^5.0.4"
  }
}

[Surfytom]

Same logs here. Trying to add to my to my chrome extension with webpack 5

[lanegoolsby]

FWIW the same sort of behavior is seen in a new Angular app.

[parkerduckworth]

Thanks for reporting back on this @lanegoolsby and @Surfytom.

Are you using typescript in your project? Can you please provide the npm, node, and typescript versions (if applicable) in your environment?

I tried to reproduce the example in your first comment @lanegoolsby, but everything works as expected on my end:

➜  /tmp npx create-react-app react-app                                                                                                                              /tmp
Creating a new React app in /private/tmp/react-app.

...

Happy hacking!
Time: 0h:00m:13s
➜  /tmp cd react-app                                                                                                                                                /tmp
➜  react-app git:(main) npm i weaviate-ts-client                                                                                                          /tmp/react-app

added 22 packages, and audited 1506 packages in 1s

234 packages are looking for funding
  run `npm fund` for details

6 high severity vulnerabilities

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
➜  react-app git:(main) ✗ npm start                                                                                                                       /tmp/react-app

> [email protected] start
> react-scripts start

(node:15691) [DEP_WEBPACK_DEV_SERVER_ON_AFTER_SETUP_MIDDLEWARE] DeprecationWarning: 'onAfterSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:15691) [DEP_WEBPACK_DEV_SERVER_ON_BEFORE_SETUP_MIDDLEWARE] DeprecationWarning: 'onBeforeSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
Starting the development server...
Compiled successfully!

You can now view react-app in the browser.

  Local:            http://localhost:3000
  On Your Network:  http://192.168.1.123:3000

Note that the development build is not optimized.
To create a production build, use npm run build.

webpack compiled successfully
^C
Time: 0h:00m:11s

[lanegoolsby]

I believe you need to add a reference to weaviate before you'll the error. By default webpack will ignore any dependencies that aren't referenced in code somewhere. Initialize the weaviate client in a class and call the schema endpoint (or something similarly simple).

My version info is:
Node: 16.15.0
npm: 8.15.1
TS: ^5.0.4 (its in the package.json)

[lanegoolsby]

For example:

import './App.css';
import weaviate, { WeaviateClient } from 'weaviate-ts-client';
import React from 'react';

function App() {
  const client: WeaviateClient = weaviate.client({
    scheme: 'http',
    host: 'localhost:8080'
  });

  async function onClick(){
    const a = await client
    .schema
    .getter()
    .do();

    console.log(a)
  }

  return (
    <div className="App">
      <button onClick={onClick}>Schema</button>
    </div>
  );
}

export default App;

[parkerduckworth]

After some investigation I was able to determine that the issue stems from the embedded module. This module imports a number of node system packages which are not available in the browser, such as child_process, fs, and crypto.
This does not affect the client's build pipeline, as the esbuild platform option is currently set to 'node'. If this were instead set to neutral or browser, the build would fail due to the packages mentioned above not being available. If I remove the imports of embedded from index.ts, the code build and runs with platform set to both neutral and browser.

Some of the node system packages have been ported to be compatible with the browser (browserify-fs, crypto-browserify), however this appears to not be possible with child_process.

I experimented with conditional imports, but have so far been unable to find a workaround which will build and run with the embedded module even conditionally available.

Still working on a solution for this. Any input is welcomed :)

[Surfytom]

Thank you for the investigation. From my little experience with Chrome extensions I think that child-process is a server side node module that can't be used on the browser (I know this because I tried to make it work for a few days on my chrome extension).

I have opted for using sockets on my chrome extension so it can communicate with a server that communicates with weaviate getting around the build issues

[Oceanswave]

Yeh, cannot use with NextJS either. Not sure why a client needs a whole bunch of server side node libs.

Edit: I solved this by just generating my own typescript client using graphql-codegen, wihch makes things much nicer.

[parkerduckworth]

@Oceanswave we recently introduced embedded db support in the client, which depended on those server-side packages. I agree that this client should not rely on any server-side dependencies.

---

The above PR removes these packages from the library, and this client will again be isomorphic. Sorry all for the discrepancy.

[parkerduckworth]

Great to hear @lanegoolsby, thanks for the follow-up