jaydenseric / extract-files Goto Github PK

A function to recursively extract files and their object paths within a value, replacing them with null in a deep clone without mutating the original value. FileList instances are treated as File instance arrays. Files are typically File and Blob instances.

Home Page: https://npm.im/extract-files

License: MIT License

JavaScript 100.00%

npm node graphql maintained esm mjs typescript

extract-files's Introduction

extract-files

A function to recursively extract files and their object paths within a value, replacing them with null in a deep clone without mutating the original value. FileList instances are treated as File instance arrays. Files are typically File and Blob instances.

Used by GraphQL multipart request spec client implementations such as graphql-react and apollo-upload-client.

Installation

For Node.js, to install extract-files with npm, run:

npm install extract-files

For Deno and browsers, an example import map:

{
  "imports": {
    "extract-files/": "https://unpkg.com/[email protected]/",
    "is-plain-obj": "https://unpkg.com/[email protected]/index.js",
    "is-plain-obj/": "https://unpkg.com/[email protected]/"
  }
}

See the function extractFiles to get started.

Requirements

Supported runtime environments:

Node.js versions ^14.17.0 || ^16.0.0 || >= 18.0.0.
Deno, importing from a CDN that might require an import map for dependencies.
Browsers matching the Browserslist query > 0.5%, not OperaMini all, not dead.

Non Deno projects must configure TypeScript to use types from the ECMAScript modules that have a // @ts-check comment:

compilerOptions.allowJs should be true.
compilerOptions.maxNodeModuleJsDepth should be reasonably large, e.g. 10.
compilerOptions.module should be "node16" or "nodenext".

Exports

The npm package extract-files features optimal JavaScript module design. It doesn’t have a main index module, so use deep imports from the ECMAScript modules that are exported via the package.json field exports:

extract-files's People

Contributors

Stargazers

Watchers

extract-files's Issues

Package "extract-files" exists but package.json "exports" does not include entry for "./public/extractFiles"

Hi,

 npx snowpack dev --polyfill-node
...
[22:24:01] [snowpack] Package "extract-files" exists but package.json "exports" does not include entry for "./public/extractFiles".

The recommendation from Snowpack common errors page --
Package exists but package.json “exports” does not include entry :

If you see this error message, that means that you’ve imported a file path not allowed in the export map. If you believe this to be an error, reach out to the package author to request the file be added to their export map.

In my yarn.lock:

apollo-upload-client@^14.1.3:
...
  dependencies:
    ...
    extract-files "^9.0.0"

Node version: v14.16.1
Snowpack version: 3.3.6

Thanks in advance,
Alexander

Reduce bundle size

While the source code is tiny, transpilation adds runtime helpers / polyfills.

See https://bundlephobia.com/[email protected]. 17.5 kB minified is crazy for package with only 100 lines, half comments.

The good news is that the extra code is being imported, so if your app or other dependencies use Babel too there is a good chance you are importing the extras already.

These are the extra imports in dist to look at eliminating:

import _classCallCheck from 'babel-runtime/helpers/classCallCheck';
import _Array$from from 'babel-runtime/core-js/array/from';
import _Object$keys from 'babel-runtime/core-js/object/keys';
import _typeof from 'babel-runtime/helpers/typeof';

DeprecationWarning: using version 9.0.0

(node:13559) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./public/" in the "exports" field module resolution of the package at PROJECT_DIR/node_modules/extract-files/package.json.
Update this package.json to use a subpath pattern like "./public/*".
(Use node --trace-deprecation ... to show where the warning was created)

Document required TypeScript libs

Version 13.0.0

I get in the new version the error:

<sourcefolder>/node_modules/apollo-upload-client/node_modules/extract-files/extractFiles.mjs(141,30)
      TS2488: Type 'any[] | FileList' must have a '[Symbol.iterator]()' method that returns an iterator.

It seems like a bug in your library in combination with typescript.

I use the library via apollo-upload-client

Optional Environment Support beside Browsers/React-Native

I know that this package is intended for browser/react-native environments.

By adding a new parameter for extractFiles, e.g. isFileValue: (value: unknown) => boolean, other higher level implementations that are not browser-specific could also leverage the logic of this package.

This is how I would imagine a Node.js implementation that uses fs.ReadStream.

"use strict";

const fs = require("fs");
const extractFiles = require("extract-files");

function isFileValue(value) {
  return value instanceof fs.ReadStream;
}

const { clone, files } = extractFiles(input, "", isFileValue);

The isFileValue function would then (if defined) used instead of the code block here:

extract-files/src/extractFiles.mjs

Lines 69 to 73 in 30292c4

    
           if ( 
        
             (typeof File !== 'undefined' && value instanceof File) || 
        
             (typeof Blob !== 'undefined' && value instanceof Blob) || 
        
             value instanceof ReactNativeFile 
        
           ) {

The default value stays the same.

function defaultIsFileValue(value) {
  return (
    (typeof File !== 'undefined' && value instanceof File) ||
    (typeof Blob !== 'undefined' && value instanceof Blob) ||
    value instanceof ReactNativeFile
  )
}

@jaydenseric What is your opinion on this? Are there any concerns? Would the impact on the bundle size be too huge? Could you imagine such a feature becoming part of this package and would you accept a pull request that implements this feature?

Don't modify original input

Hey there again!

One more from me, and this time it's a client issue : ] not necessarily a bug, more like asking for opinion.

Your extract-files module traverses the whole GraphQL request looking for files/blobs. Once found these files/blobs are pushed into the upload request while the nodes which contained them are updated with null values. I don't fully understand this but it seems like this is done so that later in the server these nodes can be "reconstructed", i.e. Upload types are inserted where Files were on the client side.

Pseudo example.

Assume that our form data looks like this:

input: {
    description: "Some text",
    mainPhoto: File<whatever is inside the file>
}

After extractFiles(body) it looks like this:

input: {
    description: "Some text",
    mainPhoto: null
}

The problem is that the request input object is modified directly and if the request fails, this input is no longer valid as it no longer has the files/blobs in it. So if the user tries to resubmit the same request, it won't work.

Since most of today's front-end frameworks work based on two-way binding between inputs and form data, this means that I have lost the files that I have selected for input. This then suggests that the simplest way to work around this would be always submitting a deep copy of the form data rather than passing it directly into the mutation, which is somewhat unusual/unnecessary.

Furthermore, creating deep copies of the form data is not exactly a viable solution either, since it is impossible to copy a File object. One needs to explicitly instantiate copy instances of File object for each and every file that's being uploaded.

Hope my explanation is clear. What are your thoughts on this? Is there a way to perhaps create a copy of the input that you receive and work on the copy while leaving the original reference intact?

Keep up the good work! : ]

Typescript type defs

Hey there!

Been using this project recently (love it), but I wanted to ask if there is any chance there might type definitions for this library?

I couldn't find any in DefinitelyTyped, so not sure if maybe someone from the community might see this that maybe rolled their own or if maybe you had this in the works.

Webpack 5 support

CRA with webpack config:

Module not found: Error: Can't resolve 'extract-files/public/ReactNativeFile'
Module not found: Error: Can't resolve 'extract-files/public/extractFiles'
Module not found: Error: Can't resolve 'extract-files/public/isExtractableFile'

Please, review this issue: jaydenseric/apollo-upload-client#230 (comment)

thanks!

"No valid exports main found for ..." Error on node 13.3.0

Error: No valid exports main found for '.../node_modules/extract-files'

This error comes up when I install apollo-upload-client and use this package in an SSR project with Nodjs v13.3.0

More relaxed engine dep?

I'm currently running node v14.10.1 which is incompatible with this package:

    "node": "^12.20 || >= 14.13"

is there any reason the dep can't just be greater than 12.20?

RN: Unable to resolve index file

Hello @jaydenseric, long time no talk. hehe.

I have an issue with extract-files today when I used it in the react-native when the metro bundling source code. It raises the error related to the main field being missing in the package.json

Because the main field is missing so it using the default value as extract-files/index but this file is not exists. Can you update it with a value?

[01:34:09] Error: While trying to resolve module `extract-files` from file `/home/runner/work/xxxx/packages/gql/src/Upload/index.ts`, the package `/home/runner/work/xxxx/packages/gql/node_modules/extract-files/package.json` was successfully found. However, this package itself specifies a `main` module field that could not be resolved (`/home/runner/work/xxxx/packages/gql/node_modules/extract-files/index`. Indeed, none of these files exist:

  * /home/runner/work/xxxx/packages/gql/node_modules/extract-files/index(.native|.ios.ts|.native.ts|.ts|.ios.tsx|.native.tsx|.tsx|.ios.js|.native.js|.js|.ios.jsx|.native.jsx|.jsx|.ios.json|.native.json|.json|.ios.svg|.native.svg|.svg)
  * /home/runner/work/xxxx/packages/gql/node_modules/extract-files/index/index(.native|.ios.ts|.native.ts|.ts|.ios.tsx|.native.tsx|.tsx|.ios.js|.native.js|.js|.ios.jsx|.native.jsx|.jsx|.ios.json|.native.json|.json|.ios.svg|.native.svg|.svg)

{
  "name": "extract-files",
  "version": "12.0.0",
  "description": "A function to recursively extract files and their object paths within a value, replacing them with null in a deep clone without mutating the original value. FileList instances are treated as File instance arrays. Files are typically File and Blob instances.",
  "license": "MIT",
  "author": {
    "name": "Jayden Seric",
    "email": "[email protected]",
    "url": "https://jaydenseric.com"
  },
  "repository": "github:jaydenseric/extract-files",
  "homepage": "https://github.com/jaydenseric/extract-files#readme",
  "bugs": "https://github.com/jaydenseric/extract-files/issues",
  "funding": "https://github.com/sponsors/jaydenseric",
  "keywords": [
    "extract",
    "file",
    "files",
    "File",
    "FileList",
    "Blob",
    "esm",
    "mjs"
  ],
  "files": [
    "extractFiles.mjs",
    "isExtractableFile.mjs"
  ],
  "sideEffects": false,
  "exports": {
    "./extractFiles.mjs": "./extractFiles.mjs",
    "./isExtractableFile.mjs": "./isExtractableFile.mjs",
    "./package.json": "./package.json"
  },
  "engines": {
    "node": "^12.22.0 || ^14.17.0 || >= 16.0.0"
  },
  "browserslist": "Node 12.22 - 13 and Node < 13, Node 14.17 - 15 and Node < 15, Node >= 16, > 0.5%, not OperaMini all, not IE > 0, not dead",
  "dependencies": {
    "is-plain-obj": "^4.0.0"
  },
  "devDependencies": {
    "@types/node": "^17.0.8",
    "coverage-node": "^5.0.1",
    "esbuild": "^0.14.11",
    "eslint": "^8.6.0",
    "eslint-config-env": "^23.0.2",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-compat": "^4.0.0",
    "eslint-plugin-import": "^2.25.4",
    "eslint-plugin-jsdoc": "^37.6.1",
    "eslint-plugin-node": "^11.1.0",
    "eslint-plugin-prettier": "^4.0.0",
    "gzip-size": "^7.0.0",
    "prettier": "^2.5.1",
    "revertable-globals": "^3.0.0",
    "test-director": "^8.0.1",
    "typescript": "^4.6.0-dev.20220110"
  },
  "scripts": {
    "eslint": "eslint .",
    "prettier": "prettier -c .",
    "types": "tsc -p jsconfig.json",
    "tests": "coverage-node test.mjs",
    "test": "npm run eslint && npm run prettier && npm run types && npm run tests",
    "prepublishOnly": "npm test"
  }
}

Are we able to continue to support non .mjs moving forwards?

Error: While trying to resolve module extract-files from file /Users/alexchin/ReactNative/trips_mobileapp/src/hooks/useSetupApolloClient.js, the package /Users/alexchin/ReactNative/trips_mobileapp/node_modules/extract-files/package.json was successfully found. However, this package itself specifies a main module field that could not be resolved (/Users/alexchin/ReactNative/trips_mobileapp/node_modules/extract-files/index

Handle circular references

This will cause an infinite loop in the recursion logic, ending with a Maximum call stack size exceeded error:

const a = {};
a.b = a;
extractFiles(a)

Perhaps the recursion logic should somehow detect when it's revisiting a node that has already been visited, and bail on further recursion? The cloned object should probably contain the same circular references as the original. I'm not sure yet the best way to do all this.

In the meantime, avoid providing extractFiles a value that may contain circular references.

Call signatures are incompatible

Hi, I have the following issue when trying to use your library :

Call signature return types 'Extraction<ExtractableFile>' and '{ clone: Body; files: Map<Extractable, ObjectPath[]>; }' are incompatible.
    The types of 'clone' are incompatible between these types.
      Type 'unknown' is not assignable to type 'Body'.

Formatted:

Here is the code I'm using :

import { NgModule } from '@angular/core';
import { ApolloModule, APOLLO_OPTIONS } from 'apollo-angular';
import { HttpLink } from 'apollo-angular/http';
import { ApolloLink, InMemoryCache } from '@apollo/client/core';
import { setContext } from '@apollo/client/link/context';

import { environment } from 'src/environments/environment';
import { HttpClientModule } from '@angular/common/http';

import extractFiles from 'extract-files/extractFiles.mjs';
import isExtractableFile from 'extract-files/isExtractableFile.mjs';

export function createApollo(httpLink: HttpLink) {
  // ...
  
  const link = ApolloLink.from([
    auth, 
    httpLink.create({
      uri: environment.API_URL,
      extractFiles: (body) => extractFiles(body, isExtractableFile),
    }),
  ]);

  const cache = new InMemoryCache();
  
  return {
    link,
    cache,
  };
}

@NgModule({
  exports: [HttpClientModule, ApolloModule],
  providers: [
    {
      provide: APOLLO_OPTIONS,
      useFactory: createApollo,
      deps: [HttpLink],
    },
  ],
})
export class GraphQLModule {}

My tsconfig file :

{
  "compilerOptions": {
    "baseUrl": "./",
    "outDir": "./dist/out-tsc",
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "noImplicitOverride": true,
    "noPropertyAccessFromIndexSignature": true,
    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": true,
    "sourceMap": true,
    "declaration": false,
    "downlevelIteration": true,
    "experimentalDecorators": true,
    "moduleResolution": "node",
    "importHelpers": true,
    "target": "ES2022",
    "module": "ESNext",
    "allowJs": true,
    "maxNodeModuleJsDepth": 10,
    "skipLibCheck": true,
    "useDefineForClassFields": false,
    "resolveJsonModule": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "lib": [
      "ES2022",
      "dom",
      "esnext.asynciterable"
    ]
  },
}

I don't know if this issue is better here or in the apollo-angular repository, so I've made it here first :p

Output object removes values instead of setting to null

I am writing a Scala implementation of the apollo-upload-server that follows the existing specification here. I ran into an issue with how the extract-files library processes the variables object when it contains a File object. The specification says that all file objects should be replaced with null, however, this line is deleting the field instead. Meaning that for an object like:

{
  "query": "mutation Upload($file: Upload!) { uploadFile(file: $file) }",
  "variables": { file: File }
}

The variables object gets converted into "variables: {}" during file extraction rather than "variable": { file: null }. Which should I be following? Or is the expected behavior just one or the other?

Error: Deprecation warning in build

error (node:3643) [DEP0148]
DeprecationWarning: Use of deprecated folder mapping "./public/" in the "exports" field module resolution of the package at /opt/build/repo/node_modules/extract-files/package.json

Add a license file

Having a clear license text & copyright statement (ideally in a LICENSE file) makes it easier to provide proper attribution and makes it easier for me to get approvals to use.

package.json uses deprecated subpath folder mappings in exports

package.json's exports uses subpath folder mappings (at https://github.com/jaydenseric/extract-files/blob/v9.0.0/package.json#L37), which is deprecated since Node v12.20.0 / v14.13.0, replaced with subpath patterns (which apparently exist since the beginning of exports maps too).
See https://nodejs.org/api/packages.html#packages_subpath_folder_mappings
The main difference is that the required paths have to exactly match, whether you use require() or import to get to the subpath (whereas with subpath folder mappings, the file extension will be automatically added when using require(), but not when using import).

Fwiw, see rollup/plugins#684 (comment) for the details of the Node algorithm applied to the specific case of require("extract-files/public/extractFiles").

imports are broken for @urql/exchange-multipart-fetch

@urql/echange-multipart-fetch has upgraded the extract-files package from version ^8.1.0 to ^11.0.0, which broke an import/export:

yarn run v1.22.11
$ craco build
Creating an optimized production build...
Failed to compile.

/Users/phil/Projects/***redacted***/node_modules/@urql/exchange-multipart-fetch/node_modules/extract-files/public/index.mjs
"export 'default' (reexported as 'ReactNativeFile') was not found in './ReactNativeFile.js'

Version 0.1.12 of @urql/echange-multipart-fetch works fine, 0.1.13 breaks my build process ( create-react-app ).

See: urql-graphql/urql#1795

FileList handler does not work against a genuine FileList

I am currently working with a GraphQL mutation passing multiple files as an input in form of [Upload!], but struggling to get it work. No matter what I do, the output from GraphQL does not seem to send the binaries, but an array of JSON.stringified Files.

Initially the problem was that I was passing an array of File instances, but after reading README of jaydenseric/apollo-upload-client, it seems that I should pass a FileList. Due to using a few 3rd party dependencies, I cannot get a native FileList directly from my code (only an array of Files), so I needed to adapt them to a FileList using a DataTransfer as follows:

    const xfer = new DataTransfer()
    this.files.forEach(f => xfer.items.add(f))
    return xfer.files

According to DataTransfer API (https://developer.mozilla.org/en-US/docs/Web/API/DataTransfer), this should return a FileList, but that one still gets stringified. I checked the related test in this project, and it seems that it expects some object with files method which is not part of FileList.

What is the actual API that should be returned? It looks almost as if it were DataTransfer that should be passed - not FileList?

Support Node version 14.4+

Because my team's project is set to node version 14.4, this package (or packages that use it) are incompatible.

$ yarn add -WD @graphql-eslint/eslint-plugin
yarn add v1.22.17
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
error [email protected]: The engine "node" is incompatible with this module. Expected version "^12.20 || >= 14.13". Got "14.4.0"
error Found incompatible module.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

Because Node version 16 is the latest stable version, I imagine other users will be hitting this problem soon if not already. If possible could this project support later versions?

[DEP0148] DeprecationWarning: Use of deprecated folder mapping
 "./public/" in the "exports" field module resolution of the package at
/Users/.../my-gatsby-project/node_modules/extract-file
s/package.json.

I believe it's coming from this module, which I don't use directly, but through some other module. Can I somehow fix the error in my repo, or it should be done in this repo ?

TypeScript support

Do you will have plan to support typescript?

	if (
	(typeof File !== 'undefined' && value instanceof File) \|\|
	(typeof Blob !== 'undefined' && value instanceof Blob) \|\|
	value instanceof ReactNativeFile
	) {