openinf/openinf-gh-file-importer

Utility that imports arbitrary files from remote GitHub repositories

@openinf/gh-file-importer

Utility that imports arbitrary files from remote GitHub repositories

The high-level goal of @openinf/gh-file-importer is to serve as a Node.js package containing a utility for importing arbitrary files from remote GitHub repos allowing users to make use of them locally. We are constantly working to improve this repository, so please feel free to contribute if you notice any omissions or errors.

Thanks!

Supported Node.js Environments

• v4：Argon (Ar)
• v6：Boron (B)
• v8：Carbon (C)
• v10：Dubnium (Db)
• v12：Erbium (Er)
• v14：Fermium (Fm)
• v16：Gallium (Ga)
• v18：Hydrogen (H)

---

Table of Contents

• Installation
• Usage
• API
• Contributing
• License

---

Installation

@openinf/gh-file-importer runs on supported versions of Node.js and is available via npm, pnpm, or yarn.

Using the npm CLI

^{See the official documentation for this command for more information.}

npm i @openinf/gh-file-importer

Using the pnpm CLI

^{See the official documentation for this command for more information.}

pnpm add @openinf/gh-file-importer

Using the Yarn 1 CLI (Classic)

^{See the official documentation for this command for more information.}

yarn add @openinf/gh-file-importer

Usage

Import the GhFileImporter constructor based on your platform.

Node.js

import { GhFileImporter } from '@openinf/gh-file-importer';

Options

Now instantiate your API. All options are optional except for destDir, which is the location where your files will be stored.

import { GhFileImporter } from '@openinf/gh-file-importer';

const DIR_TEMP = './tmp';

const ghFileImporter = new GhFileImporter({ destDir: DIR_TEMP });

await ghFileImporter.importContents('tc39', 'proposals', 'README.md');

Note: if needing to circumvent exceeding the GitHub API rate limit, be sure to have an environment variable called GITHUB_TOKEN containing a GitHub person access token.

Logging

For custom logging, pass an object with debug, info, warn, and error methods as the log option.

const ghFileImporter = new GhFileImporter({
  destDir: DIR_TEMP,
  log: {
    debug: () => {},
    info: () => {},
    warn: console.warn,
    error: console.error
  }
};

Debug

The simplest way to receive debug information is to set the log client option to console.

const ghFileImporter = new GhFileImporter({
  destDir: DIR_TEMP,
  log: console,
});

If you like to support a configurable log level, we recommend using the console-log-level module.

const ghFileImporter = new GhFileImporter({
  destDir: DIR_TEMP,
  log: require("console-log-level")({ level: "info" }),
});

API

GhFileImporter

Kind: global class

• GhFileImporter
  • new GhFileImporter(options)
  • .fetchMetadata(owner, repo, path, ref) ⇒ Promise<any>
  • .fetchFileContents(owner, repo, path, ref) ⇒ Promise<string>
  • .fetchFileContentsFromUrl(url) ⇒ Promise<string>
  • .importContents(url) ⇒ Promise<string>
  • .importContentsFromUrl(url) ⇒ Promise<string>

new GhFileImporter(options)

Creates an instance of GhFileImporter.

Throws:

• InvalidArgTypeError
• InvalidArgValueError
• InvalidPropertyValueError
• MissingArgsError
• MissingOptionError

Param	Type	Description
options	GhFileImporterOptions | undefined	The options object.

ghFileImporter.fetchMetadata(owner, repo, path, ref) ⇒ Promise<any>

Retrieves a repo or path's metadata.

Kind: instance method of GhFileImporter
Returns: Promise<any> - An object containing the metadata repo or path's metadata.
Throws:

• InvalidArgTypeError
• InvalidArgValueError
• InvalidArgsNumberError

See: https://docs.github.com/en/rest/reference/repos#get-repository-content

Param	Type	Description
owner	string	The username associated with the repository.
repo	string	The repository name.
path	string | undefined	The path to the file or folder.
ref	string | undefined	The name of the commit/branch/tag.

ghFileImporter.fetchFileContents(owner, repo, path, ref) ⇒ Promise<string>

Retrieves a path's contents.

Kind: instance method of GhFileImporter
Returns: Promise<string> - The file contents.
Throws:

• InvalidArgTypeError
• InvalidArgValueError
• InvalidArgsNumberError

Param	Type	Description
owner	string	The username associated with the repository.
repo	string	The repository name.
path	string	The path to the file or folder.
ref	string | undefined	The name of the commit/branch/tag.

ghFileImporter.fetchFileContentsFromUrl(url) ⇒ Promise<string>

Retrieves the file contents from the URL provided.

Kind: instance method of GhFileImporter
Returns: Promise<string> - The file contents.
Throws:

• InvalidArgTypeError
• InvalidArgValueError

Param	Type	Description
url	string	The string representation of a remote file URL.

ghFileImporter.importContents(url) ⇒ Promise<string>

Imports a file into the directory provided for the destDir option.

Kind: instance method of GhFileImporter
Returns: Promise<string> - The file contents.
Throws:

• InvalidArgTypeError
• InvalidArgValueError

Param	Type	Description
url	string	The string representation of a remote file URL.

ghFileImporter.importContentsFromUrl(url) ⇒ Promise<string>

Imports a file located at the supplied URL into the directory provided for the destDir option.

Kind: instance method of GhFileImporter
Returns: Promise<string> - The file contents.
Throws:

• InvalidArgTypeError
• InvalidArgValueError

Param	Type	Description
url	string	The string representation of a remote file URL.

---

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. If for whatever reason you spot something to fix but cannot patch it yourself, please open an issue.

License

This project is licensed under either of

• Apache License, Version 2.0
• MIT license

at your option.

The SPDX license identifier for this project is MIT OR Apache-2.0.

---

Show Your Support

If you like the project (or want to bookmark it) —
— give it a star ⭐️ — it will greatly encourage us.

openinf-gh-file-importer's Issues

TypeScript 4.9 - 5.x Improvements

Oops — #139 landed first w/o having first looked into it… After reading the release post, i am quite sure that this project could benefit from the new features it comes w/ and we could end up w/ a much more elegant implementation & cleaner codebase, which are both desperately needed w/ this project.

🔗 https://devblogs.microsoft.com/typescript/announcing-typescript-4-9/

Originally posted by @DerekNonGeneric in #138 (comment)

[feature request] prevent including unnecessary files in the deployed package

We should consider adding the "files" stanza to the package.json to create an explicit allowlist and prevent deploying any unnecessary files to the npm registry as part of our package. This would be preferable to using a .npmignore file, which we have been doing so far but is expected to quickly become unmanageable as more files are added.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• build(deps-dev): bump typescript to v5.4.5
• chore(deps): bump pnpm to v8.15.8
• chore(deps): bump actions/checkout action to v4
• chore(deps): bump pnpm to v9
• chore(deps): bump ubuntu to v22
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

github-actions

.github/workflows/semgrep.yml

• actions/checkout v3
• ubuntu 20.04

npm

package.json

• @types/node 20.x.x
• typescript 5.1.3
• pnpm 8.6.3

nvm

.nvmrc

• node 20

---

• Check this box to trigger a request for Renovate to run again on this repository

Hacktoberfest checklist: prepare project for contributions by following best practices

• Add the “hacktoberfest” topic to your repository to opt-in to Hacktoberfest and indicate you’re looking for contributions.

• Apply the “hacktoberfest” label to issues you want contributors to help within your GitHub or GitLab project.

• Add a CONTRIBUTING.md file with contribution guidelines to your repository.

• Choose issues that have a well-defined scope and are self-contained.

• Adopt a code of conduct to create a greater sense of inclusion and community for contributors.

• Be ready to review pull/merge requests, accepting those that are valid by merging them, leaving an overall approving review, or by adding the “hacktoberfest-accepted” label.

• Reject any spammy requests you receive by labeling them as “spam”, and any other invalid contributions by closing them or labeling them as “invalid”.

Refs: https://hacktoberfest.com/participation/#maintainers

[feature request] support default value for destination in options bag

Currently, it is not very well documented, but the options bag passed to the class constructor at initialization requires that a String be supplied for the location of the destination directory for the files/folders imported from upstream repos.

Firstly, rather than simply expecting a String, the type should be narrowed to perhaps only accept a valid filesystem path.

In the event that this field is omitted, such as in the case where the user may not necessarily be interested in the file(s) or may not have a particular preference where files are stored, a smart default for the canonical temporary directory should be used.

For Node.js environments with node:os core module available, the value returned by the os.tmpdir() method may be useful to consider. In the case of browser environments, the Window.sessionStorage() Web API may be of interest.

build: remove build products & migrate to babel for transpilation

It is preferable not to check the compiled typescript into our repo — let's remove that javascript from here. Additionally, we should use babel for this transpilation instead of the typescript compiler. We will, however, continue using tsc for type checking and type definition generation.

[feature request] detect and defer to native implementations of `fetch` global

Detect and defer to native implementations

For features that have reached the tipping point, where the polyfill registers the same name as the native feature, include code to detect a native implementation, and if one exists, defer to it. Where a native feature exists but is incomplete or buggy, consider using as much of it as possible and correcting only the deficiencies.

— https://www.w3.org/2001/tag/doc/polyfills/#detect-and-defer-to-native-implementations

Additionally, consider replacing node-fetch dependency with undici for behavior guarantees.

[windows] constraints on max length of filesystem paths

Windows has a maximum length of 260 characters (by default), but can be increased by overriding environment variables (i think). I learned about this when installing Python 3 on Windows, which allows you to increase this size. We cannot really make these kinds of OS-level configuration changes w/ a package of this nature.

We should ensure that this limit is not exceeded by this package — I am not sure what happens if it does exceed this length, but probably nothing desirable.

const MAX_PATH_LENGTH:number = 260;

I don't yet have any references to link to for this…

Something from Windows Server Docs would be great (preferably Windows Server 2022).

[bug] casing rules contrary to Web ecosystem naming conventions

According to the Web Platform Design Principles outlined by members of the W3C TAG, our API is at least in violation of guidelines laid out in the section on naming principles.

Use Web consistent names

When choosing a name for feature or API that has exposure in other technology stacks, the preference should be towards the Web ecosystem naming convention rather than other communities.

	Casing rule	Examples
Methods and properties (Web IDL attributes, operations, and dictionary keys)	Camel case	createAttribute() compatMode
Classes and mixins (Web IDL interfaces)	Pascal case	NamedNodeMap NonElementParentNode
Initialisms in APIs	All caps, except when the first word in a method or property	HTMLCollection innerHTML bgColor
Repeated initialisms in APIs	Follow the same rule	HTMLHRElement RTCDTMFSender
The abbreviation of "identity"	Id, except when the first word in a method or property	getElementById() pointerId id
Enumeration values	Lowercase, dash-delimited	"no-referrer-when-downgrade"
Events	Lowercase, concatenated	canplaythrough languagechange
HTML elements and attributes	Lowercase, concatenated	figcaption maxlength
JSON keys	Lowercase, underscore-delimited	short_name

Refs: https://w3ctag.github.io/design-principles/#casing-rules

---

GhFileImporter

Kind: global class

• GhFileImporter
  • new GhFileImporter(options)
  • .fetchMetadata(owner, repo, path, ref) ⇒ Promise.<any>
  • .fetchFileContents(owner, repo, path, ref) ⇒ Promise.<string>
  • .fetchFileContentsFromUrl(url) ⇒ Promise.<string>
  • .importContents(url) ⇒ Promise.<string>
  • .importContentsFromUrl(url) ⇒ Promise.<string>

new GhFileImporter(options)

Creates an instance of GhFileImporter.

Situation	Casing rule	Status
Classes and mixins (Web IDL interfaces)	Pascal case	ⓧ
Initialisms in APIs	All caps, except when the first word in a method or property	ⓧ

---

The API's class name and other parts will need to be re-written. The class name should likely be GHFileImporter, and the names of the methods ending in *FromUrl would need to be *FromURL are just a couple of things I was able to catch at a quick glance…

This will be a very disruptive breaking change — in this issue, we are offered a brief glimpse at how the new v3 API will be changing to be designed more like a Web Platform API, so that should be an exciting thing to look forward to for the August release happening later this month.

(JS-0323) Detected usage of the `any` type

Description

The any type can sometimes leak into your codebase. TypeScript compiler skips the type checking of the any typed variables, so it creates a potential safety hole, and source of bugs in your codebase. We recommend using unknown or never type variable.

Occurrences

There is 1 occurrence of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/openinf/gh-file-importer/issue/JS-0323/occurrences/