Giter Site home page Giter Site logo

lightning-fs's Introduction

isomorphic-git

isomorphic-git is a pure JavaScript reimplementation of git that works in both Node.js and browser JavaScript environments. It can read and write to git repositories, fetch from and push to git remotes (such as GitHub), all without any native C++ module dependencies.

Goals

Isomorphic-git aims for 100% interoperability with the canonical git implementation. This means it does all its operations by modifying files in a ".git" directory just like the git you are used to. The included isogit CLI can operate on git repositories on your desktop or server.

This library aims to be a complete solution with no assembly required. The API has been designed with modern tools like Rollup and Webpack in mind. By providing functionality as individual functions, code bundlers can produce smaller bundles by including only the functions your application uses.

The project includes type definitions so you can enjoy static type-checking and intelligent code completion in editors like VS Code and CodeSandbox.

Project status

The original author of the project (Billie Hilton) left the project, but the project is still maintained by two volunteers:

But they don't write much code, mainly do code review and try to answer to issues and on Gitter, they just don't want the project to die. So you can say that this project is community driven (as jcubic always reply to issues). Which means that if you want a feature to be implemented you need to do this yourself or find someone that is willing to write the code for you. The project have some money on OpenCollective and we can spend it on some development, if you find somoene that is willing to code in exchange to some bucks (it may be you), but we don't have a lot so don't expect to have full sallary.

If you want to help this project you're more than welcome to do so.

Supported Environments

The following environments are tested in CI and will continue to be supported until the next breaking version:


Node 10

Chrome 79

Edge 79

Firefox 72

Safari 13

Android 10

iOS 13

Upgrading from version 0.x to version 1.x?

See the full Release Notes on GitHub and the release Blog Post.

Install

You can install it from npm:

npm install --save isomorphic-git

Getting Started

The "isomorphic" in isomorphic-git means that the same code runs in either the server or the browser. That's tricky to do since git uses the file system and makes HTTP requests. Browsers don't have an fs module. And node and browsers have different APIs for making HTTP requests!

So rather than relying on the fs and http modules, isomorphic-git lets you bring your own file system and HTTP client.

If you're using isomorphic-git in node, you use the native fs module and the provided node HTTP client.

// node.js example
const path = require('path')
const git = require('isomorphic-git')
const http = require('isomorphic-git/http/node')
const fs = require('fs')

const dir = path.join(process.cwd(), 'test-clone')
git.clone({ fs, http, dir, url: 'https://github.com/isomorphic-git/lightning-fs' }).then(console.log)

If you're using isomorphic-git in the browser, you'll need something that emulates the fs API. The easiest to setup and most performant library is LightningFS which is written and maintained by the same author and is part of the isomorphic-git suite. If LightningFS doesn't meet your requirements, isomorphic-git should also work with BrowserFS and Filer. Instead of isomorphic-git/http/node this time import isomorphic-git/http/web:

<script src="https://unpkg.com/@isomorphic-git/lightning-fs"></script>
<script src="https://unpkg.com/isomorphic-git"></script>
<script type="module">
import http from 'https://unpkg.com/isomorphic-git@beta/http/web/index.js'
const fs = new LightningFS('fs')

const dir = '/test-clone'
git.clone({ fs, http, dir, url: 'https://github.com/isomorphic-git/lightning-fs', corsProxy: 'https://cors.isomorphic-git.org' }).then(console.log)
</script>

If you're using ES module syntax, you can use either the default import for convenience, or named imports to benefit from tree-shaking if you are using a bundler:

import git from 'isomorphic-git'
// or
import * as git from 'isomorphic-git'
// or
import {plugins, clone, commit, push} from 'isomorphic-git'

View the full Getting Started guide on the docs website.

Then check out the Useful Snippets page, which includes even more sample code written by the community!

CORS support

Unfortunately, due to the same-origin policy by default isomorphic-git can only clone from the same origin as the webpage it is running on. This is terribly inconvenient, as it means for all practical purposes cloning and pushing repos must be done through a proxy.

For this purpose @isomorphic-git/cors-proxy exists which you can clone or npm install. For testing or small projects, you can also use https://cors.isomorphic-git.org - a free proxy sponsored by Clever Cloud.

I'm hoping to get CORS headers added to all the major Git hosting platforms eventually, and will list my progress here:

Service Supports CORS requests
Gogs (self-hosted)
Gitea (self-hosted)
Azure DevOps (Usage Note: requires authentication)
Gitlab ❌ My PR was rejected, but the issue is still open!
Bitbucket
Github

It is literally just two lines of code to add the CORS headers!! Easy stuff. Surely it will happen.

isogit CLI

Isomorphic-git comes with a simple CLI tool, named isogit because isomorphic-git is a lot to type. It is really just a thin shell that translates command line arguments into the equivalent JS API commands. So you should be able to run any current or future isomorphic-git commands using the CLI.

It always starts with an the assumption that the current working directory is a git root. E.g. { dir: '.' }.

It uses minimisted to parse command line options and will print out the equivalent JS command and pretty-print the output JSON.

The CLI is more of a lark for quickly testing isomorphic-git and isn't really meant as a git CLI replacement.

Supported Git commands

This project follows semantic versioning, so I may continue to make changes to the API but they will always be backwards compatible unless there is a major version bump.

commands

Community

Share your questions and ideas with us! We love that. You can find us in our Gitter chatroom or just create an issue here on Github! We are also @IsomorphicGit on Twitter.

Contributing to isomorphic-git

The development setup is similar to that of a large web application. The main difference is the ridiculous amount of hacks involved in the tests. We use Facebook's Jest for testing, which make doing TDD fast and fun, but we also used custom hacks so that the same tests will also run in the browser using Jasmine via Karma. We even have our own mock server for serving git repository test fixtures!

You'll need node.js installed, but everything else is a devDependency.

git clone https://github.com/isomorphic-git/isomorphic-git
cd isomorphic-git
npm install
npm test

Check out the CONTRIBUTING document for more instructions.

Who is using isomorphic-git?

Similar projects

Acknowledgments

Isomorphic-git would not have been possible without the pioneering work by @creationix and @chrisdickinson. Git is a tricky binary mess, and without their examples (and their modules!) I would not have been able to come even close to finishing this. They are geniuses ahead of their time.

Cross-browser device testing is provided by:

BrowserStack

SauceLabs

Contributors

Thanks goes to these wonderful people (emoji key):


William Hilton

📝 🐛 💻 🎨 📖 💡 ⚠️

wDhTIG

🐛

Marc MacLeod

🤔 🔍

Brett Zamir

🤔

Dan Allen

🐛 💻 🤔

Tomáš Hübelbauer

🐛 💻

Juan Campa

🐛 💻

Ira Miller

🐛

Rhys Arkins

💻

Sean Larkin

💻

Daniel Ruf

💻

bokuweb

💻 📖 ⚠️

Hiroki Osame

💻 📖

Jakub Jankiewicz

💬 🐛 💻 💡 ⚠️

howardgod

🐛 💻

burningTyger

🐛

Melvin Carvalho

📖

akaJes
💻

Dima Sabanin

🐛 💻

Koutaro Chikuba

🐛 💻

Hubert SABLONNIÈRE

💻 ⚠️ 🤔 🔍

David Duarte

💻

Thomas Pytleski

🐛 💻

Vadim Markovtsev

🐛

Yu Shimura

🤔 💻 ⚠️

Dan Lynch

💻

Jeffrey Wescott

🐛 💻

zebzhao

💻

Tyler Smith

🐛

Bram Borggreve

🐛

Stefan Guggisberg

🐛 💻 ⚠️

Catalin Pirvu

💻

Nicholas Nelson

💻 ⚠️

Anna Henningsen

💻

Fabian Henneke

🐛 💻

djencks

🐛 💻 ⚠️

Clemens Wolff

💻 📖 ⚠️

Sojin Park

💻

Edward Faulkner

💻

Khải

🐛

Corbin Crutchley

💻 📖 ⚠️

Riceball LEE

💻 📖 ⚠️

lin onetwo

💻

林法鑫

🐛

Will Stott

💻 ⚠️

Seth Nickell

🐛

Alex Titarenko

💻

Misha Kaletsky

💻

Richard C. Zulch

💻 📖

mkizka

💻

RyotaK

🐛

Noah Hummel

💻 ⚠️

Mike Lewis

📖

Sam Verschueren

💻

Vitor Luiz Cavalcanti

📖

Shane McLaughlin

💻 📖 ⚠️

Sean Poulter

🚧

araknast

💻 ⚠️ 📖

Rafael Raab

💻 📖

Lukáš Cezner

💻 📖 ⚠️ 🐛

dead-end

💻 📖 ⚠️

Barry

💻 📖 ⚠️

Alireza Mirian

💻 📖 ⚠️ 🐛

DanilKazanov

💻 📖 ⚠️

Eyal Hisco

🐛

Sebastien

💻

Yaroslav Halchenko

📖

Alex Villarreal

💻

Ben Morrow

💻

jayree

💻 ⚠️

Lucas Martin Segurado

📖 🐛

Leon Kaucher

💻 ⚠️

Gili Shohat

💻 📖 ⚠️

Habib

💻 📖 ⚠️

Vinzent

💻

This project follows the all-contributors specification. Contributions of any kind welcome!

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

This work is released under The MIT License

FOSSA Status

lightning-fs's People

Contributors

billiegoose avatar elegaanz avatar fuzzytew avatar jcubic avatar jesseditson avatar joelspadin avatar pedroapfilho avatar raldone01 avatar tachibana-shin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lightning-fs's Issues

Integration with Emscripten

I've been using isomorphic-git with lightning-fs and it's working great so far, but the next stage of my project is calling an emscripten-built assembler. Unfortunately, lightning-fs doesn't seem to implement the mount functionality emscripten expects for integration like BrowserFS.

Is it already possible to utilize lightning-fs within emscripten? If not, I think it would be really useful for lightning-fs to work with emscripten out of the box similar to BrowserFS. The full integration from Git to executing build tools all within the same file system is really appealing.

Missing Lightning-FS from isomorphic-git

Hi,
A recently installed isomorphic-git into my project via "npm install --save isomorphic-git". But after installing there is no directory "lightning-fs" and so i cant access lightning-fs.
Is there ani solution for this ?
Thanks in advance for your answer.

Access to path module from browser

I'm using dist/lightning-fs.min.js from https://cdn.jsdelivr.net/ and it seems that path module is not exposed. It would be nice if you don't need to use browser-fs just to have path.join and path.split.

Or is there a way to access it?

Convert to TypeScript

Please convert the codebase to TypeScript, as it may help contributors make changes, and it will help with type-checking and other errors. I am working on a PR for this, and it should be here very soon.

It is possible to write in a directory as if it was a file

writeFile will happily write to a path that is currently a directory. The children of the directory will be kept, but its type will become 'file' and thus any subsequent operation will throw ENOTDIR.

Here is a minimal example.

import FS from "@isomorphic-git/lightning-fs";

const fs = new FS("demo");
const pfs = fs.promises;

const main = async () => {
  await pfs.mkdir("/foo");
  await pfs.writeFile("/foo", "bar");
  await pfs.readdir("/foo"); // throws ENOTDIR
};

main();

In Node, the following error is thrown by writeFile: Uncaught [Error: EISDIR: illegal operation on a directory, open 'test']

How to speedup writeFile?

I have a code to read a real folder from File Access API into lightningFS.

The repo is about 800kb, has 300 files.

The code works very slowly, because of writeFile (~1.3 sec).

I've read that indexedDB throttles writeFile, is that so?
How to speedup?

Here's the code, it recursively reads all dirs/files and uses fs.promises.writeFile to write them to lightningFS.

This writeFile call is the main reason for the delay, even though the data is very small.

  let fs = new LightningFS('fs', {wipe: true});

  let relPath = [''];

  async function handle(dirHandle) {
    console.time(dirHandle.name);

    for await (const entry of dirHandle.values()) {
      if (entry.kind === "file") {
        const file = await entry.getFile();
        let data = new Uint8Array(await file.arrayBuffer());
        let filePath = [...relPath, file.name].join('/');
        await fs.promises.writeFile(filePath, data);
      }

      if (entry.kind === "directory") {
        const newHandle = await dirHandle.getDirectoryHandle( entry.name, { create: false } );
        relPath.push(entry.name);
        let dirPath = relPath.join('/');
        await fs.promises.mkdir(dirPath);
        await handle(newHandle);
        relPath.pop();
      }
    }
  }

P.S. Is there any other backend for lightningFS? I need a simple in-memory strorage.
It's quite ironic that lightningFS is so sluggish for a tiny test repo.
Maybe I'm doing something wrong?

Clarify writeFile "data" Documentation

This is a minor thing, but since I just spent an embarrassingly long amount of time puzzling over it, I figured I’d mention it.

The documentation for fs.writeFile in the README mentions that "data should be a string of a Uint8Array."—which I interpreted as any data has to be converted into a Uint8Array and then encoded into a string. Looking through the source I noticed, however, that the function actually does the encoding for us if a string is passed.

Maybe this could be clarified? It might be just me, but the wording along with the amounts of time I have seen "or" misspelled as "of" really confused me here. 😅

Either way, thank you so much for this library (and the rest of Isomorphic Git!).

Happy holidays, if you celebrate.

Deleting an fs

Is there a way to reset or delete an FS (it's indexdb store)?

Compare-and-swap like operation

Hello, first of all thank you for the great libraries lightning-fs and isomorphic-git!

I am interested in implementing something like Git's push --force-with-lease which allows you to require that a ref has a particular hash before changing it (to prevent multi-tab conflicts), similar to how a compare-and-swap works.

How tricky do you think would it be to do something like this, due to the caching Lightning-FS does? Would an approach of grabbing a Web Lock, syncing the FS, and doing a read and write (while holding the lock) work?

Not thread safe

When two instances point to the same underlying IndexedDb they clobber each other. This is particularly frustrating when working with Workers, because only a single Worker can use the file system. But it also is the same underlying cause of #16.

Problem with backend option

I am creating a wrapper of my fs using lighting fs backend option but I am not able to perform these stuff:

  • create file
  • write file
  • mkdir etc

And all other operation work well like read, stat, etc...

What should I do to perform these create, write, etc stuff?

Compressed data

Hey. Do you have any experience using compressed data (like zip) with lightning-fs? My use case involves sending quite large file tree from the server and I want it to be synchronously available on the client. BrowserFS has ZipFS.

Incorrect `IDB` interface

I noticed that current IDB interface had two errors:

  1. The constructor type was typed as a regular function
  2. The readFile was mis-typed as loadFile

/lightning-fs/index.d.ts

export interface IDB {
- constructor(dbname: string, storename: string): IDB
+ new (dbname: string, storename: string): IDB
  saveSuperblock(sb: Uint8Array): TypeOrPromise<void>
  loadSuperblock(): TypeOrPromise<FS.SuperBlock>
- loadFile(inode: number): TypeOrPromise<Uint8Array>
+ readFile(inode: number): TypeOrPromise<Uint8Array>
  writeFile(inode: number, data: Uint8Array): TypeOrPromise<void>
  wipe(): TypeOrPromise<void>
  close(): TypeOrPromise<void>
}

ENOENT when calling stat on directory with slash at the end

I'm trying to use my old service worker for server like file access but it don't work with lightingFS because:

fs.stat('/foo/', (e) => { console.log(e)});

show error:

ENOENT: /foo/

the directory exists because fs.stat('/foo', ...) is working fine.

I need this to be working because I'm distinguishing directories from files using slash at the end of the directory. Probably my worker can be refactored but it would be nice if the library work the same as Node fs.

Same fs instance everywhere

Hello @wmhilton and team 😉

One liner

I think the docs should make it very important that your app needs to have one and only one instance of new FS('fs').

Details

We only tested on Firefox, but in one of our codebase we had several instances of fs (in different modules) with const fs = new FS('fs'). We hoped they would be in sync since it's the same IndexedDB behind. It seems like there is a cache layer or something with the transactions. Maybe it comes from how idb-keyval works.

Anyway, if you don't use the same fs instance when you try to readdir or any other fs operations than the instance you used with git.plugins.set('fs', fs); you will have sync problems.

I know the getting started for browser mentions a global var but it does not say much more.

Questions

  • Is my diagnose wrong?
  • What do you think about this?
  • Should we improve the docs?
  • Should we find a way for several new FS('fs') to reuse the same wrapper behind the scenes?

Thanks 😄

trying to build on sveltekit; indexedDB is not defined

ReferenceError: indexedDB is not defined
    at /home/vera/global-agora/node_modules/.pnpm/@[email protected]/node_modules/@isomorphic-git/idb-keyval/dist/idb-keyval-cjs.js:17:29
    at new Promise (<anonymous>)
    at Store._init (/home/vera/global-agora/node_modules/.pnpm/@[email protected]/node_modules/@isomorphic-git/idb-keyval/dist/idb-keyval-cjs.js:16:21)
    at new Store (/home/vera/global-agora/node_modules/.pnpm/@[email protected]/node_modules/@isomorphic-git/idb-keyval/dist/idb-keyval-cjs.js:10:14)
    at new IdbBackend (/home/vera/global-agora/node_modules/.pnpm/@[email protected]/node_modules/@isomorphic-git/lightning-fs/src/IdbBackend.js:7:19)
    at DefaultBackend.init (/home/vera/global-agora/node_modules/.pnpm/@[email protected]/node_modules/@isomorphic-git/lightning-fs/src/DefaultBackend.js:30:23)
    at PromisifiedFS._init (/home/vera/global-agora/node_modules/.pnpm/@[email protected]/node_modules/@isomorphic-git/lightning-fs/src/PromisifiedFS.js:84:27)
 ELIFECYCLE  Command failed with exit code 1.

Read folder and subfolders as a single file

I'm sorry, this library seems great but not having any experience with browser storage I'm finding the docs very counter-intuitive.
I managed to make it do most of what I need, but as the title suggests I'm trying to understand if there's a way to access a folders & its subfolders as a single file (to, for example, download it, or send it via webRTC, or anything else)

HTTP Backing Without Superblock File

It seems when a url option is passed, a .superblock.txt file must be provided at the url, listing the available files. Creating this file is not always possible, and when it is missing, the _cache.stat() check in readFile at https://github.com/isomorphic-git/lightning-fs/blob/master/src/PromisifiedFS.js#L150 throws an error, preventing the reading of any remote files.

It might be nice if there were a way to specify that files are present when the superblock file is not, or when it is missing content. A method like fs.backFile() maybe that would add an entry to the cache.

Orthogonally, that readFile line could be skipped when there is no superblock file, maybe a HEAD request performed instead.

pfs.rename deletes file if oldFilepath and newFilepath are equal

This seems related to #23. If oldFilepath and newFilepath are equal, then the file is deleted; I'm not sure if this is the default node fs behavior, but we can see this is because the old file is deleted after the new file is inserted here:

  rename(oldFilepath, newFilepath) {
    let basename = path.basename(newFilepath);
    // Note: do both lookups before making any changes
    // so if lookup throws, we don't lose data (issue #23)
    // grab references
    let entry = this._lookup(oldFilepath);
    let destDir = this._lookup(path.dirname(newFilepath));
    // insert into new parent directory
    destDir.set(basename, entry);
    // remove from old parent directory
    this.unlink(oldFilepath)
  }

A simple fix would be to check for that condition, and return without doing anything:

  rename(oldFilepath, newFilepath) {
    if (oldFilepath == newFilepath)
      return;
    ...
  }

fs.rename

fs.rename would be great and I imagine it might not be super trick to implement.

Is a PR welcome? I understand you would like to keep the API slim.

Add a way to know the current "init" status of the FS

The FS should be able to tell what is the current name with which it was initialized the last time (And maybe the options). Right now the only way to know the name is by querying fsInstance.promises._backend._store and check if it is undefined or if it has a _storeName this would be useful to not initialize the same database twice when you have multiple databases in the same thread that depend on user options

ENOENT on existing resource

Hi - thanks for the great projects.

I'm having trouble with readFile. I'm getting the following error:

Error: ENOENT: /TypescriptGeneratorTest
    at CacheFS._lookup (CacheFS.js:105)
    at CacheFS.writeFile (CacheFS.js:154)
    at superblockPromise.then (index.js:146)

with the code posted below. Note: I tried the utf8 option directly from a string and, per the code below, I'm now encoding the data to Uint8Array. The path fed to the writeFile is actually /TypescriptGeneratorTest/index.js thought the error only shows the directory - which definitely exists. What am I missing?

export function writeFileToFS (contentMetaData, content) {
    var filePath = contentMetaData.filePath
    if (!("TextEncoder" in window)) {
        alert("Sorry, this browser does not support TextEncoder...")
    } else {
        var enc = new TextEncoder(); // always utf-8
        var encodedContent = enc.encode(content)
        console.log('encoded content: ')
        console.log(encodedContent)
        window.pfs.writeFile(filePath, encodedContent, ((err) => {
            if(err) {
                console.error('could not write file to file system due to the following error: ')
                console.log(err)
            } else { // if successful, replace the rawContentItem in the store with the new one from the filesystem
                console.log('the following file was written succesfully to the file system: ' + filePath)
                readFileToStore(filePath)
            }
        }))
    }
}

Events

Are there any plans to support events? More specifically, has there been any interest in implementing the FSWatcher class from the fs API?

Data Serialisation

Hmmm, that doesn't look right:

screenshot from 2019-03-06 00-13-32

Firefox,

{
  "@isomorphic-git/lightning-fs": "^3.0.3",
  "isomorphic-git": "^0.51.12"
}

I use webpack to bundle for the browser.

Why does readdir not return an alphabetized list?

I ran into unexpected behavior in an app because LightningFS' readdir can return non-sorted strings. The following example shows the issue in Node.js (make sure you npm install --save-dev fake-indexeddb first), but the issue was first identified in the browser (Firefox and Safari):

require("fake-indexeddb/auto"); // HAS to come first for Node.js to have indexedDB
const FS = require('@isomorphic-git/lightning-fs');

const fs = new FS("testfs", {wipe: true});
const pfs = fs.promises;

(async function() {
  await pfs.mkdir('/hi');
  const max = 15;
  const randN = Array.from(Array(50), _ => Math.floor(Math.random() * max));
  // write & append fifteen files in random order with random data
  for (const r of randN) {
    const fname = `/hi/${r}`;
    pfs.writeFile(fname, Math.random());
  }
  const contents = await pfs.readdir('/hi');
  console.log(contents);
})();

The script above writes to a set of fifteen files (whose names are between '0' and '14') in random order, and then prints the output of readdir. Running this script several times shows the issue: an unsorted list of files:

$ node demo.js
[
  '13', '2', '14', '9',
  '4',  '3', '8',  '11',
  '0',  '1', '7',  '12',
  '10', '6', '5'
]
$ node demo.js
[
  '7',  '14', '4',  '1',
  '2',  '8',  '10', '3',
  '9',  '13', '11', '5',
  '12', '6',  '0'
]
$ node demo.js
[
  '8',  '11', '14', '0',
  '13', '3',  '4',  '5',
  '6',  '9',  '7',  '12',
  '10', '2'
]

This behavior persists whether I use numeric or alphanumeric filenames.

In contrast, if I use Node's fs module, readdir returns sorted strings.

Is this an IndexedDB limitation? If it is, a sort on each readdir may be too expensive. But if not, maybe there's a workaround in the library itself?

Loving the LightningFS + isomorphic-git lifestyle, thank you for your continued hard work on these projects!

feat: add native 'exists' implementation

In theory, a native implementation of exists that doesn't throw an Error and immediately catch it might be ever-so-slightly faster. Not sure what the overhead of creating an Error object is... it can be "expensive" because it creates a stack trace - although I think most browsers compute the stack trace only if needed nowadays. Hmm.

Originally posted by @wmhilton in #21 (comment)

Deleting a git repo

First of all, thank you very much for the awesome isomorphic-git and this library!

My questions is:
Is there a way to delete all the files in a git repo?
I tried to rimraf the repo directory like "/repo_name" but the files are still in IndexedDB.

Usage in Typescript

Can you please let me about using this in TypeScript. currently there are no typings available.

Error:

  Try `npm install @types/isomorphic-git__lightning-fs` if it exists or add a new declaration (.d.ts) file containing `declare module '@isomorphic-git/lightning-fs';`ts(7016)

Mutex Timeout

we are getting the following on using isomorphic-git and lightning fs in our app - is this an expected condition?

image

Allow a recursive option on rmdir

The rmdir function in newer versions of Node allows for a recursive option that when set to true makes the command behave more like a rm -rf, which I think can be very useful in many situations (such as deleting a cloned but no longer needed git repo for example).

The documentation of lightning-fs suggests that its rmdir also takes an options object, but I browsing through the source I couldn’t find anything that seemed to be using that object (I might have missed something though 😅). I feel like adding a native recursive option could really benefit the library and perhaps would even allow for a more efficient way than a recursive "rimraf" implemented on top of the library, since if I understood the structure of lightning-fs correctly it should be possible to just gather a list of all inodes contained in the directory, delete them and then delete the directory itself in a single transaction, right?

Sorry if that suggested solution is useless, but I hope the rest of my point still stands.

Keep up the great work! 🎉

Issue in desktop safari 13.1.1

An internal error was encountered in the Indexed Database server

Screenshot:

image

To reproduce, run the previous step once and this step twice from the quickstart page. The first time works, but the second time fails, and afterwards all other fs operations throw the same error.

You can also reproduce from scratch by starting with a fresh incognito window.

I found this bug in a search, but that seems to be iOS Safari. Anyone know of a workaround for this?

Potential for significant perf improvements in large repos

First, thanks for your work on this project 🙂

The current implementation is fairly slow with large repos, for instance vscode, which has around 5000 files or typescript, which has around 50k. It takes about a minute to clone vscode with --singleBranch and --depth 1, and doesn't manage to clone typescript in the ~15 minutes I waited.

By adding batching to the indexdb writes (Put all writes into a single transaction rather than one transaction per file) and changing the autoinc in the cachefs to increment a counter rather than search for the highest inode (the search means writing N files is N^2 time), I am able to see vscode clone in ~20 seconds and typescript clone in about 2 minutes. this is approx 3x slower than native for vscode and 6x slower than native for typescript.

Batching:

diff --git a/idb-keyval.ts b/idb-keyval.ts
index 45a0d97..94920ef 100644
--- a/idb-keyval.ts
+++ b/idb-keyval.ts
@@ -2,10 +2,12 @@ export class Store {
   private _dbp: Promise<IDBDatabase> | undefined;
   readonly _dbName: string;
   readonly _storeName: string;
+  readonly id: string
 
   constructor(dbName = 'keyval-store', readonly storeName = 'keyval') {
     this._dbName = dbName;
     this._storeName = storeName;
+    this.id = `dbName:${dbName};;storeName:${storeName}`
     this._init();
   }
 
@@ -44,6 +46,31 @@ export class Store {
   }
 }
 
+class Batcher<T> {
+  private ongoing: Promise<void> | undefined
+  private items: { item: T, onProcessed: () => void }[] = []
+
+  constructor(private executor: (items: T[]) => Promise<void>) { }
+
+  private async process() {
+    const toProcess = this.items;
+    this.items = [];
+    await this.executor(toProcess.map(({ item }) => item))
+    toProcess.map(({ onProcessed }) => onProcessed())
+    if (this.items.length) {
+      this.ongoing = this.process()
+    } else {
+      this.ongoing = undefined
+    }
+  }
+
+  async queue(item: T): Promise<void> {
+    const result = new Promise<void>((resolve) => this.items.push({ item, onProcessed: resolve }))
+    if (!this.ongoing) this.ongoing = this.process()
+    return result
+  }
+}
+
 let store: Store;
 
 function getDefaultStore() {
@@ -58,10 +85,17 @@ export function get<Type>(key: IDBValidKey, store = getDefaultStore()): Promise<
   }).then(() => req.result);
 }
 
+const setBatchers: Record<string, Batcher<{ key: IDBValidKey, value: any }>> = {}
 export function set(key: IDBValidKey, value: any, store = getDefaultStore()): Promise<void> {
-  return store._withIDBStore('readwrite', store => {
-    store.put(value, key);
-  });
+  if (!setBatchers[store.id]) {
+    setBatchers[store.id] = new Batcher((items) =>
+      store._withIDBStore('readwrite', store => {
+        for (const item of items) {
+          store.put(item.value, item.key)
+        }
+      }))
+  }
+  return setBatchers[store.id].queue({ key, value })
 }
 
 export function update(key: IDBValidKey, updater: (val: any) => any, store = getDefaultStore()): Promise<void> {

Counter:

diff --git a/src/CacheFS.js b/src/CacheFS.js
index ed26c57..0dc6950 100755
--- a/src/CacheFS.js
+++ b/src/CacheFS.js
@@ -5,6 +5,7 @@ const STAT = 0;
 
 module.exports = class CacheFS {
   constructor() {
+    this._maxInode = 0
   }
   _makeRoot(root = new Map()) {
     root.set(STAT, { mode: 0o777, type: "dir", size: 0, ino: 0, mtimeMs: Date.now() });
@@ -38,16 +39,7 @@ module.exports = class CacheFS {
     return count;
   }
   autoinc () {
-    let val = this._maxInode(this._root.get("/")) + 1;
-    return val;
-  }
-  _maxInode(map) {
-    let max = map.get(STAT).ino;
-    for (let [key, val] of map) {
-      if (key === STAT) continue;
-      max = Math.max(max, this._maxInode(val));
-    }
-    return max;
+    return ++this._maxInode;
   }
   print(root = this._root.get("/")) {
     let str = "";

Please let me know if you'd consider incorporating these changes... the batching should be safe, I'm not super sure about the autoinc, but I don't see a reason why it would cause issues (the main difference is deleting a file would free up its inode value in the original implementation but doesn't here, but that shouldn't be a problem AFAIK)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.