Comments (8)
Probably the work done by Roy Hashimoto over on https://github.com/rhashimoto/wa-sqlite/ could get someone who knows what they are doing most of the way there!
While it obviously has some drawbacks, given the now wide compatibility... I think spinning up a web worker and treating the worker process like a browser-internal db server is pretty elegant. This approach seems to work well with wa-sqlite
, and (with some admittedly hacky workarounds) means you can get multiple tabs talking to a single web worker. It also means you can easily use the sync API and spinning up and waiting for the pool of handles just gets done once (for most circumstances probably).
from pglite.
Hey, yes OPFS is absolutely on the roadmap. IndexedDB was the quickest route to this early Alpha, but it's not the target.
from pglite.
Yeah indexdb is ok but not good for performance. OPFS gives that performance boost. It also has sync access so it makes so much sense for Postgresql to use it. We don't need that async layer to deal with.
from pglite.
Hey, yep exactly. It's the key target for persistence.
There are a few technicalities, although OPFS offers sync file handles, opening a file is still async. There a a few tricks we can take to get around this, using asyncify (which we currently use but is a resource overhead), or keeping a pool of open files.
lots to explore.
from pglite.
Unless mistaken, wasmfs
is maybe not the best target for opfs
sync, at least according to https://sqlite.org/wasm/doc/trunk/persistence.md#vfs-opfs-sahpool and the wa-sqlite
discussions. Given it looks like pglite is going to necessarily be single process in accessing the files anyway and there is already a worker option, maybe a better target (in addition to the existing works-everywhere idb option) would be something mirroring the sqlite
vfs-opfs-sahpool
or wa-sqlite
AccessHandlePool
?
from pglite.
Hey @AntonOfTheWoods, I expect we are going to explore both routes in time. WASMFS is probably the shorter route to a first version, but may have drawbacks with CORS header requirements.
An access handle pool would work, but would probably be a new Emscripten file system, Postgres doesn't have quite the same VFS concept as SQLite so we can't do it quite the same way.
Postgres also uses a lot more file than SQLite, so its not having a handful of access handles, it's potentially hundreds.
from pglite.
Would be great to just bring your own fs instead of using predefined ones, so we could bring our own.
abstract class FileSystem {
abstract readFile(path: string): Promise<string>;
abstract writeFile(path: string, contents: string): Promise<void>;
/// other fs methods
}
import { PGLite } from "@electric-sql/pglite";
import { MemoryFS, NodeFS, IndexedDbFS } from "@electric-sql/pglite/fs";
const pg = new PGLite(new MemoryFS());
In the case of bun
its own file system claims to be faster than node, would be great.
from pglite.
@Neo-Ciber94 absolutely, that's something we are considering. All the FS are built on top of the Emscripten file systems at the moment, and there aren't really any unofficial ones out there. We would need to find the right level of abstraction to enable custom VFSs but it's certainly something that make a lot of sense to do!
from pglite.
Related Issues (20)
- `affectedRows` not being returned in UPDATE `.query` method
- initdb failing to complete in Bun HOT 1
- Generic Extension Strategy HOT 6
- `import("module")).default` is undefined HOT 3
- Calling `execProtocol` when PGlite is not ready HOT 2
- Throw error when main db instance method used inside an interactive transaction HOT 1
- Options for WASM and shareDir bundle location
- Feature: Add foreign data wrapper HOT 1
- Attempting to change column type throws error HOT 3
- Add lock to indexeddb vfs so that the database can only be opened once. HOT 10
- Able to create custom fs
- React native support
- OOB when select() with over about 1k response? HOT 2
- PGlite bindings for other runtimes? HOT 1
- missing WebSocket dependency for node environments HOT 2
- `PGlite.exec()` returns result objects with misplaced `affectedRows`
- "memory access out of bounds" or "null function or function signature mismatch" with plpgsql; but works in Postgres 13/14.5/15.1 HOT 4
- error performing snapshot: error: unrecognized node type: 67143218
- db enters invalid state, causing many subsequent errors and requiring a reload HOT 1
- Error: expected magic word 00 61 73 6d, found 3c 21 44 4f @+0 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pglite.