Comments (2)
Nice! ⭐ Sorry for the wall of text here, I'm just enthusiastic about this :)
Big picture thoughts
I had assumed people would build Earthstar apps by using the Storage API and Layers APIs provided by this Javascript implementation, and we'd achieve app portability around those APIs by making a variety of swappable classes that work in different settings (browser, node, etc).
Now there are other access patterns emerging, like this fetch API and a graphQL interface. This is a good thing:
- These are more widely understood, less specific to Earthstar. One principle of Earthstar is "use boring technologies and paradigms that are widely known"
- The network-related part of Earthstar isn't solid and standardized yet, so experimentation is good
- These allow access across languages so we don't need Earthstar implementations in every language
Standard methods across access patterns
To avoid fragmentation let's try to use the same the method names across REST, fetch, graphQL, etc.
I've just added a bunch of documentation about these methods to types.ts, especially
- Storage methods
- Query options
- Sync options. Note the "sync filters" aren't implemented yet but will be important to support.
URL details
The workspace and author address formats are designed to work well in URLs with two caveats:
- Sometimes the sigils need to be removed (
+
from workspace,@
from author) - base58 is case sensitive, but URL locations are supposed to be lowercase (at least for HTTP)
These rules make them URL-safe:
- Paths are already required to use percent-encoding for weird characters, so they work in URLs without change
- Workspaces and authors only contain alphanumeric chars,
.
,+
, and@
- Workspace and authors have a dot in the middle so they're recognized as domains by browsers
- Workspace and author parts can't start with a number (including the base58 keys)
- Paths can't start with
/@
. This avoids ambiguity betweenWORKSPACE/PATH
andWORKSPACE/AUTHOR
So except for the base58 case issue, these are all guaranteed to be valid URLs.
earthstar://WORKSPACE_NO_PLUS
earthstar://WORKSPACE_NO_PLUS?QUERY
earthstar://WORKSPACE_NO_PLUS/PATH
earthstar://WORKSPACE_NO_PLUS/AUTHOR
earthstar://AUTHOR_NO_AT@WORKSPACE_NO_PLUS/PATH
earthstar://AUTHOR_NO_AT:AUTHOR_SECRET@WORKSPACE_NO_PLUS/PATH
http://mypub.com/WORKSPACE
http://mypub.com/WORKSPACE?QUERY
http://mypub.com/WORKSPACE/PATH
http://mypub.com/WORKSPACE/AUTHOR
examples, using xxxxx
to stand in for long base58 strings
earthstar://gardening.xxxxx
earthstar://gardening.xxxxx?pathPrefix=/wiki/&limit=5
earthstar://gardening.xxxxx/wiki/Flowers
earthstar://gardening.xxxxx/@suzy.xxxxx
earthstar://suzy.xxxxx:[email protected]/wiki/Flowers
http://mypub.com/+gardening.xxxxx/wiki/Flowers
The author:password@workspace
format puts the @
in a confusing place compared with normal Earthstar sigils. It could be OK just for doing writes in the fetch API if end users never see it in the browser UI?
HTTP verbs
Let's add PUSH
and PULL
in addition to SYNC, for one-way syncs.
Pubs & swarms
The earthstar://
URLs don't include pub or swarm information. Not sure where to put it. There could be multiple pubs and multiple swarms.
There will be 2 kinds of swarms:
- the usual main swarm:
swarm key = hash(workspaceAddress)
, like DAT - secret swarms:
swarmKey = hash(workspaceAddress + swarmPassword)
Swarm passwords are just secret strings you share with friends. They let you connect only to trusted peers when you're in a very large or public workspace to protect your IP address privacy.
(Swarms aren't implemented yet)
from earthstar.
Parsing earthstar:// URLs -- case sensitivity problems
I tried shoehorning this into the javascript URL
parser. It almost worked but it lowercased the workspace, which will break the base58 workspace secret. :(
let orig = 'earthstar://+gardening.xXxXx/@suzy.xxx';
// parse it.
// URL() expects protocol to be http, so let's hack it
// also have to remove the '+'
let hacked = orig.replace('earthstar://+', 'http://');
let url = new URL(hacked)
console.log(url);
// extract info from the parsed URL
let workspace = '+' + url.host;
let author = null;
let path = null;
if (url.pathname.startsWith('/@')) {
author = url.pathname.slice(1); // remove leading slash
} else {
path = url.pathname;
}
console.log(workspace, author, path);
// --> +gardening.xxxxx @suzy.xxx null
Switch to base32?
Maybe we need to switch Earthstar keys from base58 to base32 so it can be case insensitive. It would make them 8 characters longer.
secret length:
256 bits
32 raw bytes
44 base58 characters
52 base32 characters
They would still fit in the limit of 63 characters for domain name segments (between periods).
Added issue #29 for this idea
from earthstar.
Related Issues (20)
- Make Earthstar select the best crypto driver for the detected platform HOT 1
- Ensure all doc drivers return documents of correct formats
- Cancel attachment transfers when syncing is cancelled
- Unclear how to use "earthstar identities switch" HOT 3
- Add a way to reject ingesting documents from certain users during sync
- Add a way to specify an attachment size limit during sync
- Add forgetting methods to replica HOT 1
- Persist document syncing thumbnails
- Use Web Locks API for ingestion
- Server <> Server syncing
- [FR] File size limit > 4 mb HOT 2
- Direct p2p online sync with WebRTC HOT 1
- Attachment sync not working on Deno runtime HOT 1
- Restricting the paths an applet can read and write to HOT 2
- What is this key for in `syncReplicaAndFsDir` ? HOT 1
- ReplicaDriverWeb not accessible from all environments HOT 4
- Upgrade SQLite3 dependency to 0.8.0
- Address private set intersection exploit HOT 2
- What is the rationale for the generateAuthorKeypair shortname only allowing lengths of 4 chars? HOT 1
- npm install error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from earthstar.