entropic-dev / entropic Goto Github PK
View Code? Open in Web Editor NEW🦝 :package: a package registry for anything, but mostly javascript 🦝 🦝 🦝
Home Page: https://discourse.entropic.dev/
License: Apache License 2.0
🦝 :package: a package registry for anything, but mostly javascript 🦝 🦝 🦝
Home Page: https://discourse.entropic.dev/
License: Apache License 2.0
Hello and thanks for the project. It’s not clear from the manifesto will it be possible to have package scripts? At the moment it’s very handy part of package.json
Pinning this issue (first reported by @flxwu & underlying issue identified by @itsananderson – thanks!)
If you're running into issues logging in after installing ds
, check your Node version. At the time of writing, ds
only supports Node 12+.
We're mirroring metadata, but the tarpaths are hard-coded.
There are good templates for this available.
First, congrats on the project!
I was not present in the announcement in the JS Conf EU but it looks like it was amazing.
At my company, Wasmer, we created a WebAssembly Package Manager and used GraphQL for the API. We iterated very quickly with it.
I would really recommend to use GraphQL as a first class citizen for the API, it helps both for the perspective of the server and also for any client using it.
It could also help for any other entity trying to launch a registry that is complaint with your API (since they will just need to validate the their schema against the official one).
Hopefully it's useful... keep up the great work! 👏
Is abandoning a namespace a thing?
There's a function registered as a handler for /-/whoami but it isn't implemented yet
Our first take on maintainer add & remove makes the changes directly. We should instead generate invitations that are sent to the would-be maintainer, so that they may accept or refuse.
Are blocklists a thing? I sometimes wish I could block anything from ever installing oniguruma in any project I work on.
pacote uses cacache which has a file system backing. We need to make s3 or some other kind of object store available as a pluggable storage engine.
Self-hosting for the cli is a goal within our sight.
The threat model is: somebody inherits the domain name for a server, gets a valid cert for it, and then runs an entropic with malware-injected versions of the packages from the original entropic instance. Proposal: If one entropic instance installs packages from another, they should exchange some kind of fingerprint that identifies them to each other. Sweat the details, find a good solution.
Right now we have just enough website to do an oauth flow. We need one that shows package information.
The conflation between packages and namespaces isn't so great. Right now, it's ds invite <user|namespace> --to <namespace|package>
. This is fairly clear about the order of arguments if you speak English, but what can you can invite to what isn't so obvious.
Some facts:
I suggest adding a --namespace
or --package
parameter to make it unambiguous.
ds invite potter --namespace gryffindor
: dumbledore invites potter to gryffindords accept --namespace gryffindor
: potter acceptsds invite malfoy --namespace gryffindor
: potter invites malfoyds decline --namespace gryffindor
: malfoy declinesds invite gryffindor --package quiddich-cup
: dumbledore invites gryffindor to maintain the quiddich-cup packageds accept --package quiddich-cup --as gryffindor
: potter accepts the invitation to maintain the cup on behalf of gryffindords remove malfoy --package quiddich-cup
: potter removes malfoy-the-namespace from the quiddich cupTo fill out the interactions for the invitations feature, I propose changing ds invitations
as follows:
ds invitations --packages
: list all maintainership invitations the logged-in user can acceptds invitations
: list all namespace invitations for the logged-in userIn each case, the invitations lister gives the exact commands the user can run to accept.
This is a feature we can let wait, but it would be neat to block a package-version with a security warning associated.
Here's what I'm thinking the priority list is:
It would simplify MAINTAINER's workflow if they could use GitHub, or other foreign credentials to tie to their registry account. This would simplify team management (e.g. one could setup so that anyone with push access to the repo can publish, etc.).
Document the config options, the run scripts, the requirements.
Document how to use the Docker compose thingie to develop locally.
Run scripts.
start
pre-start
pre-pre-start etc etc
test
-- to pass arguments to a script
the bash environment set up to make things run
It forwards errors to self.destStream
, which may point at different streams over time. I've tried listening on errors on each part coming through, but... no dice.
If the registry we're providing a read-through cache for is down or slow, we want to short-circuit and respond with what's in cache. Ideally we'd be able to configure the timeout.
So. Close...
Lots of options:
Publish it to a legacy-api registry so existing clients can install it.
Put the tarball somewhere existing clients can install from.
And publish it to the home entropic node so it can self-update. (Well, if it can self-update. I sense a research topic.)
Preferred option: curl to shell.
I just ripped this out when I ripped out the legacy proxy endpoints, but it would be a nice-to-have feature for the core registry.
Looks like maybe sonic-boom's "expose intrinsics" approach may need to be updated to work with Node 12?
return typeof s === "string" ? %FlattenString(s) : s
^
SyntaxError: Unexpected token %
at actualWrite (/Users/chris/projects/personal/entropic/src/node_modules/sonic-boom/index.js:242:3)
at SonicBoom.write (/Users/chris/projects/personal/entropic/src/node_modules/sonic-boom/index.js:142:5)
at Pino.write (/Users/chris/projects/personal/entropic/src/node_modules/pino/lib/proto.js:136:43)
at Pino.LOG [as error] (/Users/chris/projects/personal/entropic/src/node_modules/pino/lib/tools.js:35:26)
at Timeout._onTimeout (/Users/chris/projects/personal/entropic/src/registry/middleware/dev-only.js:36:16)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)
We don't have tests. It will help potential contributors contribute if we have tests. We should write some.
We should see if we can make it easier to build a test rig for the registry, or if the provider-function approach we have now is enough.
Key config concepts like:
This all should take place out of band of the website, preferably.
In the readme you mentioned that you want to use a content addressable storage.
There are existing content addressable systems like IPFS that you can leverage.
I’ve recently spoken with IPFS engineers and they are really interested in making IPFS easy to use for package managers so they might be open to implement features you need.
API probably looks like this:
GET /packages/package/:namespace/:name/maintainers
PUT /packages/package/:namespace/:name/maintainers/:maintainer
DEL /packages/package/:namespace/:name/maintainers/:maintainer
NB: package maintainers are a separate concept from namespace members. Namespace members implicitly get write access to all packages in the namespace.
Two api endpoints to implement here, at least:
/packages/authored-by/:namespace
: packages owned by the namespace ("namespace" is in their name0/packages/maintained-by/:namespace
: packages that the namespace has write permissions for (not necessarily in that namespace). Now that I think about it, shouldn't that be packages the user can write? Users are part of namespaces, and users have maintainership relationships with packages, so... (@chrisdickinson confirm intentions here?)Then implement cli commands to invoke these endpoints.
No, you cannot embed an ad on Vox media that creates entropic tokens for people, no matter how helpful you think it'd be.
I wish to make it clear that I am in fact giving it away, and getting it out of my gh namespace will help.
Replacing #27 in order to pin this. I want give more context on the pending software design decisions we need to make in order to build out a website (this affects #102 and #103, which are much appreciated!)
Entropic needs a website, because:
The current website doesn't have any styles, is likely inaccessible, and is hard for developers to extend. Importantly, it's also embedded into the registry service, which could present security & caching concerns down the line.
At present, Entropic runs in a single process. Because certain tasks (like syncing external packages) are too time-consuming to run in the duration of a request/response cycle, we need to split this out into multiple services. Eventually, I believe we'll need 4 services:
For the production entropic instance, @ceejbot took a clever approach with nginx proxy_pass
's so that we can simulate privileged APIs for the website to use by mounting them on the registry at non-exposed routes (for example, /private/
.) This lets us start working on the website without having completed the other two ✨new✨ apis. However, until we split out the website process from the registry process, web handlers will use HTTP clients to talk to internal handlers in the same process. We'll start this process for the registry as well.
This is quite a mouthful: this is all to say that when the website starts needing to get at registry data, please pull in @chrisdickinson and/or @ceejbot.
We'll want to consider whether we use a server-side templating language like Nunjucks to render pages in a traditional fashion, or go a single page app route using something like React + Next.js. Right now I am leaning towards server side templating using Nunjucks (or similar):
I could be wrong here, so please poke holes in this. What are we buying with a single page app?
Let us do the simplest thing that works, then iterate. I propose we use Tachyons for a base coat, then specialize with an added stylesheet. (When this gets unwieldy, that's when we should start looking into css-modules or css-in-js as an additional layer on top of tachyons.)
80% is a conservative goal, but I think it's an acceptable first goal.
Chris was muttering something about switching from mocha to ava, which is fine by me. Whatever works.
After running curl -sSL https://www.entropic.dev/install.sh | bash
, trying ds login
gives the error below:
Error: /Users/flxwu/.entropicrc
at load (/Users/flxwu/.nvm/versions/node/v8.11.4/lib/node_modules/ds/lib/config.js:21:11)
at main (/Users/flxwu/.nvm/versions/node/v8.11.4/lib/node_modules/ds/lib/main.js:19:26)
at Object.<anonymous> (/Users/flxwu/.nvm/versions/node/v8.11.4/lib/node_modules/ds/lib/main.js:49:3)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Function.Module.runMain (module.js:693:10)
at startup (bootstrap_node.js:191:16)
Things like abandonware, legacy, and git will already be in use in entropic. Other names might need to be disallowed to prevent user confusion. This might belong in an editable text file. Think it through.
We need some kind of administrative tools to provide a first-line counter to abusive content, spam content, or just plain illegal content. These tools will need to do at least the following:
It should be easy (eventually; possible is a good start) to point one entropic instance at another and make a read-only mirror of its packages. Perhaps the packages would go under a namespace that is the sitename of the site being mirrored?
We can download a package and stuff it into the cache, but we are not yet able to install it into a node modules folder in a usable format.
Implement flows for enabling and disabling 2FA, including using recovery codes.
Implement login flow for 2FA. (This looks like "oauth success -> user requires 2fa, enter otp code -> on success continue, on error retry")
Inspired by the closing keynote of JSConf EU. We have some gestures toward localizability in error codes, but we need to be rigorous about it. Also, we need to have a plan for getting there with the client.
If we have workers doing things, we suddenly need an API for the workers to talk to. This API should not be the same thing as the one we expose to the outside world. CONGRATS we've just grown a second service! Make it happen.
Invitations should be working already, more or less. We need to fill out the namespaces workflow.
First: The two-booleans approach to mark status is not so great. We need a status enum. Valid statuses for namespace_memberships:
Invitations & permissions code will need to be updated;
After that, we can fill out the namespaces feature:
Authenticated, authorized users can:
Supporting json based description of package seems pretty important, even support for incompatible version of package.json would be a plus.
And my personal preference would be also adding an option to load package description as a js object (without necessity of placing quotes) and with possibility to define scripts not only as a shell commands, but also as functions to which command line arguments and environment variables are passed.
{
name: "[email protected]/ds",
version: "0.0.0-beta",
scripts: {
test: "node test.js" // classic shell script
compile: function(args, env, shell) {
shell.run("ts " + args[0]);
}
},
dependencies: {
"@iarna/toml": "^2.2.3",
"[email protected]/figgy-pudding": "^3.5.1"
}
}
Your solution could support all these 3 formats (toml > json > js like)
It would be good to introduce the world to the project, what its goals are, what its goals aren't, and how people can help if they want to.
We want somewhere to talk about the project, with decent moderation tools. I do not think Slack is a good community support tool any more.
Issue should probably get broken up but I wanted to record the work somewhere.
We should probably discuss our approach toward command design. As an opening salvo I will say that I prefer short words for commands and no elaborate options for the targets of those commands. Argument order should preferably not be meaningful, or if meaningful unambiguous given existing context.
Example: ds add-maintainer chrisdickinson namepace/pkg
is unambiguous in this example, but hard to predict without an example. Is it ds add-maintainer target-package target-human
or does the human go first? Maybe it's great, though, because that last argument can be implicit if you're in a directory with a package.json.
ds add-maintainer user=chrisdickinson pkg=namespace/pkg
is unambiguous but kinda weird.
ds maintainers add chrisdickinson to namespace/pkg
is unambiguous if you speak English, might be hard to parse or translate.
yank
is internal jargon that means two things: 'abandon' and 'deprecate'. Do we want to just let the jargon leak out into the user experience?
The top-level readme needs to document our goals & major choices in an easily-grasped narrative. This description should not be particularly technical in focus.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.