eclipse/openvsx

An open-source registry for VS Code extensions

Eclipse Open VSX

Open VSX is a vendor-neutral open-source alternative to the Visual Studio Marketplace. It provides a server application that manages VS Code extensions in a database, a web application similar to the VS Code Marketplace, and a command-line tool for publishing extensions similar to vsce.

A public instance of Open VSX is running at open-vsx.org. Please report issues related to that instance at EclipseFdn/open-vsx.org.

Getting Started

See the openvsx Wiki for documentation of general concepts and usage of this project.

Development

The easiest way to get a development environment for this project is to open it in Gitpod.

Click Open Browser on port 3000 to see the running web application.

cli

• yarn build — build the library and ovsx command
• yarn watch — watch (build continuously)

The command line tool is available at cli/lib/ovsx.

webui

The default frontend is the one bundled in the Docker image, and is also used for testing in the development environment. It depends on the compiled library, so make sure to build or watch the library before you build or watch the default frontend.

• yarn build — build the library
• yarn watch — watch (build continuously)
• yarn build:default — build the default frontend (run webpack)
• yarn watch:default — run webpack in watch mode
• yarn start:default — start Express to serve the frontend on port 3000

The Express server is started automatically in Gitpod. A restart is usually not necessary.

server

• ./gradlew build — build and test the server
• ./gradlew assemble -t — build continuously (the server is restarted after every change)
• ./gradlew runServer — start the Spring server on port 8080
• ./scripts/test-report.sh — display test results on port 8081

The Spring server is started automatically in Gitpod. It includes spring-boot-devtools which detects changes in the compiled class files and restarts the server.

OAuth Setup

If you would like to test authorization through GitHub, you need to create an OAuth app with a callback URL pointing to the exposed port 8080 of your Gitpod workspace. You can get it by calling a script:

server/scripts/callback-url.sh github

Note that the callback URL needs to be updated on GitHub whenever you create a fresh Gitpod workspace.

After you created the GitHub OAuth app, the next step is to copy the Client ID and Client Secret into Gitpod environment variables named GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET and bound to this repository. If you change the variables in a running workspace, run scripts/generate-properties.sh in the server directory to update the application properties.

With these settings in place, you should be able to log in by authorizing your OAuth app.

Docker Compose Setup

If you prefer to quickly get started with a local, docker-based development environment, you can use the approach described in our docker compose setup. You can use our docker compose profiles, allowing you the option to either run a service directly in a docker container or to manually build and run it on your local machine.

Google Cloud Setup

If you would like to test file storage via Google Cloud, follow these steps:

• Create a GCP project and a bucket.
• Make the bucket public by granting the role "Storage Object Viewer" to allUsers.
• Configure CORS on the bucket with origin "*" and method "GET".
• Create environment variables named GCP_PROJECT_ID and GCS_BUCKET_ID containing your GCP project and bucket identifiers. If you change the variables in a running workspace, run scripts/generate-properties.sh in the server directory to update the application properties.
• Create a GCP service account with role "Storage Object Admin" and copy its credentials file into your workspace.
• Create an environment variable GOOGLE_APPLICATION_CREDENTIALS containing the path to the credentials file.

Azure Setup

If you would like to test file storage via Azure Blob, follow these steps:

• Create a file storage account and a container named openvsx-resources (a different name is possible if you change the ovsx.storage.azure.blob-container property).
• Allow Blob public access in the storage account and set the container's public access level to "Blob".
• Configure CORS in your storage account with origin "*", method "GET" and allowed headers "x-market-client-id, x-market-user-id, x-client-name, x-client-version, x-machine-id, x-client-commit"
• Create an environment variable AZURE_SERVICE_ENDPOINT with the "Blob service" URL of your storage account. If you change the variables in a running workspace, run scripts/generate-properties.sh in the server directory to update the application properties.
• Generate a "Shared access signature" and put its token into an environment variable AZURE_SAS_TOKEN.

If you also would like to test download count via Azure Blob, follow these steps:

• Create an additional storage account for diagnostics logging.
  • IMPORTANT: set the same location as the file storage account (e.g. North Europe).
  • Disable Blob public access.
• In the file storage account
  • Open the diagnostic settings (Monitoring -> Diagnostic settings (preview)).
    • Click blob.
    • Click Add diagnostic setting.
    • Select StorageRead, Transaction and Archive to a storage account.
    • Select the diagnostic storage account you created in the previous step as Storage account.
• Back to the diagnostic storage account
  • Navigate to Data Storage-> Containers
    • The insights-logs-storageread container should have been added (it might take a few minutes and you might need to do some test downloads or it won't get created).
    • Create a "Shared access token" for the insights-logs-storageread container.
      • Click on the insights-logs-storageread container.
        • Click on Settings -> Shared access token
          • Must have Read and List permissions.
          • Set the expiry date to a reasonable value
          • Set the "Allowed IP Addresses" to the server's IP address.
  • Go to Data Management-> Lifecycle management
    • Create a rule, so that logs don't pile up and the download count service stays performant.
    • Select Limit blobs with filters, Block blobs and Base blobs.
    • Pick number of days (e.g. 7).
    • Enter insights-logs-storageread/resourceId= blob prefix to limit the rule to the insights-logs-storageread container.
• You need to add two environment variables to your server environment
  • AZURE_LOGS_SERVICE_ENDPOINT with the "Blob service" URL of your diagnostic storage account. The URL must end with a slash!
  • AZURE_LOGS_SAS_TOKEN with the shared access token for the insights-logs-storageread container.
  • If you change the variables in a running workspace, run scripts/generate-properties.sh in the server directory to update the application properties.

License

Eclipse Public License 2.0

openvsx's Issues

allow to fetch raw license

Lazy loaded search results fade in very slowly

Scroll further down on https://open-vsx.org to see this effect

Conceptual change: publisher => namespace

In the Visual Studio Marketplace, the publisher field from the package.json is used to define the namespace under which an extension is published (see Publishing Extensions). One or more users can be granted the right to use a specific publisher name when publishing an extension. The actual user who published it is hidden in the marketplace, only the publisher is shown instead.

After a discussion with @svenefftinge, we'd like to follow a different approach in Open VSX.

• Rename publisher to namespace (we still use the publisher field from package.json for compatibility with VS Code extensions).
• Initially, there is no restriction on namespaces. Every user can publish to every namespace.
• The user interface shall display the actual user who published an extension.
• It shall be possible for users to claim a namespace so nobody else can publish to it. Later we'll also offer means to manage namespaces by granting access rights to other users (#5).
• We'll create a GitHub service account named open-vsx through which we'll publish extensions to open-vsx.org using scripts. These scripts will be maintained in an open source repository at github.com/open-vsx/publish-extensions
• Remove the --create-publisher flag from the CLI (#16) and introduce a command create-namespace instead.

This approach allows to keep the relationship of namespaces (i.e. publishers) to organizations or individual users, but in addition it gives us the flexibility of publishing a large number of extensions through scripts in a transparent way and still retain the original qualified names (publisher.extension). The open-vsx user is meant to create trust: instead of publishing third-party extensions in secrecy on someone's local machine, we'll do this openly and transparently.

allow to share url to rating and review page

When you open an extension page one has to click on ratings to see reviews. Right now there is no way to open such page with a link.

Add a CLI flag to create a publisher

The ovsx publish command shall have a flag --create-publisher that requests to create a publisher as specified in the package.json file. See #5.

Infinite scrolling of search results

The search endpoint already supports paging of search results. We should use this in the web UI to implement infinite scrolling by automatically loading more results as the user scrolls down.

Lazy loading of search results runs into infinite loop

Sometimes the extension list starts requesting more and more pages although the end is already reached.

browser navigation back does not navigate to the proper context

To reproduce:

• start searching something
• select an extension
• press go back

One will expect to end up on the previous search, but ends up on the landing page.

Provide database migration strategy

Some new features will require changes in the database schema, most importantly #10. We should provide a strategy for database migration so existing data can be retained when breaking changes are rolled out.

account menu is not completely visible

[security] Block SVG images?

See https://code.visualstudio.com/api/working-with-extensions/publishing-extension#publishing-extensions

While developing extensions list in not being rendered due to some server error

This is what I get in the console:

Count and display downloads

The number of downloads of an extension (version) shall be counted and displayed on the extension detail page.

This should be considered when implementing #10.

[Q] Publishing new version of extension removes previous version?

I recently published a new version of vscode.cpp on open-vsx.org. The previous version (v1.39.1) is used by the Theia example applications: https://github.com/eclipse-theia/theia/blob/master/package.json#L86

Since publishing v1.43.2, it seems the previous version is no longer served:

$ wget http://open-vsx.org/api/vscode/cpp/file/vscode.cpp-1.39.1.vsix
--2020-03-26 22:11:29--  http://open-vsx.org/api/vscode/cpp/file/vscode.cpp-1.39.1.vsix
Resolving open-vsx.org (open-vsx.org)... 34.98.99.180
Connecting to open-vsx.org (open-vsx.org)|34.98.99.180|:80... connected.
HTTP request sent, awaiting response... 404 
2020-03-26 22:11:29 ERROR 404: (no description).

Is there perhaps a URL we could use to request the latest version of an extension, that would continue working after the extension is updated?

Or maybe the file name of the extension should not contain the version?

WDYT?

[extension-detail] needs adjustments for small devices

This is how it looks:

vetur download f.txt file

I tried to download vetur and got a file with f.txt name

Further details to be shown on the extension detail page

See the VS Code Extension Manifest format.

• preview: whether the last published version is a preview
• markdown: control the Markdown rendering engine
• galleryColor, galleryTheme: control the gallery banner style
• qna: handle special values false and marketplace
• badges: render a list of badges
• dependencies, bundledExtensions: show links to extensions listed as dependencies

improve rendering of reviews

Right now stars are rendered weirdly:

[extension-list-items] need to be adjusted for small devices

Here on the tablet size, they look too narrow:

On Large phones, they look too stretched:

I'll soon make a Pr with the Fix.

Display work in progress

The UI should indicate when a server request is in progress:

• after submitting a review
• after requesting an access token
• while loading reviews or access tokens

This is especially important when a replacement is displayed in case there are no elements. For example, "You currently have no tokens" is shown initially even when there are tokens, but they have not been loaded yet. A load indicator should be shown initially instead of that text.

search works only for complete words

I cannot find my extension searching for ako for instance.

Implement namespace management

We should add a "Namespaces" section to the user settings page where one can create new namespaces and manage user roles.

The Visual Studio Marketplace has a similar page for managing publishers:
https://marketplace.visualstudio.com/manage

Improve display of non-quadratic extension icons

In extension list some images are not loaded or displayed properly.
Example:

Detect license type from file

Many extensions write SEE LICENSE IN <filename> instead of a license name in their package.json (see https://docs.npmjs.com/files/package.json#license). We should detect the license type from the file content.

Provide page titles

The web UI currently has not page title. It should change based on current route.

Scroll-bar glitch when clicking on avatar

I can see a vertical and horizontal scroll-bar when I click on my avatar. See the screencast:

Expected, I do not see the scroll-bars when I click on the avatar. Most likely related to #29.

Browser: Brave, Version 1.4.96 Chromium: 80.0.3987.132 (Official Build) (64-bit).

[cli] No message available

Whenever I try to publish an extension, I get the error message "No message available". I tried a little debugging myself, but cannot find that message in the source.

Example

$ ovsx publish --pat XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
 DONE  Packaged: /var/folders/v8/qp51rjts5jgcb8cr74d4smd00000gn/T/tmp-46513iOV0AoO5Na1U.vsix (15 files, 237.38KB)

❌  No message available

OS: macOS 10.14.6
Node: 12.14.1
nvm: 0.35.3
npm: 6.14.1
ovsx: 0.1.0-next.9b4e999

Support VS Code / VSCodium

In order to allow VS Code/ VSCodium to connect to openvsx, we need to support the proprietary protocol from MS Marketplace, which needs to be revers engineered.
I think an extra endpoint that understands that protocol and adapts it to openvsx would make sense.

Store files outside of the database

For the sake of simplicity, files associated with an extension (VSIX archive, readme, icon) are stored as byte arrays in the database. This should be changed to some file storage provider for better scalability.

Accessibility: The extension logos don't have alt text.

According to WCAG: Images must have alternate text https://dequeuniversity.com/rules/axe/3.4/image-alt

I will soon make a PR with the Fix.

Don't allow more than one review per extension and user

When a user has submitted a review, she should not be able to write another one (also to be validated in the server). The UI should offer to see and revoke the review instead (needs a new API endpoint).

Allow to unpublish extensions

how to unpublish extension from open-vsx.org once published ?

search returns strange results

It returns akosyakov.vscode-alt-sample for foo

Display icon placeholder if package.json does not have it

Support externalized strings in package.json

Strings in the package.json of an extension can be externalized for internationalization.

Example: displayName and descriptionof Deno

package.json

package.nls.json

`ovsx create-namespace` fails (Cannot find module 'didyoumean')

❯ npm install --global ovsx
/usr/local/bin/ovsx -> /usr/local/lib/node_modules/ovsx/lib/ovsx
+ [email protected]
added 68 packages from 81 contributors in 7.394s
❯ ovsx create-namespace redhat
internal/modules/cjs/loader.js:983
  throw err;
  ^

Error: Cannot find module 'didyoumean'
Require stack:
- /usr/local/lib/node_modules/ovsx/lib/main.js
- /usr/local/lib/node_modules/ovsx/lib/ovsx
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
    at Function.Module._load (internal/modules/cjs/loader.js:862:27)
    at Module.require (internal/modules/cjs/loader.js:1040:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/ovsx/lib/main.js:13:20)
    at Module._compile (internal/modules/cjs/loader.js:1151:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    at Module.load (internal/modules/cjs/loader.js:1000:32)
    at Function.Module._load (internal/modules/cjs/loader.js:899:14)
    at Module.require (internal/modules/cjs/loader.js:1040:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/local/lib/node_modules/ovsx/lib/main.js',
    '/usr/local/lib/node_modules/ovsx/lib/ovsx'
  ]
}
❯ npm -v
6.14.4
❯ node -v
v13.7.0

[security] Add CSRF protection

See https://docs.spring.io/spring-security/site/docs/current/reference/html5/#csrf

readmeUrl is empty although REAMDE.md is present

I've published from this repo: https://github.com/akosyakov/vscode-alt-sample

But readme is empty.

Validate field sizes before submitting

While writing a review or a token comment, the dialog should validate the size of the input fields to ensure the server will accept the submitted data. The limits are 2048 characters for review comments and 255 characters for other fields.

[extension-list] The input and the filter need some adjustments on mobile

I think if we were to make them stack and take up full width then they would be much nicer to use.

I'll make a Pr soon with the Fix.

Manage and display publisher details

The Visual Studio Marketplace offers to add details to a publisher, such as a logo, web page etc.

https://marketplace.visualstudio.com/manage

We should implement this, too, as soon as basic publisher management is done (#5).

Allow to expose more metadata like the cloud-workspace metadata through plug-in

Hello,

Within Eclipse Che, we're interested in using this registry. One limitation is that for some extensions, we need to specify the OCI image to use. It also includes some extra information like the memory or cpu to allocate.

After some discussions, it seems we could then package the metadata (currently a meta.yaml) file inside the vsix.
Then, the metadata could be extracted when we submit vsix file and then stored in the registry.
Thus, registry endpoint could serve the meta yaml file when asking for it.

We would need as well to display on the UI part this kind of information stored in the meta.yaml.

That said, we agree that this kind of information is optional as it might not interest at all people wanted only to use the registry with VS Code for example.

Compromise seems that to provide this logic through a 'plug-in'.

'Cloud Eclipse Che plug-in' would

• allow to extract/store more data when submitting vsix files
• add UI part reusing this extracted information

So extra logic is done in a plug-in that can be installed or not on top of the registry

Server side rendering

Currently the whole content is generated by React in the browser. Crawlers do not execute JavaScript, and so the content should be made available to them with other means.

Expose the version

The version of the server shall be made visible with an API endpoint. This version shall also be displayed in the UI, e.g. in the footer.

Reduce bundle.js size

Currently the bundle.js file produced by Webpack is quite large. We should reduce its size, and if it helps to improve loading times, split the file into smaller chunks.

extension api to return undefined if an extension is not found

Instead of returnin 404.

Otherwise, unfortunately, it is causing quite a lot of browser error logs which is not possible to disable.

publish ovsx to npmjs

Consider rating, download count and age in search result order

The order of search results should consider the average rating, download count and age of latest version. These data should be mixed into the score computed by Elasticsearch.

[webui] Proper error handling

Currently we handle errors in API requests with the function handleError (utils.ts), which displays an alert. This should be improved, e.g. with a Material-UI dialog.

Limit image size in Markdown

Extensions that display large images in their README.md can break the page layout:

https://open-vsx.org/extension/msjsdiag/debugger-for-chrome

The image size should be limited to prevent this (in a way that is compatible to devices with smaller display).