mdamien/chrome-extensions-archive

:pager: Archive all the chrome extensions (until Feb 4. 2019)

Chrome Extensions Archive: No updates since Feb 4. 2019

In maintenance: disk is full ! (2 To)

The goal is to provide a complete archive of the chrome web store with version history.

You can see the current status of what's archived and download the files here: dam.io/chrome-extensions-archive/

Installing the extensions

To install an extension, go to chrome://extensions/ and drop the file.

To avoid the auto-update, load it as an unpacked extension

Files are named as .zip but they are the exact same .crx stored on the store.

Running the scripts

scripts are python 3.5+ only

Install dependencies: pip3 install -r req.txt

Create some folders and initialize some files:

mkdir data
mkdir crawled
mkdir crawled/sitemap
mkdir crawled/pages
mkdir crawled/crx
mkdir crawled/tmp
mkdir ../site
mkdir ../site/chrome-extensions-archive
mkdir ../site/chrome-extensions-archive/ext
echo "{}" > data/not_in_sitemap.json

Crawling:

• crawl_sitemap.py: gets you the list of all the extensions in data/sitemap.json
• crawl_crx.py: use data/sitemap.json to download the crx

Site & stats:

• scan_pages_history_to_big_list.py: makes data/PAGES.json by scanning the pages you crawled
• crx_stats.py: makes data/crx_stats.json (what's currently stored)
• make_site.py: use data/crx_stats.json + data/PAGES.json to generate the site
• make_json_site.py: data/crx_stats.json + data/PAGES.json to generate JSON

Then I serve the files directly with nginx (see nginx.conf file for example)

Helping out

I have a few things in mind for the future:

• diff of extensions versions as a web interface
• malware/adware analysis
• running an alternative web store (better search, firefox support,...)

Don't hesitate to reach out (here on issues, [email protected] or @dam_io on twitter)

To propose changes, just do a PR.

chrome-extensions-archive's Issues

list index out of range

After successfully running crawl_sitemap.py,
running python3 crawl_crx.py results in errors that look like so:

ERROR: bad parsing for https://chrome.google.com/webstore/detail/qscreens-capture/hmngohhnlbaflbigbnnjilmmekaedbbg list index out of range
ERROR: no infos for https://chrome.google.com/webstore/detail/qscreens-capture/hmngohhnlbaflbigbnnjilmmekaedbbg
3%|██▉

Running python3 crawl_infos.py also results in IndexError: list index out of range

Any ideas?

Many Chrome Extension Missing

Your crawler appears to be missing many of the chrome extension available on the chrome web store.

For example: none of my chrome extension are showing on your website.
Copy URL + Title
Gist It

License

Can you include a LICENSE file?

Searching through the source code

Could you index extensions by manifest keys (e.g. permissions / optional_permissions, content_scripts), and add the ability to search through the source code of extensions that match the query?

For example, I'd like to know how many extensions use the webRequest API in a blocking fashion (=webRequest + webRequestBlocking permissions) and block "ping" requests (to make informed decisions on https://crbug.com/611453).

Show if extension was deleted from the store or removed from the sitemap

Error connecting storage site crx.dam.io

https://crx.dam.io/ Connection refused
Not SSL certificate problem, any connection is refused

letsencrypt cert expired

FYI

Add dates (or at least crawl-date) for each version

Would be helpful to know when each release was created (or at least discovered by your crawler).

Extension removed from Chrome web store

I have noticed that extension Native Lazy Tabs was removed from Chrome web store. Also I am getting the following message in chrome://extensions/
"This extension violates the Chrome Web Store policy"
Also see this comment and below:
https://www.ghacks.net/2017/06/02/speed-up-chrome-start-by-lazy-loading-tabs/#comment-4310212

What has happened? Why did they block your extension?

(view source) manifest.json has no (WebStore) key

The main .zip (.crx) downloads have the webstore key.
The view source downloads manifest.json does not. I've been looking at
gmlllbghnfkpflemihljekbapjopfjik (Bookmark Manager (by Google))
meoeeoaohbmgbocpdpnjklmfmjjagkkf (Save To Google)
Is this intentional?

If I load the source as an unpacked extension, it generates a new key and I end up with two versions: one from the webstore and my unpacked version. After updating the downloaded source manifest with the original key, the unpacked version replaces the webstore instance, which is what I had expected.

This is a great archive. I'm using the unpacked versions to recover Google extensions that have been discontinued, like the two above, discontinued 8/15/18.

Add total size of extensions stored

Note .zip is .crx on crx.dam.io extension page

Adding a few words on the
https://crx.dam.io/ext/.html page that the main .zip download link, is really the .crx file, would keep the un-annointed from stumbling.

Crawl not working: Adblock is at 3.2.1 in the archive, 3.3.2 in the store

but everything was correctly stored, looks like the problem is in the website generation

Chat no longer working

Twitch updated legacy chat, and it's not longer working with Twitch 5

Where can I download all 426.4G Dataset?

Curious about hosting costs

Hey this is a pretty neat service.

I was just wondering roughly how much it costs per month to host this (since I see you don't have any ads).

I'm guessing since you only store top 20k, extension sizes are small, and updates can be stored as deltas the physical space used won't be too much, maybe on the order of 100 Gigs or so.

Also, since the built website is static, the main cost will be from scraping and processing extension pages.

---

Also, how were you able to discover the sitemap url for the chrome webstore? I thought about doing a similar service a few years back but gave up when I thought there was no way to programmatically get a list of extensions without using a full headless browser instance.

(Incidentally, doesn't google rate limit requests? How were you able to work around that?)

popup width problem

I don't know what happens but the pop-up windows change and now is too small to be useful, I try manually but only height change.
There is a way to fix it? maybe a css file?
thanks for helping

API for Chrome Web Store

Do you have plans to create an api for Chrome Web Store like the itunes store?

Incorrect user_count

I noticed the very top entry in the list shows an incorrect user_count.
I have checked back after the list has updated to see if the user_count would update as well, but it has not.

AVG Web TuneUp CRX.DAM.IO
CRX.DAM.IO Shows:

• "user_count": "10000000"

AVG Web TuneUp CHROME.GOOGLE.COM
Official Chrome Store Shows:

• 7,669,478 users

Contributing

You should add a contribute section to the readme so people know how to contribute to this project.

I recommend you also add a requirements.txt file which would contain all the dependencies for this project and a gitter page where people can talk about this project.

All extension .zip files are corrupt

redacted

JSON of aggregated data?

Hi. Great project.

How about a json file with the aggregated data that you display white on dark grey on the extension page https://crx.dam.io/ext/chfdnecihphmhljaaejmgoiahnihplgn.html.

It looks like json but got some errors.

Greetings

search bar

there needs to be a search bar please

For all extensions on the web?

Is this only for archiving "your" extension or does it work for all extensions across the web?

How do you manage storage then?

Rename .zip to .crx

List all extension by authors

to make it easy to report the extensions of one author

display trackers and permissions

like exodus: https://reports.exodus-privacy.eu.org/reports/15369/

Monitor data ingestion

import can sometimes stall for random reasons (ssl handshake never timeout, stuck in some infinite loop, everything is crashing and the world is burning).

I should monitor that a full cycle is done everyday (download extensions + update website).

PS: timeout the stuff too, https://stackoverflow.com/questions/2281850/timeout-function-if-it-takes-too-long-to-finish

Demo site SSL is expired

Hi
Can you please fix your demo site SSL, looks like its expired - Thanks

Add removal request

Need to find a way to transparently filter out extension who requested removal and publish the removal request log.

• tranparency log: which extension (reason too)
• add a "request removal" button.

Expose metadata history

Download ChromeOS extensions

today os=linux, be able to change the os if the response is empty

Install Google Chrome Extensions from the command line

Add the ability to install, uninstall, search for Google Chrome Extensions from the command line like how you can install application on windows using chocolatey or homebrew on mac using the terminal.

Scrape extension author infos too