zotero/zotero-connectors

Chrome, Firefox, Edge, and Safari extensions for Zotero

Zotero Connectors

Building

1. git clone --recursive https://github.com/zotero/zotero-connectors.git
2. cd zotero-connectors
3. npm install
4. ./build.sh -d

The connectors are built in build/.

Running from the build directory

Chrome

1. Go to chrome://extensions/
2. Enable "Developer Mode".
3. Click "Load unpacked extension…" and select the build/browserExt directory.

Firefox

1. Go to about:debugging
2. Click "Load Temporary Add-on" and select the build/browserExt/manifest.json file.

OR

1. Get and install the Mozilla web-ext tool
2. cd to project root
3. ./scripts/firefox/run_xpi

Safari

See https://github.com/zotero/safari-app-extension

Automatic rebuilding/reloading

Chrome on MacOS

1. brew install chrome-cli
2. npm install -g gulp
3. cd to project root
4. npm install
5. build.sh -d
6. gulp watch-chrome

As files are changed, the connectors will be rebuilt automatically and Chrome will reload the extension.

Firefox

1. Get and install the Mozilla web-ext tool
2. cd to project root
3. npm install
4. build.sh -d
5. gulp watch
6. ./scripts/firefox/run_xpi (in a different terminal window)

As files are changed, the connectors will be rebuilt automatically and Firefox will reload the extension.

Others

1. cd to project root
2. npm install
3. build.sh -d
4. gulp watch

As files are changed, the connectors will be rebuilt automatically. You will need to manually reload the extension in the browser being developed for.

Requirements for packaging extensions from the command line

• Copy config.sh-sample to config.sh and modify as necessary
• Safari/Chrome extension certificates
• Google Chrome or Chromium
• xar with patch for building Safari extensions

Tests

To run tests locally, build the extension with the -d flag, and then run

$ ./test/run_tests -p c

Test files are located at src/common/test. See src/common/test/testSetup.js for test framework documentation

Developing

Zotero Connectors are built with standard tools, such as browser extension APIs, but the architecture is quite complex. This section is a short overview of some of the complexities, to make it more accessible for newcomers.

Technologies

Chrome/Firefox Browser Extension Framework

The functionality exposed on Chrome and Firefox is provided by the Chrome extension framework, which has also been adopted by Firefox. See Chrome Extension docs and Firefox Extension docs for more information.

Safari Extension Framework

For Safari specifics see https://github.com/zotero/safari-app-extension

Zotero Translator Framework

The Connectors share code with Zotero desktop application, to support translation. A basic understanding of how translation works or at least the handlers it exposes in Zotero will be highly useful in understanding the codebase.

Components

Saving resources to Zotero library is facilitated by two major components: the Zotero Connector running in the browser and either the Zotero client or zotero.org web api. The Zotero Connector itself is split into two components: code running on the webpage and a background process.

a) Injected scripts for individual webpages

Each webpage is injected (Chrome/Firefox/Safari) with a full Zotero translation framework. A Zotero.Translate.Web instance orchestrates running individual translators for detection and translation.

The translation framework shares some code with the Zotero codebase and provides custom classes concerning translator retrieval and item saving. These custom classes talk to the background process (b) of the Zotero Connector for functionality outside the translation framework, such as retrieving translator code and sending translated items either to Zotero (c) or zotero.org (d).

b) Background process

The Connector runs a background process (Chrome/Firefox/Safari) which works as a middle-layer between the translation framework running in inject scripts (a) and Zotero (c) or zotero.org (d).

The background process maintains a cache of translators and performs the initial translator detection using URL matching. Translators whose target regexp matches the URL of a given webpage are then further tested by running detectWeb() in injected scripts. A list of translators and their code is fetched either from Zotero (c) or zotero.org (d).

The background process is also responsible for updating the extension UI, kicking off translations, storing and retrieving connector preferences and sending translated items to Zotero or zotero.org. Browser specific scripts are available for BrowserExt and Safari.

c) Connector server in Zotero

When Zotero is open it runs a connector HTTP server on port 23119. The HTTP server API accommodates interactions between the Connectors and Zotero client. Calls to Zotero.Connector.callMethod(endpoint) in this codebase are translated to HTTP requests to the connector server.

Note that Zotero cannot interact with the connectors on its own accord. All communication is Connector initiated.

d) zotero.org API

When Zotero is not available item saving falls back to using zotero.org API. The interactions with zotero.org API are defined in api.js

Message passing

The only way for the background extension process and injected scripts to communicate is using the message passing protocol provided by the browsers (Chrome/Firefox/Safari). Injected scripts often need to communicate to background scripts. To simplify these interactions, calls to functions in background scripts are monkey-patched in injected scripts. These calls are asynchronous and if a return value is required, it is provided either to a callback function as the last argument of the call or as a resolving value of a promise returned.

messages.js contains the list of the monkey-patched methods. If the method value is false no response is expected, otherwise the calls provide a response. An optional pre-send processing on the background end and post-receive processing on the injected end is possible to treat values that cannot be sent as-is via the messaging protocol.

The background process registers message listeners in messaging.js. Zotero.Messaging class also provides a way to send messages to injected scripts and add custom message listeners.

The injected scripts monkey-patch methods in messaging_injected.js(BrowserExt/Safari) Zotero.Messaging class also provides a way to send messages to the background process and add message listeners.

Build process

The build process combines files from the Zotero codebase, common files to all connectors and specific files for Chrome/Firefox/Safari connectors. At the moment the build process is awkward and uses a SH script and gulp procedures. This will be reconciled in the future to only use gulp.

1. build.sh copies images and extension files
2. gulp process-custom-files initiated by build.sh performs post-processing on copied files

Contact

If you have any questions about developing Zotero Connectors you can join the discussion in the zotero-dev mailing list.

zotero-connectors's Issues

Safari connector won't authorize on 10.11

https://forums.zotero.org/discussion/50662/

Add automatic proxy redirection

Firefox webextension 5.0 standalone migration communication

Includes:

• Open a page on install explaining that standalone is now required
• Show a prompt when trying to save without standalone open for the first time, explaining saving to server. Have users check a checkbox to make sure they understand what they're doing.

Don't let documents created with processDocuments die before translation is complete

In https://forums.zotero.org/discussion/30588/, things are presumably breaking because we are reusing the same iframe for multiple document loads, so the old document dies when a new document gets loaded. We should either fix this by creating a new iframe for each load (as I think we do in Firefox) or by switching to HTML XHR (zotero/zotero#348) in which case we wouldn't have to worry about document lifetimes.

Make the chrome extension compatible with Webextension (Firefox)

Webextension are the coming new extension for Firefox. Their are very similar to the Chrome extension and should be compatible.

I was able to load the Chrome Extension in Firefox by changing the manifest.json. Simply adding the key

"applications": {
  "gecko": {
    "id": "[email protected]"
  }
},

I got the icon but nothings seem to append when I click on it.

This should help: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Porting_from_Google_Chrome

Access Date isn't saved when saving webpage to server

Thu Jan 05 2017 12:34:39 GMT-0500 (EST) is not a valid JS date

[…]

HTTP POST {"items":[{"itemKey":"83DQI7EG","itemVersion":0,"itemType":"webpage","title":"Zotero | Home","url":"https://www.zotero.org/","accessDate":""}]}

(Also, the access date needs to be UTC.)

Open new tabs after current tab

If possible, it'd be better to open a new tab immediately to the right of the current one rather than adding a tab at the end. Since Chrome doesn't seem to have Firefox's tab-jumpback feature, it can be hard to get back to the tab you were in.

RIS/etc. interception

Not sure how possible this is (using webRequest?), but it'd be good to have.

Relevant Firefox code is in mimeTypeHandler.js.

If there's no performance penalty, we could also consider checking all Content-Disposition headers for .ris filenames, which would help with sites that don't send the right Content-Type header.

No PDF-Download from Researchgate

At the moment no PDF is downloaded from Researchgate. I don't know, if this ever worked but it would be useful. :-)

Restore Z before first click in Firefox

My main concern in #72 was flashing during normal usage. To avoid scaring Z4Fx upgraders, I think we should still show the Z before the first click or client availability in Firefox.

Check for same origin fails from within a subframe at a different host

Re https://forums.zotero.org/discussion/35540/ebscohost-blocked-a-frame-error/#Item_15

Breaks at https://github.com/zotero/zotero-connectors/blob/master/src/common/inject/http.js#L42 where it's trying to access location.protocol of a parent frame, which is at a different host (serialssolutions.com)

I'm not sure why we're looking at the parent frame though.

Use background page for fetching PDFs/snapshots in Chrome

We presently use XMLHttpRequest for this because the Zotero.HTTP.doGet API is not sufficient, but XMLHttpRequest from content scripts is broken for http targets on https pages with Chrome 38+.

Improve connector server missing endpoint handling

Should have a globally handled way to notify the user about upgrading their client, if version older than connector and does not support endpoints.

Currently handled per individual endpoints, but we can switch to something better once Z4FX is phased out.

Feed autodiscovery

If a page has a feed autodiscovery link, an option should appear in the save button context menu to add it to Zotero.

Threads:

https://forums.zotero.org/discussion/67629/zotero-chrome-connector-for-feeds-suggestion

Display error when bookmarklet broken by content security policy

If we can't load our scripts because of content security policy, we should at least show an error message.

Progress window is not displayed on pages using framesets

https://github.com/zotero/zotero-connectors/blob/master/src/common/inject/progressWindow.js#L213 attempts to attach the progress window to document.body.

On pages that use frameset in place of body (e.g. http://ptx.sagepub.com/content/41/1/160.full.pdf+html?rss=1) the progress window DIV is appended to the frameset element, but per HTML5 specification it is not rendered (http://www.w3.org/html/wg/drafts/html/master/syntax.html#parsing-main-inframeset).

Restore bookmarklet to Zotero saving

https://forums.zotero.org/discussion/comment/215935/#Comment_215935

It's possible we could get this to work by running the connector server over HTTPS on a different port with a Let's Encrypt cert and having the bookmarklet try to connect to https://local.zotero.net, which would serve an A record for 127.0.0.1.

This is only relevant for people who 1) can run the client and 2) don't want to run a connector-enabled browser, so this is far from a priority, but it would probably be easy to do in Electron.

PDF stored in arXiv not downloaded from NASA/ADS

Often times a NASA/ADS link for an article will not have a PDF available for download but it will display a link to the article's arXiv page where the PDF is available (e.g.: http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:0906.2173)

If one uses the Zotero connector for Chrome (haven't tried with any other) from the NASA/ADS site, the PDF will not be saved. If the article is saved from its arXiv page, it will.

Couldn't the connector automatically check if the PDF is available in arXiv and store it even from the NASA/ADS site?

Limit progress window height and allow scrolling

Relevant when saving multiple items

This has been slowing down Gmail for the past several days

Hi,

I would appreciate if someone can duplicate this, as it's just a theory at this point -- I think Gmail made a change to their js which is causing it to behave much more slowly when the Zotero Chrome extension is enabled. Not sure how easy it'll be to troubleshoot on this end...

URL argument from processDocuments() is incorrect for cross-domain requests

Chrome extension and LexisNexis

Latest Chrome, OS X. On LexisNexis, the Zotero Chrome extension shows the "folder" icon (to indicate "multiple"?) in both list and single article views. If I click either, nothing happens—no response. If I use the "(web page)" translator, Zotero will ingest the site (but it is not the preferred translator).

The LexisNexis translator functions properly in Firefox. E.g., this article displays a "newspaper" icon in the extension.

[Chrome] DOI translator broken on HTTPS pages

CrossRef API can only be accessed over HTTP (to the best of my knowledge anyway), so we can't use protocol-relative URLs to load metadata from CrossRef.

Re https://forums.zotero.org/discussion/42279/american-economic-association-pages-stopped-working-in-chrome/

Bring OAuth prompt to front if already open

At the moment we show a translation error and spawn another prompt, which is certainly not ideal.

Save-to-server warning not showing on non-translator pages

Right now, as far as I can tell, once Zotero has been open for a page load, pref-firstUseNoClient is set to false, and so even if you close the client the prompt isn't shown on the first save attempt. Same if you click on the Z and get the connector start page. Neither of those should affect the prompt — we just want to warn the first time they're saving to the server. (Among other things, it applies in cases of connection errors too, so even if you did install the client when prompted things might not be working right.)

I'm actually not sure in what case you would ever get that prompt currently.

Save webpage items to zotero.org

Can't save snapshots, but now that we show the icon on all pages, we should at least save the basic metadata instead of showing an error when Standalone isn't open.

MS Edge extension

http://www.theverge.com/2016/3/17/11255412/windows-10-microsoft-edge-browser-extensions-beta

Display more helpful error when 127.0.0.1 returns 50x

Since Standalone never returns 50x errors (as far as I know), we should detect them and suggest checking proxy, firewall, or VPN settings to exclude 127.0.0.1.

Delete API key on uninstall

If the connector has an API key, it would be nice to delete it before the extension is uninstalled. At the moment I think we hide these keys on the website, but at some point we'll hopefully show a device list with platforms, etc., for the special client/connector keys, and it would be nice for there not to be lots of orphaned keys.

I think the only we could do this, at least in Chrome, is by setting an uninstall URL with the key as a query parameter, and have that be a special API URL that deletes the given key (which we'll have to implement).

Update translators immediately (no browser restart)

It seems that the translators are not updated when you click on "Update Translators" in the extension preference. After a browser restart, they're updated. Tested with Chromium 34

Bookmarklet fails when used with Standalone

From Chrome
The page at 'https://www.zotero.org/bookmarklet/iframe.html' was loaded over HTTPS, but ran insecure content from 'http://127.0.0.1:23119/connector/getTranslators': this content should also be loaded over HTTPS.

We can't communicate with Standalone over HTTPS at this point, but perhaps we could add or generate "on-the-fly" a self-signed certificate.

Change callback based calls to background script to only use promises

Currently injected/background page messaging has a massive complexity layer, where we support both callback as last argument to function and promise resolution API. This is necessary due to a lot (but not all) of Zotero 5.0 shared APIs moving towards promises + connector specific functions which are set up with callbacks.

As the coupling between client code and connector code is becoming smaller with Z4FX discontinuation, converting the remaining functions to use promises should be fairly straightforward and will remove a lot of headache when changes to the messaging API are needed in the future.

Improve connector OAuth page

We currently open Zotero.org with a screen like this:

CC @dstillman @fcheslack @johanneskrtek

Remove 🚫 from icon on disabled pages

Icon should just be grayed out

Safari 9 (El Capitan) connector distribution

In Safari 9 on El Capitan (and, apparently, a 10.10 dev beta, since I have it too), only Safari extensions distributed from the Safari Extensions Gallery will auto-update. We have to decide whether we want to submit the extension to the gallery or whether we want to handle updates through Standalone. If we do the latter, it'd need to be something like this:

1. We get the latest connector version either from the a hard-coded value in each release or from the daily repo update check (the same as we now do for the PDF tools). Latter might not be necessary, and it would only work if there aren't connector-compatibility-breaking changes between Standalone versions (or we did server-side user agent checking in the repo).

2. If the latest available version is later than a stored pref, or the pref value doesn't exist, we prompt the user to upgrade.

3. If they choose to upgrade, we send them to a download page, which would have to be Safari-specific so that it could show a warning if they weren't in Safari. Otherwise if someone had Chrome as the default browser they'd either 1) never get the upgrade if we updated the pref when the clicked to upgrade or 2) keep getting the prompt if we updated the pref only on the first request received from the connector. (Or maybe we can actually load a URL in Safari explicitly.)

While our usual download page would work, it'd be better to have something like 1Password's.

If we do want to update from the Safari Extensions Gallery, we should update as soon as possible, since anyone running a 10.10 or 10.11 dev or public beta will never get an update. The process for getting people on the gallery version is also unclear to me. When I try to update the 1Password extension manually, or if I uninstall and then install their previous 4.4.0 version, Safari prompts me to install either the developer version or the auto-updating gallery version, which seems awfully confusing:

So I'm inclined to just update through Standalone, particularly since version compatibility is probably an issue anyway.

This is also related to a mechanism for installing the connector at first run for all users. If we haven't either prompted for or seen either connector before, we should probably display an install prompt at startup. And in any case, we should include an "Install Browser Extensions…" menu option that opens the download page (as in 1Password).

[Safari] Collection icon broken in progress window

Connector looks for collection icon in safari_extension_base_url/images/treesource-collection.png, but the icon is placed in /images//itemTypes/treesource-collection.png by the build script.

For Chrome, all images reside directly in /images/. Is there a reason why Safari is different? Unfortunately, the code that sets the icon is shared, so this is a bit more difficult to address without rearranging the /images/ directory.

Unit testing framework

Save icon shouldn't flash to offline on every page load

This is a regression from the current version, which just changes to the next applicable icon. While that means that, for a slow page load, the icon can be briefly incorrect (and ideally we would wait to start the save until it was done loading and the right translator was ready), going to offline on every load is very distracting.

zotero: link being launched on first new tab in Firefox

No idea what's going on here. Firefox 50.1 Unbranded, no other extensions.

Steps to reproduce:

1. Remove extension.

2. Load Temporary Add-on.

3. Cmd-T

Result:

Respect automatic tagging option when saving via server

https://forums.zotero.org/discussion/comment/266420#Comment_266420

Add snapshot option to context menu in Chrome when viewing PDF

https://forums.zotero.org/discussion/35327/zotero-in-google-chrome/

It looks like Chrome's own menu is still displayed when viewing a PDF, and I still have a menu option there from another extension. We should see if we can add the save option there.

Fix message passing with chrome.runtime.sendMessage/onMessage

cd2a2c1 broke some stuff, though I'm not sure why. @simonster, any ideas here?

Localize connectors

Make sure hi-res images are used when appropriate.

Shared code with z4fx connector makes this slightly convoluted, but once the client is properly divorced we can do this easy.

Add keyboard shortcuts

Error opening new tab in Firefox

Extension error: Error: Type error for parameter createProperties (Property "openerTabId" is unsupported by Firefox) for tabs.create

I'm guessing this isn't necessary in Firefox anyway due to the tab jumpback behavior.

[Chromium] extension fails if cookies are disabled

This is rather weird - it seems that this extension fails to work if cookies are disabled.

I suspect pull requests #14 and #12 are related to this, but I am not familiar enough with the connector to actually test this.

Connectors do not save attachments on some sites

Issue:
When using Chrome/Safari connectors and saving certain items (typically while using a proxy), attachments are sometimes not downloaded.

This is a long-standing issue and a big reason preventing people from using Chrome or Safari connectors.

Some more details from what I was able to figure out (and Simon and Dan certainly already know):

When saving via connect, the connector sends an HTTP request to Standalone Zotero, which is acting as a server. In this request, Zotero sends all the info that it scrapped for the item, including urls for any attachments. Additionally, in this request Zotero includes the source URL and all the cookies for that domain (including cookies for higher level domains).

When Zotero Standalone receives the request to save the item, it attempts to fetch the attachments. It attaches the cookie that was passed down from the connector to any requests that have the same domain as the URL supplied by the connector. If the domain does not matched, no cookie is attached.

Typically, if an attachment is at a different domain and a cookie is not attached, this is not a big deal, but when content is restricted and accessed via proxy (or by logging in on a website), this prevents the attachment from being downloaded.

Example:
http://search.ebscohost.com/login.aspx?direct=true&db=aph&AN=84204840&site=ehost-live (Modernist Short Stories by Women by UTELL, JANINE) or any other EBSCOhost article with PDF
The actual entry for the article (after logging in or accessing through proxy) is located on the web.ebscohost.com domain, but the PDF is stored on the content.ebscohost.com domain. The PDF fails to download, because Zotero does not attach a cookie.

Possible solutions:

1. Zotero Standalone could relax the security and send cookies to different subdomains. While this would solve the problem at hand, I don't think it's a solution, since security is always a major concern. I don't think we want to be leaking any sensitive information.

2. Instead of Zotero Standalone trying to fetch attachments directly, Zotero Standalone could use the Connector (and possibly the Bookmarklet) as a proxy to retrieve content. The Connector would be able to handle all of the cookies natively.

One thing that I was concerned with was how do we establish such communication without having the connector act as a server (listen on a set port)? I'm not sure if some of these are real concerns, but: I don't think you want the connector acting as a general proxy for a malicious program trying to communicate with the internet and I don't think listening on a particular port would be possible via bookmarklet. I was thinking that we could solve this by having Zotero Standalone (or zotero translation server) determine which attachments (snapshots and/or PDFs) need to be fetched and then return a status code indicating that the connector should additionally supply attachments (with the list of attachments in the response). The connector would then send these files in a subsequent HTTP request. Zotero Standalone (or perhaps the connector) would provide an identifier to map requested attachments to items being saved. This would allow Standalone to only request files that are listed as item attachments, so I think from a security standpoint, this would not be very bad.

3. Connectors could pass cookies to Zotero Standalone in a more granular fashion (i.e. split up cookies for each level of the domain). I see that this would be possible in Chrome via chrome.cookie API and in Firefox via nsICookieManager2, but I didn't find anything of the sort for Safari and I'm fairly certain this would be impossible for the bookmarklet.

I'm sure there are other solutions that we can think of, so I wanted to start this issue to figure out how we can finally bridge this gap between Connectors and Firefox Plug-in. So far I think solution 2 is a very feasible option with limited drawbacks if executed correctly.

Better dialog styling

We're now displaying dialogs in a few places, such as #66. It would be good to improve the styling on those.

(We could also improve styling on the saving popup, which these are modeled after, but those are less egregious, so this is the priority.)

CC @johanneskrtek

Make context menu entry a little shorter?

"Save Zotero snapshot from Current Page" is twice as long as any other entry in my Chrome right-click context menu and it throws off the UI a bit; would "Save page to Zotero" suffice?