The mementoembed from oduwsdl

Include images from loaded CSS files

Sometimes all images on a page are loaded via CSS. MementoEmbed should interrogate the CSS files for URIs found in the background-image property.

Test with URI-Ms from Icelandic Web Archive

URI-Ms to get started:

First attempt to create a card fails

Run the server and load the page http://localhost:5550/ then enter a URI-M and click "Create a Social Card" button. For the first time the card creation request is not processed, instead the page is redirected to http://localhost:5550/?# (notice the added ?# part in the URL), the form is cleared, and a message is logged in the console, saying we have failure with data: [object Object]. Any further attempts will be processed as expected (unless the ?# is removed from the URL again.)

Test with URI-Ms from PRONI web archive

Test URI-M to get started:

http://webarchive.proni.gov.uk/20130601103808/http://www.armaghplanet.com/

Test with URI-Ms from the Internet Archive

This will likely be re-opened as new, interesting URIs are tried, but this requirement is for a base number of URIs that are tested with the Internet Archive.

To start:

Use CustomElements for cards

If these cards are intended to be used in other sites, it might be a good idea to utilize HTML CustomElements for bi-directional style isolation. We are using it in Reconstructive Banner. Here are some other resources that will help you get started with this fairly new HTML standard.

Missing thumbnail image in cards

Every time I try to request a card for a URI-M like https://web.archive.org/web/20180604110141/http://www.example.com/, the card is missing the thumbnail image and a 404 is reported in the developer console of the browser which points to a resource at http://localhost:5550/undefined. This might be a duplicate of #30.

Use unobtrusive JavaScript for event listeners

It's generally not considered a good practice to place JS as HTML attribute value in an obtrusive way so that when all the JS is extracted out into an external file, it does not leak into the markup. The click handler (such as onclick="requestEmbed();") from the following code can be removed and can be bound externally in an unobtrusive way.

<button type="button" class="btn btn-primary" onclick="requestEmbed();">Create a Social Card</button>
<button type="reset" class="btn btn-secondary" onclick="clearEmbed();">Clear URL</button>

Test with URI-Ms from Archive-It

URI-Ms to get started:

That last one is important for the "all" collection that contains all publicly-available Archive-It URI-Ms.

Add line ending to all files

Many files in this repo currently do not have a trailing newline character which is not an standard practice.

Test with URI-Ms from Bibliotheca Alexandrina Web Archive

This requirement will be delayed until we have URI-Ms to test.

"Please note that the Bibliotheca Alexandrina is currently migrating the web archive collection to a new storage system. Therefore, availability of archived webpages is not guaranteed for the time being. We appreciate your patience, and we look forward to the collection being fully available once again soon."

Test with URI-Ms from the National Library of Ireland

Some test URI-Ms to get started:

System should handle HTML META redirects

URI for test: https://web.archive.org/web/20180308084654/http://www.cs.odu.edu/

System should handle mementos containing framesets

The following URI contains framesets that create an issue when searching for a striking image: https://www.webarchive.org.uk/wayback/archive/20050726120000/http://www.isb.org.uk/index.html

Convert all instances of logger.warn to logger.warning

The system gives these messages during testing:

DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

At some point the warn function will no longer be available.

Actually verify that images exist before using them as a striking image

Sometimes the image returned generates a 404. This could be a temporary archive issue, but we should assume that it just wasn't crawled and isn't available.

Dockerfile improvements

FROM python:3.6.4-stretch

The base image is so specific that it would not update to even any non-breaking security patches (i.e., third point release in the semver scheme) automatically. For example, python:3.6.4-stretch is already stale and python:3.6.5-stretch is published. We should perhaps use python:3.6-stretch instead to accommodate any security patch releases. Better yet, if Python 3.6 is not required specifically and any later version would work then we can use python:3-stretch instead.

# TODO: publish archiveit_utilities so that we don't need to do this
RUN git clone https://github.com/shawnmjones/archiveit_utilities.git

RUN cd archiveit_utilities && pip install .

Each docker instruction adds one layer to the image. It is better to put relevant tasks and any associated cleanup should in a single layer for a cleaner image. Hence, above instructions can be consolidated in a single one.

FutureWarning: The behavior of this method will change in future versions.

From testing, this message shows up:

mementosurrogate.py:785: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.

It is related to the readability library.

Test with URI-Ms from the Portuguese Web Archive

URI-Ms to get started:

FutureWarning for the way a None object is checked

After booting the server when the first card creation request arrives, the server logs a warning.

/app/mementoembed/mementosurrogate.py:785: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
  if maxpara:

Test with URI-Ms from the National Records of Scotland

Some URI-Ms to get started:

Ensure that data URIs are handled for image selection

This URI-M contains image URIs with the data scheme:
http://perma-archives.org/warc/20180501135244/https://www.flexispy.com/

They should be processed.

Test with URI-Ms from UK Parliament Web Archive

Test URI-Ms to get started:

http://webarchive.parliament.uk/20100426094738/http://www.publications.parliament.uk/pa/cm199899/cmselect/cmagric/141/9020402.htm

Implement continuous testing with Travis CI

Ensure that a badge is placed on the README so that users understand the degree of stability of the current version of the code.

ResourceWarning: unclosed ssl.SSLSocket

Is there anything we can do about this?

ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.86.26', 50357), raddr=('207.241.225.186', 443)>

This only shows up during unit tests and appears to be related to:
psf/requests#3912

Support for CarbonDate service

MementoEmbed should at least link to the CarbonDate service for a given URI. Perhaps the CarbonDate endpoint could be made configurable?

Option to generate text snippet + thumbnail

Once the text snippets and thumbnails are implemented, this combination should not require too much additional effort.

Improve the error message for non-mementos

Right now, the following error message appears on a red background:

The URL you supplied (https://www.flexispy.com/) is not a memento or comes from an archive that is not Memento-Compliant.

For a live web resource, you can create a memento that resides on the web in the following ways:

* Using the Internet Archive's Save Page Now button.
* Saving the web page at Archive.is
* Using the ArchiveNow service.
* Using a browser plugin, like Mink.

Happy Memento Making!

The red background is not friendly and there should be links to the recommended services.

Test with URI-Ms from archive.is

To start, try this URI-M:

http://archive.is/20130508132946/http://flexispy.com/

Test with URI-Ms from the National Archives and Records Administration

Some URI-Ms to start:

Fix issue with #? in URI

It appears that MementoEmbed does not execute properly at first load. If a user submits a URI-M at /, the system reloads the page to /#? rather than submitting the request. Once this reload has occurred, subsequent URI-M submissions are successful.

Test with URI-Ms from the UK National Archives

Some URI-Ms to get started:

http://webarchive.nationalarchives.gov.uk/20081208222543/http://www.nacell.org.uk/

Support for MementoDamage service

MementoEmbed should at least link to the MementoDamage service for a given URI. Perhaps the MementoDamage endpoint could be made configurable?

Time zone missing from card date

On the bottom of the card, there is a datetime and source but the time is ambiguous, as it lacks a timezone. "GMT", "Z", or what ever is applicable ought to be appended.

MementoEmbed 6563c01 using Docker.

Ensure that the client code responds to a down server endpoint

I've noticed that the system does not give the user a good message when the server is no longer running. It should say something constructive to the user.

Force cache hits during automated testing

In spite of using a custom heuristic with cachecontrol, the system still skips the cache for some requests. Because web archives will likely block too many requests for the same URI-M, this will need to be alleviated for automated testing to be used with Travis CI.

Make sure the system identifies favicons for archives that specify them in their HTML

Specifically Archive-It and archive.is do not appear to display their favicons.

Speed up download of images when considering striking image

Currently, issues are downloaded sequentially. Using the requests-futures library, one can run multiple simultaneous requests.

Use new image scoring function for image selection

After reviewing MementoEmbed's capabilities with a variety of images, it has become clear that an image scoring function will prove to be better than just selecting the largest image.