The cds-videos from cerndocumentserver

sprint: planning

infrastructure
- puppet (#88)
- fab.py (#89)
- ~~cds|cdslabs.cern~~
- request EOS space (#90)
- installation documentation (#70)
search
- ES Schemas (#86)
- Search (facets, sort) (#71)
- Tests (#68)
files
- files-rest enabled (#87) -> (PR #95)
- previewer back-end (#92) -> (PR #93)
- previewer front-end (#85)
- list of files on detailed view (#84)
- list of files on brief view (#84)
- tests
interface
- detailed view
- home page
migration
- migration files + metadata (#91)
- EOS (#91)
- tests

update installation documentation

Change code languages by display names for language facet

Installation problems on Ubuntu

$ npm install -g node-sass clean-css requirejs uglify-js

Must install npm (i.e. apt-get install npm), duh!
node-sass internally uses the node command instead of nodejs, which is an issue with NodeJS v4.2.6 at least, so a symlink is required (i.e. sudo ln -swhich nodejs/usr/bin/node)

$ mkvirtualenv cds3

Must install virtualenv (i.e. pip install virtualenv)
Must install virtualenvwrapper (i.e. pip install virtualenvwrapper)

(cds3) $ pip install -e .

When pip have missing precompiled .whl files for a package, it will compile its C component and try to find the required lib header files. Only debian linux distributions split lib-dev packages (for header files) from lib, therefore the following packages need to be individually installed.
- libpg-dev
- libjpeg-dev
- libxml2-dev
- libxslt-dev
- libffi-dev

docker configuration problems

I tried to install cds from docker and I found some issues with ElasticSearch configuration:

The /tmp directory is not writable: JNA Warning: IOException removing temporary files: JNA temporary directory '/tmp' is not writable
(After previous problem is fixed) web and worker containers can't connect to ElasticSearch:

worker_1        | Test ES_PORT...tcp://172.17.0.148:9200...NO
web_1           | Test ES_PORT...tcp://172.17.0.148:9200...NO

Migration for files & metadata

Migrate files and metadata from legacy
Copy files to EOS

demosite: create demo records set

New scripts à la Invenio

https://github.com/inveniosoftware/invenio/tree/master/scripts

Prettify facets

Sometimes the text in the facet is too long and breaks the alignment, it would be nice to find a solution to that

We might also want to evaluate if include/exclude functionality should be added

migrator: only validate record's latest version

update and test new search developments (access rights)

python:3.5-slim fails to install lxml

After changing the basic image in docker from python:3.5 to python:3.5-slim, docker-compose build is failing because of:

gcc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isrc/lxml/includes -I/usr/local/include/python3.5m -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -w
    In file included from src/lxml/lxml.etree.c:320:0:
    src/lxml/includes/etree_defs.h:14:31: fatal error: libxml/xmlversion.h: No such file or directory
     #include "libxml/xmlversion.h"
                                   ^
    compilation terminated.
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitmhwo1u2u.c -o tmp/xmlXPathInitmhwo1u2u.o
    /tmp/xmlXPathInitmhwo1u2u.c:1:26: fatal error: libxml/xpath.h: No such file or directory
     #include "libxml/xpath.h"
                              ^
    compilation terminated.
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/local/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-9aj2a62m/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ud9ohti0-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-9aj2a62m/lxml/

After changing the image back to python:3.5 it works.
After manually checking the intermediate image, it seems that lxml is missing the libxml2-dev package.
@ddaze, did you have issues with that ?

Support CERNBox or (a text field for adding the ` url` )

Generate ES schemas

Generate ElasticSearch schema for the records.

Copy files to EOS

Puppet and fabric additional improvements

Change the invenio log location (make sure that user cds has access to this path !)
Remove fabric tasks that are not used (like DB creation for MySQL)
Configure the deploy task to not delete log files (or other files we want to preserve)
Fabric: Fix deploy config task
Add supervisorctl config in puppet (otherwise, it's impossible to run supervisorctl command to manage services)
puppet - install flower
fix index create and index destroy tasks
Fix celery not receiving tasks

fix docker installation

Request EOS space

Request EOS space for cdslabs.

tests for HD

docker - custom aliases and packages per developer

I was wondering if there is an easy way to add a user-specific file with configuration for docker that would specify additional packages to be installed and some aliases/settings to be added to bashrc ?
For example, I would like to say that I want to have:
aliases: ll = ls -al and cd ..=..
packages: vim, less, etc. (vim should probably go the list of packages by default :)
and someone else would like to have his own aliases/packages, so this settings would stay outside of git.
The most convenient way I see this is to point docker to a location on my machine (for example to: ~/.aliases and ~/.docker_packages) and make docker process those two files if they exist.
If they don't exist, simply do nothing.
@ddaze any ideas ?

Enable ``invenio-files-rest`` on the CDS app

Enable invenio-files-rest on our app.

Schema for the form

enable new search ui

MathJax currently doesn't work in the search results (it does on the detail view)

improve HB / hide source (show it for admins)

sprint: planning

front-end

back-end

conversion

Add the previewer to the detail view to preview `pdf` files

Enable invenio-previewer on detail view to preview listed files.

Migrate to sass

Bootstrap 4 will support only sass.

inveniosoftware/invenio-theme#9

Support Dropbox

`THEME_LOGO=None` breaks some templates

https://github.com/CERNDocumentServer/cds/blob/cdslabs/cds/config.py#L279

2016-06-07 11:24:10,216 ERROR: Exception on /login/ [GET] [in /opt/cds/lib/python2.7/site-packages/flask/app.py:1423]
Traceback (most recent call last):
  File "/opt/cds/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/cds/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/cds/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/cds/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/cds/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/cds/lib/python2.7/site-packages/flask_security/decorators.py", line 225, in wrapper
    return f(*args, **kwargs)
  File "/opt/cds/lib/python2.7/site-packages/flask_security/views.py", line 87, in login
    **_ctx('login'))
  File "/opt/cds/lib/python2.7/site-packages/flask_security/core.py", line 447, in render_template
    return render_template(*args, **kwargs)
  File "/opt/cds/lib/python2.7/site-packages/flask/templating.py", line 128, in render_template
    context, ctx.app)
  File "/opt/cds/lib/python2.7/site-packages/flask/templating.py", line 110, in _render
    rv = template.render(context)
  File "/opt/cds/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/opt/cds/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/cds/lib/python2.7/site-packages/invenio_oauthclient/views/../templates/invenio_oauthclient/login_user.html", line 27, in top-level template code
    {%- from "invenio_oauthclient/_macros.html" import oauth_button %}
  File "/opt/cds/lib/python2.7/site-packages/invenio_accounts/templates/invenio_accounts/login_user.html", line 27, in top-level template code
    {% from "invenio_accounts/_macros.html" import render_field, form_errors %}
  File "/opt/cds/lib/python2.7/site-packages/invenio_theme/templates/invenio_theme/page_cover.html", line 28, in top-level template code
    {%- set body_css_classes=['cover-page'] %}
  File "/opt/cds/src/cds/cds/modules/theme/templates/cds_theme/page.html", line 3, in top-level template code
    {%- set title = '%s | %s'|format(_(title), config.THEME_SITENAME) or 'CDS' -%}
  File "/opt/cds/lib/python2.7/site-packages/invenio_theme/templates/invenio_theme/page.html", line 73, in top-level template code
    {%- block body %}
  File "/opt/cds/lib/python2.7/site-packages/invenio_theme/templates/invenio_theme/page_cover.html", line 41, in block "body"
    {% block page_header %}
  File "/opt/cds/lib/python2.7/site-packages/invenio_theme/templates/invenio_theme/page_cover.html", line 45, in block "page_header"
    {%- block brand %}
  File "/opt/cds/lib/python2.7/site-packages/invenio_theme/templates/invenio_theme/page_cover.html", line 46, in block "brand"
    <a href="/"><img class="text-center masthead-brand" src="{{ url_for('static', filename=config.THEME_LOGO)}}" alt="{{_(config.THEME_SITENAME)}}" width="250px"/></a>
  File "/opt/cds/lib/python2.7/site-packages/flask/helpers.py", line 312, in url_for
    return appctx.app.handle_url_build_error(error, endpoint, values)
  File "/opt/cds/lib/python2.7/site-packages/flask/app.py", line 1641, in handle_url_build_error
    reraise(exc_type, exc_value, tb)
  File "/opt/cds/lib/python2.7/site-packages/flask/helpers.py", line 305, in url_for
    force_external=external)
  File "/opt/cds/lib/python2.7/site-packages/werkzeug/routing.py", line 1758, in build
    raise BuildError(endpoint, values, method, self)
BuildError: Could not build url for endpoint 'static'. Did you forget to specify values ['filename']?

Replace `pep257` with `pydocstyle`

Font missing

As adviced by CERN, you use PT Sans as a font on your frontend. But because it is not installed on most computers out there, it does not get loaded for most users. You should somehow load it as a webfont, e.g. by adding the following block to cds/base/tempates/page.html:

{%- block css %}
  {{ super() }}
  <link href=//fonts.googleapis.com/css?family=PT+Sans:400" rel="stylesheet" type="text/css">
{%- endblock css %}

Using Google fonts seems legit, because it is also used here: https://framework.web.cern.ch/framework/2.0/cern_toolbar/cern_toolbar-anonymous.html

Handle the urls in the UI
Celery task with feedback for downloading the remote file
Integration with the existing file model in the invenio-files-js

We will need to override https://github.com/inveniosoftware/invenio-migrator/blob/master/invenio_migrator/records.py#L132 and maybe https://github.com/inveniosoftware/invenio-migrator/blob/master/invenio_migrator/records.py#L176 to port the document types https://github.com/inveniosoftware/invenio-migrator/blob/master/invenio_migrator/legacy/bibdocfile.py#L117

Probably we should also take a look at https://github.com/inveniosoftware/invenio-records-files/blob/master/invenio_records_files/api.py#L75

It might be a good opportunity to clean up a bit bibdoc.doctypes, so far we have 158 different types - select count(distinct doctype) from bibdoc;.

37 of them have the suffix _FILES - select distinct doctype from bibdoc where doctype like '%_FILE';
2089 don't have a type - select count(*) from bibdoc where doctype=''; - and 26501are NULL - select count(*) from bibdoc where doctype is NULL;
there are lowercase, uppercase, capitalized, with spaces, with _ ...
some of them are only used once - select distinct doctype, count(id) from bibdoc group by doctype order by count(id) desc;

search results: file preview

The list of files to preview should be smaller, we should only preview the 'main' file (so not the plots, for example: https://cdslabs.cern.ch/search?page=5&size=20&q=muon&authors=Aad,%20Georges.
Also, the height of the preview thumbnail should be fixed, so there should be no jumping around when moving between files.

Sanity check on files after migration

https://its.cern.ch/jira/browse/CDS-2

Display list of files in deatail and brief view

Display the list of files for each record in detail and brief view.

cerndocumentserver / cds-videos Goto Github PK

cds-videos's People

Contributors

Stargazers

Watchers

Forkers

cds-videos's Issues

front-end

back-end

conversion

Recommend Projects

Recommend Topics

Recommend Org