serlo / metadata-exports Goto Github PK

View Code? Open in Web Editor NEW

0.0 4.0 0.0 2.17 MB

Exports of the Serlo metadata API

Home Page: https://serlo.github.io/metadata-exports/

Python 17.55% HTML 0.36% Jupyter Notebook 82.08%

metadata serlo service

metadata-exports's Introduction

Unofficial exports of the Serlo metadata API

Development

Install the Python version in .tool-versions
- You may use asdf for the installation.

Using Pipenv

Install pipenv
Run pipenv shell to activate the project's virtual environment.
Run pipenv install --dev to install the dev dependencies.
Run pipenv run lint to run the linting.
Run pipenv run format to format the code.
Run pipenv run test to run the tests in /tests.
Run exit to exit the shell

Export Metadata

Run pipenv run python download_metadata.py [output_file] to download all metadata from serlo.org
Run pipenv run python convert2rss.py [input_file] [output_file] to convert the downloaded .json into .rss

metadata-exports's People

Contributors

Watchers

metadata-exports's Issues

Add documentation for setup in README.md

feat: Add `description` from website for each RSS element

When description, fetch the description from the website -> Use the code as in https://github.com/openeduhub/oeh-search-etl/blob/master/converter/spiders/serlo_spider.py#L268-L283

Use description from website
Use description from <script type="application/ld+json"></script> of website
Use description from parsing website

Fix bug with licenses

Not every rss item has license information -> this needs to be fixed for those elements with a cc license
Filter out all elements without a license

Add script to enhance data from Metadata API

Problem

Currently we have duplicated logic in https://github.com/serlo/metadata-exports/blob/main/convert2rss.py and https://github.com/serlo/metadata-exports/blob/main/update_datenraum_nodes.py to enhance a metadata record with a description from our website. We should remove the code duplication.

Solution

https://github.com/serlo/metadata-exports/blob/main/download_metadata.py downloads metadata from our Metadata API. In an extra script we can enhance the metadata with descriptions from our website. The script should:

Load the cached descriptions from the last run:

metadata-exports/convert2rss.py

Line 299 in 2be1c1c

def get_description_cache():

go through metadata resources and add descriptions from cache / website when not present:

metadata-exports/convert2rss.py

Lines 178 to 221 in 2be1c1c

    
           def get_description( 
        
               resource: Dict[str, Any], 
        
               description_cache: Dict[str, Any], 
        
               time_passed: timedelta, 
        
           ): 
        
               if "description" in resource and isinstance(resource["description"], str): 
        
                   return resource["description"] 
        
               cached_value = description_cache.get(resource["id"], {}) 
        
               if cached_value.get("version", None) == resource["version"] and isinstance( 
        
                   cached_value.get("description", None), str 
        
               ): 
        
                   return cached_value["description"] 
        
               if time_passed > timedelta(minutes=30): 
        
                   return None 
        
               new_description = load_description_from_website(resource) 
        
               description_cache[resource["id"]] = { 
        
                   "description": new_description, 
        
                   "version": resource["version"], 
        
               } 
        
               return new_description 
        
           def load_description_from_website(resource: Dict[str, Any]): 
        
               identifier = resource.get("identifier", {}).get("value", None) 
        
               if not isinstance(identifier, int): 
        
                   return None 
        
               data = load_json_ld(f"https://serlo.org/{identifier}") 
        
               if ( 
        
                   data is not None 
        
                   and "description" in data 
        
                   and isinstance(data["description"], str) 
        
               ): 
        
                   return data["description"] 
        
               return None

make sure that the script does not run more than X minutes (let's use 20min as a setting)
save the new json in public/ as a new file -> This file should be the import of https://github.com/serlo/metadata-exports/blob/main/convert2rss.py and https://github.com/serlo/metadata-exports/blob/main/download_datenraum_nodes.py
Delete code from https://github.com/serlo/metadata-exports/blob/main/convert2rss.py and https://github.com/serlo/metadata-exports/blob/main/download_datenraum_nodes.py which is not needed any more

Publish `.rss` and `.json` file via cron job to GitHub Page

Example: https://github.com/serlo/data-pipeline-interactive-exercises/blob/main/.github/workflows/build%20and%20deploy.yml

Follow Up of #2

Add example taxonomies to dev

Filter out Exercises and Exercise Groups

Delete serlo-metadata.json and serlo-metadata.rss

Download metadata always takes 20min

This should not happen when caching works

Load description if it is not there

update_datenraum_nodes.py should check for description and load it from public/index if it is not there.

Also update dev next to demo in Datenraum

Investigate: Why does publishing to Datenraum (DEV) not work currently?

Only metadata records to Datenraum with a description

After #101 only update resources to Datenraum which have a description set -> Make sure that nodes whithout a description shall be deleted.

Fix math content in description

Generate rss each day on Midnight

Datenraum: Only update nodes when necessary

Currently all resources are always updated -> Check whether an update is needed and do only update a resource to Datenraum when it is necessary

Add support for `<sdx:subject>` and filter for only math content

Filter for all elements with about === http://w3id.org/kim/schulfaecher/s1017
Add <sdx:subject>380</sdx:subject> for those elements in the RSS feed

Use HTTP/2 for updating learning oppurtunities of Serlo to Datenraum

User story

As a developer of Serlo

I want that the cron-job for updating the metadata of Serlo to Datenraum is fast

Acceptance criteria

Rewrite https://github.com/serlo/metadata-exports/blob/main/update_datenraum_nodes.py and https://github.com/serlo/metadata-exports/blob/main/datenraum.py so that they use HTTP/2 with concurrent calls. So we might save time in updating all metadata to the Datenraum.

Add filter: No video elements in RSS feed

Filter all items in https://github.com/serlo/metadata-exports/blob/main/convert2rss.py with type === "VideoObject"

feat: Add default image to each item (SVG format)

Add image to each element with link to publisher["logo"]:

Currently https://de.serlo.org/_assets/img/serlo-logo.svg

Add Filter: Only German content

Only export items with language=de in https://github.com/serlo/metadata-exports/blob/main/convert2rss.py

Delete changes made in Demo with the edges

feat: Filter elements by subject

Publish the `.rss` file and the `.json` file on a GitHub page instead of the repo

feat: Find solution to add keywords

Is there an esay way to extract keywords from the website? (Not necessary for now)

	def get_description(
	resource: Dict[str, Any],
	description_cache: Dict[str, Any],
	time_passed: timedelta,
	):
	if "description" in resource and isinstance(resource["description"], str):
	return resource["description"]

	cached_value = description_cache.get(resource["id"], {})

	if cached_value.get("version", None) == resource["version"] and isinstance(
	cached_value.get("description", None), str
	):
	return cached_value["description"]

	if time_passed > timedelta(minutes=30):
	return None

	new_description = load_description_from_website(resource)

	description_cache[resource["id"]] = {
	"description": new_description,
	"version": resource["version"],
	}

	return new_description


	def load_description_from_website(resource: Dict[str, Any]):
	identifier = resource.get("identifier", {}).get("value", None)

	if not isinstance(identifier, int):
	return None

	data = load_json_ld(f"https://serlo.org/{identifier}")

	if (
	data is not None
	and "description" in data
	and isinstance(data["description"], str)
	):
	return data["description"]

	return None