sszepe/lama

Research tool LAMA developed during the Telling Sounds project. See following link for example data:

LAMA (Linked Annotations for Media Analysis)

About the Telling Sounds project and LAMA

The research project Telling Sounds was concerned with the impact of the digital availability of audio(visual) sources regarding new approaches in the field of musicology. The embedding of music in radio programs, reports, documentaries, and films points to the significance of these sources for a music history beyond the grand narrative of musical "heroes". The sources in question contain different kinds of music in a variety of contexts: the audio(visual) elements refer to different times, institutions, events, places, persons, repertoires, and sounds respectively, whose multiplicity, diversity, and spread across time and space challenge a merely linear understanding of history. Striving to link such sources across the boundaries of various collections and archives calls for a combination of methods, taken from oral history, media- and film-studies, music analysis, cultural-studies-informed-musicology and performance studies.

To meet this challenge, LAMA (Linked Annotations for Media Analysis) has been developed, a research software for capturing and visualizing the interaction of music and its contexts. LAMA aims to provide a collaborative means of building a corpus of relevant AV material, as well as adding metadata about the contents. Music, Speech, Picture, and Other Sounds appearing in a Clip can be described in detail. "Segments" can be used to add structural information about a Clip or single out sections and group annotations. Annotations can be about People, Places, Pieces of Music or historical periods but also political ideologies or lieux de mémoire (i.e. sites of memory). Material mainly comes from the collections of our two principal cooperating institutions, Mediathek Austria and Phonogramm Archive of the Austrian Academy of Science and Research, but could just as easily come from YouTube or other sources. A read-only version of LAMA is available [here]https://repo.mdw.ac.at/telling-sounds/lama/.

The main concept used to enter information in the system are Annotations. They are used for capturing the various persons, topics, pieces of music and other points of interest ("entities") that appear in the Clips. Besides the relevant Entity (quotes or dates are also supported), an Annotation includes a Relation (in which way something appears in the clip; e.g. "mentioned") and timecodes. Furthermore, Annotations can be marked as "interpretative", indicating that the annotated Entity does not appear directly but is nevertheless perceived to be present implicitly by the researcher. Annotations can be grouped either by the sub-element of a clip (Music, Speech etc.) they're describing or by generic, user-defined Segments, which can also be used to describe the various sections of a clip.

LAMA also offers querying capabilities, including combining queries and grouping the results by what is appearing together in the same clip, for example which pieces of music appear together with a specific topic or keyword. Moreover, connections between clips and entities can be visualized as a network, starting either from a clip or an entity.

About the application

LAMA is a web application implemented in TypeScript (React, Material UI), with a REST-backend implemented in Python (Bottle), developed in the Telling Sounds project at the mdw Vienna by Julia Jaklin and Peter Provaznik. All user actions that produce changes are stored in an event log (a SQLite database). This provides a full history, makes restoring past states or undoing unwanted actions simple, while also providing flexibility regarding changes in data representation. The data from the event log is fed into a MongoDB server, making the current application state available for querying and retrieval. This means that the event log is the source of truth for the system, from which the MongoDB representation is created. This can be done with the reset_from_eventlog.py script, either restoring the state from the current events database, or providing an exported (lama.importexport) XML file.

Note: Currently the user interface does not provide a way to delete a Clip; this can be done by inputting the IDs of the Clips to be deleted into scripts/delete_clip.py and running it.

User management

User data is stored in a separate SQLite DB (default: lama_users.db). Users can be managed through either a CLI-script (scripts/lamaherder.py) or the API (/users). However, if there are no existing users, an (probably admin) user will have to be created using the script (this could be included in the docker build).

usage: lamaherder.py [-h] [--password [PASSWORD]] [--email EMAIL] [--privileges rwa] [--active yn]
                     [--reauth yn]
                     {showusers,createuser,deleteuser,updateuser} [username]

For example: $ python3 scripts/lamaherder.py createuser admin --privileges a --password MyPass234

(Usernames need to be 2-12 lowercase letters; passwords need to be 8-16 letters/numbers with at least one uppercase/lowercase/number.)

Once a user has been created, these credentials can be used to obtain an access token.

$ curl -Ss -H 'Content-type: application/json' -d '{"username": "admin", "password": "MyPass234"}' <backend_url>/auth

Response:

{
  "username": "admin",
  "privileges": "a",
  "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NDAwOTg1MjAsInVzciI6InBwcm92YXpuaWsiLCJwcnYiOiJhIn0.KxIjvVYQpD_ffc6wfP-_dMXyPAMESYFlWeQlIOeq0DY",
  "expires": "2021-12-21T14:55:20+00:00"
}

Then create users via API:

$ curl -Ss -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NDAwOTg1MjAsInVzciI6InBwcm92YXpuaWsiLCJwcnYiOiJhIn0.KxIjvVYQpD_ffc6wfP-_dMXyPAMESYFlWeQlIOeq0DY' -H 'Content-type: application/json' -X POST -d '{"username": "testuser"}' <backend_url>/users

Response:

{"password": "RjJ4BBVtKqoq"}

If no password is set, it will be generated. See the /users routes in server.py for details.

Local docker build

This is most likely the easiest way to get a running LAMA. You could also have a look at the instance deployed at https://repo.mdw.ac.at/telling-sounds/lama/.

Empty LAMA

• cd to the project root
• $ docker build --target=lama_base -t lama:empty .
• $ docker run -it --rm --name lamatest -p 127.0.0.1:3333:3333 -p 127.0.0.1:8080:8080 lama:empty
• point your web browser at http://localhost:8080 (default user is "admin" // "Wooly4711")

LAMA with data

• $ docker build -t lama:tsdata .
• $ docker run -it --rm --name lamatest -p 127.0.0.1:3333:3333 -p 127.0.0.1:8080:8080 lama:tsdata

Local manual build

(When in doubt you could always look at the Dockerfile, which does something very similar.)

Prerequisites

• Python 3.8 or newer
• current Node.js (last version used was v16.15.0)
• MongoDB 4.4 server running at localhost:27017

Building

• cd to the project root
• we recommend creating virtualenv: $ python3 -m venv .venv
• activate virtualenv (or use .venv/bin/python): $ . .venv/bin/activate
• install python packages: $ pip install -e backend/
• $ cd frontend/
• install npm packages: $ npm install
• compile JavaScript: $ API_URL="http://localhost:3333" WS_URL="ws://localhost:3333/ws" npm run build:production (ignore warnings/errors)

Setup

• cd back to project root: $ cd ..
• create user: $ python scripts/lamaherder.py createuser admin --privileges a --password Admin123
• optionally import data: $ python scripts/replace_mongo_data.py full.json (or PhA.json)

Running

• (use two separate terminal windows)
• start MongoDB if you haven't already
• start backend (again use virtualenv): $ python -m lama.server --cors
• frontend needs a web server to work properly, for example: $ (cd frontend/dist/ && python -m http.server)
• point your web browser at http://localhost:8000 (if using the above command)

Local development

• build and setup are identical to local build, no need to compile JS though
• start backend: $ python -m lama.server --dev
• start frontend dev server: $ (cd frontend/ && npm run start)
• (the dev server provides live reloading; on Linux, if you get an error about too many open files, you could try: $ sysctl fs.inotify.max_user_watches=524288 && sysctl -p)

Additional notes for development

• Whenever the word "Connection" appears in the code, it should probably be "Annotation".
• Unfortunately, the code is not well-documented (sorry!); looking at the Props of React components and their type can really help.
• If you're seriously going to do something with this, we suggest you reach out to the developers for support.

API Routes

/annotations

POST /annotations

Create an annotation

	Route Information
Route	/annotations
Method	POST
Request Body	Entity
Response	the newly created Annotation

PUT /annotations

Update an annotation

	Route Information
Route	/annotations
Method	PUT
Request Body	Entity
Response	the updated Annotation

DELETE /annotations

Delete an annotation

	Route Information
Route	/annotations
Method	DELETE

/auth

POST /auth

Get Auth Token

	Route Information
Route	/auth
Method	POST
Request Body	Username, Password
Response	Auth Token

/clips

GET /clips

Get all clips

	Route Information
Route	/clips
Method	GET
Response	Clips and pagination information

POST /clips

Create a new clip

	Route Information
Route	/clips
Method	POST
Request Body	a ClipBasic object
Response	ClipId of the newly created Clip

PUT /clips

Update basic Metadata of an existing clip

	Route Information
Route	/clips
Method	PUT
Request Body	a ClipBasic object
Response	ClipId of the updated Clip

GET /clips/<clip_id>

Get one clip by id

	Route Information
Route	/clips/<clip_id>
Method	GET
Response	a Clip object

GET /clips/basic/<clip_id>

Get one clip basic by id

	Route Information
Route	/clips/basic/<clip_id>
Method	GET
Response	a Clip object

GET /clips/sameShelfmark/<clip_id>

Get all clip ids of clips with the given shelfmark

	Route Information
Route	/clips/sameShelfmark/<clip_id>
Method	GET
Response	List of ClipIds

/elements

POST /elements

Create an Element

	Route Information
Route	/elements
Method	POST
Request Body	Element
Response	Id of the newly created Element

PUT /elements

Update an Element

	Route Information
Route	/elements
Method	PUT
Request Body	Element
Response	Id of the updated Element

DELETE /elements

Delete an Element

	Route Information
Route	/elements
Method	DELETE

/entities

GET /entities

Get all entities

	Route Information
Route	/entities
Method	GET
Response	Entities and pagination information

POST /entities

Create a new entity

	Route Information
Route	/entities
Method	POST
Request Body	Entity data
Response	newly created Entity with Id

PUT /entities

Updade existing entity

	Route Information
Route	/entities
Method	PUT
Request Body	Entity data
Response	updated Entity

DELETE /entities

Delete existing entity

	Route Information
Route	/entities
Method	DELETE
Request Body	Entity

GET /entities/<entity_id>

Get one Entity by id

	Route Information
Route	/entities/<entity_id>
Method	GET
Response	Entity

POST /entities/byId

Get certain entities by Id

	Route Information
Route	/entities/byId
Method	POST
Request Body	List of EntityIds
Response	List of Entity objects

GET /entities/csv

Download Entities as csv

	Route Information
Route	/entities/csv
Method	GET

GET /entities/match

Get entities filtered by search text and type

	Route Information
Route	/entities/match
Method	GET
Parameters	"q": search Text; "type": comma-separated entity types
Response	List of Entity objects

POST /entities/relations

Add a relation to an Entity

	Route Information
Route	/entities/relations
Method	POST
Response	List of EntityRelationsData

GET /entities/relations/<entity_id>

Return the possible entity relations for an Entity

	Route Information
Route	/entities/relations/<entity_id>
Method	GET
Response	List of EntityRelationsData

GET /entities/suggest

Get entities by Id for an empty Autocomplete Component

	Route Information
Route	/entities/suggest
Method	GET
Parameters	"type": comma-separated entity types
Response	List of Entity objects, number: total Number of returned Entities

/export

GET /export/data

Export mongo data

	Route Information
Route	/export/data
Method	GET
Response	All data from the mongo store

GET /export/events

Export events as xml

	Route Information
Route	/export/events
Method	GET
Response	Events in xml

/graph

GET /graph/clip/<clip_id>

Get the data for the graph view (for a Clip)

	Route Information
Route	/graph/clip/<clip_id>
Method	GET
Response	Graph data for the given ClipId

GET /graph/entity/<entity_id>

Get the data for the graph view (for an Entity)

	Route Information
Route	/graph/entity/<entity_id>
Method	GET
Response	Graph data for the given EntityId

/guest

GET /guest

Set privileges for the read only version

	Route Information
Route	/guest
Method	GET

/id

GET /id/<id_>

Get Clip, Element (Music, Speech, Noise, Picture), Segment or Entity by id

	Route Information
Route	/id/<id_>
Method	GET
Response	Clip, Element (Music, Speech, Noise, Picture), Segment or Entity

/login

POST /login

Login a given User

	Route Information
Route	/login
Method	POST
Request Body	Username, Password

/logout

GET /logout

Logout

	Route Information
Route	/logout
Method	GET

/query

POST /query/block

Results for a query block

	Route Information
Route	/query/block
Method	POST
Request Body	Query block
Response	Results of the query block

POST /query/entities

Query entities

	Route Information
Route	/query/entities
Method	POST
Request Body	query: a query string
Response	List of matching Entitys

POST /query/intersection

Results for a intersected query

	Route Information
Route	/query/intersection
Method	POST
Request Body	block results
Response	Results of the intersected query

/search

GET /search/quotes

Search through quotes

	Route Information
Route	/search/quotes
Method	GET
Response	Quotes, where search string matched

/segments

POST /segments

Create a new Segment

	Route Information
Route	/segments
Method	POST
Request Body	Segment
Response	Id of the newly created Segment

PUT /segments

Update the Basic Info of a Segment

	Route Information
Route	/segments
Method	PUT
Request Body	Segment
Response	Id of the updated Segment

DELETE /segments

Delete a Segment

	Route Information
Route	/segments
Method	DELETE

PUT /segments/annotations

Update the List of Annotations belonging to a Segment

	Route Information
Route	/segments/annotations
Method	PUT
Request Body	SegmentId, List of Annotations
Response	Id of the updated Segment

/users

GET /users

Get all users

	Route Information
Route	/users
Method	GET
Response	List of all users

POST /users

Create new user

	Route Information
Route	/users
Method	POST
Response	User and newly generated password if no password was given

GET /users/<user_id>/favorites

Get favorites from a user

	Route Information
Route	/users/<user_id>/favorites
Method	GET
Response	userId, favoriteClips: List of ClipIds

POST /users/<user_id>/favorites

Set or unset a favorite clip

	Route Information
Route	/users/<user_id>/favorites
Method	POST

DELETE /users/<user_id>/favorites

Set or unset a favorite clip

	Route Information
Route	/users/<user_id>/favorites
Method	DELETE

GET /users/<username>

Get user by username

	Route Information
Route	/users/
Method	GET
Response	User

PUT /users/<username>

Update user

	Route Information
Route	/users/
Method	PUT
Response	newly generated password, if no new was given

DELETE /users/<username>

Delete user

	Route Information
Route	/users/
Method	DELETE

GET /users/me

Get username of currently loggend in user

	Route Information
Route	/users/me
Method	GET
Response	username, privileges

Schemas

Annotation

name	type	required	description	pattern
_id	string	y	Annotation ID.	-
type	Annotation	y	Always "Annotation".	-
clip	string	y	ID of the Clip the Annotation belongs to.	-
element	string	n	ID of the associated Element, if applicable.	-
relation	RBroadcastDate, RBroadcastSeries, RContributor, RDateOfCreation, RInstrument, RKeyword, RLyricsQuote, RTextQuote, RMentions, RMusicalQuote, RPerformanceDate, RPerformer, RPieceOfMusic, RProductionTechniquePicture, RProductionTechniqueSound, RQuote, RRecordingDate, RReminiscentOfMusic, RReminiscentOfNoise, RReminiscentOfPicture, RReminiscentOfSpeech, RShot, RShows, RSoundAdjective, RSoundsLike, RSoundsLikeNoise, RSpeaker, RSpeechDescriptionText	y	The Relation expressed by the Annotation.	-
target	string	n	ID of the entity being annotated or of the person associated with a quote.	-
quotes	string	n	The text of a quote, if applicable.	-
date	string	n	The date being annotated, in EDTF (level 1) format.	-
role	string	n	ID of role-Entity (of type VClipContributorRole, VFunctionInClipRole, or VMusicPerformanceRole), if applicable.	-
metaDate	string	n	Supplementary date information, for example the year the annotated person was photographed or recorded, in EDTF (level 1) format.	-
timecodeStart	integer	n	Starting timecode of Annotation, in seconds from the beginning of the clip.	-
timecodeEnd	integer	n	Ending timecode of Annotation, in seconds from the beginning of the clip.	-
confidence	certain, maybe, unsure	n	Supplementary information about how confident the annotator is about the accuracy of the annotated information.	-
attribution	string	n	Supplementary information about the origin of the Annotation's contents, for example a bibliographic reference.	-
comment	string	n	Free-form comment, that will be shown in the UI.	-
interpretative	boolean	n	Whether the Annotation describes something which is explicitly shown or mentioned, or something that is not present explicitly, but nevertheless inferred by the annotator.	-
constitutedBy	array	n	IDs of other Annotations (on the same Clip) which support the interpretative Annotation.	-
notablyAbsent	boolean	n	Whether the Annotation is about something that would be expected to be present, but isn't.	-
refersTo	string	n	ID of another Annotation (on the same Clip) that the current Annotation applies to.	-
created	string	y	Timestamp of the Annotation's creation, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
createdBy	string	y	Username of the user who created the Annotation.	-
updated	string	n	Timestamp of the Annotation's last update, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
updatedBy	string	n	Username of the user who last updated the Annotation.	-

Clip

name	type	required	description	pattern
_id	string	y	Clip ID.	-
type	Clip	y	Always "Clip".	-
effectiveId	string	n	Derived ID used for deduplication purposes.	-
title	string	y	The Clip's title, as it appears on the source website.	-
label	string	n	Optional user-defined label, in case the original title is too long or unclear.	-
subtitle	string	n	The Clip's subtitle, if applicable.	-
url	string	y	The URL (preferably permalink) of the website where the Clip can be found.	-
platform	string	y	The ID of the platform-Entity (like "Mediathek" or "YouTube") associated with the Clip.	-
collections	array	n	IDs of (user-defined) collection-Entities the Clip belongs to.	-
fileType	a, v	y	Whether the Clip is an audio or video file.	-
duration	integer	y	Duration of the Clip in seconds.	-
shelfmark	string	n	Shelfmark ("Signatur") of the Clip, if applicable.	-
language	array	y	IDs of language-Entities that apply to the Clip.	-
clipType	array	y	IDs of ClipType-Entities that best describe the Clip (for example: "Interview").	-
description	string	n	Free-form description, either copied from the source or user-provided.	-
created	string	n	Timestamp of the Clips's creation in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
createdBy	string	n	Username of the user who created the Clip in the system.	-
updated	string	n	Timestamp of the Clips's last update in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
updatedBy	string	n	Username of the user who last updated the Clip in the system.	-
updatedAny	string	n	Timestamp of last update of the Clip or any of its Annotations, in ISO8601 format.	-
updatedAnyBy	string	n	Username of the user who last updated the Clip or any of its Annotations.	-

Element

name	type	required	description	pattern
_id	string	y	The ID of the Element.	-
type	Music, Noise, Picture, Speech	y	Which aspect/layer/type of phenomenon the Element is about.	-
clip	string	y	ID of the Clip the Element belongs to.	-
label	string	y	User-provided label.	-
description	string	y	User-provided description explaining what the Element is referring to.	-
timecodes	array	y	Pairs of start and end timecodes (in seconds) identifying the relevant parts of the Clip.	-
created	string	y	Timestamp of the Element's creation in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
createdBy	string	y	Username of the user who created the Element in the system.	-
updated	string	n	Timestamp of the Element's last update in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
updatedBy	string	n	Username of the user who last updated the Element in the system.	-

Entity

name	type	required	description	pattern
_id	string	y	ID of the Entity.	-
type	AnySound, BroadcastSeries, Broadcaster, Collection, CollectiveIdentity, CreativeWork, Event, EventSeries, FictionalCharacter, Genre, Group, LieuDeMemoire, Location, Movement, Organization, Person, PieceOfMusic, Place, Platform, ProductionTechniquePicture, ProductionTechniqueSound, Repertoire, Shot, Station, Thing, TimePeriod, Topic, Topos, VActivity, VAdjective, VClipContributorRole, VFunctionInClipRole, VClipType, VEventType, VInstrument, VLanguage, VMusicArts, VMusicPerformanceRole, VSpeechGenre	y	The Entity's type.	-
label	string	y	Primary label.	-
description	string	y	Short description that ideally makes it clear what the Entity is referring to.	-
authorityURIs	array	y	Authority URIs, preferably GND, Wikidata, or MusicBrainz (for music).	-
analysisCategories	array	n	Analysis-categories assigned to the Entity.	-
attributes	object	n	Experimental attributes imported from authority data (Entity-IDs).	-
created	string	n	Timestamp of the Entity's creation in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
createdBy	string	n	Username of the user who created the Entity in the system.	-
updated	string	n	Timestamp of the Entity's last update in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
updatedBy	string	n	Username of the user who last updated the Entity in the system.	-

Segment

name	type	required	description	pattern
_id	string	y	Segment ID.	-
type	Segment	y	Always "Segment".	-
clip	string	y	ID of the Clip the Segment belongs to.	-
label	string	y	User-provided label.	-
description	string	y	User-provided description explaining what the Segment is referring to.	-
timecodes	array	y	Pairs of start and end timecodes (in seconds) identifying the relevant parts of the Clip.	-
segmentContains	array	y	IDs of Annotations belonging to the Segment, with metadata.	-
created	string	y	Timestamp of the Segment's creation in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
createdBy	string	y	Username of the user who created the Segment in the system.	-
updated	string	n	Timestamp of the Segment's last update in the system, in ISO8601 format.	[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{6})?\+00:00$
updatedBy	string	n	Username of the user who last updated the Segment in the system.	-