wikia / ask-fandom Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 1.0 181 KB

Your semantic data-based assistant from Wikia

License: MIT License

Makefile 0.21% Python 65.87% CSS 25.10% JavaScript 5.56% HTML 3.26%

alexa fandom natural-language-processing semantic-data voice-assistant

ask-fandom's People

Contributors

Stargazers

Watchers

Forkers

macbre

ask-fandom's Issues

Use WikiData / DBpedia as one of data sources

Search for entity

https://www.wikidata.org/w/api.php?action=wbsearchentities&search=Lionel%20Messi&language=en&limit=50

"Lionel Messi"
https://www.wikidata.org/wiki/Q615

{
    "searchinfo": {
        "search": "Lionel Messi"
    },
    "search": [
        {
            "repository": "",
            "id": "Q615",
            "concepturi": "http://www.wikidata.org/entity/Q615",
            "title": "Q615",
            "pageid": 899,
            "url": "//www.wikidata.org/wiki/Q615",
            "label": "Lionel Messi",
            "description": "Argentine association football player",
            "match": {
                "type": "label",
                "language": "en",
                "text": "Lionel Messi"
            }
        }
    ]
}

Get an entity

https://www.wikidata.org/w/api.php?action=wbgetentities&sites=enwiki&titles=Lionel%20Messi&languages=en

DBpedia

http://dbpedia.org/page/Lionel_Messi

$ curl -s 'http://dbpedia.org/data/Lionel_Messi.ntriples' | grep -E "height|birth"
<http://dbpedia.org/resource/Lionel_Messi>	<http://dbpedia.org/ontology/birthDate>	"1987-6-24"^^<http://www.w3.org/2001/XMLSchema#date> .
<http://dbpedia.org/resource/Lionel_Messi>	<http://dbpedia.org/ontology/birthPlace>	<http://dbpedia.org/resource/Rosario,_Santa_Fe> .
<http://dbpedia.org/resource/Lionel_Messi>	<http://purl.org/dc/terms/subject>	<http://dbpedia.org/resource/Category:1987_births> .
<http://dbpedia.org/resource/Lionel_Messi>	<http://dbpedia.org/ontology/birthDate>	"1987-06-24"^^<http://www.w3.org/2001/XMLSchema#date> .
<http://dbpedia.org/resource/Lionel_Messi>	<http://dbpedia.org/ontology/height>	"1.7"^^<http://www.w3.org/2001/XMLSchema#double> .
<http://dbpedia.org/resource/Lionel_Messi>	<http://dbpedia.org/ontology/Person/height>	"170.0"^^<http://dbpedia.org/datatype/centimetre> .

Use spaCy rule-based matching to extract knowledge from articles content

https://spacy.io/usage/rule-based-matching#matcher

An example for Elmo article: https://explosion.ai/demos/matcher?text=Elmo%20is%20a%20furry%20red%20Muppet%20monster%20with%20an%20orange%20nose%20who%20lives%20on%20Sesame%20Street.%20He%20is%20self-described%20as%20three-and-a-half%20years%20old%2C%20and%20almost%20always%20refers%20to%20himself%20in%20the%20third%20person.%20The%20character%27s%20popularity%20led%20to%20his%20own%20segments%20on%20Sesame%20Street%20including%20%22Elmo%27s%20World%22%20and%20%22Elmo%20the%20Musical%22.%0A%0AElmo%20lives%20in%20an%20apartment%20on%20Sesame%20Street%20with%20his%20mother%20Mae%2C%20his%20father%20Louie%2C%20and%2C%20in%20some%20storybooks%2C%20a%20sister%20named%20Daisy.%20He%20has%20a%20pet%20goldfish%20named%20Dorothy.%0A%0AThe%20puppet%20that%20would%20eventually%20become%20Elmo%20was%20designed%20in%201979%20by%20Caroly%20Wilcox.%5B5%5D%5B6%5D%20Intended%20as%20a%20generic%20Anything%20Muppet%20monster%2C%20labels%20used%20for%20the%20puppet%20included%20%22baby%20monster%22%2C%20%22short%20red%22%20(in%20Wilcox%27s%20sketch)%2C%20and%20%22little%20red%22%20(on%20exhibit%20at%20Center%20for%20Puppetry%20Arts%2C%20Worlds%20of%20Puppetry%2C%20Jim%20Henson%20Collection).&model=en_core_web_sm&pattern=%5B%7B%22id%22%3A0%2C%22attrs%22%3A%5B%7B%22name%22%3A%22ORTH%22%2C%22value%22%3A%22Elmo%22%7D%5D%7D%2C%7B%22id%22%3A2%2C%22attrs%22%3A%5B%7B%22name%22%3A%22POS%22%2C%22value%22%3A%22VERB%22%7D%5D%7D%5D

Verify search console for key phrases

Verify which phrases were used by our users from google search to enter our site.
Start with:

Games of Thrones wikia https://gameofthrones.fandom.com/wiki/Game_of_Thrones_Wiki
Football wikia https://football.fandom.com/wiki/Football_Wiki

Evaluate spaCy

A free, open-source library for advanced Natural Language Processing (NLP) in Python

https://spacy.io/usage/spacy-101
https://spacy.io/usage/linguistic-features
https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da (NRE - Named Entity Recognition)

Add Q&A from answers for few wikis

Web Speech API - detect and synthesize

TODO list

build a simple Flask app
expose /ask API endpoint that will respond with an answer for the provided question
create a simple web page that will allow user to say his question and hear the answer

Browser API

https://www.google.com/intl/en/chrome/demos/speech.html // https://davidwalsh.name/speech-recognition

https://w3c.github.io/speech-api/#tts-section

Add WDS library

Word-sense disambiguation (WSD) is an open problem concerned with identifying which sense of a word is used in a sentence, it might be

few available libraries:

SenseLearner: All-Words Word Sense Disambiguation Tool
https://dl.acm.org/citation.cfm?doid=1225753.1225767

KYOTO UKB graph-based WSD http://ixa2.si.ehu.es/ukb/

pyWSD: Python Implementation of Simple WSD algorithms https://github.com/alvations/pywsd

Knowledge base

Storage

Cayley

SQL

Object		Property	Value
================================================
Star Wars	composer	John Williams
dynamite	inventor	Alfred Nobel

objects:
  object_id: int
  object_name: text
  object_type: text [mapped after schema.org types]

values:
  object_id: int
  property: text
  value: text
  links_to: int [points to another object, NULL - value is not "linkable"]

RDF

@prefix wikipedia: <http://en.wikipedia.org/>
@prefix starwars: <http://starwars.fandom.com/>
@prefix dbo: <http://example.org/terms/>
@prefix schema: <http://schema.org/>

starwars:Star_Wars dbo:type schema:Movie ;
    schema:composer starwars:John_Williams .
wikipedia:Dynamite dbo:inventor wikipedia:Alfred_Nobel .

<http://dbpedia.org/resource/Semantic_Web>      <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>       <http://dbpedia.org/ontology/Software> .