Case folding
- Reduce case of words to lower except for upper-case words in middle of section.
- General Motors
- FED vs fed
- SAIL vs sail
- Group sequences of consecutive upper case letter in same sentence categorize into one word; General Motors.
Word Normalization
- U.S.A => USA. Need to specify the equivalence classes of words. An implicit equiv class for this example is deleting periods in a term.
To do
- create
apiQueryPatentView.pyto query data from PatentView- Template for API Query: https://www.dataquest.io/blog/python-api-tutorial/
- Template for JSON: https://www.dataquest.io/blog/python-json-tutorial/
- Bag-of-Words and Regex
- Do a comprehensive query on gene editing companies & set up a function to run
BagofWordson this query.- brf_sum_text
- claim
govint_raw_statement: The full government interest statement as reported on a given patent.patent_abstract: Abstract of the patent. (START HERE)patent_title: (Most likely useless)
- Clustering: Cluster the BagofWords vectors and other feasture vectors using some clustering algo.
- Optimal amount of clusters: http://www.sthda.com/english/wiki/print.php?id=239
- http://brandonrose.org/clustering
- https://pythonprogramminglanguage.com/kmeans-elbow-method/
- Use PCA and K-Means for now but later look into more complex matrix factorization techniques
- Fuzzy match the parameter
assignee_organizationwith a stock data set. Merge on fuzzymatch and date- Question? When are patents applications usually released? May need to add 1 day to patent data before match to reflect the information being observed by market
- How to map cusip to ticker.
- Or just use crosswalk on CUSIP's from NBER Cusip to PatentView Map
- Implement more general ML algos
Get unique words occuring in all text in our corpus and design a vocabulary.
The vocabulary will be a vector of size the length(of unique words in corpus). Each document will be described by a vector w/ entiries equal to the number of occurrence of words in vocabulary.
Problems
-[ ] The vector can be big and sparse so use scipy-sparse vector to represent (which one?)
-[ ] text cleaning techniques to reduce vocab
- Ignoring case
- Ignoring punctuation
- Ignoring frequent words that don’t contain much information, called stop words, like “a,” “of,” etc.
- Stemming: words to their stem (e.g. “play” from “playing”) using stemming algorithms
- Fixing misspelled words.
Enhancement
Generalize to bag-of-bigrams model.
*An N-gram is an N-token sequence of words: a 2-gram (more commonly called a bigram) is a two-word sequence of words like “please turn”, “turn your”, or “your homework”, and a 3-gram (more commonly called a trigram) is a three-word sequence of words like “please turn your”, or “turn your homework”... n-gram*
Notes
- Take a document as the input.
- Read the document line by line
- Tokenize the line (put into a vector)
- Process woords.
- https://en.wikipedia.org/wiki/Tf%E2%80%93idf
- https://s3.amazonaws.com/assets.datacamp.com/production/course_5064/slides/chapter2.pdf
- https://medium.com/@aakashtandel/the-basics-of-natural-language-programming-a-big-bag-of-words-2f2ac06638ea
** Patent Background **
- Stem cell therapy (1970s)
- Immunotherapy (1970s)
- Cas9 (RNA)
- TALE (Protein)
- Group II intron (RNA)
- Meganuclease (Protein)
- Recombinase (Protien)
- TtAgo (DNA)
- λ-beta/exo MAGE (DNA)
- RecACage (DNA)
- ZnF (Protein)
- meganucleases
- zinc finger nucleases ZFNs
- Transcription activator-like effector-based nucleases TALEN
- Clustered regularly interspaced short palindromic repeats Cas9/Crispr
- Allows to find CPC Patent code by keyword: https://worldwide.espacenet.com/classification?locale=en_EP#!/CPC=C12N15/00
- Data
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent