pydepta/pydepta

A python implementation of DEPTA

pydepta's Issues

infer() doesn't work

from pydepta import Depta

>>> d = Depta()
>>> seed = d.extract(url='http://www.iens.nl/restaurant/10545/enschede-rhodos')[5]
>>> d.infer(seed=seed, url='http://www.iens.nl/restaurant/34397/apeldoorn-de-boschvijver')

this throws the error

infer() takes at least 2 arguments (1 given)

what does infer do exactly and how can I get it working?

when I do

>>> d.infer(seed, url='http://www.iens.nl/restaurant/34397/apeldoorn-de-boschvijver')

it just gives me an empty list

annotate failed

on http://www.yellowpages.com.au/search/listings?clue=Aboriginal+%26+Torres+Strait+Islander+Associations+%26+Organisations&locationClue=All+States&selectedViewMode=&emsLocationId=

annotate the data record

integrate the sklearn to predict the data field type

after label some data manually (with xpath for example), learn from the data field and label and predict when new field arrive.

Make it clear this is not an exact implementation of DEPTA (and MDR-2)

I've read "Web Data Extraction Based on Partial Tree Alignment" paper, and it says that DEPTA uses MDR-2 data regions extraction algorithm, and that MDR-2 differs from MDR in how it constructs trees from HTML data: instead of relying on tag hierarchy it renders tree in a browser and uses bounding boxes of elements to create a tree. Also, visual information is used to compute better data regions: gap between two data records in a data region must be no smaller than any gap within a data record. I don't think pydepta implements any of this; the implementation looks more like MDR.

Maybe add a note to README that this is not a "real" DEPTA, but more like MDR + "Data extraction" part from DEPTA?

use lxml HTML.element text_content to avoid use pyquery

use itertools.groupby to replace find_subsequence?