Import DNS metadata

[sylvinus]

Import DNS metadata

[sylvinus]

Greg mentioned that Internet Archive does "survey crawls" every year with some root zones, but they are pretty spammy, as expected.

[thepieuvre]

Why not starting by indexing only domains from indexed Web pages and doing update when pages are updated? it is not complete, but it is a start.

[sylvinus]

@thepieuvre currently we reindex everything each month so we can avoid doing updates

However making sure we have the root domain indexed for each page is a good start indeed.

[indolering]

Those spammy URLs are just as valuable as non-spammy URLs.

[sylvinus]

@indolering yes there are as a dataset, but we don't want to have them in the search results, the index will be large enough as it is!

[indolering]

But wouldn't it help to know which pages those spammy domains link to?