sebastian-nagel Goto Github PK

apache-nutch-indexer-plugin

Google Cloud Search Apache Nutch Indexer Plugin

awesome-crawler

A collection of awesome web crawler,spider in different languages

browsertrix-crawler

Run a high-fidelity browser-based crawler in a single Docker container

cc-crawl-statistics

cc2dataset

Easily convert common crawl to a dataset of caption and document. Image/text Audio/text Video/text, ...

commoncrawl_notebooks

commoncrawldocumentdownload

A small tool with uses the CommonCrawl URL Index to download documents with certain file types or mime-types for mass-testing of frameworks like Apache POI and Apache Tika

cosr-back

Backend of Common Search. Analyses webpages and sends them to the index.

cotton-mask

Instructions for sewing a cotton face mask

crawler-commons

A set of reusable Java components that implement functionality common to any web crawler

data_tooling

Tools for managing datasets for governance and training.

docker-hadoop

Apache Hadoop docker image

duckdb-web

DuckDB-Web - Source code of duckdb.org

elasticsearch

Open Source, Distributed, RESTful Search Engine

ggplot

ggplot for python

hadoop

Apache Hadoop

idnet

Neural net language identification for many languages on short texts plus construction-based dialectometry

impf-botpy

Impf Bot.py 🐍⚡ – Automatisierung für den Corona ImpfterminService Bot

introduction-to-python

jwarc

Java library for reading and writing WARC files with a typed API

langdetect

Port of Google's language-detection library to Python.

news-please

news-please - an integrated web crawler and information extractor for news that just works.

nutch

Mirror of Apache Nutch

nutch-1

nutch-test-single-node-cluster

open-data-registry

A registry of publicly available datasets on AWS

open-json

Open JSON - a truly open source JSON implementation

ossym2022-robotstxt-experiments

Experiments and metrics about robots.txt captures, presentation at #ossym2022

proposal-warc-to-zstandard

A Proposal for Common Crawl to Consider Moving Compression from Gzip to Zstandard

pyathena

PyAthena is a Python DB API 2.0 (PEP 249) client for Amazon Athena.