Topic: data-processing Goto Github

👇 Here are 1003 public repositories matching this topic...

allenai / dolma
data-processing,Data and tools for generating and inspecting OLMo pre-training data.
Organization: allenai
Home Page: https://allenai.github.io/dolma/
data-processing large-language-models llm machile-learning nlp

alttch / rapidtables
data-processing,Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Organization: alttch
python python3 text-formatting data-processing library dictionary-data

apache / incubator-wayang
data-processing,Apache Wayang(incubating) is the first cross-platform data processing system.
Organization: apache
Home Page: https://wayang.incubator.apache.org/
cross-platform data-processing apache big-data data-management-platform distributed-system hadoop java jdbc middleware

asyml / forte
data-processing,Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
data-processing deep-learning information-retrieval machine-learning natural-language natural-language-processing pipeline python text-data

asyml / texar
data-processing,Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
Home Page: https://asyml.io
machine-learning natural-language-processing tensorflow deep-learning text-generation python machine-translation dialog-systems texar bert

asyml / texar-pytorch
data-processing,Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
Home Page: https://asyml.io
bert casl-project data-processing deep-learning dialog-systems gpt-2 machine-learning machine-translation natural-language-processing python pytorch roberta texar texar-pytorch text-data text-generation xlnet

benibela / xidel
data-processing,Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
User: benibela
Home Page: http://www.videlibri.de/xidel.html
xquery xml html json xpath cli command-line http web rest css-selector wget curl httpie xmlstarlet webscraper webscraping scraper datascraping data-processing

brexhq / substation
data-processing,Substation is a cloud-native, event-driven data pipeline toolkit built for security teams.
Organization: brexhq
Home Page: https://substation.readme.io
aws serverless security etl go data-engineering data-processing

bytewax / bytewax
data-processing,Python Stream Processing
Organization: bytewax
Home Page: https://docs.bytewax.io/
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data

chenghaomou / text-dedup
data-processing,All-in-one text de-duplication
User: chenghaomou
text-processing de-duplication nlp data-processing

colasgael / machine-learning-for-solar-energy-prediction
data-processing,Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
User: colasgael
data-processing machine-learning matlab neural-network python tensorflow

constellation-rs / amadeus
data-processing,Harmonious distributed data analysis in Rust.
Organization: constellation-rs
Home Page: https://constellation.rs/amadeus
data-processing data-analysis stream-processing parallel-computing distributed-computing rust

dashbitco / broadway
data-processing,Concurrent and multi-stage data ingestion and data processing with Elixir
Organization: dashbitco
Home Page: https://elixir-broadway.org
elixir genstage data-ingestion data-processing concurrent broadway

flow-php / etl
data-processing,PHP - ETL (Extract Transform Load) data processing library
Organization: flow-php
Home Page: https://flow-php.com
data-engineering data-processing etl flow-php

googlecloudplatform / data-science-on-gcp
data-processing,Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Organization: googlecloudplatform
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering

googlecloudplatform / dataflowjavasdk
data-processing,Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Organization: googlecloudplatform
Home Page: http://cloud.google.com/dataflow
google-cloud-dataflow data-science data-analysis data-mining big-data data-processing

helmholtz-analytics / heat
data-processing,Distributed tensors and Machine Learning framework with GPU and MPI acceleration in Python
Organization: helmholtz-analytics
Home Page: https://heat.readthedocs.io/
gpu tensors distributed machine-learning mpi numpy python pytorch array-api data-analytics

hstreamdb / hstream
data-processing,HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Organization: hstreamdb
Home Page: https://hstream.io/
streaming database iot stream-processing haskell real-time sql hstreamdb scale distributed-systems

hxz393 / brutalityextractor
data-processing,适用于高性能系统的多进程解压缩软件(A multiprocess decompression software for high-performance system)
User: hxz393
brute-force computational-efficiency data-processing decompression extractor high-performance optimization parallel-computing parallel-processing performance-optimization

iam-mhaseeb / skytrax-data-warehouse
data-processing,A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
User: iam-mhaseeb
python python3 database data-visualization data-analysis data-warehouse data-warehousing redshift airflow docker

infoslack / awesome-kafka
data-processing,A list about Apache Kafka
User: infoslack
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing

itechart / convtools-ita
data-processing,convtools is a python library to declaratively define conversions for processing collections, doing complex aggregations and joins.
Organization: itechart
Home Page: https://convtools.readthedocs.io
python code-generation transformations conversions data-preprocessing data-processing data-preparation functional-programming

johnkerl / miller
data-processing,Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
User: johnkerl
Home Page: https://miller.readthedocs.io
data-processing data-cleaning csv csv-format streaming-data streaming-algorithms tsv json json-data data-reduction

kousun12 / eternal
data-processing,👾~ music, eternal ~ 👾
User: kousun12
Home Page: https://eternal.rob.computer
3d-graphics art creative-coding data-processing glsl midi music node-based webaudio webgl

librecat / catmandu
data-processing,Catmandu - a data processing toolkit
Organization: librecat
Home Page: https://librecat.org
code4lib cli csv data-processing datamining ead json lido mab marc21

lithops-cloud / lithops
data-processing,A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
Organization: lithops-cloud
Home Page: http://lithops.cloud
python big-data data-processing multicloud big-data-analytics serverless distributed object-storage parallel kubernetes

markus-wa / cq
data-processing,Clojure Command-line Data Processor for JSON, YAML, EDN, XML and more
User: markus-wa
clojure yaml json edn csv command-line cli msgpack hacktoberfest xml

matousc89 / padasip
data-processing,Python Adaptive Signal Processing
User: matousc89
adaptive-filtering signal-processing data-processing data-analysis machine-learning

maykulkarni / machine-learning-notebooks
data-processing,Machine Learning notebooks for refreshing concepts.
User: maykulkarni
machine-learning python python-machine-learning machine-learning-algorithms deep-learning deep-learning-tutorial neural-networks data-science-notebook data-processing regression-models

mech-lang / mech
data-processing,🦾 Main repository for the Mech programming language. Start here!
Organization: mech-lang
Home Page: http://mech-lang.org
ide programming-environment programming-language compiler data-processing reactive-programming live-programming language robotics

microsoft / dialogpt
data-processing,Large-scale pretraining for dialogue
Organization: microsoft
dialogue machine-learning pytorch transformer text-generation dialogpt gpt-2 text-data data-processing

microsoft / godel
data-processing,Large-scale pretrained models for goal-directed dialog
Organization: microsoft
Home Page: http://aka.ms/GODEL
data-processing dialogue dialogue-systems machine-learning text-data text-generation transformers conversational-ai language-grounding grounded-generation

ml6team / fondant
data-processing,Production-ready data processing made easy and shareable
Organization: ml6team
Home Page: https://fondant.ai/en/stable/
data-processing fine-tuning foundation-models machine-learning pipeline python

msamogh / nonechucks
data-processing,Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
User: msamogh
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch

numaproj / numaflow
data-processing,Kubernetes-native platform to run massively parallel data/streaming jobs
Organization: numaproj
Home Page: https://numaflow.numaproj.io
kubernetes stream-processing data-processing k8s pipeline map-reduce hacktoberfest

nvidia / dali
data-processing,A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Organization: nvidia
Home Page: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html
fast-data-pipeline image-augmentation data-augmentation image-processing data-processing deep-learning machine-learning python neural-network gpu

onceupon / bash-oneliner
data-processing,A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
User: onceupon
Home Page: http://onceupon.github.io/Bash-Oneliner/
oneliner-commands bash data-processing linux-administration terminal linux variables grep xargs system

polyaxon / haupt
data-processing,Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
Organization: polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib

python-bonobo / bonobo
data-processing,Extract Transform Load for Python 3.5+
Organization: python-bonobo
Home Page: https://www.bonobo-project.org/
data-processing extract-transform-load bonobo python3 automation parallelization

pytlab / vaspy
data-processing,Manipulating VASP files with Python.
User: pytlab
Home Page: https://pypi.python.org/pypi/vaspy/
vasp data-processing data-visualization chemistry catalysis-informatics computational-chemistry

remotesensinginfo / rsgislib
data-processing,Remote Sensing and GIS Software Library; python module tools for processing spatial data.
Organization: remotesensinginfo
Home Page: https://www.rsgislib.org
remote sensing gis spatial data earth observation classification machine-learning remote-sensing

scramjetorg / scramjet
data-processing,Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.
Organization: scramjetorg
Home Page: https://www.scramjet.org
data-stream event-stream transformations reactive-programming javascript data-processing edge-computing python raspberry-pi data-space

sebkrantz / collapse
data-processing,Advanced and Fast Data Transformation in R
User: sebkrantz
Home Page: https://sebkrantz.github.io/collapse/
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted

senbox-org / snap-engine
data-processing,ESA Earth Observation Toolbox and Java Development Platform
Organization: senbox-org
Home Page: http://step.esa.int
data-processing data-visualization earth-observation eo raster-data remote-sensing linux macos windows

streamnative / pulsar-flink
data-processing,Elastic data processing with Apache Pulsar and Apache Flink
Organization: streamnative
apache-pulsar apache-flink data-processing batch-processing stream-processing flink flink-connector schema-registry schema sql

svenkreiss / pysparkling
data-processing,A pure Python implementation of Apache Spark's RDD and DStream interfaces.
User: svenkreiss
Home Page: https://pysparkling.readthedocs.io
data-science data-processing python apache-spark

tollwerk / data-processing-agreements
data-processing,Collection of Data Processing Agreement (DPA) and GDPR compliance resources
Organization: tollwerk
Home Page: https://tollwerk.github.io/data-processing-agreements/
gdpr dsgvo compliance data privacy dpa auftragsdatenverarbeitung adv ethics data-processing

tomwright / dasel
data-processing,Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
User: tomwright
Home Page: https://daseldocs.tomwright.me
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools

unionai-oss / pandera
data-processing,A light-weight, flexible, and expressive statistical data testing library
Organization: unionai-oss
Home Page: https://www.union.ai/pandera
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check

yord / pxi
data-processing,🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
User: yord
pxi pixie json csv ssv tsv dsv data-processing deserializer parser