https://www.youtube.com/watch?v=PAAvNmoqDq0
Welcome to the Sulfur project, where we harness the elemental power of data transformation. Just like sulfur can reshape its form, our platform reshapes, processes, and transforms data, turning it into valuable insights.
Join us on this journey of alchemy where data turns into gold through customization and innovation. Unleash the potential of Sulfur and turn raw data into refined intelligence.
-
Clone the repository:
git clone https://github.com/emreyalvac/sulfur.git
-
Navigate to the project directory:
cd sulfur
-
Install dependencies:
cargo build
-
Configure
config.yml
for your pipelines. -
To run pipelines, use the terminal:
cargo run -- --config config.yml
Here's an example of how to configure your pipelines using the config.yml
file:
sulfur:
- name: "Mongo to Redis"
cron: "0 0 * * *"
source:
type: "Mongo"
host: "example.com"
port: "5432"
user: "user"
password: "password"
database: "db_name"
collection: "collection_name"
destination:
type: "Redis"
host: "redis.example.com"
port: "6379"
password: "redis_password"
key: "data_key"
- Each pipeline is defined under the
sulfur
key. name
: A descriptive name for your pipeline.cron
(optional): The cron expression to schedule pipeline runs (e.g., "0 0 * * *" for daily runs).source
: Specifies the data source configuration.destination
: Specifies the data destination configuration.
For the source
and destination
configurations, you can specify various parameters based on the type of engine you're
using (e.g., "Database," "Redis," "BigQuery," etc.).
Remember to customize the configuration according to your project's specific requirements.
Sulfur is more than a project; it's a catalyst for data alchemy. Whether you're merging, filtering, or aggregating, Sulfur empowers you to sculpt raw data into refined insights, making your data truly valuable.
Sulfur currently supports the following data storage engines:
Engine | Type | Description | Parameters |
---|---|---|---|
ElasticSearch | Database | Use Elasticsearch as a data source or destination. | host, port, user, password, index |
MongoDB | Database | Use MongoDB as a data source or destination. | host, port, user, password, database, collection |
Redis | In-Memory Data Store | Use Redis as a data source or destination. | host, port, password, key |
BigQuery | Data Warehouse | Use Google BigQuery as a data destination. | project_id, dataset_id, table_id, credentials (service_key.json) |
We're committed to expanding the list of supported engines to give you even more flexibility. Adding new platforms is a straightforward process, allowing you to tailor Sulfur to your evolving data needs.
At Sulfur, we're committed to expanding the range of supported storage engines to cater to your evolving needs. Here's a sneak peek at some potential storage engines that might be added in the future:
- Amazon S3
- Microsoft Azure Blob Storage
- PostgreSQL
- MySQL
- SQLite
- Cassandra
- Apache Hadoop HDFS
- Amazon Redshift
- Snowflake
- Apache Kafka
- Oracle Database
- IBM Db2
- Microsoft SQL Server
- Apache Hive
- MongoDB Atlas
- Elasticsearch Service
- Redis Cloud
- Memcached
- InfluxDB
- Kafka
- RabbitMQ
Stay tuned as we continue to explore and add more storage engine options to the Sulfur platform. We're excited to provide you with a broader range of choices for your data storage needs!
At Sulfur, we're dedicated to evolving our platform to meet your needs. We're excited to introduce a feature:
The Advanced Data Transformation feature will provide a powerful toolkit for crafting precise data transformations. From mathematical operations to conditional logic, this feature grants you unparalleled control over your data.
Stay tuned as we work diligently to unveil this enhancement. Your data transformation possibilities are about to expand like never before!
Sulfur enables you to harness the power of custom Python scripts for advanced data transformation during the pipeline process. By integrating Python scripts, you can perform complex data manipulations, calculations, and enrichments before the data is forwarded to its destination.
To demonstrate the power of custom Python scripts for data transformation, we've provided an example advanced_transform
function that showcases a basic transformation:
import json
def advanced_transform(*args):
# Unpack the arguments tuple
data_string = args[0]
# Load the JSON data
data = json.loads(data_string)
# Perform your advanced transformation here
transformed_data = {
"name": "TRANSFORMED",
"original_data": data
}
# Convert the transformed data back to a JSON string
transformed_json = json.dumps(transformed_data)
return transformed_json
sulfur:
- name: "Pipeline1"
cron: "0 0 * * *"
transform:
file: './transform.py'
fn: 'advanced_transform'
source:
type: "MongoDB"
host: "mongodb.example.com"
port: "27017"
user: "user"
password: "password"
database: "db_name"
collection: "collection_name"
destination:
type: "Redis"
host: "redis.example.com"
port: "6379"
password: "redis_password"
key: "data_key"
See Contributing Guidelines for details on how to contribute to this project.
This project is licensed under the MIT License. See the LICENSE file for details.