brenofariasdasilva/dagster-portuguese-grades

Dagster Portuguese Grades Model.

Dagster - Portuguese Grades Model.

Table of Contents

• Dagster - Portuguese Grades Model.
  • Table of Contents
  • Useful documentation:
• Clone the project:
• Installation:
• How the project was created:
• Execute:
• Open:
• File Hierarchy:
• Important Notes:

Useful documentation:

• Dagster Full Documentation
• Dagster Tutorial
• API Docs
• Create Assets
• Dagstermill Asset
• Definitions
• Auto-materialize Assets
• Integrate Python Jupyter Notebook with Dagster
• Loggers
• Migration from @solid and @pipeline to @Ops and @Jobs

Clone the project:

Simply download or clone this repository and open a terminal inside of the folder that is related to it.

Installation:

pip install -e ".[dev]"
pip install dagster dagit notebook

or

make setup

How the project was created:

dagster project scaffold --name portuguese_grades
cd portuguese_grades

Execute:

make locally

Open:

Open the following link in your browser in order to see the dagit UI:

http://127.0.0.1:3000

File Hierarchy:

• Assets Files/Directory -> Assets are data artifacts that are produced and consumed by pipelines. They can be anything from raw data files (such as CSV or JSON) to processed data sets or model outputs. An asset definition typically includes the asset's name, the type of the asset (e.g. a file, a database table), and any associated metadata or configuration options. Assets can also specify dependencies on other assets, so that a pipeline will only run when all of its dependencies are available.
• Jobs -> Jobs are used to define the scheduling and execution of pipelines. They are defined using a Python script with the .py extension, and are typically stored in the jobs/ directory of a Dagster project. A job definition typically includes the name of the job, the schedule on which the job should be run (e.g. daily, weekly), and the pipeline(s) that should be executed by the job. Jobs can also specify environment variables and other configuration options that are passed to the pipeline(s) when they are executed.
• workspace.yaml -> This is a configuration file that defines the Dagster instance used in the tutorial notebooks. It specifies the location of the Dagster instance, which is important for connecting to the Dagster API.
• setup.py -> This is a Python script that contains the configuration information for building and distributing a Python package. It typically includes information such as the package name, version, author, and dependencies. It can also specify scripts to be installed with the package, and other package-specific options.
• setup.cfg -> This is a configuration file that is used to specify additional options for setuptools, the package that is commonly used to build and distribute Python packages. It can include options such as the package's entry points, the location of the package's data files, and various build and distribution options.
• pyproject.toml -> This is a configuration file that is used by modern Python build tools such as poetry and flit. It specifies the project's dependencies, build tool configuration, and metadata. It can also include options for building and packaging the project.

Important Notes:

• Each file has (or should have) comments explaining what it does, the same should be applied to functions. So, if you have any doubts, please, read the comments or contact me.
• Dagster is a project that is under development, so, some things may change in the future. From my experience, the documentation is good and there are making weekly updates. So, if you have any doubts related to error importing modules, please, check the releases and CTRL+F the module that you are trying to import in order to see if it was removed or renamed.
• Also check you Dagster installation version by running one of the following commands and compare it with the latest release number:

   dagster --version
   # OR
   make version

• If you are using a version that is not the latest, consider updating it by running one of the following commands, but keep in mind that it may break your code:

   pip install dagster dagit --upgrade
   # OR
   make upgrade