Creating a new Dagster project#

The easiest way to start building a Dagster project is by using the dagster project CLI. This CLI tool helps generate files and folder structures that enable you to quickly get started with Dagster.

Step 1: Bootstrap a new project#

You can scaffold a new project using the default project skeleton, or start with one of the official Dagster examples.

To learn more about the default files in a Dagster project, refer to the Dagster project file reference.

pip install dagster
dagster project scaffold --name my-dagster-project

pip install dagster
dagster project from-example \
  --name my-dagster-project \
  --example quickstart_etl

Step 2: Install project dependencies#

The newly generated my-dagster-project directory is a fully functioning Python package and can be installed with pip.

To install it as a package and its Python dependencies, run:

pip install -e ".[dev]"

Step 3: Start the Dagster UI#

To start the Dagster UI, run:

dagster dev

Note: This command also starts the Dagster daemon. Refer to the Running Dagster locally guide for more info.

Use your browser to open http://localhost:3000 to view the project.

Step 4: Development#

• Adding new Python dependencies
• Environment variables and secrets
• Unit testing

Adding new Python dependencies#

You can specify new Python dependencies in setup.py.

Using environment variables and secrets#

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

Refer to the Using environment variables and secrets in Dagster code guide for more info and examples.

Adding and running unit tests#

Tests can be added in the my_dagster_project_tests directory and run using pytest:

pytest my_dagster_project_tests

Next steps#

Once your project is ready to move to production, check out our recommendations for transitioning data pipelines from development to production.

Check out the following resources to learn more about deployment options:

• Dagster+ - Deploy using Dagster-managed infrastructure
• Your own infrastructure - Deploy to your infrastructure, such as Docker, Kubernetes, Amazon Web Services, etc.

Related#