Dagster+ Hybrid Architecture (evaluated, not adopted)

Not the current architecture

Dagster+ (Cloud) was evaluated but never adopted for production. Everything below describes that evaluation, not what’s actually deployed — real production is self-hosted OSS on GCP VMs (erp-{staging,prod}-dagster), with the code server, webserver, and daemon all running 24/7, same as the “Before” diagram below. See apps/dagster/README.md and infrastructure-gcp/ansible/playbooks/dagster.yml for the real setup. Kept here as a record of what was considered and why it wasn’t the direction taken — not as a migration guide.

What Would Have Changed: Self-Hosted → Dagster+ Hybrid

Before (Self-Hosted)

┌─────────────────────────────────────────────────────────┐
│              Your Infrastructure                        │
│                                                         │
│  ┌──────────────┐        ┌──────────────────────┐    │
│  │ dagster-web  │───────▶│ dagster-code-server  │    │
│  │  (Port 3000) │  gRPC  │     (Port 4000)      │    │
│  └──────────────┘        │   Runs CONTINUOUSLY  │    │
│                          │   Serves code via    │    │
│  ┌──────────────┐        │   gRPC               │    │
│  │dagster-daemon│────────┘                      │    │
│  │  (Scheduler) │                                │    │
│  └──────────────┘                                │    │
│                          ┌──────────────────────┐    │
│                          │   apps/dagster/       │    │
│                          │   (Your code)        │    │
│                          │   Loaded by code      │    │
│                          │   server              │    │
│                          └──────────────────────┘    │
└─────────────────────────────────────────────────────────┘

Components:

• dagster-code-server: Runs 24/7, serves your code via gRPC
• dagster-web: Runs 24/7, connects to code server, serves UI
• dagster-daemon: Runs 24/7, connects to code server, runs schedules
• apps/dagster/: Your pipeline code, loaded by code server

---

After (Dagster+ Hybrid)

┌─────────────────────────────────────────────────────────┐
│              Dagster+ Cloud                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │   Webserver  │  │    Daemon    │  │   Metadata   │ │
│  │     (UI)     │  │  (Scheduler) │  │   Storage    │ │
│  └──────────────┘  └──────────────┘  └──────────────┘ │
│                          ▲                              │
└──────────────────────────│──────────────────────────────┘
                           │ gRPC/HTTPS
┌──────────────────────────▼──────────────────────────────┐
│              Your Infrastructure                        │
│                                                         │
│  ┌──────────────────────────────────────────────────┐  │
│  │         dagster-agent                           │  │
│  │         Runs CONTINUOUSLY                       │  │
│  │         Connects to Dagster+                    │  │
│  │         Waits for job requests                  │  │
│  └──────────────────────────────────────────────────┘  │
│                          │                              │
│                          │ When job runs:               │
│                          ▼                              │
│  ┌──────────────────────────────────────────────────┐  │
│  │    User Code Container (ON-DEMAND)               │  │
│  │    - Pulls image: ghcr.io/.../dagster:tag       │  │
│  │    - Contains: apps/dagster/ (built in)        │  │
│  │    - Runs job                                    │  │
│  │    - Stops when done                            │  │
│  └──────────────────────────────────────────────────┘  │
│                                                         │
│  ┌──────────────────────────────────────────────────┐  │
│  │    Another Job Container (if concurrent)        │  │
│  │    (Same image, separate container)              │  │
│  └──────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

Components:

• dagster-agent: Runs 24/7, connects to Dagster+ cloud, launches containers
• User Code Containers: Launched ON-DEMAND when jobs run, stop when done
• apps/dagster/: Still your code, but now built into Docker image

---

Key Differences

Aspect	Self-Hosted	Dagster+ Hybrid
Code Server	✅ Runs 24/7 (port 4000)	❌ Not needed
Webserver	✅ Runs 24/7 (port 3000)	❌ Dagster+ manages
Daemon	✅ Runs 24/7	❌ Dagster+ manages
Agent	❌ Not needed	✅ Runs 24/7
Code Location	Loaded by code server	Built into Docker image
Job Execution	Daemon spawns processes	Agent launches containers
Metadata Storage	Your PostgreSQL	Dagster+ manages

---

What Role Does apps/dagster/ Play Now?

apps/dagster/ is still critical! It contains:

1. Pipeline Definitions (erp_pipeline/definitions.py)

   • Your jobs, assets, sensors, schedules
   • What gets executed

2. DLT Pipelines (erp_pipeline/dlt/)

   • Data extraction logic
   • Transformers

3. dbt Models (dbt/)

   • SQL transformations

4. Configuration (dagster_cloud.yaml)

   • Tells Dagster+ where to find your code

The difference: Instead of being loaded by a code server, it’s now:

1. Built into a Docker image during CI/CD
2. Pushed to GHCR (ghcr.io/ERP-Unlocked/ordermatic/dagster:tag)
3. Pulled by agent when jobs run
4. Executed in containers that start/stop per job

---

How It Works Now

1. Code Changes

# You edit apps/dagster/erp_pipeline/definitions.py
git push

2. CI/CD Builds Image

# GitHub Actions builds Docker image
docker build -t ghcr.io/.../dagster:v1.2.3 apps/dagster/
docker push ghcr.io/.../dagster:v1.2.3

3. CI/CD Notifies Dagster+

# GitHub Actions tells Dagster+ about new image
dagster-cloud serverless deploy-python-executable \
  --image ghcr.io/.../dagster:v1.2.3

4. Agent Pulls & Runs

1. Dagster+ schedules a job (via daemon)
2. Dagster+ tells agent: "Run job X"
3. Agent pulls image: ghcr.io/.../dagster:v1.2.3
4. Agent launches container with your code
5. Container runs job (executes apps/dagster/ code)
6. Container stops when done

---

What You Still Need

✅ Keep These

• apps/dagster/ - Your pipeline code (still critical!)
• Dockerfile - Builds your code into image
• CI/CD - Builds and pushes images
• Environment variables - Passed to containers

❌ No Longer Need

• dagster-code-server - Agent launches containers instead
• dagster-web - Dagster+ provides UI
• dagster-daemon - Dagster+ manages scheduling
• Dagster PostgreSQL - Dagster+ manages metadata
• workspace.yaml - Not used (code is in image)

---

Benefits

1. Simpler Infrastructure: One agent vs three services
2. Better Isolation: Each job runs in its own container
3. Automatic Scaling: Containers start/stop as needed
4. No Code Server Crashes: Code issues don’t affect agent
5. Managed UI/Daemon: Dagster+ handles reliability

---

Summary

apps/dagster/ is still your code - it just gets packaged differently:

• Before: Code server loads it from filesystem
• After: Built into Docker image, launched in containers

No code server needed because:

• Agent pulls pre-built images
• Launches containers on-demand
• Containers contain your code already

You’re essentially moving from “load code at runtime” to “package code at build time” - much more reliable!