Exchange project · JKU Linz
Automated Data Pipeline
The infrastructure side of shipping: a data pipeline that provisions and validates itself, so the apps on top just deploy.
cloud-computing-project · main
How it's built
- Prefect flows orchestrate the pipeline on a self-hosted cluster on GKE.
- Terraform provisions the GCP side — cluster, buckets, service accounts — reproducible from zero.
- GitHub Actions validates Kubernetes manifests on every push, before anything can merge.
- Environments and secrets separated per stage; no manual kubectl in the loop.
- Built during my exchange semester at JKU Linz — NYC taxi data flowing from GCS through BigQuery into Looker dashboards.