pipelines

Who knew daily updates requires effort and discipline! Anyway today was a good day, managed to get something working regarding publishing events to sns. Dagster cloud environment variables are kind of whack! You can only really drive them manually through the cloud UI. (I later found you can use graphql, still not the best though.) Lots of stuff to crack on with, lots of documentation to write up. Sidebar, it’s really hard to not work from a proper desk 🖥️

Playing around with the dagster local development and trying to build a nice DAG. Dagster has a really nice interface, although some of the naming conventions are kind of cryptic It has a I/O manager for snowflake (Nice!) Setting up and getting started with dagster is super simple. I might dig out the raspberry pi this evening and try get dagster running on it for one of my old projects.