https://matt-ellis.dev/tags/pipelines/Recent content in Pipelines on Matt EllisHugoen-GBThu, 16 Feb 2023 10:46:34 +0000https://matt-ellis.dev/rambles/data-pipelines-with-dagster/Thu, 16 Feb 2023 10:46:34 +0000https://matt-ellis.dev/rambles/data-pipelines-with-dagster/<p>Who knew daily updates requires effort and discipline!</p> <p>Anyway today was a good day, managed to get something working regarding publishing events to SNS.</p> <ul> <li>Dagster cloud environment variables are kind of whack! You can only really drive them manually through the cloud UI. <ul> <li>(I later found you can use graphql, still not the best though.)</li> </ul> </li> <li>Lots of stuff to crack on with, lots of documentation to write up.</li> </ul> <p>Sidebar, it&rsquo;s really hard to not work from a proper desk 🖥️</p>https://matt-ellis.dev/rambles/digging-around-dagster/Mon, 06 Feb 2023 11:46:56 +0000https://matt-ellis.dev/rambles/digging-around-dagster/<p>Playing around with the dagster local development and trying to build a nice DAG.</p> <ul> <li>Dagster has a really nice interface, although some of the naming conventions are kind of cryptic</li> <li>It has a I/O manager for snowflake (Nice!)</li> <li>Setting up and getting started with dagster is super simple.</li> </ul> <p>I might dig out the raspberry pi this evening and try get dagster running on it for one of my old projects.</p>