Data on Matt Ellis

A proper desk to work at

Mon, 27 Feb 2023 10:17:53 +0000

So my supposed daily ramble has been more of a ramble when you can…

Took some time off to go skiing. I came back in one piece which is surprising 🎉 Finally got my desk partially installed. It’s a solid countertop from wall to wall, super heavy and needed 6 people to lift! At somepoint I’ll stain it and setup my monitors.

Catching back up today on previous dagster work. It’s apparently crunch time.
📚 Also had chance to read some more of “Designing data intensive applications”
- Great book for understanding data engineering concepts. Helped me understand what my esteemed colleagues are talking about!

Data pipelines with dagster

Thu, 16 Feb 2023 10:46:34 +0000

Who knew daily updates requires effort and discipline!

Anyway today was a good day, managed to get something working regarding publishing events to SNS.

Dagster cloud environment variables are kind of whack! You can only really drive them manually through the cloud UI.
- (I later found you can use graphql, still not the best though.)
Lots of stuff to crack on with, lots of documentation to write up.

Sidebar, it’s really hard to not work from a proper desk 🖥️

Digging around dagster

Mon, 06 Feb 2023 11:46:56 +0000

Playing around with the dagster local development and trying to build a nice DAG.

Dagster has a really nice interface, although some of the naming conventions are kind of cryptic
It has a I/O manager for snowflake (Nice!)
Setting up and getting started with dagster is super simple.

I might dig out the raspberry pi this evening and try get dagster running on it for one of my old projects.