New features and improvements to Pipestack.
Off of last week's housekeeping, security, and performance testing, I tested a few theories to improve overall pipeline performance. The most promising is to migrate to WASI I/O streams. I spent some time this week building a successful prototype. It requires a bit of rewiring, but luckily nothing major thanks to how composable wasmCloud applications are.
CHANGELOG.md files.I want to spend a week every six (or so) weeks to clean things up and this week was the first such housekeeping week. Tech debt is real and kills velocity, so I actively counter that with regular cleanup work.
Address TODO comments, update project dependencies, refactor LLM-generated code. I also spent a few hours trying to break the system, both from a security and load test perspective. That led to a few fixes that could have been costly... :-).
What a week... I onboarded the first (besides Pipestack itself) customer! It's an agricultural company that processes sensor data of their customers' machines.
It's a proof of concept project that runs in parallel to their existing pipelines. We processed data on day two (their IT team had to approve Pipestack's security). By the end of the week, I got to see the impact Pipestack has and it was beyond impressive 🤯. I knew Wasm Components were lightweight and extremely resource efficient, but now I have actual data. Think 80%+ improvements across CPU, memory, performance, and costs!
Pipestack leverages wasmCloud, which is built on top of NATS.
The entire Pipestack infrastructure runs in a private network with no external incoming network access other than to the web app. This will change in the future when I introduce leaf nodes, a feature that lets you run individual nodes in your own infrastructure!
In preparation for that, I spent the week digging deep into NATS security. The end result: A secured NATS server with all communication now being encrypted.
Building your pipelines with the drag & drop builder is a great way to get started. Looking at the underlying YAML config in the Config view has been fun, but not very helpful :-). That's why I enabled YAML editing, with a few guards to avoid disasters.
This builds the foundation for much more useful features down the road, namely sharing pipelines with copy & paste or direct links to a builder to get started with templates.
pipeline_manager service. I initially had a LLM write most of the code, but it became a maintenance nightmare. No more!Observability! What I see is what you see. This week's focus was on exposing all logs related to your nodes in the web app's Observability view. There is more work needed to expose logs related to your pipeline overall and your execution environments. The goal is to expose anything I see in my OpenTelemetry dashboards to you.
This is it, Pipestack's initial release 🎉! Don't let that fool you though, here's what I shipped.
infra_manager to, you guessed it, manage infrastructure for individual workspaces created in Pipestackpipeline_manager used by the web app, future CLI, and a public API to manage pipelinesGET or more likely POST request to kick off a pipeline