New features and improvements to Pipestack.
End-to-end JSON support has been behind a feature flag for a few weeks. This week, I polished the user experience and removed the feature flag. You can now send JSON to a HTTP node, process it in your Wasm component and send a modified JSON payload to a HTTP endpoint of your choice.
Granular log level filters (video) and a shiny new date range selector (video) for logs 😊!
A customer asked to access their logs programmatically. Up until last week, I had an Open Telemetry Collector and self-hosted Loki, Tempo, Prometheus, and Grafana instances. This was all secured in a private network with no access to the public domain.
I had two options to approach the customer's request:
I opted to outsource it to HyperDX, which was acquired by ClickHouse. So after reading up on all that, I decided to ask for private access to the ClickStack product and migrated over to that. It was as easy as updating my Open Telemetry Collector config and changing my code to fetch logs from the ClickHouse database directly rather than the self-hosted Loki instance.
It's been six weeks since the last housekeeping, so I spent this week getting my cleaning equipment and handyman tools out of the storage to go and clean up the code.
The result:
An ever-growing platform also leads to more potential for unexpected regression bugs. I spent this week catching up on writing tests, mainly end-to-end tests as they bring the most value.
AI was a great companion to get this done with the least amount of effort. A bit of planning, then a detailed prompt and it did it's work while I planned upcoming features and provided a bit of support to the existing customers.
With two customers, there is very little needed from me in terms of support. However, I expect this to change as more customers onboard and want to be proactive.
From the beginning, I made sure to expose logs for individual nodes. This already paid off big times! I spent this week working on improving how the logs are exposed to avoid hitting the log backend unnecessarily.
Unexpected, but exciting! Customer #2 onboarded this week. They set up Pipestack to process application events such as when a new user signs up or churns, when users take certain actions, etc.
I spent the week helping them onboard and added a new feature to the logs view to filter by log level.
Another week, another opportunity to improve NATS security 😅. A few weeks ago, I put in place the first building block and this week, I built on top of that. With this week's work, each Pipestack customer gets their own NATS account. This means all communication specific to a customer is isolated from any other customer.
This is also an important piece to the puzzle of leaf nodes. Customers will be able to create their own NATS users based on the NATS account they own. Decentralized JWT authentication at work 🥰.
Off of last week's housekeeping, security, and performance testing, I tested a few theories to improve overall pipeline performance. The most promising is to migrate to WASI I/O streams. I spent some time this week building a successful prototype. It requires a bit of rewiring, but luckily nothing major thanks to how composable wasmCloud applications are.
CHANGELOG.md files.I want to spend a week every six (or so) weeks to clean things up and this week was the first such housekeeping week. Tech debt is real and kills velocity, so I actively counter that with regular cleanup work.
Address TODO comments, update project dependencies, refactor LLM-generated code. I also spent a few hours trying to break the system, both from a security and load test perspective. That led to a few fixes that could have been costly... :-).
What a week... I onboarded the first (besides Pipestack itself) customer! It's an agricultural company that processes sensor data of their customers' machines.
It's a proof of concept project that runs in parallel to their existing pipelines. We processed data on day two (their IT team had to approve Pipestack's security). By the end of the week, I got to see the impact Pipestack has and it was beyond impressive 🤯. I knew Wasm Components were lightweight and extremely resource efficient, but now I have actual data. Think 80%+ improvements across CPU, memory, performance, and costs!
Pipestack leverages wasmCloud, which is built on top of NATS.
The entire Pipestack infrastructure runs in a private network with no external incoming network access other than to the web app. This will change in the future when I introduce leaf nodes, a feature that lets you run individual nodes in your own infrastructure!
In preparation for that, I spent the week digging deep into NATS security. The end result: A secured NATS server with all communication now being encrypted.
Building your pipelines with the drag & drop builder is a great way to get started. Looking at the underlying YAML config in the Config view has been fun, but not very helpful :-). That's why I enabled YAML editing, with a few guards to avoid disasters.
This builds the foundation for much more useful features down the road, namely sharing pipelines with copy & paste or direct links to a builder to get started with templates.
pipeline_manager service. I initially had a LLM write most of the code, but it became a maintenance nightmare. No more!Observability! What I see is what you see. This week's focus was on exposing all logs related to your nodes in the web app's Observability view. There is more work needed to expose logs related to your pipeline overall and your execution environments. The goal is to expose anything I see in my OpenTelemetry dashboards to you.
This is it, Pipestack's initial release 🎉! Don't let that fool you though, here's what I shipped.
infra_manager to, you guessed it, manage infrastructure for individual workspaces created in Pipestackpipeline_manager used by the web app, future CLI, and a public API to manage pipelinesGET or more likely POST request to kick off a pipeline