Blog

All sorts of updates related to Pipestack.

← All posts

It Works! It's Live!

tl;dr A new kind of pipeline builder that combines drag-and-drop UI with language-agnostic Wasm components. Pipestack is live!

Every business ends up building some kind of glue code, workflows, event triggers, and/or data sync scripts. Eventually, hosted tools like Zapier or n8n are being considered, but these tools can be either too limiting or too complex.

A web interface, a few clicks, and you have a pipeline! Then reality hits and things either become hard to manage or UI-only tools fall short in what you can do. n8n comes closest to bridging the gap, with support for custom code, but only in JavaScript or Python.

Meet Pipestack

Ideally, you get the best of both worlds: A UI with a drag & drop interface to build your pipelines and a way to provide your own business logic, in your language of choice.

👆 This is what Pipestack provides!

I recently read Why the Component Model?. It was a short, dense page and the content hit me hard.

To process it all, I went for a walk. And I had questions:

  • Bring your own code? In your language of choice?
  • A secure, sandboxed environment for each component?
  • We're talking kilobytes per component vs. megabytes or gigabytes for Docker containers?
  • But how do you orchestrate and manage these components?
  • What about versioning and rollbacks?

I needed answers for the last two questions! Luckily, all it took to find them was a bench to sit down, a smartphone, and access to the internet.

Voilà, there it was: wasmCloud. Their tagline couldn't be more aligned with what I needed: "Wasm-native Orchestration".

Pipelines re-imagined

Wasm Components are lightweight, fast, sandboxed, and can be written in C/C++, C#, Go, JavaScript, Python, Rust. With more language support in the works. This means your team can leverage their existing skills, build systems, and experience.

After a day of reading docs and experimenting, I put together the following architecture diagram. It illustrates what a pipeline looks like at a high level as well as what individual steps (now called nodes) consist of.

There are three things you need to build a pipeline:

  • One or more source (node-in components in the diagram)
  • One or more processor to apply business logic (node-process components in the diagram)
  • One or more sink (node-out components in the diagram)

Sources

A source is a place where data for your pipeline comes from. This could be an event when a new customer signs up for your product, a change in a database, an email you received, anything.

Ideally, you don't need to write any code to configure sources. Of course, you can, if you have unique requirements.

Processors

A processor is where your code comes to life. This is where your business logic lives. You define what happens, whether it's sending an email, updating a CRM, or triggering another workflow.

You write processors in any language that compiles to Wasm Components and simply upload it to a node within your pipeline.

Sinks

A sink is a place to store the output of your processors. Given the example of a new customer signup, a sink may be your email provider to send a welcome email. Another sink may be your CRM's API. Or a Slack webhook to celebrate the win!

What's Next?

Pipestack is currently a week old and in early development, but the foundation is solid and the vision is clear. Here's what you can expect in the coming months:

Early Access

You can sign up and use Pipestack today! Before you ask, YES, Pipestack is built with Pipestack.

Early adopters get:

  • Access to the web-based pipeline builder and real-time observability
  • A HTTP source (GET or POST request to kick off your pipeline)
  • A HTTP webhook as a sink
  • Documentation and templates for building your own processor components
  • Direct feedback channel to shape the product roadmap

Component Marketplace

One of the most exciting aspects of the WebAssembly Component Model is reusability. I envision a marketplace where the community can share:

  • Common business logic processors (data transformations, validations, enrichment)
  • Connectors for popular services (Stripe, Slack, GitHub, etc.)
  • Industry-specific components (e-commerce, fintech, healthcare)

In fact, any source, processor, or sink I develop as part of Pipestack will be open source and form the foundation of that marketplace.

Get Involved

Interested in being part of the journey? Here's how you can stay connected:

  • Follow the development: Changelog
  • Join the community: Discord server for discussions and updates
  • Read the newsletter:

The future of data pipelines is component-based, language-agnostic, and developer-friendly. Let's build it together! 🚀