From my experience using AI heavily: relying on that internal list for a real workstream gets hard to manage and track. An external todo system is often easier to govern. The agent has also tended to do better when work is expressed as issues on the repo you’re in - plan the product as a project on a board, put goals and breakdown there, and let the agent pick cards instead of inventing a private backlog every session.

TL;DR: This is an observation, not a rule. Harness todos still help for short in-session steps. For longer arcs I’ve gotten more stable results when planning and ownership live outside the chat - preferably on the forge you already use for code.

What I’ve been doing

Create a project board for what you’re shipping (org-level if several repos share one product). Write the summary once: problem, goals, non-goals, constraints, what “done” means. Break that into issues with enough acceptance criteria that something can actually finish. Point the agent at the board - next Ready item, implement #142, triage bugs labeled for it. Land via PR. Reviews, CI failures, and product follow-ups go back to issues so the next turn (same agent or another) still has a durable queue.

Other project tools work - Linear, Jira, Notion, whatever your team already has. What I’ve noticed with native issues + projects next to the code is less glue: stable IDs (#142 survives chats), PRs and CI sit beside the task, humans and agents share one surface, and you spend fewer tokens re-explaining state every session. You’ve moved planning and management of what you’re building onto a board both of you can track, correlate, and pull context from.

Harness todos still have a job. Mid-issue micro-steps (“run tests,” “update the readme”) are fine as ephemeral checkboxes. They die with the session; the issue doesn’t. For a twenty-minute drive that never needs a second agent or a tomorrow-morning resume, the built-in list is often enough and a board is overhead.

This can be countered. If your work is small, solo, and finishes in one chat, external process is ceremony. If the board is theater - everything In progress, novel-length issues, no review - you just moved the mess. Some people keep a clean harness workflow and never miss a forge board. Fair. I’m not arguing universality. I’m saying for multi-session, multi-PR work, the durable queue has been the part that held up for me - and reviews still matter; the issues page is just a convenient attention surface when something needs a human or a second pass.

The agent still lives in the terminal (ideally a sandbox - How to Safely Give AI Agents a Terminal). The board is optional infrastructure for remembering what “done” was supposed to mean after the chat is gone.

Most setups still do the dangerous thing by default: run model-generated commands on a machine you care about.

Laptop. Dev VM. Shared CI runner. Sometimes worse.

I hit this while building and testing CLIs and agents. The moment you let a model shell out - npm install, curl | bash, “fix the Dockerfile,” “explore the filesystem” - you’re no longer doing chat. You’re doing untrusted remote code execution with a friendly UI.

TL;DR: Treat agent execution as an isolation problem, not a prompt problem. Prefer disposable Linux terminals that are network-isolated, resource-capped, and sandboxed with gVisor (runsc). Outbound-only BYOS access is a separate path when you need real hardware. That model is what I designed into Rexec.

Related background: Rexec as a terminal control room.

The wrong default

Teams adopt agent tooling like this:

1. Install agent CLI on a developer machine or bastion.
2. Point it at a repo.
3. Grant shell / network / package install.
4. Hope system prompts and “approve tool use” are enough.

Hope is not a control.

Models are good at sounding careful and still doing something catastrophic:

• Writing secrets into world-readable files
• rm -rf with a creative path expansion
• Exfiltrating env vars over DNS or HTTPS
• Installing packages that phone home
• Touching production kubeconfigs sitting in ~/.kube

Even without malice, agents are sloppy operators. They thrash package managers, leave half-broken state, and burn hours on “works in the agent’s world / broken in yours.”

If the blast radius is your primary workstation, every session is a production incident waiting for a bad completion.

What “sandbox the agent” actually has to mean

A sandbox for interactive humans and a sandbox for agents share plumbing, but agents change the threat model:

So the requirements I care about:

1. Disposable by default - create, run, destroy. No “that container from last Tuesday.”
2. Hard resource bounds - CPU, memory, PIDs; disk when the host can enforce it.
3. Network isolation as a first-class switch - not “we’ll add NetworkPolicy later.”
4. API/headless entry - agents shouldn’t need a human clicking xterm for every step.
5. Auditability - session recording / logs when you’re debugging “what did it do?”
6. No inbound SSH theater - prefer outbound tunnels when attaching real machines.

This is platform engineering, not vibes: terminals become infrastructure primitives with create/delete semantics, quotas, and a control plane.

How the isolation model works

I implemented this as two primitives in Rexec. You can copy the shape even if you never run our code.

Primitive 1: Cloud terminals as sandboxed containers

Each session is a Linux environment backed by Docker/Podman, with gVisor in the isolation path:

• Hard CPU / memory / PID limits
• Optional disk quotas when the host supports them
• Dropped capabilities + no-new-privileges
• Attachment to an isolated bridge (rexec-isolated) with inter-container communication disabled
• OCI runtime runsc (gVisor) so guest syscalls hit a user-space kernel, not the full host surface by default

Rexec wires this as a first-class isolation choice (OCI_RUNTIME=runsc), not a footnote. Don’t leave untrusted code on stock runc and call it a day. More product context: Rexec control-room post.

Conceptually:

Agent / CLI / UI
 │ (API or WebSocket)
 ▼
 Control plane ──create/exec/delete──► Container runtime
 │
 ├─ cgroup limits
 ├─ dropped caps
 ├─ isolated network bridge
 └─ gVisor (runsc) application kernel

The important product decision: the environment outlives a single WebSocket flap, but not your interest in it. Interactive UX can reattach via tmux/exec; the security unit is still “this sandbox, this network, these limits, this runtime.”

Primitive 2: BYOS agents (outbound only)

Sometimes the agent needs your GPU box or a lab server - not a fresh Ubuntu container.

The pattern I use: an agent process on the machine opens an outbound WebSocket to the control plane. No inbound SSH, no “open 22 to the world,” no VPN spaghetti for a demo.

Tradeoff, said plainly:

• Cloud container: strong isolation, weak access to “my weird hardware”
• BYOS: strong access to real iron, weaker isolation (it’s still that machine)

Don’t pretend BYOS is a jail. It’s mediated access with auth, session control, and audit - not a microVM escape boundary.

For untrusted agent code, prefer the container primitive. Use BYOS when the job requires the metal, and treat permissions like production access.

How agents actually use it

Once terminals are API-managed, agents stop needing to live on laptops.

Typical flow:

1. Create a sandbox (image + resource class + network mode).
2. Inject only the secrets that job needs (short-lived tokens, not your whole .env).
3. Run the agent headlessly against that sandbox.
4. Stream logs / attach if a human must intervene.
5. Delete the sandbox. Assume disk and memory are gone.

Self-host sketch (from the public Rexec path):

git clone https://github.com/PipeOpsHQ/Rexec.git
cd Rexec/docker
docker compose up --build
# UI/API on localhost:8080 - change default admin credentials immediately

Docs and deeper product context: rexec.sh/docs · source

I’m not going to pretend “one compose file” is a complete multi-tenant security program. It’s the right unit of isolation to build on.

Design choices and failure modes

Isolation is layered, not absolute

gVisor is real isolation: a smaller host-kernel attack surface than plain runc. It is still not a full hypervisor boundary.

Ladder we actually use:

1. cgroup + caps + network isolate - baseline hygiene
2. gVisor (runsc) - default stronger sandbox for Rexec cloud terminals
3. MicroVMs (Firecracker et al.) - next step when the threat model demands it (firecracker-shim-shaped work)
4. Dedicated nodes / accounts - compliance and economics, not cosplay

Pitfall: marketing “secure sandboxes” while sharing a Docker socket with the world - or advertising gVisor while still spawning on runc. If the control plane can spawn privileged containers, you’ve moved the castle gate, not closed it.

Network policy is part of the product

Default-deny between sandboxes should be boring and on.

Decide explicitly:

• Full outbound for package installs?
• Egress allowlists for registries only?
• No egress, vendored dependencies only?

Agents without egress are safer and more annoying. Agents with open egress are convenient and leaky. Pick per workload; don’t leave the default implicit.

Identity beats “shared admin shell”

Multi-user demos that all land in the same root@box teach the wrong lesson.

Sessions need:

• Authenticated users (or service accounts)
• Short-lived credentials for the sandbox
• Audit logs that answer “who/what/when”

Cost and chaos engineering for agents

Agents thrash. They’ll create five environments, install three toolchains, and OOM the host if you let them.

Quotas, concurrency caps, and aggressive TTLs aren’t “enterprise features.” They’re how you keep a sandbox fleet from becoming an expensive fork bomb with a language model at the wheel.

Practical checklist

If you’re wiring agents into your company this quarter:

1. Ban “agent has shell on my laptop” for anything that can touch secrets or prod.
2. Create/delete sandboxes per task (or per PR), not per quarter.
3. Run agent sandboxes on gVisor (runsc) - or stronger - not stock runc “because Docker default.”
4. Set egress policy deliberately; log outbound destinations if you can.
5. Cap CPU/memory/PIDs; kill zombies on a timer.
6. Record sessions for high-risk automation until you trust the loop.
7. Prefer outbound agents over inbound SSH when attaching real machines.
8. Assume breakout is possible; escalate to microVMs or dedicated nodes when the threat model says so.

Summary

Agent sandboxes fail when teams treat them as a prompt-engineering problem. They’re an isolation and lifecycle problem.

I built Rexec because I needed real multi-machine CLI testing, then watched agent workflows force the security model into the open. Disposable, network-isolated, gVisor-backed terminals with hard limits and API-driven create/delete are a better default than trusting the model on a precious machine. Copy the pattern even if you never run our compose file. You can wire the same idea with Kubernetes Jobs, RuntimeClass, Firecracker, or a cloud sandbox API, as long as local shell == trusted is off the table.

Related writing

• Rexec: The Terminal Control Room - how the product started
• Namespaces Aren’t Isolation - multi-tenant Kubernetes with gVisor, not folder labels
• firecracker-shim - when you want microVMs under Kubernetes pods
• Work catalogue - platforms and isolation work in one place

In 2014/2015 at OAU, I built a personal assistant that felt like science fiction on a student machine. It had a custom Iron Man-inspired UI. Separately, a “hey Jarvis” wake word started listening. Then it ran desktop actions: open an app, close an app, shut down the PC. It worked enough to feel real. It was also way ahead of what I could resource then: compute, models, speech stacks, distribution, time.

I didn’t have the runway to chase it properly. So it lived in the back of my head while I went deep on infrastructure, platforms, and production systems.

Here we are.

Igris is that dream with adult tools. Not a nostalgia rebuild of every pixel - a voice-only personal agent with a real architecture: a Go brain, thin Apple clients (iPhone, iPad, Mac), one WebSocket protocol, memory, tools, and a guardian presence you can feel when it’s listening vs thinking vs speaking. The product thesis is simple: you talk, it talks back; the screen is for captions and things that should be read, not a chat box pretending to be a conversation.

TL;DR: Igris is my long-cycle personal agent project. The OAU prototype is now a modern voice-native stack (Go brain, Apple clients) plus a software and 3D hardware plan: one will, many blades (phone, Mac, desk sentinels, field units).

What the 2014 version got right

Even with limited resources, the old design instincts were solid:

1. Voice first - if you have to type everything, it’s an app, not an assistant.
2. Actions, not answers - open, close, power state; agency over encyclopedia mode.
3. Presence UI - the orb / reactor aesthetic wasn’t decoration; it was state machine visualization before I had the vocabulary for that.
4. Local life integration - the PC was the body. The assistant lived where work happened.

What it lacked was everything the industry spent a decade building: reliable STT/TTS, long-context brains, tool protocols, mobile as a first-class body, and enough personal platform skill to ship a control plane instead of a demo script.

What Igris is now (software)

From the public tree (repo still uses some jarvis paths under the hood - the lineage is not subtle):

server/ Go: agent loop, memory, voice gateway - one binary
apple/ SwiftUI clients: iPhone, iPad, Mac menu bar

Brain: Go server. Default path Claude; any OpenAI-compatible mind via config (including local models). Ears/mouth: Deepgram live STT + VAD; speech out native Apple by default, or server TTS when you want it. Body: Apple-only thin clients on one full-duplex WebSocket. No web client by design. Tools: reminders, tasks, web search, local Mac integrations (calendar, mail, messages - with confirm-before-send for anything that leaves the machine), MCP servers, background coding jobs.

Architecture in one line: one protocol, many bodies. The phone is not a separate product from the Mac menu bar; both are skins over the same agent loop - the same “one will, many blades” idea the hardware plan extends.

Run the brain (dev shape):

cd server
export DEEPGRAM_API_KEY=...
export ANTHROPIC_API_KEY=sk-ant-...
go run ./cmd/jarvis
# health: curl localhost:8787/healthz

Point the Apple app at ws://…:8787/ws with a bearer token. Prefer Tailscale or private network over “exposed to the internet with a hope and a JWT.”

Source: github.com/9trocode/Igris

Face states (not wallpaper)

The face is a status display: slow pulse at rest, then clear stances for listening, assessing, speaking (with lip shapes), alert, dormant. The 2014 orb was the same idea; this version is a state machine you can read on the glass.

3D plan - guardian forms, not a speaker puck

Igris isn’t meant to live only as an icon on a phone. The design board locks four desk-scale directions - hardware with a face and a job:

Same idea as the software clients: one brain, many thin clients. The desk unit is another WebSocket body, not a separate product with its own half-broken firmware personality.

Field units - seven blades, one will

Beyond the desk, the ecosystem plan spreads the same guardian into the world: pocket, wrist, sight, home, wall watch, road, find.

• Talisman - palm summon, panic ping, location
• Vambrace - wrist vitals + silent guardian alert
• Helm - AR overlay / face match (quietly, only when you ask)
• Brazier - home shelf presence, glass-break / far-field mic
• Bastion - wall/ceiling eye that tracks motion
• Charger - in-cabin escort on the road
• Seal - coin-sized find / precision ping

Every device is another client of the same server. Phone, wrist, desk, car: same agent, different body.

What’s next

I’m building this in public as far as the repo allows. Near-term focus for me:

1. Reliability of the voice loop - barge-in, latency, silence, failure modes when STT flakes
2. Safer action surface - more confirmations, clearer audit of what the agent did
3. Bodies - Mac + iOS parity first; desk Sentinel / Wisp hardware when the protocol stays clean
4. Local-first options - brains and voices that don’t require every token to leave home
5. Field blades - pocket and home units after the software will is solid

If the 2014 version was a spark on a student PC, this is the rebuild with a real stack and a real hardware plan for a desk unit that stands watch.

Summary

I built a voice desktop agent at OAU before the ecosystem could carry it. Igris is me finishing that work: voice-only, action-capable, software clients now, desk forms and field units on the roadmap.

Repo: 9trocode/Igris

It’s also usually incomplete.

Namespaces are a scoping convenience, not a security boundary. If your platform’s isolation story stops at metadata.namespace, you’re running shared fate with extra YAML.

I learned this the expensive way while designing multi-tenant hosting for people who didn’t have (or want) a full cloud account. That path shipped as Nova on PipeOps. This post is the isolation design, not a product brochure. Product notes: How Nova Isolates Tenants on Shared Kubernetes.

TL;DR: Soft multi-tenancy needs controls that actually enforce isolation, not a namespace label. In our stack that means API identity (no raw apiserver), NetworkPolicy, resource quotas, and a stronger runtime for tenant code (gVisor / runsc, not stock runc). Capsule, NetworkPolicies, quotas, an impersonating proxy, and gVisor work together. Namespaces alone are not enough.

The failed first design (keep this story)

Our first multi-tenant attempt was naive on purpose: one beefy node pool, many users, “split the bill.”

What broke in two weeks:

• No network isolation - pods could talk sideways
• No meaningful quotas - one memory leak was everyone’s outage
• Shared trust in the API - if you can reach the apiserver as a powerful subject, the namespace label is cosplay
• Blast radius = the cluster - noisy neighbor wasn’t a metaphor

We killed it. Soft multi-tenancy without enforcement is just colocation.

What “tenant” has to mean

For application platforms (not hyperscaler-grade hostile multitenancy), a tenant is roughly:

A principal that can deploy and operate workloads without reading, mutating, or starving other principals’ workloads - and without administering the host cluster.

That implies controls for:

If you only sell “namespace,” you’ve only bought the first bullet’s directory structure.

Hard multi-tenancy (dedicated clusters/nodes, microVMs, bare metal) is a different product. Soft multi-tenancy is valid - if you raise the runtime bar and enforce the rest. We don’t stop at “shared kernel, shrug”; tenant workloads sit on gVisor so userland is isolated from the host more tightly than plain runc.

The pattern that worked for us

We needed virtual clusters without standing up a full control plane per customer (vCluster-class cost/ops). The stack:

1. Namespace (or Tenant CR → namespaces) per customer
2. Capsule as the multi-tenant operator (tenant boundaries, policy hooks)
3. ResourceQuota / LimitRange (and pod count caps)
4. NetworkPolicy default-deny + explicit allows
5. Capsule Proxy (or equivalent) so tenants never hold a kubeconfig that can see the real cluster API as cluster-admin cosplay
6. RBAC scoped to the tenant’s namespaces only
7. gVisor (runsc) as the container runtime for tenant workloads - application-kernel isolation between the pod and the host

Flow in practice:

Tenant tooling / kubectl / CI
 │
 ▼
 Impersonating API proxy ──enforces tenant scope──► Kubernetes API
 │
 ├── RBAC: only their namespaces
 ├── Admission / operator: quotas, policies
 ├── CNI NetworkPolicies: no east-west between tenants
 └── RuntimeClass → gVisor (runsc) for tenant pods

To the user it feels like “my cluster.” To the platform it’s “your namespace(s) + API mediation + network/quota teeth + a sandboxed runtime.”

Why a proxy (this is the non-negotiable)

The critical product decision: tenants never talk to the real apiserver with a wide-open credential.

They talk to a proxy that:

• Authenticates the tenant
• Impersonates a constrained identity
• Filters verbs/resources to their scope
• Rejects cluster-scoped power tools (nodes, other tenants’ namespaces, wild cluster role binds)

If you hand out a normal kubeconfig against the shared apiserver and “trust RBAC,” one mis-bound RoleBinding or aggregated ClusterRole later, isolation is a blog post, not a property.

Why NetworkPolicy is not optional

Without default-deny:

• Service discovery becomes a reconnaissance API
• One compromised app is a pivot into the estate
• “We use private clusters” doesn’t help inside the cluster

Baseline I want on every soft multi-tenant pool:

• Deny all ingress/egress by default in tenant namespaces
• Allow DNS to the cluster DNS service
• Allow egress only to what the product requires (registries, object storage, public web - deliberately)
• Allow ingress only from the platform ingress / mesh identity you own

Policy drift kills you. Treat policies as part of tenant provisioning, not a ticket after an incident.

Why quotas are a security control

People file quotas under “FinOps.” They’re also availability security.

Without them:

• One tenant schedules the node into death
• EmptyDir fills the disk
• CronJobs stampede the apiserver

Every tenant needs at least:

• CPU / memory requests+limits (LimitRange defaults)
• Namespace ResourceQuota
• Object count caps (pods, services, PVCs)
• Optional PriorityClass isolation so platform agents aren’t starved

Exceeding quota should fail their deploys - not page your whole fleet.

Why gVisor (not “just Docker”)

API and network policy stop a lot of multi-tenant pain. They do not fix “this untrusted binary is talking to the host kernel through runc.”

For multi-tenant pools we run tenant workloads on gVisor (runsc): a user-space kernel that intercepts syscalls so guest code doesn’t get the full host kernel surface by default. Same Kubernetes UX (RuntimeClass), stronger isolation story than stock containers.

What it buys:

• Smaller host-kernel attack surface for tenant pods
• A middle ground between “hope runc is fine” and “every tenant gets a VM”

What it doesn’t buy:

• Perfect multi-tenancy (nothing on a shared node is perfect)
• Free compatibility - some workloads hate gVisor’s syscall coverage; test your images
• An excuse to skip NetworkPolicy, quotas, or the API proxy

Pitfall: RuntimeClass on the YAML but nodes still defaulting to runc. If it isn’t enforced (admission / RuntimeClass default / restricted RuntimeClass), you have a blog post, not a control.

Capsule vs the alternatives (tradeoffs, not religion)

We evaluated patterns in the usual set:

We chose Capsule-class soft multi-tenancy plus gVisor because:

• No second control plane per customer
• Works with normal Kubernetes tooling through the proxy
• Policy and quota can be templated at tenant create
• Runtime isolation for tenant code without full microVM tax everywhere

We still don’t market this as bare metal. If compliance needs dedicated hardware, provision their cloud account instead (Terraform runners - see The Runner). MicroVMs (Firecracker et al.) remain the next step when gVisor isn’t enough.

Different products, different isolation SLOs.

Provisioning checklist (steal this)

When a tenant is created:

1. Create tenant object / namespace(s) with immutable labels (tenant=…)
2. Apply ResourceQuota + LimitRange
3. Apply default-deny NetworkPolicy + DNS/egress allowlist
4. Bind Role/RoleBinding only inside tenant namespaces
5. Set RuntimeClass → gVisor for tenant workloads (and enforce it)
6. Issue credentials only for the proxy (short-lived if you can)
7. Run a smoke test:
   • can deploy to own namespace
   • cannot list other namespaces
   • cannot reach another tenant’s Service ClusterIP
   • cannot create ClusterRoleBinding
   • pods actually land on runsc (not silent runc fallback)
8. Emit audit events for admin-ish verbs

Automate the smoke test. Manual “looks good” doesn’t scale.

Pitfalls I’ve seen (including ours)

1. Ingress as a free lateral path - misconfigured shared ingress controllers become universal peers.
2. Cluster-wide CRDs without tenancy - one CRD install is a shared brain; gate who can create CRs.
3. Node filesystem assumptions - hostPath is a footgun; ban it for tenants.
4. Privileged pods / CAP_SYS_ADMIN - if your PSS/PSA isn’t enforced, NetworkPolicy won’t save you.
5. “Admin kubeconfig for support” - support tooling becomes the real attack surface; impersonate with break-glass and audit.
6. Log and metric multi-tenancy - observability backends that don’t filter by tenant leak data as surely as etcd.
7. RuntimeClass theater - gVisor on paper, tenants still scheduled on runc.

Summary

We tried shared hosts without teeth. It failed. Soft multi-tenancy only became real when tenants stopped holding the real apiserver, networks defaulted to deny, quotas made noisy neighbors a local outage, and workloads ran under gVisor instead of plain runc cosplay.

A namespace is a folder label. Isolation is the proxy, the policies, the quotas, and the runtime. If your diagrams only show green boxes labeled “Namespace,” you’re selling colocation.

Write the threat model on one page, name the runtime boundary (gVisor / microVM / dedicated node), and don’t ship tenant create until the checklist above is automated.

Related writing

• How Nova Isolates Tenants - product path for cheap shared hosting
• How to Safely Give AI Agents a Terminal - same isolation mindset for untrusted agents
• Terraform multi-cloud Runner - provisioning clusters before tenancy matters
• Work catalogue

I needed real-world testing across different machines and architectures - not just “works on my laptop” and not just CI logs. The quickest thing that could give me that was: a disposable terminal I can spin up anywhere, run the binary, and throw away.

That small tool turned into something bigger: a terminal control room for cloud sandboxes and your own machines.

TL;DR: Rexec is a Terminal-as-a-Service platform: create network-isolated cloud terminals (Docker-backed), connect your own machines with an outbound agent, share sessions, record sessions, and integrate via CLI/SDKs or an embed widget.

Positioning: Rexec turns terminals into infrastructure primitives.

Terminals become API-managed sandboxes you can create, connect to (WebSocket), run commands in, share, record, lock down, and delete - with guardrails.

The Origin Story (And the Accidental Product)

Rexec started off as a small tool I needed to test our CLI and agent binary on different machine types with real-world usage.

Then the scope creep hit:

1. CLI testing across machines: “Does this binary behave the same on Ubuntu vs Alpine? AMD64 vs ARM64? Fresh box vs crusty box?”
2. Cloud shell: once you can spin terminals up quickly, it’s basically a cloud shell. So we built a proper terminal UX around it.
3. Widget mode: then we added an embeddable widget, because the terminal shouldn’t live only inside the dashboard.
4. Students + low-spec laptops: a hosted terminal with a curated toolchain is a cheat code for learning when your laptop is underpowered (or you don’t have one).
5. Agent sandboxes: once I installed opencode in a terminal and started running AI-generated code in there, it clicked - this is a safer default for agents and automation.
6. Shared expensive machines: teams can connect a single beefy box (yes, including a GPU machine) and manage access without turning SSH keys into a company-wide group chat.

Rexec is now a multipurpose tool. We use it for all of the above. And it’s open source.

What Rexec Actually Is

Rexec has two primitives:

1. Cloud terminals: disposable Linux environments running as isolated containers.
2. BYOS agents: connect your own server/laptop/Raspberry Pi into the same dashboard via an outbound WebSocket tunnel.

Everything else (CLI, SDKs, widget, collaboration, recording) is built around making those two primitives usable in real workflows.

Canonical links:

• Docs: https://rexec.sh/docs
• Resources/tutorials: https://rexec.sh/resources
• Source: https://github.com/PipeOpsHQ/rexec

How We Built It (High-Level Architecture)

Rexec is intentionally boring infrastructure:

• Compute: Docker/Podman containers with hard CPU/memory/PID limits (and optional disk quotas when the host supports it).
• Isolation: terminals attach to an isolated bridge network with inter-container communication disabled (rexec-isolated) and hardened container settings (dropped capabilities + no-new-privileges).
• Terminal UX: WebSocket streaming to xterm.js, with tmux inside the container for reconnect + scrollback.
• BYOS: agents connect outbound over WebSockets (no inbound SSH ports).
• State: PostgreSQL for users/sessions/audit logs; optional S3 for session recordings.

From the OSS README, the architecture looks like this:

[Browser UI] ←(WebSocket)→ [Rexec API] ←→ [PostgreSQL]
 │
 ├── [Container Manager] ──→ [Docker Engine]
 │
 └── [Agent Handler] ←(WebSocket)→ [Remote Agents]

Implementation stack (also from the repo):

• Frontend: Svelte + xterm.js + Tailwind CSS
• Backend: Go (Gin) + Gorilla WebSocket
• Runtime: Docker Engine
• DB: PostgreSQL

The design goal is simple: a terminal that feels native, but is disposable by default.

One implementation detail that matters: cloud terminals are kept alive (the container runs indefinitely), and interactive sessions attach via exec into a tmux session. That’s what makes disconnect/reconnect cheap.

What It Can Do (Without the Marketing)

1) Disposable, network-isolated Linux terminals

Spin up a new terminal, test something risky, then delete it. You can choose from common images (Ubuntu/Debian/Alpine/Fedora/Arch/Kali, etc) depending on the workflow.

This is the feature that makes everything else possible: once environments are cheap, you stop doing dangerous things on your laptop.

2) A real terminal UX (not a toy web console)

Rexec streams a proper terminal session (xterm.js) over WebSockets, with session persistence and collaboration.

The “instant access” trick is that you can start typing while the environment finishes provisioning in the background - no staring at progress bars.

3) Bring-your-own-server access (BYOS agent)

If you already have machines you care about (prod boxes, staging, a GPU workstation, a lab server), you can connect them to Rexec via an outbound agent. The machine shows up like a terminal you can click into.

Key point: you’re not opening inbound SSH or building VPN spaghetti.

4) Collaboration + session recording

Two features that matter when you’re doing real ops work:

• Share a session (view/control) for pair debugging or teaching.
• Record a session and replay it for documentation, auditing, or incident reviews.

5) CLI + TUI (power-user mode)

The rexec CLI lets you create/list/connect to terminals, manage snippets/macros, and register agents.

Some quick examples:

rexec login
rexec ls
rexec create --name mydev --image ubuntu:24.04
rexec connect <terminal-id>
rexec -i

It also supports shell completion and SSH integration patterns (ProxyCommand) if you’re the type of person that lives in ~/.ssh/config.

6) SDKs + API for automation and agents

Rexec exposes a REST API (and WebSockets for terminals). On top of that, there are official SDKs in multiple languages (Go, JS/TS, Python, Rust, Ruby, Java, .NET, PHP).

Example (Python-style SDK usage):

import asyncio
from rexec import RexecClient

async def main():
 async with RexecClient("https://rexec.sh", "YOUR_API_TOKEN") as client:
 container = await client.containers.create(image="ubuntu:24.04", name="sdk-demo")
 async with client.terminal.connect(container.id) as term:
 await term.write(b"echo 'Hello from Rexec!'\n")
 out = await term.read()
 print(out.decode())

asyncio.run(main())

7) Embeddable terminal widget

This is the “cloud shell inside your docs” feature.

You add a script tag, create a div, then embed a session using either:

• a share code (guest access), or
• an API token (authenticated)

Minimal embed example:

<script src="https://rexec.sh/embed/rexec.min.js"></script>

<div id="terminal" style="width: 100%; height: 400px;"></div>

<script>
 const term = Rexec.embed('#terminal', {
 shareCode: 'YOUR_SHARE_CODE'
 });
</script>

Token mode (create a new terminal from your site) looks like this:

<script>
 const term = Rexec.embed('#terminal', {
 token: 'YOUR_API_TOKEN',
 image: 'ubuntu',
 role: 'python'
 });
</script>

This is why I think Rexec is useful for education and DevRel: you can turn “run this command” into “run it here”.

Security: What’s Actually Enforced

Rexec runs arbitrary shell sessions. So security isn’t a paragraph - it’s the product.

Here’s what the open-source stack enforces for Linux terminals today (and what you can tune when self-hosting).

Container boundaries

• No privileged containers (Linux terminals run with Privileged: false) plus SecurityOpt: no-new-privileges:true
• Capabilities: CapDrop: ALL, then add back a small allowlist (including NET_BIND_SERVICE for low ports, and SYS_PTRACE for debugging/TUI tools)
• Default seccomp profile (not unconfined)
• Host info masking via masked /proc + read-only /proc/sys* paths

Network isolation model

• Terminals attach to a dedicated bridge network: rexec-isolated
• Inter-container communication is disabled (com.docker.network.bridge.enable_icc=false)

This doesn’t mean “no internet”. It means “don’t let user sandboxes talk to each other by default.”

Resource + abuse controls

• Hard memory limit with swap disabled (MemorySwap == Memory)
• CPU quota limiting (CPUPeriod/CPUQuota)
• PIDs limit (fork-bomb brake)
• Optional disk quotas when the host supports it (overlay2 quotas, typically XFS/ext4)
• Rate limiting at multiple layers (nginx at the edge + application middleware)

Auth + audit trail

• JWT auth + MFA support, plus hardened account options like single-session mode and IP allowlists
• Sensitive stored values are protected (API tokens are stored hashed, and secrets like MFA/SSH material are encrypted at rest)
• Audit logs stored in Postgres (action, IP, user-agent, JSON details)
• Session recording (optional S3 backend) for replay/auditing

Runtime isolation: gVisor

Rexec uses gVisor for stronger sandbox isolation on cloud terminals. Set the OCI runtime to runsc so guest syscalls are handled in user space instead of riding plain runc on the host kernel:

# example self-host env for the rexec service
OCI_RUNTIME=runsc

That matches the multi-tenant Kubernetes isolation story: untrusted workloads shouldn’t default to stock container runtimes. Kata (OCI_RUNTIME=kata) and Firecracker microVM terminals remain options when you need a harder boundary still.

Caveats (the honest part)

• The root filesystem is currently writable to support role/tool installation - pair that with gVisor, don’t skip the runtime.
• /tmp is mounted exec in the default profile to support some terminal tooling. Tighten it if you don’t need that.

Differentiation: Compared to the Usual Suspects

If you’re evaluating Rexec, you’re probably comparing it to one of these:

If you want a full VS Code-in-the-browser experience, use Codespaces/Gitpod. If you want governed terminals as a primitive (sandboxes + BYOS + embed + API), that’s the lane Rexec is in.

Performance & Scaling (The Boring Parts)

• Startup is “create container + start container”. For faster boots, you can prebuild rexec-* images (./scripts/build-images.sh) so basics (like SSH) are already there.
• Reconnect is fast because the session is tmux-backed, and the terminal attaches via exec. Scrollback is configured to be large (tmux history is set to 50,000 lines).
• Fairness is enforced with hard CPU/memory/PID limits, plus per-tier container/agent limits.
• Abuse prevention exists at multiple layers (edge + app). If you run a public instance, this matters.
• Stronger sandboxes: prefer OCI_RUNTIME=runsc (gVisor) for agent/multi-tenant-style isolation; Kata remains available when you need that path.
• At “real usage” scale (dozens → hundreds of concurrent terminals), this becomes capacity planning: per-terminal caps, container-host sizing, and a load balancer that handles long-lived WebSockets properly.

What It Looks Like in Practice

Scenario: release engineering across distros

You’re shipping a CLI. You want confidence it works on real environments, not just CI containers.

1. Create terminals across a few base images:

rexec create --name cli-ubuntu --image ubuntu:24.04
rexec create --name cli-debian --image debian:12
rexec create --name cli-alpine --image alpine:3.21

1. Download the binary, run the same smoke test on all three.
2. Share one session (view/control) when something breaks, and record it if you need a replayable artifact.

Scenario: SRE debugging without SSH key chaos

1. Install the agent on the box (outbound connection, no inbound SSH ports).
2. Connect from the dashboard, share the session for pair debugging, and keep an audit trail with session recording.

Scenario: education + DevRel that actually runs

Embed a terminal in docs/tutorials, hand out share codes, and let people run commands where they’re learning - without a “works on my machine” setup tax.

Scenario: agents that execute in a sandbox

If you’re using AI coding tools, the safest workflow is “generate → run → test” in a disposable environment:

1. Create a fresh terminal (or one per task).
2. Run the agent (e.g., opencode, aider) inside the sandbox.
3. Run tests in isolation.
4. Delete the terminal when you’re done.

Who Needs This (Realistically)

Rexec isn’t for everyone. If you already have perfect laptops, perfect networks, and perfect discipline, you can stop reading.

If you don’t, here’s who it’s built for:

• CLI authors and release engineers who need to validate binaries on real machines/arches without collecting laptops like Pokémon.
• SRE/DevOps teams who want disposable jump boxes and a safer way to reach machines without SSH key chaos.
• Students and bootcamps who need a real environment without a high-spec laptop.
• DevRel + docs teams who want runnable tutorials via the embed widget (or pre-configured roles).
• AI/agent builders who need a sandbox to run generated code, execute tests, and share sessions for review.
• Teams with shared expensive hardware (including GPUs): connect the box once, manage access centrally, and avoid “who has the SSH key?” as your access-control strategy.

Pitfalls (Read This Before You Paste Tokens Into Anything)

1. Treat API tokens like passwords. They give account-level access.
2. Know the trust model. A sandbox reduces blast radius; it doesn’t make malware “safe”. Don’t drop production secrets into random terminals.
3. Network isolation is about east/west by default. If you need strict egress controls, enforce them at the host/network layer.
4. Agents need outbound WebSockets. Some corporate networks break this; plan accordingly.
5. Self-hosting defaults are for dev. Change default credentials, set JWT_SECRET, and put it behind TLS.

Try It (Quick, Practical)

Cloud terminal (hosted)

1. Open https://rexec.sh and create a terminal.
2. Run something you normally don’t want on your laptop (build scripts, installer experiments, etc).
3. Share the session if you need help debugging.

Install the CLI (optional, but you’ll end up here)

1. Install rexec:

# Linux/macOS install script
curl -fsSL https://rexec.sh/install-cli.sh | bash

If you’d rather not run an install script, build from source:

git clone https://github.com/PipeOpsHQ/rexec
cd rexec
go build -o rexec ./cmd/rexec-cli
sudo mv rexec /usr/local/bin/rexec

1. Log in and create a terminal:

rexec login
rexec create --name mydev --image ubuntu:24.04
rexec connect <terminal-id>

Connect your own machine (agent)

1. Generate an agent install command from the dashboard (Settings → Agents).
2. Run the installer on your server:

curl -fsSL https://rexec.sh/install-agent.sh | sudo bash -s -- --token YOUR_TOKEN

1. Your machine should appear as a terminal card.

Self-host (open source)

If you want full control, self-host:

git clone https://github.com/PipeOpsHQ/rexec
cd rexec/docker

Rexec’s container manager talks to a Docker/Podman daemon. In the stock docker/docker-compose.yml, the API container connects to a remote Docker host over TLS (no docker.sock mount), so you must provide DOCKER_HOST + certs.

Create rexec/docker/.env:

DOCKER_HOST=tcp://YOUR_DOCKER_HOST:2376
DOCKER_TLS_VERIFY=1
DOCKER_CA_CERT="(contents of ca.pem)"
DOCKER_CLIENT_CERT="(contents of cert.pem)"
DOCKER_CLIENT_KEY="(contents of key.pem)"
JWT_SECRET="change-me"
POSTGRES_PASSWORD="change-me"

Then start the stack:

docker compose up -d --build

Open http://localhost:8080 and change defaults immediately. For the full remote-Docker deployment model (and why it’s the recommended production shape), see: https://github.com/PipeOpsHQ/rexec/blob/main/docs/DEPLOY_STANDALONE.md

For gVisor isolation when self-hosting, install runsc on the container host and set OCI_RUNTIME for the rexec service:

# docker/docker-compose.yml
environment:
 - OCI_RUNTIME=runsc

Summary

Rexec turns terminals into infrastructure primitives. It started as a way to test a CLI on real machines. It became a terminal control room: disposable gVisor-isolated cloud terminals, a BYOS agent, an embed widget for docs, SDKs for automation, and a safer sandbox for agents.

If that sounds like your workflow, start with the docs: https://rexec.sh/docs

The Problem with Fing

If you’ve ever needed to see what devices are on your network, you’ve probably used Fing. It was the go-to tool - simple, fast, and it just worked.

Back in uni days, Fing was the go-to hack tool for detecting IPs connected to the network. You could easily shop around, pick whichever IP you wanted, and assume it on your device - which would kick out whoever was logged into Intecu’s MikroTik server. It was fun. People would swear and rain curses after buying their Intecu card only to get booted off minutes later. In fact, I can’t remember buying an Intecu card with my own money throughout my time there - maybe ‘cause I was broke then. 😇

Was.

Fing has pivoted hard toward monetization:

• The original free CLI tool? Gone. Try finding a download link. I’ll wait.
• The new Fing Desktop app? Requires an account and uploads your network topology to their servers.
• Want continuous monitoring? That’ll be $4.99/month for Starter or $9.99/month for Premium.

For a tool that scans your network and shows you what’s connected, this feels… wrong.

Your network topology is sensitive data. It reveals:

• How many devices you own
• What brands you prefer (Apple household? IoT-heavy smart home?)
• When devices come online (your daily patterns)
• Potential security vulnerabilities

This data shouldn’t leave your machine. Period.

Enter OpenFing

I built OpenFing over a weekend using Zig. It’s everything the old Fing CLI was, minus the corporate baggage:

$ openfing

+==============================================================================+
| OpenFing v1.4.0 |
| Fast Network Device Scanner |
+==============================================================================+

Network Information:
--------------------
 Your IP : 192.168.1.100
 Gateway : 192.168.1.1
 Subnet : 192.168.1.0/24
 Interface : en0

Scanning....... done

+-----------------------------------------------------------------------------+
| DEVICES FOUND: 8 (via multi-method discovery)
+-----------------------------------------------------------------------------+

IP ADDRESS | MAC ADDRESS | VENDOR
------------------+--------------------+-------------------------------------
192.168.1.1 | E8:EA:4D:1D:3A:45 | Huawei (GW)
192.168.1.50 | 4C:20:B8:DB:D5:E8 | Apple
192.168.1.100 | BE:29:E5:69:04:E0 | Intel (THIS)
192.168.1.105 | B0:41:6F:0D:78:17 | Shenzhen Maxtang
192.168.1.110 | 24:0D:C2:A1:B2:C3 | Espressif (IoT)
...

Total: 8 devices

What Makes It Different

1. Zero Data Collection

Everything runs locally. There’s no account creation, no telemetry, no “anonymous usage data.” Your network scan results never leave your machine.

2. Works Without Root

Most network scanners need sudo because they send raw ARP packets. OpenFing uses a multi-method discovery approach that finds devices even without elevated privileges:

• Ping sweep + ARP cache
• mDNS/Bonjour discovery (finds Apple devices, printers, Chromecasts)
• SSDP/UPnP discovery (finds smart TVs, gaming consoles, routers)
• TCP port probing (triggers ARP entries for web servers, SSH hosts)
• NetBIOS discovery (finds Windows/Samba devices)

In my testing, the non-sudo scan found more devices than some sudo-based tools.

3. Single Binary, No Dependencies

It’s written in Zig, which means it compiles to a single static binary. No Python runtime, no Node.js, no Docker. Just download and run.

4. Auto-Updates (Optional)

OpenFing checks for updates daily in the background. If a new version is available, it tells you. You can disable this with --no-update if you prefer.

5. Deep Scan Mode

Want hostnames and open ports? Use --deep:

$ sudo openfing --deep

IP ADDRESS | MAC ADDRESS | VENDOR/HOST | PORTS
------------------+--------------------+------------------------------+----------
192.168.1.1 | E8:EA:4D:1D:3A:45 | router.local | HTTP,HTTPS
192.168.1.50 | 4C:20:B8:DB:D5:E8 | MacBook-Pro.local | SSH
192.168.1.110 | 24:0D:C2:A1:B2:C3 | esp-sensor.local | HTTP

Installation

One-Liner Install

macOS (Apple Silicon):

curl -L https://github.com/9trocode/OpenFing/releases/latest/download/openfing-macos-arm64 -o openfing && chmod +x openfing && sudo mv openfing /usr/local/bin/

macOS (Intel):

curl -L https://github.com/9trocode/OpenFing/releases/latest/download/openfing-macos-x86_64 -o openfing && chmod +x openfing && sudo mv openfing /usr/local/bin/

Linux (x86_64):

curl -L https://github.com/9trocode/OpenFing/releases/latest/download/openfing-linux-x86_64 -o openfing && chmod +x openfing && sudo mv openfing /usr/local/bin/

Linux (ARM64/Raspberry Pi):

curl -L https://github.com/9trocode/OpenFing/releases/latest/download/openfing-linux-arm64 -o openfing && chmod +x openfing && sudo mv openfing /usr/local/bin/

Build from Source

git clone https://github.com/9trocode/OpenFing.git
cd openfing
zig build -Doptimize=ReleaseFast
sudo mv zig-out/bin/openfing /usr/local/bin/

Usage Cheatsheet

openfing # Quick scan (no sudo needed)
sudo openfing # Full network scan
sudo openfing --deep # Scan with hostname + port detection
sudo openfing en0 # Scan specific interface
sudo openfing --install-deps # Install arp-scan for best results
openfing --update # Check for updates
openfing --no-update # Disable auto-update check

Why Zig?

I chose Zig for a few reasons:

1. No runtime dependencies - Compiles to a static binary that works anywhere
2. Cross-compilation is trivial - Build for Linux ARM from my Mac with one flag
3. C interop - Could easily integrate libpcap later if needed
4. Performance - It’s fast. Really fast.
5. I wanted to learn it - And building something useful is the best way

The Technical Bits

For those curious about how it works:

ARP Scanning (with sudo)

When you have root privileges and arp-scan installed, OpenFing uses it to send ARP “who-has” requests to every IP in your subnet. Every device must respond to ARP (it’s how networking works), so this finds everything that’s online.

Multi-Method Discovery (without sudo)

Without root, we can’t send raw packets. But we can be clever:

1. Ping sweep - Send ICMP echo requests to populate the ARP cache
2. mDNS queries - Apple devices and many IoT devices advertise via Bonjour
3. SSDP multicast - UPnP devices respond to discovery requests
4. TCP connect - Opening a TCP connection to common ports (22, 80, 443) triggers ARP resolution
5. Read ARP cache - After all that activity, the kernel’s ARP cache has entries for most devices

MAC Vendor Lookup

The first 3 bytes of a MAC address identify the manufacturer (called the OUI - Organizationally Unique Identifier). OpenFing has a built-in database covering the most common vendors: Apple, Samsung, Google, Amazon, Intel, Raspberry Pi, Espressif (ESP8266/ESP32), and more.

Roadmap

Things I’m considering for future versions:

• Windows support
• JSON/CSV output for scripting
• Larger OUI database (or online lookup fallback)
• Device fingerprinting (OS detection)
• Historical tracking (see new devices since last scan)
• Web UI option

FAQ: open source Fing alternative

Is OpenFing a free Fing alternative?

Yes. OpenFing is free and open source. It does the job most people wanted from the old Fing CLI: list devices on your LAN (IP, MAC, vendor) without a paid plan.

Does OpenFing upload my network topology?

No. Scans stay on your machine. No account, no cloud dashboard, no topology sync.

Where is OpenFing on GitHub?

github.com/9trocode/OpenFing - releases for macOS and Linux (x86_64 and arm64).

Is this the same as Fing Desktop / Fing Premium?

No. Those products are account-based and oriented around continuous monitoring in the cloud. OpenFing is a local CLI for discovery when you need a map of your network, not a subscription service.

Try It Out

The code is on GitHub: github.com/9trocode/OpenFing

Star it if you find it useful. Open issues if you find bugs. PRs welcome.

Related writing

If you care about local-first tools and isolation in general (not just home LAN maps):

• Namespaces Aren’t Isolation - multi-tenant Kubernetes that actually enforces boundaries
• How to Safely Give AI Agents a Terminal - sandboxes for untrusted automation
• Work catalogue - other platforms and open source I ship

When we launched PipeOps, you could provision servers on AWS, GCP, Azure - bring your own cloud account, we’ll manage it. Great for companies with cloud budgets. Not so great for side projects and solo developers.

So we thought: what if PipeOps could just…provide the servers?

The First Attempt (The Bad One)

Our first try was naive. Spin up one beefy AWS server, let multiple users deploy to it. Shared infrastructure, split the cost, everyone’s happy.

Except:

• Zero isolation between tenants
• One user’s memory leak affected everyone
• Security nightmare (shared namespace? really?)
• Couldn’t scale worth a damn

We killed it after two weeks. Back to the drawing board.

Enter Capsule

Then I found Capsule - a Kubernetes operator that turns namespaces into “virtual clusters.” Each tenant gets what looks like their own cluster, but it’s really just isolated namespaces with strict policies.

Perfect. This is what we needed.

How Nova Works Now

When you create a PipeOps-managed server through Nova:

1. We create a dedicated namespace with your name on it
2. Capsule enforces resource quotas (CPU, memory, storage)
3. Network policies isolate your traffic
4. RBAC prevents you from seeing other tenants
5. You get your own kubeconfig (via Capsule Proxy)

To you, it looks like a personal Kubernetes cluster. To us, it’s a namespace in our multi-tenant setup. You can’t access our control plane, you can’t see other tenants, you can’t break out.

The Security Model

Here’s the critical part: you never touch our actual Kubernetes API.

When you deploy, you’re talking to Capsule Proxy. It impersonates your tenant, applies your namespace’s policies, and forwards requests. If you try something you shouldn’t - access another namespace, create cluster-wide resources - the proxy says “nope.”

It’s Kubernetes inside Kubernetes. Turtles all the way down.

vs. Cloud Provider Provisioning

If you create servers through Nova on AWS/GCP/Azure, we provision actual VMs or Kubernetes clusters in your account using the Runner’s Terraform provisioning. That’s different infrastructure - you own it, we just manage it.

PipeOps-managed servers (the multi-tenant ones) are for when you don’t have (or want) a cloud account. We handle everything. You can also bring your own server using the PipeOps agent.

Resource Limits

Every Nova server gets:

• CPU quota (no hogging cores)
• Memory limits (OOM kills stay in your namespace)
• Storage caps (you can’t fill our disk)
• Pod limits (can’t spawn a million containers)

Exceed your quota? Your stuff stops working, not everyone else’s. Fair’s fair.

The Tech Stack

Nova is built on:

• Kubernetes 1.28+: Base orchestration
• Capsule: Multi-tenant operator
• Capsule Proxy: Tenant impersonation and isolation
• Network Policies: Traffic isolation
• RBAC: Permission boundaries
• Resource Quotas: Hard limits
• gVisor (runsc): Sandboxed container runtime for tenant workloads

We chose Capsule over alternatives (Hierarchical Namespaces, vCluster) because:

• Mature project, active development
• Doesn’t add another control plane
• Works with standard Kubernetes tools
• Good docs (rare in K8s land)

Cost Model

Here’s the math: our multi-tenant cluster costs $X/month to run. We split that across N tenants. Each tenant pays based on their resource usage.

Use more, pay more. Use less, pay less. Simple.

Compare to AWS: minimum $72/month for a tiny EKS cluster. Nova starts at $5.

What You Can Deploy

Anything that runs in a container:

• Web apps (Node, Python, Go, whatever)
• Databases (we support stateful sets)
• Workers and background jobs
• APIs and microservices
• Even ML workloads (if you have GPU quota)

If it fits in a Docker image and respects resource limits, Nova will run it.

The Future

We’re working on:

• Bare metal Nova (our own data centers)
• GPU instances for ML workloads
• Serverless containers (only pay when running)
• Spot instance support (cheaper, less reliable)

The goal is simple: make servers boring. Pick a size, deploy your code, forget about it.

When Not to Use Nova

Don’t use PipeOps-managed servers if you:

• Need root access to the node
• Want to run custom kernel modules
• Have compliance requiring dedicated hardware
• Need guaranteed physical isolation

In those cases, use Nova’s cloud provisioning. We’ll spin up VMs in your account instead.

Try It

Go to console.pipeops.io, click “New Server,” choose “PipeOps Managed.” Pick your resources (1 CPU, 2GB RAM, 10GB storage - starts at $5/month).

Three minutes later, you have a server. Deploy something. It just works.

No AWS account. No kubectl. No Terraform. Just a server that runs your code.

That’s how the PipeOps agent installer was born.

The Problem

Before the agent, using PipeOps meant you either:

1. Provisioned through us (AWS, GCP, Azure via Terraform)
2. Had an existing Kubernetes cluster we could talk to

But what about that Dell server under someone’s desk? Or the Hetzner box running some legacy app? Or hell, even a Raspberry Pi?

People wanted to bring their own metal. We needed a way to make any Linux machine speak PipeOps.

How It Works

Linux/Mac:

curl -fsSL https://get.pipeops.dev/k8-install.sh | bash

Windows (PowerShell):

# Install dependencies first
choco install curl wget -y

# Set your token
$env:PIPEOPS_TOKEN = "your-token"

# Run installer
curl -fsSL https://get.pipeops.dev/k8-install.sh | bash

Windows (WSL - recommended):

curl -fsSL https://get.pipeops.dev/k8-install.sh | bash

That’s it. One line. The script:

1. Detects your OS (Ubuntu, Debian, CentOS, Windows - we’re not picky)
2. Installs k3s if you don’t have Kubernetes
3. Downloads and configures the PipeOps agent
4. Registers with the control plane
5. Reports back “I’m ready, what’s next?”

Three minutes later, your random server is now a first-class PipeOps deployment target.

The Agent

Once installed, the agent is basically a lightweight Kubernetes controller that:

• Watches for deployment requests from the control plane
• Pulls container images
• Manages application lifecycles
• Reports metrics and logs back
• Handles secrets and configs

It’s the bridge between our control plane and your infrastructure. Websocket connection, encrypted, minimal overhead.

Why k3s?

We picked k3s over full Kubernetes because:

• Tiny footprint (512MB RAM minimum)
• Single binary install
• Works on ARM (Pi support for free)
• Battle-tested in production

If your server already has Kubernetes, cool - we’ll use that. If not, k3s gets you 90% of Kubernetes with 10% of the complexity.

Bring Your Own Server (BYOS)

This is what made Nova possible. Our multi-tenant platform where you can either use PipeOps-managed infrastructure or your own.

The agent is how “your own” works. Point it at any server, run the installer, boom - that machine is now part of your Nova fleet.

No cloud account needed. No Terraform modules. No kubectl fu. Just hardware and the installer script.

Security

The agent never exposes your Kubernetes API. Ever. It’s a one-way connection:

• Agent initiates connection to control plane
• Uses websockets with TLS
• Token-based auth (rotate tokens anytime)
• No inbound ports needed (firewall friendly)

Your clusters stay isolated. We don’t get API access. If the agent goes down, your apps keep running - they just stop getting updates until it reconnects.

Real-World Usage

I’ve seen this thing installed on:

• Bare metal in colocation facilities
• Old laptops running Ubuntu
• ARM servers (Raspberry Pi clusters, believe it or not)
• Hetzner dedicated servers
• DigitalOcean droplets (why not?)
• Windows servers (both native and WSL)
• Even a NAS running in someone’s closet

If it has 1GB RAM and can run containers, the agent will probably work.

The Code

The installer is a bash script with way too many comments (my fault). We detect package managers, handle different init systems, and try not to break existing setups.

When something goes wrong, we log everything. Error messages include the fix. No “contact support” BS - just “here’s what broke, here’s how to fix it.”

What’s Next

Working on:

• Auto-updates for the agent
• Better health checks
• Local dashboard (no internet? agent still works)

The goal is simple: if it has an IP address, PipeOps should be able to deploy to it.

Try It

If you have a server laying around:

curl -fsSL https://get.pipeops.dev/k8-install.sh | bash

Three minutes later, deploy something to it from the PipeOps dashboard.

That’s the magic - infrastructure should be this easy.

It’s not. Here’s what actually happens when you deploy on PipeOps, with all the details I wish other platforms documented.

Architecture Overview

Here’s the complete flow from git push to running container:

┌─────────────┐
│  Git Push   │
└──────┬──────┘
       │
       ▼
┌─────────────────────────────────────────────────────────────┐
│                    PipeOps Platform                          │
│                                                              │
│  ┌──────────┐     ┌────────────┐     ┌─────────────┐       │
│  │ Webhook  │────▶│ Controller │────▶│  RabbitMQ   │       │
│  │   API    │     │  (Go)      │     │   Queue     │       │
│  └──────────┘     └────────────┘     └──────┬──────┘       │
│                                              │              │
│                                              ▼              │
│                                       ┌─────────────┐       │
│                                       │   Runner    │       │
│                                       │   (Go)      │       │
│                                       └──────┬──────┘       │
│                                              │              │
│                  ┌───────────────────────────┼───────────┐  │
│                  │                           │           │  │
│                  ▼                           ▼           ▼  │
│          ┌──────────────┐         ┌──────────────┐  ┌────┐ │
│          │   BuildKit   │         │  Kubernetes  │  │ DB │ │
│          │    (Build)   │         │  (Deploy)    │  └────┘ │
│          └──────┬───────┘         └──────────────┘         │
│                 │                                           │
│                 ▼                                           │
│          ┌──────────────┐                                   │
│          │    Image     │                                   │
│          │   Registry   │                                   │
│          └──────────────┘                                   │
└──────────────────────────────────────────────────────────────┘

PipeOps deployment architecture: Git webhook triggers Controller, which queues jobs in RabbitMQ. Runner processes queue, executing builds with BuildKit and deployments to Kubernetes, pushing images to registry with state stored in database.

Every deployment goes through this pipeline. No shortcuts, no special cases.

The Real Stack

Everything is written in Go. The Runner is a Go service that handles the entire deployment lifecycle. We don’t shell out to docker build - we use BuildKit’s Go SDK directly.

Why Go? Because we need:

• Concurrency (hundreds of simultaneous builds)
• Performance (fast is non-negotiable)
• Type safety (infrastructure code needs to be reliable)
• Good Kubernetes client libraries

Git to Build: The Actual Flow

When you push code:

1. Webhook hits our API (signature validated - we don’t trust anyone)
2. Controller validates the project exists and is configured
3. Job goes into RabbitMQ with full context
4. Runner picks it up from the queue
5. Creates isolated workspace with unique ID
6. Clones repo (shallow clone, commit-specific)
7. Detects language if not specified
8. Executes build

Each workspace is isolated. No shared state. Workspace gets nuked after build completes or fails.

Language Detection: Pattern Matching

Not AI, not magic - just file patterns:

if fileExists("package.json") {
    return "nodejs"
} else if fileExists("requirements.txt") || fileExists("Pipfile") {
    return "python"
} else if fileExists("go.mod") {
    return "go"
} else if fileExists("pom.xml") || fileExists("build.gradle") {
    return "java"
}

Works 95% of the time. The other 5%, you tell us what you’re using.

BuildKit: Not Docker

We don’t run docker build. We use BuildKit - Docker’s build engine - but directly via its Go API.

Why BuildKit?

Parallel builds: BuildKit builds independent layers in parallel. Regular Docker is sequential.

Better caching: Content-addressable storage. If layer content hasn’t changed, it’s cached even if Dockerfile changed.

Remote caching: Build cache in S3. Team members share cache. First build is slow, rest are fast.

Build secrets: Pass secrets without baking them into images. They’re mounted at build time, never stored.

Multi-platform: Build ARM and x86 images simultaneously. Same Dockerfile, multiple architectures.

Our BuildKit Setup

We run BuildKit as privileged pods in Kubernetes with tuned configuration for:

• High concurrency - Multiple parallel build operations per pod
• Large cache retention - Aggressive caching strategy to maximize hit rates
• Fast base image pulls - Registry mirrors and geographic distribution
• Multi-platform builds - ARM64 and AMD64 support out of the box

The BuildKit configuration is optimized for throughput over individual build speed - we prioritize handling 50+ concurrent builds over making one build 10% faster.

The Build Process

We use BuildKit’s Go SDK directly - no shelling out to docker build. The flow:

1. Connect to BuildKit daemon - Each Runner maintains a pool of BuildKit connections
2. Prepare build context - Isolated workspace with source code and Dockerfile
3. Configure cache backend - S3-backed cache with content-addressable storage
4. Stream progress - Real-time build output to dashboard via WebSockets
5. Handle completion - Push image to registry or fail with actionable errors

The BuildKit SDK gives us fine-grained control over:

• Build parallelization (independent layers build simultaneously)
• Cache import/export (share cache across team)
• Secret handling (mount secrets at build time, never bake into image)
• Multi-platform targeting (ARM64 + AMD64 from same Dockerfile)

Build Args: Security Matters

We filter what gets passed as build args. Early mistake: passing all env vars. Bad idea - secrets ended up in image layers.

Now we only pass:

1. Args explicitly declared in Dockerfile with ARG
2. Essential build metadata (commit SHA, build ID, timestamps)
3. User-provided build args that match declared ARGs

Everything else is filtered out. We parse your Dockerfile, extract ARG declarations, and only pass matching variables. Undeclared args are silently dropped.

This prevents accidents like --build-arg DATABASE_PASSWORD=... ending up in image metadata that anyone can inspect with docker history.

Caching Strategy

Build cache lives in S3 with content-addressable storage. Each project gets isolated cache namespacing based on repo, branch, and Dockerfile path.

First build? Slow. Downloads all base images, installs all dependencies.

Second build? Fast. Only changed layers rebuild.

Team member builds? Fast. Shares your cache automatically.

Cache has automatic expiration and cleanup - we balance hit rates against storage costs. The system adapts: frequently-built projects get longer retention, abandoned projects get cleaned up fast.

Push to Registry

Built image gets pushed to your container registry. We support all major registries with native authentication:

DockerHub: Username/token
AWS ECR: IAM credentials with auto-renewal
GCR: Service account JSON
GitHub/GitLab: OAuth tokens
Azure ACR: Service principal

Image tag is always the commit SHA. Makes rollbacks trivial - just deploy a different SHA.

Multi-platform images? We push a manifest list. One tag, multiple architectures. Kubernetes pulls the right one.

Manifest Generation: Kubernetes Resources

Once image is built, we generate Kubernetes manifests. Not templated YAML - generated programmatically from Go structs using the official Kubernetes client libraries.

We generate a complete resource stack:

• Deployment (or StatefulSet for stateful apps)
• Service (ClusterIP, LoadBalancer, or NodePort)
• Ingress (for public apps with TLS)
• ConfigMap (non-sensitive environment variables)
• Secret (sensitive env vars, encrypted at rest)
• HorizontalPodAutoscaler (if autoscaling enabled)
• NetworkPolicy (traffic isolation rules)
• PodDisruptionBudget (for high availability)

All generated from your project configuration. The advantage of Go structs over YAML templates: type safety catches errors at compile time, not when the deploy fails.

Deployment Strategies: More Than Rolling Updates

Rolling Update: Default. Max unavailable=1, max surge=1. Gradual rollout.

Recreate: Kill everything, start new version. Fast but downtime.

Blue-Green: We deploy new version with different labels, wait for health checks, then switch service selector. Zero downtime, uses 2x resources temporarily.

Canary: Deploy with weight-based routing. 5% traffic → 25% → 50% → 100%. Istio VirtualService for traffic splitting.

You pick the strategy. We implement it correctly.

Health Checks: Not Optional

We configure three probe types:

Startup probe: Is the app starting? High timeout tolerance for slow-starting apps.

Readiness probe: Should this pod receive traffic? Failure removes pod from service endpoints immediately.

Liveness probe: Is the app alive? Failure triggers pod restart.

Default probes hit /health endpoints with tuned timeouts and thresholds based on app type. Node.js apps get longer startup windows than Go apps. Databases get different probe intervals than web servers.

Custom health endpoints? Configure them in project settings. We’ll use whatever you specify - HTTP, TCP, or exec probes.

Deployment Rollout Tracking

We don’t just apply manifests and hope. We watch the rollout using Kubernetes watch APIs to monitor deployment status in real-time.

The Runner tracks:

• Replica counts - Are new pods being created?
• Ready status - Are new pods passing health checks?
• Rollout conditions - Did we hit the progress deadline?
• Pod events - Why did that pod fail to start?

All status updates stream to the dashboard via WebSockets. You see exactly what’s happening: “Pod 2/3 ready”, “Waiting for health checks”, “Rollout complete”.

If the rollout stalls (pods not ready after deadline), we automatically rollback to the previous version. The old ReplicaSet is never deleted until the new one is healthy.

When Things Fail

Builds fail. Deployments fail. We handle it.

Build failure: Log the error, mark build as failed, send webhook, keep old version running.

Image push failure: Retry with exponential backoff (3 attempts), then fail.

Deployment failure: Monitor rollout for 5 minutes. If pods don’t become ready, automatic rollback.

Errors are actionable:

❌ Deployment failed: ImagePullBackOff

Error: Failed to pull image "registry/app:abc123"
Reason: authentication required

Fix:
1. Verify registry credentials in project settings
2. Ensure image tag exists in registry
3. Check registry is accessible from cluster

Run: pipeops project registry update

No “something went wrong.” Real errors, real fixes.

Performance Numbers

From production:

• Average build time (with cache): 2-4 minutes
• Average build time (no cache): 6-10 minutes
• Average deploy time: 45-90 seconds
• Concurrent builds: 50+ at peak
• Build success rate: 97%
• Deployment success rate: 98.5%

The 3% build failures? Usually user config (broken Dockerfile, missing dependencies).

The 1.5% deployment failures? Usually cluster issues (out of resources, network problems).

Performance Impact

The BuildKit and direct Kubernetes deployment approach significantly reduces total pipeline time. From cold cache, a typical Node.js deployment completes in 7-9 minutes. With warm cache (the common case after initial deploy), this drops to 2-3 minutes.

Why this architecture is fast:

• BuildKit’s layer caching - Smarter than Docker’s cache
• Dedicated build infrastructure - Not competing for shared runners
• Geographic distribution - BuildKit nodes close to clusters
• Direct K8s deployment - No intermediate artifacts or handoffs
• Optimized base images - We maintain language-specific optimized images

The difference compounds when deploying frequently. Multiple deploys per day means the warm cache performance becomes the norm, not the exception.

What We Don’t Do

We don’t:

• Run builds on your cluster (security risk)
• Store secrets in images (filtered out)
• Use Docker in Docker (unstable)
• Poll Git repositories (wasteful, webhooks only)
• Keep failed builds around (cleanup is automatic)

The Code

The Runner is 10,000+ lines of Go. Two years of production taught us everything that can break:

• AWS throttling ECR pushes
• BuildKit daemon crashes under load
• Kubernetes API timeouts during deploys
• Image pull failures on slow networks
• Race conditions in concurrent builds

All handled now.

The Runner is the workhorse of PipeOps. When someone clicks “Create Server,” the Runner is what actually makes multi-cloud Kubernetes real: Terraform across AWS, GCP, Azure (and lighter paths for DO/Linode), managed state, failures, and live logs.

It’s a Go service that provisions infrastructure across multiple cloud providers, manages state, handles failures, and streams logs back in real-time. The same Runner also handles application deployments once infrastructure is provisioned.

The Flow

1. Controller receives “provision cluster” request
2. Validates params (region, instance types, node count)
3. Queues job in RabbitMQ
4. Runner picks up job from queue
5. Loads our custom infrastructure modules
6. Injects customer-specific variables
7. Executes provisioning workflow
8. Streams output back to user in real-time
9. Extracts outputs (cluster endpoints, credentials)
10. Updates database with results
11. Triggers k8-agent deployment
12. Marks job complete

Average time: 10-15 minutes for a new cluster. Faster for updates.

Multi-Cloud Complexity

Supporting AWS, GCP, Azure, DigitalOcean, and Linode meant handling each provider’s quirks:

AWS: VPCs, subnets, security groups, IAM roles. Everything needs tagging. EKS takes 12 minutes to provision.

GCP: Networks, subnetworks, firewall rules, service accounts. GKE is faster (8 minutes) but weird networking defaults.

Azure: Resource groups, virtual networks, NSGs, managed identities. AKS takes forever (15 minutes). Their API is slow.

DigitalOcean: Simple but limited. K8s setup is quick (6 minutes) but features are basic.

Linode: Similar to DO. Fast but fewer regions.

The Runner abstracts all this. User picks “AWS, us-east-1, 3 nodes” and we handle the rest.

State Management

With hundreds of customers and thousands of clusters, infrastructure state management is critical.

For BYOC (Bring Your Own Cloud) deployments, each cluster’s state is stored in your cloud provider account. You bring your AWS/GCP/Azure account, we provision infrastructure in it, and state lives in your bucket.

Each BYOC cluster gets:

• Separate state file in your S3/GCS bucket
• State locking mechanisms (DynamoDB for AWS, native for GCS)
• Automated backups every hour
• Versioning enabled (can rollback if needed)

State bucket structure in your account:

s3://customer-state-bucket/
  cluster-production/
    state.tf
    backups/
      state.2024-10-31.tf
  cluster-staging/
    state.tf
    backups/
      state.2024-10-31.tf

For PipeOps-managed infrastructure (Nova multi-tenant clusters), we handle state internally with the same backup and versioning guarantees.

If state gets corrupted (it happens), we restore from backup automatically.

Infrastructure Modules

We don’t provision infrastructure from scratch for each cluster. The Runner uses a modular approach with well-tested components:

Kubernetes Module: Core cluster provisioning (EKS, GKE, AKS) Network Module: VPC, subnets, routing, security groups Essentials Module: Ingress controllers, cert-manager, monitoring stack Orchestration Layer: Combines modules into complete infrastructure stacks

These modules are versioned and locked to prevent breaking changes. Updates are tested in staging environments before gradual rollout to production clusters.

Error Handling

Infrastructure provisioning fails. A lot. Our error handling:

Cloud API failures: Retry with exponential backoff (up to 5 times)
Quota limits: Surface clear error with fix
Invalid config: Validate before execution
Partial failures: Mark what succeeded, offer cleanup or continue
Process crashes: Capture logs, save state, alert ops team

When provisioning fails, users get:

1. What went wrong
2. At what step
3. How to fix it
4. Option to retry or rollback

No “Something went wrong.” Actual information.

Concurrent Execution

The Runner handles hundreds of concurrent jobs:

• Each job gets isolated workspace
• Separate working directories
• No shared state between jobs
• Resource limits per job (CPU, memory)
• Job timeouts (30 minutes max)

We can run 50+ Terraform jobs simultaneously. The bottleneck is cloud provider APIs, not our infrastructure.

Streaming Logs

Users watch provisioning output in real-time in the dashboard. It’s WebSocket-based:

Runner → RabbitMQ → Controller → WebSocket → Browser

The provisioning process outputs to stdout/stderr, we capture it, send to queue, controller forwards to connected clients. They see everything as it happens.

Cleanup on Failure

If provisioning fails halfway, we clean up what was created. Otherwise users end up with orphaned resources costing money.

The cleanup process:

1. Mark resources that were successfully created
2. Run cleanup procedures on partial state
3. If cleanup fails, retry up to 3 times
4. If still failing, alert ops team
5. Manually investigate and resolve

We log everything for post-mortem analysis.

Variable Injection

Customers don’t write infrastructure code. They fill out a form, we generate the required configuration:

cluster_name: "customer-production"
region: "us-east-1"
node_count: 3
instance_type: "t3.medium"
enable_monitoring: true
backup_schedule: "daily"

We validate types, ranges, and dependencies before execution. Invalid config never reaches the provisioning stage.

Cost Estimation

Before provisioning, we estimate monthly costs using cloud provider pricing APIs and our infrastructure module calculations.

“This cluster will cost approximately $450/month.”

Not perfect (data transfer and storage costs are hard to predict), but close enough for budget planning.

What’s Next

Working on:

• Drift detection (alert when manual changes diverge from Terraform)
• Cost optimization recommendations
• Faster provisioning (parallel resource creation where safe)
• Better rollback mechanisms
• Support for more cloud providers

The Runner evolved from “provision clusters” to handling all infrastructure operations. Updates, scaling, configuration changes, teardowns - all go through the Runner now.

These infrastructure patterns represent two years of production learnings, edge cases, and optimizations across multiple cloud providers.

FAQ: Terraform multi-cloud

How do you do Terraform multi-cloud for Kubernetes?

One Runner service executes provider-specific Terraform modules (EKS, GKE, AKS, etc.) with a shared job queue, remote state, and the same API surface for “create / update / destroy cluster.” Customers do not write the Terraform; the platform injects validated variables.

Why not one giant multi-cloud module?

Provider quirks win. AWS, GCP, and Azure disagree on networking, IAM, and timing. Separate modules behind one orchestrator keep failures local and logs readable.

Related writing

• How PipeOps Deploys - BuildKit to Kubernetes after the cluster exists
• The PipeOps Agent - BYOS path onto metal or a VM
• Namespaces Aren’t Isolation - multi-tenant isolation once many customers share a pool