We all use CI/CD pipelines every single day to build and deploy our code. But honestly, how many of us actually know how they work under the hood?
To find out, I decided to build a minimal, high-performance CI execution engine from scratch. I wanted to understand the hard parts of distributed infrastructure, Docker internals, and sandbox security.
How We Think About Pipelines vs. How They Actually Run
When we write configurations for GitHub Actions, GitLab CI, or Jenkins, we define a simple step-by-step workflow:
Build the project.
Run the linter.
Run the test cases.
If everything passes, deploy to staging and then to production.
It looks simple on paper. But the real challenge is: How do we run these steps concurrently (at the same time) without breaking dependencies?
To solve this parallel execution problem, I built a DAG (Directed Acyclic Graph) engine scheduler. Inside the scheduler, I implemented Kahn’s Algorithm for Topological Sorting. This allows the engine to analyze the workflow graph, find out which jobs depend on each other, and automatically run independent jobs in parallel.
If a job can run in parallel, our engine runs it immediately.
Securing the Engine: Why Standard Docker Isn't Enough
When you run untrusted code in a CI pipeline, security is a massive concern.
By default, Docker uses the runc runtime. This means the containers share the host machine's Linux kernel directly. If a user runs a malicious pipeline script, they could potentially escape the container, make direct system calls (syscalls) to the host kernel, and crash or take over the entire server.
To fix this, I integrated gVisor into the engine using the runsc runtime.
gVisor acts as an application kernel virtualization layer. It intercepts all unprivileged system calls from the container. If a pipeline script tries to do something dangerous, gVisor blocks it, completely shielding our core host server from multi-tenant security risks or execution panics.
Key Takeaways
Building this project taught me a huge amount about the low-level Docker API, container lifecycles, and how crucial kernel isolation is for modern infrastructure. It proves why learning "why" code works from first principles is so much better than just making it work.
The project is fully open-source and written 100% in Go. It even includes a zero-cache integration test suite to validate concurrent jobs under realistic conditions.
If you like this architecture or want to look at the implementation patterns, check out the repository and drop a star!
