Sysadmin Hell

Building a dark software factory

This article is work in progress, I post here as I go.

I really enjoy the fact, that Google is silent on how to build a dark software factory. I mean query all you like and you get crickets. My journey started from learning about the Attractor design - https://github.com/strongdm/attractor

Steepish learning curve, but the components are clear. My first instinct was to burn some credits and do the reference implementation, but I found few guys already beat me to it, and there are well built machinery one can employ. Meet Fabro - https://fabro.sh/

That looks like a well engineered engine, but it comes with docs but no cookbooks, so I have to figure it out all on my own.

But to implement all the factory machinery, I have to first ask myself, what does my SDLC actually involve in 2026, which routines I follow, which workflows I perform myself with or without agents. Only once I formalised that, I can actually define my machinery.

My main objective is to build a machine that would satisfy the axiom that building code is cheap, but operator's / maintainer's attention is expensive. It's not to "build software as I sleep", it is designed to harness the most out of both mechanical and agentic work, whilst keeping my attention only on the details that matter.

Sysadmin

I have configured Fabro on the VPS, as well as CLI clients on my Laptops. So far it isn't clear on how am I going to drive it, but one thing for certain - reusable workflow machinery deserves its own repository.

Workflows

https://github.com/ruslanstarikov/factory-workflows - that's my repo for workflows I have created a very basic workflow - implement feature. It is my first, although, very basic piece of re-usable machinery.

digraph ImplementTask {
    graph [
        goal="Implement the requested software change"
    ]

    rankdir=LR

    start [shape=Mdiamond, label="Start"]

    implement [
        label="Implement",
        prompt="@prompts/implement.md",
        reasoning_effort="high"
    ]

    verify [
        label="Verify",
        shape=parallelogram,
        script="",
        retry_policy="none"
    ]

    exit [shape=Msquare, label="Exit"]

    start -> implement -> verify -> exit
}

And Prompt:

You are the implementation worker.

Implement the requested goal in the current repository.

Before changing code, inspect the repository and understand the relevant
architecture, conventions, tests, and existing implementation.

Make the smallest coherent change that fully satisfies the goal.

Follow the project's existing conventions rather than introducing new
patterns unnecessarily.

Add or update tests when the change requires them.

You may run targeted tests and other checks while working, but do not attempt
to bypass, disable, weaken, or modify the final verification command.

Do not create or merge a pull request.

When finished, summarize:

- what you changed
- why
- tests or checks you ran
- anything that remains uncertain

toml

_version = 1

[workflow]
graph = "workflow.fabro"

I now going to run it on one of my local repositories as such:

fabro run \
~/PhpstormProjects/factory-workflows/.fabro/workflows/implement-task/workflow.toml \
--goal "Add a health check endpoint that returns JSON with status ok" \
-I verify_command="php artisan test"

Two problems are already evident.

  1. No php on fabro workspace. I'll need to solve for that.
  2. Workflow still passed with no tests. I'll have to harden that too.

Environments

For php (or any other language) there is an environment provision in Fabro. Works real easy, all you have to do is to supply the docker container with everything you need, maybe setup some .env variables. Few caveats, you DO need git installed on the container for it to clone the repo and do the work. Worst case you'll need to build and publish your own container, but I ended up using - pionl/docker-php-laravel-ci:8.4-node-22.

Great, that took my first workflow end to end.

Goals

Second problem - we need to make sure the workflow does not pass if the tests fail. That is achievable at the goal_gate level on verify node:

    verify [
        label="Verify",
        shape=parallelogram,
        script="bash -lc ",
        retry_policy="none",
        goal_gate=true,
        on_failure="exit"
    ]

Checkpoint 1

Implement feature kinda works. I can still see a few problems with it.

  • Slow. For such a rudimentary feature it still took 10 minutes to build.
  • Funky model configuration. I configured OpenRouter from GUI, it's defaulting to Claude Sonnet without me asking. I need to find out on how to configure this properly. Also a research point, figure out how the model stylesheets work.
  • I get tons of unsupported_control warnings in my log. I need to figure this out too.