AI Factory

AI Factory · Operations handbook

Deploy, govern
and operate the Factory.

Start with a fictional scenario or prepare a private deployment. Follow access controls, agent work, human decisions and incident recovery.

Overview

In Planning → Portfolio and tasks, connect a Factory objective to its projects and business-unit tasks. The portfolio tracks due dates, ownership, dependencies and evidence.

  1. PlanThe project lead creates the tasks.
  2. AssignThe manager assigns or delegates.
  3. ReviewThe architect reviews technical work.
  4. DecideWorkbench records the sensitive decision.

GitHub Pages presents fictional cases. Compose and Kubernetes can host a private deployment with its own identities, data and model services.

Explore the architecture diagrams and 2D floor plan

Visual atlas · 05 workflows

Five workflows inside the Factory

Concept illustrations of workspaces, with links to demos and operating procedures.

  1. An illuminated path connects a coordination desk to two work areas.

    01 / Orchestration

    From project to subagent

    Assign a task, authorize its start under the business-unit policy, then review the evidence and reported token counts.

  2. The request path leads to a decision table in a review room.

    02 / Decisions

    A traceable human decision

    Have an independent approver verify the case and its revision, then find the outcome in the portfolio.

  3. A branching path connects three server areas.

    03 / Routing

    Test model flows

    Simulate a route, an outage or a privacy constraint. Compare destinations, token counts and estimated costs.

  4. An analysis table shows groups of translucent blocks beside a display.

    04 / Adoption

    The AI Usage Observatory

    Explore adoption by team, tool and period through aggregate views with a privacy threshold.

  5. A checkpoint separates workstations from a server room.

    05 / Operations

    Protect and recover

    Qualify access and network egress, then rehearse recovery by verifying the restored decision journal.

Get started

Project lead or manager

Follow a subtask, its attempt and its evidence in the portfolio.

Understand delegation

Approver or BU lead

Verify the task revision and the decision-maker’s access scope.

Prepare a decision

Platform engineer or operator

Choose a deployment mode, configure identity and plan recovery.

Compare environments

Try the public demo
  1. Open the Factory and select Planning → Portfolio and tasks.
  2. Expand the delegated work summary below an active task.
  3. Compare a reported token count with an unknown value. The examples illustrate the interface.
  4. Open the AI Flow Lab to simulate a model route and see the impact of an outage or security rule.
Qualify a private deployment in staging
  1. Prepare two separate test accounts and synthetic data. Sign in through the SSO proxy.
  2. Create an objective, a project and an active parent task. Delegate a short subtask to an enabled agent, such as summarizing three fictional facts.
  3. Check the state transitions, result and reported counters. Verify that a technical task goes through review.
  4. Submit a sensitive task, decide in the Companion with the authorized account and verify the new state in the portfolio.

Environments and prerequisites

GitHub Pages
A browser is sufficient. The dashboard, demo Companion and Flow Lab provide fictional scenarios. Some preferences, ideas and simulations stay in the browser.
Compose simulation
A Linux Docker engine, Docker Compose and an environment file based on .env.example. The sim profile uses mock providers to check startup and service flows.
Private Compose
A Docker engine dedicated to the same trust boundary, verified images, persistent storage, TLS and OIDC, a business-unit policy, operator-managed secrets and validated model routes.
Private Kubernetes
Helm, a dedicated namespace with restricted admission, a CNI that enforces NetworkPolicy, HTTPS-enforcing Ingress, OIDC and a qualified storage class. The chart installs the core and optional Alloy.

CI uses fictional identities and data. It checks PostgreSQL restart in Compose and API persistence in Kind after replacing a pod and restoring to a second volume. Qualify your own controllers, storage drivers and external services on the target platform.

Reference CI run for Compose and Kubernetes

Identity and access scope

On Compose, configure FACTORY_AUTH=sso, FACTORY_AUTH_MODE=proxy, FACTORY_TRUSTED_PROXIES, FACTORY_ROLE_MAP and FACTORY_BU_POLICY. On Kubernetes, the private profile enforces the trust boundary inside the pod and mounts the business-unit policy from a referenced secret. Keep the dashboard and API on the same origin. Each group that grants a role must carry the relevant business scope.

Factory leadership
A global admin group with scope ["*"] manages objectives and projects.
Business-unit leads
project_lead plans, manager assigns, and architect performs technical review.
Employees and agents
The policy links the verified identity to its employee record or agent catalog entry. The agent must have a valid evaluation.

Each business unit can allow an agent to start, require human authorization or block execution. Renew pending start grants after restarting the API.

Business-unit groups and access controls

Submit ideas at the coffee corner

Subagent runner

The runner handles read and preparation subtasks for configured business units and agents. An active parent can delegate one level deep, with a maximum of four subtasks including cancellations.

  1. Configure a local C3/C4 chat-sensitive use case and an evaluated agent associated with that case in the business unit.
  2. Provision a LiteLLM virtual key restricted to that case, alias and chat routes. Bind its fingerprint to the case in the strict policy. Store the raw key in your secret manager.
  3. Mount the private runner file read-only. On Compose, add docker-compose.portfolio.yml and set FACTORY_PORTFOLIO_RUNNER_SOURCE to its absolute host path. On Kubernetes, reference an existing secret via api.portfolioRunner.existingSecret and configure telemetry.litellmURL according to the chart guide.
  4. Test a denial with a different key or business unit, then run an authorized task with synthetic input.
View the private file structure

These identifiers are fictional. The key must match the virtual key provisioned for the use case.

{
  "version": 1,
  "bindings": [{
    "agent_id": "ag-support-research",
    "unit": "cc-relation",
    "usecase_id": "uc-41",
    "virtual_key": "<virtual key for this use case>"
  }]
}

Each attempt returns text, with a call bounded to 60 seconds and 512 output tokens. Review the output before using it in a business process.

Provision the key and LiteLLM route

Workbench decisions

Enable FACTORY_WORKBENCH_MODE=portfolio and FACTORY_WORKBENCH_CYBER_AUDIT=on on the protected deployment. A submitted sensitive task enters Workbench handoff after technical review when required. The API links the case to the exact task revision. The security audit remains active through the workflow.

An authorized global approver distinct from the other participants opens the Decision Companion. The decision covers the task and its evidence. CRM and ERP operations have their own execution workflows.

Approve
The task becomes Completed and may unblock its dependencies.
Reject
The task becomes Cancelled. Work that requires successful completion remains blocked.
Request changes
The task returns to Ready. A new submission opens a separate decision.

The portfolio shows the outcome and date. Workbench retains the approver and rationale within its own access scope.

Models and connectors

Models
The runner uses the local chat-sensitive route. Private assistance can use a configured internal model. Flow Lab calculates synthetic scenarios in the browser.
CRM
Salesforce, HubSpot and Twenty have read adapters that prepare a Workbench case from an opportunity. Qualify the permissions and required fields in the provider’s sandbox.
ERP
Workbench defines an execution contract for a host-supplied connector. Verify ERP-side idempotency, independent readback and reconciliation before a real write.
MCP
The local MCP server provides the Workbench operations available to its identity. Keep preparation, approval and execution in their respective scopes.

CRM, MCP and Workbench guide · Workbench modes and limits

Monitoring and incidents

Check task state, evidence and token totals in the portfolio. Distinguish provider measurements, reported values and counters that are still unknown. A partial total covers the available counts.

A subtask remains ready

Check dependency completion, agent evaluation, use-case binding and autonomy policy. In human-approval mode, ask the manager to issue the start grant.

An execution stalls or times out

Check LiteLLM, the local model, key binding and network rules. After interruption, the runner can take up to two minutes to release the attempt. The manager reviews the cause and reassigns the task to resume.

The portfolio.runner_attempt_finished and portfolio.runner_tick_failed events support triage. Authorized users can access the detailed result on the task.

A decision takes time to appear

Check that the decision was recorded on the matching case and that the API can read both databases. Let reconciliation run and reload the view. Record identifiers and timestamps for the incident, then verify access scope before sharing a trace.

The readiness check passes and an integration fails

/api/v1/ready checks local paths and API access configuration. Run a functional check against the OIDC provider, model and telemetry destinations. Verify DNS, TLS, network policy and permissions.

Logs do not reach Loki or the SIEM

Check TLS authentication, collector access to the source and network policy. On Kubernetes, verify pods/log access in the intended namespace. Test delivery to your SIEM and access to audit files separately.

Metrics, traces and operating evidence

Prepare for production

  1. Choose the revision to qualify. For a private pilot, start from audited revision 27800aa and build the images locally with the validated configuration. When a signed bundle covers this revision or a later qualified revision, verify the manifest, checksums and image attestations. Use Compose and docker-compose.release-lock.yml from that same bundle. Put every operator policy and Portfolio overlay before the lock file and keep it as the final -f argument. Release v1.1.1 and its images.txt are historical references.
  2. Prepare the tenant. Configure TLS, SSO, the business-unit policy, gateway secrets and the local model. Set FACTORY_DEMO_MODE=never and FACTORY_GUARDRAIL_MODE=block. Keep internal services on private networks.
  3. Validate the rendered Compose configuration. Use the exact target profiles and files. The JSON can contain secrets. Pipe it directly to preflight.
docker compose --project-name ai-factory \
  -f docker-compose.yml -f docker-compose.release-lock.yml \
  --env-file /secure/ai-factory/production.env --profile sso \
  config --format json | python scripts/check_production.py

This command shows the bundle path. For a source build, use the Compose files from the audited revision and pass the validated configuration to the same preflight. Keep the existing project name to retain its volumes. Add -f docker-compose.portfolio.yml and the runner file only when enabling subagents. Add --profile local-gpu for the bundled GPU service or pass --private-gpu-host NAME to preflight for each approved external private DNS name.

  1. Start and check health. Run up -d --no-build with the verified bundle or up -d --build with the audited source. Keep the files, variables and profiles used for preflight. Inspect services with docker compose ps in the same configuration.
  2. Qualify before granting access. Repeat the private walkthrough. Verify cross-unit denial and independent approval, then simulate a model outage and restart after a decision.
  3. Define operations. Assign on-call responsibilities, set budgets and retention, encrypt backups and rehearse restore in isolation.

Compose deployment and bundle verification · Harden the deployment

Kubernetes pilot

  1. Choose the demo profile with local port-forwarding for fictional qualification, or private with HTTPS, OIDC, a business-unit policy and existing secrets.
  2. If the bundle includes the chart, use its Helm archive and values.release.json. For a source rehearsal, stage the chart with python scripts/package_helm.py and use images compatible with the tested revision.
  3. Keep one API replica, a PVC and a replacement strategy without overlap. Plan the upgrade outage and rehearse restore on a separate PVC.
  4. Verify a valid sign-in, cross-unit denial and rejection of forged identity headers. Repeat the checks with your Ingress controller, CNI and OIDC provider.
  5. For Alloy, provision the destination secret and restrict egress to the Kubernetes API, DNS and the approved Loki endpoint. Qualify delivery to your SIEM as well.

The chart installs the AI Factory core and optional Alloy. Operators run the model and telemetry services and connect them as needed for the private profile.

Install the Helm chart · Operate volumes and Alloy

Backup and rollback

Keep the portfolio and Workbench databases and audit files on the private API volume. Suspend writes and processing before taking a consistent backup. Record the matching version and business-unit policy. Copying live SQLite files can miss journal writes.

Restore without replaying operations
  1. Restore to an isolated target with the qualified version and policies. Keep the runner disabled during verification.
  2. Check database integrity, pending decisions and business-unit isolation. Renew any start grants that still apply.
  3. Check active tasks and compare the restore point with confirmed events after the backup. The manager decides how to resume work.
  4. Enable the runner after qualifying a test task and a Workbench decision.

For rollback, disable the runner and retain a snapshot of recent state. For a Kubernetes file archive, save the contents of the volume and preserve the destination PVC root permissions. If the data has migrated, restore a compatible point and reconcile subsequent writes with the operator.

Portfolio and API reference