Codumentor.
← Back to blog
goalcomplexitymemoryarchitecture

Codumentor's Goals System: Autonomous Long-Lived Tasks Without the Complexity Tax

·ARDINSYS Codumentor

When we set out to build long-lived autonomous tasks into Codumentor, we inherited a common assumption: goals need a dedicated runtime. You’d expect a GoalManager singleton, a custom event loop, maybe a message broker — the usual infrastructure tax for anything that persists beyond a single request.

What we built is fundamentally different. Codumentor goals are collections of scheduled conversations coordinated through a shared filesystem. No custom runtime, no message queue, no persistent process. Just cron, SQLite, and a carefully designed directory structure that serves as both state and communication channel.

Here’s how it works, why we made those choices, and what trade-offs we accepted along the way.

The Problem: Tasks That Outlive Conversations

Chat-based AI assistants are ephemeral by nature. A conversation starts, some work happens, and the context vanishes. But many developer tasks refuse to fit that model:

  • “Keep our codebase documentation synchronized with reality”
  • “Review every PR for documentation gaps, weekly”
  • “Draft a technical blog post every Tuesday and Thursday”

These aren’t one-off requests. They’re continuous intents that need to survive across days, weeks, and months — maintaining state between runs, coordinating multiple specialized workers, and recovering gracefully from failures.

We called them “goals” because that’s what they are: long-term objectives that an autonomous agent pursues incrementally, not single-shot tasks.

The Core Insight: KB as Source of Truth

The architectural decision that shaped everything else came from a simple realization:

The knowledge base is the source of truth. The scheduler is the alarm clock; the KB is the brief.

Every worker receives the same static prompt on every tick — a thin shim pointing to its real instructions on disk:

def build_shim_prompt(*, kb_path: str, worker_name: str, goal_id: str) -> str:
    return (
        f"You are worker `{worker_name}` for goal `{goal_id}`.\n\n"
        f"Your knowledge base is at: `{kb_path}`\n\n"
        f"Read these two files first, in order, then proceed:\n"
        f"  1. `{kb_path}/workers/{worker_name}/prompt.md` "
        f"(your role + instructions)\n"
        f"  2. `{kb_path}/protocol.md` (the goal's protocol)\n\n"
        f"Then act according to those documents. Use the file tools available "
        f"in your environment to read/write inside the KB."
    )

That prompt never changes. But the files it points to — prompt.md and protocol.md — are regular files on disk. Edit them, and the next tick picks up the changes automatically. No rescheduling, no redeployment, no API call.

A human can open a goal’s directory in their editor, tweak a worker’s instructions, and the autonomous system adapts on its next scheduled run. The KB isn’t a database; it’s a shared notebook that workers read and write to.

The On-Disk Layout: Your Shared Notebook

Every goal gets a directory tree that looks like this:

<repo_root>/<storage_path>/<goal_id>/
├── definition.yaml          # Identity: goal_id, name, task, owner, status
├── protocol.md              # Contract: message formats, conventions, KPIs
├── state.yaml               # Bounded cross-tick state: plans, decisions
├── log/
│   └── YYYY-MM-DD.md        # Append-only event log
└── workers/
    └── <worker_name>/
        ├── definition.yaml  # Cron schedule, instructions, capabilities
        ├── prompt.md        # Rendered worker instructions
        ├── inbox/           # Messages from peer workers
        └── outbox/          # Results from this worker's last tick

This is deliberate. Every piece of goal state is a file you can cat, diff, and edit with your favorite text editor. No proprietary binary format, no database dumps.

Atomic Tree Creation

When a goal is created, we don’t write files one by one into the final directory — that would leave a window where readers see a partially-created goal. Instead, we stage everything in a temporary directory and rename it into place:

def create_goal_tree(base_dir: Path, *, definition, workers, template_loader) -> Path:
    # Write into a sibling tempdir, then rename — atomic on POSIX
    staging_dir = Path(tempfile.mkdtemp(prefix=f".{definition.goal_id}.tmp-", dir=str(base_dir)))
    try:
        _write_yaml(staging_dir / "definition.yaml", definition.to_dict())
        _write_text(staging_dir / "protocol.md", render_protocol_seed(...))
        _write_yaml(staging_dir / "state.yaml", {"plan": [], "decisions": [], "baselines": {}})
        (staging_dir / "log").mkdir(...)
        for w in workers:
            # Create inbox/, outbox/, definition.yaml, prompt.md
        os.rename(str(staging_dir), str(final_dir))  # Atomic on POSIX
    except Exception:
        shutil.rmtree(str(staging_dir), ignore_errors=True)
        raise

The os.rename() is atomic on POSIX systems. Either the entire goal tree appears at once, or nothing does. Combined with all-or-nothing schedule registration — we roll back every schedule we’ve created if one fails — the system never sees a half-initialized goal.

Workers Are Scheduled Conversations, Not Subagents

Here’s where the architecture gets interesting. In Codumentor, a “worker” is not a background thread or a subagent spawned by the main agent. It’s a top-level scheduled conversation — the same kind of conversation you’d have with Codumentor in the UI, but triggered by cron instead of a user typing a message.

Each tick creates a brand new conversation. There’s no shared in-memory state between runs — the KB is the only thing that persists. The implications are significant:

  • No shared memory bugs. Two workers can’t corrupt each other’s state because they don’t share any.
  • Natural isolation. Each worker runs in its own conversation context with its own system prompt.
  • Subagent capability preserved. Workers can still spawn one layer of subagents below them, respecting the existing “no recursive spawning” rule.
  • Simple debugging. Every tick is a standalone conversation you can inspect in the UI.

The Scheduler: Adaptive Polling Over Background Threads

The scheduler that fires these conversations doesn’t use a thread-per-schedule approach. Instead, it’s an adaptive poller:

# Simplified scheduler loop
async def _loop():
    while True:
        now = time.time()
        due = await store.get_due(before_ts=now)

        for schedule in due:
            await _fire(schedule)  # Overlap check → advance-then-enqueue

        soonest = await store.get_soonest_next_fire()
        sleep_duration = compute_sleep(soonest)  # Adaptive polling
        await asyncio.sleep(sleep_duration)

Three modes govern how long the scheduler sleeps:

  1. No schedules at all → idle poll interval (30 seconds)
  2. Something is already overdue → minimum poll interval (0.5 seconds)
  3. Next fire is in the future → countdown to that fire time, capped at the idle interval

New schedules can wake the poller early via an event mechanism. This is simpler than maintaining hundreds of timer threads and uses less memory.

Failure Handling Is Conservative

The scheduler takes failures seriously:

  • Exponential backoff: Failed schedules retry after 2^n seconds, capped at 300 seconds.
  • Auto-disable after 5 consecutive failures: A schedule that keeps failing isn’t worth retrying forever.
  • Overlap protection: If a previous run is still executing, the new fire is skipped and the schedule advanced.
  • 24-hour catch-up limit: On restart, schedules overdue by more than 24 hours are skipped entirely.

These aren’t arbitrary limits — they’re the result of watching what happens when autonomous systems go wrong in production.

The Original Design vs. What Shipped

Our initial design envisioned something much more monolithic: a single GoalTool class acting as a subagent manager, with goals as long-running subagent instances. The “KB-First” concept existed, but it was wrapped in a custom runtime.

What actually shipped is leaner:

Original Design What Shipped
Single GoalTool(AgenticTool) class Five separate, explicit tools
Goals as subagent instances Goals as collections of scheduled tasks
Custom GoalManager runtime Reuse existing scheduler infrastructure
onServerReady recovery hook SQLite persistence with catch-up on restart

The gap between design and implementation isn’t a failure — it’s pragmatism. By reusing the existing scheduled_agent_run job handler, we avoided building and maintaining a custom runtime. The trade-off is that workers are “dumber” — they read files instead of receiving in-memory objects — but that’s a feature, not a bug. It makes the system more debuggable and more resilient.

Real-World Example: The Nightshift Pattern

The most compelling demonstration of what goals enable is the Nightshift pattern documented in our knowledge base:

  • Commander (fires at 20:00): Analyzes the codebase, identifies improvement areas, schedules worker tasks.
  • Workers (staggered 20:30–07:00): Each performs one atomic task — fixing a bug, updating docs, refactoring a module — and saves results to the KB.
  • Reporter (fires at 07:30): Reads all worker outputs, compiles a morning briefing for the human team.

Coordination happens through KB documents — a shared notebook, not a chat room. Workers commit to branches but don’t push; human approval gates all changes. This is the “hub-and-spoke swarm” pattern: autonomous agents doing focused work, coordinated by a shared filesystem.

Security Considerations: Why No Host Filesystem Access

One design decision that might seem restrictive: goals must live inside a configured repository. No host-filesystem fallback is allowed.

def resolve_base_dir(...) -> Path:
    # Resolution order:
    # 1. Explicit base_dir (tests only)
    # 2. Plugin arg storage_repo → config.repo_context.repo_root(name)
    # 3. Agentic-memory plugin's repo_name
    # 4. Single-repo mode prefix
    # 5. ValueError — no fallback!

This isn’t a convenience limitation — it’s a security boundary. The worker sandbox translates host paths to /workspace/.... A goal directory outside the repos directory would be unreachable from inside the sandbox, creating a mismatch between what the scheduler thinks exists and what the worker can actually access.

Similarly, goal IDs are validated with a strict regex that rejects path traversal attempts:

_SLUG_RE = re.compile(r"^[a-z0-9](?:[a-z0-9_-]{0,62}[a-z0-9])?$")

def make_goal_id(name: str) -> str:
    base = slugify(name)  # lowercase, hyphen-separated, max 32 chars
    return f"{base}-{uuid.uuid4().hex[:8]}"

The random suffix prevents collisions, while the regex ensures no .., /, or Windows separators can sneak into a goal ID.

What’s Next: The Roadmap

Phase 1 gave us the basics: create, list, inspect, delete, and manually trigger goals. But the roadmap extends further:

Phase Status Capabilities
Phase 1 — Lifecycle ✅ Implemented setup_goal, list_goals, get_goal, delete_goal, run_worker_now
Phase 2 — KB Editing 🚧 Planned goal_kb_read, goal_kb_write, goal_kb_list, update_worker
Phase 2.5 — Tool Narrowing 📋 Designed Restrict worker tool registries to only what they need
Phase 3 — Setup Agent + Orchestrator 🔮 Future AI-assisted goal creation, dynamic worker coordination
Phase 4 — Chain/Fan-Out 🔮 Future Workers triggering other workers, parallel execution patterns

Phase 2 is particularly important: it gives workers the ability to read and write the KB through proper tools rather than relying on raw file system access. This enables better auditing, validation, and eventually, cross-goal communication.

The Takeaway

Codumentor’s goals system proves that you can build sophisticated autonomous task management without building a sophisticated runtime. By treating the knowledge base as the source of truth and reusing existing infrastructure — cron scheduling, SQLite, the filesystem — we got:

  • Simplicity: No custom event loop, no message broker, no persistent processes.
  • Debuggability: Every tick is a standalone conversation; every state change is a file you can diff.
  • Resilience: Atomic writes, rollback on failure, exponential backoff.
  • Extensibility: Edit a worker’s prompt.md and the next tick adapts automatically.

The system isn’t perfect — the lack of in-memory coordination means workers communicate slowly, through files, and the file-based approach has performance implications for high-frequency goals. But for the use cases we designed it for — daily blog drafting, weekly reviews, overnight improvement shifts — the simplicity pays dividends in reliability and maintainability.

The lesson for builders: don’t build a runtime when a well-designed file structure will do. Sometimes the simplest abstraction is the most powerful.

This post is based on the actual implementation in the Codumentor codebase. All code snippets are from production code.