ContextLattice

Local-first memory orchestration for AI systems with durable writes, multi-sink fanout, retrieval learning loops, and operator-grade controls.

Overview | Architecture | Installation | Integrations | Troubleshooting | Updates

Why Context Lattice

Context Lattice is built for teams running high-volume memory writes where durability and retrieval quality matter more than prompt bloat.

• One ingress contract (/memory/write) with validated + normalized payloads.
• Durable outbox fanout to specialized sinks (Qdrant, Mongo raw, MindsDB, Letta, memory-bank).
• Retrieval orchestration that merges multi-source recall and improves ranking through a learning loop.
• Local-first operation with optional cloud BYO for specific sinks.

Architecture Snapshot

Quickstart

Prerequisites

• Container app requirement: a Compose v2-compatible container runtime is required (docker compose), such as Docker Desktop, Docker Engine, or another runtime that supports Compose v2
• Supported host environments: macOS, Linux, or Windows (WSL2)
• Host machine sized for selected profile (lite vs full) with enough CPU, RAM, and disk
• gmake, jq, rg, python3, curl
• Tested baseline: macOS 13+ with Docker Desktop

1) Configure environment

cp .env.example .env
ln -svf ../../.env infra/compose/.env

2) One-command quickstart (recommended)

gmake quickstart

This command:

• creates .env if missing
• links compose env
• generates MEMMCP_ORCHESTRATOR_API_KEY if missing
• applies secure local defaults
• boots the stack
• runs smoke + auth-safe health checks

3) 60-second verify (recommended)

ORCH_KEY="$(awk -F= '/^MEMMCP_ORCHESTRATOR_API_KEY=/{print substr($0,index($0,"=")+1)}' .env)"

curl -fsS http://127.0.0.1:8075/health | jq
curl -fsS -H "x-api-key: ${ORCH_KEY}" http://127.0.0.1:8075/status | jq '.service,.sinks'

Expected:

• /health returns {"ok": true, ...}
• /status returns service and sink states (with API key)

4) Manual bootstrap (optional)

BOOTSTRAP=1 scripts/first_run.sh

MINDSDB_REQUIRED now defaults automatically from COMPOSE_PROFILES.

5) Other launch profiles

# launch using current COMPOSE_PROFILES from .env
gmake mem-up

# explicit modes
gmake mem-up-lite
gmake mem-up-full
gmake mem-up-core

# persist profile mode for future gmake mem-up
gmake mem-mode-full
gmake mem-mode-core

6) Verify health and telemetry

ORCH_KEY="$(awk -F= '/^MEMMCP_ORCHESTRATOR_API_KEY=/{print substr($0,index($0,"=")+1)}' .env)"

curl -fsS http://127.0.0.1:8075/health | jq
curl -fsS -H "x-api-key: ${ORCH_KEY}" http://127.0.0.1:8075/status | jq
curl -fsS -H "x-api-key: ${ORCH_KEY}" http://127.0.0.1:8075/telemetry/fanout | jq
curl -fsS -H "x-api-key: ${ORCH_KEY}" http://127.0.0.1:8075/telemetry/retention | jq

7) First-run toggles (optional)

scripts/first_run.sh --allow-secrets-storage
scripts/first_run.sh --block-secrets-storage
scripts/first_run.sh --insecure-local

scripts/first_run.sh now enforces secure local-first defaults unless explicitly overridden:

• loopback-only host port binding (HOST_BIND_ADDRESS=127.0.0.1)
• production auth posture (MEMMCP_ENV=production, strict API key requirement)
• private status/docs/webhook endpoints
• secrets-safe writes (SECRETS_STORAGE_MODE=redact)

Security toggles:

• --allow-secrets-storage
• --block-secrets-storage
• --insecure-local (explicit opt-out)

Agent Operator Prompt (Paste Once)

Paste this into any new agent session (ChatGPT app, Claude chat apps, Claude Code, Codex):

You must use Context Lattice as the memory/context layer.

Runtime:
- Orchestrator: http://127.0.0.1:8075
- API key: MEMMCP_ORCHESTRATOR_API_KEY from my local .env

Required behavior:
1) Before planning, call POST /memory/search with compact query + project/topic filters.
2) During long tasks, checkpoint major decisions/outcomes via POST /memory/write.
3) Before final answer, run one more POST /memory/search for recency.
4) Keep writes compact (summary, decisions, diffs), never full transcripts.
5) If memory endpoints fail, continue task and report degraded-memory mode explicitly.

Detailed playbook: docs/human_agent_instruction_playbook.md

External Agent Task Routing (Generic)

Context Lattice can queue and route tasks to external runners (Codex, OpenCode, Claude Code) and still supports internal application workers.

• External-first pattern: set agent to the external runner id (codex, opencode, claude-code, or any custom worker name).
• Internal app workers remain supported: use agent=internal or leave unassigned (agent empty / any) for orchestrator workers.
• Practical default: external runners as primary path, internal workers as fallback/secondary for high-resource systems.

ORCH_KEY="$(awk -F= '/^MEMMCP_ORCHESTRATOR_API_KEY=/{print substr($0,index($0,"=")+1)}' .env)"

# 1) Create a task targeted to any external runner id.
curl -fsS -X POST http://127.0.0.1:8075/agents/tasks \
  -H "content-type: application/json" \
  -H "x-api-key: ${ORCH_KEY}" \
  -d '{
    "title":"summarize deployment notes",
    "project":"default",
    "agent":"codex",
    "priority":3,
    "payload":{
      "action":"memory_search",
      "query":"deployment notes",
      "project":"default",
      "limit":8
    }
  }'

# 2) Runner claims only tasks assigned to its worker id (plus unassigned/any tasks).
curl -fsS -X POST "http://127.0.0.1:8075/agents/tasks/next?worker=codex" \
  -H "x-api-key: ${ORCH_KEY}"

# 3) Runner reports completion.
curl -fsS -X POST http://127.0.0.1:8075/agents/tasks/<TASK_ID>/status \
  -H "content-type: application/json" \
  -H "x-api-key: ${ORCH_KEY}" \
  -d '{"status":"succeeded","message":"completed by external runner","metadata":{"worker":"codex"}}'

Performance Profile

• Sustained write throughput target: 100+ messages/second for typical memory payloads on modern laptop-class hardware.
• Outbox protection: fanout retries, coalescing windows, and target-level backpressure to protect core durability.
• Storage pressure controls: retention runner, low-value TTL pruning, optional snapshot pruning, and external NVMe cold path support.
• Retrieval path: parallel source reads with orchestrator merge/rank loop and preference-learning feedback.

Model Runtime

• Ships with a sane local default (qwen via Ollama).
• Any OpenAI-compatible endpoint can be used when preferred.
• BYO model runtimes supported through:
  • Ollama
  • LM Studio
  • llama.cpp compatible server
  • hosted OpenAI-compatible providers

Security defaults

• SECRETS_STORAGE_MODE=redact redacts secret-like material before memory persistence/fanout.
• SECRETS_STORAGE_MODE=block rejects writes containing secret-like material (422).
• SECRETS_STORAGE_MODE=allow stores write payloads as-is (operator opt-in).
• Compose host bindings default to loopback via HOST_BIND_ADDRESS=127.0.0.1.
• Production strict mode requires MEMMCP_ORCHESTRATOR_API_KEY.

Main branch release gate (v1.0.0)

Enforce PR-only merges on main with CODEOWNERS approval (.github/CODEOWNERS is * @sheawinkler):

scripts/enable_main_branch_protection.sh main 1

If GitHub returns Upgrade to GitHub Pro or make this repository public, switch repo visibility or plan, then rerun the command.

Web 3 Ready

• IronClaw can be enabled as an optional messaging surface without changing the core local-first deployment.
• OpenClaw/ZeroClaw surfaces now run with strict secret-leakage protections by default.
• IronClaw docs and architecture conventions are excellent references for operator-facing completeness.

# optional IronClaw bridge
IRONCLAW_INTEGRATION_ENABLED=true
IRONCLAW_DEFAULT_PROJECT=messaging

# strict secret guard for openclaw/zeroclaw/ironclaw messaging surfaces
MESSAGING_OPENCLAW_STRICT_SECURITY=true

Ingress endpoints:

• POST /integrations/messaging/openclaw
• POST /integrations/messaging/ironclaw
• POST /integrations/messaging/command
• @ContextLattice task create|status|list|approve|replay|deadletter|runtime

API Surface (selected)

• POST /memory/write
• POST /memory/search
• POST /integrations/messaging/command
• POST /integrations/messaging/openclaw
• POST /integrations/messaging/ironclaw
• POST /integrations/telegram/webhook
• POST /integrations/slack/events
• POST /agents/tasks
• GET /agents/tasks
• GET /agents/tasks/runtime
• GET /agents/tasks/deadletter
• POST /agents/tasks/{task_id}/replay
• POST /agents/tasks/recover-leases
• GET /telemetry/memory
• GET /telemetry/fanout
• GET /telemetry/retention
• POST /telemetry/retention/run

Docs Index

• Runbook: docs/onprem_full_runbook.md
• Performance: docs/performance.md
• Retention operations: docs/retention_ops.md
• Storage controls: docs/storage_and_retention.md
• Orchestrator enhancements: docs/orchestrator_enhancements.md
• Launch checklist: docs/launch_checklist.md
• Submission requirements audit: docs/submission_requirements.md
• Human agent instruction playbook: docs/human_agent_instruction_playbook.md
• Public messaging package: docs/public_messaging_package.md
• Legal and licensing: docs/legal/README.md
• New repo migration plan: docs/contextlattice_repo_migration_plan.md

Pre-submit verifier:

gmake submission-preflight
python3 scripts/submission_preflight.py --online

Private/Public Sync Notes

This repository (sheawinkler/ContextLattice) is the primary codebase. Public landing collateral remains mirrored in the dedicated public overview mirror repository.

• Source: docs/public_overview/
• Sync script: scripts/sync_public_overview.sh

License

Business Source License 1.1 with change-date transition to Apache-2.0. See LICENSE and docs/legal/README.md.