ScallopBot —
your AI assistant, self-hosted.

LoCoMo benchmark evaluation

Evaluated on LoCoMo — a standardized long-conversation memory benchmark with 1,049 QA items across 5 conversations and 138 sessions. Both systems use identical models (Moonshot kimi-k2.5) and embeddings (Ollama nomic-embed-text). The system comprises 367 TypeScript source files (~63,000 lines of code) with 1,560 tests across 95 test files. ScallopBot’s hybrid retrieval with LLM reranking, temporal query detection, and score-gated context achieves F1 0.48 vs OpenClaw’s 0.38 — a 26% relative improvement.

Standardized benchmark with real embeddings (Ollama nomic-embed-text, 768-dim) and real LLM (Moonshot kimi-k2.5). Adversarial gains driven by score-gating and anti-fabrication constraints. Multi-hop gains from memory fusion, NREM dream consolidation, and increased retrieval depth. Temporal gains from date-embedded memories and regex-based temporal query detection. Full cognitive pipeline adds an estimated ~$0.02/day to base conversation cost (see the per-operation table). Design validated against 30 research works from 2023–2026 across six domains.

→ How the OpenClaw memory comparison works · What it costs to run

Small, specialized, local

Two 4B specialists distilled from ScallopBot’s own production traces, then quantized to run on local hardware. A larger model wrote the training labels; the students never trained on their own output. On a fixed, personal toolset they out-pick much larger general models — a narrow result that says nothing about general leaderboards and everything about what a specialist learns from real traces. Weights, LoRA adapters, and the full method are on Hugging Face.

114 tool-calling turns and 33 memory cases, all held out of training, same harness for every model, thinking disabled. The 35B MoE’s raw memory agreement is higher (0.88) but it returns valid structure only 57.6% of the time, so its usable score is parse-gated. Tool-calling is the clear win; memory is parity with the paid teacher. Training data was anonymized before fine-tuning, and both models, their adapters, and the recipe are public.

Self-improvement, on a leash

An optional, default-off loop distills recurring multi-tool workflows into documentation-only procedure files. Candidates are proposed from a training split and scored against a held-out split; a candidate is promoted only if it beats the frozen baseline by a required margin — a gate that cannot be disabled. Every promotion is recorded in a versioned ledger with a snapshot of the prior version, and a watchdog automatically reverts a promotion that accumulates failures. Unused machine-authored skills are recoverably archived. Machine-authored executable scripts are rejected outright — only documentation is ever written.

The loop ships disabled and stays disabled until you turn it on. Scoring is an A/B comparison judged by an LLM on a held-out split that is disjoint from the split candidates were proposed from. Skills that go unused are archived rather than deleted, and archived skills can be restored. The promotion margin is configurable upward, but the gate itself cannot be switched off.