A Mac Mini captures the screen, sends it to ClawdBot via API, receives action decisions, and executes them with human-like input simulation — all running autonomously.
This is an autonomous AI agent that plays Old School RuneScape entirely through vision, running 24/7 on a Mac Mini. It doesn't read game memory, inject code, or use any plugins — it simply looks at the screen and decides what to do, just like a human player would.
Every few seconds, the Mac Mini captures a screenshot of the game and sends it to ClawdBot (Anthropic's most powerful AI model), which returns a structured decision: what it sees, what it thinks, and what actions to take. Those actions — mouse clicks, keyboard presses, minimap navigation — are executed with human-like Bezier curve mouse paths and randomized timing.
The agent has persistent memory powered by a vector database. It remembers where it's been, what worked, what failed, and what killed it. These memories are fed back into every decision, so it learns from experience across hundreds of ticks.
Everything streams live to this dashboard — the AI's thoughts, its actions, player stats, a world map tracking its location, screenshots of what it sees, and achievement milestones. You're watching an AI explore Gielinor in real time.
Capture the game window
ClawdBot analyzes the scene
Choose actions from 30+ types
Human-like Bezier mouse paths
Store experience in vector DB
Loop forever, learn always
Real output from the perception-action loop. Every tick, the agent observes, reasons, and acts — here's what that looks like under the hood.
Real-time stats extracted from the game via vision AI. Every tree chopped, every monster slain, every level gained — tracked live.
Every component is engineered for one goal: a self-sufficient agent that plays RuneScape like a human, learns from experience, and never stops.
Sends raw screenshots to ClawdBot. The LLM observes the game world, identifies objects, NPCs, and interfaces, then decides what to do next — pure vision, zero game API access.
ClawdBotMouse movements follow Bezier curves with de Casteljau interpolation, ease-in-out acceleration, random jitter, and occasional hesitation pauses. Indistinguishable from a real human player.
Bezier CurvesChromaDB vector database stores 12 types of memory: observations, combat knowledge, navigation paths, NPC encounters, death learnings, and more. Semantic search retrieves relevant past experiences.
ChromaDB + EmbeddingsTree-based goal system with prerequisites, priority ordering, auto-cascading completion, and retry logic. Goals decompose from "Complete Tutorial Island" into atomic sub-tasks.
Goal TreeThree-level detection: repeated clicks, scene-stuck keywords, and area-stuck monitoring. Automatic recovery nudges force new approaches, exploration, and activity variation.
Self-RecoveryOSRS-themed transparent overlay shows the agent's real-time thoughts, reasoning, and actions. OBS WebSocket integration enables 24/7 autonomous livestreaming with status overlays.
OBS IntegrationClean separation of concerns allows each subsystem to evolve independently. The game loop orchestrates all components through a unified tick-based pipeline.
From Bezier curves to semantic memory, every line is crafted for autonomous gameplay.
def _bezier_points(self, start: Point, end: Point, control_points: int = 2, num_steps: int = 50) -> list[Point]: """Generate points along a Bezier curve from start to end. Creates natural-looking curved mouse trajectories.""" points = [start] # Generate random control points that create a natural arc cps = [start] for _ in range(control_points): mid_x = (start.x + end.x) / 2 mid_y = (start.y + end.y) / 2 dist = math.hypot(end.x - start.x, end.y - start.y) spread = dist * 0.3 cp = Point( int(mid_x + random.uniform(-spread, spread)), int(mid_y + random.uniform(-spread, spread)), ) cps.append(cp) cps.append(end) # De Casteljau's algorithm for smooth Bezier interpolation for i in range(1, num_steps + 1): t = i / num_steps t = self._ease_in_out(t) # Slow start, fast middle, slow end result = self._de_casteljau(cps, t) points.append(result) return points def _ease_in_out(self, t: float) -> float: """Ease-in-out for natural acceleration/deceleration.""" if t < 0.5: return 2 * t * t return -1 + (4 - 2 * t) * t
"""The LLM receives a comprehensive gameplay instruction set.""" OSRS_SYSTEM_PROMPT = """ You are an autonomous OSRS player. Analyze each screenshot and return JSON actions. ## CRITICAL RULES - ALWAYS return 2-3 actions per turn - Your LAST action should be a click_minimap to keep moving - NEVER open the Settings menu — it blocks the game view - FINISH what you start: combat, tree chopping, conversations ## ACTIVITY PRIORITY 1. CHOP TREES — click any tree to gather logs 2. COMBAT — attack nearby creatures, pick up drops 3. TALK TO NPCs — engage in conversation 4. PICK UP ITEMS — bones, coins, weapons, everything! 5. EXPLORE — walk to new towns, castles, bridges ## AVAILABLE ACTIONS - click(x, y) — left-click game coordinates - right_click(x, y) — context menu - click_minimap(x, y) — navigate via minimap - type_text(text) — type in chat - press_key(key) — press keyboard key - click_inventory_slot(slot) — interact with item - rotate_camera(direction, duration) — camera control - wait(min, max) — brief pause """
class MemoryType(Enum): """12 categories of persistent agent memory.""" OBSERVATION = "observation" # What was seen on screen ACTION = "action_result" # Successful action outcomes LOCATION = "location" # Map knowledge & landmarks NPC = "npc" # NPC behavior & dialogue QUEST = "quest" # Quest progress SKILL = "skill" # Training techniques ITEM = "item" # Item properties COMBAT = "combat" # Monster knowledge DEATH = "death" # Learn from mistakes FAILURE = "failure" # Failed attempts NAVIGATION = "navigation" # Path instructions STRATEGY = "strategy" # General approaches def get_context_for_situation(self, query: str) -> str: """Retrieve semantically relevant memories for the LLM.""" results = self.collection.query( query_texts=[query], n_results=settings.MEMORY_TOP_K, # Top 10 matches ) # Format memories as context for the LLM prompt memories = [] for doc, meta in zip(results["documents"][0], results["metadatas"][0]): memories.append(f"[{meta['type']}] {doc}") return "\n".join(memories)
@dataclass class Goal: """Hierarchical goal with prerequisites and retry logic.""" id: str name: str description: str status: GoalStatus # PENDING | ACTIVE | COMPLETED | FAILED priority: int # 1-10, higher = more urgent parent_id: Optional[str] children_ids: list[str] prerequisites: list[str] # Goal IDs that must complete first attempts: int = 0 max_attempts: int = 10 def get_next_goal(self) -> Optional[Goal]: """Select highest-priority goal with met prerequisites.""" candidates = [ g for g in self.goals.values() if g.status == GoalStatus.PENDING and all( self.goals[p].status == GoalStatus.COMPLETED for p in g.prerequisites if p in self.goals ) ] if not candidates: return None return max(candidates, key=lambda g: g.priority) def complete_goal(self, goal_id: str): """Complete a goal. Auto-cascades to parent if all children done.""" goal = self.goals[goal_id] goal.status = GoalStatus.COMPLETED # Check if parent should auto-complete if goal.parent_id and goal.parent_id in self.goals: parent = self.goals[goal.parent_id] if all(self.goals[c].status == GoalStatus.COMPLETED for c in parent.children_ids): self.complete_goal(parent.id) # Recursive cascade
Primary vision LLM
Core application language
Vector database for memory
Input automation
Image processing
Livestream integration
Thought overlay UI
Mathematical operations
24/7 hardware host
pending, active, in_progress, completed, failed, and blocked. Top-level goals ("Complete Tutorial Island") decompose into ordered sub-goals with prerequisites — you can't fight the Combat Instructor before visiting the Mining Instructor. Failed goals retry up to 10 times before permanent failure. Blocked goals are skipped and revisited. When all children complete, the parent auto-completes. Claude can also decompose goals dynamically, breaking down new challenges on the fly. The full tree persists to disk as JSON, surviving crashes and restarts.