AGENT HARNESS

12 Building Blocks to Master

A Guide to Building AI Agent Systems

Presented by Trinh The Thanh — thanhtt.com

Inspired by Daniel Walsh — The AI Automators

You Already Use AI Agents

These popular tools are all Agent systems under the hood

Claude Code

Coding Agent

Writes, tests, and deploys code autonomously using tools and file system access

Cursor / GitHub Copilot

Code Assistant Agent

Understands your codebase, suggests edits, runs commands, and iterates on feedback

Perplexity

Research Agent

Searches the web, reads sources, synthesizes answers with citations automatically

Devin

Software Engineer Agent

Plans, codes, debugs, and deploys full features end-to-end with minimal input

ChatGPT with Tools

General Purpose Agent

Browses the web, runs code, generates images, and calls APIs to complete tasks

Replit Agent

App Builder Agent

Builds full applications from a description — sets up files, writes code, and deploys

How are these built?  Let's explore the 12 building blocks →

What is an Agent Harness?

Think of it like...

An Agent Harness is like a cockpit for AI.

Just as a pilot's cockpit provides controls, instruments, and safety systems to fly a plane — an Agent Harness provides the framework, tools, and guardrails for an AI agent to complete tasks safely and effectively.

Without the harness, the AI is just a raw brain. With it, the AI becomes a capable, reliable worker.

12 Building Blocks

1

Architecture

2

Planning

3

File System

4

Delegation

5

Tool Calling

6

Memory

7

State Machine

8

Code Exec

9

Context Mgmt

10

Human Loop

11

Validation

12

Agent Skills

The Big Picture

How all 12 building blocks work together

1

Harness Architecture

Choose your design pattern

🛠  Build with:

LangGraph

CrewAI

AutoGen

OpenAI Agents SDK

Semantic Kernel

2

Planning

Think before you act

🛠  Build with:

LangGraph Planner

OpenAI o3/o4

Claude Thinking

Chain-of-Thought

ReAct Pattern

3

File System

The agent's workspace

🛠  Build with:

Docker Volumes

E2B Sandbox

AWS S3

Node.js FS

Google Drive API

4

Delegation & Models

Teamwork makes the dream work

🛠  Build with:

CrewAI Crews

AutoGen Agents

LangGraph SubGraphs

Claude Sub-agents

OpenRouter

5

Tool Calling & Guardrails

Superpowers with safety nets

🛠  Build with:

MCP Protocol

OpenAI Functions

Guardrails AI

NeMo Guardrails

LangChain Tools

6

Memory

Remember and learn

🛠  Build with:

Mem0

Pinecone

ChromaDB

Zep Memory

Redis + pgvector

7

State Machine

Know where you are

🛠  Build with:

LangGraph State

XState

Temporal.io

AWS Step Functions

Inngest

8

Code Execution

Run code safely in a sandbox

🛠  Build with:

E2B Sandbox

Docker

Modal

AWS Lambda

Jupyter Kernel

9

Context Management

Focus on what matters

🛠  Build with:

LlamaIndex

RAG Pipeline

Prompt Caching

Sliding Window

tiktoken

10

Human in the Loop

Humans stay in control

🛠  Build with:

LangGraph Interrupt

Slack Bot API

Retool

Streamlit

Inngest

11

Validation Loop

Check your own work

🛠  Build with:

Pytest / Vitest

LLM-as-Judge

Braintrust Evals

Guardrails AI

Pydantic

12

Agent Skills

Reusable superpowers

🛠  Build with:

Claude Skills

LangChain Toolkits

MCP Servers

OpenAI Plugins

Custom Prompts

Putting It All Together

A well-designed Agent Harness combines all 12 building blocks to create AI systems that are powerful, reliable, and safe.

1

Architecture

→

2

Planning

→

3

File System

→

4

Delegation

→

5

Tool Calling

→

6

Memory

7

State Machine

→

8

Code Exec

→

9

Context Mgmt

→

10

Human Loop

→

11

Validation

→

12

Skills

↓

Start with Architecture → Layer on capabilities → Build towards autonomy

Thank you!