Build the Core

Designing Your First AI System Architecture

A chatbot is not an AI system. A real AI system is an architecture — with boundaries, validation, logging, and governance. This page walks you through the minimum structure that still works in production.

Think in layers: Interface → Orchestration → Retrieval → LLM → Validation → Logging.

1. Start With the User Interface

This could be:

  • A web app (.NET, React, etc.)
  • A Power App
  • A chatbot inside Teams
  • An internal portal

The UI should:

  • Capture the user’s query
  • Pass authentication context
  • Display structured results (not just text blobs)
The UI is just the entry point. The intelligence lives behind it.

2. The Orchestration Layer (Your Brain)

This is typically your backend:

  • .NET API
  • Node service
  • Azure Function

This layer:

  • Receives the request
  • Determines the task type
  • Applies routing logic
  • Calls retrieval and LLM services

It is where guardrails live.

3. Retrieval Layer (If Using RAG)

If your system uses internal knowledge:

  • Generate embeddings for content
  • Store vectors in a database
  • Retrieve top-N similar chunks
  • Apply permission filters
Retrieval should always respect security boundaries.

4. LLM Integration Layer

This is where you:

  • Construct prompts
  • Inject retrieved context
  • Apply temperature settings
  • Select the appropriate model

You should also:

  • Set output format expectations (JSON, bullets, etc.)
  • Apply max token limits
  • Log model usage

5. Validation Layer (Often Missing)

After the LLM responds:

  • Validate JSON structure
  • Check SQL safety (if generating queries)
  • Ensure required fields exist
  • Apply business rule checks
Never trust raw LLM output in production.

6. Logging and Observability

In enterprise AI systems, you must log:

  • User ID
  • Timestamp
  • Prompt version
  • Model used
  • Tokens consumed
  • Retrieved document IDs

This enables:

  • Audit trails
  • Cost monitoring
  • Answer quality review

7. Security and Permissions

Architecture must integrate:

  • Authentication (Entra ID, OAuth)
  • Role-based access control
  • Tenant isolation
  • Data-level filtering
AI systems inherit your data risks. Design accordingly.

8. The Minimal Viable Architecture

If you’re starting small, your system should still include:

  • Frontend interface
  • Backend orchestration
  • LLM API integration
  • Basic logging
  • Output validation

Add retrieval and advanced governance as you scale.

9. The Big Picture

The difference between:

  • “A cool AI demo”

and:

  • “A production AI system”

is architecture discipline.

AI becomes powerful when wrapped in structure.

Continue the Masterclass

Next: Building a Knowledge Base with SQL + Embeddings.

Next Article Back to Writing