AI Model Integration

DEVELOPER Advanced

This page summarizes how EZ-Console wires AI models, toolsets, skills, streaming, and client tools. For skills storage, APIs, and tool-binding rules, see AI Agent Skills. For browser-side registration of page tools and prompts (useAI, registerPageAI), see React hooks & context → useAI.

Overview

• Multiple providers: OpenAI-compatible APIs and custom providers registered on the backend.
• Service layer: AIChatService resolves the org model, merges RBAC-authorized toolsets, optional skill loader, and runs ClassicChatClient (tool loops, summarization, retries).
• Low-level AIClient: Single round-trip Chat / ChatStream with toolset.ToolSets; the classic client adds iteration, auto-summarization, and client-tool handoff.
• Tool namespaces: ToolSets is map[string]ToolSet; the map key is a prefix (e.g. tool query under key mcp → mcp_query).
• Streaming: Chat HTTP uses SSE with typed events (content, tool_call, client_tool_pending, error).
• Org scope: Models and tool permissions are organization-scoped; see Multi-tenancy.

Built-in OpenAI-compatible provider

Typical config fields (also used for auto-summarization thresholds):

Field	Notes
api_key	Required; stored encrypted
model_id	Required model name
base_url	Optional; default OpenAI API base
organization_id	Optional OpenAI org
max_tokens	Optional; context window hint for summarization

Create models from System → AI Models or POST /api/ai/models. Type definitions for dynamic forms come from GET /api/ai/models/types.

Using AIChatService (recommended)

service.Service embeds *AIChatService. Prefer these entry points so toolsets, skills, and permissions stay consistent.

Non-streaming without toolsets

responses, err := svc.CreateChatCompletionWithoutToolSets(ctx, organizationID, modelID, messages, opts...)

Non-streaming with authorized toolsets (pass nil skill loader if skills are not used)

responses, err := svc.CreateChatCompletion(ctx, organizationID, modelID, messages, skillLoader, opts...)

Streaming

stream, err := svc.CreateChatCompletionStream(ctx, organizationID, modelID, messages, skillLoader, opts...)
defer stream.Close()
for {
    event, err := stream.Recv(ctx)
    if err != nil {
        if err == io.EOF {
            break
        }
        return err
    }
    switch event.EventType {
    case ai.EventTypeContent:
        // event.Content
    case ai.EventTypeToolCall:
        // event.ToolCalls
    case ai.EventTypeClientToolPending:
        // hand off to browser; see usages / AI chat controller
    case ai.EventTypeError:
        // event.Content as error text
    }
}

Common functional options include ai.WithChatMaxIterations, ai.WithChatMaxTokens, ai.WithChatAutoSummarization, ai.WithChatToolSets, and hooks such as ai.WithChatOnMessageAdded. See pkg/clients/ai WithChatOptions.

Low-level: factory and ClassicChatClient

Resolve the default (or specific) model, then use the registry. ai.GetFactory returns a ClassicChatClientFactory wrapping the registered provider factory.

aiModel, err := svc.GetDefaultAIModel(ctx, organizationID)
if err != nil {
    return err
}
factory, ok := ai.GetFactory(aiModel.Provider)
if !ok {
    return fmt.Errorf("unsupported provider: %s", aiModel.Provider)
}
client, err := factory.CreateClient(ctx, organizationID, aiModel.Config)
if err != nil {
    return err
}

messages := []ai.ChatMessage{{Role: model.AIChatMessageRoleUser, Content: "Hello"}}

responses, err := client.Exchange(ctx, messages,
    ai.WithChatMaxIterations(10),
    ai.WithChatAutoSummarization(true),
)

ClassicChatClient implements Exchange / ExchangeStream for full tool-call loops on top of your AIClient.

Implementing a custom provider

AIClient

Implement Chat and ChatStream only (single provider round-trip). Tool iteration is handled by the classic client.

func (c *CustomAIClient) Chat(ctx context.Context, messages []ai.ChatMessage, toolSets toolset.ToolSets) (*ai.ChatMessage, error) {
    // Build provider request including tools from toolSets, return assistant message
    return nil, nil
}

func (c *CustomAIClient) ChatStream(ctx context.Context, messages []ai.ChatMessage, toolSets toolset.ToolSets) (ai.ChatStream, error) {
    return nil, nil
}

AIClientFactory

Implement GetName, GetDescription, GetConfigFields, and CreateClient. Register in init():

const AIModelProviderCustom model.AIModelProvider = "custom"

func init() {
    if err := ai.RegisterFactory(AIModelProviderCustom, &CustomAIClientFactory{}); err != nil {
        panic(err)
    }
}

Optionally implement AIClientFactoryV2 (GetConfigSchema) so the admin UI gets JSON Schema + UI schema instead of legacy ConfigField conversion (pkg/clients/ai/client.go).

Toolsets

Built-in examples include utils (time, sleep, random string) and MCP. Toolsets are configured under system toolsets and exposed to the model according to role AI tool permissions.

To add a custom toolset, implement toolset.ToolSet and toolset.ToolSetFactory, call toolset.RegisterToolSet, and import your package from main so init() runs. Optional ToolSetFactoryV2 supplies JSON Schema like AI providers (pkg/toolset).

Do not use ad-hoc client-side maps for tools; the product stack expects pkg/toolset and service-layer authorization.

Skills and progressive loading

When domains / skill_ids are present, SkillService.CreateSkillLoader returns an ai.SkillLoader that injects metadata and the prefixed tool skill_loader_get_skill_content. Binding between skills and org tools is controlled by system_enable_skill_tool_binding. Details: AI Agent Skills.

Client tools (ui_*)

Browser-executed tools must match ^ui_[a-zA-Z0-9_]+$. The frontend registers them via registerPageAI in AIContext; the stream emits client_tool_pending until results are posted back. Server validation and proxy behavior live in pkg/api/ai.

Chat sessions and SSE

Session CRUD and streaming send live under /api/ai/chat/sessions (see Swag in pkg/api/ai). SSE event: message carries JSON with event_type as in the overview above.

Best practices

1. Prefer AIChatService methods so RBAC, toolsets, and skills stay aligned with the rest of the product.
2. Store secrets with password field types; keys are encrypted at rest.
3. Set max_tokens on models used for long chats so summarization thresholds are meaningful.
4. Cap tool iterations (WithChatMaxIterations) and use streaming for interactive UIs.
5. Chat permission is typically ai:chat:create (see RBAC defaults in the main repo).

Related topics

• AI Agent Skills — SKILL.md, APIs, tool binding
• Multi-tenancy — Organization-scoped AI resources
• API best practices — OpenAPI / client generation
• Authentication & authorization

---

Need help? Ask in GitHub Discussions.