LobeHub
LobeHub

Chat API Internals and Client-Server Interaction

This page is for developers changing or debugging LobeHub source code. It explains internal chat flows, event sequences, and core components.

This is not a public REST API or SDK integration guide. /webapi/chat/[provider], internal tRPC routes, store actions, and ChatService are application implementation details, not a stable external integration contract. Do not infer public authentication, request formats, or compatibility guarantees from the sequence diagram below.

For CLI, public API, or TypeScript SDK integration, start with Choose a Development Path. The snippets below explain internal control flow; they are not standalone client examples. Compare them with the source for your version.

Scope

The diagram below focuses on the client-side Agent loop. See Client-Side vs Server-Side Execution for server execution. Different transports and execution scenarios do not necessarily follow an identical path.

Here, a "preset task" is an internal helper that calls a model directly. It is not a user-created Task or Scheduled automation. For user-facing work records, see Task.

Interaction Sequence Diagram

[call_tool instruction][request_human_* instruction][finish instruction]alt[call_llm instruction]loop[Agent Plan-Execute Loop]sendMessage()Agent decides next instructiongetChatCompletionPOST /webapi/chat/[provider]Initialize ModelRuntimeChat completion requestStream back SSE responseonMessageHandle callbackExecute toolReturn tool resultRequest user interventionUser feedbackonFinish callbackfetchPresetTaskResultSend preset task requestReturn task resultReturn result via callbackCreate GeneralChatAgent + AgentRuntimeContext engineeringBuiltin / MCP / PluginPreset task scenario (bypasses Agent loop)Frontend ClientAgent Runtime LoopChatServiceBackend Chat APIModel RuntimeModel Provider APITool Execution LayerFrontend ClientAgent Runtime LoopChatServiceBackend Chat APIModel RuntimeModel Provider APITool Execution Layer

Main Process Steps

1. Client Initiates Request

After the user sends a message, sendMessage() (src/store/chat/slices/agentRun/actions/entries/conversationLifecycle.ts) creates the user message and assistant message placeholder, then calls executeClientAgent().

2. Agent Runtime Drives the Loop

The client-side Agent chat flow described here is driven by the AgentRuntime.step() loop. Preset tasks described below call the model directly and bypass this loop.

Initialization (src/store/chat/slices/agentRun/actions/transports/client/streamingExecutor.ts):

  1. Resolve agent config (model, provider, plugin list, etc.)
  2. Create the tool registry via createAgentToolsEngine()
  3. Create GeneralChatAgent (the "brain" that decides what to do next) and AgentRuntime (the "engine" that executes instructions)
  4. Bind the shared package executors to client transports via createClientRuntimeExecutors()

Execution Loop:

*.ts
TypeScript
while (state.status !== 'done' && state.status !== 'error') {
  result = await runtime.step(state, nextContext);
  // GeneralChatAgent decides: call_llm → call_tool → call_llm → finish
}

At each step, GeneralChatAgent returns an AgentInstruction based on current state, and AgentRuntime executes it via the corresponding executor:

  • call_llm: Call the LLM (see steps 3-5 below)
  • call_tool: Execute tool calls (see step 6 below)
  • finish: End the loop
  • compress_context: Context compression
  • request_human_approve / request_human_prompt / request_human_select: Request user intervention

3. Frontend Processes LLM Request

When the Agent issues a call_llm instruction, the executor calls ChatService:

  • src/services/chat/index.ts preprocesses messages, tools, and parameters
  • Modules under src/services/chat/mecha/ perform context engineering, including agent config resolution, model parameter resolution, MCP context injection, etc.
  • Calls getChatCompletion to prepare request parameters
  • Uses fetchSSE from the @lobechat/fetch-sse package to send the request to the backend API

4. Backend Processes Request

  • src/app/(backend)/webapi/chat/[provider]/route.ts receives the request
  • Calls initModelRuntimeFromDB to read user's provider config from the database and initialize ModelRuntime
  • A tRPC route apps/server/src/routers/lambda/aiChat.ts also exists for server-side message sending and structured output scenarios

5. Model Call and Response Processing

  • ModelRuntime (packages/model-runtime/src/core/ModelRuntime.ts) calls the respective model provider's API and returns a streaming response
  • Frontend processes the streaming response via fetchSSE and fetchEventSource
  • Handles different types of events (text, tool calls, reasoning, etc.)
  • Passes results back to client through callback functions

6. Tool Calling Scenario

When the AI model returns a tool_calls field in its response, the Agent issues a call_tool instruction. LobeHub supports three types of tools:

Builtin Tools: Tools built into the application, executed directly via local executors.

  • Frontend executes them directly on the client via the invokeBuiltinTool method
  • Includes built-in features like search, DALL-E image generation, etc.

MCP Tools: External tools connected via Model Context Protocol.

  • Frontend calls MCPService (src/services/mcp.ts) via the invokeMCPTypePlugin method
  • Supports three connection modes: stdio, HTTP (streamable-http/SSE), and cloud (Composio)
  • MCP tool registration and discovery is managed through MCP server configuration

Plugin Tools: Legacy plugin system, invoked via API gateway. This system is expected to be gradually deprecated in favor of the MCP tool system.

  • Frontend calls them via the invokeBuiltinTool method
  • Retrieves plugin settings and manifest, creates authentication headers, and sends requests to the plugin gateway

After tool execution completes, results are written to the message and returned to the Agent loop. The Agent then calls the LLM again to generate the final response based on tool results. The tool dispatch logic is in src/store/chat/slices/plugin/actions/pluginTypes.ts.

7. Preset Task Processing

Preset tasks are predefined functions typically triggered when users perform specific actions (bypassing the Agent Runtime loop, calling LLM directly). These tasks use the fetchPresetTaskResult method, which is similar to the normal chat flow but uses specially designed prompt chains.

Execution Timing:

  1. Agent Information Auto-generation: Triggered when users create or edit an agent
    • Agent avatar generation (via autoPickEmoji method)
    • Agent description generation (via autocompleteAgentDescription method)
    • Agent tag generation (via autocompleteAgentTags method)
    • Agent title generation (via autocompleteAgentTitle method)
  2. Message Translation: Triggered when users manually click the translate button (via translateMessage method)
  3. Web Search: When search is enabled but the model doesn't support tool calling, search functionality is implemented via fetchPresetTaskResult

Code Examples:

Agent avatar auto-generation implementation:

*.ts
TypeScript
// src/features/AgentSetting/store/action.ts
autoPickEmoji: async () => {
  const { config, meta, dispatchMeta } = get();
  const systemRole = config.systemRole;

  chatService.fetchPresetTaskResult({
    onFinish: async (emoji) => {
      dispatchMeta({ type: 'update', value: { avatar: emoji } });
    },
    onLoadingChange: (loading) => {
      get().updateLoadingState('avatar', loading);
    },
    params: merge(
      get().internal_getSystemAgentForMeta(),
      chainPickEmoji(
        [meta.title, meta.description, systemRole]
          .filter(Boolean)
          .join(','),
      ),
    ),
    trace: get().getCurrentTracePayload({
      traceName: TraceNameMap.EmojiPicker,
    }),
  });
};

Translation feature implementation:

*.ts
TypeScript
// src/store/chat/slices/translate/action.ts
translateMessage: async (id, targetLang) => {
  // ...omitted code...

  // Detect language
  chatService.fetchPresetTaskResult({
    onFinish: async (data) => {
      if (data && supportLocales.includes(data)) from = data;
      await updateMessageTranslate(id, {
        content,
        from,
        to: targetLang,
      });
    },
    params: merge(
      translationSetting,
      chainLangDetect(message.content),
    ),
    trace: get().getCurrentTracePayload({
      traceName: TraceNameMap.LanguageDetect,
    }),
  });

  // Perform translation
  chatService.fetchPresetTaskResult({
    onMessageHandle: (chunk) => {
      if (chunk.type === 'text') {
        content = chunk.text;
        internal_dispatchMessage({
          id,
          type: 'updateMessageTranslate',
          value: { content, from, to: targetLang },
        });
      }
    },
    onFinish: async () => {
      await updateMessageTranslate(id, {
        content,
        from,
        to: targetLang,
      });
      internal_toggleChatLoading(
        false,
        id,
        n('translateMessage(end)', { id }) as string,
      );
    },
    params: merge(
      translationSetting,
      chainTranslate(message.content, targetLang),
    ),
    trace: get().getCurrentTracePayload({
      traceName: TraceNameMap.Translation,
    }),
  });
};

8. Completion

When the Agent issues a finish instruction, the loop ends, and the onFinish callback is called with the complete response result.

Client-Side vs Server-Side Execution

The Agent Runtime loop execution location depends on the scenario:

  • Client-side loop (browser): Regular 1:1 chat, continue generation, group orchestration decisions. The loop runs in the browser, entry point is executeClientAgent() (src/store/chat/slices/agentRun/actions/transports/client/streamingExecutor.ts)
  • Server-side loop (queue/local): Group chat supervisor agent, sub-agent tasks, API/Cron triggers. The loop runs on the server, streaming events to the client via SSE, entry point is AgentRuntimeService.executeStep() (apps/server/src/services/agentRuntime/AgentRuntimeService/executeStep/index.ts), tRPC route at apps/server/src/routers/lambda/aiAgent.ts

Model Runtime

Model Runtime (packages/model-runtime/) is the core abstraction layer in LobeHub for interacting with LLM model providers, adapting different provider APIs into a unified interface.

Core Responsibilities:

  • Unified Abstraction Layer: Hides differences between AI provider APIs through the LobeRuntimeAI interface (packages/model-runtime/src/core/BaseAI.ts)
  • Model Initialization: Initializes the corresponding runtime instance through the provider mapping table (packages/model-runtime/src/runtimeMap.ts)
  • Capability Encapsulation: chat (streaming chat), models (model listing), embeddings (text embeddings), createImage (image generation), textToSpeech (speech synthesis), generateObject (structured output)

Core Interface:

*.ts
TypeScript
// packages/model-runtime/src/core/BaseAI.ts
export interface LobeRuntimeAI {
  baseURL?: string;
  chat?(
    payload: ChatStreamPayload,
    options?: ChatMethodOptions,
  ): Promise<Response>;
  generateObject?(
    payload: GenerateObjectPayload,
    options?: GenerateObjectOptions,
  ): Promise<any>;
  embeddings?(
    payload: EmbeddingsPayload,
    options?: EmbeddingsOptions,
  ): Promise<Embeddings[]>;
  models?(): Promise<any>;
  createImage?: (
    payload: CreateImagePayload,
  ) => Promise<CreateImageResponse>;
  textToSpeech?: (
    payload: TextToSpeechPayload,
    options?: TextToSpeechOptions,
  ) => Promise<ArrayBuffer>;
}

Adapter Architecture: Through two factory functions — createOpenAICompatibleRuntime and createAnthropicCompatibleRuntime — most providers can be integrated with minimal configuration. Currently supports over 40 model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, etc.), with implementations in packages/model-runtime/src/providers/.

Agent Runtime

Agent Runtime (packages/agent-runtime/) is LobeHub's agent orchestration engine. As described above, it is the core execution engine that drives the entire chat flow.

Core Components:

  • AgentRuntime (packages/agent-runtime/src/core/runtime.ts): The "engine" that executes the Agent instruction loop, supporting call_llm, call_tool, finish, compress_context, request_human_*, etc.
  • GeneralChatAgent (packages/agent-runtime/src/agents/GeneralChatAgent.ts): The "brain" that decides which instruction to execute next based on current state
  • GroupOrchestrationRuntime (packages/agent-runtime/src/groupOrchestration/): Multi-agent orchestration supporting speak / broadcast / delegate / executeTask collaboration modes
  • UsageCounter: Token usage and cost tracking
  • InterventionChecker: Security blacklist for managing agent behavior boundaries