New Mode - OpenMind

This guide walks you through creating a new mode for your robot system.

Project Structure

src/runtime/multi_mode/
├── config.py          # ModeConfig and ModeSystemConfig classes
├── manager.py         # ModeManager for transitions
├── cortex.py          # ModeCortexRuntime for execution
└── hook.py            # Lifecycle hooks

src/runtime/multi_mode/
├── config.py          # ModeConfig and ModeSystemConfig classes
├── cortex.py           # ModeCortexRuntime for Execution

config/
└── your_robot_modes.json5    # Mode configuration file

Configuration

Step 1: Create Configuration File

Create or modify a configuration file (e.g., your_robot_modes.json5) in the /config/ directory.

Step 2: Add Mode Definition

Add your new mode to the modes section of your configuration file. Mode Configuration Options

display_name string — Required

The human-readable name shown in the UI for this mode. Example: “Your New Mode”

description string — Required

Brief description explaining what this mode does and its purpose. Example: “Brief description of what this mode does”

system_prompt_base string — Required

The foundational system prompt that defines the agent’s behavior and purpose in this mode. Example: “You are in [mode name]. Your purpose is…“

hertz float — Required

The frequency (in Hz) at which the agent operates or processes information. Example: 1.0

timeout_seconds integer — Required

Maximum duration (in seconds) before the agent times out during execution. Example: 300

remember_locations boolean — Required

Whether the agent should persist and recall location data across interactions. Example: false

save_interactions boolean — Required

Whether to save conversation history and interactions for this mode. Example: true

agent_inputs array — Required

List of input sources or data types the agent can accept in this mode.

agent_actions array — Required

List of actions or capabilities the agent can perform in this mode.

lifecycle_hooks array — Required

Event handlers triggered at specific points in the agent’s lifecycle (startup, shutdown, etc.).

simulators array — Required

List of simulation environments or tools available to the agent in this mode.

cortex_llm object — Required

Configuration object for the language model powering the agent’s cortex.

Step 3: Configure Input Plugins

Specify which inputs your mode needs:

agent_inputs array — Required

List of input sources or data types the agent can accept in this mode. Each item in the array is an object with the following properties:

type string — Required

The input type identifier. Example: “AudioInput”

config object — Optional

Configuration options specific to this input type. Example: GoogleASRInput

Step 4: Configure LLM (Optional - Can be overwritten for each mode)

Define which LLM needs to be configured:

cortex_llm object — Required

Configures the Cortex LLM behavior.

type string — Required

The LLM provider name. Example: “OpenAILLM”

config object — Optional

Configuration options specific to this LLM type.

agent_name string — Optional

Agent name used in metadata Example: “Spot”

history_length integer — Optional

Number of past messages to remember in the conversation. Example: 10

Step 5: Configure actions

Actions define what your agent can do. You can define movement, TTS or any other actions here.

agent_actions array — Required

Defines actions the agent can execute.

name string — Required

Human-readable identifier for the action. Example: “speak”

llm_label string — Required

Label the model uses to refer to this action. Example: “speak”

implementation string — Optional

This defines the business logic, if any. If there’s none defined, we use default value passthrough. Example: “passthrough”

connector string — Required

Name of the connector. This field is the Python file name defined under actions/action_name/connector. Example: “elevenlabs_tts”

config object — Optional

Configuration options specific to this action.

exclude_from_prompt boolean — Optional

Whether to exclude this action from the LLM prompt. Default: false. Example: [{"name": "speak", "llm_label": "speak", "connector": "elevenlabstts", "config": {}, "exclude_from_prompt": false}]

Step 6: Add Lifecycle hooks (Only required for multi-mode)

A hook is a programmable event point that executes specific actions at key stages. To define lifecycle hooks, you can add the following to your config.

lifecycle_hooks array — Required

Event handlers triggered at specific points in the agent’s lifecycle. Each item in the array is an object with the following properties:

hook_type string — Required

The lifecycle event type. Allowed values: “on_startup”, “on_shutdown”, “on_entry”, “on_exit”, “on_timeout”.

handler_type string — Required

The type of handler to execute. Allowed values: “message”, “command”, “function”, “action”.

handler_config object — Required

Configuration for the handler containing one of: message, command, function, or action as a string property.

priority integer — Optional

Execution priority when multiple hooks exist for the same event.

async_execution boolean — Optional

Whether to execute the handler asynchronously.

timeout_seconds number — Optional

Maximum duration for handler execution.

on_failure string — Optional

Behavior when handler fails. Allowed values: “log”, “ignore”, “abort”.

Step 7: Add Transition Rules (Only required for multi-mode)

Add to the transition_rules section:

transition_rules array — Required

Rules that define how the agent transitions between different modes. Each item in the array is an object with the following properties:

from_mode string — Required

The source mode from which the transition originates.

to_mode string — Required

The target mode to which the agent will transition.

transition_type string — Required

The type of transition mechanism.

trigger_keywords array — Required

List of keywords that trigger this transition. Each item is a string.

priority integer — Required

Priority level for this transition rule when multiple rules match.

cooldown_seconds number — Optional

Minimum time (in seconds) before this transition can be triggered again.

Step 8: Update Default Mode (Optional)

If “new_mode” should be the starting mode, define

"default_mode": "new_mode"

Now, your new mode is ready to be tested. Deploy it directly on your robot or configure it through the docker_compose file!

Input Plugins

LLM Integration

Background

Action Plugins

Modes and Lifecycle

Recipes

​Project Structure

​Configuration

​Step 1: Create Configuration File

​Step 2: Add Mode Definition

​display_name string — Required

​description string — Required

​system_prompt_base string — Required

​hertz float — Required

​timeout_seconds integer — Required

​remember_locations boolean — Required

​save_interactions boolean — Required

​agent_inputs array — Required

​agent_actions array — Required

​lifecycle_hooks array — Required

​simulators array — Required

​cortex_llm object — Required

​Step 3: Configure Input Plugins

​agent_inputs array — Required

​type string — Required

​config object — Optional

​Step 4: Configure LLM (Optional - Can be overwritten for each mode)

​cortex_llm object — Required

​type string — Required

​config object — Optional

​agent_name string — Optional

​history_length integer — Optional

​Step 5: Configure actions

​agent_actions array — Required

​name string — Required

​llm_label string — Required

​implementation string — Optional

​connector string — Required

​config object — Optional

​exclude_from_prompt boolean — Optional

​Step 6: Add Lifecycle hooks (Only required for multi-mode)

​lifecycle_hooks array — Required

​hook_type string — Required

​handler_type string — Required

​handler_config object — Required

​priority integer — Optional

​async_execution boolean — Optional

​timeout_seconds number — Optional

​on_failure string — Optional

​Step 7: Add Transition Rules (Only required for multi-mode)

​transition_rules array — Required

​from_mode string — Required

​to_mode string — Required

​transition_type string — Required

​trigger_keywords array — Required

​priority integer — Required

​cooldown_seconds number — Optional

​Step 8: Update Default Mode (Optional)

Project Structure

Configuration

Step 1: Create Configuration File

Step 2: Add Mode Definition

display_name string — Required

description string — Required

system_prompt_base string — Required

hertz float — Required

timeout_seconds integer — Required

remember_locations boolean — Required

save_interactions boolean — Required

agent_inputs array — Required

agent_actions array — Required

lifecycle_hooks array — Required

simulators array — Required

cortex_llm object — Required

Step 3: Configure Input Plugins

agent_inputs array — Required

type string — Required

config object — Optional

Step 4: Configure LLM (Optional - Can be overwritten for each mode)

cortex_llm object — Required

type string — Required

config object — Optional

agent_name string — Optional

history_length integer — Optional

Step 5: Configure actions

agent_actions array — Required

name string — Required

llm_label string — Required

implementation string — Optional

connector string — Required

config object — Optional

exclude_from_prompt boolean — Optional

Step 6: Add Lifecycle hooks (Only required for multi-mode)

lifecycle_hooks array — Required

hook_type string — Required

handler_type string — Required

handler_config object — Required

priority integer — Optional

async_execution boolean — Optional

timeout_seconds number — Optional

on_failure string — Optional

Step 7: Add Transition Rules (Only required for multi-mode)

transition_rules array — Required

from_mode string — Required

to_mode string — Required

transition_type string — Required

trigger_keywords array — Required

priority integer — Required

cooldown_seconds number — Optional

Step 8: Update Default Mode (Optional)