Skip to main content

System Prompts Architecture

This document provides a comprehensive overview of all system prompt mechanisms in Bike4Mind, their sources, processing order, and potential interaction points where prompt leakage might occur.

Overview​

System prompts in Bike4Mind come from multiple sources and are processed in a specific priority order. Understanding this architecture is crucial for debugging prompt leakage issues and maintaining proper context isolation.

Architecture Diagram​

flowchart TB
    subgraph "System Prompt Sources"
        AS[Admin Settings<br/>Global System Files]
        US[User System Files<br/>Personal Settings]
        PS[Project System Files<br/>Project-specific]
        HC[Hard-coded Prompts<br/>ChatCompletion.ts]
    end

    subgraph "Priority & Processing"
        PP[Priority Processing<br/>1. Global Admin<br/>2. Project<br/>3. User<br/>4. Hard-coded]
        DM[Deduplication<br/>Mechanism]
    end

    subgraph "Chat Completion Pipeline"
        CC[ChatCompletion.ts]
        PF[ProjectFeature]
        FM[fabFilesToMessages]
        BM[buildAndSortMessages]
    end

    subgraph "UI Management"
        ASU[Admin Settings UI]
        USP[User System Prompts Tab]
        PSM[Project System Prompt Modal]
    end

    AS -->|SystemFiles setting| CC
    US -->|user.systemFiles| CC
    PS -->|project.systemPrompts| PF
    HC -->|Direct injection| CC

    CC --> PP
    PF --> PP
    PP --> DM
    DM --> FM
    FM --> BM

    ASU -->|Manages| AS
    USP -->|Manages| US
    PSM -->|Manages| PS

    BM -->|Final Context| LLM[LLM Model]

    style AS fill:#ff9999
    style US fill:#99ccff
    style PS fill:#99ff99
    style HC fill:#ffcc99
    style PP fill:#ff6666
    style DM fill:#6666ff

Code Locations Reference​

Key Files​

  • ChatCompletion.ts: Main processing logic (lines 355-370)
  • ChatCompletionFeatures.ts: Project system prompts (lines 254-297)
  • SystemPromptsTab.tsx: User system prompt UI
  • SystemPromptModal.tsx: Project system prompt UI
  • buildAndSortMessages (utils): Final message assembly

System Prompt Sources​

1. Admin Settings Global System Files​

Location: Admin Settings → SystemFiles setting
Storage: AdminSettings collection
Access: getSettingsValue('SystemFiles', adminSettings)

// In ChatCompletion.ts (lines 355-358)
const globalSystemFileIds =
  (getSettingsValue('SystemFiles', defaultAdminSettings) || undefined)
    ?.split(',')
    .map((id: string) => id.trim()) ?? [];

Characteristics:

  • Comma-separated list of file IDs
  • Applied to ALL users globally
  • Managed by administrators only
  • Used for core grammar rules, universal guidelines

2. User-Level System Files​

Location: User Profile → System Prompts Tab
Storage: user.systemFiles array
Structure:

interface ISystemFileEntry {
  fileId: string;
  enabled: boolean;
  systemPriority?: number;
}

Processing:

// In ChatCompletion.ts
const enabledSystemFileIds = (this.user.systemFiles ?? [])
  .filter(file => file.enabled)
  .map(file => file.fileId);

Characteristics:

  • User-specific system prompts
  • Can be enabled/disabled individually
  • Have priority ordering
  • Stored in user document

3. Project-Level System Prompts​

Location: Project Settings → System Prompts
Storage: project.systemPrompts array
Component: ProjectFeature in ChatCompletionFeatures.ts

// In ChatCompletionFeatures.ts (lines 254-297)
const allSystemPromptFileIds = this.project.systemPrompts
  .filter(prompt => prompt.enabled)
  .map(prompt => prompt.fileId);

Characteristics:

  • Project-specific instructions
  • Only active when working within a project context
  • Can be toggled on/off per project
  • Include project notebooks and summaries

4. Hard-coded System Prompts​

Location: Various places in code
Examples:

  • Default system messages in buildAndSortMessages
  • Feature-specific context messages
  • Agent-specific instructions

Processing Pipeline​

Complete Collection Flow​

sequenceDiagram
    participant U as User
    participant S as Session
    participant CC as ChatCompletion
    participant AS as Admin Settings
    participant US as User Settings
    participant PS as Project Settings
    participant FF as FabFiles Service
    participant BSM as buildAndSortMessages

    Note over U,BSM: System Prompt Collection Flow

    U->>S: Start chat in context
    S->>CC: Process message
    
    par Global System Files
        CC->>AS: Get SystemFiles setting
        AS-->>CC: "file1,file2,file3"
        CC->>CC: Parse & trim file IDs
    and User System Files
        CC->>US: Get user.systemFiles
        US-->>CC: [{fileId, enabled, priority}]
        CC->>CC: Filter enabled files
    and Project System Files
        CC->>PS: Get project.systemPrompts
        PS-->>CC: [{fileId, enabled}]
        CC->>CC: Filter enabled prompts
    and Session Knowledge
        CC->>S: Get session.knowledgeIds
        S-->>CC: [fileId1, fileId2, ...]
    end

    CC->>CC: Combine all file IDs<br/>[...global, ...user, ...project, ...session]
    
    Note over CC: ⚠️ No deduplication here!
    
    CC->>FF: fabFilesToMessages(allFileIds)
    FF-->>CC: Convert files to messages
    
    CC->>BSM: Build final message array
    
    Note over BSM: Messages combined in order:<br/>1. Previous messages<br/>2. Feature context<br/>3. System prompts<br/>4. User input
    
    BSM-->>CC: Final messages for LLM

    Note over CC,BSM: Potential Leakage Points:<br/>1. Session.knowledgeIds persists<br/>2. No file ID deduplication<br/>3. Project context not cleared<br/>4. Features accumulate context

Step 1: Collection Phase​

flowchart LR
    A[Start] --> B[Collect Global System Files]
    B --> C[Collect User System Files]
    C --> D[Collect Project System Files]
    D --> E[Collect Feature Context]
    E --> F[Merge All File IDs]

Step 2: File Processing​

All system files are processed through fabFilesToMessages:

// Combined file IDs from all sources
const allSystemFileIds = [
  ...sessionFabFileIds,      // Files attached to session
  ...messageFileIds,         // Files in current message
  ...enabledSystemFileIds,   // User's enabled system files
  ...globalSystemFileIds     // Global admin system files
];

// Process all files
const { promptMessages, convertedFabFiles } = await this.fabFilesToMessages(
  allSystemFileIds,
  quest,
  embeddingFactory,
  message,
  maxTokens,
  modelInfo
);

Step 3: Message Building​

The buildAndSortMessages function combines all messages with proper ordering:

const messages = buildAndSortMessages(
  previousMessages,           // Historical conversation
  [
    ...questMasterMessages,   // QuestMaster context
    ...mementosMessages,      // Mementos context
    ...projectMessages,       // Project system prompts
    ...urlMessages,           // URL content
    ...fabMessages,           // All system files
  ],
  userMessages,               // Current user input
  contextWindow,
  adminSettings,
  historyCount
);

Priority and Deduplication​

Priority Order (Highest to Lowest)​

  1. Global Admin System Files - Core rules, grammar
  2. Project System Prompts - Project-specific guidelines
  3. User System Files - Personal preferences
  4. Session Knowledge Files - Context-specific files
  5. Feature Context - Mementos, QuestMaster, etc.

Deduplication Mechanism​

Current Issue: The system doesn't properly deduplicate file IDs, which can lead to:

  • Same system prompt being included multiple times
  • System prompts from one context appearing in another
  • Accumulation of prompts across sessions

Problem Areas:

// Potential duplication in ChatCompletion.ts
const allFileIds = [
  ...sessionFabFileIds,     // May contain duplicates
  ...messageFileIds,        // May overlap with session files
  ...enabledSystemFileIds,  // May overlap with global
  ...globalSystemFileIds    // May be duplicated in project
];

Potential Leakage Points​

1. Session State Persistence​

Sessions maintain knowledgeIds array that persists:

// Session may accumulate system file IDs over time
session.knowledgeIds.push(...newFileIds);

2. Project Context Switching​

When switching between projects, system prompts may not be properly cleared:

// ProjectFeature adds all project system prompts
const projectFileIds = uniq([
  ...allSystemPromptFileIds,
  ...this.project.fileIds,
  ...notebookFileIds,
  ...notebookSummaryFileIds,
]);

3. Feature Context Accumulation​

Features like Mementos and QuestMaster add context that persists:

// Mementos are loaded based on tier
const hotMementos = allMementos.filter(
  memento => memento.tier === 'hot'
);

Debugging System Prompt Leakage​

1. Enable Verbose Logging​

Add logging to track system prompt sources:

logger.info('System Prompt Sources:', {
  global: globalSystemFileIds,
  user: enabledSystemFileIds,
  project: projectSystemPromptIds,
  session: session.knowledgeIds
});

2. Check for Duplicates​

const allFileIds = [...]; // Combined file IDs
const uniqueFileIds = [...new Set(allFileIds)];
logger.info(`File ID deduplication: ${allFileIds.length} → ${uniqueFileIds.length}`);

3. Monitor Context Size​

quest.promptMeta.context.systemPromptCount = systemPromptMessages.length;
quest.promptMeta.context.systemPromptTokens = calculateTokens(systemPromptMessages);

Best Practices​

1. Clear Separation of Concerns​

  • Global: Universal rules only (grammar, safety)
  • Project: Project-specific guidelines
  • User: Personal preferences and custom instructions
  • Session: Temporary context files only

2. Proper Cleanup​

// Clear session knowledge when switching contexts
session.knowledgeIds = session.knowledgeIds.filter(
  id => !systemFileIds.includes(id)
);

3. Explicit Scoping​

// Use explicit scoping for system prompts
const scopedSystemPrompts = {
  global: globalSystemFileIds,
  project: projectId ? projectSystemPrompts : [],
  user: enabledUserSystemFiles,
  session: sessionSpecificFiles
};

Common Issues and Solutions​

Issue 1: System Prompts from Previous Sessions​

Symptom: Instructions from previous conversations appear unexpectedly
Cause: Session knowledgeIds not properly cleared
Solution:

// Clear system file IDs when starting new context
const nonSystemFileIds = session.knowledgeIds.filter(
  id => !isSystemFile(id)
);

Issue 2: Project Prompts in Non-Project Context​

Symptom: Project-specific instructions appear outside project
Cause: Project context not properly cleared
Solution:

// Only load project prompts when projectId is present
if (projectId && project) {
  // Load project system prompts
}

Issue 3: Duplicate System Prompts​

Symptom: Same instruction appears multiple times
Cause: No deduplication of file IDs
Solution:

// Deduplicate before processing
const uniqueFileIds = [...new Set(allFileIds)];

Monitoring and Metrics​

Add these metrics to track system prompt behavior:

quest.promptMeta.systemPrompts = {
  sources: {
    global: globalSystemFileIds.length,
    user: enabledSystemFileIds.length,
    project: projectSystemPrompts?.length || 0,
    session: sessionSystemFiles.length
  },
  totalUnique: uniqueFileIds.length,
  duplicatesRemoved: allFileIds.length - uniqueFileIds.length,
  tokenCount: systemPromptTokens
};

Debugging Commands​

1. Add Debug Logging to ChatCompletion.ts​

// Add after line 358 in ChatCompletion.ts
logger.info('🔍 System Prompt Debug:', {
  globalSystemFiles: globalSystemFileIds,
  userSystemFiles: enabledSystemFileIds,
  sessionKnowledgeIds: session.knowledgeIds,
  messageFileIds: messageFileIds,
  projectId: projectId,
  allFileIdCount: [...sessionFabFileIds, ...messageFileIds, ...enabledSystemFileIds, ...globalSystemFileIds].length,
  uniqueFileIdCount: new Set([...sessionFabFileIds, ...messageFileIds, ...enabledSystemFileIds, ...globalSystemFileIds]).size
});

2. Track System Prompt Sources in fabFilesToMessages​

// Modify fabFilesToMessages to track sources
const fileSourceMap = new Map<string, string>();
globalSystemFileIds.forEach(id => fileSourceMap.set(id, 'global'));
enabledSystemFileIds.forEach(id => fileSourceMap.set(id, 'user'));
// ... etc

logger.info('📁 File Sources:', Object.fromEntries(fileSourceMap));

3. Add Prompt Meta Tracking​

// Add to quest.promptMeta
quest.promptMeta.systemPromptSources = {
  global: globalSystemFileIds,
  user: enabledSystemFileIds,
  project: projectSystemPromptIds || [],
  session: session.knowledgeIds.filter(id => isSystemFile(id))
};

Immediate Fixes for System Prompt Leakage​

Fix 1: Implement Deduplication​

// In ChatCompletion.ts, after collecting all file IDs
const allFileIds = [
  ...sessionFabFileIds,
  ...messageFileIds,
  ...enabledSystemFileIds,
  ...globalSystemFileIds
];

// Deduplicate
const uniqueFileIds = [...new Set(allFileIds)];

logger.info(`📊 Deduplication: ${allFileIds.length} files → ${uniqueFileIds.length} unique files`);

Fix 2: Clear Project Context​

// In ProjectFeature.getContextMessages
if (!projectId || !this.project) {
  return []; // Don't include project prompts if not in project context
}

Fix 3: Filter Session Knowledge​

// When loading session knowledge, filter out system files
const sessionKnowledgeIds = session.knowledgeIds.filter(id => {
  const isSystemFile = 
    globalSystemFileIds.includes(id) ||
    enabledSystemFileIds.includes(id) ||
    (projectSystemPromptIds && projectSystemPromptIds.includes(id));
  
  return !isSystemFile;
});

Recommendations​

  1. Implement Proper Deduplication: Add a deduplication layer before processing files
  2. Add Source Tracking: Tag each system prompt with its source for debugging
  3. Implement Scope Isolation: Ensure prompts from one scope don't leak to another
  4. Add Cleanup Hooks: Clear system prompts when context changes
  5. Enable Debug Mode: Add verbose logging for system prompt processing
  6. Add System Prompt Viewer: Create a debug UI to view active system prompts
  7. Implement Version Control: Track changes to system prompts over time