think-bigger/docs/plans/technical-implementation/backend-architecture.md

Backend Architecture Implementation Plan

This document provides detailed technical specifications for implementing the backend architecture of the Advanced Second Brain PKM System.

Architecture Overview

The backend follows a modular, service-oriented architecture designed for scalability, maintainability, and clear separation of concerns.

┌─────────────────────────────────────────────────────────────┐
│                    API Layer (FastAPI)                      │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │                 Service Layer                           │ │
│  │  ┌─────────────┬─────────────┬─────────────┬──────────┐ │ │
│  │  │ File System │   Dana      │ Knowledge   │ Embedding│ │ │
│  │  │  Service    │  Runtime    │   Graph     │ Service  │ │ │
│  │  └─────────────┴─────────────┴─────────────┴──────────┘ │ │
│  └─────────────────────────────────────────────────────────┘ │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │                 Data Access Layer                       │ │
│  │  ┌─────────────┬─────────────┬─────────────┬──────────┐ │ │
│  │  │   Neo4j     │   Vector    │   File      │  Cache   │ │ │
│  │  │  Graph DB   │   Store     │  System     │  Layer   │ │ │
│  │  └─────────────┴─────────────┴─────────────┴──────────┘ │ │
│  └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Core Components

1. API Layer (FastAPI)

Implementation Details

• Framework: FastAPI with Pydantic models
• Version: API v1 with semantic versioning
• Documentation: Auto-generated OpenAPI/Swagger docs
• Authentication: API key-based (future: JWT)
• CORS: Configured for frontend origins
• Rate Limiting: Basic implementation with Redis (future)

Key Endpoints

# File System Endpoints
GET    /api/v1/domains/{domain_id}/files          # List domain files
GET    /api/v1/files/{file_id}                    # Get file metadata
GET    /api/v1/files/{file_id}/content            # Get file content
POST   /api/v1/files/{file_id}/process            # Trigger processing

# Agent Endpoints
GET    /api/v1/domains/{domain_id}/agent          # Get agent config
PUT    /api/v1/domains/{domain_id}/agent          # Update agent config
POST   /api/v1/domains/{domain_id}/agent/execute  # Execute agent
POST   /api/v1/domains/{domain_id}/agent/test     # Test agent code

# Knowledge Graph Endpoints
GET    /api/v1/domains/{domain_id}/graph          # Get graph data
POST   /api/v1/domains/{domain_id}/graph/query    # Query graph
PUT    /api/v1/domains/{domain_id}/graph/nodes    # Update nodes
PUT    /api/v1/domains/{domain_id}/graph/edges    # Update edges

# Search Endpoints
POST   /api/v1/search/semantic                    # Semantic search
POST   /api/v1/search/hybrid                      # Hybrid search
GET    /api/v1/search/suggestions                 # Search suggestions

# Orchestrator Endpoints
POST   /api/v1/orchestrator/query                 # Cross-domain query
GET    /api/v1/orchestrator/status                # Orchestrator status
POST   /api/v1/orchestrator/agents                 # Get available agents

Error Handling

• Standardized error responses with error codes
• Detailed error logging with correlation IDs
• Graceful degradation for service failures
• Client-friendly error messages

2. Service Layer

File System Service

Responsibilities:

• Monitor domain directories for changes
• Handle file I/O operations securely
• Manage file metadata and indexing
• Coordinate document processing

Key Classes:

class FileSystemService:
    def __init__(self, config: FileSystemConfig)
    def watch_directory(self, path: str) -> AsyncGenerator[FileEvent, None]
    def get_file_metadata(self, file_path: str) -> FileMetadata
    def read_file_content(self, file_path: str, offset: int = 0, limit: int = -1) -> bytes
    def process_file(self, file_path: str) -> ProcessingResult

Configuration:

@dataclass
class FileSystemConfig:
    watch_paths: List[str]
    excluded_patterns: List[str]
    max_file_size: int = 100 * 1024 * 1024  # 100MB
    processing_timeout: int = 300  # 5 minutes
    concurrent_workers: int = 4

Dana Runtime Service

Responsibilities:

• Execute Dana code in sandboxed environment
• Manage agent lifecycle
• Handle Dana REPL interactions
• Provide debugging and error reporting

Key Classes:

class DanaRuntimeService:
    def __init__(self, config: DanaConfig)
    def execute_code(self, code: str, context: Dict[str, Any]) -> ExecutionResult
    def start_repl_session(self, agent_id: str) -> ReplSession
    def validate_code(self, code: str) -> ValidationResult
    def get_agent_status(self, agent_id: str) -> AgentStatus

Sandboxing Strategy:

• Restricted execution environment
• Limited standard library access
• Resource usage monitoring
• Timeout enforcement
• Error isolation

Knowledge Graph Service

Responsibilities:

• Manage graph database operations
• Handle node/edge CRUD operations
• Execute graph queries and traversals
• Provide graph analytics and visualization data

Key Classes:

class KnowledgeGraphService:
    def __init__(self, config: GraphConfig)
    def create_node(self, node_data: NodeData) -> NodeId
    def create_edge(self, edge_data: EdgeData) -> EdgeId
    def query_graph(self, query: GraphQuery) -> QueryResult
    def get_subgraph(self, center_node: NodeId, depth: int) -> GraphData
    def calculate_centrality(self, method: str) -> CentralityResult

Graph Schema:

// Node Types
(domain:Domain {id: string, name: string, path: string})
(document:Document {id: string, title: string, type: string})
(chunk:TextChunk {id: string, content: string, embeddings: list})
(concept:Concept {id: string, name: string, strength: float})
(agent:Agent {id: string, domain_id: string, dana_code: string})

// Edge Types
(:domain)-[:CONTAINS]->(:document)
(:document)-[:HAS_CHUNK]->(:chunk)
(:chunk)-[:CONTAINS_CONCEPT]->(:concept)
(:domain)-[:HAS_AGENT]->(:agent)
(:concept)-[:RELATED_TO {weight: float}]->(:concept)

Embedding Service

Responsibilities:

• Generate text embeddings for semantic search
• Manage vector storage and indexing
• Perform similarity searches
• Handle batch processing and caching

Key Classes:

class EmbeddingService:
    def __init__(self, config: EmbeddingConfig)
    def generate_embeddings(self, texts: List[str]) -> List[List[float]]
    def search_similar(self, query_embedding: List[float], top_k: int) -> SearchResults
    def batch_process_chunks(self, chunks: List[DocumentChunk]) -> ProcessingStats
    def update_index(self, new_embeddings: List[Tuple[str, List[float]]]) -> None

Embedding Pipeline:

1. Text preprocessing and chunking
2. Batch embedding generation
3. Vector normalization
4. Index updates with FAISS
5. Metadata storage in graph database

3. Data Access Layer

Database Abstraction

• Repository pattern for data access
• Connection pooling and retry logic
• Migration management
• Backup and recovery procedures

Caching Strategy

• Redis for session and API caching
• In-memory LRU cache for frequent queries
• CDN integration for static assets (future)

Implementation Phases

Phase 1A: Core Infrastructure (Week 1)

1. Set up FastAPI application structure
2. Implement basic configuration management
3. Create database connection layer
4. Set up logging and monitoring
5. Implement health check endpoints

Phase 1B: File System Integration (Week 2)

1. Implement file system watcher
2. Create file metadata extraction
3. Set up document processing pipeline
4. Implement secure file I/O operations
5. Add file change event handling

Phase 1C: Dana Runtime (Week 3)

1. Integrate Dana language runtime
2. Implement sandboxed execution
3. Create agent lifecycle management
4. Set up REPL functionality
5. Add error handling and debugging

Phase 1D: Knowledge Graph (Week 4)

1. Set up Neo4j connection and schema
2. Implement basic CRUD operations
3. Create graph query interface
4. Add centrality calculations
5. Implement graph visualization data endpoints

Performance Considerations

Scalability

• Horizontal scaling with load balancer
• Database read replicas for queries
• CDN for static content delivery
• Background job queues for heavy processing

Optimization Strategies

• Connection pooling for databases
• Embedding batching and caching
• Graph query optimization
• File system operation parallelization

Monitoring and Observability

• Structured logging with correlation IDs
• Performance metrics collection
• Health checks for all services
• Error tracking and alerting

Security Implementation

Data Protection

• Local data sovereignty enforcement
• File system permission validation
• Secure temporary file handling
• Input sanitization and validation

API Security

• Rate limiting implementation
• Request validation with Pydantic
• CORS configuration
• API key authentication

Runtime Security

• Dana code sandboxing
• Resource usage limits
• Execution timeouts
• Error isolation

Testing Strategy

Unit Testing

• Service layer testing with mocks
• Data access layer testing
• API endpoint testing
• Error condition testing

Integration Testing

• End-to-end API workflows
• Database integration tests
• File system operation tests
• Cross-service communication tests

Performance Testing

• Load testing for API endpoints
• Database query performance tests
• File processing throughput tests
• Memory usage profiling

Deployment Architecture

Development Environment

• Local Docker Compose setup
• Hot reload for development
• Debug logging enabled
• Local database instances

Production Environment

• Containerized deployment
• Orchestration with Kubernetes
• Production database configuration
• Monitoring and alerting setup

CI/CD Pipeline

• Automated testing on commits
• Docker image building
• Security scanning
• Deployment automation

Migration and Rollback

Data Migration

• Versioned database migrations
• Backward compatibility for APIs
• Data export/import capabilities
• Rollback procedures for deployments

Service Updates

• Blue-green deployment strategy
• Feature flags for gradual rollouts
• Monitoring during deployments
• Automated rollback triggers

This architecture provides a solid foundation for the Advanced Second Brain PKM System, ensuring scalability, maintainability, and security while supporting the complex requirements of multi-agent knowledge management. docs/plans/technical-implementation/backend-architecture.md