Crate search_engine_backend

Expand description

§Search Engine Backend

A Rust-based search engine backend that integrates with Azure Cognitive Search and CosmosDB to provide web crawling, indexing, and search capabilities.

§Features

Web Search: Full-text search across indexed web content
Domain Indexing: Crawl and index web domains with robots.txt compliance
Search Analytics: Track search queries, performance metrics, and usage patterns
Azure Integration: Uses Azure Cognitive Search and CosmosDB for scalable storage
REST API: Clean HTTP API for all operations

§Architecture

The application is structured around several key services:

SearchService: Handles search operations via Azure Cognitive Search
StorageService: Manages data persistence in Azure CosmosDB
IndexerService: Orchestrates web crawling and content indexing
Config: Application configuration management

§Example Usage

use search_engine_backend::{Config, AppState, create_router, StorageService, SearchService, IndexerService};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load configuration
    let config = Arc::new(Config::from_env()?);
     
    // Initialize services
    let storage_service = Arc::new(StorageService::new(config.clone()).await?);
    let search_service = Arc::new(SearchService::new(config.clone()).await?);
    let indexer_service = Arc::new(IndexerService::new(
        config.clone(),
        storage_service.clone(),
        search_service.clone()
    ).await?);
     
    // Create application state
    let app_state = AppState {
        config,
        search_service,
        storage_service,
        indexer_service,
    };
     
    // Create router and start server
    let app = create_router(app_state);
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
    axum::serve(listener, app).await?;
     
    Ok(())
}

Modules§

config 🔒: Configuration Management
indexer 🔒
search 🔒: Search Service
storage 🔒: Storage Service

Macros§

log_and_capture: Macro to log and capture messages for the dashboard

Structs§

AppState: Application state containing all service instances and configuration.
Args: Command-line arguments for the Search Engine Backend application.
Config: Main application configuration structure.
ForceIndexResponse: Response structure for force indexing operations.
ForceProcessQueueResponse: Response structure for force queue processing operations.
IndexRequest: Request structure for indexing operations.
IndexResponse: Response structure for indexing operations.
IndexerService
SearchQuery: Query parameters for search requests.
SearchResponse: Response structure for search operations.
SearchResult: Individual search result item.
SearchService
SearchStatistic: Represents search analytics and statistics data.
StatsResponse: Response structure for search statistics.
StorageService: Storage service for Azure CosmosDB operations.
TopQueriesResponse: Response structure for top search queries.
TopQuery: Individual top query item.

Functions§

admin_force_index_handler: HTTP handler for force indexing operations.
admin_force_process_queue_handler: HTTP handler for force queue processing operations.
admin_stats_handler: HTTP handler for retrieving search statistics.
admin_top_queries_handler: HTTP handler for retrieving top search queries.
create_router: Creates the main application router with all endpoints configured.
default_limit 🔒: Default limit for search results when not specified.
generate_system_activity_log 🔒: Generates a system activity log for the statistics dashboard.
health_handler: HTTP handler for health check operations.
index_handler: HTTP handler for domain indexing operations.
search_handler: HTTP handler for search operations.
stats_page_handler: HTTP handler for the statistics dashboard page.
validate_admin_auth 🔒: Validates admin authentication from request headers.

Crate search_engine_backendCopy item path