mirror of
https://github.com/bookorbit/bookorbit.git
synced 2026-11-03 00:11:37 -06:00
feat(entity-manager): pre-computed duplicate detection for large datasets
Replace the live self-join scan (O(N^2), times out at 100K+ authors) with a background-computed candidates table served instantly at read time. Key changes: - Add entity_duplicate_candidates and entity_duplicate_scan_status tables - Background job batches 1K authors at a time (~58s total for 200K authors) - Scan endpoint reads from the indexed table (<10ms regardless of scale) - Candidates are invalidated after merge/delete, not recomputed from scratch - Add GET /duplicates/status and POST /duplicates/refresh endpoints - Frontend updates clusters locally after merge/dismiss (no re-scan) - Status bar shows last computed time and recompute button with progress - Author browse in entity manager now shows all authors, not just those with books
This commit is contained in:
@@ -73,6 +73,17 @@ export interface DuplicateScanResponse {
|
||||
pageSize: number;
|
||||
}
|
||||
|
||||
export type DuplicateScanState = 'idle' | 'computing' | 'done' | 'error';
|
||||
|
||||
export interface DuplicateScanStatus {
|
||||
entityType: EntityType;
|
||||
state: DuplicateScanState;
|
||||
computedAt: string | null;
|
||||
totalPairs: number | null;
|
||||
threshold: number | null;
|
||||
progressPct: number | null;
|
||||
}
|
||||
|
||||
// Operation results
|
||||
export interface MergeResult {
|
||||
targetId: number | string;
|
||||
|
||||
Reference in New Issue
Block a user