feat(entity-manager): pre-computed duplicate detection for large datasets

Replace the live self-join scan (O(N^2), times out at 100K+ authors) with a
background-computed candidates table served instantly at read time.

Key changes:
- Add entity_duplicate_candidates and entity_duplicate_scan_status tables
- Background job batches 1K authors at a time (~58s total for 200K authors)
- Scan endpoint reads from the indexed table (<10ms regardless of scale)
- Candidates are invalidated after merge/delete, not recomputed from scratch
- Add GET /duplicates/status and POST /duplicates/refresh endpoints
- Frontend updates clusters locally after merge/dismiss (no re-scan)
- Status bar shows last computed time and recompute button with progress
- Author browse in entity manager now shows all authors, not just those with books
This commit is contained in:
neon
2026-05-10 07:32:32 +00:00
parent e59608ceb3
commit ce72cfd988
19 changed files with 1218 additions and 141 deletions
+11
View File
@@ -73,6 +73,17 @@ export interface DuplicateScanResponse {
pageSize: number;
}
export type DuplicateScanState = 'idle' | 'computing' | 'done' | 'error';
export interface DuplicateScanStatus {
entityType: EntityType;
state: DuplicateScanState;
computedAt: string | null;
totalPairs: number | null;
threshold: number | null;
progressPct: number | null;
}
// Operation results
export interface MergeResult {
targetId: number | string;