claude/kochwas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(search): persistenter Thumbnail-Cache in SQLite, Default-TTL 30 Tage
All checks were successful
Build & Publish Docker Image / build-and-push (push) Successful in 54s
Details

Browse Source 
Vorher: In-Memory-Map, TTL 30 Minuten. Container-Neustart verwarf den
kompletten Cache, also musste nach jedem Deploy jede Suche wieder alle
Seiten laden.

Jetzt:
- Neue Tabelle thumbnail_cache (url PK, image, expires_at)
- Default-TTL 30 Tage, per Env KOCHWAS_THUMB_TTL_DAYS konfigurierbar
  (7, 365, was der User will — is alles ok laut Nutzer)
- Negative Cache: Seiten ohne Bild werden mit image=NULL gespeichert,
  damit wir nicht jede Suche die gleiche kaputte Seite wieder laden
- Lazy-Cleanup: pro searchWeb-Aufruf werden abgelaufene Zeilen via
  DELETE ... WHERE expires_at <= now() weggeräumt (Index-Scan, billig)

Migration 003_thumbnail_cache.sql: nicht-destruktiv, nur neue Tabelle.
Bestehende DB bekommt sie beim nächsten Start automatisch dazu.

Tests (99/99):
- Neuer Cache-Test: zweiter searchWeb für dieselbe URL macht keinen
  Page-Fetch mehr und liest die image-Spalte aus SQLite.
This commit is contained in:
hsiegeln
2026-04-17 18:34:29 +02:00
parent 1712263fd1
commit 4d90d51501
3 changed files with 81 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
tests/integration/searxng.test.ts
View File

@@ -184,6 +184,34 @@ describe('searchWeb', () => {
}
});
it('SQLite cache: second search does not re-fetch the page', async () => {
let pageHits = 0;
const pageServer = createServer((_req, res) => {
pageHits += 1;
res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
res.end('<html><head><meta property="og:image" content="https://cdn.example/c.jpg"></head></html>');
});
await new Promise<void>((r) => pageServer.listen(0, '127.0.0.1', r));
const addr = pageServer.address() as AddressInfo;
const pageUrl = `http://127.0.0.1:${addr.port}/cached`;
try {
const db = openInMemoryForTest();
addDomain(db, '127.0.0.1');
respondWith([{ url: pageUrl, title: 'C', content: '' }]);
const first = await searchWeb(db, 'c', { searxngUrl: baseUrl });
const second = await searchWeb(db, 'c', { searxngUrl: baseUrl });
expect(first[0].thumbnail).toBe('https://cdn.example/c.jpg');
expect(second[0].thumbnail).toBe('https://cdn.example/c.jpg');
expect(pageHits).toBe(1); // second call read from SQLite cache
const row = db
.prepare('SELECT image FROM thumbnail_cache WHERE url = ?')
.get(pageUrl) as { image: string };
expect(row.image).toBe('https://cdn.example/c.jpg');
} finally {
await new Promise<void>((r) => pageServer.close(() => r()));
}
});
it('filters out forum/magazine/listing URLs', async () => {
const db = openInMemoryForTest();
addDomain(db, 'chefkoch.de');