mirror of
https://github.com/bookorbit/bookorbit.git
synced 2026-11-03 00:11:37 -06:00
fix(kobo): forward x-kobo-userkey and decode the composite sync token in the catch-all proxy (#1138)
* fix(kobo): forward x-kobo-userkey and decode the composite sync token in the catch-all proxy The catch-all proxy in kobo-device.controller.ts forwards device requests to storeapi.kobo.com through KoboProxyService, which built its outgoing headers from a static allow-list (FORWARD_HEADERS). The list covered accept, accept-language, authorization, content-type, user-agent, and a curated set of x-kobo-* headers, but two of the headers that matter for store-API authentication and cursor tracking were missing: x-kobo-userkey, the device's per-account secret that authorizes most Kobo store endpoints, and x-kobo-synctoken, the upstream cursor the device carries on every store call. Without x-kobo-userkey, every store interaction proxied through the catch-all shipped without credentials, so Kobo either rejected the call or treated the device as unauthenticated. The "Connecting to OverDrive" dialog that issue #738 describes is the borrow endpoint, /v1/library/borrow, hanging because Kobo cannot reconcile a borrow into a user's library without the user key. The same gap also left library books a user borrowed on libbyapp.com (which is just OverDrive pushed to kobo.com) unreachable from the device, because every proxy-bound sync the device makes lands at Kobo without the credentials it needs to enumerate the user's entitlements. The Kobo Plus / Kobo Store entitlement fix in 2.6.0 (closes #789) only patched the one explicit library/sync call. That call already decodes the BookOrbit composite sync token via kobo-sync-token.ts and overrides the outgoing header with the Kobo cursor Kobo understands, so the store page works. Every other store call still went through the catch-all proxy with the device's PX.<base64> composite on it, which Kobo treats as a fresh cursor and answers for. The borrow endpoint in particular was the most visible casualty, but anything else proxied through @All('*') was affected the same way. Switch the proxy to a deny-list (DROP_HEADERS) for the headers RFC 7230 names hop-by-hop or otherwise scoped to a single connection (host, connection, content-length, content-encoding, transfer-encoding, keep-alive, proxy-authenticate, proxy-authorization, te, trailers, upgrade), and forward everything else the device sends. x-kobo-userkey now reaches Kobo on every store call, so the borrow endpoint can authorize the user's library and the libby-arrived entitlements surface on the next /v1/library/sync. Add a forward() path that decodes the device's composite sync token before relaying, and forwards the Kobo cursor Kobo can use instead. A raw Kobo token (no PX. prefix) is forwarded verbatim, the way devices that synced with Kobo directly before BookOrbit was put in the path already do. A composite token whose cursor is undefined has no Kobo-side value, so omit the header entirely on the way out rather than leak our composite to Kobo. The extraHeaders and omitHeaders escape hatches stay in place, so the explicit library/sync call can still pin the upstream cursor (and the no-cursor path still drops the header) without going through the new decode-on-forward path. The librarySync controller in kobo-sync.controller.ts already uses those exacts knobs for the store page, so this change does not touch it; this commit is the catch-all side of the same fix. Closes #738 * fix(kobo): honor RFC 7230 Trailer and Connection fields, and emit sanitized proxy logs Three small corrections surfaced by code review on the catch-all proxy that delivers Kobo store entitlements. The deny-list shipped with the wrong name for one of the RFC 7230 hop-by-hop fields. `trailers` (plural) does not match the standard `Trailer` header that arrives in mixed case, so a Trailer field set by an upstream could leak through on the way to the device. Replace it with `trailer`. RFC 7230 says the fields listed in `Connection` are also hop-specific, and a proxy must strip them before forwarding. The two header-copy paths only checked DROP_HEADERS, so a `Connection: x-kobo-tracking` header from the device or from upstream could let a nominated field slip through. Parse the Connection header in both directions, normalize case-insensitively, and add its fields to the ignore set alongside DROP_HEADERS. The new readConnectionFieldNames helper does the splitting once and is reused on both sides of the relay. Add a structured start/end/fail log around the relay, replacing the previous single-line warn that interpolated a request-derived targetUrl and a raw err.message. The repo's logging conventions in AGENTS.md name an external HTTP call as a non-trivial operation that warrants start/end/fail logs, and require sanitizeLogValue() for every dynamic value that lands inside a quoted log field. The fail line now carries durationMs, errorClass, sanitized error, and sanitized targetUrl. The 502 response with the "Upstream Kobo API unavailable" body is preserved. Tests cover the new paths: Trailer is stripped, Connection-nominated fields are stripped on the request side, and Connection-nominated fields are stripped on the response side. All 10,486 server tests pass; lint and typecheck clean. CodeRabbit also asked to instantiate KoboProxyService through Test.createTestingModule. AGENTS.md says the same thing, but every sibling test in the kobo module uses direct construction with no testing module, so changing only this file would invert the local pattern. The convention is already inconsistent across the module and is not enforced by any lint rule; that is a separate cleanup better suited to its own PR. * test(kobo): drop unused makeHeaders helper in proxy service tests makeHeaders was a placeholder identity function that returned its argument unchanged. Inline the headers object literals at the 9 call sites. * fix(kobo): scrub BookOrbit credentials and the composite token in every relay Move the sync-token substitution into request() so forwardTagDelete, which relays with no options, stops handing Kobo a PX. token it cannot read. Apply it before omitHeaders and extraHeaders so store sync keeps both escape hatches. Strip BookOrbit's session cookies and the reverse proxy forwarding headers: these routes sit on the BookOrbit origin, so a browser carrying a session reached them with our auth cookies attached and relayed them to Kobo. Restore the SSRF regression case that covers the allowed hostname in userinfo. * test(maintenance): wait for the cover sweep instead of a fixed turn budget The detached sweep does real filesystem I/O, so 50 setImmediate turns settled it locally but not on a loaded runner. The drain loop then gave up silently and the next call failed as "The cover sweep is not complete", pointing at the wrong place. Poll the store to a deadline and fail loudly when it truly hangs. --------- Co-authored-by: Daniel Chong <daniel@chong.nz> Co-authored-by: neon <18559658+neonsolstice@users.noreply.github.com>
This commit is contained in:
Daniel Chong
GitHub
@@ -1,5 +1,6 @@
|
||||
import { BadRequestException } from '@nestjs/common';
|
||||
|
||||
import type { KoboProxyRequestOptions } from './kobo-proxy.service';
|
||||
import { KoboProxyService } from './kobo-proxy.service';
|
||||
|
||||
function makeReply() {
|
||||
@@ -43,6 +44,9 @@ describe('KoboProxyService', () => {
|
||||
accept: 'application/json',
|
||||
host: 'localhost:3000',
|
||||
'x-kobo-deviceid': 'dev123',
|
||||
'x-kobo-userkey': 'user-secret',
|
||||
'x-kobo-synctoken': 'raw-kobo-token',
|
||||
'content-length': '17',
|
||||
},
|
||||
body: { hello: 'world' },
|
||||
};
|
||||
@@ -50,14 +54,24 @@ describe('KoboProxyService', () => {
|
||||
|
||||
await service.forward(req as never, reply as never, 'device-1');
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith('https://storeapi.kobo.com/v1/library/sync?since=1', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'x-kobo-deviceid': 'dev123',
|
||||
},
|
||||
body: '{"hello":"world"}',
|
||||
});
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
'https://storeapi.kobo.com/v1/library/sync?since=1',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
headers: expect.objectContaining({
|
||||
accept: 'application/json',
|
||||
'x-kobo-deviceid': 'dev123',
|
||||
'x-kobo-userkey': 'user-secret',
|
||||
'x-kobo-synctoken': 'raw-kobo-token',
|
||||
}),
|
||||
body: '{"hello":"world"}',
|
||||
}),
|
||||
);
|
||||
const sentHeaders = fetchMock.mock.calls[0][1].headers as Record<string, string>;
|
||||
expect(sentHeaders).not.toHaveProperty('host');
|
||||
expect(sentHeaders).not.toHaveProperty('content-length');
|
||||
// Raw Kobo tokens (no PX. prefix) reach Kobo verbatim, so the device's own cursor is forwarded.
|
||||
expect(sentHeaders['x-kobo-synctoken']).toBe('raw-kobo-token');
|
||||
expect(reply.status).toHaveBeenCalledWith(200);
|
||||
expect(reply.header).toHaveBeenCalledWith('content-type', 'application/json');
|
||||
expect(reply.header).toHaveBeenCalledWith('x-custom', 'ok');
|
||||
@@ -119,18 +133,21 @@ describe('KoboProxyService', () => {
|
||||
|
||||
await service.forward(req as never, reply as never, 'device-1');
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith(`https://storeapi.kobo.com/v1/library/${entitlementId}/state`, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
authorization: 'Bearer kobo-oauth-token',
|
||||
'content-type': 'application/json',
|
||||
'user-agent': 'Kobo Touch',
|
||||
'x-kobo-appversion': '4.45.23697',
|
||||
'x-kobo-deviceid': 'device-id',
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
`https://storeapi.kobo.com/v1/library/${entitlementId}/state`,
|
||||
expect.objectContaining({
|
||||
method: 'PUT',
|
||||
headers: expect.objectContaining({
|
||||
accept: 'application/json',
|
||||
authorization: 'Bearer kobo-oauth-token',
|
||||
'content-type': 'application/json',
|
||||
'user-agent': 'Kobo Touch',
|
||||
'x-kobo-appversion': '4.45.23697',
|
||||
'x-kobo-deviceid': 'device-id',
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
}),
|
||||
);
|
||||
expect(reply.status).toHaveBeenCalledWith(200);
|
||||
expect(reply.send).toHaveBeenCalledWith(Buffer.from('{"RequestResult":"Success"}'));
|
||||
});
|
||||
@@ -154,6 +171,236 @@ describe('KoboProxyService', () => {
|
||||
expect(reply.send).toHaveBeenCalledWith({ message: 'Upstream Kobo API unavailable' });
|
||||
});
|
||||
|
||||
describe('over-the-device-borrow flow', () => {
|
||||
it('forwards the Kobo userkey and decodes the BookOrbit composite sync token before proxying a borrow', async () => {
|
||||
const service = new KoboProxyService();
|
||||
const upstream = {
|
||||
status: 200,
|
||||
headers: new Headers({ 'content-type': 'application/json' }),
|
||||
arrayBuffer: vi.fn().mockResolvedValue(new TextEncoder().encode('{"success":true}').buffer),
|
||||
};
|
||||
const fetchMock = vi.fn().mockResolvedValue(upstream);
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
const composite = `PX.${Buffer.from(JSON.stringify({ snapshotId: 4, koboSyncToken: 'kobo-cursor-9' })).toString('base64')}`;
|
||||
const req = {
|
||||
method: 'POST',
|
||||
url: '/api/v1/kobo/dev/v1/library/borrow',
|
||||
headers: {
|
||||
accept: 'application/json',
|
||||
'content-type': 'application/json',
|
||||
'x-kobo-userkey': 'device-user-key',
|
||||
'x-kobo-deviceid': 'device-id',
|
||||
'x-kobo-synctoken': composite,
|
||||
},
|
||||
body: { BookId: '9780123456789' },
|
||||
};
|
||||
const reply = makeReply();
|
||||
|
||||
await service.forward(req as never, reply as never, 'dev');
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
'https://storeapi.kobo.com/v1/library/borrow',
|
||||
expect.objectContaining({
|
||||
headers: expect.objectContaining({
|
||||
'x-kobo-userkey': 'device-user-key',
|
||||
'x-kobo-synctoken': 'kobo-cursor-9',
|
||||
}),
|
||||
body: JSON.stringify({ BookId: '9780123456789' }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('omits the sync token entirely when the device sent no Kobo cursor', async () => {
|
||||
const service = new KoboProxyService();
|
||||
const upstream = {
|
||||
status: 200,
|
||||
headers: new Headers(),
|
||||
arrayBuffer: vi.fn().mockResolvedValue(new ArrayBuffer(0)),
|
||||
};
|
||||
const fetchMock = vi.fn().mockResolvedValue(upstream);
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
const composite = `PX.${Buffer.from(JSON.stringify({ snapshotId: 4 })).toString('base64')}`;
|
||||
const req = {
|
||||
method: 'POST',
|
||||
url: '/api/v1/kobo/dev/v1/library/borrow',
|
||||
headers: {
|
||||
'x-kobo-userkey': 'device-user-key',
|
||||
'x-kobo-synctoken': composite,
|
||||
},
|
||||
body: {},
|
||||
};
|
||||
|
||||
await service.forward(req as never, makeReply() as never, 'dev');
|
||||
|
||||
const sentHeaders = fetchMock.mock.calls[0][1].headers as Record<string, string>;
|
||||
expect(sentHeaders).not.toHaveProperty('x-kobo-synctoken');
|
||||
expect(sentHeaders['x-kobo-userkey']).toBe('device-user-key');
|
||||
});
|
||||
});
|
||||
|
||||
describe('credentials that must not reach Kobo', () => {
|
||||
function stubFetch() {
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
status: 200,
|
||||
headers: new Headers(),
|
||||
arrayBuffer: vi.fn().mockResolvedValue(new ArrayBuffer(0)),
|
||||
});
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
return fetchMock;
|
||||
}
|
||||
|
||||
async function sentHeaders(headers: Record<string, string>, options?: KoboProxyRequestOptions): Promise<Record<string, string>> {
|
||||
const fetchMock = stubFetch();
|
||||
await new KoboProxyService().request({ method: 'GET', url: '/api/v1/kobo/dev/v1/library/sync', headers } as never, 'dev', options);
|
||||
return fetchMock.mock.calls[0][1].headers as Record<string, string>;
|
||||
}
|
||||
|
||||
const composite = `PX.${Buffer.from(JSON.stringify({ snapshotId: 4, koboSyncToken: 'kobo-cursor-9' })).toString('base64')}`;
|
||||
|
||||
// Every caller gets the substitution, not just the ones that remember to ask: forwardTagDelete
|
||||
// relays through request() with no options and must not hand Kobo a token it cannot read.
|
||||
it('substitutes the Kobo cursor for the composite token even with no options', async () => {
|
||||
expect((await sentHeaders({ 'x-kobo-synctoken': composite }))['x-kobo-synctoken']).toBe('kobo-cursor-9');
|
||||
});
|
||||
|
||||
it('drops a composite token it cannot decode rather than forwarding it', async () => {
|
||||
expect(await sentHeaders({ 'x-kobo-synctoken': 'PX.not-base64-json' })).not.toHaveProperty('x-kobo-synctoken');
|
||||
});
|
||||
|
||||
it('still lets omitHeaders drop the token the substitution would have written', async () => {
|
||||
const headers = await sentHeaders({ 'x-kobo-synctoken': composite }, { omitHeaders: ['x-kobo-synctoken'] });
|
||||
expect(headers).not.toHaveProperty('x-kobo-synctoken');
|
||||
});
|
||||
|
||||
it('still lets extraHeaders override the token the substitution would have written', async () => {
|
||||
const headers = await sentHeaders({ 'x-kobo-synctoken': composite }, { extraHeaders: { 'x-kobo-synctoken': 'server-cursor' } });
|
||||
expect(headers['x-kobo-synctoken']).toBe('server-cursor');
|
||||
});
|
||||
|
||||
// These routes sit on the BookOrbit origin, so a browser carrying a session reaches them with
|
||||
// our auth cookies attached. Kobo must never receive those; cookies it set itself still ride.
|
||||
it('strips the BookOrbit session cookies while keeping the rest of the jar', async () => {
|
||||
const headers = await sentHeaders({ cookie: 'access_token=secret; kobo_session=keep-me; refresh_token=secret; wsid=abc' });
|
||||
expect(headers.cookie).toBe('kobo_session=keep-me; wsid=abc');
|
||||
});
|
||||
|
||||
it('sends no cookie header at all when only BookOrbit cookies were present', async () => {
|
||||
expect(await sentHeaders({ cookie: 'access_token=secret; refresh_token=secret' })).not.toHaveProperty('cookie');
|
||||
});
|
||||
|
||||
it('drops the reverse proxy forwarding headers that describe our hop', async () => {
|
||||
const headers = await sentHeaders({
|
||||
forwarded: 'for=10.0.0.5',
|
||||
'x-forwarded-for': '10.0.0.5',
|
||||
'x-forwarded-host': 'books.example',
|
||||
'x-forwarded-proto': 'https',
|
||||
'x-real-ip': '10.0.0.5',
|
||||
'x-kobo-userkey': 'device-user-key',
|
||||
});
|
||||
|
||||
expect(headers).not.toHaveProperty('forwarded');
|
||||
expect(headers).not.toHaveProperty('x-forwarded-for');
|
||||
expect(headers).not.toHaveProperty('x-forwarded-host');
|
||||
expect(headers).not.toHaveProperty('x-forwarded-proto');
|
||||
expect(headers).not.toHaveProperty('x-real-ip');
|
||||
expect(headers['x-kobo-userkey']).toBe('device-user-key');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hop-by-hop header handling', () => {
|
||||
it('strips the singular Trailer header field on the way out', async () => {
|
||||
const service = new KoboProxyService();
|
||||
const upstream = {
|
||||
status: 200,
|
||||
headers: new Headers(),
|
||||
arrayBuffer: vi.fn().mockResolvedValue(new ArrayBuffer(0)),
|
||||
};
|
||||
const fetchMock = vi.fn().mockResolvedValue(upstream);
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
await service.forward(
|
||||
{
|
||||
method: 'GET',
|
||||
url: '/api/v1/kobo/dev/v1/library/sync',
|
||||
headers: {
|
||||
trailer: 'x-internal-state',
|
||||
'x-kobo-deviceid': 'device-id',
|
||||
},
|
||||
} as never,
|
||||
makeReply() as never,
|
||||
'dev',
|
||||
);
|
||||
|
||||
const sentHeaders = fetchMock.mock.calls[0][1].headers as Record<string, string>;
|
||||
expect(sentHeaders).not.toHaveProperty('trailer');
|
||||
expect(sentHeaders['x-kobo-deviceid']).toBe('device-id');
|
||||
});
|
||||
|
||||
it('drops header fields nominated by the device Connection header before forwarding', async () => {
|
||||
const service = new KoboProxyService();
|
||||
const upstream = {
|
||||
status: 200,
|
||||
headers: new Headers(),
|
||||
arrayBuffer: vi.fn().mockResolvedValue(new ArrayBuffer(0)),
|
||||
};
|
||||
const fetchMock = vi.fn().mockResolvedValue(upstream);
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
await service.forward(
|
||||
{
|
||||
method: 'GET',
|
||||
url: '/api/v1/kobo/dev/v1/library/sync',
|
||||
headers: {
|
||||
connection: 'x-kobo-tracking, x-kobo-internal',
|
||||
'x-kobo-tracking': 'should-be-stripped',
|
||||
'x-kobo-internal': 'also-stripped',
|
||||
'x-kobo-deviceid': 'device-id',
|
||||
},
|
||||
} as never,
|
||||
makeReply() as never,
|
||||
'dev',
|
||||
);
|
||||
|
||||
const sentHeaders = fetchMock.mock.calls[0][1].headers as Record<string, string>;
|
||||
expect(sentHeaders).not.toHaveProperty('x-kobo-tracking');
|
||||
expect(sentHeaders).not.toHaveProperty('x-kobo-internal');
|
||||
expect(sentHeaders['x-kobo-deviceid']).toBe('device-id');
|
||||
});
|
||||
|
||||
it('drops header fields nominated by the upstream Connection header before relaying the response', async () => {
|
||||
const service = new KoboProxyService();
|
||||
const upstream = {
|
||||
status: 200,
|
||||
headers: new Headers({
|
||||
connection: 'x-kobo-tracking, x-kobo-internal',
|
||||
'x-kobo-tracking': 'should-be-stripped',
|
||||
'x-kobo-internal': 'also-stripped',
|
||||
'x-kobo-deviceid': 'device-id',
|
||||
}),
|
||||
arrayBuffer: vi.fn().mockResolvedValue(new TextEncoder().encode('{}').buffer),
|
||||
};
|
||||
const fetchMock = vi.fn().mockResolvedValue(upstream);
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
const reply = makeReply();
|
||||
await service.forward(
|
||||
{
|
||||
method: 'GET',
|
||||
url: '/api/v1/kobo/dev/v1/library/sync',
|
||||
headers: {},
|
||||
} as never,
|
||||
reply as never,
|
||||
'dev',
|
||||
);
|
||||
|
||||
expect(reply.header).not.toHaveBeenCalledWith('x-kobo-tracking', expect.anything());
|
||||
expect(reply.header).not.toHaveBeenCalledWith('x-kobo-internal', expect.anything());
|
||||
expect(reply.header).toHaveBeenCalledWith('x-kobo-deviceid', 'device-id');
|
||||
});
|
||||
});
|
||||
|
||||
describe('request', () => {
|
||||
function stubUpstream(headers: Record<string, string>, body = '[]') {
|
||||
const fetchMock = vi.fn().mockResolvedValue({
|
||||
@@ -168,7 +415,7 @@ describe('KoboProxyService', () => {
|
||||
const syncRequest = {
|
||||
method: 'GET',
|
||||
url: '/api/v1/kobo/dev/v1/library/sync?Filter=ALL',
|
||||
headers: { authorization: 'Bearer kobo-jwt', 'x-kobo-synctoken': 'PX.composite' },
|
||||
headers: { authorization: 'Bearer kobo-jwt', 'x-kobo-synctoken': 'PX.composite', 'x-kobo-userkey': 'user-secret' },
|
||||
};
|
||||
|
||||
it('returns the upstream response with lowercased headers instead of piping it', async () => {
|
||||
|
||||
@@ -1,26 +1,56 @@
|
||||
import { BadRequestException, Injectable, Logger } from '@nestjs/common';
|
||||
import type { FastifyReply, FastifyRequest } from 'fastify';
|
||||
|
||||
import { sanitizeLogValue } from '../../../common/utils/log-sanitize.utils';
|
||||
import { decodeSyncToken } from './kobo-sync-token';
|
||||
|
||||
const KOBO_API_BASE = 'https://storeapi.kobo.com';
|
||||
const KOBO_API_HOSTNAME = 'storeapi.kobo.com';
|
||||
|
||||
const FORWARD_HEADERS = [
|
||||
'accept',
|
||||
'accept-language',
|
||||
'authorization',
|
||||
'content-type',
|
||||
'user-agent',
|
||||
'x-kobo-affiliatename',
|
||||
'x-kobo-appversion',
|
||||
'x-kobo-deviceid',
|
||||
'x-kobo-devicemodel',
|
||||
'x-kobo-deviceos',
|
||||
'x-kobo-deviceosversion',
|
||||
'x-kobo-platformid',
|
||||
'x-kobo-synctokenversion',
|
||||
];
|
||||
// Headers that must never be forwarded. Connection-specific headers are defined by RFC 7230 as
|
||||
// scoped to a single hop and reset by every proxy, and `host` would otherwise point Kobo at the
|
||||
// BookOrbit host that the device just spoke to. Everything else the device sends (including
|
||||
// `x-kobo-userkey`, which authorizes most Kobo API calls, and `x-kobo-synctoken`, which carries
|
||||
// the upstream cursor) is forwarded, with two escape hatches callers can use:
|
||||
// * `extraHeaders` overrides a forwarded value (the sync-token-aware entry points use this to
|
||||
// hand Kobo its own cursor instead of the BookOrbit composite).
|
||||
// * `omitHeaders` strips a forwarded header (used to drop our composite when we have no Kobo
|
||||
// cursor to substitute).
|
||||
const DROP_HEADERS = new Set([
|
||||
'host',
|
||||
'connection',
|
||||
'keep-alive',
|
||||
'proxy-authenticate',
|
||||
'proxy-authorization',
|
||||
'te',
|
||||
'trailer',
|
||||
'transfer-encoding',
|
||||
'upgrade',
|
||||
'content-length',
|
||||
'content-encoding',
|
||||
// Added by the operator's reverse proxy; describes our hop, not the device's, and tells Kobo
|
||||
// about the deployment's internal addressing.
|
||||
'forwarded',
|
||||
'x-forwarded-for',
|
||||
'x-forwarded-host',
|
||||
'x-forwarded-proto',
|
||||
'x-real-ip',
|
||||
]);
|
||||
|
||||
const HOP_BY_HOP_HEADERS = ['transfer-encoding', 'connection', 'content-encoding', 'content-length'];
|
||||
const SYNC_TOKEN_HEADER = 'x-kobo-synctoken';
|
||||
|
||||
// BookOrbit's own session cookies. The device token in the path is the credential for these
|
||||
// routes, so a browser that carries a BookOrbit session to one must not hand it to Kobo.
|
||||
const BOOKORBIT_COOKIE_NAMES = new Set(['access_token', 'refresh_token']);
|
||||
|
||||
/** Drops BookOrbit's session cookies from a Cookie header, keeping any the device set itself. */
|
||||
function stripBookOrbitCookies(cookieHeader: string): string {
|
||||
return cookieHeader
|
||||
.split(';')
|
||||
.filter((pair) => !BOOKORBIT_COOKIE_NAMES.has(pair.split('=')[0].trim()))
|
||||
.join(';')
|
||||
.trim();
|
||||
}
|
||||
|
||||
export type KoboProxyResponse = {
|
||||
status: number;
|
||||
@@ -30,11 +60,26 @@ export type KoboProxyResponse = {
|
||||
|
||||
export type KoboProxyRequestOptions = {
|
||||
extraHeaders?: Record<string, string>;
|
||||
/** Drops headers FORWARD_HEADERS would otherwise copy from the device request. */
|
||||
/** Drops headers the device would otherwise copy. */
|
||||
omitHeaders?: string[];
|
||||
timeoutMs?: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Reads the `Connection` header from a header bag, splits it on commas, lowercases each field
|
||||
* name, and returns the result. RFC 7230 §6.1 says every field listed in `Connection` is
|
||||
* hop-specific: a proxy must remove those fields before forwarding the message.
|
||||
*/
|
||||
function readConnectionFieldNames(headers: Record<string, string | string[] | undefined>): string[] {
|
||||
const raw = headers['connection'];
|
||||
if (raw === undefined) return [];
|
||||
const values = Array.isArray(raw) ? raw : [raw];
|
||||
return values
|
||||
.flatMap((v) => v.split(','))
|
||||
.map((name) => name.trim().toLowerCase())
|
||||
.filter((name) => name.length > 0);
|
||||
}
|
||||
|
||||
@Injectable()
|
||||
export class KoboProxyService {
|
||||
private readonly logger = new Logger(KoboProxyService.name);
|
||||
@@ -43,11 +88,34 @@ export class KoboProxyService {
|
||||
async request(req: FastifyRequest, deviceToken: string, options: KoboProxyRequestOptions = {}): Promise<KoboProxyResponse> {
|
||||
const targetUrl = this.resolveTargetUrl(req, deviceToken);
|
||||
|
||||
// Fastify lowercases header keys, but defensive: lowercase the lookup keys too.
|
||||
const sourceHeaders = req.headers as Record<string, string | string[] | undefined>;
|
||||
const connectionFields = new Set(readConnectionFieldNames(sourceHeaders));
|
||||
const headers: Record<string, string> = {};
|
||||
for (const key of FORWARD_HEADERS) {
|
||||
const val = req.headers[key];
|
||||
if (val) headers[key] = Array.isArray(val) ? val[0] : val;
|
||||
for (const [key, value] of Object.entries(sourceHeaders)) {
|
||||
if (value === undefined) continue;
|
||||
const lowerKey = key.toLowerCase();
|
||||
if (DROP_HEADERS.has(lowerKey)) continue;
|
||||
if (connectionFields.has(lowerKey)) continue;
|
||||
const raw = Array.isArray(value) ? (value[0] ?? '') : value;
|
||||
|
||||
// Kobo cannot read the BookOrbit composite token, so it gets its own cursor or nothing.
|
||||
// Applied before omitHeaders and extraHeaders so callers can still override either way.
|
||||
if (lowerKey === SYNC_TOKEN_HEADER) {
|
||||
const koboCursor = decodeSyncToken(raw).koboSyncToken;
|
||||
if (koboCursor !== undefined) headers[lowerKey] = koboCursor;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (lowerKey === 'cookie') {
|
||||
const kept = stripBookOrbitCookies(raw);
|
||||
if (kept.length > 0) headers[lowerKey] = kept;
|
||||
continue;
|
||||
}
|
||||
|
||||
headers[lowerKey] = raw;
|
||||
}
|
||||
|
||||
for (const key of options.omitHeaders ?? []) {
|
||||
delete headers[key.toLowerCase()];
|
||||
}
|
||||
@@ -76,28 +144,49 @@ export class KoboProxyService {
|
||||
return { status: upstream.status, headers: responseHeaders, body: Buffer.from(await upstream.arrayBuffer()) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Relays a request to Kobo, dropping the connection-specific headers and replacing the device's
|
||||
* BookOrbit composite sync token with the Kobo cursor it carries. Kobo's API does not understand
|
||||
* `PX.<base64>` and treats it as a fresh cursor, so a borrow call that uses it lands Kobo in a
|
||||
* state where the device has an entitlement BookOrbit never sees on this iteration.
|
||||
*/
|
||||
async forward(req: FastifyRequest, reply: FastifyReply, deviceToken: string): Promise<void> {
|
||||
const startedAt = Date.now();
|
||||
const targetUrl = this.resolveTargetUrl(req, deviceToken);
|
||||
const deviceId = (req as unknown as { koboDevice?: { deviceId: number } }).koboDevice?.deviceId;
|
||||
|
||||
this.logger.debug(`[kobo.proxy] [start] deviceId=${deviceId ?? 'unknown'} - upstream Kobo relay started`);
|
||||
|
||||
try {
|
||||
const response = await this.request(req, deviceToken);
|
||||
this.sendUpstream(reply, response);
|
||||
this.logger.debug(
|
||||
`[kobo.proxy] [end] deviceId=${deviceId ?? 'unknown'} durationMs=${Date.now() - startedAt} upstreamStatus=${response.status} - upstream Kobo relay completed`,
|
||||
);
|
||||
} catch (err: unknown) {
|
||||
const errorClass = err instanceof Error ? err.name : 'UnknownError';
|
||||
const errorMessage = sanitizeLogValue(err instanceof Error ? err.message : 'unknown error');
|
||||
const sanitizedUrl = sanitizeLogValue(targetUrl);
|
||||
this.logger.warn(
|
||||
`[kobo.proxy] [fail] deviceId=${deviceId ?? 'unknown'} durationMs=${Date.now() - startedAt} errorClass=${errorClass} error="${errorMessage}" targetUrl="${sanitizedUrl}" - upstream Kobo relay failed`,
|
||||
);
|
||||
reply.status(502).send({ message: 'Upstream Kobo API unavailable' });
|
||||
}
|
||||
}
|
||||
|
||||
/** Relays an upstream response to the device, dropping the headers that must not be forwarded. */
|
||||
sendUpstream(reply: FastifyReply, response: KoboProxyResponse): void {
|
||||
reply.status(response.status);
|
||||
const connectionFields = new Set(readConnectionFieldNames(response.headers));
|
||||
for (const [key, value] of Object.entries(response.headers)) {
|
||||
if (!HOP_BY_HOP_HEADERS.includes(key)) {
|
||||
reply.header(key, value);
|
||||
}
|
||||
const lowerKey = key.toLowerCase();
|
||||
if (DROP_HEADERS.has(lowerKey)) continue;
|
||||
if (connectionFields.has(lowerKey)) continue;
|
||||
reply.header(lowerKey, value);
|
||||
}
|
||||
reply.send(response.body);
|
||||
}
|
||||
|
||||
async forward(req: FastifyRequest, reply: FastifyReply, deviceToken: string) {
|
||||
const targetUrl = this.resolveTargetUrl(req, deviceToken);
|
||||
|
||||
try {
|
||||
this.sendUpstream(reply, await this.request(req, deviceToken));
|
||||
} catch (err) {
|
||||
this.logger.warn(`Proxy failed for ${targetUrl}: ${(err as Error).message}`);
|
||||
reply.status(502).send({ message: 'Upstream Kobo API unavailable' });
|
||||
}
|
||||
}
|
||||
|
||||
private resolveTargetUrl(req: FastifyRequest, deviceToken: string): string {
|
||||
const rawUrl = req.url;
|
||||
const prefix = `/api/v1/kobo/${deviceToken}`;
|
||||
|
||||
@@ -9,6 +9,8 @@ import { MissingResourcesService } from './missing-resources.service';
|
||||
|
||||
const user = { id: 7, isSuperuser: true } as never;
|
||||
|
||||
const SWEEP_SETTLE_TIMEOUT_MS = 10_000;
|
||||
|
||||
describe('MissingResourcesService', () => {
|
||||
let appDataPath: string;
|
||||
let coversRoot: string;
|
||||
@@ -54,9 +56,13 @@ describe('MissingResourcesService', () => {
|
||||
|
||||
async function runSweep(service: MissingResourcesService): Promise<void> {
|
||||
await service.startSweep(user);
|
||||
// The sweep runs detached; drain the microtask/IO queue until it settles.
|
||||
for (let attempt = 0; attempt < 50 && store.isRunning(user.id); attempt += 1) {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
// The sweep runs detached over real filesystem I/O, so wait on the store rather than draining a
|
||||
// fixed number of turns: a loaded runner outlasts any turn budget, and giving up silently left
|
||||
// the sweep running into the next call, which failed as an unrelated "sweep is not complete".
|
||||
const deadline = Date.now() + SWEEP_SETTLE_TIMEOUT_MS;
|
||||
while (store.isRunning(user.id)) {
|
||||
if (Date.now() >= deadline) throw new Error(`cover sweep did not settle within ${SWEEP_SETTLE_TIMEOUT_MS}ms`);
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user