A Kobo dogear was stored as an empty yellow highlight. Land it as a
BookOrbit bookmark linked to the device's id, serve it back as a dogear,
sync deletions both ways, and move dogears already stored as highlights.
Report only books on that Kobo from checkforchanges: a re-registered
device was told about books it never had, so real changes were fetched
outside its sync and never applied. Serve a known annotation at its last
location instead of failing the whole pull when this copy cannot place it.
Fixes#1374
A book's first announcement went to the Kobo before it was downloaded,
when no Kobo location could be computed, so it carried a bare percentage
and the book opened at its start. Compute the location from the copy
being offered for bookmarks sent to the device; positions it reports
still wait for the transfer.
When a book's primary file changed, the Kobo kept the copy built from the
old file but its positions no longer converted, so it was sent a bare
percentage, reset to the start, and that reset was saved over real
progress. Sync each device against its copy's source file instead, which
BookOrbit's sibling sync carries to the other EPUBs, and serve the new
primary when the device downloads the book again.
Fixes#1786
A replacement stayed ambiguous after the user confirmed it, so every
position the Kobo sent was discarded and annotation pulls failed with 409.
Confirming now makes it the device's copy and moves the stored bookmark
onto it. It still cannot drag saved progress backwards, and that is
judged by position, since its percentages run on a different scale.
Ref #1786
Every position conversion resolved the delivered artifact by hashing the
whole source file, so a multi-gigabyte read-along outlasted the Kobo's 30 s
request timeout. Remember each source's hash by its file signature and
hash again only after the file changes. Reusing the stamped Kobo location
on device syncs (defect 3) is left out: that stamp is not per device.
Ref #1786
A replacement byte-identical to the device's copy was never downloaded, so
it stayed offered and every bookmark the device pushed was discarded.
Keep the current delivery instead, and tell the user in the replacement
panel that the device already has this copy.
Ref #1786
A read-along save can hold the sentence being narrated without a text
position, and the Kobo was then sent a bare percentage, so it opened the
book at its start. Turn the narration marker into a text position and a
Kobo location when the saved progress has no cfi.
Ref #1786
A replacement prepared before the device ever received the book had no
earlier delivery to fall back to, so every sync failed the book. It now
stops standing for the book and the book gets an ordinary first offer.
Cancelling a replacement the device has not fetched now clears it, hands
the book back to the earlier delivery, and re-announces it. Update the
download E2E suite for the stale-offer replacement from #1816.
Ref #1786
A store cursor over the carry limit was dropped while the device was still
told to continue, so the store restarted at its first page forever. Stop
store paging for that sync instead, and carry Kobo's cursor verbatim in a
compact PX2 token so the limit can rise to 7,000 characters. Tokens in the
old format still decode.
Fixes#1679
When a book's file changed after it was offered to a Kobo, every download
returned 409 because the offer could no longer be rebuilt. A device download
now retires that offer and serves a fresh one from the current file, and the
device bookmark adopts the copy it downloaded. Log the conflict code when a
sync defers a delivery. Also drop a dev-only relocate log from the reader.
Fixes#1816
fix(kobo): reconcile counter estimates on their counter interval (#1796)
Stop counting offline Kobo reading twice. Record when each counter
estimate's interval began, split measured sessions across intervals by
wall-clock overlap, and subtract measured seconds instead of dropping the
estimate, so both arrival orders converge.
Closes#1667
Co-authored-by: neon <18559658+neonsolstice@users.noreply.github.com>
feat(metadata): extract metadata from file paths with regex rules (#1669)
Let libraries define ordered regex rules that read title, authors, series,
series number and dates from each book's path relative to its library folder.
Run rules in a bounded worker pool, honour source precedence and field locks,
and apply them on scans, uploads and watcher moves. Also retry unprobed M4B
tag writes with the mp4 muxer.
Closes#1351
Co-authored-by: neon <18559658+neonsolstice@users.noreply.github.com>
Show the global rule and each library's own rule on Field Rules-style tabs, with live counts per
condition, the score histogram and the books a run would take. Store only what a library changes,
run every library with its own rule, and record each library's run. Also fix the DjVu queue test
that assumed which request waits, and inventory the DjVu reader routes.
Reader › Fonts now uses the Server Fonts ledger, family sheet and upload tray for a reader's own
fonts, and lists the server's fonts with a show/hide switch and a preview sheet. The shared pieces
move to features/settings/fonts and the ledger columns narrow to give the specimen room.
Server: additive GET /fonts/usage and PATCH /fonts/families, upload error codes, and a 409 instead
of a 500 when a reader uploads a second file for the same face.
Changes, Sign-ins, Security and All views replace the filter form. Repeats fold into one row,
sign-ins fold per person, day and network, and rows name their targets; events now snapshot
the target's name so deleted ones keep it. The drawer adds the event's facts, the target's
history, what the same person did around it, and one-click scoping. New additive routes:
/audit-log/summary, /sign-ins and /export; retention is editable from the page.
Each family is a row with its specimen, a weight grid of the styles readers get and how readers
use it; its sheet previews the face on reader pages and inside the reader's own font list. Uploads
answer per file in one tray, removing a family confirms with its cost, and a rename is one request.
Server: additive GET /server-fonts/usage (counts only), PATCH /server-fonts/families, upload error codes.
Show each custom field on one row with its type, libraries and how many
books are filled, and edit, create, archive and delete it in the settings
sheet. Order Edit Metadata and card values by the order set on this page,
and add usageByLibrary to the admin field list, an additive field.
Replace the card grid with medals, a showcase, category shelves, a record book and a
detail sheet. Add GET /achievements/hints, which explains how to unlock what is still
locked from the user's own library and reading history, backed by a new page-count
index. Year-Round Reader now reports its months read as progress.
Library, smart scope, collection, authors and series pages share a new
BrowseHeader: the place and its actions on one row, then search, sort,
filters and facets in one bar that folds groups instead of wrapping.
Format chips count from the new POST /libraries/:id/books/facets, built
on the listing's own query; scopes and collections gain filter and sort.
Three defects, found while planning the Android client:
- The email module's providers, preferences, templates, recipient
groups, tests and send requests had no types in packages/types, so
every client mirrored them from server code. They are typed now, from
the JSON the routes return and accept, and the services and DTOs
declare them. No response changes.
- Narration audio paths were decoded twice: the router decodes the
wildcard, and the route decoded it again, so a name holding a literal
% resolved to the wrong entry or none. A path is now decoded once and
matched to the playlist's hrefs exactly, with an exact entry name
preferred. Each request also read the whole audio entry into memory;
it is now streamed, read from its offset when stored and inflated and
skipped when deflated, with the same headers and Range handling, and
an open range at or past the end is a 416. Both narration audio
routes skip response compression, which dropped Content-Length.
- A playlist's section starts left out any clip of unknown length.
They now use the media:duration each overlay declares when a clip's
own length is unknown. A seq that interleaves par and seq children
was reordered by the XML parser's grouping; SMIL is now read in
document order.
Three book request defects, found while planning the Android client:
- The sweep that cancels a self-serve request nobody picked a release
for timed it from createdAt. A self-fulfiller who took on an older
request somebody else made inherited that clock, so a request taken
on more than six hours after it was made was cancelled at the next
sweep and dismissed for the person who asked for it. It is now timed
from when the request became self-serve, the decidedAt that creating
it self-serve and taking it on both set.
- The grab routes kept Fastify's 1 MiB body limit while their DTO takes
a base64 .torrent of up to 4 Mi characters and the service 2 MiB
decoded, so a .torrent over about 768 KiB was refused with a 413
before the DTO was read. Both grab routes now declare a body limit
that fits the DTO.
- Approving with a new library and no folder kept the request's stored
folder, which belongs to its old library, and was refused with
SUBMIT_FOLDER_NOT_IN_LIBRARY; nothing the approver could send cleared
it. Rerouting without a folder now leaves the old one behind.
Five defects in resumable uploads, found while planning the Android
client's transfers:
- A create under a key whose session had failed, been cancelled or
expired handed that dead session back, and the unique index refused a
new one, so a client that keeps its key across Try again could never
succeed. A dead session now releases its key, keeping its row and
outcome, and a new session is made under it.
- A failed import was stored as UPLOAD_IMPORT_FAILED whatever the error,
with the raw exception message, which can carry server paths. A coded
upload error now keeps its own code and message; anything else is
stored with a fixed message and the raw error goes to the log.
- Upload capabilities listed every reachable library whatever its type,
and the session routes had no type check. Only book libraries are
offered or accepted, as on the older library upload route.
- Contents that do not match the format were refused only at complete,
after every byte was sent, and the session stayed receiving. The first
chunk is now checked against the format's signature, and refused
contents end the session and free its staged bytes.
- A chunk's body was read while the session's lock was held, with no
timeout, so a connection gone half-open blocked every later chunk,
complete and cancel for that session until the server restarted. The
body is now read before the lock is taken, and a chunk that sends
nothing for two minutes is given up.
The response shapes and request fields are unchanged.
Replace the lattice placeholder with a typesetting engine that measures real glyphs, balances line
breaks and parses series, subtitles and sort-form authors before drawing.
Add a placeholderCoverStyle display preference, Bauhaus by default, with a style switcher and sample
shelf in Book Covers settings. Colours follow the series, author or title, so a cover stays the same
when genres arrive later; only Paperback's bands use genre.
Scanned books are commonly distributed as DjVu, which the scanner only stored as a supplement.
Decode pages on the server with DjVuLibre so DjVu imports with a cover and page count, opens in
the image reader with its embedded OCR text, and can be the preferred format of an ordinary email
recipient. Migration 0113 widens the recipient format check to allow it.
Decoding runs on validated private snapshots behind a bounded, reader-first queue, and
revalidation and cached pages are answered without a decoder. Released iOS builds cannot open
DjVu, so series continuation offers it only to callers sending includeDjvu=true, and the
dashboard and notebook read file never name it. The first full scan after upgrading promotes
stored .djvu supplements to content.
Use the source finish day when importing completed KOReader books so
completion timelines reflect reading history instead of the sync date.
Preserve newer manual edits and retain fallback behavior for unusable dates.
Closes#1783
Copy unchanged ZIP records without inflating and recompressing book content
so metadata saves avoid processing every image, chapter, and font.
Use bounded file I/O and validate archive structure while preserving atomic
replacement and original entry compression. Cover ZIP64, descriptor variants,
byte preservation, repeated saves, and filesystem failures in regression tests.
Closes#1798
Resolve OPF and Dublin Core namespaces so valid prefixed metadata retains
book language and avoids incorrect English dictionary lookups.
Preserve element order and identifiers, and cover reader metadata and
narration with unit and HTTP regression tests.
Closes#1789
Library access roles had no effect on metadata editing: only the global Edit metadata permission
was checked, so a Viewer could edit every library they could see and an Editor could not be
limited to one library. Require Editor or Owner on the book's library for every metadata write,
including bulk edits, locks, covers, file renames, entity manager changes and author edits, and
hide the matching actions in the web client.
Keep existing access levels when an admin saves a user's library list. The old replace reset every
row to Viewer, which would now silently revoke editing on each user save.
Existing grants are Viewer, so after upgrading, admins must set Editor on each library where a
non-superuser should keep editing metadata. The released iOS app already gates editing on Editor.
Closes#1778
The authorization matrix fails for any live route that is neither inventoried nor in its frozen
gap baseline, and GET /server-identity arrived without an entry. Record it as public with no
permission or library access, which is exactly what the route serves.
Local connection-address builds created server_identity from a migration whose ledger row is
dated after 0109 to 0111. Drizzle applies only migrations newer than the newest recorded one, so
a database that ran one of those builds would skip the release migrations recorded before it,
including 0111_add_koreader_session_tombstones, on its next upgrade.
Drop that one record by its exact hash before migrating. 0112 creates the same table only when it
is missing, so the installation UUID survives and the table is recorded under its released
migration. A journal that owns the hash keeps the record, and builds without this step never
touch it, so the server that ran the unreleased build still restarts normally until upgraded.
The iOS app lets an account reach one server through several addresses, such as a home network
IP and a public name. Before it sends credentials to an alternative address it needs to know the
address leads to the same installation, so expose a public, credential-free GET /server-identity
returning a persisted UUID, and add the same optional instanceId to app-info with the
server-connection-identity feature flag.
The UUID lives in a singleton server_identity row shared by every replica, created on first use
and cached per process. An unreadable identity omits instanceId instead of failing app-info,
which clients rely on for feature detection. Existing routes and bodies are unchanged and the
web client does not call the new route; the iOS app does.
The migration uses IF NOT EXISTS because an unreleased build created the same table as 0109.
Keep the narration marker a reading progress write leaves out. The
save turned an omitted positionSeconds, mediaOverlayFragment or
mediaOverlaySectionIndex into null and wrote it, so any client saving
only its text position erased the marker another client had left. An
explicit null still clears it. The audiobook-to-ebook sync now moves
narrationUpdatedAt and the narration percentage with the marker it
writes, so a comparison of stamps no longer prefers an older text
position.
Answer TTS requests as their types say. A position never saved is
JSON null rather than an empty 200 body, and the book preferences PUT
answers providerId, voiceId, speed and bookId, with the provider as a
string. TtsPosition gains the updatedAt the route has always sent.
Refuse with a 400 what used to reach the database as a 500: a null
speed for the account (now left unchanged), a provider id that names
no provider, a voiceId over 200 characters, a cfi, fragment or Kobo
location longer than its column, and a fractional or oversized
section index. Voices for a provider id that is not a number answer
404.
Match the reader's blocks in chapter text: tag names now match whole,
so <picture>, <param>, <link>, <track> and SVG shapes no longer open
blocks foliate does not count. The chapter href is percent-decoded,
and cached text is keyed by the file's modified time.
Build a narration playlist once per copy of the file instead of on
every overlay audio request, answer an unsatisfiable range with 416
and ignore an unreadable Range header. The iOS app now plays online
read-along audio from this route in 512 KB ranges, so it is asked
several times per clip.
Give plain synthesis the captioned path's 120 s limit beside the
client's own cancel signal, and voice listing and connection tests a
30 s one.
Fix the server gaps found while porting book detail actions to Android,
checked against the iOS app:
- audit: record a metadata refresh only when it writes; a preview,
which every editor lookup takes, was logged as "Refreshed metadata"
- personal note: leave the note alone when the body has no note key,
as the other partial routes do; null still clears it, and the web
and every iOS build send the key
- covers: answer re-extract-cover with 409 when every slot it would
write is locked, as the other cover writes do, instead of an empty
count a client cannot tell from a file with no art
- types: describe the cover search response as the bare array the
route returns
Fix the server and web gaps found while porting the PDF and comic
readers and offline downloads to Android, checked against the iOS app:
- KOReader: answer a pull for a PDF or comic with the page number when a
BookOrbit client wrote last, instead of a percentage with no page
- comics: key the zip offset, RAR and CB7 caches by file id and
modification time, so pages are not served from stale offsets after a
metadata write-back rewrites the archive
- comics: answer an unreadable archive with 422 rather than a page count
of 0, and extract CB7 with its folders so same-named pages in
different folders no longer overwrite each other
- book files: send an ETag and Last-Modified on serve and download and
honour If-None-Match and If-Range, clamp a range that runs past the
end, answer suffix ranges with 206, and honour Range on download,
which advertised it but sent the whole file
- book files: leave raw file routes uncompressed, so FB2 and unknown
formats keep their Content-Length
- progress: reject a fractional or negative page number with 400 rather
than failing with 500 at the database
- types: declare the comic page-count response shared by the web and
the apps
- web: open a percentage-only PDF position at its estimated page, and
flush pending PDF and comic progress before the reading session ends
Preserve reset-hold notices when recovery requests fail and prevent
stale responses from replacing another book's progress. Refresh holds
after permission changes and rereads, and cover recovery and session
deletion with regression tests.
Ref #1355
Deleting a KOReader session left its page events stored, so the next upload within the
session gap re-derived it. Record the deleted span in koreader_session_tombstones and skip
those events during derivation. Show the reset-hold notice on the Reading Log as well,
since Reset is reachable there and a held device otherwise reads as unexplained progress.
Refs #1355
A stream copy of a large audiobook onto a network share outlived the fixed 60s ffmpeg timeout
and failed the write. FFmpeg now reports progress and is stopped only once its output stops
growing for max(2 min, bytes written at 1 MiB/s), which also covers the silent flush on close.
Fixes#1769
Honor Retry-After for Newznab searches and NZB downloads so transient
rate limits do not immediately end automatic fulfillment. Bound retries
and preserve deadlines without restarting the outer retry budget.
Add unit coverage for rate-limit headers, exhaustion, cancellation,
redirects, and suppression of nested retries. Verify automatic grab
recovery with a real HTTP reproduction.
Fixes#1749
A voice or speed change invalidated the playing block's end handler, so the next block was never
requested, and a change during loading left the player loading forever. Hand the playing block over
to the new settings, refetch any block playback was waiting on, and cancel superseded requests in
the client and at the provider. Audio that arrives during a pause no longer resumes playback.
Fixes#1734
Fold six tabs into Recipients, Mail servers, Templates and History, where only problems get
colour and one star sets each default through email_preferences. Replace the send dialog with
a sheet that preselects the default recipient and follows the send. Server additions are
optional fields and routes, so released iOS builds keep working.
Probe equally ranked comic runs within the existing search budget to prevent reprints from hiding original issues. Require an unambiguous result for automatic matching while preserving manual selection and saved-ID refreshes.
Add regression coverage for the reported Generation X match, padded issue numbers, competing runs, provider failures, deadlines, and cache isolation.
Fixes#1238
Read every EPUB 3 role refinement so authors and narrators survive
regardless of role order, including creators who hold both roles.
Preserve EPUB 2 role precedence and contributor handling, and add
regression coverage for ordering, normalization, and deduplication.
Fixes#1344
A cached Hardcover match outranked a corrected Hardcover ID or ISBN, so progress kept going to the
wrong book. Record the identifiers each link came from and re-check it when they change, honour the
metadata edition on a match, and release the old book's read links so they are never reused.
Closes#1488
FFmpeg picks its ipod muxer from the .m4b temp name, and that muxer rejects the
MP3 an Audiobookshelf copy merge leaves inside an MP4. Probe the audio codec and
use the mp4 muxer for anything but AAC, ALAC and AC-3, so other files keep their
iTunes file type. The rename collision log now names the path that collided.
Closes#1593
Keep disabled Prowlarr limits from being reapplied through tracker metadata,
cached releases, or stale source rows after a failed synchronization.
Cover policy changes, synchronization, both torrent transports, client defaults,
and the PostgreSQL join with regression tests.
Closes#1625
Nest 12 runs global guards on gateway messages, and the HTTP JWT guard
crashed every subscribe on a missing Authorization header. Skip it for
sockets, which authenticate in handleConnection, and make subscribe
handlers wait for that handshake. Library rooms now require library
access, and a real-socket test runs the app's global guards.
Closes#1743
Audiobookshelf reports a progress row's creation time as its start, while
clients set the finish date freely, so a backfilled finish can predate it.
One such row broke user_book_status's date check and the whole status batch
with it. Keep the finish date and drop the start, and store the database's
reason on a failed run instead of the failed SQL.
Closes#1678
The iOS library filter offers languages as chips and needs the whole list, not a typeahead. A blank
term on /metadata/languages now returns every language value on a visible book, most common first
and capped at 200, under the same library and content-filter scope as the search. Every other
suggestion endpoint still answers a blank term with nothing.
The web client never sends a blank term. Released iOS builds do, when the Smart Shelf language
picker opens, and now see the list there instead of a search prompt. The iOS filter calls this
route too, though nothing in the web client shows it.
Ref bookorbit/bookorbit-ios-support#54
epubBuilder writes the OPF root as `mlns=` for `xmlns=`. Once a metadata save
bound `opf:` on <package>, foliate matched OPF elements by namespace, found no
spine, and failed with "reading 'children'". Match by namespace only when the
root is in it, and have the writer add the missing default namespace so
rewritten files also open in released iOS builds.
Closes#1680
The player follows each track's stored sort order, which overlapping
folder scans, Add File, and imports joining a book could leave out of
step with the file names the Files tab shows. Realign a book's tracks
after each of those writes, under the book row lock.
Closes#1681
neon
François-Xavier Payet
Dennis