Squash of feat/koreader-widget-board. KOReader plugin 2.0.0 gets the widget-board dashboard, redesigned
book details, menus, lists and dialogs, Decor page and reading companions. Highlights, notes and bookmarks
now stay in step across web, Kobo, KOReader and the phones (defects D-00 to D-20, migration 0121), and
reading progress and narration hand off cleanly between devices.
A Kobo downloads a cover only for a new entitlement, so a changed cover
sent as changed metadata left the book with a text-only cover until the
home screen fetched a small tile. Record the cover id on each device
snapshot row and re-announce the book when it moves; rows from before
the change learn it silently. Serve tile-size requests at a full-screen
size so the sleep screen never stretches a tile.
A Kobo asks for each cover at its screen size but always got the
400x600 grid thumbnail, so sleep screens and full-screen covers were
blurry. Scale the full cover to fit the requested box instead, behind a
new Full-size covers setting that is on by default. Drop the stale
UserStorage entries from the authorization matrix gap baseline.
Fixes#924
When a Kobo syncs after finishing a book, its finished state closes the
reading attempt before the offline sessions arrive, and those sessions
opened a second attempt, marking the book as reread or read twice.
Activity dated within the finished read now joins it, activity from
before it began joins nothing, and an undated start never widens the
range. Update the reading-attempts E2E Kobo test to use a real device
that holds the book.
Fixes#1224Fixes#1182
A Kobo dogear was stored as an empty yellow highlight. Land it as a
BookOrbit bookmark linked to the device's id, serve it back as a dogear,
sync deletions both ways, and move dogears already stored as highlights.
Report only books on that Kobo from checkforchanges: a re-registered
device was told about books it never had, so real changes were fetched
outside its sync and never applied. Serve a known annotation at its last
location instead of failing the whole pull when this copy cannot place it.
Fixes#1374
A replacement stayed ambiguous after the user confirmed it, so every
position the Kobo sent was discarded and annotation pulls failed with 409.
Confirming now makes it the device's copy and moves the stored bookmark
onto it. It still cannot drag saved progress backwards, and that is
judged by position, since its percentages run on a different scale.
Ref #1786
A replacement byte-identical to the device's copy was never downloaded, so
it stayed offered and every bookmark the device pushed was discarded.
Keep the current delivery instead, and tell the user in the replacement
panel that the device already has this copy.
Ref #1786
A replacement prepared before the device ever received the book had no
earlier delivery to fall back to, so every sync failed the book. It now
stops standing for the book and the book gets an ordinary first offer.
Cancelling a replacement the device has not fetched now clears it, hands
the book back to the earlier delivery, and re-announces it. Update the
download E2E suite for the stale-offer replacement from #1816.
Ref #1786
fix(kobo): reconcile counter estimates on their counter interval (#1796)
Stop counting offline Kobo reading twice. Record when each counter
estimate's interval began, split measured sessions across intervals by
wall-clock overlap, and subtract measured seconds instead of dropping the
estimate, so both arrival orders converge.
Closes#1667
Co-authored-by: neon <18559658+neonsolstice@users.noreply.github.com>
feat(metadata): extract metadata from file paths with regex rules (#1669)
Let libraries define ordered regex rules that read title, authors, series,
series number and dates from each book's path relative to its library folder.
Run rules in a bounded worker pool, honour source precedence and field locks,
and apply them on scans, uploads and watcher moves. Also retry unprobed M4B
tag writes with the mp4 muxer.
Closes#1351
Co-authored-by: neon <18559658+neonsolstice@users.noreply.github.com>
Show the global rule and each library's own rule on Field Rules-style tabs, with live counts per
condition, the score histogram and the books a run would take. Store only what a library changes,
run every library with its own rule, and record each library's run. Also fix the DjVu queue test
that assumed which request waits, and inventory the DjVu reader routes.
Reader › Fonts now uses the Server Fonts ledger, family sheet and upload tray for a reader's own
fonts, and lists the server's fonts with a show/hide switch and a preview sheet. The shared pieces
move to features/settings/fonts and the ledger columns narrow to give the specimen room.
Server: additive GET /fonts/usage and PATCH /fonts/families, upload error codes, and a 409 instead
of a 500 when a reader uploads a second file for the same face.
Changes, Sign-ins, Security and All views replace the filter form. Repeats fold into one row,
sign-ins fold per person, day and network, and rows name their targets; events now snapshot
the target's name so deleted ones keep it. The drawer adds the event's facts, the target's
history, what the same person did around it, and one-click scoping. New additive routes:
/audit-log/summary, /sign-ins and /export; retention is editable from the page.
Each family is a row with its specimen, a weight grid of the styles readers get and how readers
use it; its sheet previews the face on reader pages and inside the reader's own font list. Uploads
answer per file in one tray, removing a family confirms with its cost, and a rename is one request.
Server: additive GET /server-fonts/usage (counts only), PATCH /server-fonts/families, upload error codes.
Replace the card grid with medals, a showcase, category shelves, a record book and a
detail sheet. Add GET /achievements/hints, which explains how to unlock what is still
locked from the user's own library and reading history, backed by a new page-count
index. Year-Round Reader now reports its months read as progress.
Library, smart scope, collection, authors and series pages share a new
BrowseHeader: the place and its actions on one row, then search, sort,
filters and facets in one bar that folds groups instead of wrapping.
Format chips count from the new POST /libraries/:id/books/facets, built
on the listing's own query; scopes and collections gain filter and sort.
Scanned books are commonly distributed as DjVu, which the scanner only stored as a supplement.
Decode pages on the server with DjVuLibre so DjVu imports with a cover and page count, opens in
the image reader with its embedded OCR text, and can be the preferred format of an ordinary email
recipient. Migration 0113 widens the recipient format check to allow it.
Decoding runs on validated private snapshots behind a bounded, reader-first queue, and
revalidation and cached pages are answered without a decoder. Released iOS builds cannot open
DjVu, so series continuation offers it only to callers sending includeDjvu=true, and the
dashboard and notebook read file never name it. The first full scan after upgrading promotes
stored .djvu supplements to content.
Use the source finish day when importing completed KOReader books so
completion timelines reflect reading history instead of the sync date.
Preserve newer manual edits and retain fallback behavior for unusable dates.
Closes#1783
Resolve OPF and Dublin Core namespaces so valid prefixed metadata retains
book language and avoids incorrect English dictionary lookups.
Preserve element order and identifiers, and cover reader metadata and
narration with unit and HTTP regression tests.
Closes#1789
Library access roles had no effect on metadata editing: only the global Edit metadata permission
was checked, so a Viewer could edit every library they could see and an Editor could not be
limited to one library. Require Editor or Owner on the book's library for every metadata write,
including bulk edits, locks, covers, file renames, entity manager changes and author edits, and
hide the matching actions in the web client.
Keep existing access levels when an admin saves a user's library list. The old replace reset every
row to Viewer, which would now silently revoke editing on each user save.
Existing grants are Viewer, so after upgrading, admins must set Editor on each library where a
non-superuser should keep editing metadata. The released iOS app already gates editing on Editor.
Closes#1778
The authorization matrix fails for any live route that is neither inventoried nor in its frozen
gap baseline, and GET /server-identity arrived without an entry. Record it as public with no
permission or library access, which is exactly what the route serves.
The iOS app lets an account reach one server through several addresses, such as a home network
IP and a public name. Before it sends credentials to an alternative address it needs to know the
address leads to the same installation, so expose a public, credential-free GET /server-identity
returning a persisted UUID, and add the same optional instanceId to app-info with the
server-connection-identity feature flag.
The UUID lives in a singleton server_identity row shared by every replica, created on first use
and cached per process. An unreadable identity omits instanceId instead of failing app-info,
which clients rely on for feature detection. Existing routes and bodies are unchanged and the
web client does not call the new route; the iOS app does.
The migration uses IF NOT EXISTS because an unreleased build created the same table as 0109.
Fix the server and web gaps found while porting the PDF and comic
readers and offline downloads to Android, checked against the iOS app:
- KOReader: answer a pull for a PDF or comic with the page number when a
BookOrbit client wrote last, instead of a percentage with no page
- comics: key the zip offset, RAR and CB7 caches by file id and
modification time, so pages are not served from stale offsets after a
metadata write-back rewrites the archive
- comics: answer an unreadable archive with 422 rather than a page count
of 0, and extract CB7 with its folders so same-named pages in
different folders no longer overwrite each other
- book files: send an ETag and Last-Modified on serve and download and
honour If-None-Match and If-Range, clamp a range that runs past the
end, answer suffix ranges with 206, and honour Range on download,
which advertised it but sent the whole file
- book files: leave raw file routes uncompressed, so FB2 and unknown
formats keep their Content-Length
- progress: reject a fractional or negative page number with 400 rather
than failing with 500 at the database
- types: declare the comic page-count response shared by the web and
the apps
- web: open a percentage-only PDF position at its estimated page, and
flush pending PDF and comic progress before the reading session ends
Preserve reset-hold notices when recovery requests fail and prevent
stale responses from replacing another book's progress. Refresh holds
after permission changes and rereads, and cover recovery and session
deletion with regression tests.
Ref #1355
Deleting a KOReader session left its page events stored, so the next upload within the
session gap re-derived it. Record the deleted span in koreader_session_tombstones and skip
those events during derivation. Show the reset-hold notice on the Reading Log as well,
since Reset is reachable there and a held device otherwise reads as unexplained progress.
Refs #1355
Fold six tabs into Recipients, Mail servers, Templates and History, where only problems get
colour and one star sets each default through email_preferences. Replace the send dialog with
a sheet that preselects the default recipient and follows the send. Server additions are
optional fields and routes, so released iOS builds keep working.
A cached Hardcover match outranked a corrected Hardcover ID or ISBN, so progress kept going to the
wrong book. Record the identifiers each link came from and re-check it when they change, honour the
metadata edition on a match, and release the old book's read links so they are never reused.
Closes#1488
Keep disabled Prowlarr limits from being reapplied through tracker metadata,
cached releases, or stale source rows after a failed synchronization.
Cover policy changes, synchronization, both torrent transports, client defaults,
and the PostgreSQL join with regression tests.
Closes#1625
Audiobookshelf reports a progress row's creation time as its start, while
clients set the finish date freely, so a backfilled finish can predate it.
One such row broke user_book_status's date check and the whole status batch
with it. Keep the finish date and drop the start, and store the database's
reason on a failed run instead of the failed SQL.
Closes#1678
The player follows each track's stored sort order, which overlapping
folder scans, Add File, and imports joining a book could leave out of
step with the file names the Files tab shows. Realign a book's tracks
after each of those writes, under the book row lock.
Closes#1681
Book Dock finalize, which also files book requests, left each audio file's
duration empty until the next library scan. The player derives progress only
from those, so a filed audiobook stayed at 0% and never reached Continue
Listening. Finalize now measures them once metadata is applied.
Refs #1757
Map <Genre> to genres and <Tags> to tags instead of using tags as a genre
fallback, which dropped them whenever a Genre was present. Read the
ComicBookInfo genre field so those comics keep their genres.
Fixes#1572
Add original and text-only EPUB delivery choices across Kobo, KOReader,
OPDS, web downloads and email while keeping source files unchanged.
Persist server defaults, device overrides and captured artifact identities
so retries and guided replacement retain the selected rendition.
Preserve saved progress across first opens, replacement resets and stale
updates, and project annotations against the copy delivered to each device.
Handle XML resource references, large inodes, retry backoff and background
worker state correctly. Consolidate the schema changes into one migration
and retain released-client request and response compatibility.
Verify with the full server and client checks, delivery and sync HTTP suites,
independent code reviews, and live KOReader download/settings/reader proof.
Fixes#1496Closes#1603Closes#1714Fixes#1723
Ref #1633
Ref #1508
Ref #1621
Ref #1136
Repeated searches and refreshes asked every provider again, which is slow and runs into provider
rate limits. Keep each provider's answer in Postgres, shared by all users, and reuse it until an
admin-set lifetime runs out. Search screens mark saved results and offer a fresh search, and
explicit refreshes stay live, falling back to a saved answer only when a provider fails.
Empty answers are only kept for API providers that report failures reliably, and a cache-only
request never calls a provider in cooldown. Book Dock's manual refetch stays live-only because
the dock has nowhere to report a fallback. Released iOS builds keep working: every new request
field is optional and newer clients gate it on the metadata-search-cache server feature.
Keep books and comic archives under the existing ebook request kind so searches, ownership checks,
format preferences and release profiles agree without changing the API contract. Cover the combined
formats and library access rules with unit and end-to-end tests.
Ref #1748
Lead fetches with the primary file, give Book Dock its library's rules and read-along media,
default audiobook fields to sources that carry them, and flag providers that never fill a field.
Joining a format now fills gaps only and respects locks; a move picks its primary as a scan would.
Link two ordinary requests as companions, filed into one book, with a Both search,
grouped list rows, a tabbed release picker and a book-page request for the missing
format. Retire comic requests, keeping old rows readable. File Book Dock covers by the
field rules with a shape check, and re-pick a moved book's primary by its new library.
Use file identities for root-level imports and renames to match scanner
grouping and prevent unrelated books from sharing metadata.
Cover repeated scans, metadata write-back, and stable reading and file
identities with regression tests.
Fixes#1599
Return matching item counts with create and update responses so the sidebar
can display the saved scope count without requiring a page reload.
Reuse the listing count logic and verify visibility and shared sidebar state.
Carry ISBN-10 and ISBN-13 independently from accepted providers through
Auto-fill previews, saves and background refreshes. Preserve existing
identifiers and field locks, validate provider checksums and avoid
combining different editions.
Classify embedded ISBNs correctly and support ISBN-10 fallback in
search, Book Dock confidence and file naming.
The walker sorted a book's files by file name, so tracks in disc folders that reuse names, such as
CD 1/01.mp3 and CD 2/01.mp3, interleaved in playback, merged chapters, and written track numbers.
Order them by their whole path instead, everywhere a track order is chosen.
A rescan that only reorders a book's existing tracks now moves every audio bookmark and the
listening progress percentage to the same moment of the same track, and rebuilds merged chapters.
Disc folders are never skipped by incremental scans, so existing libraries are fixed by their next
scan.
Also inventory the notebook and dashboard highlights routes in the authorization matrix, and pin the
app locale in two date tests that depended on the machine's locale.
Books often carry several files, such as an EPUB beside a PDF or an audiobook split into tracks.
Write-back reached only the primary file and the audio tracks, so the other formats kept stale
metadata.
Add an opt-in per-library mode that writes every content file of a book. Target selection,
capability reporting and the Files tab now share one role-aware selector, covers resolve per
medium, and results report per-format counts. Read-along EPUBs get their own toggle, off by
default, and their own size limit of 1000 MB, so they no longer follow the EPUB settings.
Live testing on copies of real libraries surfaced write-path bugs, fixed here as well:
- tag-only writes to chaptered M4B files failed because FFmpeg copied the chapter data track
- an undecodable cover failed every file instead of falling back to metadata only
- failure reasons stored the FFmpeg command line and absolute paths; they are now sanitized
- size limits used the stored size, so an unknown size bypassed them; writes now check the disk
- atomic replace dropped the file mode and ownership
- the EPUB temp name failed on long file names and could clash with a user file
- tracks in disc folders got duplicate titles
- a PDF worker that exited without a result hung its write forever; it now fails and times out
- null on a NOT NULL library setting returned 500 instead of 400
Generates migrations 0104 and 0105. Every new API field is optional, and released iOS builds
decode none of them.
Ref #694
Refresh dashboard and statistics caches immediately after reading history,
projection, reset, import, and shared deletion changes. Coalesce bulk
notifications while keeping committed changes visible to concurrent reads,
and isolate listener failures from successful writes.
Skip unchanged Hardcover attempt updates and protect cache generations
against stale in-flight loads. Cover date edits, projection-only resumes,
rollback, bulk failures, user scope, and cache races with regression tests.
Ref #1695
Register the identity of delivered bytes so Kobo conversions and audio
removal continue matching their source books in KOReader. Keep download
registration recoverable without allowing database lock waits to stall it.
Preserve manual corrections across cached device identities, make review
semantics explicit, and keep cache invalidation bounded for large libraries.
Add unit, database, lifecycle and scale regressions with one generated migration.
Closes#1697
Align collection visibility across OPDS and KOReader with the web catalog.
Count only books the viewer can access in one aggregate query, and cover
public access, private revocation, and filtered counts with regression tests.
Closes#1701
Allow native KEPUB files to exchange precise reading progress, bookmarks
and highlights with Kobo while preserving sync permissions and settings.
Use the original KEPUB archive for Kobo spans and project its added
wrappers away for shared EPUB CFI and KOReader XPointer positions.
Preserve existing EPUB conversion behavior and API compatibility with
released iOS clients.
Cover native KEPUB context, canonical positions and two-way exchange with
unit tests and paired EPUB/KEPUB API E2E scenarios.
Ref bookorbit/bookorbit-ios-support#36
In folder-as-book libraries a pattern that files several books under one folder either failed on
the folder_path unique index or silently merged a different book into the folder's owner, deleting
its reading history. Check the folder owner on every rename path and merge only when both books
share a title and primary author; otherwise skip with a clear reason. The bulk preview now reports
these as collisions.
Fixes#1682
Add read-only /notebook routes that list highlights, journal entries, bookmarks and reviews
across every book a reader can open, as one cursor-paged stream with per-kind counts, a By Book
listing, a daily review deck, a seeded review shuffle, On this day and a library-wide trash. Each
kind is read through its own index-ordered arm cut at the page size before a UNION ALL merge, so a
deep page costs what the first does, and book labels travel once per page in a sidecar map.
Scope follows library access and content filters. Everything is additive and advertised as the
notebook-hub app-info feature. The iOS app calls these routes and the web client does not yet, so
they look unused from the client alone. No migration: the extra indexes for the edited sort and
search wait until the chapter-editor migrations land.
Add per-book journal entries keyed by a client id, a star toggle that does not bump the annotation
version pushed to devices, a per-user colour legend, and an edit route so bookmark title and note
changes reach the server. All of it is additive and advertised through app-info features, so iOS
sends new fields only to servers that support them. Also sort annotations by numeric CFI steps
instead of comparing CFI strings.
Convert complete source CFIs on the server to avoid chapter-start resets
and browser-generated locators that disagree with CREngine. Preserve
engine whitespace, Unicode offsets, element anchors and long text nodes.
Handle prefixed OPF namespaces and retry outdated annotation conversions.
Closes#1621Closes#1288
neon
François-Xavier Payet
Dennis