feat(metadata): extract metadata from file paths with regex rules (#1669)
Let libraries define ordered regex rules that read title, authors, series,
series number and dates from each book's path relative to its library folder.
Run rules in a bounded worker pool, honour source precedence and field locks,
and apply them on scans, uploads and watcher moves. Also retry unprobed M4B
tag writes with the mp4 muxer.
Closes#1351
Co-authored-by: neon <18559658+neonsolstice@users.noreply.github.com>
Clip the hidden genre measuring strip so its full single-line width no
longer widens the scrollable middle column, and let the rating row and
reading activity stats wrap in narrow columns.
Put every auth page on a shared Ex Libris plate with signature-line fields, a show-password toggle and a Caps Lock hint.
Show sign-in messages outside the password form, with Retry, a reset link on lockout and a reset confirmation.
Add language and endpaper pickers (eight repeats) to the auth toolbar, replacing the background and corner controls.
Show the global rule and each library's own rule on Field Rules-style tabs, with live counts per
condition, the score histogram and the books a run would take. Store only what a library changes,
run every library with its own rule, and record each library's run. Also fix the DjVu queue test
that assumed which request waits, and inventory the DjVu reader routes.
Reader › Fonts now uses the Server Fonts ledger, family sheet and upload tray for a reader's own
fonts, and lists the server's fonts with a show/hide switch and a preview sheet. The shared pieces
move to features/settings/fonts and the ledger columns narrow to give the specimen room.
Server: additive GET /fonts/usage and PATCH /fonts/families, upload error codes, and a 409 instead
of a 500 when a reader uploads a second file for the same face.
Changes, Sign-ins, Security and All views replace the filter form. Repeats fold into one row,
sign-ins fold per person, day and network, and rows name their targets; events now snapshot
the target's name so deleted ones keep it. The drawer adds the event's facts, the target's
history, what the same person did around it, and one-click scoping. New additive routes:
/audit-log/summary, /sign-ins and /export; retention is editable from the page.
Each family is a row with its specimen, a weight grid of the styles readers get and how readers
use it; its sheet previews the face on reader pages and inside the reader's own font list. Uploads
answer per file in one tray, removing a family confirms with its cost, and a rename is one request.
Server: additive GET /server-fonts/usage (counts only), PATCH /server-fonts/families, upload error codes.
Show each custom field on one row with its type, libraries and how many
books are filled, and edit, create, archive and delete it in the settings
sheet. Order Edit Metadata and card values by the order set on this page,
and add usageByLibrary to the admin field list, an additive field.
Replace the card grid with medals, a showcase, category shelves, a record book and a
detail sheet. Add GET /achievements/hints, which explains how to unlock what is still
locked from the user's own library and reading history, backed by a new page-count
index. Year-Round Reader now reports its months read as progress.
Remove AppHeader. The sidebar gains a Search row that opens a Cmd+K palette (books, authors,
series, places, settings pages and actions), a utility dock above the footer, and a Reading
group below the lists. Every page opens with a shared PageTitleRow; phones without one get a
slim bar. Settings headers gain a full description, and the top glow defaults to 50% with one
reach and strength for every style. The Appearance panel opens past the sidebar, reorders its
controls and names the current pick, and keeps each pastel accent under its vivid one.
Describe the six entity sections once in useSidebarSections and render them with one v-for in
both the expanded list and the rail, instead of twelve hand-written copies. Move data loading to
useSidebarDataSync, scan and import progress to useSidebarLibraryActivity, and the footer to
SidebarFooterBar. The rendered sidebar is unchanged.
Library, smart scope, collection, authors and series pages share a new
BrowseHeader: the place and its actions on one row, then search, sort,
filters and facets in one bar that folds groups instead of wrapping.
Format chips count from the new POST /libraries/:id/books/facets, built
on the listing's own query; scopes and collections gain filter and sort.
Three defects, found while planning the Android client:
- The email module's providers, preferences, templates, recipient
groups, tests and send requests had no types in packages/types, so
every client mirrored them from server code. They are typed now, from
the JSON the routes return and accept, and the services and DTOs
declare them. No response changes.
- Narration audio paths were decoded twice: the router decodes the
wildcard, and the route decoded it again, so a name holding a literal
% resolved to the wrong entry or none. A path is now decoded once and
matched to the playlist's hrefs exactly, with an exact entry name
preferred. Each request also read the whole audio entry into memory;
it is now streamed, read from its offset when stored and inflated and
skipped when deflated, with the same headers and Range handling, and
an open range at or past the end is a 416. Both narration audio
routes skip response compression, which dropped Content-Length.
- A playlist's section starts left out any clip of unknown length.
They now use the media:duration each overlay declares when a clip's
own length is unknown. A seq that interleaves par and seq children
was reordered by the XML parser's grouping; SMIL is now read in
document order.
Three book request defects, found while planning the Android client:
- The sweep that cancels a self-serve request nobody picked a release
for timed it from createdAt. A self-fulfiller who took on an older
request somebody else made inherited that clock, so a request taken
on more than six hours after it was made was cancelled at the next
sweep and dismissed for the person who asked for it. It is now timed
from when the request became self-serve, the decidedAt that creating
it self-serve and taking it on both set.
- The grab routes kept Fastify's 1 MiB body limit while their DTO takes
a base64 .torrent of up to 4 Mi characters and the service 2 MiB
decoded, so a .torrent over about 768 KiB was refused with a 413
before the DTO was read. Both grab routes now declare a body limit
that fits the DTO.
- Approving with a new library and no folder kept the request's stored
folder, which belongs to its old library, and was refused with
SUBMIT_FOLDER_NOT_IN_LIBRARY; nothing the approver could send cleared
it. Rerouting without a folder now leaves the old one behind.
Five defects in resumable uploads, found while planning the Android
client's transfers:
- A create under a key whose session had failed, been cancelled or
expired handed that dead session back, and the unique index refused a
new one, so a client that keeps its key across Try again could never
succeed. A dead session now releases its key, keeping its row and
outcome, and a new session is made under it.
- A failed import was stored as UPLOAD_IMPORT_FAILED whatever the error,
with the raw exception message, which can carry server paths. A coded
upload error now keeps its own code and message; anything else is
stored with a fixed message and the raw error goes to the log.
- Upload capabilities listed every reachable library whatever its type,
and the session routes had no type check. Only book libraries are
offered or accepted, as on the older library upload route.
- Contents that do not match the format were refused only at complete,
after every byte was sent, and the session stayed receiving. The first
chunk is now checked against the format's signature, and refused
contents end the session and free its staged bytes.
- A chunk's body was read while the session's lock was held, with no
timeout, so a connection gone half-open blocked every later chunk,
complete and cancel for that session until the server restarted. The
body is now read before the lock is taken, and a chunk that sends
nothing for two minutes is given up.
The response shapes and request fields are unchanged.
Replace the lattice placeholder with a typesetting engine that measures real glyphs, balances line
breaks and parses series, subtitles and sort-form authors before drawing.
Add a placeholderCoverStyle display preference, Bauhaus by default, with a style switcher and sample
shelf in Book Covers settings. Colours follow the series, author or title, so a cover stays the same
when genres arrive later; only Paperback's bands use genre.
Scanned books are commonly distributed as DjVu, which the scanner only stored as a supplement.
Decode pages on the server with DjVuLibre so DjVu imports with a cover and page count, opens in
the image reader with its embedded OCR text, and can be the preferred format of an ordinary email
recipient. Migration 0113 widens the recipient format check to allow it.
Decoding runs on validated private snapshots behind a bounded, reader-first queue, and
revalidation and cached pages are answered without a decoder. Released iOS builds cannot open
DjVu, so series continuation offers it only to callers sending includeDjvu=true, and the
dashboard and notebook read file never name it. The first full scan after upgrading promotes
stored .djvu supplements to content.
Use the source finish day when importing completed KOReader books so
completion timelines reflect reading history instead of the sync date.
Preserve newer manual edits and retain fallback behavior for unusable dates.
Closes#1783
Copy unchanged ZIP records without inflating and recompressing book content
so metadata saves avoid processing every image, chapter, and font.
Use bounded file I/O and validate archive structure while preserving atomic
replacement and original entry compression. Cover ZIP64, descriptor variants,
byte preservation, repeated saves, and filesystem failures in regression tests.
Closes#1798
Resolve OPF and Dublin Core namespaces so valid prefixed metadata retains
book language and avoids incorrect English dictionary lookups.
Preserve element order and identifiers, and cover reader metadata and
narration with unit and HTTP regression tests.
Closes#1789
Library access roles had no effect on metadata editing: only the global Edit metadata permission
was checked, so a Viewer could edit every library they could see and an Editor could not be
limited to one library. Require Editor or Owner on the book's library for every metadata write,
including bulk edits, locks, covers, file renames, entity manager changes and author edits, and
hide the matching actions in the web client.
Keep existing access levels when an admin saves a user's library list. The old replace reset every
row to Viewer, which would now silently revoke editing on each user save.
Existing grants are Viewer, so after upgrading, admins must set Editor on each library where a
non-superuser should keep editing metadata. The released iOS app already gates editing on Editor.
Closes#1778
The authorization matrix fails for any live route that is neither inventoried nor in its frozen
gap baseline, and GET /server-identity arrived without an entry. Record it as public with no
permission or library access, which is exactly what the route serves.
Local connection-address builds created server_identity from a migration whose ledger row is
dated after 0109 to 0111. Drizzle applies only migrations newer than the newest recorded one, so
a database that ran one of those builds would skip the release migrations recorded before it,
including 0111_add_koreader_session_tombstones, on its next upgrade.
Drop that one record by its exact hash before migrating. 0112 creates the same table only when it
is missing, so the installation UUID survives and the table is recorded under its released
migration. A journal that owns the hash keeps the record, and builds without this step never
touch it, so the server that ran the unreleased build still restarts normally until upgraded.
The iOS app lets an account reach one server through several addresses, such as a home network
IP and a public name. Before it sends credentials to an alternative address it needs to know the
address leads to the same installation, so expose a public, credential-free GET /server-identity
returning a persisted UUID, and add the same optional instanceId to app-info with the
server-connection-identity feature flag.
The UUID lives in a singleton server_identity row shared by every replica, created on first use
and cached per process. An unreadable identity omits instanceId instead of failing app-info,
which clients rely on for feature detection. Existing routes and bodies are unchanged and the
web client does not call the new route; the iOS app does.
The migration uses IF NOT EXISTS because an unreleased build created the same table as 0109.
Keep the narration marker a reading progress write leaves out. The
save turned an omitted positionSeconds, mediaOverlayFragment or
mediaOverlaySectionIndex into null and wrote it, so any client saving
only its text position erased the marker another client had left. An
explicit null still clears it. The audiobook-to-ebook sync now moves
narrationUpdatedAt and the narration percentage with the marker it
writes, so a comparison of stamps no longer prefers an older text
position.
Answer TTS requests as their types say. A position never saved is
JSON null rather than an empty 200 body, and the book preferences PUT
answers providerId, voiceId, speed and bookId, with the provider as a
string. TtsPosition gains the updatedAt the route has always sent.
Refuse with a 400 what used to reach the database as a 500: a null
speed for the account (now left unchanged), a provider id that names
no provider, a voiceId over 200 characters, a cfi, fragment or Kobo
location longer than its column, and a fractional or oversized
section index. Voices for a provider id that is not a number answer
404.
Match the reader's blocks in chapter text: tag names now match whole,
so <picture>, <param>, <link>, <track> and SVG shapes no longer open
blocks foliate does not count. The chapter href is percent-decoded,
and cached text is keyed by the file's modified time.
Build a narration playlist once per copy of the file instead of on
every overlay audio request, answer an unsatisfiable range with 416
and ignore an unreadable Range header. The iOS app now plays online
read-along audio from this route in 512 KB ranges, so it is asked
several times per clip.
Give plain synthesis the captioned path's 120 s limit beside the
client's own cancel signal, and voice listing and connection tests a
30 s one.
The seed names the icon for Epic Voyager (long_book_3) book-marked,
which lucide ships only as an alias of book-bookmark. The web's icon
map had every other seeded name but not this one, so the badge drew the
HelpCircle fallback. The map now has it, and a test reads the seed and
fails if any seeded icon would fall back.
Fix the server gaps found while porting book detail actions to Android,
checked against the iOS app:
- audit: record a metadata refresh only when it writes; a preview,
which every editor lookup takes, was logged as "Refreshed metadata"
- personal note: leave the note alone when the body has no note key,
as the other partial routes do; null still clears it, and the web
and every iOS build send the key
- covers: answer re-extract-cover with 409 when every slot it would
write is locked, as the other cover writes do, instead of an empty
count a client cannot tell from a file with no art
- types: describe the cover search response as the bare array the
route returns
Fix the server and web gaps found while porting the PDF and comic
readers and offline downloads to Android, checked against the iOS app:
- KOReader: answer a pull for a PDF or comic with the page number when a
BookOrbit client wrote last, instead of a percentage with no page
- comics: key the zip offset, RAR and CB7 caches by file id and
modification time, so pages are not served from stale offsets after a
metadata write-back rewrites the archive
- comics: answer an unreadable archive with 422 rather than a page count
of 0, and extract CB7 with its folders so same-named pages in
different folders no longer overwrite each other
- book files: send an ETag and Last-Modified on serve and download and
honour If-None-Match and If-Range, clamp a range that runs past the
end, answer suffix ranges with 206, and honour Range on download,
which advertised it but sent the whole file
- book files: leave raw file routes uncompressed, so FB2 and unknown
formats keep their Content-Length
- progress: reject a fractional or negative page number with 400 rather
than failing with 500 at the database
- types: declare the comic page-count response shared by the web and
the apps
- web: open a percentage-only PDF position at its estimated page, and
flush pending PDF and comic progress before the reading session ends
Preserve reset-hold notices when recovery requests fail and prevent
stale responses from replacing another book's progress. Refresh holds
after permission changes and rereads, and cover recovery and session
deletion with regression tests.
Ref #1355
Deleting a KOReader session left its page events stored, so the next upload within the
session gap re-derived it. Record the deleted span in koreader_session_tombstones and skip
those events during derivation. Show the reset-hold notice on the Reading Log as well,
since Reset is reachable there and a held device otherwise reads as unexplained progress.
Refs #1355
A stream copy of a large audiobook onto a network share outlived the fixed 60s ffmpeg timeout
and failed the write. FFmpeg now reports progress and is stopped only once its output stops
growing for max(2 min, bytes written at 1 MiB/s), which also covers the silent flush on close.
Fixes#1769
Grant cache write access so stale processing can save progress across runs.
Raise the operation budget to 1000 to cover the current backlog in one run.
Retain the policy of 60 inactive days before marking stale and 14 additional
days before closure.
Honor Retry-After for Newznab searches and NZB downloads so transient
rate limits do not immediately end automatic fulfillment. Bound retries
and preserve deadlines without restarting the outer retry budget.
Add unit coverage for rate-limit headers, exhaustion, cancellation,
redirects, and suppression of nested retries. Verify automatic grab
recovery with a real HTTP reproduction.
Fixes#1749
The Book Dock discarded every upload failure, so a rejected file looked like nothing happened.
List failed files with a translated reason, offer retry where it can help, and stop a failed
token refresh, abort or timeout from stalling the upload queue. Dropped files of unsupported
types are reported instead of ignored.
Refs #1739
A voice or speed change invalidated the playing block's end handler, so the next block was never
requested, and a change during loading left the player loading forever. Hand the playing block over
to the new settings, refetch any block playback was waiting on, and cancel superseded requests in
the client and at the provider. Audio that arrives during a pause no longer resumes playback.
Fixes#1734
Fold six tabs into Recipients, Mail servers, Templates and History, where only problems get
colour and one star sets each default through email_preferences. Replace the send dialog with
a sheet that preselects the default recipient and follows the send. Server additions are
optional fields and routes, so released iOS builds keep working.
Rewrite absolute publisher font sizes (keywords and px/pt lengths) as each EPUB loads so they
follow the reader's root size, falling back to the original value when the reader applies no
styles. Skip fixed-layout books and SVG content, and cover the rewrite and the foliate loader
hook with regression tests.
Fixes#1276
Probe equally ranked comic runs within the existing search budget to prevent reprints from hiding original issues. Require an unambiguous result for automatic matching while preserving manual selection and saved-ID refreshes.
Add regression coverage for the reported Generation X match, padded issue numbers, competing runs, provider failures, deadlines, and cache isolation.
Fixes#1238
Inset and center hover metadata within the circular portrait so the book count and last-added date remain readable. Add regression coverage for both shapes and metadata states.
Fixes#1413
Read every EPUB 3 role refinement so authors and narrators survive
regardless of role order, including creators who hold both roles.
Preserve EPUB 2 role precedence and contributor handling, and add
regression coverage for ordering, normalization, and deduplication.
Fixes#1344
Keep EPUB loading functional on browsers without Object.groupBy or Map.groupBy.
Replace the native calls with local helpers and cover missing APIs with EPUB 2
and EPUB 3 metadata regression tests.
Closes#1683
Use English Wiktionary's multilingual definitions endpoint to avoid HTTP 501
responses from language-specific hosts. Keep filtering definitions by the
book's language and cover host selection, language aliases, inflection
lookups, URL encoding, and popover retry recovery with regression tests.
Closes#1439
A cached Hardcover match outranked a corrected Hardcover ID or ISBN, so progress kept going to the
wrong book. Record the identifiers each link came from and re-check it when they change, honour the
metadata edition on a match, and release the old book's read links so they are never reused.
Closes#1488
FFmpeg picks its ipod muxer from the .m4b temp name, and that muxer rejects the
MP3 an Audiobookshelf copy merge leaves inside an MP4. Probe the audio codec and
use the mp4 muxer for anything but AAC, ALAC and AC-3, so other files keep their
iTunes file type. The rename collision log now names the path that collided.
Closes#1593
The table rendered a dash in every column of a collapsed series row,
including Authors, although the row already carries the first
volume's authors and the grid and list views show them. Render them
as the same read-only author chips book rows use.
Closes#1645
Keep disabled Prowlarr limits from being reapplied through tracker metadata,
cached releases, or stale source rows after a failed synchronization.
Cover policy changes, synchronization, both torrent transports, client defaults,
and the PostgreSQL join with regression tests.
Closes#1625
Nest 12 runs global guards on gateway messages, and the HTTP JWT guard
crashed every subscribe on a missing Authorization header. Skip it for
sockets, which authenticate in handleConnection, and make subscribe
handlers wait for that handshake. Library rooms now require library
access, and a real-socket test runs the app's global guards.
Closes#1743
Audiobookshelf reports a progress row's creation time as its start, while
clients set the finish date freely, so a backfilled finish can predate it.
One such row broke user_book_status's date check and the whole status batch
with it. Keep the finish date and drop the start, and store the database's
reason on a failed run instead of the failed SQL.
Closes#1678
neon
dependabot[bot]
Dennis