Migrations/bookorbit
1
0
Fork 0
mirror of https://github.com/bookorbit/bookorbit.git synced 2026-11-03 00:11:37 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(metadata): add external HTTP metadata providers

Browse Source 
Register book-metadata-provider services by address, with no code change per
provider; Settings lists ready-made ones to install. Ranking prefers a record of
the book's own medium. LubimyCzytac moves out to its own provider and Aladin,
whose API closes, is retired; their ids stay. Adds migrations 0129 and 0130.
See docs/METADATA_PROVIDERS.md.

Ref #1405
This commit is contained in:
neon
2026-10-22 21:05:06 +00:00
parent 9e49f496f1
commit b50b8da56c
186 changed files with 59209 additions and 2550 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/BUILDING_METADATA_PROVIDERS.md
+372
View File
@@ -0,0 +1,372 @@
# Building a Metadata Provider
A metadata provider is a small web service that answers BookOrbit's metadata searches from a source
it does not know: a regional catalogue, a fan wiki, a manga database, your own spreadsheet. BookOrbit
calls it over HTTP and never runs its code, so you write it in any language, release it on your own
schedule, and nobody has to change BookOrbit for it.
This guide takes you from an empty folder to a published image that anyone can add to BookOrbit.
To add a provider someone else made, see [External Metadata Providers](METADATA_PROVIDERS.md).
## How it works
```text
BookOrbit your provider the source
| GET /search?query=Dune -------> | |
| | -- its own API or pages ----> |
| | <---------------------------- |
| <---- { "matches": [ ... ] } --- | |
```
Your provider answers three requests:
- `GET /provider/info`: who it is (name, a permanent key, version, icon, what it can do).
- `GET /search`: books matching a title, author or ISBN, best first.
- `GET /book/{id}`: one book by the id your provider gave it earlier. Optional, but recommended.
BookOrbit caches answers, merges them with its other sources through field rules, stores your ids
on the books it matches, and shows your icon and colour wherever your results appear.
## Before you start
- **Check the source's terms.** You are responsible for how your provider uses its source. Prefer an
official API over scraping pages: APIs change less often, and many sites forbid scraping.
- **Choose the key now, and never change it.** The key (such as `mangabaka`) is how BookOrbit files
every id and rating your provider gives it. A provider that changes its key is a new provider to
BookOrbit, and users have to add it again.
- **Choose a language you will keep maintaining.** There are starters in Rust and Go. Rust is what
the published providers use: the same provider ran in about a third of Go's memory. Go needs
nothing but `go run`. Python, Node or anything else that serves JSON works too.
## Quick start
The [bookorbit-metadata-providers](https://github.com/bookorbit/bookorbit-metadata-providers)
repository has a working starter in Rust and one in Go, and a conformance checker. Both starters
serve one made-up book and behave the same; pick the language you prefer.
1. **Copy a starter.**
```bash
git clone https://github.com/bookorbit/bookorbit-metadata-providers.git
```
For Rust:
```bash
cp -R bookorbit-metadata-providers/starters/rust my-provider
```
Or for Go:
```bash
cp -R bookorbit-metadata-providers/starters/go my-provider
```
2. **Run it**, so you can see every endpoint working before you change anything. In Rust,
`cargo run`; in Go, `go run .`. Then:
```bash
curl "http://localhost:8080/search?query=earthsea"
```
3. **Name it.** Change `info()` in `src/main.rs` (Rust) or `main.go` (Go): give it your `name` and
the `key` you chose.
4. **Connect your source.** In `src/source.rs` or `source.go`, replace the example with calls to
your source:
- Search returns matches best first, at most the query's limit.
- Lookup returns the book for an id, or nothing (`None`, `nil`) when the id does not exist.
- Return an error for anything else that goes wrong. Never hide a failure behind an empty list:
to BookOrbit, an empty list means "this book is not there", and it remembers that.
5. **Check it against the contract.** With your provider running:
```bash
cd bookorbit-metadata-providers/conformance && go run . -url http://localhost:8080 -query "Dune"
```
It prints a line per check and ends with the number of failures and warnings. Fix every failure;
each warning says what BookOrbit will do instead.
6. **Try it in BookOrbit.** Add it under **Settings > Metadata > Providers > External providers**
with **Allow private address** on, choose **Test**, turn it on and search for a book.
## Recommendations
**Identity**
- Serve `/provider/info` with a `name`, a `key` and a `version`.
- Bump `version` whenever your answers change shape: new fields, different mapping, a fixed bug.
BookOrbit keeps cached answers per version, so an unchanged version keeps serving old ones.
- Give it an icon (`avatarUrl`) and a `color`, so your results are recognisable at a glance.
**Answers**
- Put the best match first, and return a handful rather than everything: BookOrbit uses at most 50,
and a person picks from the first few.
- Leave out what is not a book on a shelf: reviews, articles, interviews, collections pages.
- Support `GET /book/{id}` and declare `lookup: true` when your source has stable ids. Refreshing a
book by id is faster and more accurate than searching for it again.
- Declare what you can fill with `supportedFields` and `mediaTypes`, so BookOrbit only asks you for
what you have.
- Give each record its own `mediaType` when your source has more than one: a light novel is
`book`, its manga `comic`. When titles are otherwise alike, BookOrbit prefers the record that
matches the medium of the book being matched, so a novel's EPUB gets the novel.
**Data**
- Plain-text descriptions: strip HTML, Markdown and site markup, and keep paragraphs.
- Write people's names the way the book does. Give years as four digits, ISBNs without hyphens,
ratings on a 0-5 scale.
- Return the largest cover your source has, with `coverWidth` and `coverHeight` when you know them.
- Fill only what you know. A missing field is better than a guessed one.
**Failures and limits**
- Answer `429` with `Retry-After` when your source throttles you. BookOrbit backs off.
- Answer `502` or `503` when your source fails. BookOrbit may then answer from its cache.
- Answer `"partial": true` when some matches could not be fetched, so the answer is not cached as
complete.
- Pace your calls to the source inside the provider, whatever BookOrbit asks for: several BookOrbit
servers, or other apps, may share one provider. Declare a `rateLimit` too, so BookOrbit does not
queue more than you can serve.
- Identify yourself to the source with a `User-Agent` such as `my-provider/1.2.0`.
**Safety and privacy**
- Keep it stateless and read-only. It needs no database and no volume.
- Run as a non-root user in a minimal image (`scratch` or distroless). Both starters' Dockerfiles
already do both.
- Support an `AUTH_TOKEN` (both starters do) for anyone who exposes it beyond their own network.
- Log failures, not the titles people look up.
- Say in your README which details you send to your source. BookOrbit sends you the title, author
and ISBN of each book it asks about.
## Test it
- **Unit tests with real answers.** Save a few real responses from your source into `testdata/`,
trimmed to the fields you read, and test your mapping against them. Tests then run offline and
keep passing when the source is down. Both published providers do this.
- **The conformance checker** before every release: it checks the endpoints, types, limits, status
codes, icon and colour the way BookOrbit reads them.
- **In BookOrbit:** **Test** in the provider's settings runs a real search and shows what came back.
## Package it
Ship it as a container image. Each starter's `Dockerfile` builds a static binary into an empty
`scratch` image that runs as a non-root user: about 1 MB for the Rust starter, 5 MB for the Go one.
```bash
docker build -t my-provider .
```
```bash
docker run --rm -p 8080:8080 my-provider
```
Build for both `linux/amd64` and `linux/arm64`: many people run BookOrbit on a Raspberry Pi or another
ARM machine.
```bash
docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/your-name/my-provider:0.1.0 --push .
```
## Publish it
GitHub Container Registry (GHCR) is free for public images and sits next to your code.
1. **Push your code to GitHub.**
2. **Add a release workflow** as `.github/workflows/publish.yml`. Pushing a tag such as `v0.1.0` then
builds both architectures and publishes the image:
```yaml
name: Publish
on:
push:
tags: ["v*"]
permissions:
contents: read
packages: write
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/metadata-action@v5
id: meta
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
- uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
```
3. **Release:** bump `version` in `/provider/info`, then tag and push.
```bash
git tag v0.1.0 && git push origin v0.1.0
```
4. **Make the image public.** GitHub creates new packages as private. Open the package from your
profile's **Packages** tab, then **Package settings > Change visibility > Public**.
5. **Write the README people need:** what the provider covers, a Compose snippet with a pinned
version, the address to enter in BookOrbit, its environment variables, what it sends to its
source, and any known gaps.
6. **Tell people about it** in a
[Community Discussion](https://github.com/bookorbit/bookorbit/issues/new?template=discussion.yml).
To have it offered under **Available to install** in every BookOrbit, open a pull request that
adds it to `directory.json` in
[bookorbit-metadata-providers](https://github.com/bookorbit/bookorbit-metadata-providers).
### Versions
Use semantic versions and tag each image with its full version (`0.1.0`), its minor line (`0.1`)
and, if you like, `latest`. Tell users to pin a version, so an update never surprises them.
Once people use your provider:
- Keep its key forever.
- Keep answering for ids you issued before. BookOrbit stores them on people's books and looks them
up again on every refresh; an id that suddenly returns 404 tells BookOrbit the book is gone.
- Bump `version` with every change to your answers.
## Reference
The contract builds on the
[book-metadata-provider](https://github.com/FlyinPancake/book-metadata-provider) specification. Every
field below except `name`, `providerId` and `title` is optional.
### `GET /provider/info`
```json
{
"name": "MangaBaka",
"key": "mangabaka",
"version": "0.1.0",
"avatarUrl": "https://mangabaka.org/images/maomao/180x180.png",
"color": "#e30047",
"mediaTypes": ["book", "comic"],
"lookup": true,
"searchParams": ["limit"],
"supportedFields": [
"title",
"description",
"cover",
"authors",
"seriesName",
"seriesIndex"
],
"bookUrlTemplate": "https://mangabaka.org/manga/{id}",
"idMaxLength": 12,
"rateLimit": { "maxConcurrency": 1, "minIntervalMs": 2000 },
"coverShape": "portrait"
}
```
| Field | What it does |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Display name, up to 100 characters. Required |
| `key` | Permanent identity: 2-32 lowercase letters, digits or hyphens, starting with a letter. Built-in names such as `goodreads` are taken. Defaults to a slug of `name` |
| `version` | Your release. Part of BookOrbit's cache key |
| `avatarUrl` | Your icon (see [Icon and colour](#icon-and-colour)) |
| `color` | `#rrggbb` brand colour |
| `mediaTypes` | Any of `book`, `audiobook`, `comic`. Defaults to `book` and `audiobook` |
| `lookup` | `true` when you serve `GET /book/{id}` |
| `searchParams` | Optional search parameters you want: `isbn`, `limit` |
| `supportedFields` | Field rules you can fill: `title`, `subtitle`, `description`, `cover`, `audioCover`, `authors`, `publisher`, `isbn`, `publishedYear`, `language`, `pageCount`, `communityRating`, `seriesName`, `seriesIndex`, `genres`, `narrators`, `duration`, `abridged`. Defaults to all |
| `bookUrlTemplate` | A link to a book's page, with `{id}` in it, shown on the book |
| `idMaxLength` | Longest id you issue, at most 512 |
| `rateLimit` | `maxConcurrency` (1-4, default 2) and `minIntervalMs` (0-60000, default 0): how hard BookOrbit may call you |
| `coverShape` | `portrait` or `square`, when all your covers are one shape |
| `authenticationRequired` | `true` when your provider needs an `Authorization` header |
### `GET /search`
BookOrbit sends `query` (the title), `author` and `mediaType` (`book` or `audiobook`), plus `isbn` and
`limit` if you listed them in `searchParams`. It only searches when it has a title or an ISBN.
Answer with your matches, best first:
```json
{
"matches": [
{ "providerId": "1692", "title": "BERSERK", "authors": ["MIURA Kentaro"] }
]
}
```
### `GET /book/{id}`
The `id` is one of your `providerId` values, URL-encoded. Answer with one book in the same shape as
a match, or `404` when it does not exist.
### A book
| Field | Notes |
| ------------------------------------ | ------------------------------------------------------------------ |
| `providerId` | Your id for it, a string. Required; a match without one is dropped |
| `title` | Required |
| `mediaType` | `book`, `comic` or `audiobook`: what this record is |
| `subtitle` | |
| `authors`, `narrators` | Lists of names |
| `description` | Plain text, up to 50,000 characters |
| `publisher` | |
| `publishedYear` | `"1968"` |
| `publishedDate` | `"1968-11-01"` |
| `languageCode` | `"en"` |
| `isbn13`, `isbn10` | Or a single `isbn` of either length |
| `pages` | Page count |
| `duration` | Audiobook length in seconds |
| `genres` | List of names |
| `series` | `[{ "series": "Earthsea Cycle", "sequence": "1" }]` |
| `cover`, `coverWidth`, `coverHeight` | A public `http(s)` image URL, and its size in pixels |
| `rating`, `ratingCount` | Community rating on a 0-5 scale, and how many people rated |
| `url` | The book's page on your source |
### Status codes
| You answer | BookOrbit |
| ------------------------------------- | --------------------------------------------------------- |
| `200` | Uses the answer. An empty `matches` list means none found |
| `404` from `/book/{id}` | Treats the id as gone |
| `429` with `Retry-After` | Waits that long before asking again |
| `401`, `403` | Reports the authorization header as refused |
| Anything else, a timeout, or bad JSON | Counts a failure, and may answer from its cache |
BookOrbit waits as long as the admin set for the provider: 10 seconds by default, between 2 and 30.
It reads at most 2 MB of an answer and the first 50 matches, and follows at most 3 redirects.
### Icon and colour
BookOrbit downloads the image at `avatarUrl` when the provider is added, tested, or rechecked every
six hours, and serves its own copy, so readers' browsers never contact you. It keeps PNG, WebP, JPEG,
GIF and ICO images up to 256 KB, judged by the file's contents; SVG is never kept. A square image of
64 pixels or more looks best. `color` must be six hex digits, such as `#2c98cd`, or BookOrbit ignores
it. Without an icon BookOrbit shows your initials; without a colour, a neutral one.
## FAQ
**Do I need to change BookOrbit?** No. An admin adds your provider by its address.
**Can one container serve several providers?** Yes. Serve each under its own path, such as
`http://pack:8080/goodreads/` and `http://pack:8080/audible/`, each with its own key, and add each
address separately.
**Can I see what BookOrbit asked?** Log the request path and status, not the query: the query is
someone's reading.
docs/METADATA_PROVIDERS.md
+145
View File
@@ -0,0 +1,145 @@
# External Metadata Providers
BookOrbit comes with its own metadata sources: Google Books, Goodreads, Hardcover and others. When
your library needs something they do not cover well, such as manga, Chinese books or Russian-language
science fiction, you can add an **external provider**: a small service that runs beside BookOrbit and
answers its metadata searches.
Once added, an external provider works like a built-in source. It shows up in manual search, Book
Dock, auto-fetch and library refresh, it can fill field rules, and it gets its own icon and colour
wherever it appears. Adding one needs no BookOrbit update.
## Ready-made providers
| Provider | Good for | Image |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- |
| Douban | Chinese books from [book.douban.com](https://book.douban.com): titles, authors, publishers, ratings, ISBN lookups. No covers | `ghcr.io/bookorbit/douban-provider` |
| FantLab | Russian-language science fiction and fantasy from [fantlab.ru](https://fantlab.ru): titles, authors, series, genres, ratings, ISBN lookups | `ghcr.io/bookorbit/fantlab-provider` |
| J-Novel Club | Light novels and manga published in English by [J-Novel Club](https://j-novel.club). Reads the volume number from the book's title | `ghcr.io/bookorbit/jnovelclub-provider` |
| Libris | Swedish books from [Libris](https://libris.kb.se), the National Library of Sweden's catalogue | `ghcr.io/bookorbit/libris-provider` |
| LubimyCzytac | Polish books from [lubimyczytac.pl](https://lubimyczytac.pl): titles, authors, series, genres | `ghcr.io/bookorbit/lubimyczytac-provider` |
| MangaBaka | Manga, manhwa, manhua and light novels from [mangabaka.org](https://mangabaka.org). Reads the volume number from the book's title | `ghcr.io/bookorbit/mangabaka-provider` |
Their source, and what each one returns, is in
[bookorbit-metadata-providers](https://github.com/bookorbit/bookorbit-metadata-providers). To make
your own, see [Building a Metadata Provider](BUILDING_METADATA_PROVIDERS.md).
### LubimyCzytac and Aladin used to be built in
Neither is built into BookOrbit any more. Nothing on your books is lost: their ids and links stay
where they were. Field rules that name either one skip it, and **Settings > Metadata > Providers**
says what to do.
- **LubimyCzytac** now runs as an external provider. Run its image and add it as below; field rules
that name it pick up where they left off.
```yaml
services:
lubimyczytac:
image: ghcr.io/bookorbit/lubimyczytac-provider:0.1.0
restart: unless-stopped
```
- **Aladin** is retired. Aladin stopped issuing Open API keys on 4 September 2026 and ends the API on
30 October 2026, with no replacement, so there is no provider to run. Take it out of your field
rules. The Aladin key BookOrbit stored was deleted when it updated.
## Set one up
**Settings > Metadata > Providers** has two tabs: **Built-in** for the sources BookOrbit ships with,
and **External** for the ones you run. Under **Available to install**, the External tab lists the
ready-made providers with a **Copy Compose** button and an **Add** button that fills in the address
for you, so steps 1 to 3 below are mostly done for you.
1. **Run the provider next to BookOrbit.** Add it to the same Docker Compose file, so the two share
a network. It needs no ports published and no volumes:
```yaml
services:
mangabaka:
image: ghcr.io/bookorbit/mangabaka-provider:0.1.0
restart: unless-stopped
```
Start it with `docker compose up -d`.
2. **Add it in BookOrbit.** Open **Settings > Metadata > Providers > External** and choose **Add** on
its card, or **Add provider** for one that is not listed. The address is the service name and
port: `http://mangabaka:8080`.
3. **Turn on Allow private address.** A container beside BookOrbit is on a private network, and
BookOrbit refuses private addresses unless you allow it for that provider. Leave it off for a
provider hosted on the internet.
4. **Save, then choose Test.** A good test shows how many results a sample search found and how
long it took. The provider appears in the list with its own icon.
5. **Turn it on.** It now answers manual searches: open a book, **Edit Metadata > Search**, and its
results show beside the built-in sources.
6. **Let auto-fetch use it.** A new provider is in no field rule yet. Choose **Use in field rules** to
add it to the end of every rule it can fill, or add it to individual rules yourself in
**Settings > Metadata > Field Rules**. Move it up in the rules where it should win: for a manga
library, MangaBaka belongs ahead of the general sources for title, series and cover.
### Tips
- **Pin a version** such as `0.1.0` rather than `latest`, and update on purpose. After updating,
choose **Test** to pick up the new version straight away; otherwise BookOrbit notices within six
hours.
- **Keep it private.** A provider only has to be reachable from BookOrbit. Do not publish its port to
the internet unless you also set its `AUTH_TOKEN` (see its README) and enter the same value in
BookOrbit's **Authorization header** field.
- **Authorization headers are stored encrypted** with `BOOK_REQUEST_ENCRYPTION_KEY`, the same key
book requests use (see `server/.env.example`). If that key changes, the provider shows **Header
unreadable** and is skipped until you enter the header again.
- **More than one at once is fine.** Each provider has its own key, ids and ratings, and BookOrbit
asks each at the pace it asks for.
## What a provider can see
BookOrbit sends a provider the title, author and ISBN of each book it asks about, and the provider's
own id for a book it has already matched. It sends nothing else about your library or your users.
A provider passes those details on to the service it gets its data from, so a provider for a public
site means that site sees which books you look up.
The list under **Available to install** is read from the
[bookorbit-metadata-providers](https://github.com/bookorbit/bookorbit-metadata-providers) repository
on GitHub, at most once a day and only when an admin opens the External tab. To keep BookOrbit from
contacting GitHub for it, set `METADATA_PROVIDER_DIRECTORY_URL=off`; the list that came with your
version is shown instead.
## Status
| Status | What it means |
| ----------------- | --------------------------------------------------------------------------------- |
| Ready | Working, but turned off |
| Active | Working and turned on |
| Last check failed | The six-hourly check failed; it keeps answering with what it last reported |
| Unreachable | Three checks in a row failed; it is skipped until one succeeds. Choose **Test** |
| Key changed | The provider now reports a different identity. Remove it and add it again |
| Incompatible | Something answered at that address, but not a metadata provider BookOrbit can use |
| Header unreadable | The saved authorization header can no longer be decrypted. Enter it again |
## Removing a provider
**Remove provider** keeps the ids and ratings it stored on your books and its place in field rules,
so adding it again later picks up where it left off. **Remove and delete its data** also deletes those
ids, ratings and its cached answers.
## Troubleshooting
- **"That address is on a private network"**: turn on **Allow private address** for it.
- **"Could not reach a provider at that address"**: the address is what BookOrbit, not your browser,
can reach. Inside Docker that is the service name (`http://mangabaka:8080`); `localhost` would be
the BookOrbit container itself. Check the provider is running with `docker compose ps`.
- **"Answered, but not with usable provider info"**: the address is wrong or points at something
else. Open `<address>/provider/info` from the BookOrbit host; a provider answers with JSON that has
a `name`. A missing or extra path in the address is the usual cause.
- **"A provider with the same key is already registered"**: that provider is already added. Each
provider can be added once.
- **Two providers show the same name**: BookOrbit adds the host, such as "FantLab (fantlab:8080)", so
they can be told apart.
- **No results for a book you know is there**: the provider searches by title and author as they are
in BookOrbit. For MangaBaka, a title like "Berserk Vol. 3" works; for FantLab, Russian titles find
more than English ones.