Migrations/bookorbit
1
0
Fork 0
mirror of https://github.com/bookorbit/bookorbit.git synced 2026-11-03 08:21:39 -06:00
Code Issues Packages Projects Releases Wiki Activity
Files
2bfd4d1ff45f4b66b67b1c7c7b6cba0c25b82ed8
bookorbit/docs/BUILDING_METADATA_PROVIDERS.md
T
neon b50b8da56c feat(metadata): add external HTTP metadata providers 
Register book-metadata-provider services by address, with no code change per
provider; Settings lists ready-made ones to install. Ranking prefers a record of
the book's own medium. LubimyCzytac moves out to its own provider and Aladin,
whose API closes, is retired; their ids stay. Adds migrations 0129 and 0130.
See docs/METADATA_PROVIDERS.md.

Ref #1405
2026-10-23 06:11:04 +00:00

373 lines
20 KiB
Markdown
Raw Blame History

# Building a Metadata Provider
A metadata provider is a small web service that answers BookOrbit's metadata searches from a source
it does not know: a regional catalogue, a fan wiki, a manga database, your own spreadsheet. BookOrbit
calls it over HTTP and never runs its code, so you write it in any language, release it on your own
schedule, and nobody has to change BookOrbit for it.
This guide takes you from an empty folder to a published image that anyone can add to BookOrbit.
To add a provider someone else made, see [External Metadata Providers](METADATA_PROVIDERS.md).
## How it works
```text
BookOrbit your provider the source
| GET /search?query=Dune -------> | |
| | -- its own API or pages ----> |
| | <---------------------------- |
| <---- { "matches": [ ... ] } --- | |
```
Your provider answers three requests:
- `GET /provider/info`: who it is (name, a permanent key, version, icon, what it can do).
- `GET /search`: books matching a title, author or ISBN, best first.
- `GET /book/{id}`: one book by the id your provider gave it earlier. Optional, but recommended.
BookOrbit caches answers, merges them with its other sources through field rules, stores your ids
on the books it matches, and shows your icon and colour wherever your results appear.
## Before you start
- **Check the source's terms.** You are responsible for how your provider uses its source. Prefer an
official API over scraping pages: APIs change less often, and many sites forbid scraping.
- **Choose the key now, and never change it.** The key (such as `mangabaka`) is how BookOrbit files
every id and rating your provider gives it. A provider that changes its key is a new provider to
BookOrbit, and users have to add it again.
- **Choose a language you will keep maintaining.** There are starters in Rust and Go. Rust is what
the published providers use: the same provider ran in about a third of Go's memory. Go needs
nothing but `go run`. Python, Node or anything else that serves JSON works too.
## Quick start
The [bookorbit-metadata-providers](https://github.com/bookorbit/bookorbit-metadata-providers)
repository has a working starter in Rust and one in Go, and a conformance checker. Both starters
serve one made-up book and behave the same; pick the language you prefer.
1. **Copy a starter.**
```bash
git clone https://github.com/bookorbit/bookorbit-metadata-providers.git
```
For Rust:
```bash
cp -R bookorbit-metadata-providers/starters/rust my-provider
```
Or for Go:
```bash
cp -R bookorbit-metadata-providers/starters/go my-provider
```
2. **Run it**, so you can see every endpoint working before you change anything. In Rust,
`cargo run`; in Go, `go run .`. Then:
```bash
curl "http://localhost:8080/search?query=earthsea"
```
3. **Name it.** Change `info()` in `src/main.rs` (Rust) or `main.go` (Go): give it your `name` and
the `key` you chose.
4. **Connect your source.** In `src/source.rs` or `source.go`, replace the example with calls to
your source:
- Search returns matches best first, at most the query's limit.
- Lookup returns the book for an id, or nothing (`None`, `nil`) when the id does not exist.
- Return an error for anything else that goes wrong. Never hide a failure behind an empty list:
to BookOrbit, an empty list means "this book is not there", and it remembers that.
5. **Check it against the contract.** With your provider running:
```bash
cd bookorbit-metadata-providers/conformance && go run . -url http://localhost:8080 -query "Dune"
```
It prints a line per check and ends with the number of failures and warnings. Fix every failure;
each warning says what BookOrbit will do instead.
6. **Try it in BookOrbit.** Add it under **Settings > Metadata > Providers > External providers**
with **Allow private address** on, choose **Test**, turn it on and search for a book.
## Recommendations
**Identity**
- Serve `/provider/info` with a `name`, a `key` and a `version`.
- Bump `version` whenever your answers change shape: new fields, different mapping, a fixed bug.
BookOrbit keeps cached answers per version, so an unchanged version keeps serving old ones.
- Give it an icon (`avatarUrl`) and a `color`, so your results are recognisable at a glance.
**Answers**
- Put the best match first, and return a handful rather than everything: BookOrbit uses at most 50,
and a person picks from the first few.
- Leave out what is not a book on a shelf: reviews, articles, interviews, collections pages.
- Support `GET /book/{id}` and declare `lookup: true` when your source has stable ids. Refreshing a
book by id is faster and more accurate than searching for it again.
- Declare what you can fill with `supportedFields` and `mediaTypes`, so BookOrbit only asks you for
what you have.
- Give each record its own `mediaType` when your source has more than one: a light novel is
`book`, its manga `comic`. When titles are otherwise alike, BookOrbit prefers the record that
matches the medium of the book being matched, so a novel's EPUB gets the novel.
**Data**
- Plain-text descriptions: strip HTML, Markdown and site markup, and keep paragraphs.
- Write people's names the way the book does. Give years as four digits, ISBNs without hyphens,
ratings on a 0-5 scale.
- Return the largest cover your source has, with `coverWidth` and `coverHeight` when you know them.
- Fill only what you know. A missing field is better than a guessed one.
**Failures and limits**
- Answer `429` with `Retry-After` when your source throttles you. BookOrbit backs off.
- Answer `502` or `503` when your source fails. BookOrbit may then answer from its cache.
- Answer `"partial": true` when some matches could not be fetched, so the answer is not cached as
complete.
- Pace your calls to the source inside the provider, whatever BookOrbit asks for: several BookOrbit
servers, or other apps, may share one provider. Declare a `rateLimit` too, so BookOrbit does not
queue more than you can serve.
- Identify yourself to the source with a `User-Agent` such as `my-provider/1.2.0`.
**Safety and privacy**
- Keep it stateless and read-only. It needs no database and no volume.
- Run as a non-root user in a minimal image (`scratch` or distroless). Both starters' Dockerfiles
already do both.
- Support an `AUTH_TOKEN` (both starters do) for anyone who exposes it beyond their own network.
- Log failures, not the titles people look up.
- Say in your README which details you send to your source. BookOrbit sends you the title, author
and ISBN of each book it asks about.
## Test it
- **Unit tests with real answers.** Save a few real responses from your source into `testdata/`,
trimmed to the fields you read, and test your mapping against them. Tests then run offline and
keep passing when the source is down. Both published providers do this.
- **The conformance checker** before every release: it checks the endpoints, types, limits, status
codes, icon and colour the way BookOrbit reads them.
- **In BookOrbit:** **Test** in the provider's settings runs a real search and shows what came back.
## Package it
Ship it as a container image. Each starter's `Dockerfile` builds a static binary into an empty
`scratch` image that runs as a non-root user: about 1 MB for the Rust starter, 5 MB for the Go one.
```bash
docker build -t my-provider .
```
```bash
docker run --rm -p 8080:8080 my-provider
```
Build for both `linux/amd64` and `linux/arm64`: many people run BookOrbit on a Raspberry Pi or another
ARM machine.
```bash
docker buildx build --platform linux/amd64,linux/arm64 -t ghcr.io/your-name/my-provider:0.1.0 --push .
```
## Publish it
GitHub Container Registry (GHCR) is free for public images and sits next to your code.
1. **Push your code to GitHub.**
2. **Add a release workflow** as `.github/workflows/publish.yml`. Pushing a tag such as `v0.1.0` then
builds both architectures and publishes the image:
```yaml
name: Publish
on:
push:
tags: ["v*"]
permissions:
contents: read
packages: write
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/metadata-action@v5
id: meta
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
- uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
```
3. **Release:** bump `version` in `/provider/info`, then tag and push.
```bash
git tag v0.1.0 && git push origin v0.1.0
```
4. **Make the image public.** GitHub creates new packages as private. Open the package from your
profile's **Packages** tab, then **Package settings > Change visibility > Public**.
5. **Write the README people need:** what the provider covers, a Compose snippet with a pinned
version, the address to enter in BookOrbit, its environment variables, what it sends to its
source, and any known gaps.
6. **Tell people about it** in a
[Community Discussion](https://github.com/bookorbit/bookorbit/issues/new?template=discussion.yml).
To have it offered under **Available to install** in every BookOrbit, open a pull request that
adds it to `directory.json` in
[bookorbit-metadata-providers](https://github.com/bookorbit/bookorbit-metadata-providers).
### Versions
Use semantic versions and tag each image with its full version (`0.1.0`), its minor line (`0.1`)
and, if you like, `latest`. Tell users to pin a version, so an update never surprises them.
Once people use your provider:
- Keep its key forever.
- Keep answering for ids you issued before. BookOrbit stores them on people's books and looks them
up again on every refresh; an id that suddenly returns 404 tells BookOrbit the book is gone.
- Bump `version` with every change to your answers.
## Reference
The contract builds on the
[book-metadata-provider](https://github.com/FlyinPancake/book-metadata-provider) specification. Every
field below except `name`, `providerId` and `title` is optional.
### `GET /provider/info`
```json
{
"name": "MangaBaka",
"key": "mangabaka",
"version": "0.1.0",
"avatarUrl": "https://mangabaka.org/images/maomao/180x180.png",
"color": "#e30047",
"mediaTypes": ["book", "comic"],
"lookup": true,
"searchParams": ["limit"],
"supportedFields": [
"title",
"description",
"cover",
"authors",
"seriesName",
"seriesIndex"
],
"bookUrlTemplate": "https://mangabaka.org/manga/{id}",
"idMaxLength": 12,
"rateLimit": { "maxConcurrency": 1, "minIntervalMs": 2000 },
"coverShape": "portrait"
}
```
| Field | What it does |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Display name, up to 100 characters. Required |
| `key` | Permanent identity: 2-32 lowercase letters, digits or hyphens, starting with a letter. Built-in names such as `goodreads` are taken. Defaults to a slug of `name` |
| `version` | Your release. Part of BookOrbit's cache key |
| `avatarUrl` | Your icon (see [Icon and colour](#icon-and-colour)) |
| `color` | `#rrggbb` brand colour |
| `mediaTypes` | Any of `book`, `audiobook`, `comic`. Defaults to `book` and `audiobook` |
| `lookup` | `true` when you serve `GET /book/{id}` |
| `searchParams` | Optional search parameters you want: `isbn`, `limit` |
| `supportedFields` | Field rules you can fill: `title`, `subtitle`, `description`, `cover`, `audioCover`, `authors`, `publisher`, `isbn`, `publishedYear`, `language`, `pageCount`, `communityRating`, `seriesName`, `seriesIndex`, `genres`, `narrators`, `duration`, `abridged`. Defaults to all |
| `bookUrlTemplate` | A link to a book's page, with `{id}` in it, shown on the book |
| `idMaxLength` | Longest id you issue, at most 512 |
| `rateLimit` | `maxConcurrency` (1-4, default 2) and `minIntervalMs` (0-60000, default 0): how hard BookOrbit may call you |
| `coverShape` | `portrait` or `square`, when all your covers are one shape |
| `authenticationRequired` | `true` when your provider needs an `Authorization` header |
### `GET /search`
BookOrbit sends `query` (the title), `author` and `mediaType` (`book` or `audiobook`), plus `isbn` and
`limit` if you listed them in `searchParams`. It only searches when it has a title or an ISBN.
Answer with your matches, best first:
```json
{
"matches": [
{ "providerId": "1692", "title": "BERSERK", "authors": ["MIURA Kentaro"] }
]
}
```
### `GET /book/{id}`
The `id` is one of your `providerId` values, URL-encoded. Answer with one book in the same shape as
a match, or `404` when it does not exist.
### A book
| Field | Notes |
| ------------------------------------ | ------------------------------------------------------------------ |
| `providerId` | Your id for it, a string. Required; a match without one is dropped |
| `title` | Required |
| `mediaType` | `book`, `comic` or `audiobook`: what this record is |
| `subtitle` | |
| `authors`, `narrators` | Lists of names |
| `description` | Plain text, up to 50,000 characters |
| `publisher` | |
| `publishedYear` | `"1968"` |
| `publishedDate` | `"1968-11-01"` |
| `languageCode` | `"en"` |
| `isbn13`, `isbn10` | Or a single `isbn` of either length |
| `pages` | Page count |
| `duration` | Audiobook length in seconds |
| `genres` | List of names |
| `series` | `[{ "series": "Earthsea Cycle", "sequence": "1" }]` |
| `cover`, `coverWidth`, `coverHeight` | A public `http(s)` image URL, and its size in pixels |
| `rating`, `ratingCount` | Community rating on a 0-5 scale, and how many people rated |
| `url` | The book's page on your source |
### Status codes
| You answer | BookOrbit |
| ------------------------------------- | --------------------------------------------------------- |
| `200` | Uses the answer. An empty `matches` list means none found |
| `404` from `/book/{id}` | Treats the id as gone |
| `429` with `Retry-After` | Waits that long before asking again |
| `401`, `403` | Reports the authorization header as refused |
| Anything else, a timeout, or bad JSON | Counts a failure, and may answer from its cache |
BookOrbit waits as long as the admin set for the provider: 10 seconds by default, between 2 and 30.
It reads at most 2 MB of an answer and the first 50 matches, and follows at most 3 redirects.
### Icon and colour
BookOrbit downloads the image at `avatarUrl` when the provider is added, tested, or rechecked every
six hours, and serves its own copy, so readers' browsers never contact you. It keeps PNG, WebP, JPEG,
GIF and ICO images up to 256 KB, judged by the file's contents; SVG is never kept. A square image of
64 pixels or more looks best. `color` must be six hex digits, such as `#2c98cd`, or BookOrbit ignores
it. Without an icon BookOrbit shows your initials; without a colour, a neutral one.
## FAQ
**Do I need to change BookOrbit?** No. An admin adds your provider by its address.
**Can one container serve several providers?** Yes. Serve each under its own path, such as
`http://pack:8080/goodreads/` and `http://pack:8080/audible/`, each with its own key, and add each
address separately.
**Can I see what BookOrbit asked?** Log the request path and status, not the query: the query is
someone's reading.
Reference in New Issue View Git Blame Copy Permalink