Skip to content

docs: refresh README and testing guidance #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ryanbarlow97 merged 2 commits into from
Oct 8, 2026
Merged

docs: refresh README and testing guidance #136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
10573f3
docs: refresh README and testing guidance
ryanbarlow97 Oct 8, 2026
c6f69d8
docs: propagate failures in test command example
ryanbarlow97 Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ It connects the Minecraft world with a browser experience: players can explore t
- **Character creation and profiles** — create characters and manage their profiles, wardrobes, and kits through the website.
- **Custom skin submissions** — submit supported cosmetic skins through an in-game-code-linked workflow.
- **Drink design** — create BreweryX recipes and submit custom drinks for staff review.
- **Accounts and staff tools** — sign in with Discord, manage linked accounts and supporter status, and use role-controlled player, activity, rail, and permission panels.
- **Player knowledge hub** — browse the wiki's guides to crafting, professions, equipment, economy, and other server features.

## Connected to the server
Expand All @@ -25,22 +26,22 @@ It connects the Minecraft world with a browser experience: players can explore t

Technical documentation is maintained in [TF-Minecraft/Docs](https://github.com/TF-Minecraft/Docs).

Each branch gets a preview at `https://<branch>.tfminecraft.net`, merging to `main`
deploys `https://dev.tfminecraft.net`, and **Actions → Deploy** puts `main` live on
`https://www.tfminecraft.net`. See [Previews and deploys](https://github.com/TF-Minecraft/Docs/blob/main/projects/ProvinceSystem/DEPLOY.md).
[Previews and deployment](https://github.com/TF-Minecraft/Docs/blob/main/projects/ProvinceSystem/DEPLOY.md)

## Tests

CI runs the frontend Vitest suite and the backend pytest suite, then builds the
Next.js frontend. To run the same suites from the repository root:
CI uses Node 22 and Python 3.12. From the repository root, install dependencies
and run the frontend Vitest and backend pytest suites with the same JUnit output:

```sh
(cd frontend && npm ci && npm test)
(cd backend && python -m pip install -r requirements.txt -r requirements-dev.txt && python -m pytest -q)
(cd frontend && npm ci && npm test -- --reporter=default --reporter=junit --outputFile.junit=../test-results/frontend.xml) &&
(cd backend && python -m pip install -r requirements.txt -r requirements-dev.txt && python -m pytest -q --junitxml=../test-results/backend.xml)
```

The suites cover the website and API in isolation; they do not run a Minecraft
server or the companion plugins.
CI uploads `test-results/frontend.xml` and `test-results/backend.xml`, then builds
the Next.js frontend. No coverage threshold is enforced. The suites exercise
website components, application logic, and APIs in isolation; they do not run a
live Minecraft server, companion plugins, or real external-service flows.

## License

Expand Down
153 changes: 3 additions & 150 deletions backend/src/coreprotect/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,154 +1,7 @@
# CoreProtect reader

The staff panel's player pages read the Minecraft server's live CoreProtect
SQLite database: who has played, their sessions, and their recent actions.
`src/auth/players.py` joins that with `discord_links`, `users` and
`character_roster`; the routes are `GET /admin/players`,
`/admin/players/{uuid}`, `/sessions` and `/activity` (`view_players`, mod and
above). Movement, `GET /admin/players/{uuid}/movement` and `/admin/movement`
for everyone, is for admins and the owner (`view_player_movement`).
Read-only CoreProtect activity, session, and movement data for the staff panel.

## Settings
[Configuration and technical reference](https://github.com/TF-Minecraft/Docs/blob/main/projects/ProvinceSystem/docs/integrations/coreprotect.md)

| Variable | Meaning |
| --- | --- |
| `COREPROTECT_DB` | Path to `database.db` inside the backend container. Unset: the directory still lists linked players and characters, and CoreProtect sections say they are unavailable. |
| `COREPROTECT_SERVER` | Short id that scopes caches and page cursors (default `main`). |
| `COREPROTECT_SERVER_LABEL` | World name shown with the data, for example `Vardera`. |
| `COREPROTECT_PING_SECONDS` | The server's `player-pings` interval (default 60; `0` if pings are off). Used to guess whether someone is online, to end crashed sessions and to tell a gap in a movement path. |
| `COREPROTECT_MAP_WORLD` | The CoreProtect world the site's map shows: the server's `level-name` (default `TFMC_Map`). Movement in other worlds is summarised, not drawn. |

Mount the CoreProtect **directory** read-only, so SQLite can see a hot
`-journal`. Set `disable-wal: true` in CoreProtect's `config.yml` (then
`/co reload`): our fork uses WAL by default, and through a read-only mount
SQLite can open a WAL database only while its `-wal` and `-shm` files already
exist. CoreProtect closes its connections between batches, which removes
them, so in WAL mode the site mostly reports `cannot_open`. The option is
appended to `config.yml` once and applied on every start or reload; CoreProtect
only ever appends missing options to that file, so updates keep it. The paths are host-specific, so they belong in the host's
`docker-compose.override.yml`, not the repository's compose files:

```yaml
services:
backend:
volumes:
- /home/amp/.ampdata/instances/TFMCMain01/Minecraft/plugins/CoreProtect:/coreprotect:ro
environment:
- COREPROTECT_DB=/coreprotect/database.db
- COREPROTECT_SERVER=main
- COREPROTECT_SERVER_LABEL=Vardera
```

Docker resolves the bind as root, so the AMP tree needs no permission
changes. The mount also exposes CoreProtect's `config.yml`, which holds
connection settings for database engines the server does not use.

## Keeping CoreProtect's writer unblocked

With `disable-wal: true` the database uses a rollback journal, so a
reader's lock holds off CoreProtect's commits. `reader.py` explains the guards: autocommit, one
reader per process, a 1.5 s budget per request (waiting for the lock
included), and every statement fully fetched before the next. The backend
runs as one uvicorn process; running more would need the reader limit
shared between them. Never open the file with `immutable=1`: on Main that
read torn pages while CoreProtect wrote.

Candidate scans are indexed seeks by player; final-page metadata reads use
row IDs. The only whole-table reads are the small tables holding
CoreProtect names, including `co_user` and `co_username_log` (a few hundred
to a few thousand rows). Activity pages examine at most `SCAN_LIMIT` rows per
table, so a sparse filter stops early and returns a cursor to continue
(`searched_to`).

## What is shown

For moderators, chat is never read and commands are cut to their first word
inside SQL. Admins and the owner (`view_player_messages`) also get chat and
whole commands, each cut to 512 characters; every page that shows any is
recorded in `admin_audit` as `player.messages.view` (which rows, never the
text) before it is returned, and the page is refused if that record cannot
be written. Sign text is never selected. After selecting the final activity page,
point reads
fetch at most 64 KiB per item, entity, or identity blob for displayed rows. After closing the
reader, a bounded, read-only Java-stream/NBT decoder extracts only custom names
and MMOItems type/ID; raw metadata, lore, and other tags never enter API responses.
Decompression is capped at 256 KiB, with depth and node limits. Malformed,
oversized, or unsupported metadata retains the vanilla label.

Item and container labels include the recorded MMOItems identity beside the
saved name (including alloys). Named vanilla items are explicitly marked renamed.
Historical mob kills show their saved name as a name, not a proven mob type. New
kills from the custom-identity CoreProtect build additionally show the MythicMobs
ID from the `coreprotect:mythic` block-metadata marker. No history rewrite is
needed. Custom-ID search and ItemsAdder block identity are outside this change.

Each activity entry also carries `target_info`: the name the panel shows
(`Short Grass`, not `short_grass`), where it came from (vanilla, MMOItems,
MythicMobs, a player) and the ids behind it. Vanilla names come from
Minecraft's own `en_us.json`, which is Mojang's to publish, not ours: the
backend image runs `scripts/build_minecraft_names.py` to fetch the pinned
client jar, check its SHA-1 and keep the block, item and entity names
(about 94 KB) in `data/minecraft_names.json`, which git ignores. Bump the
pinned version there when the server updates. Without the file, ids are
tidied instead. Block changes use block names (`wheat` is "Wheat Crops"),
items use item names ("Wheat"); only `minecraft:` ids are looked up.
Activity responses also return `server_label` and `map_world`, so the panel
calls the map's world by its name (Vardera). The panel merges consecutive
changes to the same block within a minute into one row; the API does not.

RPCharacters chat sent by command (`/looc hi`, `/fooc`, `/me`, `/shout`…;
the aliases are `CHANNELS` in `activity.py`, from Main's `chat.yml`) is chat:
for admins it comes back as kind `chat` with its `channel` and the text after
the command, under the Chat filter and audited like chat; moderators still get
only the command word, with the `channel` named. Plain chat goes to the
player's current channel, which CoreProtect does not record, so it is worked
out (`channel_inferred`): RPCharacters keeps the channel picked with
`/channel <name>` in memory until the player quits, so a line's channel is
the last valid switch since their latest login, else RP. That looks back at
most 5,000 commands; further than that, or with no login recorded, the
channel is left unknown. Character-creation answers typed in chat are not
channel chat but are labelled as if they were.

Sessions are rebuilt from login, logout and ping rows; see `sessions.py` for how crashed sessions end.

Movement is a player's session rows in a window (up to 7 days for one
player, 24 hours for everyone): logins, logouts and a position ping once a
minute, so the path between pings is a guess. Every request is recorded as
`player.movement.view` (who, which player or everyone, the window and how
many rows, never positions) before it is returned, and refused if that
record cannot be written. Rows are read newest first in batches of 5,000,
one statement each; a window holding more than the limit keeps the newest
rows and reports `complete_from`.

## Deployment gate

`backend/benchmarks/coreprotect_reader.py` measures both sides:

```sh
# Lock hold time per request against a live database (opens it immutable, benchmark only)
python3 benchmarks/coreprotect_reader.py hold --no-lock /path/to/database.db
# A disposable database at Main's size, then writer impact on it (contend writes)
python3 benchmarks/coreprotect_reader.py synth /tmp/main-synth.db
python3 benchmarks/coreprotect_reader.py contend /tmp/main-synth.db --readers 1
```

Acceptance: `hold` p99 ≤ 100 ms and max ≤ 500 ms; `contend` with no writer
failures and writer transaction p99/max up by no more than 100/500 ms.

Results on 2026-10-06:

- Movement, inside Main's backend container (60 requests): everyone over
24 h (26,900 rows, 10 statements) 33 ms per request, longest statement
58 ms, usually under 8 ms; one player over 7 days (3,155 rows) 4 ms. Do
not time live reads through `~/work/amp-readonly`: it is a FUSE mount that
does not pass file locks through, so SQLite sees CoreProtect's journal as
hot and fails with "attempt to write a readonly database".

- `hold` on live Main (7.4 GB, 242 players, 1,455 requests): p99 8.6 ms,
max 18.5 ms.
- `contend` on a 6.5 GB synthetic Main, one reader (as in production):
writer p99 +10 ms, max +18 ms, no failures, no reader errors.
- Three readers making back-to-back requests: writer p99 +18 ms, max
+14 ms, throughput −5%, no writer failures; 2% of reader requests gave
up as busy or out of budget. The synthetic writer spills its cache on
every batch, so this is harsher than CoreProtect.
Technical documentation is maintained in [TF-Minecraft/Docs](https://github.com/TF-Minecraft/Docs).
132 changes: 3 additions & 129 deletions backend/src/luckperms/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,133 +1,7 @@
# LuckPerms in the staff panel

The website shows and edits LuckPerms without touching its database. One
server's TFMCWeb (Main's) is the bridge: it publishes a snapshot of LuckPerms
to the site and applies changes that staff queue on the site, through the
LuckPerms API. Dev and Main share one LuckPerms MariaDB, so exactly one server
may apply changes. Another server may publish to its own site (Dev publishes to
dev.tfminecraft.net) without applying, which makes that site read-only.
Permission snapshots and queued staff changes through the TFMCWeb bridge.

## Rights
[Configuration and technical reference](https://github.com/TF-Minecraft/Docs/blob/main/projects/ProvinceSystem/docs/integrations/luckperms.md)

| Who | May |
| --- | --- |
| mod | view everything: groups, tracks, every player's groups and permissions, change history |
| admin | add or remove the groups in `policy.yaml` `admin_groups` on players, promote/demote players along tracks between those groups, set or unset player permissions matching `admin_permissions` but not `root_only_permissions` (such as `armourshop.admin`) |
| root | everything, including staff groups, other permissions, meta (prefix/suffix/weight) on players, group definitions and tracks |

A listed group counts as an admin group only while everything it inherits, in
any context, is listed too, so a root edit to a group definition cannot hand
admins a staff group by inheritance. Unknown permissions, wildcards and regex
nodes are root-only.

Admins may not change anything on their own Minecraft account, on a player
whose linked website account is admin or root, or on a player who holds or
inherits a root-only group (in-game staff, linked or not). Rights are checked
when a change is queued and again when the bridge collects it, so a demotion
in between cancels it (`no_longer_allowed`). Every change carries a
reason and is written to `admin_audit`, and LuckPerms logs it (`/lp log`) with
the source `web:<discord name>`.

## Plugin protocol

All routes need the primary `X-Plugin-Key`; secondary keys are refused.

### Snapshot

`PUT /luckperms/plugin/snapshot` (JSON, at most 16 MiB):

```json
{
"server": "main",
"generated_at": 1791321779,
"revision": 1791321779123,
"hash": "<sha-256 hex of the canonical groups/tracks/users JSON>",
"groups": [{"name": "staff", "display_name": null, "weight": 200, "nodes": [NODE]}],
"tracks": [{"name": "staff", "groups": ["staff_player", "staff_inactive", "staff"]}],
"users": [{"uuid": "…", "name": "drefvelin", "nodes": [NODE]}]
}
```

`NODE` is `{"key": "group.staff", "value": true, "contexts": {"server": ["main"]}, "expiry": 0}`;
`expiry` is unix seconds, 0 for permanent; `contexts` maps each key to its values (empty object for global).
`users` holds every user with at least one stored node (`UserManager.searchAll` with an empty key prefix).
The site replaces its whole mirror in one transaction and answers `{"ok": true}`.

`revision` orders everything the bridge reports: it strictly increases across snapshots and
results, also across restarts (the bridge uses `max(last + 1, current time in ms)`). The site
ignores a snapshot or result state whose revision is not above the newest it has stored
(answering `{"ok": true, "stale": true}`), so a request delayed past a newer one cannot undo it.

`POST /luckperms/plugin/snapshot/unchanged` `{"hash": "…", "revision": …}` answers `{"ok": true}` and
marks the mirror current when the hash matches the stored one, else `{"ok": false, "need_full": true}`.

The bridge builds a snapshot every `snapshot-seconds` (default 30) and right after applying changes,
on the same single worker thread that applies changes, so a snapshot never predates an applied change.

### Changes

`GET /luckperms/plugin/changes` returns up to 20 changes in id order and records the poll time
(the site shows the bridge as connected while polls are recent):

```json
{"changes": [{
"id": 12,
"target_type": "user", // user | group | track
"target": "5b0c…-uuid", // user UUID, group name or track name
"target_name": "drefvelin",
"actor_name": "web:w.o.n", // LuckPerms action source name
"actor_uuid": null, // the actor's linked Minecraft UUID, if any
"description": "parent add staff", // LuckPerms action log text
"ops": [OP],
"guard": {"admin_groups": ["default", "commoner"]} // admin changes only
}]}
```

A group or track change is handed out on its own, and nothing else is handed out until its
result arrives, so later changes are checked against the definitions it leaves.

`guard` is on changes an admin made. The mirror can lag in-game edits, so the bridge checks live
LuckPerms before mutating: the player must hold no group (any context) outside `admin_groups`
(else `target_is_staff`), and every group an `add_node` grants must inherit only groups in
`admin_groups` (else `root_only`).

Ops, applied in order to one holder and saved once:

| target | op | fields | precondition |
| --- | --- | --- | --- |
| user, group | `add_node` | `node` | the exact node (key, value, contexts, expiry) is absent |
| user, group | `remove_node` | `node` | the exact node is present |
| group | `create_group` | | the group does not exist |
| group | `delete_group` | | the group exists |
| track | `create_track` | | the track does not exist |
| track | `delete_track` | | the track exists |
| track | `set_groups` | `groups` (ordered) | the track exists and every group exists |

The bridge checks every precondition in op order against a staged copy of the live LuckPerms data
(so create-then-add and remove-then-add work) before mutating anything; if one fails, nothing is
mutated or saved and the result is `{"ok": false, "error": "<code>"}` with codes
`node_exists`, `node_missing`, `group_exists`, `group_missing`, `track_exists`, `track_missing`,
`bad_op`, `bad_target`, `save_failed`. A `group.<name>` node whose group does not exist is `group_missing`.
If saving fails after mutating a loaded holder, the bridge reloads that holder from storage so the
server does not keep an unsaved change in memory. After saving a user it pushes a user update; after
a group or track change it pushes a full update. It submits one LuckPerms action per change
(source UUID: the actor's linked Minecraft UUID, else the nil UUID).

`POST /luckperms/plugin/changes/results`:

```json
{"results": [{"id": 12, "ok": true, "error": null, "revision": 1791321780456, "state": STATE}]}
```

`STATE` (optional) is the target after the change: a user `{"uuid", "name", "nodes"}`, a group
`{"name", "display_name", "weight", "nodes"}` or `null` (deleted, or tracks). The site updates its
mirror for that target at once. The bridge keeps results it could not post and posts them again
on later ticks until the site answers `ok`.

Site side, a change is `pending` until fetched, then `sent`, and the site never offers it again: a
change is applied at most once. A sent change with no result after 300 s becomes `unknown` (staff
are told to check the player); a late result still settles it. A change not fetched within
10 minutes `expired`. Both timeouts are settled on staff reads too, so they happen even when the
bridge is gone. New changes are refused with `bridge_offline` unless the bridge polled in the
last 60 s, and always on a site with `LUCKPERMS_READ_ONLY=1` (dev.tfminecraft.net, whose server
shares LuckPerms with Main).
Technical documentation is maintained in [TF-Minecraft/Docs](https://github.com/TF-Minecraft/Docs).
Loading