Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

Dit is de centrale locatie voor alle informatie over de website en de .ComCom.

Wil je iets weten over het gebruik van de website? Klik hier.

Wil je een suggestie doen voor de website of iets vragen aan de .ComCom (bijvoorbeeld dat we een stukje content aanpassen)? Klik hier.

Wil je iets weten over hoe de .ComCom werkt, of over wat je kunt leren bij de .ComCom? Klik hier.

The rest will be dedicated to explaining technical details, the choices we made, how to setup a development environment, how to deploy to production (and what that even means). We use English for the sake of future compatibility and so that it is easy to share anything written here online (and because many terms don’t have Dutch equivalents).

Changelog

Hier houden we alle veranderingen bij. Dit gaat specifiek om code, niet om content. De frontend wordt vaker geüpdate dan hier weergegeven (en heeft ook geen specifieke versies), maar de belangrijke veranderingen zijn wel hier samengevat.

Frontend (0232c32) - 2024-04-05

Changed (frontend)

  • (February/March) Jesper has updated a lot of content, including most of the commission photos
  • (March) Theme has been updated from winter to spring
  • (March) Records have been updated

Added (frontend)

  • A new “Pas aan” feature has been added for admins to manage the klassementen. This makes use of the 2.2 backend update.
    • It allows you to start new classifications
    • You can select classifications and modify their start, end and freeze date
    • You can make the database recompute the points based on whether or not you want to show the points added after freezing
  • Members can see when an event was last added and when it was frozen
  • (February) An Oud Leden Dodeka (OLD) page has been added

2.2.1 - 2024-01-10

Tip heeft code aan deze release code bijgedragen.

Released into production on 2024-01-11.

Deploy

  • Fix production image version

2.2.0 - 2024-01-10

Note: this version was not released into production.

This is the first full tidploy release on the backend with the mailserver moved to a dedicated provider. This should make all required features for the classifications fully work.

Senne en Tip hebben code aan deze release bijgedragen. We are now at 591 commits for the backend (including the 150+ commits that were from the deployment repository).

Added (backend)

  • <server>/admin/class/get_meta/{recent_number}/ (this returns the most recent classifications of each type, recent_number/2 for each type)
  • <server>/admin/class/new/ (creates a new points and training classification)
  • <server>/admin/class/modify/ (modifies a classification)
  • <server>/admin/class/remove/{class_id} (removes classification with id class_id)
  • <server>/member/get_with_info/{rank_type}/ (Will replace <server>/members/class/get/{rank_type}/, but added with different name to avoid breaking change, this also returns last_updated and whether the classification is frozen)
  • The last_updated column is now updated when new events are added.
  • (Senne) <server>/admin/update/training has been added (not yet in use by frontend) as the start of the training registration update.
  • New update_by_unique store method for updating an entire row based on a simple where condition.

Internal (backend)

  • The backend and deployment repository have been merged together, making deployment and versioning a lot easier. We now use a bunch of Nu scripts instead of shell scripts and the deployment will now use tidploy. This done in a series of commits from 4602abd to around 414bc23.
  • Logging has been much improved throughout the backend, so errors can now be properly seen. Error handling has also been improved (especially for the store module). Some common startup errors now have better messages that indicate what’s wrong. Documentation has been improved and all database queries now properly catch some specific errors. See #31.

2.1.1 - 2023-12-05

Tip heeft code aan deze release code bijgedragen.

Released into production on 2023-12-05.

Fixed (backend)

  • Remove debug eduinstitution value during registration

2.1.0 - 2023-12-05

Note: this version was not released into production.

Matthijs en Tip hebben code aan deze release bijgedragen.

Stats

We are now at 359 commits on the backend and 922 commits on the frontend.

Added (frontend)

  • An explanation on the classification has been added to the classification page.
  • There is now an NSK Meerkamp role.

Fixed (frontend)

  • Last names are no longer all caps on the classification page and are shown in full.

Added (authpage)

  • Add Dodeka logo to all pages, using new Title component.

Changed (authpage)

  • Use flex layout to make alignment better when pressing “forgot password”, so no the layout doesn’t jump slightly
  • Update node version to v20
  • Update dependencies

Fixed (authpage)

  • Educational institution is now recorded if not changed from default (TU Delft)

Added (backend)

  • Admin: Synchronization of the total points per user and events, as well as a more consistent naming scheme for the endpoints. All old endpoints are retained for backwards compatibility. Furthermore, admins can now request additional information about events on a user, event or class basis (see the PR).
    • <server>/admin/class/sync/ (Force synchronization of the total points per user, without having to add a new event)
    • <server>/admin/class/update/ (Same as previous <server>/admin/ranking/update, which still exists for backwards compatibility)
    • <server>/admin/class/get/{rank_type}/ (Same as previous <server>/admin/classificaiton/{rank_type}/, which still exists for backwards compatibility)
    • <server>/admin/class/events/user/{user_id}/ (Get all events for a specific user_id, with the class_id and rank_type as query parameters)
    • <server>/admin/class/events/all// (Get all events, with the specific class_id and rank_type as query parameters)
    • <server>/admin/class/users/event/{event_id}/ (Get all users for a specific event_id)
  • Member: Only renames, as described above.
    • <server>/members/class/get/{rank_type}/ (Same as previous <server>/members/classificaiton/{rank_type}/, which still exists for backwards compatibility)
    • <server>/members/profile/ (Same as previous <server>/res/profile, which still exists for backwards compatibility)

Changed (backend)

  • Types: The entire backend now passes mypy’s type checker (see the PR)!
  • Better context/dependency injection: The previous system was not perfect and it was still not easy to write tests. Lots of improvements have been made, utilizing FastAPI Depends and making it possible to easily wrap a single function call to make the caller testable. See #64, #65, #70 and #71.
  • Better logging: Logging had been lackluster while waiting for a better solution. This has now arrived with the adoption of loguru. Logging is now much more nicely formatted and it will be easily possible in the future to collect and show the logs in a central place, although that is not yet implemented. Some of the startup code has also been refactored as part of the logging effort.
  • Check for role on router basis: For certain routers, we now check whether they are requested by admins or members for all routes inside the router, making it harder to forget to add a check. The header checking logic has also been refactored and some tests have been added. Much better than the manual if check we did before. This also includes some minor refactor and fixes for access token verification.
  • There are now different router tags, which makes it easier to find all the different API endpoints in the OpenAPI docs view.

Fixed (backend)

  • An error is no longer thrown on the backend when a password reset is requested for a user that does not exist.

Internal (backend)

  • Live query tests: in the GitHub Actions CI we now actually run some tests against a live database using Actions service containers. This means we can be much more sure that we did not completely break database functionality after passing the tests. PR
  • Add request_id to logger using loguru’s contextualize
  • Added logging to all major user flows (signup, onboard, change email/password), also allowing the display of reset URL’s etc. so email doesn’t have to be turned on during local development

2.0.1 - 2023-10-25

Tip heeft code aan deze release bijgedragen.

Released into production on 2023-10-25.

Fixed (backend)

  • Fix update email: If you requested an email change twice, but only confirmed this after they were both sent, it is no longer to change it twice. After changing it using either one, the other one is invalidated.
  • Changed package structure so it is possible to extract the schema package and load it on the production server to run database schema migrations.

2.0.0 - 2023-10-17

Note: this version was not released into production.

Leander, Matthijs en Tip hebben code aan deze release bijgedragen.

Added (backend)

  • Admin: Roles, using OAuth scope mechanism, as well as classifications stored in the database, computed based on each event.
    • <server>/admin/scopes/all/ (Get scopes for all users)
    • <server>/admin/scopes/add/ (Add scope for a user)
    • <server>/admin/scopes/remove/ (Remove scope for a user)
    • <server>/admin/users/ids/ (Get all user ids)
    • <server>/admin/users/names/ (Get all user names, to match for rankings)
    • <server>/admin/ranking/update (Add an event for classifications)
    • <server>/admin/classification/{rank_type}/ (See current points for a specific classification)
  • Member:
    • <server>/members/classification/{rank_type}/ (See current points for a specific classification, changes hidden after certain point)

Added (frontend)

  • Member -> Classification page
  • Admin -> Classification page and Add new event
  • Roles can be changed in user overview

Changed (backend)

  • Major refactor of backend code, which separates auth code from app-specific code
  • Updated some major dependencies, including Pydantic to v2
  • Database schema update:
    • classifications table added to store a classification, which lasts for half a year and can be either “points” or “training”.
    • class_events table added, which stores all events that have been held (borrel, NSK, training, …). Possibly related to a specific classification.
    • class_event_points table added, which stores how many points a specific user has received for a specific event. In general, users will have the same amount of points per event, but this flexibility allows us to change that later.
    • class_points table added, which stores the aggregated total points of a user for a specific classification. When an event is added, this table should be updated using the correct update query.

Changed (postgres)

  • Updated from Postgres 14 to 16

Changed (redis)

  • Updated from Redis 6.2 to 7.2

1.1.0 - 2023-3-18

Released into production.

Changed (backend)

  • Update dependencies, including updating Python to 3.11 and SQLAlchemy 2

Fixed (server)

  • Docker conatiner no longer accumulates core dumps, crashing the server after 1-2 weeks

1.0.0 - 2023-1-11

Initial release of the FastAPI backend server, PostgreSQL database and Redis key-value store. Released into production on 2023-1-11.

Added (backend)

  • Login: Mostly OAuth 2.1-compliant authentication/authorization system, using the authorization code flow. Authentication is done using OPAQUE:
    • <server>/oauth/authorize/ (Authorization Endpoint initialize)
    • <server>/oauth/callback/ (Authorization Endpoint after authentication)
    • <server>/oauth/token/ (Token Endpoint)
    • <server>/login/start/ (Start OPAQUE password authentication)
    • <server>/login/finish/ (Finish OPAQUE authentication)
  • Registration: Registration/onboarding flow, which requires confirmation of AV`40 signup.
    • <server>/onboard/signup/ (Initiate signup, board will confirm)
    • <server>/onboard/email/ (Confirm email)
    • <server>/onboard/confirm/ (Board confirms signup)
    • <server>/onboard/register/ (Start OPAQUE registration)
    • <server>/onboard/finish/ (Finish OPAQUE registration and send registration info)
  • Update: Some information needs to be updated or changed.
    • <server>/update/password/reset/ (Initiate password reset)
    • <server>/update/password/start/ (Start OPAQUE set new password)
    • <server>/update/password/finish/ (Finsih OPAQUE set new password)
    • <server>/update/email/send/ (Initiate email change)
    • <server>/update/email/check/ (Finish email change after authentication)
    • <server>/update/delete/url/ (Delete account start, create url)
    • <server>/update/delete/check/ (Confirm deletion after authentication)
  • Admin: Get information only with admin account.
    • <server>/admin/users/ (Get all user data)
  • Members: Get information for members.
    • <server>/members/birthdays/ (Get member birthdays)

Added (authpage)

  • Login page
  • Registration page
  • Other confirmation pages necessary for backend functionality

Added (frontend)

  • Profile page
  • Leden page -> Verjaardagen
  • Admin page -> Ledenoverzicht
  • Use React Query for getting data from backend
  • Use Context from React for authentication state
  • Redirect pages necessary for OAuth
  • Confirmation pages for info update

Added (server)

  • Docker container that contains the FastAPI backend server, as well as static files for serving the authpage.

Added (postgres)

  • Docker conatiner with PostgreSQL server

Added (redis)

  • Docker container with Redis server

Pre-1.0.0

The frontend went live in June 2021 and before the release of the backend, was regularly updated using a rolling release schedule. The frontend is not versioned.

Hoe gebruik je de website?

Ben je bestuur, en wil je iets over de adminfunctionaliteiten weten? Klik hier.

Ben je lid van D.S.A.V. Dodeka en wil je meer weten over wat je op de website kan doen? Dan kun je op de volgende pagina’s kijken:

Ben je lid van een commissie en wil je weten hoe je de website kunt gebruiken? Dan kun je bij de volgende onderwerpen terecht. Staat wat je wil er niet bij, neem dan contact met ons op. Er is veel mogelijk!

Account aanmaken

Leuke pagina’s

Klassementen

Records

Commissiemail

Vraag het bestuur voor e-mail en wachtwoord voor @dsavdodeka.nl account. Gebruiken jullie nog een @gmail.com account of een andere e-mail? Laat het dan aan het bestuur weten en zorg ervoor dat dit account wordt verwijderd.

Bestuur

Nieuw lid accepteren

Update klassement

Rollen

Vragen en suggesties

.ComCom

Commissie

De huidige commissie is:

  • Tijmen Hoedjes (voorzitter, o.a. QQ B6)
  • Xylander Parqui (QQ B7)
  • Ymme van Komen
  • Mei Chang van der Werff
  • Anne-Wil van Werkhoven
  • Jefry el Bhwash

Geschiedenis

Voormalige leden (op volgorde van commissie verlaten):

  • Laura Geurtsen
  • Donne Gerlich (QQ B2)
  • Nathan Douenburg
  • Aniek Sips (QQ B3)
  • Pien Abbink
  • Senne Drent
  • Leander Bindt
  • Jefry el Bhwash (o.a. QQ B4)
  • Sanne van Beek (QQ B5)
  • Matthijs Arnoldus (is een recordaantal aantal van 5 commissiejaren lid geweest)
  • Tip ten Brink
  • Liam Timmerman
  • Jesper van der Marel
  • Sanne Diepen

De commissie is opgericht in bestuursjaar 2 (2020/2021), door Matthijs, Jefry, Laura en Donne. Met Nathan als designer hebben ze in korte tijd een hele mooie website in elkaar gezet, die live ging in 2021. Dat jaar werd ook de eerste 24-uursvergadering gehouden. De website was vooral een bron van informatie en een uithangbord voor de vereniging, en natuurlijk het thuis van de Spike. Daarnaast was de .ComCom verantwoordelijk voor de e-mail.

De website was statisch, wat betekent dat er geen server was die data kon opslaan en dat je dus ook niet in kon loggen. Tip werd lid van Dodeka en ook meteen lid van de .ComCom in 2021/2022 en ging hard aan de slag om een “backend” te bouwen, waarmee wel ingelogd zou kunnen worden. Ook Pien werd lid en hielp mee met het bouwen van nog meer mooie pagina’s.

Dit project ging live in januari 2023. Dat jaar krijgen we ook een nieuw lid, Leander, die nieuwe functies maakt voor de backend. Ondertussen waren Matthijs en Pien hard bezig om de content te onderhouden, iets wat een steeds grotere taak werd.

2023/2024 zag veel leden komen en gaan, terwijl de content beter dan ooit werd onderhouden door o.a. Jesper. Achter de schermen werd gewerkt aan de trainingsinschrijving, maar er ging bijvoorbeeld ook een systeem voor de klassementen live. Jefry, lid van het eerste uur, nam helaas afscheid. Met zijn opgedane kennis lukte het hem om voorzitter te worden van de Lustrumcommissie.

Wat kun je doen leren?

Bij de .ComCom is enorm veel te doen en dus natuurlijk ook heel veel te leren. De taken kunnen in de volgende vier belangrijke delen worden opgesplitst:

  • Design: de website moet er natuurlijk mooi uitzien. Dat is echter niet het enige, want de website moet ook gebruiksvriendelijk zijn. De user interface (UI), dus hoe de gebruiker omgaat met de website, en de user experience (UX), dus de ervaring van de gebruiker, moeten ook top zijn. UI/UX design is dus misschien nog wel belangrijker en daar is dus enorm veel over te leren.
  • Content: de website bevat naast nieuws belangrijke informatie over o.a. hoe je lid wordt en wanneer de trainingstijden zijn, ook bijvoorbeeld informatie over huidige en toekomstige wedstrijden en activiteiten. Die moeten constant geüpdate worden. De website is een uithangbord voor de vereniging en speelt een belangrijke rol bij het aantrekken van nieuwe leden. Dit maakt de “content” (de tekst en plaatjes) erg belangrijk. Je zult expert worden in het doorspitten van de FOCUS-archieven en het schrijven van tekstjes.
  • Programmeren: een groot deel van het werk dat bij de .ComCom wordt gedaan, is inderdaad simpelweg programmeren. Het is wel best anders dan je misschien gewend bent bij de vakken die je op de uni volgt. Het zijn namelijk geen Python plotjes of algoritmes in Java. Het gaat hier om het bouwen en onderhouden van een grote applicatie (die ondertussen 3+ jaar oud is) met meerdere componenten. Hieronder geven we nog meer detail.
  • Systeemadministratie: dit is niet het meest sexy taakje, maar wel heel belangrijk. We kunnen namelijk ook privéinformatie. Daarnaast mogen de systemen niet zomaar uitvallen en moeten we de juiste keuze maken tussen aanbieders van servers.

Programmeren

Programmeren, coderen, developen, er zijn veel woorden voor wat wij doen. Je kunt het zelfs software engineering of architecture noemen. De volgende dingen bouwen wij allemaal:

  • Een website, de “frontend”, (JavaScript, HTML, CSS) met het React framework. Naast statische pagina’s komen er ondertussen steeds meer dynamische pagina’s bij, die up-to-date informatie ophalen van een server, die vervolgens moet worden gemanipuleerd. Daarnaast moet worden bijgehouden of je wel of niet bent ingelogd en welke onderdelen van de website je kunt zien. Het bestuur moet een tabel kunnen zien met informatie over alle leden, en dit aan kunnen passen. Als data verandert op de server, moet je dat ook meteen kunnen zien.
  • Een server die een toegankelijk is via een API, de “backend” (geschreven in Python), die reageert op aanvragen (requests) van de website. Deze slaat alle wachtwoorden op in een vorm die wij niet kunnen lezen en kan bewijzen of je echt bent wie je zegt dat je bent. Daarnaast moet de backend informatie op kunnen halen uit een database. Hiervoor gebruiken we SQLite.

Systeemadministratie

  • Beheren van de server zelf (Ubuntu Linux), updates uitvoeren, de toegang veilighouden.

Overview

The technical information is divided into four parts:

  • Setup: which only teaches how to get everything running on your local machine, so you can dive into the code and start developing right away.
  • Architecture: a deeper look at the architecture of the entire application, detailing why we made certain decisions and explaining things on a higher-level than you would find by looking at the comments and documentation in the source code.
  • Developing: details what kind of things you need to modify if you want to make changes. Contains tips on how to easily do actualy ‘development’, what files matter most, among other things.
  • Deployment: once you have developed something, it needs to actually go live. This section details all the steps you have to go through to deploy the code into the real world, how to administer the servers and related tasks.

Definitely look at Setup, and also at Deployment if you actually do an update. Otherwise, Developing should be what you look at next. Architecture is only necessary when you want to make big changes or understand things better.

Setup

Here you can find information on how to set everything up, both the frontend, backend and database, so you can start developing right away.

No matter what you do, you’ll need to install Git, so check out the instructions for that.

If you are only doing things on the frontend, all you need to know is how to setup the frontend.

If you are developing the backend, you will probably also want to test things on the frontend. Set up the backend with Backend setup. The current backend uses SQLite databases configured through dodeka/backend/envs/<environment>/.env. It should just work automatically and you should not have to perform any special set up.

Git

Terminal cheat sheet

# See what branch you are on and what changes you have.
git status

Installation

To share code with each other, we use a program called Git. We use Git to download the “online” version of the source code (which is hosted on GitHub) so that we can work on it locally on our own machines. We also use Git to again upload it to the server.

For an in-depth guide to Git, checkout the Pro Git book. For an overview of the most important commands, checkout GitLab’s cheat sheet and this other one. Want to perform some specific action? Check out Git Flight Rules. Also, ChatGPT is pretty good at Git nowadays.

You can also use a GUI client instead of the command line (although I nowadays recommend against that), such as the one integrated into Visual Studio Code (with an extension such as GitLens), GitKraken or GitHub Desktop.

But first, you’ll need to install Git itself. On Windows, use the link at the top here (the 64-bit standalone installer). On Linux, use your system package manager (instructions here). On macOS, use HomeBrew.

For the Windows installer, I recommend against adding the links to your context menu (so disable Windows Explorer integration). Git Bash Profile for Windows Terminal can be useful (if you’ve installed it). As default editor I recommend something like Notepad++ or Notepad. For the rest the default options should be okay. Be sure to use the recommended option for “Adjusting your PATH environment”.

GUI

If you’re using a GUI program (so GitKraken, VS Code or GitHub Desktop), use their documentation to login to GitHub (make sure you have an account). Go to the frontend or backend for further instructions.

Why do I recommend using the command line? Because a lot of developer tools work with it exclusively. Because it’s simple to develop one, they usually have the most features and offer you the most control. At the same time, they also usually allow you to make more mistakes and can have a steeper learning curve, although ChatGPT has made things a lot easier nowadays.

To be able to download and upload repositories, you’ll still need to login to GitHub. For that, I recommend using the gh CLI (GitHub CLI). Install it by following the instructions at https://cli.github.com/. The gh CLI is also used by the backend to auto-download the auth binary, so it’s required for backend development.

If you haven’t already, I recommend installing “Windows Terminal”, which is much, much better than the standard Command Prompt. You can find it on the Microsoft Store (if the link doesn’t work, just search for Windows Terminal). You might also want to install PowerShell 7.

Once you have gh installed (you might need to restart your terminal), run gh auth login and follow the steps to login to your GitHub account (make sure you have one!). Finally, go to the frontend or backend for further instructions.

Frontend setup

For more information on developing the frontend, see the section on developing the frontend.

Setting up the frontend is the easiest. The frontend is entirely developed and deployed from the DSAV-Dodeka/DSAV-Dodeka.github.io repository. Like in the other pages, we will assume some familiarity with Git and GitHub and have Git installed (see the guide on Git).

Before you start, find some location on your computer where you want to store all the files. Try and put it somewhere you can easily find it again. For example, make a folder called “Git” in your “Documents” folder.

Next, be sure you have access to a terminal. Windows Terminal, which you can download from the Microsoft Store, is highly recommended. Once you have that installed, go to the “Git” folder or whatever location you chose, and right-click. It should then have a “Open in Terminal” option.

Otherwise, you can type Ctrl+L and then copy the file path (using e.g. Ctrl+C). This could be a path like C:\Users\tip\Documents\Git (where “tip” is replaced by your username). Then, just open Windows Terminal (just search for it on your computer), and type cd "<path>", where <path> is the path you just copied, so in my case that would be cd C:\Users\tip\Documents\Git.

You now have that folder open in your terminal, it should say something like:

PS C:\Users\tip\Documents\Git >

Ensure you are in that folder for the next part!

Downloading the code

The next step is to clone the repository to your computer. Because we store all our images inside the repository, the full history contains countless copies of large images. In the past, we didn’t properly optimize them so sometimes there were multiple versions of very large images. Thankfully, you don’t have to download the full history. Instead, when cloning, run the following command:

git clone https://github.com/DSAV-Dodeka/DSAV-Dodeka.github.io.git dodekafrontend --filter=blob:none

The --filter=blob:none option executes a “partial clone”, in which all blobs (so the actual file contents) of old commits are not downloaded. They are only downloaded once you actually switch to a commit. This saves a lot of disk space on your computer and makes the clone much faster.

After this is done, a new folder called dodekafrontend will have appeared. This is where all the code is located! I recommend editing the code by using Visual Studio Code and then opening that folder.

Installing NodeJS

The next step is to install fnm (https://github.com/Schniz/fnm), which will help us install NodeJS, which is a JavaScript runtime (a program that runs JavaScript code) based on the same internal engine as Google Chrome.

Open Windows Terminal (install it from the Microsoft Store if you don’t have it) and do this:

winget install Schniz.fnm

Open a new PowerShell terminal and run the following command:

if (-not (Test-Path $profile)) { New-Item $profile -Force }

This creates a “profile” file that will we then edit with the following command:

Invoke-Item $profile

Then, at the end of the file, copy-paste the following:

# Fast Node Manager to run NodeJS
fnm env --use-on-cd --shell powershell | Out-String | Invoke-Expression

Then, open a new PowerShell window and run:

fnm use 24

If it asks for yes or no, type “y” and press enter to install. This will actually install version 24 of NodeJS. Then, once you open a new terminal with PowerShell and type node -v, it should respond something like v24.x.x (where x.x can be any number).

Check https://github.com/Schniz/fnm for more instructions if something does not work, or feel free to ask ChatGPT. Otherwise, you can always ask Tip for help.

Installing dependencies

Next, open the command line in the root folder of the project. This can be easily done by opening it in a IDE (integrated development environment, I recommend using VS Code) and then opening a terminal there. Or, you can do what we did earlier (open it in file explorer, right-click or use Ctrl+L to get the path). Remember, open the dodekafrontend folder, not the Git folder or wherever you put it. Then, to install all dependencies, run:

npm install

Running the website locally

Once this is done, we can actually run the website using the command:

npm run dev

Under the hood, this will use Vite to bundle and build our project into actual HTML, CSS and JavaScript that our browser can run. The command will start a dev server, which will allow you to access the website in your browser using something like localhost:3000 (the port, 3000 in this case, might be different).

Backend setup

The backend is developed from the dodeka repository, in the backend folder.

Downloading the code

The backend is in a separate repository from the frontend. Clone it next to your dodekafrontend folder:

git clone https://github.com/DSAV-Dodeka/dodeka.git dodeka

Run all commands from the dodeka/backend folder!

  • Install uv. uv is like npm, but then for Python. I recommend using the standalone installer.
  • Ensure you have gh (GitHub CLI) installed and authenticated (gh auth login). See Git setup for instructions. This is needed to auto-download the auth binary.
  • Then, set up a Python environment. Use uv python install inside the ./backend directory, which should then install the right version of Python, or use one you already have installed.
  • Next, sync the project using uv sync --frozen. This will also set up a virtual environment at ./.venv.
  • Then I recommend connecting your IDE to the environment. In the previous step uv will have created a virtual environment in a .venv directory. Point your IDE to that executable (the file named python or python.exe in .venv/bin, ask an LLM or look at uv’s README for more details) to make it work.
  • Now run uv run dev inside the dodeka/backend folder. This will start two programs:
    • The auth server, based on faroe through tiauth-faroe
    • The Python backend itself, which builds on top of the very basic HTTP server freetser
  • On startup, you will see something like:
[Thread-6 (run_startup_actions)] [backend] Root admin bootstrapped: root_admin@localhost (user_id=7_root_admin)
[Thread-6 (run_startup_actions)] [backend] Root admin credentials: email=root_admin@localhost, password=yttU06VMelR5d7MU4bVt65RcSlmnOGSkIn0Qt0vQWaA

You can use those credentials to log in as admin through http://localhost:3000 (assuming that’s where your frontend is running).

Architecture

update start The site has three main parts:

  • dodekafrontend: React Router static frontend, deployed through GitHub Pages.
  • dodeka/backend: Python backend using freetser and SQLite.
  • dodeka/backend/auth: Go auth server using tiauth-faroe.

Use this book for orientation. For exact backend behavior, prefer dodeka/backend/docs/spec.md and dodeka/backend/README.md.

update end

Frontend vs Backend

The frontend is the website code that runs in the browser. It owns pages, styling, navigation, forms, and calls to the backend.

The backend runs on the server. It owns private state: users, registrations, permissions, imported Volta data, session validation, and member-only data.

The auth server is a separate Go process. It owns signup, login, password flows, email verification codes, and session tokens. The Python backend validates those sessions and maps them to Dodeka permissions.

The frontend talks to:

  • the Python backend through wrappers in dodekafrontend/src/functions/backend.ts
  • Faroe auth through dodekafrontend/src/functions/faroe.ts

The backend exposes public API routes from dodeka/backend/src/apiserver/app.py. It also has a private loopback API for the auth server and backend CLI tools.

Keep the split simple:

  • public content belongs in the frontend repository
  • private or member-only data belongs in the backend database
  • passwords and auth protocol belong in Faroe
  • exact backend rules belong in dodeka/backend/spec.md

Developing

Prerequisites

Development isn’t scary, but it’s probably new. Lots of jargon and all kinds of tools will be thrown around. Just let it come, you can only learn by doing.

Below I’ll introduce some concepts that are necessary to understand to develop for the website. Some of it you’ll know about or heard of already.

File system

Servers

Linux

Command line

Version control / Git

Browsers / JavaScript

Package managers

NodeJS / npm

Cheatsheet

This contains the most important information to get you up and running and productive.

Frontend

Open the command line in the folder where you downloaded DSAV-Dodeka.github.io / open the terminal in VS Code (Ctrl+`)

Run the website locally

(In your dodekafrontend folder, containing the package.json)

npm run dev

The website is now available in your browser at http://localhost:3000.

Update dependencies

npm install

Backend

(In your dodeka/backend folder, containing the pyproject.toml)

uv run dev

Git

Open the command line in the folder where you downloaded DSAV-Dodeka.github.io / open the terminal in VS Code (Ctrl+`)

General workflow

# go to the main branch
git checkout main
# update the repository
git pull
# go to a new branch (replace branchname with your desired name, no spaces or capital letters allowed)
git switch -c "branchname"
# add all edited files to future commit
git add -A
# commit the changes (change the description to something useful)
git commit -m "commit description"
# upload changes to github.com (replace 'branchname' with what you used earlier)
# in case you already pushed this branch before, you can just do git push
git push --set-upstream origin branchname

Status

See the current status (shows what branch you are on)

git status

Go to a branch

If you want to go to a particular branch, say ‘branch-xyz’, do:

git checkout branch-xyz

Update a branch

If you want to update your current branch with changes on github.com:

git pull

If you have changes locally, this might not work.

Delete all local changes (BE CAREFUL)

If you did some stuff you don’t know how to revert, but also don’t care to save it, do (be careful!):

git reset --hard

Frontend development

For more in-depth details on why certain decisions were made, see the frontend/backend architecture section.

The frontend development can be divided roughly into three sections:

  • Updating the content (don’t modify pages, just the text and images). See here. For information on images and how to optimize all images, see here.
  • Adding new static pages (design-focused page design). See here.
  • Creating dynamic pages (and integrating them with the backend). See the section in the backend here.

React and React Router

We use React Router 7 for client-side routing, which means code on the page itself handles navigation between subpages. This is also known as a “single-page application”. For more details, see the section on architecture.

Routes

Routes are defined in src/routes.ts using React Router’s configuration API.

Basic route

import { type RouteConfig, route } from "@react-router/dev/routes";

export default [
  route("contact", "./pages/contact/contact/contact.tsx"),
] satisfies RouteConfig;

This makes the page available at /contact.

Layout with nested routes

Most routes use a shared layout (navigation bar, footer). Wrap routes in layout():

import { type RouteConfig, layout, route, index } from "@react-router/dev/routes";

export default [
  layout("./pages/layout.tsx", [
    index("./pages/home/home.tsx"),           // Shows at /
    route("contact", "./pages/contact/contact/contact.tsx"),  // Shows at /contact
  ]),
] satisfies RouteConfig;

The layout component uses <Outlet /> to render child routes:

import { Outlet } from "react-router";

export default function AppLayout() {
  return (
    <div>
      <NavigationBar />
      <Outlet />  {/* Child routes render here */}
      <Footer />
    </div>
  );
}

Route prefixes

Group related routes under a common path prefix:

import { prefix, route, index } from "@react-router/dev/routes";

...prefix("vereniging", [
  index("./pages/vereniging/vereniging/vereniging.tsx"),      // /vereniging
  route("bestuur", "./pages/vereniging/bestuur/bestuur.tsx"),  // /vereniging/bestuur
  route("commissies", "./pages/vereniging/commissies/commissies.tsx"),  // /vereniging/commissies
]),

Dynamic routes

Use :param for dynamic segments:

route(":wedstrijdPath", "./pages/wedstrijden/eigen/wedstrijd-loader.tsx"),

Access the parameter in your component:

import { useParams } from "react-router";

function Wedstrijd() {
  const { wedstrijdPath } = useParams();
  // ...
}

Creating a new page

  1. Create a folder in src/pages/ with your page name (lowercase)
  2. Add a .tsx file for the component and optionally a .css/.scss file for styles
  3. Export a default function component
  4. Add the route to src/routes.ts

Example page (src/pages/example/example.tsx):

import PageTitle from "$components/PageTitle";
import "./example.css";

function Example() {
  return (
    <div>
      <PageTitle title="Example Page" />
      <p>Hello world!</p>
    </div>
  );
}

export default Example;

Then in src/routes.ts:

route("example", "./pages/example/example.tsx"),

Adding to the navigation bar

Edit src/components/Navigation Bar/NavigationBar.tsx to add your page to the menu. The desktop and mobile menus are currently JSX in that component, not exported data arrays.

Prerendering

Static pages are prerendered at build time for better performance and SEO. Configure prerendering in react-router.config.ts:

import type { Config } from "@react-router/dev/config";

export default {
  appDirectory: "src",
  ssr: false,  // No server-side rendering, just static prerendering
  async prerender() {
    return [
      "/",
      "/contact",
      "/vereniging",
      "/vereniging/bestuur",
      // ... add your static pages here
    ];
  },
} satisfies Config;

When you add a new static page, add its path to the prerender() array. Dynamic paths can be generated programmatically:

async prerender() {
  const dynamicPaths = getDynamicPaths().map((p) => `/prefix${p}`);
  return [
    "/",
    "/static-page",
    ...dynamicPaths,
  ];
}

Pages not in this list will still work but are rendered client-side only. Only add routes that don’t change based on e.g. the user or other dynamic state.

.tsx vs .jsx

For new components, prefer .tsx, which ensures proper TypeScript checking. This makes development easier by providing hints about available properties and prevents bugs.

Content

The content can be found primarily in ./src/content. There you can find many JSON files. JSON (JavaScript Object Notation) is a format that can easily be read by a machine. In the actual pages, we import these files and then read them, putting the text actually on the website.

The images can be found in ./src/images. Import images directly from the relevant page/component, often through the configured aliases such as $images.

Images

Avoid uploading very large images. They make the repository slower to clone/update and they make the website slower, because images are usually the largest browser downloads.

Before committing new images:

  • Resize photos to the size the page actually needs.
  • Prefer .webp for photos.
  • Run npm run build in dodekafrontend; Vite lists generated asset sizes, which makes the largest images easy to spot.
  • If one image dominates the asset list, optimize or replace it before committing.

Images

Optimizing images

This script only works on Linux (or WSL).

Dependencies

  • img-optimize - https://virtubox.github.io/img-optimize/ (optimize.sh)

  • imagemagick - https://imagemagick.org/script/download.php (convert)

  • jpegoptim

  • optipng

  • cwebp

The last 3 can be installed on Debian/Ubuntu using:

sudo apt install jpegoptim optipng webp

Once you’ve downloaded the first script, run the following script from the img-optimize main folder (be sure to replace <DSAV-Dodeka repository location> by the correct path):

#!/bin/bash
# Script by https://christitus.com/script-for-optimizing-images/ (Chris Titus)
# Modified by Tip ten Brink

FOLDER="<DSAV-Dodeka repository location>/src/images"

#resize png or jpg to either height or width, keeps proportions using imagemagick
find ${FOLDER} -iname '*.jpg' -o -iname '*.png' -exec convert \{} -verbose -resize 2400x\> \{} \;
find ${FOLDER} -iname '*.jpg' -o -iname '*.png' -exec convert \{} -verbose -resize x1300\> \{} \;
find ${FOLDER} -iname '*.png' -exec convert \{} -verbose -resize 2400x\> \{} \;
find ${FOLDER} -iname '*.png' -exec convert \{} -verbose -resize x1300\> \{} \;
# Optimize.sh is the img-optimize script
./optimize.sh --std --path ${FOLDER}

We convert the images to a size of max 2400x1300, as higher resolutions don’t make a big difference.

Backend development

  • It is recommended to open the dodeka/backend folder in VS Code, not dodeka.
  • If you have not set it up yet, start with Backend setup.

Continuous Integration (CI)

Tests (including some additional tests that run against a live database) and all the above tools are all run in GitHub actions. If you open a Pull Request, these checks are run for every commit you push. If any fail, the “check” will fail, indicating that we should not merge.

VS Code settings

VS Code doesn’t come included with all necessary/useful tools for developing a Python application. Therefore, a few extensions have been added in dodeka/backend/.vscode/extensions.json. VS Code should automatically recommend them for you to install.

Routes

The backend route table currently lives in handler_with_client() in dodeka/backend/src/apiserver/app.py. Check the route_table dictionary in that function for exact paths, methods, and permission requirements before adding frontend calls or documenting API behavior.

Integrating the backend/frontend

The database is the only place you can securely store private information. Everything stored in the repositories can easily be accessed by anyone. In the future, we might want to make an easier way to store private content.

So, if you want to display private backend data on the frontend, load it with an HTTP request. The current frontend uses native fetch wrappers in dodekafrontend/src/functions/backend.ts and React Query hooks/options in dodekafrontend/src/functions/query.ts.

A query

A “query” is basically an automatic function that, once the page loads, will load whatever function you ask it to and keep it up to date. It can be enabled based on whether or not someone is authenticated.

Use the current files as the source of truth:

  • Add low-level HTTP calls in dodekafrontend/src/functions/backend.ts.
  • Add React Query hooks or queryOptions in dodekafrontend/src/functions/query.ts.
  • For backend route names and permissions, check the route table in dodeka/backend/src/apiserver/app.py.
  • For backend commands, ports, and auth details, prefer dodeka/backend/README.md over copying commands here.

Minimal pattern:

// backend.ts
export async function loadThing(): Promise<Thing> {
  const response = await get("/some/backend/route/", true);
  if (!response.ok) throw new Error(await response.text());
  return response.json();
}

// query.ts
export function useThing(enabled: boolean) {
  return useQuery({
    queryKey: ["thing"],
    queryFn: backend.loadThing,
    enabled,
  });
}

tiauth-faroe

The backend auth server is a Go binary that uses the tiauth-faroe library.

The source lives in backend/auth/. Check backend/auth/go.mod for the current Go and tiauth-faroe versions; do not copy a version from this book page.

For now, tiauth-faroe is managed by Tip. Ask him if you want something updated or if the CI is broken. Feel free to replace it with something else, there are many other solutions to authentication. Just make sure you know what you are doing!

Running in development

When you run uv run dev from the backend/ directory, the auth server is started automatically alongside the Python backend. If the auth binary is missing or does not match backend/auth/auth-binary.sum, it is downloaded from the pinned GitHub release asset using the gh CLI. So everything should work out of the box and should not require any additional work!

Running auth server standalone (expert)

You can also run the auth server standalone, but this requires installing the Go programming language:

cd backend/auth
go run . --env-file envs/test/.env

The pinned binary version is set in dodeka/backend/auth/auth-binary.sum, using SHA256 hashes of release assets for each platform. See dodeka/backend/README.md for the current release and update-auth commands. If you compiled a binary locally and want to use it, pass --local-auth to uv run dev.

Updating the dependency in dodeka (expert)

After a new tiauth-faroe version is tagged and pushed:

cd backend/auth
go get github.com/tiptenbrink/tiauth-faroe/tiauth@<version>
go mod tidy

go get updates go.mod to the new version. go mod tidy cleans up go.sum by removing checksums for old versions that are no longer needed. Without go mod tidy, go.sum retains hashes of all previously resolved versions (this is a security feature, not a bug).

Building (expert)

cd backend/auth
go build -o auth .

CI builds are in .github/workflows/auth-ci.yml. Release binaries are built and attached by .github/workflows/auth-release.yml when an auth/v* tag is pushed.

Developing the deployment setup

Deployment

This is the starting point for running and updating the deployed backend.

Use these commands only on the production server as the backend user.

Quick Status

cd /home/backend/dodeka/deploy
uv run dodeka-status.py

For systemd directly:

sudo systemctl status dodeka-auth-demo dodeka-backend-demo
sudo systemctl status dodeka-auth-production dodeka-backend-production

Deploy Backend Changes

cd ~/dodeka
git pull

If Python dependencies changed:

cd ~/dodeka/backend
uv sync --frozen --no-dev

If auth code changed:

cd ~/dodeka/backend/auth
go build -o auth .

Restart the right environment:

~/dodeka/deploy/restart-demo.sh
~/dodeka/deploy/restart-production.sh

Logs

Follow logs:

journalctl -u dodeka-auth-demo -u dodeka-backend-demo -f
journalctl -u dodeka-auth-production -u dodeka-backend-production -f

Recent logs:

journalctl -u dodeka-auth-demo -u dodeka-backend-demo --since "1 hour ago"

Start And Stop

Auth should start before backend.

sudo systemctl stop dodeka-backend-demo dodeka-auth-demo
sudo systemctl start dodeka-auth-demo dodeka-backend-demo

sudo systemctl stop dodeka-backend-production dodeka-auth-production
sudo systemctl start dodeka-auth-production dodeka-backend-production

Demo Admin

cd /home/backend/dodeka/backend
uv run --frozen --no-dev ba --env demo get-admin-credentials

Reset Demo

This wipes demo auth and backend databases.

~/dodeka/deploy/reset-demo.sh

Manual Backup

~/dodeka/deploy/dodeka-db-backup.sh demo
~/dodeka/deploy/dodeka-db-backup.sh production

For restore, see Backups and restore.

Board Admin Account

The real board admin account is bestuur@dsavdodeka.nl. The root_admin@localhost account is only a bootstrap account.

First-time setup:

cd ~/dodeka/backend
uv run --frozen --no-dev ba --env production board-setup

After the board completes signup:

uv run --frozen --no-dev ba --env production grant-admin bestuur@dsavdodeka.nl

Yearly handover:

uv run --frozen --no-dev ba --env production board-renew

Grant admin to another existing user:

uv run --frozen --no-dev ba --env production grant-admin someone@example.com

SMTP

~/dodeka/deploy/enable-smtp.sh production smtp-relay.gmail.com

Demo can send real email too:

~/dodeka/deploy/enable-smtp.sh demo smtp-relay.gmail.com

If SMTP is disabled, fetch a signup token manually:

cd ~/dodeka/backend
uv run --frozen --no-dev ba --env demo get-token signup bestuur@dsavdodeka.nl

More

Server

Our server is hosted by Hetzner, a German cloud provider. Our server is an unmanaged Linux Ubuntu virtual machine (VM). VM means that we do not have a full system to ourselves, but share it with other Hetzner customers. We have access to a limited number of cores, memory and storage.

It is unmanaged because we have full control over the operating system. We need to keep it up to date ourselves. The choice for Ubuntu was also made by us. It has no GUI, only a command line, so getting familiar with the Unix command line is very helpful. By default, it uses the bash shell.

Webportal

The webportal for the server can be accessed from https://console.hetzner.cloud. The account we use is dsavdodeka@gmail.com. You need 2FA to log in.

The most important things you can do from the portal are:

  • See graphs of CPU, disk and network load, as well as memory usage
  • Manage backups
  • Access to root console

Connecting: SSH

To connect to the server, we use SSH. To be able to connect, you need to have an “SSH key” configured. To add one, you must first generate a private-public SSH keypair.

Then, the public part must be added to the ~/backend/.ssh/authorized_keys file. Note, this file requires some specific permissions, so if something is not working check whether these are correct.

Currently, we have the following important SSH settings:

PermitRootLogin no
PasswordAuthentication no

This ensure you can only log in with an SSH key, not with a simple user password.

We only allow logins through the backend user (see next section), so keep /root/.ssh/authorized_keys empty.

It is recommended to not add SSH keys through the web console, as these are not easily visible inside the authorized_keys file.

If you no longer have access to any keys, use the web console to log in as root, then change user to backend su backend and edit the authorized_keys file.

Connecting

Connecting is simple, simply do ssh backend@<ip address>. If your default identity is not a key that has access, you might need to use the -i flag to select the right key on your client.

Once you have done this, you have access to the server as if it’s your PC’s own command line.

root vs backend user

To keep things safe, try to avoid using the root user as much as possible. Instead, use backend. You can use sudo to run priviledged commands and and if necessary, log in as root using su root.

Keeping it up to date

To keep the server up to date, ocassionally run:

sudo apt update
sudo apt install

Ocasionally, Ubuntu itself might also get an update. It is best to only update once there is a new LTS version.

File locations

Currently, the dodeka repository is in /home/backend/dodeka.

Environments

Each environment (demo, production) runs two systemd services: an auth server (dodeka-auth-{env}) and a Python backend (dodeka-backend-{env}). Both need to be started separately, but they connect to each other automatically. Always start/restart auth before the backend.

Service Names

EnvironmentAuth serviceBackend service
Demododeka-auth-demododeka-backend-demo
Productiondodeka-auth-productiondodeka-backend-production

Files

Environment files live in:

  • dodeka/backend/envs/demo/.env
  • dodeka/backend/envs/production/.env
  • dodeka/backend/auth/envs/demo/.env
  • dodeka/backend/auth/envs/production/.env

Database files live next to those environment files.

Operations

Common deployment commands now live on the main Deployment page.

For backups and restore, see Backups.

For a new server, see Setting up from scratch.

Hetzner

Preparing SSH

First we rebuild the image from Ubuntu 24.04.

We reset the root password using Rescue -> Reset root password. I recommend then changing it to a new password once inside again (through passwd root).

Login using the console GUI.

Go to /etc/ssh and change the SSH server settings sshd_config:

PermitRootLogin no
PasswordAuthentication no

Then we create a new user (-m creates home directory, then we add them to sudo group):

useradd -m backend
adduser backend sudo

Then do passwd backend and set up a password.

Switch to the user using su backend.

The default shell might not be bash (for example if your prompt starts with only ‘$’), in that case run:

chsh --shell /bin/bash

Create a new directory mkdir /home/backend/.ssh. Enter this directory (using cd) and then do nano authorized_keys to open/create a new file there.

Paste in your SSH public key (generate one using ssh-keygen -t ed25519 -C "your_email@example.com") (the public key looks something like ssh-ed25519 .... tiptenbrink@tipc) and save the file (Ctrl-X). If copy-paste is not working, maybe try a different browser and make sure you’re not in GUI mode or similar.

Then, ensure the file has the correct permissions:

chmod 700 /home/backend/.ssh && chmod 600 /home/backend/.ssh/authorized_keys

Then you can log in with ssh backend@<ip address here> (make sure it uses the proper private key, so you might have to use the -i option).

Note that it’s often had to find out what’s going on when it’s not working. Be sure that the string in authorized_keys precisely matches your public key. Note that sometimes copy-pasting can not work correctly and some characters are changed (like = to -, or _ to -) or maybe you missed one character at the beginning or end. It really needs to match!

SSH niceties

Install “xauth” (while logged in as root)

apt install xauth

If you’re using a nice terminal emulator, you might have to add some xterm files. Consult the documentation of your terminal for details.

From this point on we never need to be logged in as root anymore! Always log in via ssh from your terminal

Dependencies

Update packages

sudo apt update
sudo apt upgrade

You might have to reboot after this: reboot.

Install basic C compiler and other useful packages

sudo apt install unzip build-essential sqlite3 zstd

Install NodeJS

Using fnm:

curl -o- https://fnm.vercel.app/install | bash

Be sure to re-login/start new terminal.

Set-up NodeJS 24:

fnm use 24

Verify with node -v, should return something starting with v24.

Install Python (with uv)

First, we install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then install Python (free-threading build) with:

uv python install 3.14t

Install Go

Check dodeka/backend/auth/go.mod for the Go version required by the auth server, then download that version for linux amd64/x64 from:

https://go.dev/doc/install

For example:

curl -OL https://go.dev/dl/go<version>.linux-amd64.tar.gz

Then put it in the install location with:

sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go<version>.linux-amd64.tar.gz

Finally, add go to your path by appending the following to the end of ~/.profile:

export PATH="$PATH:/usr/local/go/bin"

Install backend (and frontend)

We did not strictly need to install NodeJS and the frontend as we will not host the frontend ourselves. However, I will explain how to here, since that can be useful as a demo.

Frontend

First, let’s clone the frontend to the home directory.

git clone https://github.com/DSAV-Dodeka/DSAV-Dodeka.github.io.git frontend --filter=blob:none

Set up reverse proxy

Installing Caddy

We will install Caddy. Go to their docs for the precise command (the one with stable and containing sudo apt update among other things).

Caddyfile

demo.dsavdodeka.nl {
    # Set the document root for all requests
    root * /home/backend/frontend/build/client
    # Enable static file serving
    file_server
    # Try to serve the requested file, fall back to SPA entry point if not found
    try_files {path} __spa-fallback.html
}

# Backend API reverse proxy
backend.dsavdodeka.nl {
    # Create a named matcher "private" that matches:
    # - /private (exact match, no trailing slash)
    # - /private/* (any path starting with /private/)
    @private path /private /private/*
    # Return 403 Forbidden for any request matching @private, before it reaches the proxy
    respond @private 403

    # Forward all other requests to the production backend service
    reverse_proxy localhost:12980
}

# Auth service reverse proxy
auth.dsavdodeka.nl {
    # Forward all requests to the production auth service
    reverse_proxy localhost:12970
}

# Backend API reverse proxy
backend-demo.dsavdodeka.nl {
    # Create a named matcher "private" that matches:
    # - /private (exact match, no trailing slash)
    # - /private/* (any path starting with /private/)
    @private path /private /private/*
    # Return 403 Forbidden for any request matching @private, before it reaches the proxy
    respond @private 403

    # Forward all other requests to the demo backend service
    reverse_proxy localhost:12880
}

# Auth service reverse proxy
auth-demo.dsavdodeka.nl {
    # Forward all requests to the demo auth service
    reverse_proxy localhost:12870
}

Permissions

The Caddy server runs as the caddy user and if you want to run the demo using a file server, you will have to give it permission to access the files you generate when building the frontend. Therefore, we will create a new group and add both the backend and caddy users to it and give it access to the frontend build output.

The following snippets shows all commands you have to run.

# Create a shared group called "webdata"
sudo groupadd webdata
# groupadd: creates a new group
# webdata: the name of the group to create

# Add both users to the webdata group
sudo usermod -aG webdata backend
sudo usermod -aG webdata caddy
# usermod: modify a user account
# -a: append (add to group without removing from other groups)
# -G: specify supplementary group(s) to add the user to
# webdata: the group to add them to
# backend/caddy: the username to modify

# Change ownership of the client folder
sudo chown -R backend:webdata /home/backend/frontend/build/client
# chown: change file owner and group
# -R: recursive (apply to all files and subdirectories)
# backend:webdata: set owner to "backend" and group to "webdata"
# /home/...: the target directory

# Set permissions: owner full, group read+execute
sudo chmod -R 750 /home/backend/frontend/build/client
# chmod: change file mode/permissions
# -R: recursive
# 750: octal permission code
#   7 (owner): read(4) + write(2) + execute(1) = full access
#   5 (group): read(4) + execute(1) = can read and traverse
#   0 (others): no access

# Make new files inherit the group (setgid bit)
sudo chmod g+s /home/backend/frontend/build/client
# g+s: set the setgid bit on the directory
# This means new files/folders created inside will inherit
# the "webdata" group instead of the creator's primary group

# Ensure parent directories are traversable
chmod 755 /home/backend
chmod 755 /home/backend/frontend
chmod 755 /home/backend/frontend/build
# 755: 
#   7 (owner): full access
#   5 (group): read + execute
#   5 (others): read + execute
# "execute" on a directory means permission to traverse/enter it

# Restart caddy to pick up the new group membership
sudo systemctl restart caddy
# systemctl: control the systemd service manager
# restart: stop and start the service
# caddy: the service name

# Log out and back in (or run `newgrp webdata`) for backend user to pick up the new group
# newgrp webdata: starts a new shell with webdata as the active group

Backup volume creation

When you create a volume in the Hetzner Cloud Console and attach it to a server, it gets mounted with an auto-generated name like /mnt/HC_Volume_104307139. This guide shows how to rename the mount point to /mnt/backup.

Ensure you have a Hetzner Cloud volume attached to the server!

Step 1: Identify Your Volume

ls -l /dev/disk/by-id/

Look for something like scsi-0HC_Volume_XXXXXXXX.

Step 2: Unmount the Volume

sudo umount /mnt/HC_Volume_XXXXXXXX

Step 3: Create New Mount Point

sudo mkdir /mnt/backup

Step 4: Update /etc/fstab

Edit fstab:

sudo nano /etc/fstab

Remove the old auto-generated line, then add:

/dev/disk/by-id/scsi-0HC_Volume_XXXXXXXX /mnt/backup xfs discard,nofail,defaults 0 0

Step 5: Mount and Set Ownership

sudo mount /mnt/backup
sudo chown backend:backend /mnt/backup

Step 6: Verify

df -Th /mnt/backup
ls -la /mnt/backup

Step 7: Clean Up Old Mount Point

sudo rmdir /mnt/HC_Volume_XXXXXXXX

Step 8: Test Persistence

Reboot and confirm the volume mounts correctly:

sudo reboot
# after reboot:
lsblk

Restic

Installation

Download and install restic v0.18.1:

curl -L https://github.com/restic/restic/releases/download/v0.18.1/restic_0.18.1_linux_amd64.bz2 | bunzip2 > /home/backend/.local/bin/restic && chmod +x /home/backend/.local/bin/restic

After installing, you can update to the latest version with:

restic self-update

Note: Check the Restic GitHub releases page to verify you’re getting the latest version, as v0.18.1 may no longer be current.

Initialize Repository

First, create a password file (be sure to replace it with an actual randomly generated password!):

echo "your-secure-password-here" > /mnt/backup/.restic-password
chmod 600 /mnt/backup/.restic-password

Then initialize the repository:

restic init --repo /mnt/backup/restic --password-file /mnt/backup/.restic-password

The backup cron job and log rotation are set up automatically by install-services.sh (see Environments).

Backups

The database is backed up using Restic, which provides incremental, encrypted, deduplicated backups. Each backup is compressed with zstd before being stored. Because Restic deduplicates across snapshots, frequent backups are storage-efficient.

The backup scripts live in deploy/ in the dodeka repository. The implementation source of truth is dodeka/deploy/dodeka-db-common.sh, dodeka-db-backup.sh, dodeka-db-cron.sh, and dodeka-db-restore.sh.

How it works

  1. The SQLite database is exported with VACUUM INTO (produces a clean, compacted copy)
  2. The export is compressed with zstd --rsyncable
  3. The compressed file is uploaded to a Restic repository at /mnt/backup/restic

Each snapshot is tagged (e.g. db_dodeka, env_production) so you can filter when listing or restoring.

Setup

See server from scratch how to set it up on the server.

The backup scripts expect restic at /home/backend/.local/bin/restic, and sqlite3 and zstd installed on the server. The install steps are in server from scratch.

Setting up the cron job

The cron script (dodeka-db-cron.sh) runs a backup and then prunes old snapshots according to the retention policy. A pre-made crontab file is provided that backs up both production and demo every 12 minutes (with a 6-minute offset so they don’t overlap).

Install it with:

crontab ~/dodeka/deploy/dodeka_crontab

This replaces your current crontab. The schedule is:

  • Production: every 12 minutes (:00, :12, :24, :36, :48)
  • Demo: every 12 minutes at 6-minute offset (:06, :18, :30, :42, :54)

Output is appended to /home/backend/log/backup.log.

Retention policy

The cron script automatically prunes old snapshots with the following retention:

KeepCount
Latest10
Hourly24
Daily7
Weekly4
Monthly6

Manual backup

~/dodeka/deploy/dodeka-db-backup.sh production
~/dodeka/deploy/dodeka-db-backup.sh demo

Restore

# Restore from same environment
~/dodeka/deploy/dodeka-db-restore.sh production

# Clone production data to demo
~/dodeka/deploy/dodeka-db-restore.sh demo --from production

The restore script will:

  1. List available snapshots and ask you to pick one (or type latest)
  2. Stop the backend service
  3. Back up the current database (tagged pre-restore, so you can undo)
  4. Download and decompress the selected snapshot
  5. Replace the database file (clearing WAL/SHM files)
  6. Restart the service

Listing snapshots

# Prefer the helper used by the scripts when possible.
# Production snapshots:
restic -r /mnt/backup/restic --password-file /mnt/backup/.restic-password snapshots --tag "db_dodeka,env_production"

# Demo snapshots:
restic -r /mnt/backup/restic --password-file /mnt/backup/.restic-password snapshots --tag "db_dodeka,env_demo"

Administration

Google Workspace

Google Workspace is used to manage e-mail.

In order to send e-mails from the backend, we use SMTP.

We use the following two Google Support articles.

https://web.archive.org/web/20250517200708/https://support.google.com/a/answer/176600?hl=en https://web.archive.org/web/20250508111359/https://support.google.com/a/answer/2956491

We choose an SMPT relay, currently called “SMTP voor e-mail versturen via code”, with TLS required, only allowed IP addresses and SMTP authentication disabled. SMTP authentication is disabled because IP addresses are used to authenticate, which means we don’t have to worry about app passwords or other complexity.

Currently, allowed IP addresses should only be home networks of .ComCom members that need to test it as well as the backend Hetzner server.

To send an email, all you need is to connect to smtp-relay.gmail.com:587 using SMPT (with TLS enabled) from an allowed IP address. Then, you can just send an email with SMTP.

In Python, using smtplib works perfectly. However, in some languages (like Go), it’s important to make sure the local address name is set to your actual IP address. This sometimes requires sending an EHLO request first.