# ansible-polytech-2024 This is the repo that will serve as the support for the ASR practical work session of january. You should probably work in this directory and add code to it, it is going to be easier than creating a new ansible repository from scratch. You are also welcome to commit to this repository to checkpoint your work, as well as push it into any repository that polytech gives you access to. The goal here is to deploy a Synapse Matrix server, an opensource decentralised messaging app, using industry standard tools such as ansible and docker. By the end of this lab you'll be able to message each other in an end to end encrypted fashion using your own servers! ## How it's going to work * Log into https://gitea.plil.fr with your school credentials * Then _fork_ [this repository](https://gitea.plil.fr/tmaurice/polytech-ansible) (you need to do that only once per group) * Modify the repo's code as you see fit and remember to *commit and push often*. This is how Xavier and myself are going to assess progress and be able to grade your work. Optionally I would recomment you to add this repository as a remote so you can pull fixes should I have to push emergency fixes to the instructions (it happens more or less every year): ``` $ git remote add upstream https://git.plil.fr/tmaurice/polytech-ansible # If you want to get the latest code merged with your branch $ git pull upstream master ``` :warning: Please use explicit and descriptive commit messages when committing to git. ### Optional setup: a gitea runner for continous integration You can if you so wish deploy a [gitea runner](https://gitea.com/gitea/act_runner) somewhere, preferably on your zabeth. I would recommend you set the runner up for your user, and enable the "actions" in the settings of your newly created repo. This way every time you push your code, the checks in `.gitea/workflows` will be ran and you will have a live preview of the state of your code (how is the linting doing, what is the code quality, etc etc). If the consensus is "fuck that ain't nobody got time for that" I can also deploy one from my own server somewhere and you'll be able to use the runner for the entire length of the lab. :warning: you can run your gitea actions locally by running `make action` after installing the [`act` binary](https://github.com/nektos/act) The default provided CI pipelines only check for YAML and Ansible correctness, but you could imagine workflows where after these tests are run and are successful, we trigger an ansible run directly from the CI and the runner itself. Feel free to try :) ## Setup :warning: This setup has to be done on your Zabeth host. You will need to install a few things to get started, buckle up. Firstly you need to export some environment variables in your working terminal to be able to talk to the outside world using the school's proxy: ``` export https_proxy=http://proxy.polytech-lille.fr:3128 ``` Also, depending on the operating system installed on your virtual machines (Ubuntu server should not have the issue, Debian might), you should install python 3: ``` # apt install python3 ``` ### The virtualenv Since Ansible is written in python and we don't want to install it in the system, you will need to create a virtual environment. These are used to have your python stuff installed, without making them available system-wide, we are doing this to avoid polluting your lab machine with things that won't be used after today. To create the virtualenv you need to run the following: ``` $ python3 -m venv ~/.ansible-venv # then you want to "activate" the venv, you will need to do this for every new terminal you open $ . ~/.ansible-venv/bin/activate ``` :warning: you need to run `~/.ansible-venv/bin/activate` every time you want to open a new terminal and use ansible in it, otherwise it just won't work because the ansible binary won't be found. ### Install ansible Install ansible via pip _after entering the virtualenv_ ``` $ pip install -r requirements.txt ``` At this point you should have ansible installed. ### Install the docker role Install the docker role using ansible galaxy (ansible galaxy is a sort of package manager for ansible). ``` $ ansible-galaxy install -r requirements.yml ``` At this point you should be good to go! ### Generate an SSH key if you don't have one already You'll need an SSH key if you don't have one already If `ssh-keygen` complains about the key already existing, just reuse the existing key in case someone else needs it. ``` $ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -P '' ``` :warning: In real life, don't use `-P ''` because it creates your SSH key without a passphrase, it is ok for this lab, not for real life. :warning: Never, ever, EVER commit an unencrypted SSH keys to a git repository. If you need to add the key to the repository for some reason I would recommend you look up something like [sops](https://github.com/getsops/sops) to store an encrypted version of the secret in the repository. Lastly, look into `group_vars/all.yml`, go at the end of the file and add the created _public_ key in the root_user.default_root_keys (from `~/.ssh/id_ed25519.pub`, or any other keys you created before hand). If you want me to be able to hop on your machines to debug stuff please add my SSH key too: ``` ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPrw78OSJACq5MFXHrhAr2bPpnTNxwLE85mzij8gKmCs thomas@thonkpad ``` :warning: Before the first run of Ansible, if you don't have your SSH key installed on the machine, put it inside the `/root/.ssh/authorized_keys` file inside *both* your VMs, and `chmod 500` the file. ### One more thing, update your inventory You can now update your `inventory` file by modifying it with your new values (hostnames and ip addresses for the machines you'll be working with). The IP addresses you should put in there are the IPs that you have set in the VLAN50 (or the DNS names you have associated with it, if applicable) ### Read the roles to understand how everything works ! Ansible runs `playbooks`, which are collections of `roles` that in turn are a collection of `tasks`. `tasks` are instructions like "install this package", "copy this file", "create this directory", "install this service", "create this container" and so on and so forth. I have very much documented the example roles in `./roles` and I would _greatly_ encourage you to read them to understand how to do basic stuff in ansible such as copying a file, starting a service and so on. If you do not do that, you will be lost and won't understand anything that is coming at you. The playbooks on the other hand are stored in the `./playbooks/` directory, feel free to look at them to inspire your own playbooks for this lab. ### Check everything works properly You should now be able to actually `run` ansible to execute the `base.yml` playbook. ``` $ ansible-playbook -vi inventory -l all playbooks/base.yml ``` The `-i` flag specifies the inventory file to use, the `-l` file limits which hosts it applies to, either by hostname or group name, here we apply it to all the hosts. If you cannot connect to the host with a `permission denied` error, rerun the command using the `-k` flag, to make ansible ask you for the host password. ``` # make sure the sshpass program is installed on the Zabeth $ sudo apt install -y sshpass $ ansible-playbook -vi inventory -k -l all playbooks/base.yml ``` If that does not work, add it to the `~/.ssh/authorized_keys` in the `root` home folder on every one of your virtual machines, make sure the `authorized_keys` file has `600` permissions. If the permissions are too wide then sshd will refuse to use it. While you are at it I would recommend you install docker as well using the `docker.yml` playbook Good, now you are good to go ! ## Install a database server To deploy Synapse and Mastodon, you need to deploy a database server. We are going to use Postgres in this lab. You will for this need to use the `community.postgres_db` module for this. The `community.*` modules are modules written by the community and available to everyone, you will encounter similar modules when you will want to start deploying docker container! For more details to do this, I refer you to [the official ansible postgres documentation](https://docs.ansible.com/ansible/latest/collections/community/postgresql/index.html). Note that you will need to install (with ansible !) - `postgresql` - `python3-psycopg2` If you are feeling adventurous you can install postgresql using [docker](https://github.com/levonet/ansible-docker-postgres). Either way, you need to install this on your DB machine, which should be the one with the least amount of RAM. Go check the configuration of `/etc/postgresql/15/main/pg_hba.conf` and `/etc/postgresql/15/main/postgresql.conf` to make sure you can connect from `0.0.0.0`. On the `pg_hba.conf` file add a line like that at the end: ``` host all all scram-sha-256 ``` You can (and should) use the [postgresql_pg_hba](https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_pg_hba_module.html#ansible-collections-community-postgresql-postgresql-pg-hba-module) module for that. This module controles how clients can connect/authorise themselves to postgres. The netmask should look like `A.B.C.D/E` like `0.0.0.0/0` for everyone, for example. Also change the `listen_addresses` instruction to `'*'` in `postgresql.conf`. Then create the [user](https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_user_module.html#ansible-collections-community-postgresql-postgresql-user-module) and the [database](https://docs.ansible.com/ansible/latest/collections/community/postgresql/postgresql_db_module.html#ansible-collections-community-postgresql-postgresql-db-module). Please note that you need to pass the following parameters to the database you are creating, otherwise Synapse will most likely complain about encoding issues and refuse to start: ```yaml encoding: "UTF-8" lc_collate: "C" lc_ctype: "C" ``` If postgres coplains about encoding, you can also add `template: template0` to the above dictionnary. To create the user and the database you will need to `become` the postgres user on the server, you can achieve this by adding the `become: true` and `become_user: postgres` directives to your relevant tasks. Bonus point if you store the password in an encrypted fashion in your repository with either [ansible vault](https://docs.ansible.com/ansible/latest/vault_guide/vault_managing_passwords.html) or [sops](https://github.com/getsops/sops). If you end up using sops, please also encrypt your secret with the following age public key so I am able to decrypt them later on `age18rkuwwpzl3az5gr093uhvk7cwg348eajxsm9fjansur5qa97csfs597zh6`, this can be achieved running `sops --add-age ` ## Install Nginx You need to install nginx (or apache2, or traefik, whichever you are more comfortable with, but be aware I'll only be able to help with nginx and traefik as I have no idea how apache2 works). Use ansible for this, and install the `nginx-extras` package. ## Create a certificate for our deployments While you are at it you might as well create a DNS `A` fields to point at OpenWRT (which will forward it to nginx). To do this use Gandi and one of the domains you probably already have from other labs. We need to use a certificate to secure HTTPS communication, the Matrix protocols require it. This can be done manually for the moment and automated later, as it is not super straightforward. I refer you to the [documentation on certbot + nginx](https://galaxy.ansible.com/ui/standalone/roles/geerlingguy/certbot/) I would recommend that you create a `matrix.` certificate, then back them up somewhere safe as Letsencrypt has pretty aggressive rate limiting with regard to certificate creations. ## Let's install Synapse Synapse is the messaging app I told you about. ### Preconfigure synapse I would recommend you read quickly through this [synapse install guide](https://hub.docker.com/r/matrixdotorg/synapse/) so you understand what you are doing before touching anyting! In essence you need to run the following command in your lab machine: ``` $ docker run -it --rm \ -v $(pwd)/synapse-data:/data \ -e SYNAPSE_SERVER_NAME=matrix. \ -e SYNAPSE_REPORT_STATS=no \ matrixdotorg/synapse:latest generate ``` :warning: careful this is not the same command as in the documentation above, this will create the files you need in a `synapse-data` directory in your working directory, which is probably what you want in order to send them to your host via ansible. Create a new role and copy over these files to `roles/synapse/files`. Next customise the configuration file. If you feel like using postgres (you already set it up so why not, right ?) check the [docs](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#database). Another approach that would be cleaner, but potentially harder, would be to tell ansible to run the configuration bootstrap command on the synapse machine if the `homeserver.yaml` file cannot be found, and then potentially customise the file. ## Create the synapse docker container Normally your `synapse` module does not have any tasks yet, so go ahead and create a `roles/synapse/tasks/main.yml` file. You need to use the [`community.docker.docker_container`](https://docs.ansible.com/ansible/latest/collections/community/docker/docker_container_module.html#examples) module for this. In essence you need to make sure of a few things: - you want your container to port forward port 8008 in the host to 8008 in the container either that or you can just use the `network_mode: host` in the definition - you want to mount the directory in which you copied the `homeserver.yaml`, `matrix..log.config` and `matrix..signing.key` to `/data` in the container. For example, if I wanted to start a `redis` container that forwards 6379 and mounts `/tmp` I would do something like ```yaml --- - name: creates the redis container community.docker.docker_container: name: "redis" image: "{{ redis_image }}:{{ redis_image_tag }}" state: started recreate: yes ports: - 6379:6379/tcp volumes: - /tmp:/tmp restart_policy: "unless-stopped" ``` Note `restart_policy: "unless-stopped"` that ensures that your container is going to: - Be started on boot - Restart in case it crashes :information_source: Note that I used `{{ redis_image }}` in the manifest above. These are variables that are available in the same fashion as they are for [templates](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template_module.html) which use the [jinja2](https://jinja.palletsprojects.com/en/3.1.x/templates/) markup language. If you want to create a default variable for your module, you can create a `roles//defaults/main.yml` that will contain something like this for example ```yaml --- redis_image: redis redis_image_tag: latest ``` You can override this variable if you want in any of the following: - `vars` section of a playbook file - `host_vars/.yml` file for host specific overrides - `group_vars/.yml` file for group specific variables Anyways back to synapse, create a playbook real quick just containing the synapse role ```yaml --- - hosts: all roles: - synapse ``` Then apply it like we have seen above (`ansible-playbook -vi inventory -l playbooks/synapse.yml`) You should now have a running container by now, that has a directory mounted with your configuration. Check your container works with the `docker ps` command, you should see something like ``` root synapse.lil.maurice.fr ~ # docker ps | grep synapse 95aba77cc320 matrixdotorg/synapse:v1.74.0 "/start.py" 2 weeks ago Up 6 days (healthy) 8009/tcp, 0.0.0.0:8008->8008/tcp, 8448/tcp synapse ``` ### Configure the nginx reverse proxy You can see from the configuration that matrix listens on port `8008` for federation and client connexions. This means you will need to create one reverse proxy in your Nginx configuration, that will listen on port 443, that will utilise the TLS certificate you have generated earlier, and listens on the servername `matrix.`. To do that, update the configuration of your Nginx role to upload a file to `/etc/nginx/sites-enabled/matrix` looks like something like ```nginx server { listen 443 ssl; server_name matrix.; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA; ssl_prefer_server_ciphers on; ssl_session_cache shared:SSL:10m; ssl_session_tickets off; ssl_certificate .crt; ssl_certificate_key .key; location / { proxy_pass http://127.0.0.1:8008; } } ``` After a reload of nginx it should work properly. ### Configure openwrt To make you able to talk to synapse, you need to configure openwrt to send all packets from ports 443 and 8448 (the default federation port) to your docker host on port 443. You can do that either with the UI or with the command line config ### Create an admin user Note about the matrix addresses, a matrix address looks like this: ``` @: ``` For example my address is `@thomas:matrix.maurice.fr`, the localpart is `thomas` and the homeserver is `matrix.maurice.fr` In the following step you can create one or two users (for your team) to connect to the homeserver, pick the localpart carefully because that will be your username. Get into your synapse host and run ``` $ docker exec -it synapse register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml ``` Fill in the localpart, the password and you should be good to go. ### Test it works #### Check you can login The Matrix people wrote a nice web based application for using the protocol, called [Riot](https://riot.im/app/). Go to Riot, log in by specifying the homeserver URL, username and password, and verify the login goes smoothly. If it does not, check the logs. #### Test the federation works The federation is required to work for you to be able to speak to users on other servers ! Enter your hostname into the [matrix federation tester](https://federationtester.matrix.org/) and check the federation works #### Test by sending a message to another address Fire a message to `@thomas:matrix.maurice.fr`, or better yet, send a message to one of your colleague on another homeserver. ### Congratulations you reached the end ! This is pretty cool, however there is more you can do in the optional objectives of the lab, I would suggest you have a look at: * Setup a Gitea runner and make sure your tests pass * Add more tests ? * Using a mechanism (sops/ansible vault/hashicorp vault) to store secrets in the repository and make it accessible to the gitea runner * Finally, run ansible _from the CI_ so your playbooks get applied when you push new code to the repository