--- tags: - Slide section: infrastructure kind: - explain --- ## Infrastructure Mint System A short introduction into the infrastructure of Mint System. --- ### Drivers Early on we wanted to be able to self-host web application. ERP-Systems are operating system for companies - They have to be hosted reliably - Companies want know where their data is Infrastructure as Code: - Every system / server can be rebuilt from code - The deployment of configuration must be automated Run apps with Docker containers: - Manage containers not apps - Every decent web app provides Docker images --- ### Hardware We don't want to manage hardware. We use these providers: - Hetzner - Exoscale - Infomaniak - Ungleich And service partners. --- ### Ansible Everything is done with Ansible. There is a role for that: - Install and configure the OS - Manage firewalls and access rights - Deploy Docker containers - Manage backups and cron jobs - Build Wireguard networks --- ### Hosting We provide a managed server service. Every customer has a server. No shared environments. Every application is a hosting offer. --- ### Scripts For things that cannot be automated there are scripts: Ansible roles rely on these helper scripts. Helper scripts work independent of Ansible. --- ### Monitoring To monitor servers, containers and applications we use Prometheus/Grafana. Prometheus provides "Exporters" for many applications. Data collection and visualization is simple. --- ### Backup Backups are defined in Ansible and are done using [[Restic]]. - In Ansible inventory a backup set is defined - Ansible creates a cron job to run the backup - The backup job uses helper scripts to create local backups - The local backups are snapshoted with restic to the backup server - All backup data on the backup server is mirrored to an S3 bucket --- ### Proxy On every server there is Nginx-instance running. The Nginx config is generated by Ansible: Some roles require specific configs: Certificates are managed with certbot and let's encrypt. --- ### Auth We a running Keycloak and integrate it using OAuth/OpenID Connect. It is possible to manage all Keycloak config in Ansible. --- ### Security We try to apply best practices: - Update our applications / Docker images - Linux server patching - Basics: fail2ban, ssh pubkey, named users Nginx WAF with OWASP has been tried. It was too much effort to train. --- ### Challenges **Configuration drift** One an Ansible role or a config is update we have to apply this change to all hosts. This rarely done. **Proxy config not integrated** The proxy config is not provided by the Ansible role. It would be better if the application provides the config. **Backup verification** We cannot be sure that all backups are working. So far they did.