A new server infrastructure

Ecrit le

So I’ve redone my setup and everything seems working without collapsing on itself.

Yes, we’ve understood that with the title. Can you go in deeper details please ?

Choosing the virtualisation environment

The choosen virtualisation environment is Proxmox. Not only because people I know use it, also because I heard good feedback about it.

I also had the opportunity to (very quickly) have a try at it and see what it looks like.

Discovering Proxmox and various tests

I started by installing Proxmox in a virtual machine on my machine to see how the ISO behaves. Nothing very complicated at the moment. However, the installer doesn’t let you choose how the disk will be partitioned in the end.

With the installation finished, I got out the UI1 on the IP address assigned to the virtual machine by libvirt. I enter the root password that goes well and I’m left with trying it out and getting familiar with it.

Networking

How to get packets from outside to the containers ?

The first “obstacle” that I’ve encountered was the following: how will get from one external IP and distribute everything in the containers ?

For now, it was a good thing: the final setup will end up being behind a NAT and I couldn’t do otherwise. So I’ve done a not very clean double-NAT but it works. Below an example of firewall rule:

# Inbound packets
iptables -t nat -A PREROUTING -i $INTERFACE -p tcp --dport 80 -j DNAT --to xxx.xxx.xxx.xxx:80

# Outbound packets
iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o $INTERFACE -j MASQUERADE

The encounter of this little difficulty allowed me to learn and better understand how a packet flows through iptables (and nftables while I’m at it) and also to see that, despite my slim network knowledge, I can do something :p

The installation with libvirt with the VM network card configured as a NAT was fine since that’s what I would have had in the end.

Inter-container communication

This aspect is managed by a virtual network card created from scratch in /etc/network/interfaces. I had to search the web a little bit to find out which directives to use, but it wasn’t that complicated. The packets going to the containers arrive on this interface via the pre-routing of iptables and the packets to the outside take the IP address of the hypervisor with -j MASQUERADE.

Not the best approach but it gets the job done.

Storage

In terms of storage, I have enough space. It’s about how to configure Proxmox to take advantage of the storage spaces that are at its disposal and how to use them.

While poking around, I found a storage.cfg file in /etc/pve. This file allows you to do exactly what I mentioned above. Was it necessary to put the specifications of the mount points in /etc/fstab? I don’t think so.

I also messed a bit with the backups to see how they work. And finally, I also did an installation test on an existing Debian to learn things.

We get on the ride

I start the dance by stopping (and disabling the automagic start of) the various services available and backing up the current setup. Then I install Proxmox as it has been installed on my test setup. Except the weather’s getting worse. Already.

I had a strange error from the pveproxy about malformed JSON. So be it. I’m trying to see where this file could be before I give up and do a complete reinstallation of Debian and then put Proxmox on it.

Once the reinstallation was done, I started by installing Nginx compiled by hand from the sources using a script in a container. Then I left Nginx aside to focus on the mail server.

With the mail server up and running, I continued with Nginx. And so on: Postgersql, Gitea, Synapse, Lutim, Etherpad, the discord bot…

It was repetitive: rsync of the files, some just a simple scp, adapt the configurations. Anyway. The kind of thing that might have ellipsed into an edited video :p

Long story short, putting everything back in place took me two good days full of commands and adjustments. Sometimes forgotten detauls, sometimes mistakes such as:

  • assigning the same IP to two containers and wonder why everything breaks;
  • forgetting to change some paths in the service configurations;
  • putting the main storage of the container on the hard drive instead of the SSD RAID array.

At the end of the second day, everything is in order and functional. I went back to some containers to make minor adjustments to the configuration.

What about backups?

Well, yes, the container backups, how do I do that?

While reading the documentation, I noticed that it is necessary to allocate storage space via Proxmox’s pvesm utility and then assign it to the container as a mounting point.

Except that, by default, the allocated spaces are limited, for reasons that may be practical depending on the use case. Except that it’s not what I want, especially with the database and files downloaded by Synapse or Gitea’s Git repositories. However, there is a way to create a volume that will still be in a backup and that will have no disk limit2 with pvesm, precisely with the --format subvol option.

Example (see man pvesm for more details):

pvesm alloc data 150 dessert.subvol 0 --format subvol
# pvesm alloc <volume> <container ID> <name> <size>

Then mount the sub-volume with:

pct set 150 -mp0=data:150/dessert,mp=/mnt/dessert,backup=1

And with that, the volume will be backed up at the next vzdump.

In the end, I’m happy with the installation I have put together. This will allow me, among other things, to separate things properly without clashing them, especially with emails. I think it’s also a good gateway to virtualization. To be continued.


  1. User interface 

  2. In a manner of speaking. It will be limited by the disk space you have on the volume