Filesystem configuration ======================== TPA allows you to define a list of ``volumes`` attached to each instance. This list comprises both platform-specific settings that are used during provisioning and filesystem-level settings used during deployment. First, ``tpaexec provision`` will use the information to create and attach volumes to the instance (if applicable; see platform-specific sections below for details). Then it will write a simplified list of volumes (containing only non-platform-specific settings) as a host var for the instance. Finally, ``tpaexec deploy`` will act on the simplified list to set up and mount filesystems, if required. Here’s a moderately complex example from an AWS cluster: .. code:: yaml instances: - Name: one … volumes: - device_name: root volume_type: gp2 volume_size: 32 - device_name: /dev/xvdf volume_type: io2 volume_size: 64 iops: 5000 vars: volume_for: postgres_data encryption: luks - device_name: /dev/xvdb ephemeral: ephemeral0 vars: mountpoint: /mnt/scratch In this example, the EC2 instance will end up with a 32GB EBS root volume, a 64GB io2 volume (provisioned-iops EBS volumes) mounted as /opt/postgres/data, and a /tmp/scratch filesystem provided by an instance-store (“ephemeral”) volume, whose number and size are determined by the instance type. The details are documented in the section on AWS below, but settings like ``volume_type`` and ``volume_size`` are used during provisioning, while settings under ``vars`` like ``volume_for`` or ``mountpoint`` are written to the inventory for use during deployment. .. NOTE ephemeral0 instance store:: nowadays most of the internal storage is NVMe in which volumes are automatically enumerated and assigned a device name by AWS, hence you might need to modify `device_name` in your config.yml to whatever was given after the provision phase. default_volumes --------------- Volumes are properties of an instance. You cannot set them in ``cluster_vars`` , because they contain platform-specific settings. The :ref:`instance_defaults ` mechanism makes special allowances for volume definitions. Since volume definitions in a large cluster may be quite repetitive (especially since we recommend that instances in a cluster be configured as close to each other as possible, you can specify ``default_volumes`` as shown here: .. code:: yaml instance_defaults: default_volumes: - device_name: root volume_type: gp2 volume_size: 32 - device_name: /dev/xvdf volume_size: 100 instances: - Name: one … - Name: two volumes: - device_name: /dev/xvdf volume_size: 64 - device_name: /dev/xvdg volume_size: 64 … - Name: three volumes: - device_name: /dev/xvdf volume_type: none - Name: four volumes: [] Here every instance will have a 32GB root volume and a 100GB additional volume by default (as is the case for instance ``one`` , which does not specify anything different). Instance ``two`` will have the same root volume, but it overrides ``/dev/xvdf`` to be 64GB instead, and has another 64GB volume in addition. Instance ``three`` will have the same root volume, but no additional volume because it sets ``volume_type: none`` for the default ``/dev/xvdf`` . Instance ``four`` will have no volumes at all. An instance starts off with whatever is specified in ``default_volumes`` , and its ``volumes`` entries can override a default entry with the same ``device_name`` , remove a volume by setting ``volume_type`` to ``none`` , add new volumes with different names, or reject the defaults altogether. (This behaviour of merging two lists is specific to ``default_volumes`` . If you set any other list in both ``instance_defaults`` and ``instances`` , the latter will override the former completely.) Platform AWS ------------ On AWS EC2 instances, you can attach EBS volumes. .. code:: yaml instances: - Name: one … volumes: - device_name: root volume_type: gp2 volume_size: 32 encrypted: yes … - device_name: /dev/xvdf volume_type: io1 volume_size: 32 iops: 10000 delete_on_termination: false … - device_name: /dev/xvdg ephemeral: ephemeral0 … TPA translates a ``device_name`` of ``root`` to ``/dev/sda`` or ``/dev/xvda`` based on the instance type, so that you don’t need to remember (or change) which one to use. The ``volume_type`` specifies the EBS volume type, e.g., ``gp2`` (for “general-purpose” EBS volumes), ``io1`` for provisioned-IOPS volumes (in which case you must also set ``iops: 5000`` ), etc. The ``volume_size`` specifies the size of the volume in gigabytes. Set ``encrypted: yes`` to enable EBS encryption at rest. (This is an AWS feature, enabled by default in newly-generated TPA configurations, and is different from :ref:`LUKS encryption ` , explained below.) Set ``delete_on_termination`` to ``false`` to prevent the volume from being destroyed when the attached instance is terminated (which is the default behaviour). Set ``ephemeral: ephemeralN`` to use a physically-attached `instance store volume `_ , formerly known as an ephemeral volume. The number, type, and size of available instance store volumes depends on the instance type. Not all instances have instance store volumes. Use instance store volumes only for testing or temporary data, and EBS volumes for any data that you care about. For an EBS volume, you can also set ``snapshot: snap-xxxxxxxx`` to attach a volume from an existing snapshot. Volumes restored from snapshots may be extraordinarily slow until enough data has been read from S3 and cached locally. (In particular, you can spin up a new instance with ``PGDATA`` from a snapshot, but expect it to take several hours before it is ready to handle your full load.) If you set ``attach_existing: yes`` for a volume, and there is an existing unattached EBS volume with matching Name/type/size/iops, a new volume will not be created when launching the instance, but instead the existing one will be attached to the instance the first time it starts. Reattached EBS volumes do not suffer from the performance limitations of volumes created from snapshots. Platform bare ------------- TPA has no control over what volumes may be attached to pre-provisioned ``bare`` instances, but if you define ``volumes`` with the appropriate ``device_name`` , it will handle ``mkfs`` and ``mount`` for the devices if required. Platform Docker --------------- Docker containers can have attached volumes, but they are bind-mounted directories, not regular block devices. They do not need to be separately initialised or mounted. As such, the configuration looks quite different. .. code:: yaml instances: - Name: one platform: docker … volumes: - /host/path/to/dir:/tmp/container/path:ro - named_volume:/mnt/somevol:rw You may recognise these volume specifications as arguments to ``docker run -v`` . The volumes are attached when the container is created, and there are no further actions during deployment. RAID arrays ----------- On AWS EC2 instances, only RAID 0 is recommended by Amazon. You can create RAID volumes with a similar command: .. code:: shell sudo mdadm --create --verbose /dev/md0 --level=0 --name=MY_RAID --raid-devices=number_of_volumes device_name1 device_name2 This example will attach the block device named ``/dev/md0`` . The handling of ``volume_for`` or ``mountpoint`` during deployment happens as the same as with any other volume. TPA will handle ``mkfs`` and ``mount`` for it. .. code:: yaml - Name: one … volumes: - device_name: /dev/md0 vars: volume_for: postgres_data LUKS encryption --------------- TPA can set up a LUKS-encrypted device: .. code:: yaml instances: - Name: one … volumes: - device_name: /dev/xyz vars: encryption: luks luks_volume: mappedname volume_for: … If a volume with ``encryption: luks`` set is not already initialised, TPA will use ``cryptsetup`` to first ``luksFormat`` and then ``luksOpen`` it to map it under ``/dev/mapper/mappedname`` before handling filesystem creation as with any other device. (To avoid any possibility of data loss, TPA will refuse to set up LUKS encryption on a device that contains a valid filesystem already.) If you create a LUKS-encrypted ``volume_for: postgres_data`` , TPA will configure Postgres to not start automatically at boot. You can use ``tpaexec start-postgres clustername`` to mount the volume and start Postgres (and ``stop-postgres`` to stop Postgres and unmap the volume). The LUKS passphrase is generated locally and stored in the vault. Filesystem creation and mounting -------------------------------- If any ``device`` does not contain a valid filesystem, it will be initialised with ``mkfs`` . .. code:: yaml instances: - Name: one … volumes: - device_name: /dev/xyz vars: volume_for: … fstype: ext4 fsopts: - -cc - -m 2 mountopts: defaults,relatime,nosuid readahead: 65536 owner: root group: root mode: "0755" You can specify the ``fstype`` (default: ext4), ``fsopts`` to be passed to mkfs (default: none), and ``mountopts`` to be passed to mount and written to fstab (see below). TPA will set the readahead for the device to 16MB by default (and make the value persist across reboots), but you can specify a different value for the volume as shown above. There are two ways to determine where a volume is mounted. You can either specify a ``mountpoint`` explicitly, or you can set ``volume_for`` to ``postgres_data`` , ``postgres_wal`` , ``postgres_tablespace`` or ``barman_data`` , and TPA will translate the setting into an appropriate mountpoint for the system. Once the ``mountpoint`` is determined, the ``device`` will be mounted there with the given ``mountopts`` (default: ``defaults,noatime`` ). An entry will also be created for the filesystem in ``/etc/fstab`` . You may optionally specify ``owner`` , ``group`` , or ``mode`` for the volume, and these attributes will be set on the ``mountpoint`` . Remember that at this very early stage of deployment, you cannot count on the ``postgres`` user to exist. In any case, TPA will (separately) ensure that any directories needed by Postgres have the right ownership and permissions, so you don’t have to do it yourself.