Virtualize

Data Virtualization

In any given test data environment, data is needed. Conventionally, a user creates a separate database using a backup of a (production) environment or restores an existing test database with a backup. This is a process which reserves logical space on the machine where the database is located, while essentially being a copy of data that already exists somewhere else. When creating multiple copies of this same environment, this leads to linearly increasing storage costs.

Btrfs (“Butter FS” or “Better FS“) provides a solution to this problem by creating a file system that can immediately copy existing data without assigning that data its own logical space, and by instead creating a referral to the original data. Only when the data in this copy is edited, a new “Chunk“ of data is created, and the copy understands not to refer to the original data, but to this edited chunk whenever that specific data is being read. For unedited data, the referral to the original data is still used.

This means that if an original environment is copied, the only storage this copy uses on the file system is that of any edited data. This means that overall – less storage is used. Furthermore, whenever existing data is copied, because only a reference is created to existing data initially, cloning or restoring data from a snapshot is (nearly) instantaneous.

Core Principles

Now that we understand how the btrfs file system can facilitate efficient copying of data, we need to understand how this ties into the Runtime interface, and how the end-user can use this functionality. In one sentence:

Virtualize operates by using a pre-existing virtual machine, which contains Docker containers, which, in turn, contain a database image and the desired data.

Concretely, this means that the end-user only interacts with this virtual machine through the Runtime interface after the initial set-up is completed. From the Runtime interface, all of the day-to-day tasks carried out on the virtual machine can be done. This may contain, for example:

• Resetting an existing environment’s data to a previous snapshot

• Copying an existing environment, and creating a new container with the same image

• Reading container logging, or logging returned from the virtual machine

Mostly, when working with the currently offered DATPROF tools, executing runs and designing templates is completely unaffected.

VM Requirements

Because the configuration of a VM can be complex, DATPROF distributes a pre-existing VM which you can use as a starting template, with all the necessary settings already configured.

Should you want to configure your own, follow these specifications:

• A virtual machine which can be connected to externally, running a Linux distribution such as Ubuntu. On it, the following must at the very least be configured:

  • Docker, with SSL configured

  • OpenSSH, for configuring the connection to the VM from a client

  • BTRFS – a BTRFS file-system must be available.

The Virtualize Menu

In order to access Virtualize, first click on Virtualize in the bottom left corner of the Runtime menu, under the administration header.

Adding a container host

To add a container host, press Connect container host at the top-right corner of the container host overview.

Name

A name for the new container host. This name will also be displayed in the container host overview screen.

Host

The IP-address or host-name of the machine the virtual machine is hosted on.

Port

The port over which to connect to the virtual machine.

Save

Saves the current configuration, and navigates the user back to the container host overview screen.

Cancel

Discards any modifications, and navigates the user back to the container host overview screen.

Editing existing container hosts

To edit an existing container host, click on the Options drop-down box on any given row in the container host overview. After this, select Edit. The resulting overview is identical to that shown under Adding a container host, with the only difference being that pre-configured data is present in the data fields.

Removing existing container hosts

To remove an existing container host, click on the Options drop-down box on any given row in the container host overview. After this, select Remove.

Removing an existing container host does not impact the actual virtual machine at all. This merely removes the stored connection profile from Runtime.

Creating a virtualized environment

The process of creating an environment within your container host is largely identical to the regular process of creating an environment, however, because we are using docker to create a container, we need to supply some additional information.

To get started with creating a virtualized environment; select or create a Group in the main menu of Runtime, and press Add Environment in the top-right group overview screen.

Environment name

Enter the name of the environment you’re creating. This name will be displayed within the group overview screen and is used to identify the environment.

Database type

The type of SQL database that’s being added. This is a drop-down menu that currently contains the following options:

• IBM DB2 for iSeries

• IBM DB2 for LUW (Linux, Unix & Windows)

• IBM DB2 for z/OS (Mainframe)

• SQL Server (Which also serves as the Azure SQL connection type)

• Oracle

• PostgreSQL

• Other.. (default connector)

Environment type

In order to create a virtualize environment, the Create container database option must be selected. For the default environment type (indicated by Existing database option); please refer to the Environments chapter of the documentation.

Database settings

Container host

This is a drop-down menu that lists all configured container hosts from the Virtualize page. The selected container host’s connection data is used for any further operations within this environment.

Docker image

This is a free text field where the user can specify the name of any Docker image to be used. The name of this image must be a direct match, and any image used must be available within the Docker environment on the virtual machine beforehand.

Volume mapping

The volume mapping dictates where all data used is stored, in case the container is brought offline. The Host volume name refers to a location on the Virtual Machine where a volume can be created, and the Container volume refers to the location inside the container where any database information is stored. Generally, the latter uses a standard location, defined within a given database image.

Environment

Here, the user can (and in some cases must) pass environment variables to the container. For certain images/databases, an environment variable is needed to successfully create a database.

An example of a database image which cannot start without an environment variable is the PostgreSQL image. if the user does not supply a POSTGRES_PASSWORD and POSTGRES_USER key, deployment will fail.

After all the above steps have been completed, the user can press Add environment to save the current configuration.

Environment overview

Once a container database environment has been created, its settings can be reviewed by selecting the overview from the group overview. In general, once the environment has been configured, its usage is fairly similar to that of an ordinary environment. Two major differences are the Container and Snapshots sub-menus, which allow the user to configure and control the Virtualize specific aspects of this environment.

Container

Container actions

Using the container actions a user can give instructions to the container. Among these, the user has to option to:

• Start – start an exited or paused container.

• Pause – pause a running container.

• Stop – exit a running or paused container.

• Restart – exit a running or paused container, and then start it again.

• Clone as new environment – Creates a new environment using the same environment settings and the same image. takes the user to a new settings menu where the user must input a new environment name, and input a new port on the container host to be used for communication.

Container information

This section contains information about the container as returned by Docker.

Status

Portrays the current status of the container. This is a one-on-one representation of Docker’s container statuses. Available statuses are Running, Paused, Exited, Restarting, Dead, and Created.

Image

The name of the used database image. This is configured by the user whilst creating the environment, and should match the name of the image as returned when querying Docker.

To illustrate functionality, we’ll be using the Postgres 14 image. A valid name for the image in Runtime would be postgres:14 , as this is the name docker returns, plus the tag given to this image.

Container Name

The name of the container used.

Created

A timestamp of when this container was initially created, including the date and timezone.

Environment

The environment section displays all used environment variables, including those automatically generated by the chosen image.

Volume configuration

Displays the configured storage locations for the docker volumes on the host (virtual machine) and the container.

Port configuration

Displays the configured port configuration between the database container and the host (virtual machine).

Host

This must be an open port on the container host.

Container

This must be an open port over which the database communicates. For certain databases, this is likely to be the default port for the database (5432 for PostgreSQL, 1433/1434 for MS SQL Server, etc.)

Logging

Command log

Displays the logging for all commands executed on the host (virtual machine) regarding the environment. For instance, when first creating the environment, the command log logs the creation of the data volume directory on the host.

Container log

Displays the logging for any activity inside of the container, such as the logging that the database image generates.

Cloning existing containers to create new environments

The user is able to quickly clone an existing virtualize environment

Creates a new environment using the same environment settings and the same image. takes the user to a new settings menu where the user must input a new environment name, and input a new port on the container host to be used for communication.

Snapshots

The snapshots menu allows a user to create and restore snapshots of a container database. A snapshot is a stored copy of a container at a given timestamp. Concretely, this means that the user can save a database, and quickly roll back to a previous snapshot at any time. This can be desirable when, for instance, developing templates where an incorrect template configuration can damage aspects of your dataset.

Create snapshot

When first opening the snapshots sub-menu in a new environment, this is the only option available to the user. Creating a snapshot creates a point-in-time reference to the data in the database in that moment, and is a differential back-up. This means that a snapshot only saves all changes made since the last snapshot, and does not save the entire state of all the data. This significantly reduces the amount of space needed to make a back-up, and means creating one is significantly faster.

If the environment contains one or more snapshots already, this button will be displayed in the top-right corner of the snapshot overview menu.

After creating an initial snapshot, all existing snapshots will be listed in the snapshot overview.

Snapshot Overview

In the snapshot overview, all existing snapshots are listed, along with information pertaining to the snapshot.

Created By

Lists the username of the creator of a snapshot. This username refers to the one assigned to a given user in the User management portion of Runtime.

Created

The timestamp of creation for a snapshot.

Restoring from a snapshot

The user can restore from a snapshot by clicking Restore on any given entry in the snapshot overview. This will lead to the loss of any unsaved data in the container, and roll-back the data in the container to represent the data as it was during the timestamp of the snapshot.

Due to how btrfs handles restoring snapshots, this process is (nearly) instantaneous.

Deleting a snapshot

A snapshot can be deleted by pressing delete on any given entry in the snapshot overview. This process is not reversible.