NAME

vmstated — Manage and run virtual machine in supervised manner

SYNOPSIS

vmstated [-h] [-c configdir] [-f] [-p pidfile] [-v]

DESCRIPTION

vmstated is a user-space daemon that manages bhyve virtual machines. It provides hook scripts for easy network and storage setup and maintains records on a virtual machine's state while it is running.

It provides a socket file on which utilities like vmstatedctl(1) can send commands and retrieve information about its state and about the virtual machines vmstated is managing.

Virtual Machine Configuration

vmstated maintains a list of virtual machines, which are defined and configured via directories and files under /usr/local/etc/vmstated.

It expects one sub-directory per virtual machine. The name of the sub directory may be chosen arbitrarily and does not have any impact on virtual machine configuration, for example /usr/local/etc/vmstated/vm0.

The contents of this virtual machine specific sub-directory define the configuration of the virtual machine and can also provide user defined hook scripts, which are called by vmstated when the associated virtual machine changes its state.

vmstated expects a UCL-formatted configuration file called config within the sub-directory. This file contains the specification of the virtual machine.

The following variables can be set for a virtual machine:

name: Specify the name of the virtual machine configuration. If omitted, vmstated assumes the parent tag name as the virtual machine name.
configfile: path to the bhyve configuration file that specifies the layout and settings for the virtual machine.
scriptpath: define an alternate path holding hook scripts instead of the default /usr/local/etc/vmstated/<name>.
os: Specify the operating system of the virtual machine.
osversion: Specify the operating system version of the virtual machine.
(owner): designate the name of the virtual machine's owner. This user may start and stop the virtual machine using vmstatedctl(1).
(group): designate the name of the virtual machine's owning group. Any user in that group may start and stop the virtual machine using vmstatedctl(1).
description: A brief description outlining the purpose and use of the virtual machine.
maxrestart: The number of times a virtual machine may restart within a set timeframe (see maxrestarttime) before it is put into failure mode. If no value is set, this value is set to 3 by default.
maxrestarttime: The timeframe in seconds in which the value "maxrestart" is being applied. For example, if maxrestart is set to 3 and maxrestarttime is set to 10, a virtual machine that attempts to restart 4 times within 10 seconds will put into failure mode. If no value is set, this value is set to 30 by default.
(bootrom): The path to the bootrom file to use, i.e. /usr/local/share/uefi-firmware/BHYVE_UEFI.fd
generate_acpi_tables: Equivalent to the bhyve_config option "acpi_tables"
vmexit_on_halt: Equivalent to the bhyve_config option "x86.vmexit_on_hlt"
memory: Amount of memory in megabytes to use for the virtual machine
cpus: The number of VCPUs to assign to the virtual machine
sockets: The number of VCPU sockets to apply
cores: The number of VCPU cores to apply

Variables in brackets are parsed by vmstated but are not yet applied to virtual machine configurations. They will be integrated in a future version.

Settings in a config file that modify virtual machine configuration are combined with those in the file referenced by "configfile"; vmstated does this by appending the generated configuration to the file specified by "configfile". Generated settings thus override any predefined settings found in the "configfile" file.

By default, vmstated takes the filename specified in variable "configfile" and appends ".generated" as suffix to construct the filename for the combined configuration file of original bhyve_config file settings and those listed in the config file.

Hook Scripts

vmstated allows placement of hook scripts within a virtual machine's configuration directory. It supports running scripts whenever the virtual machine enters one of the following states:

start_network	The virtual machine's network is being prepared, i.e. tap interface(s) are being created and added to bridge devices.
start_storage	The virtual machine's storage is being prepared, i.e. connections to remote storage is established, a volume is snapshotted before the virtual machine starts, etc.
running	The virtual machine has reached its running state.
restarting	The virtual machine has begun the process of restarting; this state may be assumed at any state before and after the virtual machine process terminates, depending on whether the restart was triggered from within the virtual machine or from outside via a management command sent to `vmstated`. Hence there is no guarantee that the virtual machine is still running or has shut down when this state is assumed.
restarted	The virtual machine has completed its restart cycle and is about to reenter into the running state.
stopping	Similar to the restarting state, the stopping state may be assumed any time during the actual shutdown of a virtual machine process; depending on the origin of the shutdown trigger, this state is either assumed before a TERM signal is sent to the bhyve(8) process or after it has already terminated.
stop_storage	The virtual machine's storage is being shut down and disconnected; any clean up steps are taken to make sure a later restart can re-connect to the virtual machine's storage again.
stop_network	The virtual machine's network is being shut down and cleaned up; the steps taken during the start_network phase are reversed to restore the system to its' original state.
stopped	The virtual machine is now stopped. It can be restarted by the user.

Hook scripts are executed by vmstated synchronously - it waits for each process to complete before it continues.

vmstated expects each hook script to return exit code 0 on successful completion. Any other exit code will lead vmstated to assume the state change failed and move the virtual machine into a failure state.

To make vmstated execute a script for a particular state, place an executable (binary or shell script) with the state's name in the virtual machine's configuration directory.

Output Logging

vmstated creates one separate log file under /var/log/vmstated for each virtual machine recording the output of bhyve(8) as well as of any hook scripts it calls during state transitions. The log filenames are set to the virtual machine's name and the suffix ".log".

The output from hook scripts is also included in each virtual machine's log file.

Log files record stdout as well as stderr output.

Failure Modes

If a virtual machine fails to start correctly, i.e. either due to misconfiguration, disk issues or other problems making bhyve(8) fail to complete the boot process, vmstated will keep track of the number of times it attempted to restart it. If the number of restart attempts exceeds the value of "maxrestart" within the last number of seconds set by value "maxrestarttime", vmstated will enter the virtual machine into failure mode and no longer attempt a restart.

Virtual machines are also entered into failure mode when one of its hook scripts indicates a failure by returning a non-zero exit code.

To restart a virtual machine that entered into failure mode, you first need to reset its state. You can do so by using vmstatedctl(1) with argument "failreset".

OPTIONS

-c configdir: specify configdir to be used as configuration directory instead of /usr/local/etc/vmstated.
-f: run vmstated in foreground mode
-h: print usage summary and exit immediately
-v: be more verbose when logging

FILES

/usr/local/etc/vmstated - default configuration directory path
/var/log/vmstated - default log directory path
/var/run/vmstated.pid - holds PID once vmstated has successfully started
/var/run/vmstated.sock - UNIX socket through which vmstatedctl(1) communicates with vmstated

EXAMPLES

To configure a virtual machine called "bsdvm", create a directory /usr/local/etc/vmstated/bsdvm and then create a new configuration /usr/local/etc/vmstated/bsdvm/config with the following contents:

bsdvm {
	configfile = /usr/local/etc/vmstated/bsdvm/bhyve_config;
}

Place a valid bhyve configuration file under /usr/local/etc/vmstated/bsdvm/bhyve_config. The file can be generated by calling bhyve(8) with the desired options as well as -o config.dump=1 and pipe the output into the bhyve_config file.

bhyve -c 2 -m 4G -w -H \
             -s 0,hostbridge \
             -s 4,ahci-hd,disk.img \
             -s 5,virtio-net,tap0 \
             -s 29,fbuf,tcp=[::]:5900,w=800,h=600 \
             -s 30,xhci,tablet \
             -s 31,lpc -l com1,stdio \
             -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
	     -o config.dump=1 \
             bsdvm > /usr/local/etc/vmstated/bsdvm/bhyve_config

Once the configuration file has been generated, make sure to remove the line "config.dump=1" from it:

sed -i '' s@config.dump=1@@g \
	/usr/local/etc/vmstated/bsdvm/bhyve_config

To add tap device "tap0" to bridge "bridge0" before the virtual machine starts and to destroy this tap interface when it shuts down, we can create a new script /usr/local/etc/vmstated/bsdvm/start_network with the following contents:

#!/bin/sh
ifconfig tap0 create
ifconfig bridge0 addm tap0

as well as /usr/local/etc/vmstated/bsdvm/stop_network:

ifconfig tap0 destroy

Make sure to make both scripts executable:

chmod 755 /usr/local/etc/vmstated/bsdvm/start_network
chmod 755 /usr/local/etc/vmstated/bsdvm/stop_network

Finally, you can now use vmstatedctl(1) to start the virtual machine:

vmstatedctl start bsdvm

You can then check the virtual machine's log file for output and progress information:

tail -f /var/log/vmstated/bsdvm.log

EXIT STATUS

The vmstated utility exits 0 on success, and >0 if an error occurs. It may fail for one of the following reasons:

It failed to read the configuration directory.
It failed to acquire necessary resources.
It wasn't launched with necessary credentials.

HISTORY

The vmstated command first appeared in FreeBSD 13.2.

AUTHORS

Christian Moerz <freebsd@ny-central.org>

CAVEATS

vmstated is work in progress. The programming interface, hook script architecture and the inner workings may change in future releases. While some degree of testing has gone into its development, this is meant as a reference implementation and brain teaser for those involved with bhyve and jails state management.

Hence, it will likely change inside and outside due to new insights that were learned from the development process. Expect having to redo hook scripts and configuration files when upgrading to a new version in the future.

At the moment, vmstated must be started as root. As of version 0.07, it is not making use of capsicum(4) to limit its system interaction capabilities.

Feature variables like "owner" and "group" can be defined in a config file but are not yet respected by vmstated when authorizing users to communicate with vmstated via its socket. By default this socket is not world-writeable and thus requires root level privileges to communicate through the use of vmstatedctl(1).

With its initial release, vmstated does not yet persist its state when shutting down. If you shut down vmstated without first shutting down the virtual machines it started, those bhyve(8) processes will become "orphaned" and will have to be shut down and cleaned up manually - as no hook scripts will be called.

Configuration names are not yet checked for their uniqueness by vmstated. Hence, if you label two virtual machines with the same name, expect undefined behavior.

Finally, vmstated does not allow to be started multiple times. If another instance of it is already running, it will print an error message and terminate.

BUGS

There are some known issues:

There is no way yet to make vmstated reload its configuration files. If you modify configuration files, you need to restart vmstated for it to re-read settings from the file system. However, unless you first shut down any running virtual machines, you will end up with orphaned bhyve(8) processes.
if a hook script fails to complete within a reasonable amount of time, it may cause vmstated to hang because it waits for the completion of every hook script before it continues processing events and commands.
if vmstated fails to correctly initialize after reading in its configuration files and forking; this may lead to situations where vmstated actually fails to start correctly but still returns exit code zero upon daemonizing.

There may be additional bugs lurking in the dark. Particularly with

parsing configuration files with syntax errors,
handling out of disk errors
handling out of memory errors
and more...