NAME

appjail-tutorial — A tutorial introduction to AppJail

DESCRIPTION

AppJail is out of the box in many cases and does not need configuration files or third-party applications to work properly, however, this does not mean that AppJail is suitable for all kind of users without configuring it and does not mean that you cannot take advantage of third-party projects to provide more features.

NOTE: Sections are unordered, so you can jump to the section you really need.

This tutorial is divided into the following sections:

Introduction: Introduction to the most basic commands to download FreeBSD components, create a jail and list the jails we have created.
External Interface: Configuring the external interface, so that jails can access the outside and clients can connect to applications inside the jails using features such as NAT and port forwarding.
Configuration: Configuring some basic parameters explicitly to improve performance and stability.
Loopback Interface: Configuring the loopback interface if we want to use LinuxJails in combination with Virtual Networks.
Packet Filter: Configuring our packet filter with pf(4) anchors, so that jails can use NAT and port forwarding thanks to Virtual Networks.
Virtual Networks: Quick explanation of how to create a jail and join it to a Virtual Network.
DNS: Using meaningful names instead of dotted IPv4 addresses is very useful in many cases, so AppJail can use the Domain Name System ⟨DNS⟩ to take advantage of this feature.
RC Scripts: AppJail uses some RC scripts to start jails at startup, enable NAT for an entire network instead of per jail, supervise jails, update DNS information, etc. All of them are optional and you may or may not need them.
Resource Limits: Uncontrolled applications can exacerbate our system resources, but having an application inside a jail is very useful since we can limit the resources to what it really needs.
ZFS: The most advanced file system available today for FreeBSD and valid for AppJail is what we want to use when possible.

INTRO

To create a very basic jail, only two things are needed: obtaining the FreeBSD components ⟨base.txz, lib32.txz, ...⟩ and creating the jail using those components.

# appjail fetch
# appjail update release
# appjail quick jbasic start

appjail-fetch(1) will download MANIFEST and the components listed in COMPONENTS specified in your AppJail configuration file. By default, only (base.txz) is downloaded. MANIFEST contains the component checksums, so appjail-fetch(1) can check if they are incorrect.

After retrieving the components, update your release using appjail-update(1). At this point you can create jails using those components with appjail-quick(1). In the previous case only the start option is specified, so our jail will start after its creation.

That's all! You have a jail! A very simple and basic jail with no customization, no colors and no powers. Be patient, you'll wonder how many things you can do with AppJail, from basic ⟨create jails, start them, destroy them, ...⟩ to more advanced ⟨deploy complex applications using Makejails, each of them in separate jails with an active supervisor to avoid time of inactivity, ...⟩

List the jails you have created using appjail-jail(1) with the list subcommand.

# appjail jail list
STATUS  NAME    TYPE  VERSION       PORTS  NETWORK_IP4
UP      jbasic  thin  14.0-RELEASE  -      -

# appjail login jbasic

Stop your jail.

# appjail stop jbasic

And destroy it.

# appjail jail destroy jbasic

EXTERNAL INTERFACE

To avoid wasting precious time getting the external interface every time AppJail ⟨and its child processes!⟩ runs, it's a good idea to explicitly configure an external interface in your AppJail configuration file. More than performance it's about stability, which is more relevant when you have more than one interface you want to use with your jails, so AppJail will only use the one you explicitly configure.

Another recommendation is to change the name of your external interface. This is not necessary for AppJail, but it is necessary for you, as it is much easier to change one interface for another without your jails noticing under normal circumstances. Another reason is when you can use the same interface name on almost, if not all, of your servers, so you have fewer things to consider. AppJail is designed with the aforementioned problem in mind, so it is dynamic in almost all its phases, however, we recommend changing the interface name when possible.

# sysrc ifconfig_<interface>_name="<custom name>"
# sysrc ifconfig_<custom name>="<options>"

For example: Suppose we want to use em0 as our external interface and we want to change its name to jext and configure it using DHCP.

# sysrc ifconfig_em0_name="jext"
# sysrc ifconfig_jext="DHCP"

You can apply the same steps on another system but with a different interface, say, if_re(4).

For the above changes to take effect, we must restart etc/rc.d/netif.

# service netif restart

We recommend performing the above steps physically, so that any errors can be fixed as soon as possible. Or at least you should have some other way to fix any networking related errors.

The last step is to put the EXT_IF parameter in your AppJail configuration file, which is commonly /usr/local/etc/appjail/appjail.conf:

EXT_IF=jext

CONFIGURATION

As mentioned, AppJail doesn't require a configuration file to work properly, but it's a good idea to have one to get better performance and stability, so we only use what we explicitly configure. Not all parameters are listed here, these are just a complement, see appjail-conf(5)

EXT_IF: External Interface. In almost all cases, the interface you use to access the network.
See EXTERNAL INTERFACE.
ON_IF: The name or group of the network interface to transmit packets on. In almost all cases, it must have the same value as EXT_IF.
See EXTERNAL INTERFACE.
FREEBSD_VERSION: Default FreeBSD version without patch level that jails will use.
FREEBSD_ARCH: Default FreeBSD architecture that jails will use.
IMAGE_ARCH: Default architecture used by AppJail images.
SHORTEN_DOMAIN_NAMES: It is used to shorten the domain name of your jails, so that you can communicate between them using only their name, i.e. redis instead of redis.ajnet.appjail when using the DNS system.
ENABLE_ZFS: If you plan to take advantage of ZFS with AppJail, set this option.

Configuration example:

EXT_IF=jext
ON_IF=jext
FREEBSD_VERSION=14.0-RELEASE
FREEBSD_ARCH=amd64
IMAGE_ARCH=amd64
SHORTEN_DOMAIN_NAMES=1
# Remove the # character if you want to use ZFS with AppJail.
#ENABLE_ZFS=1

LOOPBACK INTERFACE

Since LinuxJails uses aliasing in combination with Virtual Networks, we need to clone a loopback interface.

# sysrc cloned_interfaces+="lo1"
# sysrc ifconfig_lo1_name="appjail0"

PACKET FILTER

An application inside a jail is not as useful when they need to communicate with external clients. To enable this, we need to enable pf(4), pflog(4) and add some anchors to our pf.conf(5).

# sysrc pf_enable="YES"
# sysrc pflog_enable="YES"

/etc/pf.conf:

nat-anchor "appjail-nat/jail/*"
nat-anchor "appjail-nat/network/*"
rdr-anchor "appjail-rdr/*"

Restart the rc scripts:

service pf restart
service pflog restart

Some AppJail features require you to enable IPv4 forwarding.

# sysrc gateway_enable="YES"
# sysctl net.inet.ip.forwarding=1

VIRTUAL NETWORKS

In early versions of AppJail, when you want to create a jail that is part of a Virtual Network, you must first explicitly create the virtual network. You can do this if you want, but we recommend that you leave that responsibility to AppJail. AppJail has the ability to automatically create a virtual network with some default values when you create a jail.

# appjail quick jtest \
	start \
	overwrite=force \
	virtualnet=":<random> default" \
	nat

If you want more details see appjail-nat(1), appjail-network(1) and appjail-quick(1), but basically we have created a jail named jtest which will have an interface with a randomly chosen name thanks to the <random> keyword. The left part of the character : is to indicate the virtual network that we want to use. If we leave that part empty ⟨as we do⟩ will use the default virtual network. The default virtual network is created if it does not exist.

DNS

AppJail does not come with a DNS server nor does it officially support one, but it does come with a configuration for dns/dnsmasq and generates a file similar to hosts(5) that can be consumed by DNSMasq or any other. In theory, you can use any other DNS server; see appjail-dns(8) for more details.

# appjail-dns
10.0.0.1        ajnet.appjail
10.0.0.2        jtest jtest.ajnet.appjail

As mentioned, this script generates a file similar to hosts(5), so we only need to tell a DNS system how to consume it. In the case of DNSMasq, we just need to enable, configure and start some RC scripts.

# sysrc appjail_dns_enable="YES"
# sysrc dnsmasq_enable="YES"
# sysrc dnsmasq_conf="/usr/local/share/appjail/files/dnsmasq.conf"
# touch /var/tmp/appjail-hosts
# service dnsmasq start
# service appjail-dns start

That's all, but we have a new problem: what IP address should our jails use to send DNS queries? That depends entirely on your environment, but in many cases, or at least for AppJail, you only need a private IPv4 address, so we'll configure a tap(4) interface and set a single IPv4 address. We recommend using this IP address instead of the host IP address because it is much easier to migrate between environments this way: if you use the host IP address and move to another environment with different network parameters, you must change resolv.conf(5) for each jail, which is not really hard even when you have many jails, but it is preferable to change things as little as possible.

# sysrc cloned_interfaces+="tap0"
# sysrc ifconfig_tap0_name="ajdns"
# sysrc ifconfig_ajdns="inet 172.0.0.1/32"
# service netif cloneup
# service netif start ajdns

The next step is to decide how our jails copy resolv.conf(5). There are many ways, but we recommend the most trivial and simplest: set DEFAULT_RESOLV_CONF in appjail.conf(5) to a resolv.conf(5) file, so that AppJail copies it instead of /etc/resolv.conf. Why is it preferable to use an explicitly resolv.conf(5) pathname instead of /etc/resolv.conf? Some applications can modify /etc/resolv.conf, so our jails will break their connections due to DNS issues.

DEFAULT_RESOLV_CONF="/usr/local/etc/appjail/resolv.conf"

Our resolv.conf(5) file at /usr/local/etc/appjail/ should be very simple.

nameserver 172.0.0.1

Now our jails can use a DNS hostname to communicate with another jail. That is fine, but we might want to do the same task on the host, so we'll need to configure /etc/resolv.conf to point to the address we configured for the ajdns interface. Very trivial, the problem is the one we mentioned: some applications can modify that file, but a solution may be to set the schg flag, preventing the modification of that file. Consider whether this will break your existing applications, but in many cases it will not.

After successful configuration, you can resolve DNS hostnames to IPv4 addresses.

# appjail jail list -j jtest
STATUS  NAME   TYPE  VERSION       PORTS  NETWORK_IP4
UP      jtest  thin  14.0-RELEASE  -      10.0.0.2
# host jtest.ajnet.appjail
jtest.ajnet.appjail has address 10.0.0.2

If you set SHORTEN_DOMAIN_NAMES=1 in your AppJail configuration file, you can use only the jail name.

# host jtest
jtest has address 10.0.0.2

RC SCRIPTS

Some RC scripts are part of AppJail and are commonly used to perform a task in the background or only during startup.

etc/rc.d/appjail: This RC script has the responsibility of starting jails at startup in the background and stopping them in the foreground. The reason for starting jails in the background is that the user probably doesn't want to wait for each of its jails to say (I'm up and running!) The stop part must be in the foreground for rc.shutdown(8) to work correctly.
etc/rc.d/appjail-dns: This RC script is responsible for updating the DNS information, that is, updating the hostname with the IPv4 address of the jail.
See DNS.
etc/rc.d/appjail-health: This RC script will start any healthcheckers you configure for any of your jails in the background. Note that the purpose of this RC script and healthcheckers is not to run forever: it will run only until no more healthcheckers are running. You must restart this RC script every time you add a new healthchecker or change a parameter. This means that if you exhaust your recovery attempts, the healthchecker will stop and won't run until you run it again. We recommend that you don't run healthcheckers forever, fix your application!
etc/rc.d/appjail-natnet: You can perform NAT per jail or per network. Typically, NAT is performed per jail, but there are some advantages to performing NAT per network as you only need to do it once and not more. Of course, it may not be wise to perform NAT for an entire network. This RC script is responsible for performing NAT per network at startup.

RESOURCE LIMITS

For resource limits to work in AppJail and in general, you must enable RACCT in your loader.conf(5) file and reboot your system.

kern.racct.enable=1

ZFS

To take advantage of this amazing, powerful and advanced file system with AppJail, you must enable it using ENABLE_ZFS=1 in your AppJail configuration file. There are other parameters you should consider, such as ZPOOL, the pool you want to use, which by default is zroot; ZROOTFS, the datasets root part, which by default is appjail; and ZOPTS, parameters passed to zfs-create(8), which by default is -o compress=lz4. You need to escape the shell characters for the last parameter.

AUTHORS

Jesús Daniel Colmenares Oviedo <DtxdF@disroot.org>

BUGS

Don't mix ZFS and non-ZFS file systems!

AppJail assumes that the user has correctly configured before use whether it want to use ZFS or not, i.e. it sets ENABLE_ZFS=1 or ENABLE_ZFS=0. If you already have an AppJail installation with data, you must remove or migrate it.

In any case if you have started jails, you must stop them all.

service appjail stop

⟨ZFS: Removing⟩ Assuming ZPOOL is zroot and ZROOTFS is appjail:

zfs destroy -Rf zroot/appjail

⟨non-ZFS: Removing⟩ Assuming PREFIX is /usr/local:

chflags -R 0 /usr/local/appjail
rm -rf /usr/local/appjail
rm -f /var/log/appjail.log
rm -rf /var/log/appjail

⟨Migrating⟩ Change the LOGDIR and PREFIX parameters in your AppJail configuration file to a different path or create a backup and delete those directories.

⟨ZFS or non-ZFS⟩ After cleaning you must decide whether you want to use ZFS or not. Review any ZFS-related AppJail configuration parameters before enabling it. After you set ENABLE_ZFS=1 and run AppJail, the datasets are automatically created in the pool you specified.

NOTES

AppJail Documentation