NAME

qjail-vnet-howto — Details explaining how to use qjail to build a vnet jail system.

DESCRIPTION

This howto will strive to explain how the qjail utility is used in a real world application.

BACKGROUND DESCRIPTION

Prior to FreeBSD RELEASE 12.0 Vimage was experimental software that required a compile of the kernel to enable it. With FreeBSD RELEASE 12.0 Vimage became included in the base kernel no longer requiring a kernel compile to enable it. The jail(8) utility was updated to take advantage of the service provided by the Vimage emvironment being called a vnet jail. In the Vimage and vnet literature it says that a vnet jail has it's own ip stack and can run it's own firewall. Without coming right out and saying it the impression is given that vnet jails connect directly to the public internet. This is not correct. The fact is that all packet traffic traveling in or out of a vnet jail passes through the hosts firewall. Knowing this fact you can see that having a vnet jail perform NAT is useless and even having a firewall is redundant. Any way, FreeBSD's three firewalls; ipfilter, pf, and ipfw have been made Vimage aware. If you want to run a firewall in your vnet jail, best practice is to run the same firewall in your vnet jail as what is run on the host and let the host do the NATing. This is the default used by qjail.

Vnet jails created by qjail use the bridge/epair method. This method requires a class "C" ipv4 ip address allocation. What this means is if you assigned this ipv4 address 10.0.20.5 the vnet jail would encumber this ipv4 address range 10.0.20.0 - 10.0.20.255. For each additional vnet jail just bump the 3rd octal by one as in 10.0.21.5 and everything will work out just fine.

When assigning one of the private ipv4 addresses to a vnet jail it will cause that vnet jail to have access to the public internet if the host NATs that ip address. According to RFC 1918, the following IP address ranges are available for private IPv4 networks. This is intended for Local Area Networks and jails.
#
# 10.0.0.0 - 10.255.255.255
# 172.16.0.0 - 172.31.255.255
# 192.168.0.0 - 192.168.255.255
#

If you assign an ipv4 address that contains 127 in the left side octal then that vnet jail will only have local access. Meaning it will not be able to access the public internet. The class "C" rule applies here also. 127.0.0.1 is reserved and can not be assigned or be part of a class"C" allocation.

Creating a vnet jail with qjail is a two step process. First you create a normal non-vnet jail. The single ipv4 address you assign will become the ip address of the epairXb interface inside of the vnet jail. The second step is to "config -v" where you enter none, ipf, pf, or ipfw for the firewall you want to run in the vnet jail. The firewall will be ready to run with a simple rule set that lets everything pass accept for the "whois" command. Vnet jails allow the ping command to work and when you issue the whois command from inside of the vnet jail it will be blocked. Ping works and whois don't from inside of the vnet jail means everything is working. If it don't work check your host firewall rules.

The vnet jail will also be populated with some sh scripts to help manage your vnet firewall that are prefixed with the selected firewall name. Once you login to your vnet jail using the console command you can list the firewall helper scripts by issuing "ls /usr/local/bin/ipf.*" if ipf was the firewall you selected.

The following is based on a host system having network access to the public internet verified by issuing the "ping -c 2 8.8.8.8" command and the "whois frebsd.org" command on the host.

The following sequence of qjail commands were issued using the physical keyboard attached to a desktop type of computer running RELEASE 12.1. This gateway host uses the ipfilter firewall that provides the NAT services for public internet access. For brevity, the "ifconfig command" output shown below has had all the interface details removed execpt for bridge10 and the the epair information.

VNET JAIL CONFIG SETUP

Step 1. Enter pkg install qjail to Install the qjail software.

Step 2. Enter qjail install to install the qjail environment.

Step 3. Enter qjail create -4 10.10.110.10 v110 creates a normal
non-vnet directory tree type jail. This is the network ip
address used to communicate with the vnet jail. The name of the
jail is v110. A vnet jail gets a class "C" block of ip addresses
assigned to it. /24 is apended to the above -4 coded ip address.
This would allocate ip addresses 10.10.110.1 through
10.10.110.254 to the vnet jail with 10.10.110.10 being assigned
to the vnet jail and 10.10.110.2 being assigned as the default
route ip address for the vnet jail.

When you create additional vnet jails you must be aware of this.
So use an ip address scheme of something like this 10.10.111.10,
10.10.112.10 and so on.

Step 4. Enter qjail config -v none v110 To change the non-vnet
jail into a vnet jail.

The "-v none" means this vnet jail will not be using any
firewall. The "-v value" options are none, ipf, pf, ipfw.
Read "man qjail" config -v section for more details about
firewall selection.

Step 5. Enter qjail start v110 Start this jail.

Step 6. Enter ifconfig -a to see what the host's network looks like.

bridge10: flags+8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
description: qjail_vnet_jail_only_bridge
ether 02:3e:ba:a7:58:0a
inet 10.10.110.2 netmask 0xffffff00 broadcast 10.0.50.255
md6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddwlay
maxage 20 holdcut 5 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE>
ifmaxaddr 0 port 4 priority 128 path cost 2000 epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
description: qjail_vnet_jail_v110
options=8<VLAN_MTU>
ether 02:c1:00:00:04:0a
inet6 fe80::c1:ff:fe00:40a%epair1a prefixlen 64 scopeid
nd6 options=21<PERFORMNUD.AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

qjail defines it's bridge as bridge10 so hopefully it's name is unique enough not to get stepped on by some other host function.   The bridge has 1 member for the epair. epair1a is on the host and its matching pair epair1b is in the vnet jail. The number 1 is incremented by 1 for each additional vnet jail created. So if there were two vnet jails, there also would be a epair2a member shown.

Step 7. Enter qjail list List status of all defined jails. STATUS JID NIC IP Jailname ------ ---- ------ --------------- ------------------------------- DRV 0 7 fxp0 10.10.110.10 v110

D means this is a directory tree type of jail R means this jail is running V means this is a vnet jail 0 means this vnet jail has no firewall 7 means this is the seventh time a jail has been started

Step 8. Enter qjail console v110 This will log you into the
vnet jail. From there you can issue any host console commands
you want. Enter "exit" to leave the console.

FreeBSD 12.1-RELEASE-p8 GENERIC Welcome to your FreeBSD jail. v110 /root >  

Step 9. Enter ifconfig -a to see what the Vnet jail network looks like. epair1b: flags=8843<UP.BTOADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
options=8<VLAN_MTU>
ether 02:C1:00:00:05:0B
inet 10.10.110.10 netmask 0xffffff00 broadcast 10.0.110.255
inet6 fe80::c1:ff;fe00:50b%epair1b prefixlen 64 scopeid 0x2
nd6 options=21<PREFORMNUD,AAUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Dbase-T <full-duplex>)
staatus: active
groups: epair v110 /root >

Here we see the 10.10.110.10 ip address has been assigned to the epair1b interface in the vnet jail.

Step 10. Enter ping -c 2 8.8.8.8 Check to see if jail has public access. PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=43 time=37.337 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=43 time=35.983 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 35.983/36.660/37.337/0.677 ms v110 /root >

Step 11. Enter exit to leave the jail console.

Step 12. Enter qjail stop v10 to stop the jail.

Trouble shooting network problems.

1. Cannot access the public internet. Check the hosts Firewall rules or NAT for errors or omissions.

2. Cannot ping the hosts default route ip address. Most likely you don't know what the default gateway route ipv4 address is. On the host issue the "route -n get default" command and use the gateway ip address in the ping command issued from the vnet jails console.

3. A test you can do is try to do a ssh login on the host to the vnet jails ip address. First you have to prepare the vnet jail. Use the console command to login to the vnet jail. Edit the vnet jails rc.conf file adding the sshd_enable="YES" statement. Then issue "pw adduser test -m -w yes" to create a user account named test with password of test. Exit the jails console. Issue "qjail restart v110" and then issue "ssh 10.10.110.10 -p 22 -l test" and see if you can access the vnet jail this way. The host has to be runing ssh for this to work.

An alternate method to enable ssh in a jail is to issue this command from the host console "qjail config -h v110" followed by "qjail restart v110". A login id of v110 with a pw of v110 will be automatically created.

4. Be aware that you can run any of the 3 firewalls on the GATEWAY host and no firewall in the LAN vnet jail and have conplete control of the LAN vnet jail's network traffic from the GATEWAY host's firewall.

5. There are some common utilities that require the "bfp" device to function in a jail such as dhclient(8) and tcpdump(1). There is a customized ruleset number 50 that adds the "bfp" device. The first time you use the "qjail config -b 50" command it will be created automatically.

ALSO SEE

qjail(8), qjail-intro(8) qjail-drive-traffic(8), qjail-ipv6-testing

AUTHOR

Joe Barbish ⟨qjail1@a1poweruser.com⟩