Details explaining how to use qjail to build a VIMAGE
vnet jail system.
This howto will strive to explain how the qjail utility is used in a real world
application Utilizing a documentation approach that fills in the blanks of how
all the separate network functions fit together.
VIMAGE (virtualized network stack) is a highly experimental feature.
After you compile vimage into your kernel, every boot from there on will issue
this warning message as part of the boot console log messages. This is
FreeBSD's way of saying, USE IT AT YOUR OWN RISK.
It's assumed that you know how to compile your kernel. That's not going to be
covered here. With FreeBSD RELEASE 11.0, Vimage still has to be compiled into
your host's kernel before the jail(8) vnet function will work. The
"option VIMAGE" is the only statement you have to add to your kernel
definition source. Compiling in the bridge, epair, and the firewall you plan
to use is NOT necessary. The starting of the following discussion is based on
a running vimage kernel already being booted and having network access to the
public internet verified by issueing the "ping -c 2 220.127.116.11" command
on the host.
The following sequence of qjail commands were issued on a desktop type of
computer running RELEASE 11.0. This computer is a node on a small LAN hard
cabled to the gateway computer who's firewall provides the NAT services for
public internet access. The gateway is also running 11.0 and uses the ipfilter
firewall. All "ifconfig command" output shown below has had the lo0
interface details removed for brevity.
Step 1. Enter pkg install qjail
to Install the qjail software.
Step 2. Enter qjail install
to install the qjail environment.
Step 3. Enter qjail create -4 10.10.110.10 v10
To create a
directory tree type jail. This is the network ip address used
to communicate with the vnet jail. The name of the jail is v10.
Step 4. Enter qjail config -w fxp0 -v none v10
To change the
non-vnet jail into a vnet jail. fxp0 is the interface device
name connecting this computer to the LAN. You could use vtnet0
or vlan0 interface devices if so desired.
The "-v none" means this vnet jail will not be using any
firewall. The "-v value" options are none, ipfw, pf, ipf. At
this time, IE: 11.0, the ipfw firewall is the only one that
functions. The host must be using IPFW also for things to work.
Read "man qjail" config -v section for more details about
Step 5. Enter qjail start v10
Start this jail.
Step 6. Enter ifconfig -a
to see what the host's network looks like.
inet 10.0.10.7 netmask 0xfffffff0 broadcast 10.0.10.15
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active bridge10:
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddwlay
maxage 20 holdcut 5 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE>
ifmaxaddr 0 port 4 priority 128 path cost 2000
member: fxp0 flags=143<LEARNING,DISCOVER,AUTOEDGE>
ifmaxaddr 0 port 1 priority 128 path cost 200000 epair1a:
inet6 fe80::c1:ff:fe00:40a%epair1a prefixlen 64 scopeid
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
The fxp0 network device has been assigned an IP address of 10.0.10.7 by the host
system because of this ifconfig_fxp0="DHCP" statement being in the
host's rc.conf file. qjail defines it's bridge as bridge10 so hopefully it's
name is unique enough not to get steped on by some other host function. The
bridge has 2 members, one for the host's network device and one for the epair.
epair1a is on the host and its matching pair epair1b is in the vnet jail. The
number 1 is incremented by 1 for each additional vnet jail created. So if
there were two vnet jails, there also would be a epair2a member shown.
Step 7. Enter qjail list
List status of all defined jails. STATUS JID NIC
IP Jailname ------ ---- ------ --------------- -------------------------------
DRV 0 7 fxp0 10.10.110.10 v10
D means this is a directory tree type of jail R means this jail is running V
means this is a vnet jail 0 means this vnet jail has no firewall 7 means this
is the seventh time a jail has been started
Step 8. Enter qjail console v10
Auto login to this jails console. FreeBSD
11.0-RELEASE-p7 (Vimage) #0 r313172: Fri Feb 3 17:02:05 EST 2017 Welcome to
your FreeBSD jail. v10 /root >
Step 9. Enter ifconfig -a
to see what the Vnet jail network looks like.
epair1b: flags=8843<UP.BTOADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
inet 10.0.110.10 netmask 0xff000000 broadcast 10.255.255.255
inet6 fe80::c1:ff;fe00:50b%epair1b prefixlen 64 scopeid 0x2
media: Ethernet 10Gbase-T (10Dbase-T <full-duplex>)
groups: epair v10 /root >
Here we see the 10.10.110.10 ip address has been assigned to epair1b in the vnet
Step 10. Enter ping -c 2 18.104.22.168
Check to see if jail has public access.
PING 22.214.171.124 (126.96.36.199): 56 data bytes 64 bytes from 188.8.131.52: icmp_seq=0 ttl=43
time=37.337 ms 64 bytes from 184.108.40.206: icmp_seq=1 ttl=43 time=35.983 ms ---
220.127.116.11 ping statistics --- 2 packets transmitted, 2 packets received, 0.0%
packet loss round-trip min/avg/max/stddev = 35.983/36.660/37.337/0.677 ms v10
Step 11. Enter exit
to leave the jail console.
1. Cannot access the public internet. Firewall rules or NAT problem on the host
or gateway computer.
2. Cannot ping the hosts default route ip address. Most likely you don't know
what the default gateway route ip address is. On the host issue the
"route get default" command and use the gateway ip address in the
ping command issued from the vnet jails console.
3. If nothing you try seems to solve the problem, reboot the host so the host
firewall and vimage in the kernel can shake hands anew. Then most likly what
you were doing starts working. Your experiencing the highly experimental part
4. Another test you can do is try to do a ssh login on the host to the vnet
jails ip address. First you have to prepare the vnet jail. Use the console
command to login to the vnet jail. Edit the vnet jails rc.conf file adding the
sshd_enable="YES" statement. Then issue "pw adduser test -m -w
yes" to create a user account named test with password of test. Exit the
jails console. Issue "qjail restart v10" and then issue "ssh
10.10.110.10 -p 22 -l test" and see if you can access the vnet jail this
5. Be aware that a vnet jail running no firewall and the host is running ipfw
does not work in the sense that vnet jail pings to the host or public ip
addresses just time out. If the host and the vnet jail both have ipfw running
then the vnet jail will have public internet access if both the host and vnet
jail ipfw rules are correct. I suggest you start with a single ipfw rule of
"pass all" in both the host's and the vnet's ipfw rules file and
then work from there. Keep in mind that just because the vnet jail has it's
own network stack does not mean it's traffic is bypassing the hosts firewall.
Another thing to be aware of is ipfw on the host means all vnet jails have to
also be running ipfw.
6. Be aware that you can run pf or ipfilter firewalls on the host and no
firewall in the vnet jail and have conplete control of the vnet jail's network
traffic from the host's firewall.
7. Qjail does enable you to create a vnet jail using pf or ipfilter running in
it but this requires that the host is running the same firewall as the vnet
jail. Things in the vnet jail's firewall will look like it's working, but add
a deny rule for outbound port 43 to the vnet jail's firewall. Then issue a
whois command from the vnet console and you will see it work when it should
not. These two firewalls run in the hosts kernel to access the network stack
and have not been modified to work with vimage yet.
8. There are some common utilities that require the "bfp" device to
function in a jail such as dhclient(8) and tcpdump(1). There is a customized
ruleset number 50 that adds the "bfp" device. The first time you use
the "qjail config -b 50" command it will be created automatically.
qjail(8), qjail-intro(8) qjail-howto(8), qjail-ipv6-testing