|
|
| |
SYNTH(1) |
FreeBSD General Commands Manual |
SYNTH(1) |
synth —
custom package repository builder for FreeBSD and
DragonFly
synth |
[help | configure |
version | status |
purge-distfiles] |
synth |
[rebuild-repository |
prepare-system |
upgrade-system] |
synth |
[everything |
status-everything] |
synth |
[status | build |
just-build | install |
force | test]
[file | port-origin ...] |
The synth program is an advanced concurrent ports
building tool aimed at system administrators that prefer or require the
building of packages from source rather than installing official binary
packages. synth will build packages in a clean
environment can exactly mirror the system that they are built on, it will
create local package respositories and install
pkg(8)
repository configuration file that causes locally built packages to be used
with the highest priority, all while allowing the system to fully upgraded
with a single command.
When synth is executed with the
version command, an unknown command, or no command at
all, it will display a short block of text containing an identification,
summary, version number, copyright, usage, and a summary of valid commands.
The help command displays lists each valid command with
a short synopsis of the command's function. It also elaborates the several
commands expect one or more port origins to be provided, or a path to a file
with the same information.
The configure command launches an interactive menu for
providing the user with a clean method to configure
synth . It works on the concept of profiles, and the
default profile “LiveSystem” is automatically installed with
values that are determined by scanning the system. Most of the configurable
items are directories that are used to construct the build environment, but
they aren't limited to that.
Ports
directory
- If the PORTSDIR environment variable is set when the initial profile is
created, it establishes the initial value. Otherwise,
synth will extract the value of PORTSDIR from
/usr/share/mk and use that. If neither source
points to a valid path, /usr/dports and
/usr/ports will be successfully tried. If those
don't exist either, synth will exit with an error
message.
Packages
directory
- This is the location of the local repository metadata and all the packages
that are built by
synth . It clearly must have lots
of available disk space. The default value is
/var/synth/live_packages but it should be altered
if the system has a /var partition with limited
space.
Distfiles
directory
- This indicates where all the ports source distribution files are stored,
and the initial value is determined by scanning the existing
configuration. The defaults are defined by ports and are
/usr/ports/distfiles for
(FreeBSD) and
/usr/distfiles for
(DragonFly). Most users will want to keep the
initial value as that directory should already contain distfiles.
Port
options directory
- This is the directory where all the selected options for ports are cached.
The initial value comes from a system scan, so chances are it has the
correct value. However, if the user would like a separate configuration
area for port options, they would create the new directory and set this
value accordingly (but the user would have to ensure the new location is
passed to the port when configuring port options later in that case).
Build
logs directory
- A log is produced for every port built (subsequent builds of the same port
will overwrite previous logs). This item dictates where those logs are
stored, which is /var/log/synth by default. If the
/var partition is limited, the user will
definitely want to change this location as the uncompressed logs can
potentially consume gigabyte of space.
Build
base directory
- This is the mount point for all the builders. When tmpfs is used, this
location requires very little space as it contains empty directories that
serve as mount points. If tmpfs is not used, then this location should
have access to a lot available diskspace as it will provide the work area
and virtual /usr/local for each builder. The
default location of /usr/obj/synth-live should be
fine for most users (it generally has lots of space and tmpfs is the
default build mode).
System
root directory
- Most users will use the base system as the basis for the build
environment, and thus the default setting of / is
correct. Advanced users may use DESTDIR with installworld to create an
alternative build base (e.g. i386 cross-build on amd64 architecture or a
modified base that serves as a fallout test for an upcoming feature).
Those would be the users that would modify this value.
Compiler
cache directory
- If ccache is properly installed on the system, set this value to the
location of the cache. All the builders will leverage it. It is disabled
by default. It can be disabled again by entering any invalid path as a new
value, e.g. “none”.
Num.
concurrent builders
- The represents the number of ports that can be simultaneously built on the
system. The selected value is influenced by the number of physical cores
and hyperthreads the system has, the amount of memory on the system, the
amount of available swap, and if that swap is a solid-state drive or not.
Generally memory is the limiting resource when tmpfs is used, so the
default value for the number of builders is generally 75% of the number of
CPUs of the system. The user is free to balance jobs versus builders as
well.
Max.
jobs per builder
- If memory is constrained, it's often a better performance tradeoff to
reduce the number of builders and increase the number of jobs per builder.
The default value varies depending on the system, but it will never exceed
5.
Use
tmpfs for work area
- If the initial scan shows there is at least 1.25 Gb per builder, this item
will default to “true”. Building on tmpfs is a big speed
spot and it's recommended if the memory resources allow it. This item is
directed at the work area for building the port
Use
tmpfs for /usr/local
- Before a port can start building, all the build and library dependencies
have to be installed into the localbase. Extracting, installing and later
deinstalling is a lot of work for a disk, so using tmpfs here is another
performance boost. It is recommended that users with adequate memory leave
this enabled.
Display
using ncurses
- During the concurrent building, the default display is constructed using
ncurses. It shows a lot of good information and it should be used. The
display will fall back to a text mode if the terminal doesn't support it.
This setting forces text mode unconditionally, but few users will want or
need to use this option.
Fetch
prebuilt packages
- When active, this option will scan the external repository for suitable
prebuilt packages to satisfy build requirements. To be considered
suitable, the ABI, options, and dependencies of the remote package must
all match requirements. If a package is located, it will be fetched and
placed in the packages directory and later incorporated into the local
repository. This option is not active by default, meaning that
synth will normally build everything from source
and ignore external repositories.
To create alternative profiles, press the “>” key
at the prompt. A new menu will appear that provides options to switch to
another existing profile or create a new one that can be tailored and named.
Any newly created profile becomes the new default profile, but this can be
easily changed by repeating the profile selection process.
Pressing the Alphabetic key associated with each configuration
item brings up a prompt to change the value. Directory entries must be valid
paths except in the case of ccache. The boolean values are modified with a
single keypress (“T” or “F”) and the numeric
items require positive integers. After changing the values, the old menu
returns and altered values are marked with an asterisk. Simply entering the
“Enter” key (carriage return) saves the changes while pressing
the “Escape” key will discard them.
If more than one profile exists, an option to delete profiles will
appear on the menu. By pressing the “<” key, a new menu
will appear that lists all the inactive profiles. Entering the number
associated with one of those profiles will immediately delete the profile.
The deletion cannot be undone.
The upgrade-system command automates the full upgrade of
the system upon which synth is installed. It is
expected that the ports tree has been updated to the latest version before the
command is executed. Using the ports tree as a reference,
synth will query
pkg(8) to
determine what is currently installed and which software has newer versions
available. It will rebuild those packages and any package that depends
libraries they contain or that have a runtime dependency on the rebuild
packages. In addition, any port that has changed with respect to its options,
dependencies, or ABI will be rebuilt as well. After analysis, the concurrent
builder will begin building the set of packages that have been identified as
requiring a rebuild. When that phase is complete, the local repository will be
rebuilt without waiting for confirmation. Finally,
synth will command
pkg(8) to
upgrade the system using the local repository, effectively bringing it
completely up to date.
The prepare-system command has the same functionality of
the upgrade-system command with the exception that the
packages will not be installed. Once the local repository is fully rebuilt,
the command exits.
The rebuild-repository command will perform a full
sanity check on all packages present in the packages directory and remove the
ones which fail the check. The local Synth repository will be created (or
recreated) using the remaining packages. This step is also done as part of
several other commands, so it mainly exists for scripting purposes.
The status command with no arguments performs a dry-run
of the upgrade-system command. It will not delete
obsolete packages, nor rebuild the local repository. It will list all the
ports that will be rebuild along with a total, and it also logs the same
information to /var/synth/synth_status_results.txt
since the full list is often longer than the terminal height.
The just-build command takes a list (one or more) of
port origins (an origin is the combination of a category and the port
directory, e.g. lang/python27) and it will builds the packages of those ports
if necessary. If analysis determines the packages are already up to date,
nothing is done.
This is the first of several commands that take arguments. The
arguments are either an unlimited number of port origins, e.g.
synth just-build editors/joe editors/nano editors/libreoffice
or they are limited to one, a path to a file, e.g.
synth just-build /tmp/build.list
A file that is equivalent to the first example would contain:
editors/joe
editors/nano
editors/libreoffice
When the building task is complete, synth exits.
The build command differs from the
just-build command by asking the user if they wish to
rebuild the local repository when the building is complete. The answer is
usually “N” (for “no”) because rebuilding the
repository can take a few minutes to complete, and it only makes sense to do
when all the building is complete. Should the user opt to rebuild the
repository, synth will then ask if the user wishes to
install the packages that come from the ports list on the system.
The install command is similar to the
build command except that it will not ask permission
to rebuild the repository or install the resulting packages. It will just do
it all in sequence.
The force command is similar to the
build command except that any packages related to the
ports list will be deleted first, regardless if they are up to date or not.
This results in that every port on the given list will build. As with the
build command, synth will
request permission to rebuild the repository and possibly install the packages
when the building phase is complete.
The test command will pre-delete any existing packages
related to the ports list before rebuilding them with extra DEVELOPER_MODE
tests and settings. When the building is complete, it will just exit without
offering to rebuild the repository.
This version of the status command provides a dry-run
and reports on what would happen if the just-build
command was executed with the same arguments. The output and logging is
similar to the solo status command, but the build
scope is limited to the given ports list.
The everything command is not meant for most users. It
will attempt to build every port in the ports tree, which can take several
days depending on the power and resources of the building machine. Even
downloading the tens of gigabytes worth of distfiles consumes significant
time!
The status-everything command is just a dry-run for the
everything command. Since
synth works through increment building (that is, it
will not build a port if an up to date package already exists for it and all
the packages dependencies are also in place) , the resultant build list can be
much shorter than the full list of ports in the tree. Only those rare users
that wish to build the entire set of ports would use this command.
This is a useful command that will first scan the entire ports tree to gather a
complete list of distfiles used by the ports tree, and then it will scan the
contents of the configured distfiles directory. All source distribution files
that cannot be matched to at least one port in the tree will be removed. This
can result in gigabytes of recovered disk space if significant time has passed
since the distfiles directory was last cleaned.
PORTSDIR
- The
PORTSDIR environment variable is only checked
on the very first signficant command of synth
(meaning everything except help and version) when the default profile is
created. It is also checked when adding new profiles. The variable
dictates the default location of the ports tree with a higher priority
than the standard locations.
SYNTHPROFILE
- When this environment variable is set to name of an existing
synth profile, that profile will be loaded when
synth is launch rather than the active profile. It
is a temporary override; removing the variable will allow the default
profile to load again.
WHYFAIL
- When this environment variable is defined to any value, the sanity check
routines will provide more detail on dependency and option check failures
on a separate output line.
ENTERAFTER
- This environment variable only applies with the
test command, and then only when a single port is
specified. If both of those conditions are satisfied, and the variable has
the value of:
- extract
- patch
- configure
- build
- stage
- install
- deinstall
then an interactive build will be attempted. First, all the port's
prerequisites are built normally and synth will
return to text mode. If everything built successfully, the specified port
will build up to and including the point specified by ENTERAFTER. At that
point, control will be given to the user by providing them with a tcsh
shell with the current directory set at the root of the builder. When the
user is done, they should terminate the shell with the
exit command to signal
synth to clean up and terminate. This is a
developer tool to troubleshoot port build problems and it is not intended
for the average user.
LOCK
- This environment variable only applies with the
test command. When it is defined to any value, the
builder's localbase is remounted read-only prior to the configure phase
and restored to read/write permissions after building is complete. This is
a diagnostic tool aimed at pinpointing file system violations as a port is
not supposed to write to localbase prior to staging. Note that the
localbase tmpfs setting will considered as False in this mode, so building
may be noticeably slower than normal.
SELFTEST
- This environment variable only applies with the
test command. When it is defined, the
“test” target is executed as part of the build cycle; this
target is not executed under any other circumstances.
The following files may be used or created:
- synth.ini
- This is the configuration file. It is automatically generated and
manipulated using the interactive configuration menu. There is no need to
touch this file other than to remove unwanted profiles as currently that
is not possible from the program itself. It is normally located at
/usr/local/etc/synth/synth.ini
- <profile>-make.conf
- This is an optional, user-provided file. If it exists, the builder's
/etc/make.conf will be appended with the contents
of this file. For the default profile, the file would normally be located
at /usr/local/etc/synth/LiveSystem-make.conf
- <profile>-environment
- This is an optional, user-provided file. It contains a list of name-value
pairs joins with an equals sign (e.g. HTTP_PROXY=http://proxyserver:8888),
one pair per line. If the file exists, these variables will be defined in
the builder environment. For the default profile, the file would normally
be located at
/usr/local/etc/synth/LiveSystem-environment
- <profile>-private.key
- This is one of two files required to support the signing of the repository
by a local RSA key. It is the output of the
openssl
genrsa command; see the EXAMPLE section for how to generate it. The
file permissions are required to be set at 400 as well. For the default
profile, the file is normally located at
/usr/local/etc/synth/LiveSystem-private.key
- <profile>-public.key
- This is the second of two files required to support the signing of the
repository by a local RSA key.
synth must confirm
the public counterpart of the signing key is available because its path is
required for the generated pkg repository configuration file. For the
default profile, the file is normally located at
/usr/local/etc/synth/LiveSystem-public.key
- <profile>-signing_command
- This is one of two files required to support the signing of the repository
by an external signing server. It consists of a single line containing the
command; see the EXAMPLE section. For the default profile, the file is
normally located at
/usr/local/etc/synth/LiveSystem-signing_command
- <profile>-fingerprint
- This is the second of two files required to support the signing of the
repository by an external signing server. It consists of a single line
containing the SHA 256 hash of the public key that is the counterpart of
the key that signed the repository. For the default profile, the file is
normally located at
/usr/local/etc/synth/LiveSystem-fingerprint
- <profile>-index
- The introduction of flavors within FreeBSD ports destroyed the 1:1
relationship between port directories and packages. Now any port can
generated any number of packages. This requires scanning every port in the
tree to determine the true origin range. This is a time consuming task, so
Synth caches the scan results and reuses it as long as all ports files are
older than the generated index. When Synth detects the index is
out-of-date or if it doesn't exist, it is automatically generated. One
index exists for each profile, with the default profile index normally
located at /var/cache/synth/LiveSystem-index
- synth_status_results.txt
- Whenever a status command is run, a list of ports that would build with
the actual command is produced at
/var/synth/synth_status_results.txt
- synth_prefetch_list.txt
- Whenever a status command is run, and the option to use suitable prebuilt
packages has been selected, a list of packages that will be fetched as a
result will be written to
/var/synth/synth_prefetch_list.txt
- index.html
- For every build run, Synth installs a web summary at the Report
subdirectory of the build logs directory. For the default profile, the
file would be located at
/var/log/synth/Report/index.html. To remotely view
the web report with a browser, a web server must be present and configured
to expose the logs directory. The web report is dynamic and updated 10
times per minute. The complete history is available and searchable, with
quick-filters on "Built", "Failed",
"Ignored" and "Skipped" fields as well as on the
"No." column which is useful to examine cascaded skips. The
search filter can be quickly cleared by clicking on the "Total"
field.
- port logs
- Every single attempt at building a port results in a build log. The name
of the log is always in the form of [category]___[port].log so that
subsequent builds will overwrite previous attempts. They are created in
the directory specified by the configuration.
- hook_run_start
- If this file exists, and it is executable (or it is a symbolic link that
leads to an executable file) then it will be executed by
synth at the start of the build. The same
requirement exists for all hooks. This hook, and the next five hooks all
define the following common environment variables:
PROFILE , DIR_PACKAGES ,
DIR_REPOSITORY , DIR_PORTS ,
DIR_OPTIONS ,
DIR_DISTFILES , DIR_LOGS ,
DIR_BUILDBASE . Uniquely, this hook also defines
PORTS_QUEUED which is the number of ports that
synth will attempt to build. This file must be
colated with synth.ini, so that would be
/usr/local/etc/synth/hook_run_start for most
installations.
- hook_run_end
- This hook is fired when the build completes. In addition to the common
definitions, it defines
PORTS_BUILT ,
PORTS_FAILED ,
PORTS_IGNORED , and
PORTS_SKIPPED in the environment, which represents
the results of the build.
- hook_pkg_success
- This hook is fired whenever a builder successfully packages a port. In
addition to the common definitions, it defines
RESULT=success , ORIGIN ,
and PKGNAME in the environment.
- hook_pkg_failure
- This hook is fired whenever a builder fails to package a port. In addition
to the common definitions, it defines
RESULT=failure , ORIGIN ,
and PKGNAME in the environment.
- hook_pkg_ignored
- This hook is fired prior to the start of a build when
synth prescans the queue for ports that are to be
ignored according to the ports tree. Once the build has started, this hook
is never fired again. In addition to the common definitions, it defines
RESULT=ignored , ORIGIN ,
and PKGNAME in the environment.
- hook_pkg_skipped
- This hook is fired whenever a package gets ignored or fails which in turns
cascades into a number of ports getting removed from the build queue due
to the dependency's failure. This may fire before the build starts due to
ignored prescanning, or any time during the build. In addition to the
common definitions, it defines
RESULT=skipped ,
ORIGIN , and PKGNAME in the
environment.
During development, the interrupt signal (SIGINT) was captured and handled. It
turns out this handling extended to the builders, so any event resulting in a
SIGINT in the build would lock up the builder permanently. There's no apparent
workaround for this given the design, so pressing Control-C is no longer
handled. It will stop the build, but it will leave everything in bad state.
The next significant call to synth will attempt to
clean that up though. If a user wants to stop the build, the best approach is
to type the Control-Q combination. The program will shut down as soon as it
can. During the building phase no new builds will start after the shutdown
begins, but the active builders will continue until completion.
At this time, synth can only be run by the root user.
A common question is "What does the Impulse indicator on the display
mean?" It is the package build rate (packages/hour) over the last 500
seconds. For the first roughly ten minutes of a bulk run, the
“Impulse” rate and the “Pkg/Hour” rate will be
identical. After that, the “Impulse” build rate will differ from
the rate averaged over the entire elapsed time, often by a great amount. It's
just a way to gauge the package production rate over the previous few minutes.
Synth is written in Ada, and thus an Ada compiler or cross-compiler is required
to build it. For FreeBSD, that means
synth is currently limited to amd64 and i386. It is
feasible to port GNAT to the ARMv7 architecture as this has already been done
(See lang/gnatdroid-armv7 port) but it's not a trivial task to do it. Other
architectures would be challenging, although not impossible, to support.
To activate the signing of default profile's repository with a local RSA key,
generate the two key files:
% cd /usr/local/etc/synth
% openssl genrsa -out LiveSystem-private.key 2048
% chmod 0400 LiveSystem-private.key
% openssl rsa -in LiveSystem-private.key -out \
LiveSystem-public.key -pubout
To activate the signing of default profile's repository with an
external signing server, two files need to be created. It's assumed the
signing server has already been configured as described by the
pkg-repo(8)
man page, and that the keys are located in the repo.pub and repo.key files.
On the signing server which is 192.168.0.44 in this example:
% sha256 -q repo.pub > LiveSystem-fingerprint
% echo "/usr/bin/ssh 192.168.0.44 /usr/local/bin/sign.sh" \
> LiveSystem-signing_command
Then move both files to /usr/local/etc/synth directory on the
build server. The build server root user needs to have a passphrase-free SSH
key access to the signing server in order for Synth to sign the
repository.
synth was conceived long before coding began in December
of 2015. It was designed to be system-agnostic from the beginning, and in
particular support for FreeBSD and
DragonFly was added simulaneously. It was hoped that
it would attract users of the older PortMaster and PortUpgrade ports
management tools by providing them with a superior and well-maintained
alternative.
synth was written entirely by
John Marino ⟨marino@FreeBSD.org⟩ and
released under the Internet Software Consortium license.
Visit the GSP FreeBSD Man Page Interface. Output converted with ManDoc. |