NAME

portscout - A tool to scan for new versions of FreeBSD ports.

SYNOPSIS

    portscout build

    while (lazy) {
        portscout rebuild
        portscout check
        portscout showupdates
    }

DESCRIPTION

Portscout is an automated system designed to search for new versions of software available in the FreeBSD ports tree. It is primarily designed for use by FreeBSD port maintainers, who can avoid trailing around dozens of websites looking for updates. However, I hope that others might find it useful too.

The current version of Portscout is also capable of checking OpenBSD's ports, NetBSD's pkgsrc, and also a generic list of software from an XML file.

SYSTEM REQUIREMENTS

The following software is required to run Portscout:

    - PostgreSQL or SQLite
    - POSIX-compatible system
    - The FreeBSD ports tree
    - Berkeley make
    - Perl 5.6+

Plus we need a few Perl modules:

    - URI
    - DBD::Pg or DBD::SQLite
    - Net::FTP
    - Proc::Queue
    - LWP::UserAgent
    - MIME::Lite
    - XML::XPath
    - XML::XPath::XMLParser
    - JSON

SQLite support is currently limited to non-forking mode only. That is, if you decide to use SQLite, Portscout will only check one port at a time; this will severely limit Portscout's speed/efficiency. It is therefore suggested that SQLite only be used for relatively light workloads.

INITIAL SET-UP

Initialise Database

The recommended database backend is PostgreSQL.

Option One: PostgreSQL

Create database:

    # createuser -U pgsql -P portscout
    # createdb -U pgsql portscout

Execute the included pgsql_init.sql script via "psql":

    # psql portscout portscout < sql/pgsql_init.sql

This will create the database tables for you.

Option Two: SQLite

Create a database file with the included script:

    # sqlite3 /var/db/portscout.db < sql/sqlite_init.sql

Configure Portscout

Review portscout.conf, and check it suits your needs. The defaults should be reasonable for most people. You can reduce "num_children" and "workqueue_size" if you don't want Portscout sucking up all your resources.

Please note that Portscout's internal defaults differ from the defaults in portscout.conf - this is because without a config file, Portscout tries to be "portable" and use its own directory for storing things under, whereas if a config file is found, it assumes it is installed and being used "system-wide".

Any of the options in portscout.conf can also be set on the fly on the command line. For example:

    $ portscout --precious_data --num_children=8

Update Ports Tree (FreeBSD Only)

Ensure your ports tree is up to date.

Populate Database

We need now to populate the database with the software we want to check.

Option One: FreeBSD

If you're using Portscout to check FreeBSD ports, run:

    $ portscout build

This takes around 70 minutes for me. Basically, Portscout is extracting all the information it needs from the ports tree. Ports (by virtue of make) is slow; the database we're building is much faster. After this initial build, we will do incremental 'builds', only updating what has changed.

Option Two: Other Software Repositories

If you would like to check another software repository/source, Portscout has several options.

Firstly, NetBSD and OpenBSD's ports trees are supported by the standard "Ports" backend. See portscout.conf for details on how to configure this. Make sure you've got the correct "make" at hand for Portscout if you're checking either of these from another operating system (e.g. FreeBSD).

Caveat: neither of the above have been well-tested, and support should probably be considered experimental.

Secondly, you can use the "XML" backend for a finite list of software that you want to manage by hand. See Portscout::DataSrc::XML for more information.

REGULAR OPERATION

Update Ports Tree (FreeBSD Only)

Ensure your ports tree is up to date.

Incremental Database Update

If your ports tree / data source was updated since your last build/rebuild, ensure Portscout knows about the changes:

    $ portscout rebuild

Run Version Checks

    $ portscout check

This will instruct Portscout to search for new distfiles for each port in the database. It will take several hours for a complete ports tree scan.

View Results

Now you've got some results, you can view them.

Option One: HTML Reports

    $ portscout generate

This will put HTML pages inside "html_data_dir" - existing pages will be deleted.

Option Two: E-Mail Reports

    $ portscout mail

This will send out an e-mail message to the maintainers of ports with updates. The e-mail messages are opt-in; you will need to add addresses to the database before any e-mails are sent out.

Option Three: Console Summary

    $ portscout showupdates

This will output a summary of software with outstanding updates. It is recommended if you're checking a limited set of software/ports.

ADDING E-MAIL ADDRESSES

If you want to send e-mail reports to maintainers of updated of software, the e-mail addresses need to be registered with Portscout. This is a safeguard to ensure no one gets e-mails they don't want.

Use the following to manage these e-mail "subscriptions":

    $ portscout add-mail dave@example.net

    $ portscout remove-mail john@localhost

    $ portscout show-mail

UPGRADING

When upgrading, check the sql directory for any relevant database schema upgrade scripts. If there were multiple schema updates between the previous version of Portscout and the version to which you have upgraded, be sure to run each script in sequence to arrive at the latest database version.

CHECKING ALGORITHM

For anyone interested in how Portscout operates, here is a high-level summary of the checking algorithm in use:

 Test 1:
   1) Order master sites using previous reliability data.
   2) Attempt to get an FTP listing or web server index from each site.
   3) Extract version from files found; compare to current version.
   4) Skip other tests if new or current version is found.

 Test 2:
   1) Increment each part of the port's version string and attempt to
      download file, e.g. for 1.4.2, try 2.0.0, 1.5.0 and 1.4.3

The last test is not yet included in Portscout, but I may add it at some point, depending on the results of testing:

 Test 3:
   1) Locate port's WWW page and spider site in an attempt to find a
      page that looks like a "download area". Scan page for possible
      links to new files.

BUGS AND LIMITATIONS

Portscout tries to make a reasonable guess when it encounters version strings in a different format to the original distname (e.g. 3.2, 3.6-pre7), but this is difficult and error-prone since vendor version schemes vary wildly.
The only real problem at the moment is version strings which seem to to count backwards (e.g. 2.11 -> 2.2).
There's some difficulty in deciding what to do with trailing zeros in version guesses. Currently, they are left intact, but this is not always going to be the right course of action. In other words, from 4.3.9, will the next major version be 4.4.0 or 4.4?
The restrict_* variables don't affect generate/mail.
Portscout doesn't handle ports with multiple distfiles very well.
At least one port (archivers/zip, as of 2010-04-28) doesn't provide a version string in the vendor's format. portscout doesn't know what to do in this case, although the version could theoretically be ascertained from the distfile name.