GSP
Quick Navigator

Search Site

Unix VPS
A - Starter
B - Basic
C - Preferred
D - Commercial
MPS - Dedicated
Previous VPSs
* Sign Up! *

Support
Contact Us
Online Help
Handbooks
Domain Status
Man Pages

FAQ
Virtual Servers
Pricing
Billing
Technical

Network
Facilities
Connectivity
Topology Map

Miscellaneous
Server Agreement
Year 2038
Credits
 

USA Flag

 

 

Man Pages
URLWATCH-INTRO(7) urlwatch 2.25 Documentation URLWATCH-INTRO(7)

urlwatch-intro - Introduction to basic urlwatch usage

1.
Run urlwatch once to migrate your old data or start fresh
2.
Use urlwatch --edit to customize jobs and filters (urls.yaml)
3.
Use urlwatch --edit-config to customize settings and reporters (urlwatch.yaml)
4.
Add urlwatch to your crontab (crontab -e) to monitor webpages periodically

The checking interval is defined by how often you run urlwatch. You can use e.g. crontab.guru <https://crontab.guru> to figure out the schedule expression for the checking interval, we recommend not more often than 30 minutes (this would be */30 * * * *). If you have never used cron before, check out the crontab command help <https://www.computerhope.com/unix/ucrontab.htm>.

On Windows, cron is not installed by default. Use the Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler> instead, or see this StackOverflow question <https://stackoverflow.com/q/132971/1047040> for alternatives.

Every time you run urlwatch(1), it:
  • retrieves the output of each job and filters it
  • compares it with the version retrieved the previous time ("diffing")
  • if it finds any differences, it invokes enabled reporters (e.g. text reporter, e-mail reporter, ...) to notify you of the changes

Each website or shell command to be monitored constitutes a "job".

The instructions for each such job are contained in a config file in the YAML format <https://yaml.org/spec/>. If you have more than one job, you separate them with a line containing only ---.

You can edit the job and filter configuration file using:

urlwatch --edit


If you get an error, set your $EDITOR (or $VISUAL) environment variable in your shell, for example:

export EDITOR=/bin/nano


While you can edit the YAML file manually, using --edit will do sanity checks before activating the new configuration file.

Each job must have exactly one of the following keys, which also defines the kind of job:
  • url retrieves what is served by the web server (HTTP GET by default),
  • navigate uses a headless browser to load web pages requiring JavaScript, and
  • command runs a shell command.

Each job can have an optional name key to define a user-visible name for the job.

You can then use optional keys to finely control various job's parameters.

See urlwatch-jobs(5) for detailed information on job configuration.

You may use the filter key to select one or more filters to apply to the data after it is retrieved, for example to:
  • select HTML: css, xpath, element-by-class, element-by-id, element-by-style, element-by-tag
  • make HTML more readable: html2text, beautify
  • make PDFs readable: pdf2text
  • make JSON more readable: format-json
  • make iCal more readable: ical2text
  • make binary readable: hexdump
  • just detect changes: sha1sum
  • edit text: grep, grepi, strip, sort, striplines

These filters can be chained. As an example, after retrieving an HTML document by using the url key, you can extract a selection with the xpath filter, convert this to text with html2text, use grep to extract only lines matching a specific regular expression, and then sort them:

name: "Sample urlwatch job definition"
url: "https://example.dummy/"
https_proxy: "http://dummy.proxy/"
max_tries: 2
filter:
  - xpath: '//section[@role="main"]'
  - html2text:
      method: pyhtml2text
      unicode_snob: true
      body_width: 0
      inline_links: false
      ignore_links: true
      ignore_images: true
      pad_tables: false
      single_line_break: true
  - grep: "lines I care about"
  - sort:
---


See urlwatch-filters(5) for detailed information on filter configuration.

urlwatch can be configured to do something with its report besides (or in addition to) the default of displaying it on the console.

reporters are configured in the global configuration file:

urlwatch --edit-config


Examples of reporters:

  • email (using SMTP)
  • email using mailgun
  • slack
  • discord
  • pushbullet
  • telegram
  • matrix
  • pushover
  • stdout
  • xmpp

See urlwatch-reporters(5) for reporter configuration options.

urlwatch(1), urlwatch-jobs(5), urlwatch-filters(5), urlwatch-config(5), urlwatch-reporters(5), cron(8)

2022 Thomas Perl
March 15, 2022 urlwatch 2.25

Search for    or go to Top of page |  Section 7 |  Main Index

Powered by GSP Visit the GSP FreeBSD Man Page Interface.
Output converted with ManDoc.