Managing Traffic Ops Extensions¶
Traffic Ops supports two types of extensions. ‘Check Extensions’ are analytics scripts that collect and display information as columns in the table under ‘Monitor’ -> ‘Cache Checks’ in Traffic Portal. ‘Data Source Extensions’ provide ways to add data to the graph views and usage APIs.
Check Extensions¶
Check Extensions are scripts that, after registering with Traffic Ops, have a column reserved in the “Monitor”->”Cache Checks” view (“Health”->”Server Checks” in the legacy UI) and usually run periodically using cron
. Each extension is a separate executable located in $TO_HOME/bin/checks/
on the Traffic Ops server (though all of the default extensions are written in Perl, this is in no way a requirement; they can be any valid executable). The currently registered extensions can be listed by running /opt/traffic_ops/app/bin/extensions -a
. Some extensions automatically registered with the Traffic Ops database (to_extension
table) at install time (see traffic_ops/app/db/seeds.sql
). However, cron
must still be configured to run these checks periodically. The extensions are called like so:
$TO_HOME/bin/checks/<name> -c "{\"base_url\": \",https://\"<traffic_ops_ip>\", \"check_name\": \"<check_name>\"}" -l <log level>
name: | The basename of the extension executable |
---|---|
traffic_ops_ip: | The IP address or Fully Qualified Domain Name (FQDN) of the Traffic Ops server |
check_name: | The name of the check e.g. CDU , CHR , DSCP , MTU , etc… |
log_level: | A whole number between 1 and 4 (inclusive), with 4 being the most verbose. Implementation of this field is optional |
It is the responsibility of the check extension script to iterate over the servers it wants to check and post the results. An example script might proceed by logging into the Traffic Ops server using the HTTPS base_url
provided on the command line. The script is hard-coded with an authentication token that is also provisioned in the Traffic Ops User database. This token allows the script to obtain a cookie used in later communications with the Traffic Ops API. The script then obtains a list of all caches to be polled by accessing the Traffic Ops /api/1.3/servers
REST endpoint. This list is then iterated, running a command to gather the stats from that cache. For some extensions, an HTTP GET request might be made to the Apache Traffic Server (ATS) astats
plugin, while for others the cache might be pinged, or a command might run over SSH. The results are then compiled into a numeric or boolean result and the script POSTs the result back to the Traffic Ops using the /api/1.3/servercheck/
endpoint. A check extension can have a column of ‘s and
‘s (CHECK_EXTENSION_BOOL) or a column that shows a number (CHECK_EXTENSION_NUM).A simple example of a check extension of type CHECK_EXTENSION_NUM that will show 99.33 for all servers of type EDGE is shown below:
Script here.
Currently, the following Check Extensions are available and installed by default:
- Cache Disk Usage Check (CDU)
- This check shows how much of the available total cache disk is in use. A “warm” cache should show 100.00.
- Cache Hit Ratio Check (CHR)
- The cache hit ratio for the cache in the last 15 minutes (the interval is determined by the
cron
entry). - Differential Services CodePoint Check (DSCP)
- Checks if the returning traffic from the cache has the correct DSCP value as assigned in the delivery service. (Some routers will overwrite DSCP)
- Maximum Transmission Unit Check (MTU)
- Checks if the Traffic Ops host (if that is the one running the check) can send and receive 8192B packets to the
ip_address
of the server in the server table. - Operational Readiness Check (ORT)
- See Configuring Traffic Server for more information on the ORT script. The ORT column shows how many changes the
traffic_ops_ort.pl
script would apply if it was run. The number in this column should be 0 for caches that do not have updates pending. - Ping Check - 10G, ILO, 10G6, FQDN
The
bin/checks/ToPingCheck.pl
script checks basic IP connectivity, and in the default setup it checks IP connectivity to the following:- 10G
- Is the
ip_address
(the main IPv4 address) from the server table ping-able? - ILO
- Is the
ilo_ip_address
(the lights-out-management IPv4 address) from the server table ping-able? - 10G6
- Is the
ip6_address
(the main IPv6 address) from the server table ping-able? - FQDN
- Is the Fully Qualified Domain name (the concatenation of
host_name
and.
anddomain_name
from the server table) ping-able?
- Traffic Router Check (RTR)
- Checks the state of each cache as perceived by all Traffic Monitors (via Traffic Router). This extension asks each Traffic Router for the state of the cache. A check failure is indicated if one or more monitors report an error for a cache. A cache is only marked as good if all reports are positive. (This is a pessimistic approach, opposite of how TM marks a cache as up, “the optimistic approach”)
Data Source Extensions¶
Data Source Extensions work in much the same way as Check Extensions, but are implemented differently. Rather than being a totally external executable, a Data Source Extension must be written in Perl 5, as they are injected via manipulation of the $PERL5LIB
environment variable. These extensions are not very well-documented (as you may be able to tell), and support for extending them may be phased out in future releases.
Example Cron File¶
The cron file should be edited by running crontab -e
as the traffops
user, or with sudo
. You may need to adjust the path to your $TO_HOME
to match your system.
PERL5LIB=/opt/traffic_ops/app/local/lib/perl5:/opt/traffic_ops/app/lib
# IPv4 ping examples - The 'select: ["hostName","domainName"]' works but, if you want to check DNS resolution use FQDN.
*/15 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"10G\", \"select\": [\"hostName\",\"domainName\"]}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
*/15 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"10G\", \"select\": \"ipAddress\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
*/15 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"10G\", \"name\": \"IPv4 Ping\", \"select\": \"ipAddress\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1
# IPv6 ping examples
*/15 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"10G6\", \"name\": \"IPv6 Ping\", \"select\": \"ip6Address\", \"syslog_facility\": \"local0\"}" >/dev/null 2>&1
*/15 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"10G6\", \"select\": \"ip6Address\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
# iLO ping
18 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"ILO\", \"select\": \"iloIpAddress\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
18 * * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"ILO\", \"name\": \"ILO ping\", \"select\": \"iloIpAddress\", \"syslog_facility\": \"local0\"}" >/dev/null 2>&1
# MTU ping
45 0 * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"MTU\", \"select\": \"ipAddress\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
45 0 * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"MTU\", \"select\": \"ip6Address\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
45 0 * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"MTU\", \"name\": \"Max Trans Unit\", \"select\": \"ipAddress\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1
45 0 * * * root /opt/traffic_ops/app/bin/checks/ToPingCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"MTU\", \"name\": \"Max Trans Unit\", \"select\": \"ip6Address\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1
# FQDN
27 * * * * root /opt/traffic_ops/app/bin/checks/ToFQDNCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"FQDN\"" >> /var/log/traffic_ops/extensionCheck.log 2>&1
27 * * * * root /opt/traffic_ops/app/bin/checks/ToFQDNCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"FQDN\", \"name\": \"DNS Lookup\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1
# DSCP
36 * * * * root /opt/traffic_ops/app/bin/checks/ToDSCPCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"DSCP\", \"cms_interface\": \"eth0\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
36 * * * * root /opt/traffic_ops/app/bin/checks/ToDSCPCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"DSCP\", \"name\": \"Delivery Service\", \"cms_interface\": \"eth0\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1
# RTR
10 * * * * root /opt/traffic_ops/app/bin/checks/ToRTRCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"RTR\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
10 * * * * root /opt/traffic_ops/app/bin/checks/ToRTRCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"RTR\", \"name\": \"Content Router Check\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1
# CHR
*/15 * * * * root /opt/traffic_ops/app/bin/checks/ToCHRCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"CHR\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
# CDU
20 * * * * root /opt/traffic_ops/app/bin/checks/ToCDUCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"CDU\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
# ORT
40 * * * * ssh_key_edge_user /opt/traffic_ops/app/bin/checks/ToORTCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"ORT\"}" >> /var/log/traffic_ops/extensionCheck.log 2>&1
40 * * * * ssh_key_edge_user /opt/traffic_ops/app/bin/checks/ToORTCheck.pl -c "{\"base_url\": \"https://localhost\", \"check_name\": \"ORT\", \"name\": \"Operational Readiness Test\", \"syslog_facility\": \"local0\"}" > /dev/null 2>&1