Traffic Ops

Introduction

Traffic Ops uses a PostgreSQL database to store the configuration information, and the Mojolicious framework to generate the user interface and REST APIs.

Software Requirements

To work on Traffic Ops you need a CentOS 7+ environment that has the following installed:

  • Carton 1.0.12

    • libpcap (plus development library - usually “libpcap-dev” or “libpcap-devel”)
    • libpq (plus development library - usually “libpq-dev” or “libpq-devel”)
    • cpan JSON
    • cpan JSON::PP
  • Go 1.8.3

  • Perl 5.10.1

  • Git

  • PostgreSQL 9.6.6

  • Goose

Traffic Ops Project Tree Overview

traffic_ops/ - The root of the Traffic Ops project

  • app/ - Holds most of the Perl code base

    • bin/ - Directory for scripts, cronjobs, etc

    • conf/

      • development/ - Development (local) specific configuration files.
      • misc/ - Miscellaneous configuration files.
      • production/ - Production specific configuration files.
      • test/ - Test (unit test) specific configuration files.
    • db/ - Database related area.

      • migrations/ - Database Migration files.
    • lib/

      • API/ - Mojo Controllers for the /API area of the application.

      • Common/ - Common Code between both the API and UI areas.

      • Extensions/ - Contains Data Source Extensions

      • Fixtures/ - Test Case fixture data for the ‘to_test’ database.

        • Integration/ - Integration Tests.
      • MojoPlugins/ - Mojolicious Plugins for Common Controller Code.

      • Schema/ - Database Schema area.

        • /Result - DBIx ORM related files.
      • /Test - Common Test.

      • UI/ - Mojolicious Controllers for the Traffic Ops UI itself.

      • Utils/

        • Helper/ - Common utilities for the Traffic Ops application.
    • log/ - Log directory where the development and test files are written

    • public/

    • css/ - Stylesheets
    • images/ - Images
    • js/ - Javascripts
    • script/ - Mojo Bootstrap scripts.
    • t/ - Unit Tests for the UI.
    • api/ - Unit Tests for the API.
    • t_integration/ - High level tests for Integration level testing.
    • templates/ - Mojo Embedded Perl (*.ep) files for the UI.
  • bin/ - holds executables related to Traffic Ops, but not actually a part of the Traffic Ops server’s operation

  • build/ - contains files that are responsible for packaging Traffic Ops into an RPM file

  • client/ - API endpoints handled by Go

  • client_tests/ - lol

  • doc/ - contains only a coverage-zone.json example (?) file

  • etc/ - configuration files needed for the Traffic Ops server

    • cron.d/ - holds specifications for Cron jobs that need to be run periodically on Traffic Ops servers
    • init.d/ - contains the old initscripts-based job control for traffic ops
    • logrotate.d/ - specifications for the Linux logrotate utility for Traffic Ops log files
    • profile.d/traffic_ops.sh - sets up common environment variables for working with Traffic Ops
  • experimental/ - includes all kinds of prototype and/or abandoned tools and extensions

    • ats_config/ - an attempt to provide an easier method of obtaining and/or writing configuration files for Apache Trafficserver cache servers
    • auth/ - a simple authentication server that mimics the authentication process of Traffic Ops, and provides a detailed view of a logged-in user’s permissions and capabilities
    • goto/ - an Angular (1.x) web page backed by a Go server that provides a ReST API interface for mySQL servers
    • postgrest/ - originally probably going to be a web server that provides a ReST API for postgreSQL servers, this only contains a simple - albeit unfinished - Docker container specification for running postgreSQL client tools and/or server(s)
    • server/ - a living copy of the original attempt at re-writing Traffic Ops in Go
    • traffic_ops_auth/ - proof-of-concept for authenticating, creating and deleting users in a Traffic Ops schema.
    • url-rewriter-nginx/ - Docker container specification for a modification to the NginX web server, meant to make it suitable for use as a caching server at the Edge-tier or Mid-tier levels of the Traffic Control architecture
    • webfront/ - a simple HTTP caching server written from the ground-up, meant to be suitable as a caching server at the Edge-tier or Mid-tier levels of the Traffic Control architecture
  • install/ - contains all of the resources necessary for a full install of Traffic Ops

    • bin/ - binaries related to installing Traffic Ops, as well as installing its prerequisites, certificates, and database
    • data/ - almost nothing
    • etc/ - this directory left empty; it’s used to contain post-installation extensions and resources
    • lib/ - contains libraries used by the various installation binaries
  • testing/ - holds utilities for testing the Traffic Ops API, as well as comparing two separate API instances (for e.g. comparing a new build to a known-to-work build)

  • traffic_ops_golang/ - has all of the functionality that has been re-written from Perl into Go

  • vendor/ - contains “vendored” packages from third party sources

Perl Formatting Conventions

Perltidy is for use in code formatting.

Database Management

The app/db/admin.pl script is for use in managing the Traffic Ops database tables. This essentially serves as a front-end for Goose Below is an example of its usage.

Note

For proper resolution of Perl library paths and SOL statement files, it’s recommended that this script be run from the app directory

db/admin.pl [options] command

Options:

--env An environment specification. One of development, integration, production, or test. Default is development.

Note

The $MOJO_MODE environment variable is set to the value of the environment

Commands:

createdb
Creates the database for the current environment
create_user
Creates the user defined for the current environment
dbversion
Displays the database version that results from the current sequence of migrations
down
Rolls back a single migration from the current version
drop
Drops the database for the current environment
drop_user
Drops the user defined for the current environment
load_schema
Sets up the database for the current environment according to the SQL statements in app/db/create_tables.sql
migrate
Runs a migration on the database for the current environment
patch
Patches the database for the current environment using the SQL statements from the app/db/patches.sql
redo
Rolls back the most recently applied migration, then run it again
reset
Creates the user defined for the current environment, drops the database for the current environment, creates a new one, loads the schema into it, and runs a single migration on it
reverse_schema
Reverse engineers the app/lib/Schema/Result/* files from the environment database
seed
Executes the SQL statements from the app/db/seeds.sql file for loading static data
show_users
Displays a list of all users registered with the PostgreSQL server
status
Prints the status of all migrations
upgrade
Performs a migration on the database for the current environment, then seeds it and patches it using the SQL statements from the app/db/patches.sql file

The environments are defined in the app/db/dbconf.yml file, and the name of the database generated will be the name of the environment for which it was created. To use the admin.pl script, you may need to add traffic_ops/lib and traffic_ops/local/lib/perl5 to your PERL5LIB environment variable.

Installing The Developer Environment

To install the Traffic Ops Developer environment:

  1. Clone the Traffic Control repository from GitHub.

  2. Install the local dependencies using Carton.

  3. Set up a role (user) in PostgreSQL

  4. Use the setup command of the admin.pl script (see Database Management for usage) to set up the traffic_ops database(s).

    Example Output:

    $ db/admin.pl --env=development setup
    Using database.conf: conf/development/database.conf
    Using database.conf: conf/development/database.conf
    Using database.conf: conf/development/database.conf
    Using database.conf: conf/development/database.conf
    Using database.conf: conf/development/database.conf
    Using database.conf: conf/development/database.conf
    Executing 'drop database to_development'
    Executing 'create database to_development'
    Creating database tables...
    Warning: Using a password on the command line interface can be insecure.
    Migrating database...
    goose: migrating db environment 'development', current version: 0, target: 20150210100000
    OK    20141222103718_extension.sql
    OK    20150108100000_add_job_deliveryservice.sql
    OK    20150205100000_cg_location.sql
    OK    20150209100000_cran_to_asn.sql
    OK    20150210100000_ds_keyinfo.sql
    Seeding database...
    Warning: Using a password on the command line interface can be insecure.
    
  5. (Optional) To load the ‘KableTown’ example/testing data set into the tables, use the setup_kabletown.pl script located in app/bin/db/.

    Note

    To ensure proper paths to Perl libraries and resource files, the setup_kabletown.pl should be run from within the app/ directory.

  6. Run the postinstall script, located in install/bin/

  7. To start Traffic Ops, use the start.pl script located in the app/bin directory. If the server starts successfully, the STDOUT of the process should contain the line [<date and time>] [INFO] Listening at "http://*:3000", followed by the line Server available at http://127.0.0.1:3000 (using default settings for port number and listening address, and where <date and time> is an actual date and time in ISO format).

    Note

    To ensure proper paths to Perl libraries and resource files, the start.pl script should be run from within the app/ directory.

  8. Using a web browser, navigate to the given address: http://127.0.0.1:3000

  9. A prompt for login credentials should appear. Assuming default settings are used, the initial login credentials will be

    User name:admin
    Password:password
  10. Change the login credentials.

Test Cases

Use prove (should be installed with Perl) to execute test cases. Execute after a carton install of all required dependencies:

  • To run the Unit Tests: prove -qrp  app/t/
  • To run the Integration Tests: prove -qrp app/t_integration/

Note

As progress continues on moving Traffic Ops to run entirely in Go, the number of passing tests has increased. This means that the tests are not a reliable way to test Traffic Ops, as they are expected to fail more and more as functionality is stripped from the Perl codebase.

The KableTown CDN example

The integration tests will load an example CDN with most of the features of Traffic Control being used. This is mostly for testing purposes, but can also be used as an example of how to configure certain features. To load the KableTown CDN example and access it:

  1. Be sure the integration tests have been run

  2. Start the Traffic Ops server. The MOJO_MODE environment variable should be set to the name of the environment that has been loaded.

  3. Using a web browser, navigate to the address Traffic Ops is serving, e.g. http://127.0.0.1:3000 for default settings

  4. For the initial log in:

    User name:admin
    Password:password

Extensions

Traffic Ops Extensions are a way to enhance the basic functionality of Traffic Ops in a custom manner. There are three types of extensions:

Check Extensions
These allow you to add custom checks to the “Monitor”->”Cache Checks” view.
Data Source Extensions
These allow you to add statistic sources for the graph views and APIs.

Extensions are managed using the $TO_HOME/bin/extensions command line script

See also

For more information see Managing Traffic Ops Extensions.

Extensions at Runtime

The search path for Data Source Extensions depends on the configuration of the PERL5LIB environment variable, which is pre-configured in the Traffic Ops start scripts. All Check Extensions must be located in $TO_HOME/bin/checks

Example PERL5LIB Configuration
export PERL5LIB=/opt/traffic_ops_extensions/private/lib/Extensions:/opt/traffic_ops/app/lib/Extensions/TrafficStats

To prevent Data Source Extension namespace collisions within Traffic Ops all Data Source Extensions should follow the package naming convention ‘Extensions::<ExtensionName>

TrafficOpsRoutes.pm

Traffic Ops accesses each extension through the addition of a URL route as a custom hook. These routes will be defined in a file called TrafficOpsRoutes.pm that should be present in the top directory of your Extension. The routes that are defined should follow the Mojolicious route conventions.

Development Configuration

To incorporate any custom Data Source Extensions during development set your PERL5LIB environment variable with any number of colon-separated directories with the understanding that the PERL5LIB search order is from left to right through this list. Once Perl locates your custom route or Perl package/class it ‘pins’ on that class or Mojolicious Route and doesn’t look any further, which allows for the developer to override Traffic Ops functionality.