Synthetic Monitoring with iontoolkit

During the initial production runs of iontk, I found new states for the devices in my customer’s fleet, and a new use for iontk.

The new path

Normal monitoring, like SolarWinds or Palo Alto’s controller, can give me stats, but iontk checks the path all the way through the device:

controller 
    |
toolkit session (requires DNS resolution from the site)
    |
ION CLI
    |
run a command from the onsite router
    |
output

Due to the error handling in iontk, I can tell how far the ION got in the process. I can fingerprint each class of failure, map it to the most likely resolutions, and automate those resolutions.

fmt.Errorf("element %s: status %s: %s", eid, resp.Status, body) // controller
fmt.Errorf("dial: %w", err)                                     // toolkit session
fmt.Errorf("waiting for login prompt: %w", err)                 // ION CLI
fmt.Errorf("waiting for password prompt: %w", err)              // ION CLI
fmt.Errorf("waiting for shell: %w", err)                        // ION CLI
fmt.Errorf("write %q: %w", cmd, err)                            // command
fmt.Errorf("read after %q: %w", cmd, err)                       // output

For example, there is a state where the ION will show online in the controller, but will not respond to the toolkit sessions and will drop pings to the management interface. This is a hung management plane, and can lead to an onsite failure. The fix is simply a remote reboot.