terminates on the aggregator. Then another multicast
stream is generated and goes into the ad inserter via
ACR01, DTR02, and ACR02. Ads are inserted into
the incoming second multicast stream, and thereafter
a third multicast stream is generated and goes into the
EQAM via ACR02 and DTR02. The third stream ter-
minates on the EQAM, which ultimately reaches the
subscriber.
2.3 Video Service Operations and
General Issues
The current state of video service operations is not
healthy, with a large number of video service impair-
ments taking a long time for troubleshooting. Accord-
ing to our survey on a cable service provider, about
60% of the incidents that occurred were due to video
processing and transport. The average duration of an
incident was 231 minutes. The video service deliv-
ery problems result in service impairments such as
blackout caused by video stream outage, video qual-
ity degradation (e.g., block noises on the screen), and
frame drops caused by packet loss or packet delivery
delay. Such problems usually originate in cable cuts
(intentional or accidental), hardware/software failure
or misconfiguration of routers, link equipment, or ser-
vice appliances such as ad-inserters, aggregators, etc.,
failed failover of routers, and router or link capacity
overload resulting from failover.
The followings are the dominant factors that lead
to a long mean time for troubleshooting (i.e. detecting
and repairing) video service delivery problems in the
routing network.
• Limited end-to-end visibility for simple flows
A CSP’s network operations center (NOC) does
not have accurate real-time, end-to-end visibility
of the network flows involved in video service de-
livery. This results in a lot of guesswork based
on sketchy information despite there being many
kinds of probes and monitoring systems in place.
• Silos (“tribal knowledge”) - no complete picture
of current/future state
Due to many factors such as complexity of the
multiple types of technology stacks (networking
gear, protocols), incomplete monitoring systems,
insufficient documentation and ongoing standard-
ization, loss/movement of technical personnel,
etc., there is no complete picture of the current
or future state of the system.
• Complex video services and routing flows
IP multicast video routing generally uses ar-
cane and complex technology with many im-
plementation variations in terms of network-
ing protocols. There are multiple tech-
nology stacks, and there are no standard
books/recipes that can help operators troubleshoot
video service delivery problems more efficiently.
Bundling/aggregating/multiplexing channels to
reduce bandwidth further complicates the picture
because internal systems do not alert the NOC
based on their internal monitoring, and usually the
first sign of trouble is customer care getting calls
from customers or getting news from social me-
dia. Furthermore, local advertisement insertion
results in even more complexity.
2.4 Problem Statement and Goals
The goal is to reduce service impairment time; which
is equal to the customer-experienced service interrup-
tion time. This will result in the improvement of cus-
tomer satisfaction as well as network operation ef-
ficiency, which means the operating team will have
more time for planning future network improvements
or considering new services less time will be spent
troubleshooting. To shorten service impairment time,
we need to reduce both the time need to identify issues
(MTTD; mean time to detect) and the time to restore
services (MTTR; mean time to restore/resolve/repair)
as described in Figure 3. We set the goal for reduc-
tion of the current service impairment time to 50%
that with our proposed solution as the cable service
provider requested.
3 CABLE NETWORK QoS
ANALYSIS SOLUTION
3.1 High-level Description
As stated in Section 2.4, the goal is to reduce
MTTD/MTTR of issues on a cable service network
with limited end-to-end visibility complex network
technologies, complex video services, and a complex
routing network. We developed a new analytics solu-
tion that supports two main use cases listed as follows.
(1) Topology Discovery and Visualization
The solution provides network topology diagram
views highlighting multicast trees of a video stream
for a specific channel (i.e., a single-program trans-
port stream) or a bundle of channels (i.e., a multiple-
program transport stream). To speed issue identifica-
tion, these views include both the multicast tree vi-
sualized on a physical topology diagram in multiple
QoS Analysis on Cable Video Delivery Networks
87