(12) United States Patent Luft (45) Date Of Patent: May 18, 2010

US 7,719,995 B2 Page 3 “CloudShield PacketWorks IDE: Developers' Toolkit for CS-2000 In-Network Computing Platform.” 2005. retrieved Feb. 28, 2007). Retrieved from the Internet: CloudShield Technologies, Inc., www. cloudshield.com. CloudShield, 212 Gibraltar Drive, Sunnyvale, California 94089. “CS-2000 Content Processing Platform: Multi-Function Systems for Controlling and Securing Next Generation Services.” 2006. retrieved Feb. 28, 2007. Retrieved from the Internet: CloudShield Technologies, Inc., www.cloudshield.com. CloudShield, 212 Gibraltar Drive, Sunnyvale, California 94089. “Efficient Network Resource Control—A Source of Competitive Advantage.” Sep. 2005. Operax, Tegeluddsvägen 92, SE-115 28, Stockholm, Sweden. “Ellacoya Introduces VolP Quality Reporter (VQR).” online Ellacoya Networks, Inc., 2007. retrieved Apr. 19, 2007 Retrieved from the Internet: URL: http://www.ellacoya.com/products/var. shtml . “Ellacoya Multimedia Service Manager (MSM) PacketCable Multi media-based Application” online Ellacoya Networks, Inc., 2007. retrieved Apr. 19, 2007 Retrieved from the Internet: URL: http:// www.ellacoya.com/products/multimediaMgr.shtml . “Ellacoya's Scalable Distributed Architecture.” online Ellacoya Networks, Inc., 2007. retrieved Apr. 19, 2007 Retrieved from the Internet: URL: http://www.ellacoya.com/products/architecture. com. Ellacoya Networks, 7 Henry Clay Drive, Merrimack, NH03054 USA. “Managed DDoS Security Services: An Arbor Networks/ CloudShield Solution.” Aug. 2004. CloudShield Technologies, Inc., www.cloudshield.com. CloudShield, 212 Gibraltar Drive, Sunny vale, California 94089. “Meeting the Multi-Service IP QoS Challenges.” 2005. retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92, SE-1 1528, Stockholm, Sweden. “On the Road to IP Multi-Service.” 2005. retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92, SE-1 1528, Stockholm, Sweden. “Operax Bandwidth Manager 5500: QoS control for the access, aggregation and edge.” 2006. retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92. SE-1 1528, Stockholm, Sweden. “Operax Bandwidth Manager 5700: QoS control for IP/MPLS Net works.” 2006. retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92, SE-1 15 28, Stockholm, Sweden. “Operax guarantees QoS for Video Telephony. QoS guaranteed value added services.” 2005. retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92, SE-1 1528, shtml . Stockholm, Sweden. “Guarantee QoS for IPTV with Operax Bandwidth Manager.” retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax. com. Operax, Tegeluddsvägen 92, SE-1 1528, Stockholm, Sweden. “Operax in IP Multimedia Subsystem (IMS): Advanced QoS control for IMS/NGN.” 2006. retrieved Apr. 19, 2007. Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92, SE-1 1528, 2006. Stockholm, Sweden. “IMS and Operax Bandwidth Manager—Guaranteed QoS.” retrieved Apr. 19, 2007). 2006. Retrieved from the Internet: www. operax.com. Operax, Tegeluddsvägen 92, SE-1 1528, Stockholm, “Operax Triple Play: Service Control System.” 2005. retrieved Apr. 19, 2007). Retrieved from the Internet: www.operax.com. Operax, Tegeluddsvägen 92, SE-1 1528, Stockholm, Sweden. Sweden. “Prolexic DNS Shield on CS-2000: Prolexic and CloudShield Inte "Integrating Applications with the CloudShield ONSP” Aug. 2004. grated Solution for DNS Service Protection.” 2005. retrieved Feb. 28, 2007. Retrieved from the Internet: CloudShield Technologies, CloudShield Technologies, Inc., www.cloudshiled.com. CloudShield, 212 Gibraltar Drive, Sunnyvale, California 94089. “Integrated Content Control for An All IP World.” retrieved Feb. 28, 2007. Retrieved from the Internet: CloudShield Technologies, Inc., www.cloudshiled.com. CloudShield, 212 Gibraltar Drive, Sunny vale, California 94089. “IP Service Control System.” 2005. retrieved Apr. 19, 2007 Retrieved from the Internet: Ellacoya Networks, Inc., www.ellacoya. Inc., www.cloudshield.com. CloudShield, 212 Gibraltar Drive, Sunnyvale, California 94089. “A Unified Policy Management Framework: IPService Creation and Assurance in Converged Networks.” Nov. 2006. Operax, Tegeluddsvägen 92, SE-1 1528, Stockholm, Sweden. * cited by examiner

U.S. Patent May 18, 2010 Sheet 1 of 15 100 Y APPLICATIONLAYER (LAYER 7) 135 PRESENTATIONLAYER (LAYER6) 130 SESSION LAYER (LAYER5) 125 TRANSPORTLAYER (LAYER4) 120 NETWORKLAYER (LAYER3) 115 DATALINKLAYER (LAYER2) 110 PHYSICALLAYER (LAYER 1) 105 FIG. 1 (PRIOR ART) US 7,719,995 B2

U.S. Patent May 18, 2010 UNICAST CONTENT FLOW -o- UNICAST CONTROL FLOW - --- Sheet 2 of 15 US 7,719,995 B2 2 05 CONTENT 3E is E/ PROVIDER EEsi 210 H EROUTER 220 a CLENT SUBSCRIBER CLENT SUBSCRIBER Ya O N O O CLIENT SUBSCRIBER 205 FIG. 2 (PRIOR ART)

U.S. Patent May 18, 2010 UNICAST CONTENT FLOW -o- UNICAST CONTROL FLOW - --- Sheet 4 of 15 US 7,719,995 B2 CLIENT SUBSCRIBER (A) CONTENT PROVIDER CLIENT SUBSCRIBER 410 CLIENT SUBSCRIBER (N) 415 FIG. 4

U.S. Patent May 18, 2010 Sheet 5 of 15 US 7,719,995 B2 500 505 1st CLENT SUBSCRIBER ISSUES UNICAST REQUEST FOR CONTENT FROM CONTENT PROVIDER (1ST UNICAST CTRL FLOW) 510 APPROUTER INTERCEPTS AND INTERPRETS 1ST UNICAST REQUEST 515 520 APPROUTER FORWARDS 1ST UNICAST REQUEST TO CONTENT PROVIDER APP ROUTER INSPECTS UNICAST TRAFFIC FROM OTHER CLIENT SUBSCRIBERS TOIDENTIFY SUBSEQUENT REQUESTS FOR SAME CONTENT FROM OTHER CLIENT SUBSCRIBERS 530 525 FORWARD TRAFFIC TOY NO DESTINATION SMLAR UNICAST ROST? YES 535 QUEUE REDUNDANT REQUESTS 540 CONTENT ARRIVES 545 YES FORWARD UNICAST CONTENT FLOW TO 1ST CLIENTSUBSCRIBER 550 REPLICATE UNICAST CONTENT FLOW FROM CONTENT PROVIDER TO OTHER CLIENTSUBSCRIBERS FIG. 5 M

U.S. Patent May 18, 2010 Sheet 6 of 15 US 7,719,995 B2 9 NEITO B(EGI8NOS)G NEITO X7NIT VO NOIS E XHONIEN T;TWOISAHd *1IMSN OWE T—H 1T—»MSOWHIT NOHf]

U.S. Patent May 18, 2010 Sheet 10 of 15 US 7,719,995 B2 1000 1005 M CLIENT CONSUMERS RECEIVE INSERTION POINT TRIGGER 1010 APP ROUTER RECEIVES GMP DEPART REQUESTS TOEXIT MULTICAST SESSION FORCURRENT CHANNEL 1015 APP ROUTER RECEIVES 1ST UNICAST REQUEST FORSKF OF COMMERCIAL CHANNEL 1020 APP ROUTER FORWARDS 1ST CTRL SIGNAL REQUEST TO REGIONALLOCAL SERVER 1025 APP ROUTERIDENTIFIES & QUEUES SUBSEQUENT REDUNDANT REQUESTS FROM OTHER CLIENT CONSUMERS 1030 APPROUTER RECEIVES UNICAST CONTENTFLOW WITHSKF FROM REGIONALLOCAL SERVER 1035 1040 1045 APP ROUTER REPLICATES UNICAST CONTENT FLOW FOR ALL QUEUED REQUESTS APP ROUTER TRANSMITS REPLICATED UNICAST CONTENT FLOWS TO EACH CLIENT CONSUMER APPROUTER RECEIVESIGMP JOIN REQUESTS FROM CLIENT CONSUMERS TO JOIN MULTICAST BROADCAST FIG. 10

U.S. Patent May 18, 2010 Sheet 11 of 15 US 7,719,995 B2 SERVICE PROVIDER SUBSCRIBER SUBSCRIBER POLICY 1108A 1104A MANAGER(S) 1110 SUBSCRIBER 1108B SERVICE PROVIDER 1104B a A.ON CORE NETWOR ACCESS NETWORK S 1102 ELEMENT 1101 1106 1112 SUBSCRIBER 1108M SERVICE PROVIDER 1 104N N 1100 F.G. 11

U.S. Patent May 18, 2010 Sheet 12 of 15 US 7,719,995 B2 1200 ATCA CHASSIS TRAFFICBLADE PHYSICAL LAYER LINKLAYER NETWORK LAYER oNPU a PHY BACKPLANE HGIG XGM SP4.2TB HOST SP4.2 SP42 BEE O K. O K BEE C T K. D. C. C T K C T K. DC O K FULL MESH INTER CONNECT COMPUTE BLADE SERVICE HGG LAYER oCOMPUTE NODE C O a SP42 BE SP4.2 BEE OK D. D K. D. C. C 1202 K. 1204 FIG. 12

U.S. Patent May 18, 2010 US 7,719,995 B2 Sheet 14 of 15 –) – L– C “ SECRJS

U.S. Patent May 18, 2010 Sheet 15 Of 15 US 7,719,995 B2 F.G. 15 12022 N / 12023 21N3, N Sps ANSA-XVXX/SAxe, NY, XN-4 as v 1218 iXSyxxix. N. 35XX-2N V X V, ZXAfta ZAlte YA 7N a . XX Xax SYY s 2\k AZ 5x8 AZXYYAXKASN w A K NXY-a2S, 2 7 SRXR3Kxh2 S-SA 4Alex(\tfZ27 N NasySXX272s2 Sassax2, Say2 \ 12007 12009

US 7,719,995 B2 1. 2 a single destination. Unicast packets using the IPv4 or IPv6 standard, have specified header/footer fields set, which iden tify the packets as unicast packets. A flow of related unicast packets is referred to as a unicast flow. The term unicast exists APPLICATION DRIVEN FAST UNICAST FLOW REPLICATION CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority benefit, under 35 U.S.C. S119(e), of U.S. Provisional Application No. 60/719,890, filed on Sep. 9, 2005, the contents of which are incorporated herein by reference. in contradiction to multicast, which is communication 10 TECHNICAL FIELD This disclosure relates generally to network packet pro cessing, and in particular but not exclusively, relates to uni cast flow replication. 15 BACKGROUND INFORMATION A data packet (also known as a "packet’) is a block of user data with address and administration information attached, usually in a packet header and/or footer, which allows a network to deliver the data packet to the correct destination. Examples of data packets include, but are not limited to, Internet Protocol (IP) packets, asynchronous transfer mode (ATM) cells, Ethernet frames, synchronous optical network (“SONET) frames, and Frame Relay packets. Typically, related data packets having similar characteristics (e.g., related to the same application) and common source and destination addresses are referred to as a packet flow. FIG. 1 represents the Open Systems Interconnect (“OSI) model of a layered network protocol stack 100 for transmit ting data packets. Each layer installs its own header in the data packet being transmitted to control the packet through the network. A physical layer (layer 1) 105 is used for the physi cal signaling. A next layer, data link layer (layer 2) 110. enables transferring of data between network entities. A net work layer (layer3) 115 contains information for transferring variable length data packet between one or more networks. For example, IP addresses are contained in network layer 115, which allows network devices (also commonly referred to a network elements) to route the data packet. Layer 4, transport layer 120, provides transparent data transfer between end users. A session layer (layer 5) 125, provides the mechanism for managing the dialogue between end-user applications. A presentation layer (layer 6) 130 provides independence from difference in data representation (e.g. encryption, data encod ing, etc.). The final layer is an application layer (layer 7) 135, which contains the actual data used by the application send ing or receiving the packet. While protocol stacks often do not exactly follow the OSI model, it is commonly used to describe control flows. Since conventional routers are limited to layer-3 process ing, which include the IP address processing to make routing decisions, router 220 is labeled as an L3 router. FIG. 3 illus 25 flows. 30 35 execute protocol stacks 305 on the client side, content pro vider 210 executes a protocol stack 310 on the serverside, and L3 router 220 executes a protocol stack 320. Protocol stacks 305 and 310 terminate the unicast flows, since they are the ultimate source or destination of the unicast flows. In other words, the server application executing on content provider 210 and the client applications executing on client subscribers 205 parse data communicated in the unicast flows all the way to the application layer to extract the specific application data. In the case of unicast control flows 325, this is illustrated with termination dots 330. In contrast, unicast 40 control flow 325 is merely parsed up to the network layer in L3 router 220. L3 router 220 only unwraps or analyzes the packets of unicast control flows 325 to the network layer to effect routing decision, but is unaware of higher layer data and therefore unable to interpret application data wrapped in the application layer. L3 router 220 does not access the appli cation data and therefore does not terminate unicast control 45 flows 325. Similarly, unicast content flows 335 pass through L3 router 220 without termination and without the applica tion data being parsed or analyzed. BRIEF DESCRIPTION OF THE DRAWINGS 50 55 Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following fig ures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. FIG. 1 (Prior Art) is a block diagram illustrating the Open Systems Interconnect model of a layered network protocol stack. 60 FIG. 2 (Prior Art) is a block diagram illustrating unicast flows between a server and multiple clients via a layer-3 rOuter. The predominant form of transmission on local area net works (“LANs), wide area networks (“WANs), and within the Internet is unicast transmission. Unicast is the term used to describe communication where a piece of information is sent from one point to another. In other words, there is just one sender and just one receiver. A unicast packet is a packet that has been transmitted from a single source and is addressed to trates the layered network protocol stacks associated with each network element in FIG. 2. Client subscribers 205 networks. FIG. 2 illustrates a typical client-server model where a plurality of client subscribers 205 receive and consume con tent transmitted by a content provider 210 (the server) trans ported via a network 215 (e.g., the Internet, a wide area network, a metro network, a local area network, or the like). Network 215 may include one or more routers 220 to switch packets carrying the content through network 215 in packet between a single sender and multiple receivers, and anycast, which is communication between any sender and the nearest of a group of receivers in a network. Unicast packets are distinct from multicast and anycast packets not only in the techniques used to route them through a network, but also in the values and fields set in their packet headers/footers. Accordingly, FIG. 2 illustrates content provider 210 trans mitting independent and distinct unicast content flows to each client subscriber 205. A unicast content flow is simply a unicast flow of data packets which provides content (e.g., media content, video on demand (“VoID), IP television (“IPTV), streaming audio, etc.) for consumption by client subscribers 205. Client subscribers 205 may communicate back to the content provider 210 (e.g., to select the particular content desired) with control packets transmitted in unicast 65 FIG. 3 (Prior Art) is a block diagram illustrating network protocol stacks executed by a server, multiple clients, and a layer-3 router. FIG. 4 is a block diagram illustrating unicast flow replica tion by an application router, in accordance with an embodi ment of the invention.

US 7,719,995 B2 4 Throughout this specification, several terms of art are used. These terms are to take on their ordinary meaning in the art from which they come, unless specifically defined herein or the context of their use would clearly suggest otherwise. A (“unicast flow” is defined herein as a plurality of related packets, cells, or frames flowing into or out of a network node and having a common Source? destination address pair. A uni 3 FIG. 5 is a flow chart illustrating a process for unicast flow replication by an application router, in accordance with an embodiment of the invention. FIG. 6 is a block diagram illustrating network protocol stacks executed by a server, multiple clients, and an applica tion router, in accordance with an embodiment of the inven tion. FIG. 7 (Prior Art) is a block diagram illustrating a system for delivering multicast content from a head end content server to a plurality of client consumers. FIG. 8A (Prior Art) is a block diagram illustrating a mul ticast content flow including primary key frames and delta vectors for transporting Internet Protocol (“IP) television (“IPTV) content. FIG. 8B (Prior Art) is a block diagram illustrating second ary key frames generated by regional or local servers to effec tuate fast channel changes for an IPTV delivery system. FIG. 9 is a block diagram illustrating a system for deliver ing multicast content from a head end content server to a plurality of client consumers using unicast flow replication between regional/local servers and the client consumers, in cast flow is distinct from a multicast flow in that a unicast flow 10 ferent header/footer fields, as is known in the art of network communications. 15 405, in accordance with an embodiment of the invention. The flows transmitted between client subscribers 415 and content provider 410, thereby reducing bandwidth consumption along link 420 between APP router 405 and content provider 410. In one embodiment, APP router 405 includes sufficient 25 dance with an embodiment of the invention. FIG. 11 is a block diagram illustrating an exemplary con figuration using an application aware network element in a metro area network, inaccordance with an embodiment of the 30 FIG. 12 is a block diagram illustrating interconnections between traffic blades and compute blades of an application aware network element, in accordance with an embodiment 35 FIG. 13 is a block diagram illustrating a compute blade, in FIG. 14 is a block diagram illustrating a traffic blade, in accordance with an embodiment of the invention. 45 DETAILED DESCRIPTION als, etc. In other instances, well-known structures, materials, 50 55 or operations are not shown or described in detail to avoid obscuring certain aspects. Reference throughout this specification to ("one embodi ment” or (“an embodiment’ means that a particular feature, structure, or characteristic described in connection with the 60 embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases ("in one embodiment” or (“in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular fea tures, structures, or characteristics may be combined in any Suitable manner in one or more embodiments. the unicast content flows to all client subscribes 415 that have routes all unicast content and control flows between content 40 of the invention. Embodiments of a method, apparatus, and system for uni cast flow replication are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materi control flows are redundant (i.e., requesting similar content), and terminate redundant unicast requests for the same con tent. When redundant unicast requests for the same content are received, APP router 405 forwards one unicast request for the content on to content provider 410, terminates and queues Subsequent redundant unicast requests, and then replicates requested the same unicast content. When compared to the functionality provided by L3 router 220 illustrated in FIG. 2, L3 router 220 simply and blindly accordance with an embodiment of the invention. FIG. 15 is a schematic diagram illustrating one configura tion of an application aware network element implemented via an Advanced Telecommunication and Computing Archi tecture (ATCA) chassis, in accordance with an embodiment line rate processing power and intelligence to intercept uni cast flows traversing its datapaths, analyze or decode them all the way to the application layer, interpret the application data contained therein to determine whether two or more unicast invention. of the invention. FIG. 4 is a block diagram illustrating a system 400 for unicast flow replication using an application (APP) router illustrated embodiment of system 400 includes APP router 405, a content provider 410, client subscribers A, B, to N (collectively client subscribers 415), and a network 420. APP router 405 acts to terminate and replicate unicast accordance with an embodiment of the invention. FIG. 10 is a flow chart illustrating a process for delivering multicast content from a head end content server to a plurality of client consumers using unicast flow replication between regional/local servers and the client consumers, in accor is addressed to a single destination, whereas a multicast flow is not address to a single destination. Furthermore, unicast and multicast techniques use different protocols and set dif 65 provider 210 and client subscribers 205, without regard as to whether it is forwarding redundant unicast requests and con tent. While content provider 210 must lease sufficient band width along link 220 at potentially considerable cost, content provider 410 can lease substantially lower bandwidth along link 420 at potentially significant cost savings, and rely on network 420 (and APP router 405) to manage replication of redundant unicast flows

(12) United States Patent Luft USOO771.9995B2 (10) Patent No.: US 7,719,995 B2 (45) Date of Patent: May 18, 2010 (54) APPLICATION DRIVEN FAST UNICAST