ConnectX -2 VPI Dual Port QSFP And SFP Card User’s Manual
ConnectX -2 VPI Dual Port QSFP and SFP Card User’s ManualP/N: MHZH29-XTR, MHZH29-XSRRev 1.0www.mellanox.com
Rev 1.0NOTE:THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND ANDSOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOTMET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/ORTHE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEEOR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS ORIMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENTSHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; ORBUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAYFROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.Mellanox Technologies350 Oakmead ParkwaySunnyvale, CA 94085U.S.A.www.mellanox.comTel: (408) 970-3400Fax: (408) 970-3403Mellanox Technologies, Ltd.PO Box 586 Hermon BuildingYokneam 20692IsraelTel: 972-4-909-7200Fax: 972-4-959-3245 Copyright 2009. Mellanox Technologies, Inc. All Rights Reserved.Mellanox , ConnectX , InfiniBlast , InfiniBridge , InfiniHost , InfiniRISC , InfiniScale , and InfiniPCI are registeredtrademarks of Mellanox Technologies, Ltd.BridgeX and Virtual Protocol Interconnect are trademarks of Mellanox Technologies, Ltd.All other marks and names mentioned herein may be trademarks of their respective companies.ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualDocument Number: 31572Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.0Table of ContentsTable of ContentsList of FiguresList of Tables345Revision History67About this Manual7Chapter 18Chapter 2Overview1.1 Adapter Cards1.2 Mellanox Part Numbering Legend1.3 Finding the GUID/MAC and Serial Number on the Adapter Cards8910VPI Adapter Card Installation122.12.22.32.42.51212121316Hardware and Software RequirementsInstallation InstructionsSafety WarningsCables and ModulesCable LengthsChapter 3 Driver Software and Firmware173.1 Driver Software3.2 Updating Adapter Card Firmware3.3 Single Adapter Card Firmware Update171717Chapter 4 Adapter Card Interfaces4.14.24.34.418I/O InterfacesPowerMemoryVPD Layout for these Adapter Cards18202021Chapter 5 Replacing a Tall Bracket With a Short Bracket235.1 Remove the Existing Bracket from the Adapter Card5.2 Installing the New Bracket2323Appendix A Specifications25A.1 Board Mechanical Drawing and DimensionsA.2 EMC Certification StatementsA.3 MHZH29-X[ST]R Specifications252628Appendix B Interface Connectors Pinout29B.1 I2C-Compatible Connector PinoutB.2 PCI Express x8 Connector PinoutB.3 QSFP Connector PinoutB.4 SFP Connector Pinout29293032Appendix C Avertissements de sécurité d’installation (Warnings in French)34Appendix D Sicherheitshinweise (Warnings in German)36Mellanox Technologies3
Rev 1.0List of FiguresFigure 1:Component Side9Figure 2:Card Product Label11Figure 3:SFP Transceiver Module14Figure 4:Module With Locking Mechanism Closed14Figure 5:Module With Locking Mechanism Open14Figure 6:Connector Orientation15Figure 7:Port Numbering18Figure 8:Physical and Logical Link Indications19Figure 9:I2C Connector20Figure 10: Flash Jumper20Figure 11: Adapter Card23Figure 12: Placing the Bracket on the Card24Figure 13: Schematic of the ConnectX-2 MHZH Adapter Card25Figure 14: Compatible Connector Plug and Pinout29Figure 15: Connector and Cage Views31Figure 16: Rear View of Module With Pin Placement324Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.0List of TablesTable 1:Revision History Table6Table 2:Documents List7Table 3:Mellanox Cards Part Numbering Key9Table 4:Hardware and Software Requirements12Table 5:Jumper Configuration20Table 6:Specifications for MHZH29-XTR28Table 7:I2C-compatible Connector Pinout29Table 8:SFP Connector Pinout33Mellanox Technologies5
Rev 1.0Revision HistoryThis document was printed on 11/3/09.Table 1 - Revision History Table6DateRevOctober20091.0Mellanox TechnologiesComments/ChangesInitial Release
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.0About this ManualThis User’s Manual describes Mellanox Technologies ConnectX -2 VPI IB and Ethernet PCIExpress x8 adapter cards. It provides details as to the interfaces of the board, specifications,required software and firmware for operating the board, and relevant documentation.Intended AudienceThis manual is intended for the installer and user of these cards.The manual assumes basic familiarity with InfiniBand and Ethernet networks and architecturespecifications.Related DocumentationTable 2 - Documents ListMellanox Firmware Tools (MFT) User’sManualDocument no. 2204UGUser’s Manual describing the set of MFT firmware managementtools for a single InfiniBand node.See http://www.mellanox.com Downloads Firmware ToolsIBTA Specification Release 1.2.1InfiniBand Architecture SpecificationIEEE Std 802.3 SpecificationThis is the IEEE Ethernet CI Express 2.0 SpecificationsIndustry Standard PCI Express 2.0 Card Electromechanical Specification, Rev 1.3.Online Resources Mellanox Technologies Web pages: http://www.mellanox.com Mellanox Technologies Firmware download Web page:http://www.mellanox.com Downloads FirmwareDocument ConventionsWhen discussing memory sizes, MB and MBytes are used in this document to mean size in megabytes. The use of Mb or Mbits (small b) indicates size in mega bits.Mellanox Technologies7
Rev 1.01OverviewOverviewThis document is a User’s Manual for Mellanox Technologies network VPI adapter cards based onthe MT25408 ConnectX -2 VPI integrated circuit device. The cards described in this manual havethe following main features: IBTA v1.2.1 compliant IEEE Std 802.3 compliant QSFP port for connecting InfiniBand traffic at 10Gb/s (SDR), 20Gb/s (DDR), or 40Gb/s (QDR) SFP port for connecting Ethernet traffic at 10 Gb/s Compliant with QSFP MSA spec Rev. 1.0 Compatible with copper cables and optical cables with the use of QSFP and SFP connectors PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s ‘Media detect circuit’ with powered connector support for the use of active cables and externalPHY fiber solutions EU Restriction of Hazardous Substances (RoHS) compliant 10 Gb/s Ethernet1.1Adapter CardsThe MHZH29 card is a VPI adapter card with a 40Gb/s InfiniBand QSFP connector and a 10GigESFP connector.The QSFP connector is compatible with: Gigabit Ethernet InfiniBand Architecture Specifications Fibre ChannelThe SFP connector is compatible with 10GigE.Table 3 lists the VPI adapter cards described in this manual.Table 3 - Adapter Cards List8Ordering PartNumber (OPN)IB QSFPPort SpeedETH SFP Port SpeedShort /TallBracketRoHSAdapter ICPart NumberMHZH29-XTR40Gb/s 0Gb/s (QDR)10Gb/sShortRoHS-R6MT25408B0-FCCR-QIMellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.0Figure 1: Component Side1.2Mellanox Part Numbering LegendTable 4 describes the Mellanox Technologies adapter cards part numbering legend.Table 4 - Mellanox Cards Part Numbering KeyAdapter HAdapterTypeH InfiniBand Host Channel Adapter, N Ethernet Network Interface Card,TMediaE CX4 SDR, G CX4 DDR, J CX4 QDR,Q QSFP QDR, R QSFP DDR, T UTP,Z one SFP connector and one QSFP connectorSSiliconH ConnectX , S InfiniHost III Lx , T InfiniHost ,A InfiniHost III Ex,S InfiniHost III Lx, T InfiniHost## ports1 1, 2 2,Mellanox Technologies9
Rev 1.0OverviewTable 4 - Mellanox Cards Part Numbering KeyAdapter CardOPNMHTS#I-XBRFieldDecoderIHost InterfaceX PCI-X, 4 PCIe x4, 8 PCIe x8, 9 PCIe (SerDes @ 5.0 GT/s)GGeneration blank Initial product generation-SeparatorXMemorySizeX MemFree, 1 128MB, 2 256MB, 3 512MBBBracketS Short, T Tall, N NoneRRoHS blank non RoHS, C RoHS R-5 w/ Exemption, R RoHS R-6 Lead-FreeFor example, the part number MHZH29-XTR describes Mellanox Technologies’ ConnectX-2 cardwith dual ports one SFP and one QSFP, a PCIe2.0 x8 5.0GT/s interface, no on-board memory(mem-free), a short PCI bracket, and RoHS R5 compliance. Using the legend, field M M to indicate a Mellanox Technologies product, field H H to indicate an InfiniBand Adapter Card, field T Z to indicate QSFP (IB QDR) and SFP (10GigE), field S H to indicate the ConnectX family, field # 2 to indicate two ports, field I 9 to indicate PCI Express 2.0 x8 running at 5.0GT/s, field X X to indicate no on-board memory, field B T to indicate a tall bracket, and field R R to indicate RoHS R6 (lead free) compliance1.3Finding the GUID/MAC and Serial Number on the Adapter CardsAll Mellanox adapter cards have a label on the printed side of the adapter card that has the cardserial number, the card MAC for Ethernet protocol and the card GUID for InfiniBand protocol.VPI Cards have both a MAC and a GUID.10Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.0Figure 2: Card Product LabelS/N:MT0744X00012REV: X1P/N:MHZH29-XTRGUID: 0002C9030005646 CM AC: 0002C901122642 FCMade in ILMellanox Technologies11
Rev 1.0VPI Adapter Card Installation2VPI Adapter Card Installation2.1Hardware and Software RequirementsBefore installing the adapter card, please make sure that the system meets the hardware and software requirements listed in Table 5, “Hardware and Software Requirements”. Refer to Chapter3,“Driver Software and Firmware” on page 17 for download and installation instructions.Table 5 - Hardware and Software RequirementsRequirementHardwareDescriptionPCI Express x8 or x16 slotsSoftwareOperatingSystems/Distributions For Windows IB see http://www.mellanox.com Downloads InfiniBandSW/ Windows Drivers Ethernet see http://www.mellanox.com Downloads EthernetSW/ Windows Drivers 2.2For Linux, both the InfiniBand and Ethernet drivers are in the Mellanox OpenFabrics Enterprise Distribution (OFED) software package available via the MellanoxOpenFabrics Web sitehttp://www.mellanox.com Downloads InfiniBand/VPI SW/ Linux DriversInstallation InstructionsRead all installation instructions before connecting the equipment to the power source.2.2.1 Installation Instructions as per Host MachineThe adapter cards listed in Table 3 on page 8 are standard PCI Express cards, each with a standardx8 edge connector. Please consult the host machine documentation for instructions on how toinstall a PCI Express card.2.3Safety Warnings1. Installation InstructionsRead all installation instructions before connecting the equipment to the powersource.2. Over-temperatureThis equipment should not be operated in an area with an ambient temperatureexceeding the maximum recommended: 55 C (131 F).To guarantee proper air flow, allow at least 8cm (3 inches) of clearance aroundthe ventilation openings.12Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.03. During Lightning - Electrical HazardDuring periods of lightning activity, do not work on the equipment or connector disconnect cables.4. Copper InfiniBand Cable Connecting/DisconnectingCopper InfiniBand cables are heavy and not flexible, as such they should becarefully attached to or detached from the connectors. Refer to the cable manufacturer for special warnings and instructions.5. Equipment InstallationThis equipment should be installed, replaced, or serviced only by trained andqualified personnel.6. Equipment DisposalDisposal of this equipment should be in accordance to all national laws andregulations.7. Local and National Electrical CodesThis equipment should be installed in compliance with local and national electrical codes.8. Exposure to RadiationCaution – Use of controls or adjustment or performance of procedures otherthan those specified herein may result in hazardous radiation exposure.CLASS 1 LASER PRODUCT and reference to the most recent laser standardsIEC 60 825-1:1993 A1:1997 A2:2001 and EN 60825-1:1994 A1:1996 A2:20012.4Cables and ModulesThis card can be connected using direct copper cables or optical cables using an SFP transceivermodule. Copper cables can be both active and passive.Mellanox Technologies13
Rev 1.0VPI Adapter Card InstallationThe adapter cards are shipped without optical modules. Approved modules must be purchasedfrom Mellanox. The OPNs for the approved Mellanox modules are MFM1T02A-SR andMFM1T02A-LR. The figure below shows the Mellanox approved SFP module.Note: SR and LR modules not recommended by Mellanox may not work with the adapter.Figure 3: SFP Transceiver Module2.4.1 Inserting the Optical Transceiver ModuleTo insert the module into the cage:1. Open the module’s locking mechanism – see Figure 4 and Figure 5.2. Make sure that the male connectors on the module will align with the female connectors insideof the cage. Also check that there is no dirt or foreign matter in the module or in the cage.Figure 4: Module With Locking Mechanism ClosedFigure 5: Module With Locking Mechanism Open3. Insert the module into the adapter card module cage.4. Close the locking Mechanism.To remove the module from the cage:1. Unlock the locking mechanism by opening the handle.2. Pull the module out of the cage.2.4.2 Cable InstallationAll cables can be inserted or removed with the unit powered on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. The GREEN LED indicator willlight when the physical connection is established (that is, when the unit is powered on and a cableis plugged into the port with the other end of the connector plugged into a functioning port). Afterplugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When a logical connection is made the YELLOW LED will come on. When data is beingtransferred the YELLOW LED will blink.14Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.0Note: When installing cables make sure that the latches engage.Note: Always install and remove cables by pushing or pulling the cable and connector in astraight line with the card.To remove a cable, disengage the locks and slowly pull the connector away from the port receptacle. Both LED indicators will turn off when the cable is unseated.Cables, especially long copper cables, can weigh a substantial amount. Makesure that the weight of the cable is supported on its own and is not hangingfrom the adapter card.2.4.2.1 Inserting a Cable into the Adapter Card1. Support the weight of the cable before connecting the cable to the adapter card. Do this by usinga cable holder or tying the cable to the rack.2. Determine the correct orientation of the connector to the card before inserting the connector. Donot try and insert the connector up side down. This may damage the adapter card.3. Insert the connector into the adapter card. Be careful to insert the connector straight into thecage. Do not apply any torque, up or down, to the connector cage in the adapter.4. Make sure that the connector locks in place.Figure 6: Connector OrientationTie theCable to theRack180o2.4.2.2 Removing a Cable from the Adapter Card1. Pull on the latch release mechanism thereby unlatching the connector and pull the connector outof the cage.2. Do not apply torque to the connector when removing it from the adapter card.Mellanox Technologies15
Rev 1.0VPI Adapter Card Installation3. Remove any cable supports that were used to support the cable’s weight.2.5Cable LengthsMellanox Cards support up to 40 Gb/s IB and Ethernet over the QSFP port, and up to 10Gb/s onthe SFP connector.2.5.1 InfiniBand ConnectivityThese Mellanox Cards support QSFP passive copper connectivity up to 7 meters, and active cablesupport up to 50 meters.2.5.2 Ethernet ConnectivityThese Mellanox Cards support ethernet connectivity as defined in IEEE Std 802.3. ae standard.The described adapter cards can be connected to switches and routers using cable lengths as specified in the tables below.Table 6 - Max SR Cable LengthsCableMax Length of Approved CableOM-282mOM-3330mTable 7 - Max LR Cable LengthsCableMax Length of Approved CableSMF10 KmThese cards are able to support direct attached copper cables. Check www.mellanox.com Products Cables for cable recommendations regarding direct attached copper cables.16Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s Manual3Driver Software and Firmware3.1Driver SoftwareRev 1.0For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) softwarepackage available via the Mellanox OpenFabrics Web site at:http://www.mellanox.com Downloads InfiniBand/VPI SW/Drivers. Follow the installationinstructions included in the download package.3.2Updating Adapter Card FirmwareEach card is shipped with the latest version of qualified firmware at the time of manufacturing.Firmware is updated occasionally, and the most recent firmware can be obtained fromhttp://www.mellanox.com Downloads Firmware.3.3Single Adapter Card Firmware UpdateFirmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools (MFT) package. This package is available for download, along with its user’s manual,from the Mellanox Firmware Tools page. See http://www.mellanox.com Downloads Firmware Tools.A firmware binaries table lists a binary file per adapter card. The file name of each such binary iscomposed by combining the firmware name, the firmware release version, and the card part number. Please contact Mellanox or your assigned Field Application Engineer if you cannot find thefirmware binary for your adapter card. This may happen if the product is not yet available for general distribution.Mellanox Technologies17
Rev 1.0Adapter Card Interfaces4Adapter Card Interfaces4.1I/O InterfacesEach adapter card includes the following interfaces: QSFP Optical Connector SFP Ethernet Connector PCI Express x8 edge connector I/O panel LEDs I2C compatible connector (for debug)Port 1 connects to the IB port of the device, while port 2 connects to the Ethernet port of thedevice. See Figure 7,“Port Numbering”Figure 7: Port NumberingPort 1IB portPort 2Ethernet4.1.1 InfiniBand InterfaceThe ConnectX-2 VPI device (MT25408B0) is compliant with the InfiniBand Architecture Specification, Release 1.2.1. It has a compliant 4X InfiniBand port, with four Tx/Rx pairs of SerDes.Each of the VPI adapter cards (listed in Table 3 on page 8) based on this device provides access toits port by means of a QSFP connector.This port utilizes a ‘media detect circuit’ that supports active copper cables and fiber solutions tobe connected to the InfiniBand port connectors.4.1.2 Ethernet SFP InterfaceThe ConnectX -2 device is compliant with the IEEE Std 802.3ae 10 Gigabit Ethernet and theIEEE Std 802.3aq 10GBASE LRM. The SFP port has one Tx/Rx pair of SerDes. Each of the cards(listed in Table 3 on page 8) based on this device provides access to their Ethernet port by meansof copper or optical cables.18Mellanox Technologies
ConnectX-2 VPI InfiniBand and Ethernet Adapter Card User’s ManualRev 1.
Rev 1.0 Overview 8 Mellanox Technologies 1 Overview This document is a User’s Manual for Mellanox Technologies network VPI adapter cards based on the MT25408 ConnectX -2 VPI integrated circuit device.The cards described in this manual have the following main features: IBTA v1.2.1 compliant
call Mikayel Ishmael at 561-868-3557 The classes are: VPI 0100 Reading VPI 0200 Math VPI 0300 Language VPI 0400 Strategies. Class size can range from only 1 student to as many as 15 students. . information, call the VPI Lab at 561-868-3795. Thank you for visiting the VPI web page! Title: The Vocational Preparatory Instruction Lab (VPI)
Since SystemVerilog Assertions API is an extension above the VPI, we will hereafter refer to it as VPI. The general flow of using VPI to extract information is a two-step process: 1. 55 2. Obtain the properties of assertions. Calls to standard VPI routines, such as and , can derive a list of
www.mellanox.com Mellanox Technologies ConnectX -4 Ethernet Single and Dual QSFP28 Port Adapter Card
Dual Port Cylinder Covers Can be used on all 1600 based dual port engines. This is the clean way to move on up to dual port cylinder heads. 1988 Dual Port Cylinder Covers - Black (set of 2) 1993 Dual Port Cylinder Covers - Chrome (set of 2) Deflector Plates Stock Factory Deflector Plates are used on all Type-1, 2 & 3 VW engines. They are .
2010 1.5 Minor formatting changes Nov. 22nd, 2010 1.4 Added Generation D card Mini Form Factor Oct. 28th, 2010 1.3 Added Technical Support and Firmware and Software Updates sections to the Pref-ace section Added RoCE Added ConnectX EN PXE Added Set up section, This is a sanity check after card installation. Added KCC statement to Appendix A
Doosan Heavy Ind. DP World Cochin JNPT Port of Dubai Port of El Callao Port of Everglades Evergreen Fantuzzi-Reggiane Port of Felixstowe Fels crane FMC Technologies Port of Gioia Tauro Port of Gothenburg Port of Guanghzou Port of Hamburg Port of Heidland Port of Helsinki Port of Ho Ci Minh
NxM ("N by M") switch ! Often N M ! Goals " High throughput ! Best is MIN(sum of inputs, sum of outputs) " Avoid contention " Good scalability ! Linear size/cost growth Input Port Input Port Input Port Input Port Input Port Input Port Output Port Output Port Output Port Output Port Output Port Output Port Switch Fabric
Python for Scientific Programming Paul Sherwood CCLRC Daresbury Laboratory p.sherwood@dl.ac.uk. Overview Introduction to the language – (thanks to David Beazley) Some important extension modules – free tools to extend the interpreter – extending and embedding with C and C Chemistry projects in Python – some examples of science projects that use Python Our experiences – a GUI for .
