Lessons Learned From PEP-II LLRF And Longitudinal Feedback
WEOBM02 EPAC June 2008Lessons learned from PEP-II LLRF and Longitudinal FeedbackJohn D. Fox.Themis Mastorides, Claudio Rivetta, Dan Van Winkle, Jiajing XuStanford Linear Accelerator CenterDmitry TeytelmanDimtel, Inc.Work supported by the U.S. Department of Energy under contract #DE-AC02-76SF00515
WEOBM02 EPAC June 2008PEP-II LLRF and Broadband FeedbackKicker response reconstruction from a timing sweep at ALS1.5What Was Expected? LLRF Longitudinal Coupled-Bunch Feedback1Beam response, auOriginal CDR Goals 1991/199328 Oct 960.50 0.5 Design and Methodology 1What Happened? 1.50 Commissioning Experience Major Upgrades Final PEP-II operations, performanceWhat wasn’t foreseen (via 2 examples) LFB - Impact of Noise in processing channel LLRF - Impact of Nonlinear Signal ProcessingImportant Lessons Learned -Summary123Time, ns456
WEOBM02 EPAC June 2008PEP-IITworingsof2.2kmcircumference. CDR 1991,updated 1993 Goal of factory e e- collidermachine L 3E33 e- HER 1.5A (0.75/1A 1991) e LER 2.14 A (2.14 A 1991)RF concerns High Beam Loading Impedance of cavityfundamental, detuning - lowmode coupled bunchinstabilities Reliability -extensive R&D effort in Vacuum, Feedback and RF systemsInstability Concerns - operation well beyond stability thresholds in three planes HOM Impedances of cavities - HOM driven coupled-bunch instabilities HOM Dampers - Still the need for coupled-bunch fast feedback (238 MHz sampling rate)
WEOBM02 EPAC June 2008Where we started, Where we finishedYear/run LER stations LER cavities HER stations HER cavities I HER I LERL1998244( 1 parked) 16( 4 parked) 0.6A 1.0A 1.2E33Run 1245200.91.53.0E33Run 2365201.01.74.4E33Run 3366221.11.96.3E33Run 4368261.52.59.0E33Run 5a489261.73.01.0E34Run 5b489261.92.91.2E34Run 64811281.93.01.2E34Run 74811282.1A 3.2A 1.2E34HER reconfigured 4 cavity - two cavity station in Run 3, subsequently added 2 cavity stationsThe operating configuration, gap voltages, tunes, etc. were constantly changingHER current - 2x design LER Current -1.8x design Luminosity 4X designLOM Growth rates HER 1.2 ms-1 LER 3.0 ms-1 (design - simulation was damped!)HOM growth ratesHER 3x design LER growth rates 0.45 ms-1 (5.6x design)The PEP-II collider holds the record for stored charge in a storage ring (3.213 A at 3 GeV).Were we successful in the feedback and LLRF areas because it was easy and we overdesigned/overestimated things?
WEOBM02 EPAC June 2008PEP-II low-level RF feedback loops: TopologyTuner loops - standard tuning for minimum reflected powerKlystron operating point supportstationreferencegap loopklys sat. loopripple looperrorklystron Ripple loop adjusts a complex modulator to maintain constantgain and phase shift through the klystron/modulator system.mod.RFreference Klystron saturation loops maintain constant saturation headroomDirect feedback loop (analog) Causes the station to follow the RF reference adding regulation ofthe cavity voltage Extends the beam-loading Robinson stability limit Lowers the effective fundamental impedance seen by the beamComb filter (digital) ΣRFcavitiesmod.direct RF loopband limitedkick signalHVPStunerloopcomb filtersbeamlongitudinal multi-bunchto wideband feedback systemkickerBPMLoop technologyTuner, Klys sat - EPICSComb. Gap, (ripple) - digitalDirect ,(Ripple) - analog Adds narrow gain peaks at synchrotron sidebands to further reduce the residual impedanceGap feedback loop (digital) Removes revolution harmonics from the feedback error signal to avoid saturating the klystron ongap synchronous phase transientsLongitudinal feedback uses RF as low-frequency “woofer” kicker for modes /- 10
WEOBM02 EPAC June 2008How was required LLRF performance estimated?LLRF -Origin and design/modelling F. Pedersen, Stuart Craig (Chalk River), Rich Tighe (SLAC) Linear frequency domain models Concerns about non-linear klystron, impact on impedance controlLLRF station model, beam model - nonlinear time domain simulations (Tighe, Rivetta, Mastorides) Macro-bunch structure, low-mode dynamics with Non-Linear Klystron 1994 model - No criteria for stability, robustness beyond trajectories in ms time windowsDesign proceeded based on initial simulations. Little criteria for Noise, Dynamic range issues, I&Qmismatches, other technical imperfectionsLER ring modal analysisGrowth rates (ms 1)20.030.020.010.00501050.80.600.4 5Mode number0.2 100Time (ms)Open LoopPoleClosed LoopPole0 1 2 100.015Oscillation frequencies (Hz)Phase (deg)0.025jω1 50510oLGDW damping lxImpedance control4500netdamping d l4300growthrate σl1/t41003900 10 50Mode number510ox
WEOBM02 EPAC June 2008PEP-II RF Station, LLRFEach Station 1.2 MW 476 MHz Klystron VXI-based LLRF electronics 2 or 4 RF cavities, with HOM loads HV power supply, Interlocks, etc.
WEOBM02 EPAC June 2008LFB Systems DesignBPMKicker structureBeamKicker oscillatorComb generatorlocked to 9/4 frfPowerLNA1071 MHzamplifierTiming and controlPhase servoDSPFarm of digitalsignal processorsHoldbuffer, DAC ADC, downsamplerLow-pass filter QPSK modulatorWoofer linkLow-pass filterMaster oscillatorlocked to 6 frf2856 MHzLow group-delay channelTo RF stationsDSP at 9.81 MHzA DSP based flexible, programmable system (can run arbitrary FIR or IIR Filters)Developed for PEP-II, ALS, DAΦNE (later BESSY-II, PLS and SPEAR). Multi-crate VXI/VMEDetection at 6 F RF , correction at 9/4 RF (options 11/4, 13/4)9Scalable VME processing array, up to 3.2 10 MAC/sec.Sampling, A/D and D/A at 500 MHz (238 MHz PEP-II)Downsampling to reduce computational load (match processing rate to synchrotron oscillationfrequency). Original “woofer” taken at DSP farm D/A wideband output.
WEOBM02 EPAC June 2008How was required LFB performance estimated?Linear Growth Rates, Gain and Time-domain Simulation (tracking, cavity HOM estimates, with downsampledFIR filter)Thresholds, Growth Rates - from cavity HOM measurements/estimatesBeam tests - 1 bunch (SPEAR), ALS 4 processor “Quick Prototype”Resolution of front end modelled and lab-tested (noise kept small for high DSP gain)Required kicker power - estimated from injection error (amp expense, minimize installed power)Dynamics estimates from simulation. Filter completely programmableBeam-LLRF Simulations predict stable Low-mode behavior - LER issue at ultimate currentsInsurance policy - design in a low-mode “Woofer” channelε 0100 ns 110ΣΣ Kicker φ Σ vb 3100Amplitude (dB) 0.5positive λl 410negative λlSynhrotron Radiation damping dr 510 1 1.5 2 2.50204060Frequency (MHz)80100120 30npBPMCombXvml Re(λ ) (ms1)10FilterBeamMasterOscillator 2vk200040006000Frequency (Hz)80001000012000
WEOBM02 EPAC June 2008Features Anticipated and ImplementedLFBprogrammable 80 processor DSP reconfigurable array500 MS/sec. A/D, D/A, Downsampler - table driven 2 ns bucket spacingGrow-damp dynamics measurements (via dual-port memory, codes)monitoring functions (Signal MUX, RMS detectors front and back-end)Woofer outputLLRFSoftware controlled broadband (direct - analog) and comb (IIR digital) loopsSoftware based low frequency digital regulators via EPICSBuilt-in network analyzer (via time domain excitation, response functions)Fault filesWoofer inputThe days of NIM Modules with Pots - are over. Software intensive systems, with VxWorks. GUI, etc.
WEOBM02 EPAC June 2008Commissioning Experience LFBALS - extensive experience with the “quickprototype” made commissioning fastVXI system commissioned at ALS 1994Historical DataRun 4Run 5bRun 6Run 70.35Growth Rates (ms 1)developed control filters, EP-II (1998)0.05800100012001400160018002000Current (mA)Thresholds - cavity fundamental driven low modes300 mA (Simulation had them damped!)0Damping Rates (ms 1)HOM Thresholds - consistent with cavity HOMmeasurements - damping rates per simulationBeam has RF power supply noise (very differentfrom ALS) 0.1 0.2 0.3May 31st 2007June 1st 2007Aug 13th 2007Feb 9th 2008 0.4 0.5130014001500160017001800190020002100Current (mA)6750Woofer required for low mode controlcoordination with operations on synchronousphase, timing LFBFrequency (Hz)operational issues 6700665015001550160016501700Current (mA)1750180018501900
WEOBM02 EPAC June 2008Unexpected Impact of “noise” in ReceiverUnanticipated - amount of “noise” on beam fromRF systems (unlike ALS and SPEAR Experience)Ratio 0.58111; Θ 2.288; Θvar 44.01; H var 1.0384; Gain ratio 11050Many sources, predominantly klystron HVPS,LLRF processing, phase distribution noise 5 10Impact of driven motion vs. HOM instability0123Magnitude of Filter TF45610121012(28.1 at 6.595 kHz)50Quantizing noise in A/D, Rcvr noise insignificantHER LFB receiver noise (rms A/D counts)0010 5002HER 1700 mA Red4Phase of Filter TF68( 90.6 degrees at 6.595 kHz)Counts200deg1000 110 100HER Receiver No Beam Cyan 200HER A/D 50 Ohm Termination Green0246Freq (kHz)8100246Freq. (kHz)8
WEOBM02 EPAC June 2008Low-Frequency Noise leads to saturation and runaway HOM controlAt 1900 mA - 2100 mA in HER, unexpectedtransient saturation effects, loss of controlVery hard to diagnose, not a steady state situation,infrequent transient effectsmagnitude of 720 Hz constantly changing with RFsystem configurations, operating points, activestations, maintenance etc.solutions via better 720 Hz control in LLRF andwoofer ( more kicker amp power would help,too)
WEOBM02 EPAC June 2008Commissioning Experience LLRFIssues with configurations of direct and comb loops -stations oscillatingSpurious signals in RF output, klystron oscillationsInitial configuration method - network analyzer no beam, gain/phase marginsInstabilities (Station and/or Beam) at currentmanual tweaking of the direct and comb loopstrade-off of station stability vs. beam stability - low mode growth rates much faster than anticipatedIQA Module #1 Fault File LR45 Klystron Output Forward PEP II LLRF tutorial450IQA Module #1 Fault File LR45 Klystron Output Forward PEP II LLRF tutorial8040060350300Magnitude (dB)power kW40250200150200100 205000246time ms81012 400100200300400500Frequency (kHz)600700800900
WEOBM02 EPAC June 2008Major Developments/upgrades - LLRFModel based configuration - reconfigure loops at current. (Dynamics changes with current)Fault file methodology, weekly reports and analysis- understand origins of operational problemsLow Group delay Woofer - necessary above 1500 mA (HER), 2A (LER). Rapid low-mode growthKlystron linearizer - effort to address fast low-mode growth rates - but not with expected resultExtensive re-investment in LLRF-Beam simulation models. Led to improved Driver Amplifieraddresses limits of impedance control, allows comb rotation, more optimal station configurationsDirect: Fr 475.9 0.1 MHz; G 5.268 0.009; Td 430.8 0.6 ns; φ 164.1 0.1 deg10To HER back end moduleDataEthernet 10 20 1000 50005001000Phase (degrees)Comb: Gc 0.2042 0.0009; Tc 5590 2 ns; φc 30.1 0.3 deg2001000Offset&filterboard #1 (HER)Linux 16ADC 5000Frequency (kHz)5001000EPICS IOCDACGVA 200 FPGA boardFPGAXC4085XLADACOffset&filterboard #2 (LER)To LER back end module 100 200 1000From HER and LER LFB phase monitorsGain (dB)Fit0Front panel status LEDs,trigger inputsParallel portdriver
WEOBM02 EPAC June 2008Understanding the impact of nonlinear processingFFT klystron output (LR42 Ibeam 1900mA)0 20LLRF Signals - Dynamic range 90 dB!Non-linear behavior in loop - imperfections 40Klystron - obviously non-linear 60We missed - medium power amplifier 80very significant impact- SS vs. LS gain 100Image frequency generationNot realized for 7 years - understood via model 120 1000 5000freq (kHz)154014301320100 10 20 30 40 504711000Amplitude Response50Gain (dB) relative ( approx 30 dB)Output Power (dBm)Output Signal (single Sideband test)50020W CarrierNo Carrier1211109876472473474475476477freq (MHz)4784794804815471472473474475476477frequency (MHz)478479480481
WEOBM02 EPAC June 2008Final PEP-II Run April 2008LER LLRF and LOM control limit at 3100 mA without comb rotation, new LLRF amplifierReplacing Driver Amp, useof “comb rotation” provided30%improvementinimpedance controlRun 6/7 - new amps, combrotationThis was possible, with New Modelling3.53Growth Rates (ms 1)Run 5B - old amps, originalLLRF config42.5Run 5b configurationRun 5b configurationNew LER42 AmpRun 6, low comb gainEnd of Run 6 configurationApril 2008Run 7 configuration21.51 New control technique0.5 Careful operating pointselectionMany Accelerator issues(e.g. bunch length, heating,optics, choise of gap voltage, etc.)0150020002500Beam Current (mA)30003500
WEOBM02 EPAC June 2008Lessons LearnedLLRF Modelling - key to understanding non-linear effects Non-linear klystron was less significant than drive amplifierWithout models - impossible to sort out effects, see if things were worse than they should beModels - prediction of limits, identification of nonlinear amp, new control techniques (comb rotation)This took years, but was invaluableFault Files - so much informationIQA Module #1 Fault File LR42 Klystron Output Forward 14 Jul 2007 16:39:38600PEP-II Experience - needed a full-time RF expertjust to understand complexities of faults500LFB - transient domain measurements key tounderstanding dynamics400power kWwho is the customer for this information?300200Unanticipated - saturation limit from RF systemFlexibility of DSP architecture key to unexpectedapplicationsNew Accelerator Diagnostics developed10000102030time ms405060
WEOBM02 EPAC June 2008What wasn’t foreseen -more lessonsLFB - thermal problems with beam induced power inkickers kW power levels -SC/DIN/EIA connectors? Cable fires (several systems)RF operational task - management of so many stations Individual station dynamics- unique station tostation Individually configured stationsImpact of non-linear Klystron/Preampcomplexity of fault file analysisR&D project - continual changes and performancepush in the machineHow is this consistent with the operating machine?
WEOBM02 EPAC June 2008Summary - Lessons learnedWere we successful in the feedback and LLRF areas because it was easy and we overdesigned/overestimated things?LLRF and RF dynamics Complexity of RF system, stability of low modes, operational issues - completely underestimated Unexpected low-mode instabilities - 2 woofers- klystron linearizer- finding nonlinear amp Operational intensity, issues of constantly moving configurations (klystrons on/off, gap voltages) Manpower/skill of operational support woefully underestimated/under supportedBroadband (coupled-bunch) longitudinal feedback Essential techniques developed at ALS and other facilities - tremendous benefit to PEP-II Very lucky (wise?) design choices for detection frequency, scalability of output powerWhat features were essential for success Flexibility (reprogrammability, modular architecture), close ties to modelling/measurements Most important element- Creative, highly curious group with concurrent physics/technology skills Diverse set of interesting challenges
WEOBM02 EPAC June 2008AcknowledgmentsThanks to D. Andersen, L. Beckman, M. Browne, P. Corredoura, J. Dusatko, M. Minty, P. McIntosh,C. Limborg, S. Prabhakar, W. Ross, J. Sebek, R. Tighe, U. Wienands, A. Young, H. Hindi, I. Linscott(Stanford), M. Tobiyama, E. Kikutani (KEK), A. Drago, A. Ghigo, F. Marcellini, M. Serio (LNFINFN), Shaukat Khan (BESSY), H. Kang ( POSTECH),W. Barry, J. Byrd, J. Corlett, G. Lambertson,G. Stover and M. Zisman (LBL) for numerous discussions, advice and contributions.The PEP-II LLRF Systems were inspired by Flemming Pedersen ( CERN) and developed by P.Corredoura, R. Tighe, L. Sapozhnikov,B. Ross, J. Judkins, H. Schwarz, A. Hill (SLAC) and manyothers. System software was designed and coded by S. Allison, R. Claus, K. Krauter, K. Luchini andD. Teytelman (SLAC). We thank P. Baudrenghien and J. Tuckmantel for LHC-related collaboration.The PEP-II LFB digital processing architecture and modules were skillfully designed and developedby G. Oxoby, J. Olsen, J. Hoeflich and B. Ross (SLAC) - System software was designed and codedby R. Claus (SLAC), I. Linscott (Stanford), K. Krauter, S. Prabhakar and D. Teytelman (SLAC)The wideband longitudinal kicker for ALS and PEP-II was designed and developed by F. Voelker andJ. Corlett (LBL). The kicker for DAFNE (PEP-II) was designed by R. Boni, A. Gallo, F. Marcellini,et.al.Special thanks to Boni Cordova-Grimaldi (SLAC) for fabrication expertise and to the ALS, SPEAR,PEP-II, DAFNE, KEKB, PLS and BESSY-II operations groups for their consistent good humor andhelp. Work supported by U.S. Department of Energy contract DE-AC03-76SF0051
WEOBM02 EPAC June 2008Longitudinal Feedback System FeaturesMultiprocessor architecture fully implements ALS/BESSY-II/DA Φ NE/PEP-II/PLS/SPEARrequirements. Scalable, flexible architecture for up to 8192 bunches with up to 500 MHz samplingrates. DSP processor -VME card,4 AT&T DSP 1610s VME interface - Bus master for data distribution Downsampler- 500 MHz A/D and VXI Sequencer Hold Buffer -500 MHz D/A and VXI Ring Buffer Timing - VXI oscillators ( RF , 6 RF , 9 4 RF ) Front-end - Comb filter followed by 6 RF (3 GHz) phase detector - 600 MHz IF bandwidth Back-end - AM modulator transfers baseband kick to QPSK’ed carrier (1125 MHz, 1071 MHz,1196 MHz, 1375 MHz). Software - VxWorks operating system for configuration and control with EPICS-based userinterface Data analysis in Matlab, Automated diagnostics and setup tools Link error checking, temperature monitoring
WEOBM02 EPAC June 2008Grow/damp measurement example from PLSA 30 ms long data set with 15 ms open-loop section.All filled bunches participate in the modal motion.Transformation to the even-fill eigenmode basissimplifies the picture - there are three strong eigenmodesin this transient. Fitting complex exponentials to themodalmotion we extract estimates of the modaleigenvalues for both open and closed-loop parts of thetransient.A single measurement like this only characterizes theinstabilities and the feedback at a single acceleratoroperating point.A very powerful technique is to measure modaleigenvalues as a function of beam current, RF systemconfiguration, etc.
WEOBM02 EPAC June 2008LFB Flexibility -Quadrupole instability control40 increased operating currents20 quadrupole mode longitudinalinstabilities have appeared (theinstalled system suppresses thedipole modes).Gain (dB)DAFNE e /e-collider at LNF0 200200 100 two parallel control paths fordipole and quadrupole modes. quadrupole control has beensuccessful, allowing a 20%increase in luminosity.Phase (deg)We implemented a novel quadrupolecontrol filtersoftware programmability ofthe DSP farmDualNotch 40204060Frequency (kHz)80100120204060Frequency (kHz)801001200 100 2000ALS added passive harmonic cavities (to address Tousheck-limited lifetime) - unanticipated effectgiant tune change with current. Stability required a novel negative group delay IIR Filter
WEOBM02 EPAC June 2008Where We finished - LOM and HOM controlLast year of OperationsPlan to push from 1.2E34 to 2E34 - via current increase, bunch length, optics changesLongitudinal stability- Data from LER 2900 mAGrowth/Damping rate isn’t the issue. Strange interfering signals at 1100 Hz, etc. are a mystery
WEOBM02 EPAC June 2008Ultimate/Practical Limits to Instability ControlWhat Limits the Maximum Gain (e.g. fastest growth rate, or allowed impedance)?Several MechanismI). Noise in feedback filter bandwidth, limits on noise saturation. Gain is from several stages Front End (BPM to baseband signal) gain limited by required oscillation dynamic range, steady-stateoffsets (synchronous phase transients, orbit offsets)Processing Block - gain limited by noise in filter bandwidth. Quantizing noise (broadband) is onesystem limit - noise from RF system or front-end circuitry may also contribute. Narrowband filtershelp with broadband noise. Broad filter bandwidths help with reduced sensitivity to machine tunes,operating point - or variations of dynamics with currentPower stages - gain scales with kicker impedance, sqrt(output power). An expensive way to increasegain (more kickers, more output power).Output power (actually maximum kicker voltage) determines maximum oscillation amplitude fromwhich linear (non-saturated) control is possible. Saturated behavior is complicated
WEOBM02 EPAC June 2008Ultimate/Practical Limits to Instability ControlII) Stability of the feedback loop itself, (e.g. limits on phase shift and gain vs. control frequency)Related to time delay between pickup, processing, and actuatorFor circular machines (systems with kick signal applied on later turn than pickup)limit set by revolution time, fastest growth rates, and filter phase slope over control bandAppropriate for optimal control theory applicationsLQRRobust ControlUncertain SystemsNegative group delay over a portion of the frequency band is possible, but for causal systems you paythe price in increased phase slope away from the negative region
1 1.5 Time, ns Beam response, au Kicker response reconstruction from a timing sweep at ALS 28 Oct 96. WEOBM02 EPAC June 2008 . 1300 1400 1500 1600 1700 1800 1900 2000 2100 0.5 0.4 0.3 0.2 0.1 0 Current (mA) Damping Rates (ms . ( more kicker amp power would help,too) WEOBM02 EPAC June 2008
PEP Orientation Goals: The purpose of Module A, PEP Orientation Session is to give you, as a prospective mentor or mentee a general understanding of PEP. It is intended to reach both mentors and mentees and create enthusiasm for becoming part of the PEP experience. The PEP Orientation S
All the amps had third-order IMD down between -40 and -50 dB PEP. QST review Elecraft KPA1500 amp listed third-order IMD at -30 dB PEP. Flex PowerGenius XL -30 dB on 20m, -27 dB PEP on 10 & 6 meters. SPE Expert 1.5K-FA ARRL measured -30 dB PEP on 20 meters. -30 dB is 6 t
LESSONS_LEARNED_REPORT BI Project Page 1 PROJECT LESSONS LEARNED REPORT Project Name: Business Intelligence (BI) Prepared by: Diane Kleinman Date: June 15, 2009 Project Close-Out Discussion A Lessons Learned meeting was held on 6/12/09. The summarized lessons learned survey results are attached to this document. Attendees: Janet Heller Vel Angamthu
As the centralized lessons learned capability for the Army, CALL synthesizes input from across the ALLP community and disseminates pertinent lessons learned information to units to help plan, prepare, and execute mission requirements. This collaboration allows TRADOC, as the lead for Army lessons learned, to provide
Apr 19, 1995 · Lessons Learned Major Lessons Learned Lessons Learned through Response/Recovery Operations Lessons Learned from Other Agencies Statistics Introduction, Summary of Fatalities and Injuries Exhibits Exhibit A - Murrah Building Floor Plan Image of Floors 1 and 2 (73Kb) Imag
TOPIC 12 Understand Fractions as Numbers 8 LESSONS 13 DAYS TOPIC 13 Fraction Equivalence and Comparison 8 LESSONS 12 DAYS TOPIC 14 Solve Time, Capacity, and Mass Problems 9 LESSONS 11 DAYS TOPIC 15 Attributes of Two-Dimensional Shapes* 5 LESSONS 9 DAYS TOPIC 16 Solve Perimeter Problems 6 LESSONS 8 DAYS Step Up Lessons 10 LESSONS 10 DAYS TOTAL .
Learned.” The USFA acknowledges the effort of the individuals responsible for producing those legacy works. The updated content from those two publications is coupled in this report with a stronger focus on learning from lessons learned. The lessons learned by first responders and emergency managers in the April 2011 tornado outbreak in
The purpose of a lessons learned activity following a cyber incident is to reflect, learn and improve. Lessons learned from the incident should be used to improve security measures and the incident handling process itself. This paper is an overarching lesson learned report for SEPA. To produce this paper, information has been
