Monitoring From Home
Monitoring from HomeBruce Howardfor the Control Room and Operations teams17 March 2020
Shifts for ICARUS: very quick summary Until now, we’ve had a pretty smooth operations with both Fermilab local shifters and othershifters who have traveled to Fermilab to take shifts We had started to think towards off-site Remote Operations Centers (ROCs) at otherinstitutions Original testing was performed at CSU, and recently Umut has been running some testsfrom CERN. Additional interest These centers were not planned to be fully commissioned for some time (setup, testing,approval procedures, etc.)The rapidly changing environment due to COVID-19 has forced us to start monitoring the filland related tasks away from ROC-West, and in general, away from any institution This lasted from 14 February until now with 24/7 shifts from Fermilab’s ROC-WestAdditional steps necessary now (e.g. VNC) - hope to make less burdensome in futureThis talk represents the work of many individuals both in terms of setting up tools and in termsof testing them. Thanks to all who are helping make “monitoring from home” possible.Monitoring from Home B. Howard2
Some basic points This talk is NOT a tutorial on how to take shift, however I will quickly show a few things/tools that are useful independent of ifa person takes shift from ROC-West or connects from homeWiki page! Has links to many usefulpages explaining what to do on shift and how.This picture is a link Monitoring from Home B. Howard3
Some basic points This talk is NOT a tutorial on how to take shift, however I will quickly show a few things/tools that are useful independent of ifa person takes shift from ROC-West or connects from homeMessage on COVID-19: link to remote shift connections. We’ll get back to this Wiki page! Has links to many usefulpages explaining what to do on shift and how.This picture is a link Monitoring from Home B. Howard4
Some basic points This talk is NOT a tutorial on how to take shift, however I will quickly show a few things/tools that are useful independent of ifa person takes shift from ROC-West or connects from homeWiki pages!Monitoring from Home B. Howard5
Some basic points This talk is NOT a tutorial on how to take shift, however I will quickly show a few things/tools that are useful independent of ifa person takes shift from ROC-West or connects from homeDocument with some usefulthings for shifts (Zoom) butmostly on setting up a shift stationThis picture is a link Log in as icarus user on doc-dbMonitoring from Home B. Howard6
Some basic points This talk is NOT a tutorial on how to take shift, however I will quickly show a few things/tools that are useful independent of ifa person takes shift from ROC-West or connects from homeDocument with Expert Contact Info!This picture is a link Log in as icarus user on doc-dbMonitoring from Home B. Howard7
Some basic points This talk is NOT a tutorial on how to take shift, however I will quickly show a few things/tools that are useful independent of ifa person takes shift from ROC-West or connects from homeIf you log into the ECL and click on Projects, or log directly into the Projects pagehttp://dbweb0.fnal.gov/ECL/sbnfd/PL/indexyou will see this. Go to Configurations to view links to the cameras to monitor,passwords for the VNC sessions, etc.Monitoring from Home B. Howard8
Remote monitoring/shiftingin response to COVID-19Monitoring from Home B. Howard9
Status Many of the tools have been available offsite before the response, but was onlyexercised by a select group of experts Examples: DAQ and subsystem testing by experts, checking Online Monitoring,a handful of people on experiment had iFix accounts Due to the evolving situation as the world deals with COVID-19, to continue takingshifts, monitoring the fill, and taking noise data and DAQ tests, it has becomenecessary to swiftly move to a “monitoring from home” scheme We have put together a set of instructions to connect to things remotely, and havetaken steps over the last few days to test and streamline many of these things See the following agenda item for more on the various tests. I’ll go through howto connect.Cryogenics monitoring is first and foremost goal - we are thinking it’s reasonable toask for DAQ/noise run monitoring, but may need to re-evaluateMonitoring from Home B. Howard10
Remote Connections Wiki page Thanks to Wes for putting together a Wiki page based on the preliminary instructions— describes what is needed to connect to the various pages/tools, and how We’ve been editing this page over the last few days and based on tests to try to geteverything together for Remote Shift capabilitiesThis page goes through most things,we’ll try to make more palatable hereThis picture is a link to the page Monitoring from Home B. Howard11
Pre-requisites Just like for any shift, you will need the basic accounts: Fermilab services account (ECL) - this is the account you use for servicedesk tickets, Fermilab email if you have one, etc. And an ECL accountFermilab Kerberos account (ssh/sign into Linux machines) - this is what you need to set up to ssh toFermilab Linux machines like to reach the VNC our DAQ machinesFor remote shifting/monitoring the following are also needed: Fermilab VPN (Virtual Private Network) - allows you to essentially extend Fermilab’s network (theinfamous FGZ network) to your computer, allowin you to access things as if you were on-site Fermilab “FERMI” (Windows) domain account - this allows you sign into Fermilab Windows machines likeiFix (cryogenics monitoring) iFix account - this allows you to log in to view the cryogenics monitoring system Remote Desktop Program compatible with your Operating System Accounts on the ICARUS (SBN-FD) DAQ computing cluster to be able to log in and connect to the DAQ(Run Control and Online Monitoring) and DCS (slow controls) sessionsMonitoring from Home B. Howard12
Basics & FERMI (Windows) Domain Won’t get into Services or Kerberos, will assume you have one already. If you don’t remember yourpasswords, call the service desk to try to get them reset ( 1 630-840-2345) To get the FERMI (Windows) domain account set up, try the following. Thanks to Justin M. for puttingv1 together and to Aiwu Z. who provided additional insight from his interaction w/Service Desk:1. Call Fermilab Service Desk at 630-840-2345 & follow instructions to talk to Service Desk Agent(Option 1)2. Tell the agent that you wish to have your Fermi Domain account reset. They will ask you for yourFermilab ID. To verify your identity they will prompt you with your security question and check youranswer3. If you have not yet received a password for your Fermi domain account, they will give you thetemporary one that they sentThere seems to be some Organizational Unit (OU) and they may ask what OU you want your Fermidomain account in. They don’t seem to have a Neutrino Division OU, but in a case where the personwas in the PPD (Particle Physics Division) OU already, this worked fine. Once you have a FERMI domain account set up, email me (howard AT fnal DOT gov) to have you setup with an iFix accountMonitoring from Home B. Howard13
Fermilab VPN The Fermilab VPN allows you to extend the Fermilab network to your computer Access to DAQ cluster, monitoring pages, cameras, that live behind Fermilab’s network For many shift activities, being connected to VPN is important if not physically located on-site We want to lessen the need for this for remote shifting during physics data taking, but it’s not there yetVarious Fermilab web pages explain this procedure: iki/VPN https://fermi.service-now.com/kb view.do?sysparm article KB0560Really the three lines on this sbndcode page is what you want to follow:1. You get a token (if off-site, you probably want software token, generally a smart phone/tablet app. Can also get ahardware token - USB device - from service desk2. You set up the token (you’ll make a PIN and pair the app with your Services account to VPN)3. Get the Cisco client (explained on webpage) and then use it to connect!Note: when you log in, you’ll use the app to get a 6 digit pass code, this plus the 4 digit PIN are the “secondpassword” when you sign in, using the style XXXXYYYYYY where XXXX PIN, and YYYYYY RSA Token (the firstpassword is your Services account password)Monitoring from Home B. Howard14
Fermilab VPN Really the pages linked from the list of 3 things on sbndcode site on lastslide seem to have good pictures, but I’ll include a few bits here to showthe general idea:RSA code fromIn my applicationsLog in from this screenapp on my phoneServices UsernameServices PasswordPIN RSA Code (XXXXYYYYYY)Monitoring from Home B. Howard15
Account on ICARUS (SBN-FD) DAQ/computing clusterForm to fill out, withexample text This picture is a link!Once you have the account, letsomeone know - the ICARUSdaq group, CR Ops group, BH, etc.Monitoring from Home B. Howard16
Remote Desktop Remote Desktop is a tool that lets a user connect from their computer toanother computer, but unlike ssh which opens a terminal connection, youactually can interact with the screen as if you were controlling the computer Exampleconnecting toiFix machinefrom myUbuntu:Monitoring from Home B. Howard17
Remote Desktop Most operating systems have a program that allow RDP (Remote Desktop Protocol) connections: Windows: Remote Desktop Connection (seems to be default in Windows 10, not sure aboutWindows 8 but seems to exist anyway) Mac: Microsoft Remote Desktop (can get from App Store) Ubuntu: Remmina (my Ubuntu had it by default) Scientific Linux: Vinagre (probably there by default - may be called Remote Desktop Viewer) This covers most people. If you’re running something else, Google probably knows and candirect you.Once you have your Fermi Domain account set up and have been added to iFix, then you followthe general procedure here to log in (screenshots for various OS on next page):Monitoring from Home B. Howard18
Remote DesktopMacUbuntuWindowsMonitoring from Home B. HowardSci Linux19
Setting up sessions to monitor/interact Once you have the accounts and have kerberized (kinit username@FNAL.GOV) and set up VPNyou can then go ahead and set up the various monitoring tools and screens: Web cams at building on the ECL- Projects- Configuration page Online Monitoring: n/icarusminargon.wsgi/introduction Then you can open 3 tunnels for the various tools: DAQ VNC (interact with Run Control, OnMon) DCS VNC (interact with Slow Controls - EPICS) Grafana web page (shows the status of the DAQ: data rate, triggers, alarms) Note: VNC is virtual network computing, another type of remote desktop program, usedin this case to control Linux systemsYou can follow the instructions on the web page by opening a terminal and issuing thecommands, or you can run scripts — see next slideMonitoring from Home B. Howard20
Setting up sessions to monitor/interact For the machines in ROC-West, the work of several people (Sci Linux Architecture team, the SBN CR Opsteam, etc.) had put together scripts and desktop icons to launch tunnels for the VNCs and Grafana With some alterations, we make available scripts to make the tunnels for you and hopefully makemore seamlessWe have put a git repo hosted in GitHub to store & update scripts related to Control Rooms: SBN CR We have put together a subdirectory “CR-From-Home”This image is link!Monitoring from Home B. Howard21
Setting up sessions to monitor/interact For the machines in ROC-West, the work of several people (Sci Linux Architecture team, the SBN CR Opsteam, etc.) had put together scripts and desktop icons to launch tunnels for the VNCs and Grafana With some alterations, we make available scripts to make the tunnels for you and hopefully makemore seamlessWe have put a git repo hosted in GitHub to store & update scripts related to Control Rooms: SBN CR We have put together a subdirectory “CR-From-Home”This image is link!git clone https://github.com/brucehoward-physics/SBN CR.gitor go to this page and click “Clone or Download” and download as zip.Monitoring from Home B. Howard22
Setting up sessions to monitor/interact There is a readme file in the CR-From-Home directory If you’re using a Linux distribution, you can run the shell script setup icons.sh to create3 clickable icons in the folder (one for DAQ VNC, one for DCS VNC, and one for Grafana) If you’re not using a Linux distribution (e.g. Mac), then just run the shell scripts in theCR-From-Home/scripts/ directory instead Then you just have to run each program by double clicking the “.desktop” file (afteryou run once and “allow” it to run, it should then be an icon for later uses)The scripts are designed to make things easy. If you’re running a Mac or Linuxdistribution and have Firefox, they should open the Firefox page along with the tunnelto save yourself a step. If not, they will just open the tunnels and you’ll have to opena browser and go to the necessary pages (see the Wiki page!)Let us know if you run into any issues running these scripts. We can fix them. If youdo find issues and need a solution quickly, then remember there are terminal commandson the Wiki page.Monitoring from Home B. Howard23
Slack and shift communications Shifters should make sure to get an SBN Slack account. We have a channel #icarus-shiftoperations to discuss things happening on shift, have experts contact shifters, passinformation between the outgoing and incoming shifters, etc. There is also the shift Zoom meeting room. To account for international shifters and the cellplans/etc., probably we should have shifters called into this room unless they are in anotherZoom meeting for some period of time. And in these cases are where Slack would be especially handy.It’s ideal if shifters provide some way of reaching them in their starting forms (i.e. instead ofpicking ROC-West, pick Other and then give info) Experts and Coordinators need to be able to contact you some way while you are on shiftMonitoring from Home B. Howard24
Fermilab services account (ECL) - this is the account you use for service desk tickets, Fermilab email if you have one, etc. And an ECL account Fermilab Kerberos account (ssh/sign into Linux machines) - this is what you need to set up to ssh to Fermilab
telemetry 1.24 Service P threshold_migrator 2.11 Monitoring P tomcat 1.30 Monitoring P trellis 20.30 Service P udm_manager 20.30 Service P url_response 4.52 Monitoring P usage_metering 9.28 Monitoring vCloud 2.04 Monitoring P vmax 1.44 Monitoring P vmware 7.15 Monitoring P vnxe_monitor 1.03 Monitoring vplex 1.01 Monitoring P wasp 20.30 UMP P .
What is Media Monitoring and How Do You Use it Monitoring: a history of tracking media What is monitoring? Getting started with monitoring The Benefits and Uses of Monitoring Using media monitoring to combat information overload Tools to maximize monitoring and measurement efforts Using media monitoring to develop media lists
There is no place like home . Welcome home. It is my home away from home. People have different ideas of home. For you, home might mean where you grew up, or where . your family and friends live now. For other people, home might mean the place where they live right now and would like to stay in the future. Because the idea of home is so important
SIRIUS monitoring relays: Perfect protection of machines and systems Monitoring relays 3UG451 / 461 / 463 monitoring relays for line and single-phase voltage monitoring – as 3UG481 / 483 also for IO-Link 10 6* 3RR21/22 monitoring relays for direct mounting on contactors for multi-phase current monitoring – as 3RR24 also for IO-Link 12 7 .
2.2 Monitoring surveys 7 3 Monitoring habitat 8 3.1 Food supply - direct measurement 9 3.2 Food supply - indirect measurements 9 4 Monitoring protocol summary 10 4.1 Monitoring otters 10 4.2 Monitoring habitat 11 SECTION 2:REVIEW OF ASSESSMENT TECHNIQUES AND PROTOCOL RATIONALE 13 1 Introduction 13 1.1 Monitoring otter populations 13
2016 American Home Shield Landmark Home Warranty 2016 American Home Shield OneGuard Home Warranties 2015 Fidelity National BPG Home Warranty 2014 American Home Shield HSA Home Warranty 2013 NRG Energy Allied Warranty 2012 Direct Energy Home Warranty of America 2002 Brera Capital Partners 2-10 Home Buyers Warranty .
Home Confinement and Electronic Monitoring Home Confinement. Home confinement or house arrest—with and without electronic monitoring (or EM)—is an intermediate community corrections program designed to restrict the activities of juvenile offenders in the community. Home confinement res
Bently Nevada Monitoring Systems 7200 Series (Dial Monitoring) 3500 Series (Computer Monitoring) we have two types of vibration monitoring system the Bently nevada 7200 series vibration monitoring system. It is a dial monitoring analog system installed at our plant on Air Compressor partially and on Most of the TPs etc.
