from: Paul Sichta to: John Wertenbaker date: Dec 5, 2018, 11:48 AM subject: Re: [PPPL] NSTX-U Central I&C LogBook John, Is there a cheat sheet or document that says how the DAS link status gets to EPICS. Some IDL program on mdspc or on skylark? -ps ====================================================================================================== Paul, Oh, boy! That's a long story!!! When I log into Skylark, I run a script called camacmonitorstart. I call it like this ./camacmonitorstart & This is the content of this script: source /usr/local/mdsplus_pppl/etc/setup_epics_mds.sh ./linkwatchdog -d & idl camacmonitorbatch.pro This is the contents of the "linkwatchdog" script: #!/bin/bash # linkwatchdog # this looks at the EPICS status bits for all the links, and emails # jwertenbaker@pppl.gov if any link is down, or if any crate is off line echo Linkwatchdog is running while true # this is Linux's poor excuse for the lack of a goto do # executes lines from the 'while true' to 'done' lines forever sleep 60 failure=0 #clear failure flag epics1crates=$(caget -t -w 10 ch_Link1Health_SEL) das0crates=$(caget -t -w 10 ch_DAS0_sts_LI) das1crates=$(caget -t -w 10 ch_DAS1_sts_LI) das2crates=$(caget -t -w 10 ch_DAS2_sts_LI) epics1link=$(caget -t -w 10 ch_Link1_367status_MBBI.SEVR) das0link=$(caget -t -w 10 ch_DAS3_367status_BO.SEVR) das1link=$(caget -t -w 10 ch_DAS1_367status_BO.SEVR) das2link=$(caget -t -w 10 ch_DAS2_367status_BO.SEVR) if [ ${epics1crates// /} = "0" ]; then epics1crates="" else failure=1 #set failure flag if there is any failure epics1crates="EPICS1 crates off line" fi if [ ${das0crates// /} = "1" ]; then das0crates="" else failure=1 #set failure flag if there is any failure das0crates="DAS0/A crates off line" fi if [ ${das1crates// /} = "1" ]; then das1crates="" else failure=1 #set failure flag if there is any failure das1crates="DAS1/B crates off line" fi if [ ${das2crates// /} = "1" ]; then das2crates="" else failure=1 #set failure flag if there is any failure das2crates="DAS2/C crates off line" fi if [ ${epics1link// /} = "NO_ALARM" ]; then epics1link="" else echo -n "epics1 down, but waiting for another failure before emailing" sleep 25 epics1link=$(caget -t -w 10 ch_Link1_367status_MBBI.SEVR) if [ ${epics1link// /} = "NO_ALARM" ]; then epics1link="" else failure=1 #set failure flag if there is any failure epics1link="EPICS1 link down!" echo -n "epics1 down the second time. will email" fi fi if [ ${das0link// /} = "NO_ALARM" ]; then das0link="" else failure=1 #set failure flag if there is any failure das0link="DAS0/A link down!" fi if [ ${das1link// /} = "NO_ALARM" ]; then das1link="" else failure=1 #set failure flag if there is any failure das1link="DAS1/B link down!" fi if [ ${das2link// /} = "NO_ALARM" ]; then das2link="" else failure=1 #set failure flag if there is any failure das2link="DAS2/C link down!" fi let echocounter=$echocounter+1 if [ $echocounter == 2 ]; then echo -n "Linkwatchdog is running as of ";date echocounter=0 fi if [ "$failure" = 1 ]; then echo Subject: CAMAC Problems! > EMAILMESSAGE.txt echo $epics1crates >>EMAILMESSAGE.txt echo $das0crates >>EMAILMESSAGE.txt echo $das1crates >>EMAILMESSAGE.txt echo $das2crates >>EMAILMESSAGE.txt echo $epics1link >>EMAILMESSAGE.txt echo $das0link >>EMAILMESSAGE.txt echo $das1link >>EMAILMESSAGE.txt echo $das2link >>EMAILMESSAGE.txt date >>EMAILMESSAGE.txt cat EMAILMESSAGE.txt # print out what is emailed echo sendmail jwertenb@pppl.gov < EMAILMESSAGE.txt # Send email to jwertenbaker sleep 600 fi done This does lots of CAGET's from EPICS. If any of them indicate a failure, it sends me an email. I could include anybody else on this email, if that is desired. CAMACMONITORBATCH.PRO is an IDL program. Its contents are: .run ezcaIDL.pro wait, 5 .run camacmonitor.pro I don't know what ezcaIDL.pro is, but my program won't run without it. This is the contents of camacmonitor.pro: ;+------------------------------------------------------------- ; NAME: ; Camacmonitor ; PURPOSE: ; Get crate status from CTS SHO/CRA ; Send that information to EPICSRV ; Sends wakeup command to all crates every minute ; Regularly reads/writes two H367 modules, outputs of which go to ; H322 monitored by EPICS, which will alert machine operators ; CATEGORY: ; CAMAC ; CALLING SEQUENCE: ; $ source /usr/local/mdsplus_pppl/etc/setup_epics_mds.sh ; $ idl ; IDL> .run ezcaIDL.pro ; IDL> .run camacmonitor.pro ; INPUTS: ; No explicit inputs. ; KEYWORD PARAMETERS: ; None ; OUTPUTS: ; Writes log file to EPICSRV, and data to H367 modules ; CAPUT's to ch_DAS0_01_LI thru ch_DAS0_62_LI ; CAPUT's to ch_DAS1_01_LI thru ch_DAS1_62_LI ; CAPUT's to ch_DAS2_01_LI thru ch_DAS2_62_LI ; CAPUT's to ch_DAS0_sts_LI, ch_DAS1_sts_LI, and ch_DAS2_sts_LI ; COMMON BLOCKS: ; None ; EXAMPLE: ; $ source /usr/local/mdsplus_pppl/etc/setup_epics_mds.sh ; $ idl ; IDL> .run ezcaIDL.pro ; IDL> .run camacmonitor.pro ; NOTES: ; MODIFICATION HISTORY as CRATEMONITOR: ; create: 29-March-2000, P.Roney, PPPL ; edit: 26-Octover-2000, PR, reformat output as table with up to 4 cols ; minimize time html file is open ; edit: Mar 4, 2009 John Wertenbaker, port over to linux ; read list of crates like SCANCRATE does ; edit: Mar 23, 2009 John Wertenbaker, combine with Linkmonitor ; Add CAPUT's to EPICS status pages ; edit: Feb 19, 2016 John Wertenbaker, change remotects to remoteshowcrate ; ; MODIFICATION HISTORY as LINKMONITOR: ; create: 03-April-2000, P.Roney, PPPL ; edit: port to linux: Feb 12, 2009 J. Wertenbaker, PPPL ; edit: Mar 23, 2009 John Wertenbaker, combine with Cratemonitor ; ; MODIFICATION HISTORY AS CAMACMONITOR: ; edit: March 23,2009 John Wertenbaker, change its name to CAMACMONITOR.PRO ; edit: Jan 26, 2010 John Wertenbaker, Modify for separate Skylark and ; MDSPC. Convert from GKC to GKA. ; Replaced "SPAWN, 'mdscts sho/cra', outputArray" with ; "SPAWN, 'remoteshowcrate', outputArray" ; edit: Feb 22, 2010 John Wertenbaker, remoteshowcrate causes memory leak, ; changed in Skylark to just print out names of crates and not status. ; Crate status read directly from L-2 using N(30) F(1) A(0) ; ; ---------------------------------------------------- ; This will work with the GKA1xx, GKA2xx, GKA3xx format. ; It will NOT work if GKA is mixed with GKB, or GKB is mixed with GKC ; on-line=2, non-existant=1, and off-line=0 FUNCTION KeepAlive, CrateSlot ; This sets up a function that tests the link with a 367 module A0 = 0 F0 = 0 ; read F16 = 16 ; write data = 0 CAMPIOW,CrateSlot,A0,F0,data CAMPIOW,CrateSlot,A0,F16,data status = CAMXANDQ() RETURN, status END ; this initializes the program that enables CAputs ; spawn, "idl -e '.run ezcaIDL.pro'" ;********* THIS DOES NOT WORK ON SKYLARK!!!********* cainit camv2 stat100=strarr(65) ; Initialize stat100 variable array stat200=strarr(65) stat300=strarr(65) for loop2 = 1, 62 do begin ;this loop initalizes the stat array to "1" stat100(loop2)=1 stat200(loop2)=1 stat300(loop2)=1 endfor print print, 'CAMACMONITOR has been started' print Showcrates: count=1 nCrates = 0 SPAWN, 'remoteshowcrate', outputArray ;This has been set up to read the CTS table on MDSPC, ;it then sends results to outputArray nLines = N_ELEMENTS(outputArray) ; get # of lines for "mdscts sho/cra" isitonline = strArr(nLines) ; Initialize isitonline variable array FOR loop1 = 0, nLines-1 do begin isitonline(loop1) = 0 ; on line = 0, crate is assumed to be off unless proven otherwise CAMPIOW,strmid(outputarray(loop1),0,6)+':N30',0,1,ccstat ;read L-2 status if (ccstat and 16727239) eq 0 then isitonline(loop1) = 2 ;crate has been proven on line if (camx() ne 1) then isitonline(loop1) = 0 ;'NO X' means it's off line CAMPIOW,strmid(outputarray(loop1),0,6)+':N30',0,17,0 ;wake up crate test=strmid(outputarray(loop1),3,3)+0 ; set up to write status in proper place FOR loop2 = 1, 62 do begin if (test eq loop2+100) then stat100(loop2)=isitonline(loop1) if (test eq loop2+200) then stat200(loop2)=isitonline(loop1) if (test eq loop2+300) then stat300(loop2)=isitonline(loop1) if nlines lt 2 then stat100(loop2)=0 ; this flags all crates off if nlines lt 2 then stat200(loop2)=0 ; if remoteshowcrate is not working if nlines lt 2 then stat300(loop2)=0 ENDFOR ENDFOR summary100=1 ;set summary bit to indicate on line summary200=1 ;and if just 1 crate is off line, the bit goes to 0 summary300=1 for loop2 = 1, 62 do begin ; This loop CAPUT's crate status for each crate if (loop2 gt 9) then begin status=caput('ch_DAS0_'+strtrim(string(loop2),2)+'_LI',stat100(loop2)) status=caput('ch_DAS1_'+strtrim(string(loop2),2)+'_LI',stat200(loop2)) status=caput('ch_DAS2_'+strtrim(string(loop2),2)+'_LI',stat300(loop2)) endif if (loop2 lt 10) then begin status=caput('ch_DAS0_'+'0'+strtrim(string(loop2),2)+'_LI',stat100(loop2)) status=caput('ch_DAS1_'+'0'+strtrim(string(loop2),2)+'_LI',stat200(loop2)) status=caput('ch_DAS2_'+'0'+strtrim(string(loop2),2)+'_LI',stat300(loop2)) endif if stat100(loop2) eq 0 then summary100=0 ; set to 0 if it is off line if stat200(loop2) eq 0 then summary200=0 if stat300(loop2) eq 0 then summary300=0 stat100(loop2)=1 ; resets the stat100 variable after it has been used on the caput stat200(loop2)=1 stat300(loop2)=1 endfor status=caput('ch_DAS0_sts_LI',summary100) status=caput('ch_DAS1_sts_LI',summary200) status=caput('ch_DAS2_sts_LI',summary300) testlinkstatus: ; This writes to the 367's every 10 seconds to check the links Astatus = KeepAlive('GKA160:N1') ; will change to GKA260 if we get 3 SCSII cards Bstatus = KeepAlive('GKA261:N1') ; will change to GKB261 if we get 3 SCSII cards Cstatus = KeepAlive('GKA362:N1') ; will change to GKC262 if we get 3 SCSII cards WAIT, 10 ; pause for 10 seconds count=count+1 if count lt 7 then goto, testlinkstatus ;check link every 10 seconds, crates every 60 seconds if STRMID(SYSTIME(0), 15, 1) eq '0' and STRMID(SYSTIME(0), 14, 1) eq '0' then print, format='("CAMAC monitor is running as of ", A )', systime(0) ;This prints a reminder every hour that CAMAC monitor is still running ;For some reason, if you don't print something occasionally, the program blows up GOTO, Showcrates finish: end This started out being Phyllis's IDL program. You can read about it in the comments section in the first 60 or so lines of the program. I had to port it over to IDL on Skylark. Basically, I check the status of all the DAS crates every minute, send a wakeup command every minute, check the link every 10 seconds, and CAPUT statuses to EPICS. I tried to comment things as best as I can, but, since I'm not really a programmer, I know a real programmer can do better. Eventually this thing should probably be consolidated into one program in some other language than IDL, but Greg and I never had the time to do it. One other note: This blows up on the first of every month, when MDSPC is rebooted. I always have to stop IDL from TOP, and restart camacmonitorstart on the first of every month. John John Wertenbaker (jwertenb@pppl.gov) Princeton University Plasma Physics Laboratory Admin Wing, AD-111 MS-9 C-site (609)-243-3361 ====================================================================================================