We need python3 package tkinter installed on ds-proto-daq in order to run the steering module GUI. 

I don't have root privileges.

I have installed the python3-tkinter package from yum. I checked that "import tkinter" works from a python3 prompt.

> V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) 

There was a problem with CMakeLists for the V1725 driver, and the frontend was only connecting to 1 board. This is now fixed, and all the boards should have the same (TTL) logic level.

We have tried to connect the steering module to the DAQ pc via Luigi's USB-Ethernet adapter. 

That does not appear to work. We'll receive another USB-Ethernet adapter tomorrow, and it should be configured to have a static IP address : 192.168.121.1 and NetMask 255.255.255.0
For the time being, we are using Tom's laptop. 

For a few days we were unable to access ds-proto-daq remotely. We were also unable to reach the outside world from ds-proto-daq.

To gain more control over the network settings, I have disabled the automatic configuration by dracut. I created the file /etc/dracut.conf.d/no-ifcfg.conf with the single line `omit_dracutmodules+="ifcfg"`.

I then set the following contents in /etc/sysconfig/network-scripts:

# cat ifcfg-enp0s31f6
NAME=enp0s31f6
DEVICE=enp0s31f6
ONBOOT=yes
NETBOOT=yes
UUID=4887c207-13da-4424-8307-116ef2163fd8
IPV6INIT=no
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
HWADDR=18:31:BF:2E:00:12
BOOTPROTO=dhcp
IPV6_PRIVACY=no
AUTOCONNECT_PRIORITY=1
DNS1=137.138.16.5
DNS2=137.138.17.5
PEERDNS=no


# cat ifcfg-enp5s0
TYPE=Ethernet
PROXY_METHOD=None
BROWSER_ONLY=no
BOOTPROTO=static
DHCPCLASS=
IPADDR=192.168.1.1
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
HWADDR=68:05:CA:8E:66:5C
ONBOOT=yes
DEVICE=enp5s0


Network settings are now correct, and have survived a reboot. We are able to talk to both the local network and the outside world. We are also able to connect to ds-proto-daq from lxplus, and via the web proxy.

After booting, the elog was only showing logs from before July, and was missing the "For Shifters" category. It appears we were loading the incorrect config file. The running command was:

/usr/sbin/elogd -D -c /etc/elogd.cfg -s /usr/share/elog/ -d /var/lib/elog/logbooks/

while it should have been:

/usr/sbin/elogd -D -x -c /home/dsproto/online/elog/elogd.cfg -s /usr/share/elog


rc.local was set up to run the latter command, but rc.local~ had the prior version. Perhaps that was being picked up. I have deleted the rc.local~ version, and will see if elog starts correctly next time we reboot.

This morning I installed the xrootd-client and python3-devel packages from yum. For the latter to work, I needed to do a yum update. This was long overdue, and updated 1200+ packages.

I have compiled a new version of ROOT 6 that links to Python 3. This will allow us to use ROOT and midas from the same python scripts.

I have created a new script that simplifies the adjustment of self-trigger thresholds (e.g. to change between "noise" runs with a low threshold and "normal" runs with a higher threshold).

It supports either setting the threshold to a certain number of ADC beneath baseline, or just shifting all the current thresholds by a given amount. It can also be used to just print the current baseline of each channel. Baseline calculation is done "live" by reading a few events.

The code is available in dsproto_analyzer, and is documented at https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation (scroll to "Adjusting self-trigger thresholds"). We used it recently to change to a noise run, and it worked well.

I've written a new script that will set all the V1725 baselines to the same level, by adjusting the DAC values. All the baselines are currently set to 15500.

The script is documented in the "Adjusting V1725 baselines" section of https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation.

I also added a new option to the "adjust self trigger threshold" script that allows the user to specify an exact ADC threshold (rather than doing it in relation to baseline). E.g. if you know that the baselines are all currently 15500, and you want to set a threshold ~200mV below, you could set a self-trigger threshold of 13900 (1mV is ~ 8ADC).

I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).

I made a new page that makes it easier to switch between laser/noise/physics runs. 

It handles setting the most common chronobox trigger settings, and the V1725 self-trigger thresholds. 
For the latter, it assumes that the baselines are at 15500. A suggested workflow would be to run
"equalize_baselines.py" in the morning to ensure all channels are at 15500, then you can use this
page to set the self-trigger thresholds for the rest of the day/week without using the python scripts.

The page is available at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type

When we start the HV program (CAEN_SY4527), it spews error messages of the form:
"16:58:30.035 2019/11/13 [CAEN_SY4527,ERROR] [dd_sy4527.cxx:280:fParam_get,ERROR] Param Not Found Type : 0"

This is because the LV module in the SY5527 crate doesn't support some of the parameters that the driver wants to read 
(e.g. it support "RUpTime" rather than "RUp"). For our current usage, the errors are benign. But it does add to the list 
of deficiencies of the driver provided by core midas:
* No automatic discovery of total number of channels
* Confusing ODB structure
* No discovery of which features a module supports
* Odd detection of whether a channel is on/off
* ODB status doesn't update if module settings are changed with a different interface (e.g. SSH or Caen's Java-based GUI).

All these limitations are imposed by the core HV driver "class" in midas. Assuming we will be using Caen HV modules for the
foreseeable future, I think I should write a new slow control frontend that is less generic, but fixes all the above issues.

A few months ago I created a new midas frontend for CAEN HV crates. It behaves more like what users expect - changes made through other interfaces (e.g. ssh or the Java-based GUI) will be reflected in the ODB. The ODB structure is created dynamically based on which modules are present in the crate, and the parameters that each module supports (e.g. whether you specify a ramp-up rate in V/s or a ramp-up time in s).

When I first wrote the frontend, ds-proto-daq was offline. Pierre reminded me today that I should actually deploy the new code.

The new program code is at ~/online/dsproto_sy4527/caen_hv*. The executable is at ~/online/bin/caenhv_fe. The frontend is available on the "Programs" page in mhttpd. If we want to run the old HV frontend, it is still available at ~/online/bin/sy4527.

Connection parameters for the frontend are found in the ODB at /Equipment/CAEN_HV/Settings/Global. Currently the crate appears to be off, so the frontend will refuse to start properly as it cannot connect to 192.168.1.9. If the frontend is started with the crate on, it will create more directories in /Equipment/CAEN_HV/Settings (one for each module). The "Voltage" page on the experiment now points to a generic webpage that will work with arbitrary combinations of modules.

IV of 25 SiPM put in ivdata_191119 directory

I have added a backup of the CERN proto-0 ELOG at TRIUMF, in case ds-proto-daq is unavailable in future.

The backup ELOG is at https://ladd00.triumf.ca/elog-ds/. There are other logbooks hosted in the same place (requiring a username/password), but the CERN backup logbooks (at the bottom of the list) are available to all. I have configured things so that the TRIUMF replicas are read-only. To add new entries one must still use the CERN logbooks.

The logbooks are rsynced every hour via cron.

/home was getting very full on ds-proto-daq, so I moved the history files onto the /data disk. There is now a symlink from ~/online/history to /data/dsproto/history/.

I have changed `/Logger/Channels/0/Settings/Filename` to start with a dot, so files will be hidden until they are complete. 
The current value is `.run%05dsub%03d.mid`.

This will help Pablo with his script for transferring files to EOS.

frontend process still alive but slow, causing lot of rejected triggers.

stopped and restarted the feov1725 process to fix this

Error during data taking  1573416708 21:11:48.151 2019/11/10 [feov1725MTI00,INFO] V1725 PLL loss lock Board:0 (vmeAcq=0x0)

It stop saving event but run in process. Impossible to stop the run. I tried to reset feov1725MTl00. After that error: 

1573416857 21:14:17.792 2019/11/10 [feChronoEsper,INFO] Client 'feov1725MTI00' on database 'ODB' removed by db_cleanup called by cm_periodic_tasks because pid 9724 does not exist

Run stopped after some time. Error 21:21:19.121 2019/11/10 [mhttpd,ERROR] [midas.cxx:3951:cm_transition_call,ERROR] cannot connect to client "feov1725MTI00" on host localhost, port 40944, status 503   ╳

Trying to restart feov1725MTI00. Status initializing

It is not possible to initialize 

The connection to rootana is lost.

No data taking can be started.