Gas pocket: ON

Fields: wire gate 3780 V, 1st ring: 4500 V, cathode: 7000 V

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 300 mbarg

SiPM HV: 65 V (noise run)

Comment: noise conditions good. Rate ~30Hz. Look for S2s!

I have installed the python3-tkinter package from yum. I checked that "import tkinter" works from a python3 prompt.

> V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) 

There was a problem with CMakeLists for the V1725 driver, and the frontend was only connecting to 1 board. This is now fixed, and all the boards should have the same (TTL) logic level.

For a few days we were unable to access ds-proto-daq remotely. We were also unable to reach the outside world from ds-proto-daq.

To gain more control over the network settings, I have disabled the automatic configuration by dracut. I created the file /etc/dracut.conf.d/no-ifcfg.conf with the single line `omit_dracutmodules+="ifcfg"`.

I then set the following contents in /etc/sysconfig/network-scripts:

# cat ifcfg-enp0s31f6
NAME=enp0s31f6
DEVICE=enp0s31f6
ONBOOT=yes
NETBOOT=yes
UUID=4887c207-13da-4424-8307-116ef2163fd8
IPV6INIT=no
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
HWADDR=18:31:BF:2E:00:12
BOOTPROTO=dhcp
IPV6_PRIVACY=no
AUTOCONNECT_PRIORITY=1
DNS1=137.138.16.5
DNS2=137.138.17.5
PEERDNS=no


# cat ifcfg-enp5s0
TYPE=Ethernet
PROXY_METHOD=None
BROWSER_ONLY=no
BOOTPROTO=static
DHCPCLASS=
IPADDR=192.168.1.1
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
HWADDR=68:05:CA:8E:66:5C
ONBOOT=yes
DEVICE=enp5s0


Network settings are now correct, and have survived a reboot. We are able to talk to both the local network and the outside world. We are also able to connect to ds-proto-daq from lxplus, and via the web proxy.

After booting, the elog was only showing logs from before July, and was missing the "For Shifters" category. It appears we were loading the incorrect config file. The running command was:

/usr/sbin/elogd -D -c /etc/elogd.cfg -s /usr/share/elog/ -d /var/lib/elog/logbooks/

while it should have been:

/usr/sbin/elogd -D -x -c /home/dsproto/online/elog/elogd.cfg -s /usr/share/elog


rc.local was set up to run the latter command, but rc.local~ had the prior version. Perhaps that was being picked up. I have deleted the rc.local~ version, and will see if elog starts correctly next time we reboot.

This morning I installed the xrootd-client and python3-devel packages from yum. For the latter to work, I needed to do a yum update. This was long overdue, and updated 1200+ packages.

I have compiled a new version of ROOT 6 that links to Python 3. This will allow us to use ROOT and midas from the same python scripts.

/home was getting very full on ds-proto-daq, so I moved the history files onto the /data disk. There is now a symlink from ~/online/history to /data/dsproto/history/.

I have created a new script that simplifies the adjustment of self-trigger thresholds (e.g. to change between "noise" runs with a low threshold and "normal" runs with a higher threshold).

It supports either setting the threshold to a certain number of ADC beneath baseline, or just shifting all the current thresholds by a given amount. It can also be used to just print the current baseline of each channel. Baseline calculation is done "live" by reading a few events.

The code is available in dsproto_analyzer, and is documented at https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation (scroll to "Adjusting self-trigger thresholds"). We used it recently to change to a noise run, and it worked well.

The V1725s have been changed to have board config 0x50 rather than 0x10. This means that they will now trigger on the leading edge of the pulse rather than the tailing edge. There is now much less jitter in the location of pulses in the digitized waveforms.

I've written a new script that will set all the V1725 baselines to the same level, by adjusting the DAC values. All the baselines are currently set to 15500.

The script is documented in the "Adjusting V1725 baselines" section of https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation.

I also added a new option to the "adjust self trigger threshold" script that allows the user to specify an exact ADC threshold (rather than doing it in relation to baseline). E.g. if you know that the baselines are all currently 15500, and you want to set a threshold ~200mV below, you could set a self-trigger threshold of 13900 (1mV is ~ 8ADC).

The CAEN VME crate now gets its IP address from the ds-proto-daq DHCP server. To do this, I set the IP address of the crate to 0.0.0.0 (using the front panel interface) then reset the crate (not just a power cycle - pressing the reset button is required).

I am now able to ping the crate, but am NOT able to load the webpage it is supposed to serve (I just get a "connection refused"). I can't see any configuration settings that would disable the webpage or would cause it to be served on a non-standard port. I may have missed something though.

> Not sure if the trigger threshold is correctly set, there is an error:

Sorry, I made a change to the baseline calculation and forgot to update the threshold tool. It's updated now.

The thresholds currently seem to be set to 12801 - this is 2699 below the nominal baseline of 15500.

I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).

I made a new page that makes it easier to switch between laser/noise/physics runs. 

It handles setting the most common chronobox trigger settings, and the V1725 self-trigger thresholds. 
For the latter, it assumes that the baselines are at 15500. A suggested workflow would be to run
"equalize_baselines.py" in the morning to ensure all channels are at 15500, then you can use this
page to set the self-trigger thresholds for the rest of the day/week without using the python scripts.

The page is available at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type

I have changed `/Logger/Channels/0/Settings/Filename` to start with a dot, so files will be hidden until they are complete. 
The current value is `.run%05dsub%03d.mid`.

This will help Pablo with his script for transferring files to EOS.

The analysis server from Roma (darkside01) is now connected to the local network. It is not yet connected to CERN.

There are 2 network ports on the rear of the machine:
* The port labelled "1":
 * MAC: 9c:71:3a:22:d5:62
 * Name: enp2s0f0
 * Configured for DHCP
 * Connected to local switch
 * Assigned IP 192.168.1.10 by ds-proto-daq
* The port labelled "2":
 * MAC: 9c:71:3a:22:d5:63
 * Name: enp2s0f1
 * Configured for DHCP
 * Should be connected to CERN network

I have set up password-less ssh between ds-proto-daq and darkside01. 

There are 3 problems connecting to the CERN network:
1) The long cable that was created does not work. I tried using it to connect ds-proto-daq to socket 0001/06 (i.e. the socket ds-proto-daq is normally connected to) and there is no link light - the cable is broken.
2) Socket 0001/07 does not seem to be operational. I tried connecting ds-proto-daq to socket 0001/07 using the cable we know works, and there was no link light. Either CERN have disabled the connection intentionally, or it is broken.
3) When I used the good cable to connect to the good socket (0001/06), CERN claimed the machine was unregistered. Marco told me that the interfaces have MACs 9c:71:3a:22:d6:4[ab] (rather than 9c:71:3a:22:d5:6[23]), so presumably the wrong MAC addresses were registered with CERN.

So, we need to register the correct MACs, fix the cable, and either get 0001/07 working or connect to a different socket.

I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

The cable has been fixed. Socket 0001/007 is still not active.

It looks like there's a memory leak in the high voltage driver, and we were running out of memory (using a lot of swap). That *may* be related to the recent instability, but I'm not sure.

If things are more stable now, then the memory leak was probably the cause. If not, then we should try stopping the V1725 program, power-cycling the VME crate, then starting the V1725 program.


Yi: restarted the VME crate twice, now it seems like smooth.

When we start the HV program (CAEN_SY4527), it spews error messages of the form:
"16:58:30.035 2019/11/13 [CAEN_SY4527,ERROR] [dd_sy4527.cxx:280:fParam_get,ERROR] Param Not Found Type : 0"

This is because the LV module in the SY5527 crate doesn't support some of the parameters that the driver wants to read 
(e.g. it support "RUpTime" rather than "RUp"). For our current usage, the errors are benign. But it does add to the list 
of deficiencies of the driver provided by core midas:
* No automatic discovery of total number of channels
* Confusing ODB structure
* No discovery of which features a module supports
* Odd detection of whether a channel is on/off
* ODB status doesn't update if module settings are changed with a different interface (e.g. SSH or Caen's Java-based GUI).

All these limitations are imposed by the core HV driver "class" in midas. Assuming we will be using Caen HV modules for the
foreseeable future, I think I should write a new slow control frontend that is less generic, but fixes all the above issues.