The last elog didn't go out cleanly.  Modified the elogd.cfg to point to the proxy.  Try again.

Darkside people seem to be doing some tests at CERN next week.  It looks like they aren't going to use our DAQ (they will I think just use CAEN tools).  But we took 
opportunity to make TRIUMF DAQ work again.  There was a couple issues

1) I needed to inver the busy signal for input 0 to the chronobox in order to get any triggers out of the system.  Not sure why; I don't think I had to do that before.  But it is 
working now.

2) The CERN web proxy was not initially working.  Somehow I 'fixed' it by changing the proxy configuration variable of SERVICE_NAME from 'midas' to 'dummy'.  I don't 
understand why that fixed it.  But now you can see the DAQ here:

https://m-darkside.web.cern.ch/chronobox/

People:  Simone, Sam, Tom, Edgar

Tested IV curve python script that runs using MIDAS. Measured resistance consistant with resistor specification. elog:46/1

The resistance is attached to ch11, and we called the script (channels go from 0 to 24):
chnum=11 ; python3 iv_curve.py --no_steering --caen_chan $chnum --stop_v 3 --step_v 0.001 --filename ivcurve_`date +%s`_$chnum.txt 

Issues:

No readout of trip condiotion therefore python script overides power supply trip and attemps to bring back voltage to next value. Potentially dangerous. Script only terminates with the final voltage value. See attached plot elog:46/1

Script terminated with error message:

Traceback (most recent call last):
  File "iv_curve.py", line 287, in <module>
    iv.run()
  File "iv_curve.py", line 269, in run
    self.set_caen_and_wait_for_readback(voltage)
  File "iv_curve.py", line 223, in set_caen_and_wait_for_readback
    raise RuntimeError("CAEN HV didn't respond to request to set voltage to %s; latest readback voltage is %s" % (voltage, rdb))
RuntimeError: CAEN HV didn't respond to request to set voltage to 2.976; latest readback voltage is 0.052000001072883606

We need python3 package tkinter installed on ds-proto-daq in order to run the steering module GUI. 

I don't have root privileges.

I have installed the python3-tkinter package from yum. I checked that "import tkinter" works from a python3 prompt.

> V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) 

There was a problem with CMakeLists for the V1725 driver, and the frontend was only connecting to 1 board. This is now fixed, and all the boards should have the same (TTL) logic level.

We have tried to connect the steering module to the DAQ pc via Luigi's USB-Ethernet adapter. 

That does not appear to work. We'll receive another USB-Ethernet adapter tomorrow, and it should be configured to have a static IP address : 192.168.121.1 and NetMask 255.255.255.0
For the time being, we are using Tom's laptop. 

For a few days we were unable to access ds-proto-daq remotely. We were also unable to reach the outside world from ds-proto-daq.

To gain more control over the network settings, I have disabled the automatic configuration by dracut. I created the file /etc/dracut.conf.d/no-ifcfg.conf with the single line `omit_dracutmodules+="ifcfg"`.

I then set the following contents in /etc/sysconfig/network-scripts:

# cat ifcfg-enp0s31f6
NAME=enp0s31f6
DEVICE=enp0s31f6
ONBOOT=yes
NETBOOT=yes
UUID=4887c207-13da-4424-8307-116ef2163fd8
IPV6INIT=no
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
HWADDR=18:31:BF:2E:00:12
BOOTPROTO=dhcp
IPV6_PRIVACY=no
AUTOCONNECT_PRIORITY=1
DNS1=137.138.16.5
DNS2=137.138.17.5
PEERDNS=no


# cat ifcfg-enp5s0
TYPE=Ethernet
PROXY_METHOD=None
BROWSER_ONLY=no
BOOTPROTO=static
DHCPCLASS=
IPADDR=192.168.1.1
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
HWADDR=68:05:CA:8E:66:5C
ONBOOT=yes
DEVICE=enp5s0


Network settings are now correct, and have survived a reboot. We are able to talk to both the local network and the outside world. We are also able to connect to ds-proto-daq from lxplus, and via the web proxy.

After booting, the elog was only showing logs from before July, and was missing the "For Shifters" category. It appears we were loading the incorrect config file. The running command was:

/usr/sbin/elogd -D -c /etc/elogd.cfg -s /usr/share/elog/ -d /var/lib/elog/logbooks/

while it should have been:

/usr/sbin/elogd -D -x -c /home/dsproto/online/elog/elogd.cfg -s /usr/share/elog


rc.local was set up to run the latter command, but rc.local~ had the prior version. Perhaps that was being picked up. I have deleted the rc.local~ version, and will see if elog starts correctly next time we reboot.

This morning I installed the xrootd-client and python3-devel packages from yum. For the latter to work, I needed to do a yum update. This was long overdue, and updated 1200+ packages.

I have compiled a new version of ROOT 6 that links to Python 3. This will allow us to use ROOT and midas from the same python scripts.

I have created a new script that simplifies the adjustment of self-trigger thresholds (e.g. to change between "noise" runs with a low threshold and "normal" runs with a higher threshold).

It supports either setting the threshold to a certain number of ADC beneath baseline, or just shifting all the current thresholds by a given amount. It can also be used to just print the current baseline of each channel. Baseline calculation is done "live" by reading a few events.

The code is available in dsproto_analyzer, and is documented at https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation (scroll to "Adjusting self-trigger thresholds"). We used it recently to change to a noise run, and it worked well.

I've written a new script that will set all the V1725 baselines to the same level, by adjusting the DAC values. All the baselines are currently set to 15500.

The script is documented in the "Adjusting V1725 baselines" section of https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation.

I also added a new option to the "adjust self trigger threshold" script that allows the user to specify an exact ADC threshold (rather than doing it in relation to baseline). E.g. if you know that the baselines are all currently 15500, and you want to set a threshold ~200mV below, you could set a self-trigger threshold of 13900 (1mV is ~ 8ADC).

I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).

I made a new page that makes it easier to switch between laser/noise/physics runs. 

It handles setting the most common chronobox trigger settings, and the V1725 self-trigger thresholds. 
For the latter, it assumes that the baselines are at 15500. A suggested workflow would be to run
"equalize_baselines.py" in the morning to ensure all channels are at 15500, then you can use this
page to set the self-trigger thresholds for the rest of the day/week without using the python scripts.

The page is available at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type

When we start the HV program (CAEN_SY4527), it spews error messages of the form:
"16:58:30.035 2019/11/13 [CAEN_SY4527,ERROR] [dd_sy4527.cxx:280:fParam_get,ERROR] Param Not Found Type : 0"

This is because the LV module in the SY5527 crate doesn't support some of the parameters that the driver wants to read 
(e.g. it support "RUpTime" rather than "RUp"). For our current usage, the errors are benign. But it does add to the list 
of deficiencies of the driver provided by core midas:
* No automatic discovery of total number of channels
* Confusing ODB structure
* No discovery of which features a module supports
* Odd detection of whether a channel is on/off
* ODB status doesn't update if module settings are changed with a different interface (e.g. SSH or Caen's Java-based GUI).

All these limitations are imposed by the core HV driver "class" in midas. Assuming we will be using Caen HV modules for the
foreseeable future, I think I should write a new slow control frontend that is less generic, but fixes all the above issues.

A few months ago I created a new midas frontend for CAEN HV crates. It behaves more like what users expect - changes made through other interfaces (e.g. ssh or the Java-based GUI) will be reflected in the ODB. The ODB structure is created dynamically based on which modules are present in the crate, and the parameters that each module supports (e.g. whether you specify a ramp-up rate in V/s or a ramp-up time in s).

When I first wrote the frontend, ds-proto-daq was offline. Pierre reminded me today that I should actually deploy the new code.

The new program code is at ~/online/dsproto_sy4527/caen_hv*. The executable is at ~/online/bin/caenhv_fe. The frontend is available on the "Programs" page in mhttpd. If we want to run the old HV frontend, it is still available at ~/online/bin/sy4527.

Connection parameters for the frontend are found in the ODB at /Equipment/CAEN_HV/Settings/Global. Currently the crate appears to be off, so the frontend will refuse to start properly as it cannot connect to 192.168.1.9. If the frontend is started with the crate on, it will create more directories in /Equipment/CAEN_HV/Settings (one for each module). The "Voltage" page on the experiment now points to a generic webpage that will work with arbitrary combinations of modules.

Somehow the trigger rate was not matching the trigger source.
Find out that Link 3 was not collecting and possibly holding the fragment assembly in the main thread.
Swap Link3 <-> Link0 on the V1725, restarted.
Needs further investigation!

Date rate is fine now! CPU load is balanced on all 4 threads (~25%)
- irqbalance disabled
- change affinity for A3818 to cpu9: /etc/rc.local add: echo 0200 > /proc/irq/136/smp_affinity
  Check : watch -n 0.1 'cat /proc/interrupts'

Maximum Trigger rate (HW buffer not rising) 1950 Evt/s => 200MB/s 
for event size of 100KB composed of 4 banks with 32us per channel.

Summary of setup of chronobox on network (mostly done by Pierre):

1) Hook up USB connection from chronobox to ds-proto-daq.  Start serial-USB connection by doing

/home/dsproto/online/dsproto_daq/serialusb

can login through serial link.  Added the ssh key for dsproto@ds-proto-daq to the authorized_keys file on
chronobox linux.

ctrl-a, ctrl-x to stop picocom

2) On ds-proto-daq, setup dhcpd server.  Configuration in this file

/etc/dhcp/dhcpd.conf

dhpcd is bond to the second NIC only.  Configure DHCP to give IP 192.168.1.3 to chronobox.  In /etc/hosts set
192.168.1.3 to have hostname chronobox

Power cycle chronobox; it successfully gets IP:

[dsproto@ds-proto-daq dsproto_daq]$ ping chronobox
PING chronobox (192.168.1.3) 56(84) bytes of data.
64 bytes from chronobox (192.168.1.3): icmp_seq=1 ttl=64 time=0.180 ms

3) Can see the chronobox webpage locally as

http://chronobox:8080

4) Some esper tool to access data:  did following to setup esper

  734  yum install python2-pip
  735  pip install esper-tool
  736  yum install ncurses
  738  yum install ncurses-devel
  739  pip install esper-tool

Can then connect to the chronobox by doing

esper-tool interactive http://chronobox:8080

This is as much as I understand at this point... more exploring now...

5) Pierre unplugged the SD card to take it back to Bryerton's room.  But I guess this was bad.

lots of errors on serialUSB link now and the webpage doesn't work anymore:

[ 1246.665269] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.904770] EXT4-fs error: 339 callbacks suppressed
[ 1246.679162] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.904780] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: inode #2: com[ 1246.692884] blk_partition_remap: fail for partition 2
m syslogd: reading directory lblock 0
[ 1246.709020] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.929199] blk_partition_remap: fail for partition 2
[ 1246.717394] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.942971] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: inode #2: com[ 1246.731255] blk_partition_remap: fail for partition 2
m syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.953760] blk_partition_remap: fail for [ 1246.747337]
blk_partition_remap: fail for partition 2
partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.967532] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: [ 1246.763481] blk_partition_remap: fail for partition 2
inode #2: comm syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.978320] blk_partition_rem[ 1246.779511] blk_partition_remap:
fail for partition 2
ap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.992091] EXT4-fs error (device mmcblk0p2): ext4_find[
1246.795586] blk_partition_remap: fail for partition 2
_entry:1437: inode #2: comm syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1243.002885] blk_[ 1246.811671] blk_partition_remap: fail for
partition 2
partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1243.016656] EXT4-fs error (device mmcblk0p[ 1246.827750]
blk_partition_remap: fail for partition 2

The SDcard is currently required for operation of the device, the ext3/4 filesystem will immediately fail upon removal.


> Summary of setup of chronobox on network (mostly done by Pierre):
> 
> 1) Hook up USB connection from chronobox to ds-proto-daq.  Start serial-USB connection by doing
> 
> /home/dsproto/online/dsproto_daq/serialusb
> 
> can login through serial link.  Added the ssh key for dsproto@ds-proto-daq to the authorized_keys file on
> chronobox linux.
> 
> ctrl-a, ctrl-x to stop picocom
> 
> 2) On ds-proto-daq, setup dhcpd server.  Configuration in this file
> 
> /etc/dhcp/dhcpd.conf
> 
> dhpcd is bond to the second NIC only.  Configure DHCP to give IP 192.168.1.3 to chronobox.  In /etc/hosts set
> 192.168.1.3 to have hostname chronobox
> 
> Power cycle chronobox; it successfully gets IP:
> 
> [dsproto@ds-proto-daq dsproto_daq]$ ping chronobox
> PING chronobox (192.168.1.3) 56(84) bytes of data.
> 64 bytes from chronobox (192.168.1.3): icmp_seq=1 ttl=64 time=0.180 ms
> 
> 3) Can see the chronobox webpage locally as
> 
> http://chronobox:8080
> 
> 4) Some esper tool to access data:  did following to setup esper
> 
>   734  yum install python2-pip
>   735  pip install esper-tool
>   736  yum install ncurses
>   738  yum install ncurses-devel
>   739  pip install esper-tool
> 
> Can then connect to the chronobox by doing
> 
> esper-tool interactive http://chronobox:8080
> 
> This is as much as I understand at this point... more exploring now...
> 
> 5) Pierre unplugged the SD card to take it back to Bryerton's room.  But I guess this was bad.
> 
> lots of errors on serialUSB link now and the webpage doesn't work anymore:
> 
> [ 1246.665269] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.904770] EXT4-fs error: 339 callbacks suppressed
> [ 1246.679162] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.904780] EXT4-fs error (device mmcblk0p2):
> ext4_find_entry:1437: inode #2: com[ 1246.692884] blk_partition_remap: fail for partition 2
> m syslogd: reading directory lblock 0
> [ 1246.709020] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.929199] blk_partition_remap: fail for partition 2
> [ 1246.717394] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.942971] EXT4-fs error (device mmcblk0p2):
> ext4_find_entry:1437: inode #2: com[ 1246.731255] blk_partition_remap: fail for partition 2
> m syslogd: reading directory lblock 0
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.953760] blk_partition_remap: fail for [ 1246.747337]
> blk_partition_remap: fail for partition 2
> partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.967532] EXT4-fs error (device mmcblk0p2):
> ext4_find_entry:1437: [ 1246.763481] blk_partition_remap: fail for partition 2
> inode #2: comm syslogd: reading directory lblock 0
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.978320] blk_partition_rem[ 1246.779511] blk_partition_remap:
> fail for partition 2
> ap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.992091] EXT4-fs error (device mmcblk0p2): ext4_find[
> 1246.795586] blk_partition_remap: fail for partition 2
> _entry:1437: inode #2: comm syslogd: reading directory lblock 0
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1243.002885] blk_[ 1246.811671] blk_partition_remap: fail for
> partition 2
> partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1243.016656] EXT4-fs error (device mmcblk0p[ 1246.827750]
> blk_partition_remap: fail for partition 2

a) Bryerton implemented new version of the firmware.  New features:

1) run start/stop state
2) at run start 6 events are sent with 200ms separation
3) there is a greater set of counters about trigger, as well as configuring the trigger.

b) chronobox webpage can be seen here:

https://ds-proto-daq.triumf.ca/chronobox/

The mod_tdm page gives configuration of run state and trigger.  In particular

- button to start/stop run
- button to do manual trigger
- configure which channels are TOP and which are BOTTOM
- configure the number of TOP or BOTTOM channels that need to figure
- DecisionType: true= TOP AND BOTTOM groups must fire; false = TOP OR BOTTOM groups can fire.

c) Start and stop run can also be done on command line with esper tool:

esper-tool write -d true 192.168.1.3 mod_tdm run
esper-tool write -d false 192.168.1.3 mod_tdm run

d) I modified the V1725 frontend to integrate the chronobox start/stop.

At run start
- configure V1725s
- start chronobox with the command-line esper-tool call

At end run
- add deferred transition function which stops run (with esper-tool), then checks whether all the events have
cleared from ring buffers.
- once ring buffers are cleared, finish stopping the V1725s.

In the long run should somehow do the start/stop commands with some http post command, rather than command line
to esper-tool.

e) With the start/stop run could start to compare the timestamps of V1725s and confirm that they matched (ie,
all V1725s got reset at the same time).  Also, all the events are cleared from buffers.

f) However, I found that the frontend program still consistently failed with this error when the trigger rate
was above the maximum sustainable:

Deferred transition.  First call of wait_buffer_empty. Stopping run
[feov1725MTI00,ERROR] [v1725CONET2.cxx:685:ReadEvent,ERROR] Communication error: -2
[feov1725MTI00,ERROR] [feoV1725.cxx:654:link_thread,ERROR] Readout routine error on thread 0 (module 0)
[feov1725MTI00,ERROR] [feoV1725.cxx:655:link_thread,ERROR] Exiting thread 0 with error
Stopped chronobox run; status = 0
Segmentation fault


Next steps:

1) Fix the seg-fault for high rate running.
2) More detailed timestamp checking, with cm_msg(ERRORs)
3) Bryerton is now working on the event FIFO on chronobox.  That will be next thing to integrate.