Lazy logger has caught up with the backlog, and continues to transfer new files to darkside01 as they are written.
EOS currently contains data for runs 822-1201.

I have deleted the following runs from ds-proto-daq:
* runs < 700 that were taken with gzip compression
* runs 800-1100 that are available on EOS and darkside01

The following files remain:
* runs 700-800 that haven't been copied to EOS (but are on darkside01). If they're of interest, they should be copied to EOS.
* runs 1200+ that haven't been copied to EOS yet (but will be soon).

Disk usage has improved from 85% to 33%.

When we start the HV program (CAEN_SY4527), it spews error messages of the form:
"16:58:30.035 2019/11/13 [CAEN_SY4527,ERROR] [dd_sy4527.cxx:280:fParam_get,ERROR] Param Not Found Type : 0"

This is because the LV module in the SY5527 crate doesn't support some of the parameters that the driver wants to read 
(e.g. it support "RUpTime" rather than "RUp"). For our current usage, the errors are benign. But it does add to the list 
of deficiencies of the driver provided by core midas:
* No automatic discovery of total number of channels
* Confusing ODB structure
* No discovery of which features a module supports
* Odd detection of whether a channel is on/off
* ODB status doesn't update if module settings are changed with a different interface (e.g. SSH or Caen's Java-based GUI).

All these limitations are imposed by the core HV driver "class" in midas. Assuming we will be using Caen HV modules for the
foreseeable future, I think I should write a new slow control frontend that is less generic, but fixes all the above issues.

Run 1186

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 100 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 10k

V1725 offline, run stopped manually

Run 1188

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 100 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 70k

Trigger rate became very low, run stopped manually

Run 1189

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1190

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  10

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1191

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  15

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1192

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1193

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  10

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1194

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  15

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1195

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 10k

I have changed `/Logger/Channels/0/Settings/Filename` to start with a dot, so files will be hidden until they are complete. 
The current value is `.run%05dsub%03d.mid`.

This will help Pablo with his script for transferring files to EOS.

Run 1016

Gas pocket: ON (thinkness 7mm)

Fields: Off

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 201 mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1017

Gas pocket: ON (thinkness 7mm)

Fields:  drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 201 mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1018

Gas pocket: ON (thinkness 7mm)

Fields:  drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 217 mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1019

Gas pocket: ON (thinkness 7mm)

Fields:  drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 240

SiPM HV: 65 V

Number of Events: 10K

Run 1020 Laser Run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Comment: Recieved this error message but run seemed to continue ok and data appeared to be written: 19:56:04.839 2019/11/09 [feov1725MTI00,ERROR] [feoV1725.cxx:1048:read_trigger_event,ERROR] Error: did not receive a ZMQ bank after 100.000000 ms. Stopping run.   â•³

Run 1021 Laser Run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Attemted the following run, and it started but the rate was shoing 0 and it got stuck on 5 events

Was unable to connect to Chronobox webpage even after turning VME crate off and on.

Abandoned run and will attempt to fix in the morning.

Run 1073

Gas pocket: ON (thickness unknown)

Fields:  drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1105

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  265 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1106

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1107

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1108

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  242 mbarg

SiPM HV: 65 V

Number of events: 10k

Comment: Webpage of Chronobox doens't work, skipped Laser runs.

Not sure if the trigger threshold is correctly set, there is an error:

Traceback (most recent call last):
  File "/home/dsproto/online/dsproto_analyzer/threshold_tool.py", line 71, in <module>
    main()
  File "/home/dsproto/online/dsproto_analyzer/threshold_tool.py", line 47, in main
    (mean, rms) = fit_res
ValueError: too many values to unpack (expected 2)

the trigger threshold for all the runs in this Elog is 2699 ADCc below the baseline (see Elog 89)
 

Run 943

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 945

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 946

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 948

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 949

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 951

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 952

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 954

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

> Not sure if the trigger threshold is correctly set, there is an error:

Sorry, I made a change to the baseline calculation and forgot to update the threshold tool. It's updated now.

The thresholds currently seem to be set to 12801 - this is 2699 below the nominal baseline of 15500.

Confirmed this ETTT configuration is working.

                     ETTT Enabled [22..21] = b10
                      |
                      v
Data [0x811C] = 0x 00 4 D 013C

                ETTT Time [47..32]
                 |  Ch Mask[16..0]  Time[31..0]
                 |    TTTTTT           |
Header 1         v    v    v           v
0xa0001914 0x00 0025 ff 0xff1ca598 0xe9c6e8a1   < event 1
0xa0001914 0x00 0025 ff 0xff1ca599 0xe9c82e25   < event 2

dTime :  0x25e9c82e25 &#8722; 0x25e9c6e8a1 = 0x14584 => 83332
Time interval: 8ns  => 666.7e-6s => Freq: 1500Hz corresponding the current trigger rate

PAA

I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

Notes on day:

1) Fixed the problem with the network interfaces.  Now the computer boots with the correct network configuration; outside world visible 
and private network on.

2) fan tray on VME crate seemed to be broken.  Got another VME crate from pool and installed it.  This VME crate seems to be working 
well. 

3) Recommissioned the DAQ setup.  Found a couple small bugs related to the V1725 self-trigger logic.  Fixed those and the V1725 self-
triggers seem to be working correctly.

4) Tried to install new CDM from TRIUMF (with ssh access), but clocks didn't stay synchronized.  Will bring module back to TRIUMF.

5) Added some code to V1725 frontend for clearing out the ZMQ buffers of extra events at the end of the run.  This is to protect 
against the case where the chronobox is triggering too fast for the V1725s.

Several points:

1) Gave a series of tutorial on DAQ to DS people yesterday and today.  Got a bunch of feedback, which I will pass on when I'm back at TRIUMF.

2) The computer ds-proto-daq was offline when I got in to lab this morning.  Hmm, not clear what is wrong with computer.  Didn't happen the 
first day.  Maybe another power blip?  Maybe we need a UPS for this DAQ machine, to protect it from power blips.

3) Using instructions from Luke, reconfigured the CDM to use the clock from the chronobox. 

4) Added scripts for putting the chronobox and the CDM into a sensible state.  Scripts are 

/home/dsproto/online/dsproto_daq/setup_chronobox.sh
/home/dsproto/online/dsproto_daq/setup_cdm.sh

The scripts need to be rerun whenever the chronobox or VME crate are power cycled.

5) Fix some bugs and added some new plots to online monitoring.  In particular, added a bunch of plots related to the chronobox data.

6) Found some problems with monitoring of chronobox trigger primitives, which I passed onto Bryerton.

The problem reported in ID 117 is somehow fixed this morning.

Temp at start: 

A: 242.7K    B: 278.16K    C: 86.4K    D: 240.89K

Temp at End:

A: 248.87K    B: 278.24K    C: 86.37K    D: 247.75K

Took iv curves, data is on DAQ PC here: /home/dsproto/online/dsproto_sy4527/ivdata_191120

IV of 25 SiPM put in ivdata_191119 directory

Increased drift and extraction fields.

Current settings: wire gate 5100 V, 1st ring: 6100 V, cathode: 11100 V

P.S. Increased by in the morning by Yi.

I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).

I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.

I installed a PCIe 1Gbps network card and configured it as a private network.  The PC (ds-proto-daq) is
192.168.1.1.  I guess we can make the chronobox 192.168.1.2.

[root@ds-proto-daq ~]# ifconfig enp5s0
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::bb27:5db:f778:d584  prefixlen 64  scopeid 0x20<link>
        ether 68:05:ca:8e:66:5c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 4145 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf72c0000-f72e0000