We still have the noise issue.

Run 1219

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1220

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1221

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1222

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1223

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1224

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1226

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 40k

Run 1228

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 15k

Run 1229

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 40k

"ERROR INFO: 10:38:10.924 2019/11/09 [feov1725MTI00,ERROR] [feoV1725.cxx:1048:read_trigger_event,ERROR] Error: did not receive a ZMQ bank after 100.000000 ms. Stopping run."

Emailed to Ben.

Run 999

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  233 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1000

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  235 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1001

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  238 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1002

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  238 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1003 Laser run (manually stopped)

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Comment: Triger rate too low (~8 Hz), manually stopped

Run 1004 Laser run (manually stopped)

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Comment: Triger rate too low (~8 Hz), manually stopped

Run 1105

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  265 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1106

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1107

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1108

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  242 mbarg

SiPM HV: 65 V

Number of events: 10k

Comment: Webpage of Chronobox doens't work, skipped Laser runs.

Drive folder with data files and some pictures:  https://drive.google.com/drive/folders/1Sr27kQTOO2kciBY7v-wYBP06tWLVEjiP

Lab notes @ CERN

11/10/2020

11:00 – First look at MB2.  Noticed on some of the PDMs that some of the tile PCB are more delaminated than others.  PDM 25 also had the FEB connectors more inserted than any other PDMs.  No other differences noticed.

11:30 – Connected SMU and switching matrix to MB2.  Power w/o steering module.  Checked that all channels match by running photo current tests with and w/o a flashlight on one corner.  Using the computer form Pisa with Labview scripts from Matteo.  Everything is ok.

12:00 – Replace plastic shields around Proto-0 for IV curve measurements in dark conditions.  PDM 1 breakdown is around 63V.  Looks ok.  PDM 2 is over 65V…. Seems high.

12:30 - PDM 6 People walking, disrupted the light conditions.  IV curve will be jagged.  Will go for lunch.  Closing door and turning off exterior lights.  Current will decrease

15:30 – IV curves look fine.  …/MB2_retriveval_CERN/data_files/iv_warm_1

16:00 – Took forward IV curves.  Saved as …/MB2_retriveval_CERN/data_files/forward_warm_1.  They look fine.  Checked old pictures of the lamination of the tile PCB and they have always varied in thickness, from before the LNGS LN test.

16:45 – Power FEBs on to look at noise spectra with 20V bias on the tiles.  

18:00 – PDM 2, 4, 6, and 25 have very odd power spectra.  George said they look like they’re not connected.

18:30 – Saving FFT to disk for analysis.  Saved as …/MB2_retriveval_CERN/data_files/noise_warm_1

19:00 – PDM 2,4,6 and 25 have one of the differential lines disconnected.  Using a light and looking at the differential outputs on the scope, this is obvious.  20V bias and FEBs powered at 5V.  Saved as …/MB2_retriveval_CERN/data_files/waveform_light_1

19:30 – Started another round of reverse IV curves in dark conditions.  Saved as …/MB2_retriveval_CERN/data_files/iv_warm_2

21:15 – IV still going.  EP will come back to shut off the LV supplies

12/12/2020

10:00 – Checking connectivity of all the differential lines.  One side of PDM 6 is not connected.  2, 4, and 25 are connected.  Doesn’t explain what we saw yesterday.  Channel 2 and 4 now are responding to light on both differential lines.  The only thing that has changed is unplugging the external signal cable and plugging it back in.  PDM 6 is the only one now not responding on both lines.

10:30 – Checking the power spectra.  PDM 4 and 6 are bad on both differential lines.  PDM 2 looks fine.  Seems to be non-repeatable.

11:00 – Disconnected the signal patch board from the finger strip.  Holding the patch and reconnecting the entire signal path and pinging now shows PDM 6 and 25 have one differential line disconnected.

11:30 – Checking the power spectra on every PDM channel directly from the fingerstrip with a flat cable.

12:30 – PDMs 2, 4, 6 power spectra from the fingerstrip look normal.  Both connections are there.  Light response is normal.  Channel 13, however, now has one differential line disconnected and the other looks abnormal.  Saving waveform data with 60Hz light on as waveform_light_2.

All of this is pointing to multiple connection issues…

15:00 – It looks like PDM 13 is disconnected from the fingerstrip, or at least it is pulled further away than the others.  This is likely also due to the removal of the patch.  PDM 2 also looks pulled a bit further.

15:30 – Unmounted MB2 and put on the stand to do an optical inspection.  The microscope didn’t allow a close enough look at the wirebonds to search for any broken ones.  A close look by eye didn’t reveal any obvious damage.

16:00 – Packed everything up.  MB2 needs a closer look, likely needs to be disassembled.

I added javascript webdisplay of the V1725 waveforms to the Darkside setup.  You can see the waveform display here:

https://ds-proto-daq.triumf.ca/CS/webdisplay

You can also see the webdisplay that comes built into the JSROOT webserver here:

https://ds-proto-daq.triumf.ca/rootana/

For Pierre: I went through and reset the baselines for all the channels.

A little progress on a couple fronts:

1) We got the baseline DAC setting working correctly from both the CAEN command line tool and from the MIDAS
frontend.  Needed to add some extra sleep before the DAC settings were modified.

2) We figured out how to set the threshold for self-triggering on each channel. It is actually register 0x1n60
for this firmware.  The threshold is defined relative to the online calculated baseline. The threshold is
settable from the ODB.  However, the polarity of the triggering is currently hard-coded to be negative (bit 6 of
register 0x1n64).

3) We could confirm that the LVDS outputs from the V1725 appeared and disappeared as expected when the
self-trigger threshold was changed.

4) We added some basic documentation in Markdown here:

https://bitbucket.org/ttriumfdaq/dsproto_daq/src/master/

Summary of setup of chronobox on network (mostly done by Pierre):

1) Hook up USB connection from chronobox to ds-proto-daq.  Start serial-USB connection by doing

/home/dsproto/online/dsproto_daq/serialusb

can login through serial link.  Added the ssh key for dsproto@ds-proto-daq to the authorized_keys file on
chronobox linux.

ctrl-a, ctrl-x to stop picocom

2) On ds-proto-daq, setup dhcpd server.  Configuration in this file

/etc/dhcp/dhcpd.conf

dhpcd is bond to the second NIC only.  Configure DHCP to give IP 192.168.1.3 to chronobox.  In /etc/hosts set
192.168.1.3 to have hostname chronobox

Power cycle chronobox; it successfully gets IP:

[dsproto@ds-proto-daq dsproto_daq]$ ping chronobox
PING chronobox (192.168.1.3) 56(84) bytes of data.
64 bytes from chronobox (192.168.1.3): icmp_seq=1 ttl=64 time=0.180 ms

3) Can see the chronobox webpage locally as

http://chronobox:8080

4) Some esper tool to access data:  did following to setup esper

  734  yum install python2-pip
  735  pip install esper-tool
  736  yum install ncurses
  738  yum install ncurses-devel
  739  pip install esper-tool

Can then connect to the chronobox by doing

esper-tool interactive http://chronobox:8080

This is as much as I understand at this point... more exploring now...

5) Pierre unplugged the SD card to take it back to Bryerton's room.  But I guess this was bad.

lots of errors on serialUSB link now and the webpage doesn't work anymore:

[ 1246.665269] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.904770] EXT4-fs error: 339 callbacks suppressed
[ 1246.679162] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.904780] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: inode #2: com[ 1246.692884] blk_partition_remap: fail for partition 2
m syslogd: reading directory lblock 0
[ 1246.709020] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.929199] blk_partition_remap: fail for partition 2
[ 1246.717394] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.942971] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: inode #2: com[ 1246.731255] blk_partition_remap: fail for partition 2
m syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.953760] blk_partition_remap: fail for [ 1246.747337]
blk_partition_remap: fail for partition 2
partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.967532] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: [ 1246.763481] blk_partition_remap: fail for partition 2
inode #2: comm syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.978320] blk_partition_rem[ 1246.779511] blk_partition_remap:
fail for partition 2
ap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.992091] EXT4-fs error (device mmcblk0p2): ext4_find[
1246.795586] blk_partition_remap: fail for partition 2
_entry:1437: inode #2: comm syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1243.002885] blk_[ 1246.811671] blk_partition_remap: fail for
partition 2
partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1243.016656] EXT4-fs error (device mmcblk0p[ 1246.827750]
blk_partition_remap: fail for partition 2

1) Install and tweak elog:

[root@ds-proto-daq ~]# cat /etc/elogd.cfg 
[global]
port = 8084
MTP host = trmail.triumf.ca
Use Email Subject = {$logbook} $Subject
Remove on reply = Author
Quote on reply = 1
URL = https://ds-proto-daq/elog/

[DS Prototype]
Theme = default
Comment = ELOG for DS Prototype MIDAS DAQ
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem, Problem Fixed, Configuration, Other
Options Category = General, Hardware, Digitizer, Trigger, MIDAS, Software, Other
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type
Email all = lindner@triumf.ca

[root@ds-proto-daq ~]# systemctl start elogd.service
[root@ds-proto-daq ~]# systemctl is-active elogd.service
active
[root@ds-proto-daq ~]# systemctl enable elogd.service

2) Tweak apache and restart 

[root@ds-proto-daq ~]# grep elog /etc/httpd/conf.d/ssl-ds-proto-daq.conf 
ProxyPass /elog/ http://localhost:8084/ retry=1
[root@ds-proto-daq ~]# systemctl restart httpd

3) change MIDAS to use elog

[dsproto@ds-proto-daq bin]$ odbedit 
[local:dsproto:R]/>cd Elog/
[local:dsproto:R]/Elog>create STRING URL
String length [32]: 256
[local:dsproto:R]/Elog>set URL https://ds-proto-daq.triumf.ca/elog/DS+Prototype/

1) Install and tweak elog:

[root@ds-proto-daq ~]# cat /etc/elogd.cfg 
[global]
port = 8084
MTP host = trmail.triumf.ca
Use Email Subject = {$logbook} $Subject
Remove on reply = Author
Quote on reply = 1
URL = https://ds-proto-daq/elog/

[DS Prototype]
Theme = default
Comment = ELOG for DS Prototype MIDAS DAQ
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem, Problem Fixed, Configuration, Other
Options Category = General, Hardware, Digitizer, Trigger, MIDAS, Software, Other
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type
Email all = lindner@triumf.ca

[root@ds-proto-daq ~]# systemctl start elogd.service
[root@ds-proto-daq ~]# systemctl is-active elogd.service
active
[root@ds-proto-daq ~]# systemctl enable elogd.service

2) Tweak apache and restart 

[root@ds-proto-daq ~]# grep elog /etc/httpd/conf.d/ssl-ds-proto-daq.conf 
ProxyPass /elog/ http://localhost:8084/ retry=1
[root@ds-proto-daq ~]# systemctl restart httpd

3) change MIDAS to use elog

[dsproto@ds-proto-daq bin]$ odbedit 
[local:dsproto:R]/>cd Elog/
[local:dsproto:R]/Elog>create STRING URL
String length [32]: 256
[local:dsproto:R]/Elog>set URL https://ds-proto-daq.triumf.ca/elog/DS+Prototype/

Couple weeks of work documented in one elog...

V1725 serial numbers: 455, 392, 460, 462, 474

firmware: DPP-ZLE+

The 3818 kernel module not getting loaded on start-up... it is in /etc/rc.local, but not getting called

[root@ds-proto-daq ~]# grep 3818 /etc/rc.local 
# Load A3818 driver...
/sbin/insmod  /home/dsproto/packages/A3818Drv-1.6.0/src/a3818.ko

Called manually for now...

Reworked and cleaned up DEAP version of V1725 code
- Updated register map
- Removed smartQT code

Quick test: the V1725 seems to be getting busy at a maximum rate of 10kHz (with no samples being saved though).

Got preliminary documentation from CAEN for ZLE-plus firmware... posted here:

/home/dsproto/packages/CAEN_ZLE_Info

In particular, manual shows how the data structure is different for ZLE-plus data banks (as compared to V1720 ZLE data banks).

Fixed the /etc/rc.local setup so that the A3818 driver is loaded and mhttpd/mlogger is started on reboot.

Looking at V1725-ZLE register list.  I don't entirely understand how we are going to do the trigger outputs and the busy.  The manual makes it clear that the triggers from each pair of channels are combined together and send to the trigger logic.  So there should be 8 trigger primitives from the board.  The LVDS IO connector allows to configure groups of 4 outputs as being for either the trigger outputs or the busy/veto outputs.  So I guess I will configure the first two groups to output the trigger primitives and the third group to output the BUSY information.

Added code to do the ADC calibration; added extra equipment for reading out ADC temperatures periodically.

Renewed the lets-encrypt SSL certificate... but haven't set up automatic renewal yet...

[root@ds-proto-daq ~]# certbot renew --apache
[root@ds-proto-daq ~]# systemctl restart httpd

Analyzer program basically working.  Pushed to bitbucket:

https://bitbucket.org/ttriumfdaq/dsproto_analyzer/src

Pierre figured out that NIM crate not working.  We now see LVDS outputs from the individual channels firing. 
Each set of two different channels is ganged together into a single self-trigger output.  By setting the
register 0x1n84 to 3 we enable so that if either input channel fires then the self-trigger for that group fires.

Bryerton provided CDM outputting 50MHz clock; all V1725s now running with external clock.

Modified frontend to readout 4 modules and 16 channels per module.

Still need to modify the analyzer to show data for all 4 modules.

I modified the analyzer so that it shows data from all four modules.

I find that there is evidence of corruption of the V1725 ADC data on a couple channels.   You can see an example of a waveform with corruption in the attachment.  You see that there seems to be a bunch of fluctuations of exactly 16 
ADC counts, which seems unphysical.  So far I see this problem in these channels:

module 0, ch 0
module 0, ch 1
module 0, ch 13
module 1, ch 4
module 3, ch 6

I saw corruption similar to this on the V1730 readout.  CAEN does have a scheme for ADC calibration (by poking a register), which worked well for the V1730.  I thought that I implemented this ADC calibration for the V1725, but I might 
have messed it.  I'll look at this again.

Added pair of 10TB hard-drives in raid-1 to ds-proto-daq.  MIDAS data files will get written to this raid volume.

[dsproto@ds-proto-daq tmp]$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        9.1T  180M  8.6T   1% /data
[dsproto@ds-proto-daq tmp]$ cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdc1[1] sdb1[0]
      9766435648 blocks super 1.0 [2/2] [UU]
      [>....................]  resync =  0.9% (88214784/9766435648) finish=4372.8min speed=36886K/sec
      bitmap: 73/73 pages [292KB], 65536KB chunk

unused devices: <none>

First check the maximum trigger rate and maximum data rate for different sample lengths (for each channels):

sample length      Maximum rate   MB/s    CPU % (per thread)
16us               0.44kHz        231     20
6.4us              1.04kHz        214     20  
3.2us              1.95kHz        198     22   
1.6us              3.12kHz        160     26   
0.8us              4.88kHz        127     30   (what are these threads doing?)

Look at the code more.  See that there is a maximum size for the block transfer of 10kB.  Increase this to 130kB
(which is the maximum amount that this board can make per event).  Now find

sample length     Maximum rate       MB/s     CPU % (per thread)
16us              0.68kHz            346      7
6.4us             1.46kHz            300      10 
1.6us             3.6kHz             190      25

Good.  So for long samples we are actually slightly above the maximum transfer rate of 85MB/s*4 = 340MB/s

Tried writing out the data to disk at the maximum 345MB/s rate; the DAQ can't keep up.  Maximum rate was more
like 270MB/s.
I think the mlogger was actually fine.  But I think the write-to-disk speed of the harddrive could not keep up.
 So I think we are limited 
by hardware in that case.  We would need a large raid array to be able to write faster.

I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.

It turns out that the ZLE V1725 firmware we are using only supports reading out up to 4000 samples per channel.
 We need 80000 samples to readout 200us, which is requirement.

So we switch the V1725s to use the backup firmware on the board, which is the standard waveform firmware. 
Firmware version is 17200410.

Expected data size for event with 200us of data: 200000ns * 1/4 ns/sample * 16 ch * 4 boards * 2 bytes/sample =
 6.4MB per event

Measured max event rate of 60Hz with 385MB/s with 200us readout.

Needed to increase max event size, set buffer organization to 6 and set almost full to 32 in order to
accommodate the larger event size.

Changed some registers for different firmware:
- V1725_SELFTRIGGER_LOGIC

More tests needed

I installed a PCIe 1Gbps network card and configured it as a private network.  The PC (ds-proto-daq) is
192.168.1.1.  I guess we can make the chronobox 192.168.1.2.

[root@ds-proto-daq ~]# ifconfig enp5s0
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::bb27:5db:f778:d584  prefixlen 64  scopeid 0x20<link>
        ether 68:05:ca:8e:66:5c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 4145 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf72c0000-f72e0000  

I retested the chronobox trigger generation:

1) Inserting moderate sized pulse into channel 8 of V1725-0.
2) Configured threshold of V1725 so that channel triggers LVDS pulse into chronobox
3) Fan-out trigger out from chronobox to all V1725s.
4) busy signal from each V1725 fed into the chronobox
5) Start run, then push trigger from 20Hz up to 200Hz
6) System running stably!  Actual trigger rate about 60.2Hz.  The almost_full condition is set to 32 on the
V1725s and the estored on each board fluctuates below 32.

https://ds-proto-daq.triumf.ca/HS/Buffers/eStored?hscale=300&fgroup=Buffers&fpanel=eStored&scale=10m

The busy light on the V1725 never comes on, good.

The next thing we need is some way to start/stop the trigger generation on the chronobox, so that at
begin-of-run triggers do not get sent before the V1725s are finished configuration.

Other notes:

a) It turned out that the register to set the V1725 channel trigger threshold was different for RAW vs ZLE
firmware (0x1080 vs 0x1060); after fixing that the channel threshold seemed to work as expected.
b) Map for chronobox NIM cables:

channel 1-4: busy IN from V1725
channel 5,7,8: trigger OUT from chronobox
channel 6: clock OUT from chronobox

a) Bryerton implemented new version of the firmware.  New features:

1) run start/stop state
2) at run start 6 events are sent with 200ms separation
3) there is a greater set of counters about trigger, as well as configuring the trigger.

b) chronobox webpage can be seen here:

https://ds-proto-daq.triumf.ca/chronobox/

The mod_tdm page gives configuration of run state and trigger.  In particular

- button to start/stop run
- button to do manual trigger
- configure which channels are TOP and which are BOTTOM
- configure the number of TOP or BOTTOM channels that need to figure
- DecisionType: true= TOP AND BOTTOM groups must fire; false = TOP OR BOTTOM groups can fire.

c) Start and stop run can also be done on command line with esper tool:

esper-tool write -d true 192.168.1.3 mod_tdm run
esper-tool write -d false 192.168.1.3 mod_tdm run

d) I modified the V1725 frontend to integrate the chronobox start/stop.

At run start
- configure V1725s
- start chronobox with the command-line esper-tool call

At end run
- add deferred transition function which stops run (with esper-tool), then checks whether all the events have
cleared from ring buffers.
- once ring buffers are cleared, finish stopping the V1725s.

In the long run should somehow do the start/stop commands with some http post command, rather than command line
to esper-tool.

e) With the start/stop run could start to compare the timestamps of V1725s and confirm that they matched (ie,
all V1725s got reset at the same time).  Also, all the events are cleared from buffers.

f) However, I found that the frontend program still consistently failed with this error when the trigger rate
was above the maximum sustainable:

Deferred transition.  First call of wait_buffer_empty. Stopping run
[feov1725MTI00,ERROR] [v1725CONET2.cxx:685:ReadEvent,ERROR] Communication error: -2
[feov1725MTI00,ERROR] [feoV1725.cxx:654:link_thread,ERROR] Readout routine error on thread 0 (module 0)
[feov1725MTI00,ERROR] [feoV1725.cxx:655:link_thread,ERROR] Exiting thread 0 with error
Stopped chronobox run; status = 0
Segmentation fault


Next steps:

1) Fix the seg-fault for high rate running.
2) More detailed timestamp checking, with cm_msg(ERRORs)
3) Bryerton is now working on the event FIFO on chronobox.  That will be next thing to integrate.

> f) However, I found that the frontend program still consistently failed with this error when the trigger rate
> was above the maximum sustainable:
> 
> Deferred transition.  First call of wait_buffer_empty. Stopping run
> [feov1725MTI00,ERROR] [v1725CONET2.cxx:685:ReadEvent,ERROR] Communication error: -2
> [feov1725MTI00,ERROR] [feoV1725.cxx:654:link_thread,ERROR] Readout routine error on thread 0 (module 0)
> [feov1725MTI00,ERROR] [feoV1725.cxx:655:link_thread,ERROR] Exiting thread 0 with error
> Stopped chronobox run; status = 0
> Segmentation fault

I sort of 'fix' this problem.  There is some sort of problems between the V1725 readout	thread and the system call
to esper-tools to stop the run.  Some collision between the system resources for these two calls causes the readout
thread ReadEvent call to fail.  I 'fix' the problem by adding in the end_of_run part a 500us pause of the readout
threads before I make the system call to esper-tool.  

Odd.  In principle I think that the system calls and the readout threads are running on different cores.  So not
clear what the problem was.  Should figure out better diagnosis and fix problem properly.