Changed custom size to 500 (20 us), tested ok run 681

I modified the analyzer so that it shows data from all four modules.

I find that there is evidence of corruption of the V1725 ADC data on a couple channels.   You can see an example of a waveform with corruption in the attachment.  You see that there seems to be a bunch of fluctuations of exactly 16 
ADC counts, which seems unphysical.  So far I see this problem in these channels:

module 0, ch 0
module 0, ch 1
module 0, ch 13
module 1, ch 4
module 3, ch 6

I saw corruption similar to this on the V1730 readout.  CAEN does have a scheme for ADC calibration (by poking a register), which worked well for the V1730.  I thought that I implemented this ADC calibration for the V1725, but I might 
have messed it.  I'll look at this again.

A little progress on a couple fronts:

1) We got the baseline DAC setting working correctly from both the CAEN command line tool and from the MIDAS
frontend.  Needed to add some extra sleep before the DAC settings were modified.

2) We figured out how to set the threshold for self-triggering on each channel. It is actually register 0x1n60
for this firmware.  The threshold is defined relative to the online calculated baseline. The threshold is
settable from the ODB.  However, the polarity of the triggering is currently hard-coded to be negative (bit 6 of
register 0x1n64).

3) We could confirm that the LVDS outputs from the V1725 appeared and disappeared as expected when the
self-trigger threshold was changed.

4) We added some basic documentation in Markdown here:

https://bitbucket.org/ttriumfdaq/dsproto_daq/src/master/

It turns out that the ZLE V1725 firmware we are using only supports reading out up to 4000 samples per channel.
 We need 80000 samples to readout 200us, which is requirement.

So we switch the V1725s to use the backup firmware on the board, which is the standard waveform firmware. 
Firmware version is 17200410.

Expected data size for event with 200us of data: 200000ns * 1/4 ns/sample * 16 ch * 4 boards * 2 bytes/sample =
 6.4MB per event

Measured max event rate of 60Hz with 385MB/s with 200us readout.

Needed to increase max event size, set buffer organization to 6 and set almost full to 32 in order to
accommodate the larger event size.

Changed some registers for different firmware:
- V1725_SELFTRIGGER_LOGIC

More tests needed

The V1725s have been changed to have board config 0x50 rather than 0x10. This means that they will now trigger on the leading edge of the pulse rather than the tailing edge. There is now much less jitter in the location of pulses in the digitized waveforms.

09:15:33.955 2019/11/12 [feov1725MTI00,ERROR] [feoV1725.cxx:663:link_thread,ERROR] Exiting thread 3 with error

It looks like there's a memory leak in the high voltage driver, and we were running out of memory (using a lot of swap). That *may* be related to the recent instability, but I'm not sure.

If things are more stable now, then the memory leak was probably the cause. If not, then we should try stopping the V1725 program, power-cycling the VME crate, then starting the V1725 program.


Yi: restarted the VME crate twice, now it seems like smooth.

Pierre figured out that NIM crate not working.  We now see LVDS outputs from the individual channels firing. 
Each set of two different channels is ganged together into a single self-trigger output.  By setting the
register 0x1n84 to 3 we enable so that if either input channel fires then the self-trigger for that group fires.

Bryerton provided CDM outputting 50MHz clock; all V1725s now running with external clock.

Modified frontend to readout 4 modules and 16 channels per module.

Still need to modify the analyzer to show data for all 4 modules.

For Reference,

I put a simple schematic for the Trigger/Run control.

Bryerton, please have a look. Let's try to issue 3..5 SW trigger before opening the HW trigger.

I retested the chronobox trigger generation:

1) Inserting moderate sized pulse into channel 8 of V1725-0.
2) Configured threshold of V1725 so that channel triggers LVDS pulse into chronobox
3) Fan-out trigger out from chronobox to all V1725s.
4) busy signal from each V1725 fed into the chronobox
5) Start run, then push trigger from 20Hz up to 200Hz
6) System running stably!  Actual trigger rate about 60.2Hz.  The almost_full condition is set to 32 on the
V1725s and the estored on each board fluctuates below 32.

https://ds-proto-daq.triumf.ca/HS/Buffers/eStored?hscale=300&fgroup=Buffers&fpanel=eStored&scale=10m

The busy light on the V1725 never comes on, good.

The next thing we need is some way to start/stop the trigger generation on the chronobox, so that at
begin-of-run triggers do not get sent before the V1725s are finished configuration.

Other notes:

a) It turned out that the register to set the V1725 channel trigger threshold was different for RAW vs ZLE
firmware (0x1080 vs 0x1060); after fixing that the channel threshold seemed to work as expected.
b) Map for chronobox NIM cables:

channel 1-4: busy IN from V1725
channel 5,7,8: trigger OUT from chronobox
channel 6: clock OUT from chronobox

Notes on day:

1) Fixed the problem with the network interfaces.  Now the computer boots with the correct network configuration; outside world visible 
and private network on.

2) fan tray on VME crate seemed to be broken.  Got another VME crate from pool and installed it.  This VME crate seems to be working 
well. 

3) Recommissioned the DAQ setup.  Found a couple small bugs related to the V1725 self-trigger logic.  Fixed those and the V1725 self-
triggers seem to be working correctly.

4) Tried to install new CDM from TRIUMF (with ssh access), but clocks didn't stay synchronized.  Will bring module back to TRIUMF.

5) Added some code to V1725 frontend for clearing out the ZMQ buffers of extra events at the end of the run.  This is to protect 
against the case where the chronobox is triggering too fast for the V1725s.

Several points:

1) Gave a series of tutorial on DAQ to DS people yesterday and today.  Got a bunch of feedback, which I will pass on when I'm back at TRIUMF.

2) The computer ds-proto-daq was offline when I got in to lab this morning.  Hmm, not clear what is wrong with computer.  Didn't happen the 
first day.  Maybe another power blip?  Maybe we need a UPS for this DAQ machine, to protect it from power blips.

3) Using instructions from Luke, reconfigured the CDM to use the clock from the chronobox. 

4) Added scripts for putting the chronobox and the CDM into a sensible state.  Scripts are 

/home/dsproto/online/dsproto_daq/setup_chronobox.sh
/home/dsproto/online/dsproto_daq/setup_cdm.sh

The scripts need to be rerun whenever the chronobox or VME crate are power cycled.

5) Fix some bugs and added some new plots to online monitoring.  In particular, added a bunch of plots related to the chronobox data.

6) Found some problems with monitoring of chronobox trigger primitives, which I passed onto Bryerton.

Run 703 is being writte on disk.

16 channels of board 00. 

Laser trigger at 10 Hz

ended with

[feoV1725.cxx:1033:read_trigger_event,ERROR] Error: did not receive a ZMQ bank. Stopping run.   ]

Implemented a simple charge integration on the midas display, most of the channels look almost as good as this one.

Nothing on the daq side changed, a part from the DAC value moved to 5000, allowing a slightly greater dynamic range.

Run 718 has 25 channel readout Board 0 ch 0-15 and Board 0 ch 0-7. 

Laser triggered, some noise, maybe not so bad as is look here.

RMS of baseline is 200 ADC count. Single PE peak is about 100-200 ADC count from baseline.

Looking carefully, some very nice scintillation event are found.

Run 747 , 250 k events

as requested by alessandro 40% post trigger, acquisition window 16 us.

Laser Run with Vbias=65 V , new recommended value from PE group

500 k events acquired

Trigger setup with a threshold of 1500 ADC count wrt to baseline, on just two channels.

This is also to limit the trigger to 50 Hz, since the busy still did not work.

100 K events on disk.

Majority of 5, threshold at 1000 ADC count, 13 channels enabled (10 trigger signals)  50 Hz, 30 us window.

58 k events recorded.