Somehow the trigger rate was not matching the trigger source.
Find out that Link 3 was not collecting and possibly holding the fragment assembly in the main thread.
Swap Link3 <-> Link0 on the V1725, restarted.
Needs further investigation!

Date rate is fine now! CPU load is balanced on all 4 threads (~25%)
- irqbalance disabled
- change affinity for A3818 to cpu9: /etc/rc.local add: echo 0200 > /proc/irq/136/smp_affinity
  Check : watch -n 0.1 'cat /proc/interrupts'

Maximum Trigger rate (HW buffer not rising) 1950 Evt/s => 200MB/s 
for event size of 100KB composed of 4 banks with 32us per channel.

Ramp down the fields, in 200V steps.
Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Ramp UP the fields, in 100V steps.

Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Pierre figured out that NIM crate not working.  We now see LVDS outputs from the individual channels firing. 
Each set of two different channels is ganged together into a single self-trigger output.  By setting the
register 0x1n84 to 3 we enable so that if either input channel fires then the self-trigger for that group fires.

Bryerton provided CDM outputting 50MHz clock; all V1725s now running with external clock.

Modified frontend to readout 4 modules and 16 channels per module.

Still need to modify the analyzer to show data for all 4 modules.

The V1725s have been changed to have board config 0x50 rather than 0x10. This means that they will now trigger on the leading edge of the pulse rather than the tailing edge. There is now much less jitter in the location of pulses in the digitized waveforms.

09:15:33.955 2019/11/12 [feov1725MTI00,ERROR] [feoV1725.cxx:663:link_thread,ERROR] Exiting thread 3 with error

It looks like there's a memory leak in the high voltage driver, and we were running out of memory (using a lot of swap). That *may* be related to the recent instability, but I'm not sure.

If things are more stable now, then the memory leak was probably the cause. If not, then we should try stopping the V1725 program, power-cycling the VME crate, then starting the V1725 program.


Yi: restarted the VME crate twice, now it seems like smooth.

The connection to rootana is lost.

No data taking can be started.

Run 1197

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1198

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1200

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1201

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Data taking has to be stopped because of the noises.

Noise problem seems like fixed after remaking the Al foil cover.

Run 1208

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1209

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

At the end of run 1209, a lot of noise show up again.

We still have the noise issue.

Run 1219

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1220

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1221

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1222

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1223

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1224

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1226

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 40k

Run 1228

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 15k

Run 1229

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 40k

The DAQ seems to be ready for tests with proto-0 tomorrow.  

I had to invert the first busy input in order to get chronobox to produce triggers.  I modified the setup script

setup_chronobox.sh

so this is now the default.  Not sure why necessary.

Another interesting fact; it seems the chronobox only asks for a DHCP IP when it first boots.  I think that
chronobox and ds-proto-daq were rebooted at the same time; ds-proto-daq dhcp server was probably not running
when chronobox asked for IP.  chronobox got IP fine when it was power cycled.

Run 973

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 974

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 975

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 976

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 978 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 979 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

After laser run:

Switch Laser OFF

In Chronobox:

DIsable "External Triger Enable [ext_trig_ena]0\[0]

Input "0x3F3F3F7F" in "Enable Channel [ch_enable] 

Run 1022

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  159 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1024

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  186 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1025

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  207 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1026

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  225 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1028 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1029 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 956

Gas pocket: ON (thickness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 957

Gas pocket: ON (thickness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 958

Gas pocket: ON (thickness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 960

Gas pocket: ON (thickness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 961

Gas pocket: ON (thickness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Trigger setup with a threshold of 1500 ADC count wrt to baseline, on just two channels.

This is also to limit the trigger to 50 Hz, since the busy still did not work.

100 K events on disk.

Majority of 5, threshold at 1000 ADC count, 13 channels enabled (10 trigger signals)  50 Hz, 30 us window.

58 k events recorded.

Majority of 5, threshold at 1000 ADC count, 24 channels enabled (16 trigger signals)  100 Hz, 30 us window.

38 k events recorded.

Majority of 8, threshold at 1000 ADC count, 24 channels enabled (16 trigger signals)  10 Hz, 200 us window, trigger at 9.5 microseconds.

10 k events recorded.

By direct inspection pileup seems at the 10 % level in the 200 us window.

Gas pocket: ON

Fields: OFF

Threshold: 400 ADCc below baseline

Threshold extend:  5x16 = 80ns

Comment: Noise appeared on event 4200