Drive folder with data files and some pictures:  https://drive.google.com/drive/folders/1Sr27kQTOO2kciBY7v-wYBP06tWLVEjiP

Lab notes @ CERN

11/10/2020

11:00 – First look at MB2.  Noticed on some of the PDMs that some of the tile PCB are more delaminated than others.  PDM 25 also had the FEB connectors more inserted than any other PDMs.  No other differences noticed.

11:30 – Connected SMU and switching matrix to MB2.  Power w/o steering module.  Checked that all channels match by running photo current tests with and w/o a flashlight on one corner.  Using the computer form Pisa with Labview scripts from Matteo.  Everything is ok.

12:00 – Replace plastic shields around Proto-0 for IV curve measurements in dark conditions.  PDM 1 breakdown is around 63V.  Looks ok.  PDM 2 is over 65V…. Seems high.

12:30 - PDM 6 People walking, disrupted the light conditions.  IV curve will be jagged.  Will go for lunch.  Closing door and turning off exterior lights.  Current will decrease

15:30 – IV curves look fine.  …/MB2_retriveval_CERN/data_files/iv_warm_1

16:00 – Took forward IV curves.  Saved as …/MB2_retriveval_CERN/data_files/forward_warm_1.  They look fine.  Checked old pictures of the lamination of the tile PCB and they have always varied in thickness, from before the LNGS LN test.

16:45 – Power FEBs on to look at noise spectra with 20V bias on the tiles.  

18:00 – PDM 2, 4, 6, and 25 have very odd power spectra.  George said they look like they’re not connected.

18:30 – Saving FFT to disk for analysis.  Saved as …/MB2_retriveval_CERN/data_files/noise_warm_1

19:00 – PDM 2,4,6 and 25 have one of the differential lines disconnected.  Using a light and looking at the differential outputs on the scope, this is obvious.  20V bias and FEBs powered at 5V.  Saved as …/MB2_retriveval_CERN/data_files/waveform_light_1

19:30 – Started another round of reverse IV curves in dark conditions.  Saved as …/MB2_retriveval_CERN/data_files/iv_warm_2

21:15 – IV still going.  EP will come back to shut off the LV supplies

12/12/2020

10:00 – Checking connectivity of all the differential lines.  One side of PDM 6 is not connected.  2, 4, and 25 are connected.  Doesn’t explain what we saw yesterday.  Channel 2 and 4 now are responding to light on both differential lines.  The only thing that has changed is unplugging the external signal cable and plugging it back in.  PDM 6 is the only one now not responding on both lines.

10:30 – Checking the power spectra.  PDM 4 and 6 are bad on both differential lines.  PDM 2 looks fine.  Seems to be non-repeatable.

11:00 – Disconnected the signal patch board from the finger strip.  Holding the patch and reconnecting the entire signal path and pinging now shows PDM 6 and 25 have one differential line disconnected.

11:30 – Checking the power spectra on every PDM channel directly from the fingerstrip with a flat cable.

12:30 – PDMs 2, 4, 6 power spectra from the fingerstrip look normal.  Both connections are there.  Light response is normal.  Channel 13, however, now has one differential line disconnected and the other looks abnormal.  Saving waveform data with 60Hz light on as waveform_light_2.

All of this is pointing to multiple connection issues…

15:00 – It looks like PDM 13 is disconnected from the fingerstrip, or at least it is pulled further away than the others.  This is likely also due to the removal of the patch.  PDM 2 also looks pulled a bit further.

15:30 – Unmounted MB2 and put on the stand to do an optical inspection.  The microscope didn’t allow a close enough look at the wirebonds to search for any broken ones.  A close look by eye didn’t reveal any obvious damage.

16:00 – Packed everything up.  MB2 needs a closer look, likely needs to be disassembled.

I have added a backup of the CERN proto-0 ELOG at TRIUMF, in case ds-proto-daq is unavailable in future.

The backup ELOG is at https://ladd00.triumf.ca/elog-ds/. There are other logbooks hosted in the same place (requiring a username/password), but the CERN backup logbooks (at the bottom of the list) are available to all. I have configured things so that the TRIUMF replicas are read-only. To add new entries one must still use the CERN logbooks.

The logbooks are rsynced every hour via cron.

A few months ago I created a new midas frontend for CAEN HV crates. It behaves more like what users expect - changes made through other interfaces (e.g. ssh or the Java-based GUI) will be reflected in the ODB. The ODB structure is created dynamically based on which modules are present in the crate, and the parameters that each module supports (e.g. whether you specify a ramp-up rate in V/s or a ramp-up time in s).

When I first wrote the frontend, ds-proto-daq was offline. Pierre reminded me today that I should actually deploy the new code.

The new program code is at ~/online/dsproto_sy4527/caen_hv*. The executable is at ~/online/bin/caenhv_fe. The frontend is available on the "Programs" page in mhttpd. If we want to run the old HV frontend, it is still available at ~/online/bin/sy4527.

Connection parameters for the frontend are found in the ODB at /Equipment/CAEN_HV/Settings/Global. Currently the crate appears to be off, so the frontend will refuse to start properly as it cannot connect to 192.168.1.9. If the frontend is started with the crate on, it will create more directories in /Equipment/CAEN_HV/Settings (one for each module). The "Voltage" page on the experiment now points to a generic webpage that will work with arbitrary combinations of modules.

Temp at start: 

A: 242.7K    B: 278.16K    C: 86.4K    D: 240.89K

Temp at End:

A: 248.87K    B: 278.24K    C: 86.37K    D: 247.75K

Took iv curves, data is on DAQ PC here: /home/dsproto/online/dsproto_sy4527/ivdata_191120

IV of 25 SiPM put in ivdata_191119 directory

Run 1258  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 3000us, 1500us pre-trigger

No field

SIPM HV: 65V

Number of events: 10k

--> stopped after 4.7k (end of the data taking at 11:00)

Run 1255  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 1000us, 500us pre-trigger

No field

SIPM HV: 65V

Number of events: 10k

Run 1256  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 1000us, 500us pre-trigger

No field

SIPM HV: 65V

Number of events: 10k

--> Stop after 1.7k evts (10:16)

Run 1257  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 1000us, 500us pre-trigger

No field

SIPM HV: 65V

Number of events: 10k

Run 1252 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Low trigger rate - must restart manually 

Run 1253 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1254 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Lazy logger has caught up with the backlog, and continues to transfer new files to darkside01 as they are written.
EOS currently contains data for runs 822-1201.

I have deleted the following runs from ds-proto-daq:
* runs < 700 that were taken with gzip compression
* runs 800-1100 that are available on EOS and darkside01

The following files remain:
* runs 700-800 that haven't been copied to EOS (but are on darkside01). If they're of interest, they should be copied to EOS.
* runs 1200+ that haven't been copied to EOS yet (but will be soon).

Disk usage has improved from 85% to 33%.

PDM 15 and PDM 19 strange behaviour, lot of noise

Run 1239  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1240  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1241 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1242 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1243 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1244 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1245 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1246 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1247 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1248 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1249 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1250 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1251 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1252 Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

No field

SIPM HV: 65V

Number of events: 40k

Run 1234  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

 Drift field 200 V/cm Extraction 3.78 kV/cm

SIPM HV: 65V

Number of events: 200k

Problem with DAQ at 40k, force to restart manually 

Run 1235  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

 Drift field 200 V/cm Extraction 3.78 kV/cm

SIPM HV: 65V

Number of events: 200k

Problem with DAQ at 70k, force to restart manually 

Run 1237  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

 Drift field 200 V/cm Extraction 3.78 kV/cm
 

SIPM HV: 65V

Number of events: 40k

Run 1238  Physics run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Threshold: external clock,  set to 50 Hz

Trace length: 200us, 100us pre-trigger

 Drift field 200 V/cm Extraction 3.78 kV/cm

SIPM HV: 65V

Number of events: 40k

Run 1232 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1233 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

We still have the noise issue.

Run 1219

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1220

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1221

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1222

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1223

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1224

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1226

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 40k

Run 1228

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 15k

Run 1229

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 40k

Noise problem seems like fixed after remaking the Al foil cover.

Run 1208

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1209

Gas pocket: ON (thinkness 9mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

At the end of run 1209, a lot of noise show up again.

Run 1197

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1198

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1200

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1201

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 3

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  30 mbarg

SiPM HV: 65 V

Number of events: 20k

Data taking has to be stopped because of the noises.

Run 1186

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 100 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 10k

V1725 offline, run stopped manually

Run 1188

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 100 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 70k

Trigger rate became very low, run stopped manually

Run 1189

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1190

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  10

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1191

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 200 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  15

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1192

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1193

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  10

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1194

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  15

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1195

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 500 ADCc below baseline

Coincidence: 2

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5

TPC Pressure:  26 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1171

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1172 Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1173

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 4 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1174

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend: 5x16 = 80ns TPC

Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1175

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 6 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1176

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 500 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1177

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 500 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1178

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 500 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 4 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1179 Gas pocket: OFF

Fields: 0 V/cm

Threshold: 500 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1180

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 500 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 6 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1181

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 200us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 100k

Problem with DAQ, forced to stop

Run 1182

Gas pocket: OFF

Fields: 0 V/cm

Threshold: 200 ADCc below baseline

Trace length: 200us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend: 5x16 = 80ns

TPC Pressure: 80 mbarg

SiPM HV: 65 V

Number of events: 100k

Run 1166 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Trigger rate went to 0, run was stopped manually.

Run 1169 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1170 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

I have set up automatic copying of data from ds-proto-daq to darkside01, using the lazy logger.

There are 2 lazy logger channels set up - one for lz4 files and one for gz files. It will take some time for all the files to copied over. Note that lazy logger writes a message into the midas log for every file that it copies. The lazy logger channels can be started from the Program page like all the other midas-related programs.

Once files have been copied to darkside01 (by lazy logger) and to EOS (by Pablo/Edgar) we can delete files from ds-proto-daq. We will probably not be able to do this until tomorrow (copying is quite slow).

Run 1148

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

problem DAQ, forced to stop manually

Run 1149

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 380 Hz

Run 1150

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 310 Hz

Run 1151

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  250 Hz

Run 1152

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  150 Hz

Run 1153

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  90 Hz

Run 1154

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  390 Hz

Run 1155

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  400 Hz

Run 1156

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  400 Hz

Run 1157

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:   400 Hz

Run 1158

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:   400 Hz

Run 1159

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1160

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1161

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1163

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1164

Gas pocket: OFF

Fields: 500 V/cm

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1121 

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 50 V (lower than breakdown voltage) 

Number of events: 5k

Run 1122 

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 63 V

Number of events: 5k

Without blanket 

Run 1125

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 5k

Without blanket 

Run 1126

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 67 V

Number of events: 3.5k

trigger rate went down to 1 Hz, run was stopped manually

Without blanket 

Run 1127

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 67 V

Number of events: 2k

Without blanket 

Run 1128

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 67 V

Number of events: 5k

With blanket 

trigger rate went down to 1 Hz, run was stopped manually

problem with DAQ forced to restart manually 

Run 1129

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 67 V

Number of events: 5k

With blanket 

trigger rate went down to 1 Hz, run was stopped manually

problem with DAQ forced to restart manually 

jsroot_server stops from time to time 

1573739641 14:54:01.099 2019/11/14 [Logger,INFO] Client 'jsroot_server' on buffer 'SYSTEM' removed by cm_periodic_tasks because process pid 26693 does not exist

Run 1130

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 67 V

Number of events: 5k

With blanket 

Run 1131

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 5k

With blanket 

Run 1132

Gas pocket: OFF

Fields:  OFF

Threshold: external clock, 50Hz

Trace length: 1ms total, 600 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 5k

With blanket 

Run 1133

Gas pocket: OFF

Fields: OFF

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1134

Gas pocket: OFF

Fields: OFF

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1135

Gas pocket: OFF

Fields: OFF

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1136

Gas pocket: OFF

Fields: OFF

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1137

Gas pocket: OFF

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 2 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1138

Gas pocket: OFF

Fields: OFF

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1139

Gas pocket: OFF

Fields: OFF

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1140

Gas pocket: OFF

Fields: OFF

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1141

Gas pocket: OFF

Fields: OFF

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1142

Gas pocket: OFF

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 3 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1143

Gas pocket: OFF

Fields: OFF

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1144

Gas pocket: OFF

Fields: OFF

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1145

Gas pocket: OFF

Fields: OFF

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1146

Gas pocket: OFF

Fields: OFF

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Run 1147

Gas pocket: OFF

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Coincidence: 5 PDMs

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

It is not possible to access to the Chronobox page

Run 1109 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1110 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Run 1112

Gas pocket: ON (thinkness 7mm)

Fields: OFF 

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1113

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Run 1114

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Run 1115

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

The chronobox isn't serving the human-interactive webpage that should be visible at https://m-darkside.web.cern.ch/chronobox/ - it responds with "File not found". Oddly, it is still responding to API calls at /read_var, /write_var etc.

So shifters can still change between laser/noise/physics runs using the "Run type" page at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type (which uses the API), but can't as easily monitor / sanity check the chronobox behaviour.

In the first instance, I think it would be useful to power-cycle the chronobox. If that doesn't work, I'm not sure how to proceed.

Run 1105

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  265 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1106

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1107

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1108

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  242 mbarg

SiPM HV: 65 V

Number of events: 10k

Comment: Webpage of Chronobox doens't work, skipped Laser runs.

When we start the HV program (CAEN_SY4527), it spews error messages of the form:
"16:58:30.035 2019/11/13 [CAEN_SY4527,ERROR] [dd_sy4527.cxx:280:fParam_get,ERROR] Param Not Found Type : 0"

This is because the LV module in the SY5527 crate doesn't support some of the parameters that the driver wants to read 
(e.g. it support "RUpTime" rather than "RUp"). For our current usage, the errors are benign. But it does add to the list 
of deficiencies of the driver provided by core midas:
* No automatic discovery of total number of channels
* Confusing ODB structure
* No discovery of which features a module supports
* Odd detection of whether a channel is on/off
* ODB status doesn't update if module settings are changed with a different interface (e.g. SSH or Caen's Java-based GUI).

All these limitations are imposed by the core HV driver "class" in midas. Assuming we will be using Caen HV modules for the
foreseeable future, I think I should write a new slow control frontend that is less generic, but fixes all the above issues.

Run 1099

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  250 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1100

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  250 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1101

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  250 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1102

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  250 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1103

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  250 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1096 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1097 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Problem: The access to the Chronobox page is not possible.  Rate 400 Hz 

Run 1092

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  220 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1093

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  220 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1094

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  220 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1095

Gas pocket: ON (thinkness 7mm)

Fields: OFF 

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  220 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1090

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  220 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1091

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 400 ADCc below baseline

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  220 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1084

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1085

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1086

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 100us total, 50 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 6k

Run crashed: 09:31:55.239 2019/11/12 [feov1725MTI00,ERROR] [feoV1725.cxx:663:link_thread,ERROR] Exiting thread 0 with error

It looks like there's a memory leak in the high voltage driver, and we were running out of memory (using a lot of swap). That *may* be related to the recent instability, but I'm not sure.

If things are more stable now, then the memory leak was probably the cause. If not, then we should try stopping the V1725 program, power-cycling the VME crate, then starting the V1725 program.


Yi: restarted the VME crate twice, now it seems like smooth.

09:15:33.955 2019/11/12 [feov1725MTI00,ERROR] [feoV1725.cxx:663:link_thread,ERROR] Exiting thread 3 with error

The problem reported in ID 117 is somehow fixed this morning.

Attemted the following run, and it started but the rate was shoing 0 and it got stuck on 5 events

Was unable to connect to Chronobox webpage even after turning VME crate off and on.

Abandoned run and will attempt to fix in the morning.

Run 1073

Gas pocket: ON (thickness unknown)

Fields:  drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252mbarg

SiPM HV: 65 V

Number of Events: 10K

17:30

Took a set of IV curves using 0.5V steps from 30-50V and 0.05 steps above 50V.

Curves all look as expected, although reducing the step size makes the breakdown voltage harder to see on the plots.

Will use 0.02V steps in future.

Data can be found on dsproto machine at: /home/dsproto/online/dsproto_sy4527/ivdata_191111

Summary plot attached to this entry.

19:15

Refilled outer dewar for 10 mins

Run 1065

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 100us total, 80 us post-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  260 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1066

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 500 ADCc below baseline

Trace length: 100us total, 80 us post-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  260 mbarg

SiPM HV: 65 V

Number of events: 20k

Run 1067

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Trace length: 100us total, 80 us post-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  260 mbarg

SiPM HV: 65 V

Number of events: 20k

The connection to rootana is lost.

No data taking can be started.

Error during data taking  1573416708 21:11:48.151 2019/11/10 [feov1725MTI00,INFO] V1725 PLL loss lock Board:0 (vmeAcq=0x0)

It stop saving event but run in process. Impossible to stop the run. I tried to reset feov1725MTl00. After that error: 

1573416857 21:14:17.792 2019/11/10 [feChronoEsper,INFO] Client 'feov1725MTI00' on database 'ODB' removed by db_cleanup called by cm_periodic_tasks because pid 9724 does not exist

Run stopped after some time. Error 21:21:19.121 2019/11/10 [mhttpd,ERROR] [midas.cxx:3951:cm_transition_call,ERROR] cannot connect to client "feov1725MTI00" on host localhost, port 40944, status 503   ╳

Trying to restart feov1725MTI00. Status initializing

It is not possible to initialize 

Run 1054

Gas pocket: ON (unknow)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 100 ADC below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 400 Hz 

Run 1050

Gas pocket: ON (unknow)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  245 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1051

Gas pocket: ON (unknow)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  260 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1052

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  245 mbarg

SiPM HV: 65 V

Number of events: 10k

Stopped. Problem with DAQ

Run 1041

Gas pocket: OFF

Fields: OFF

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 360 Hz

Run 1042

Gas pocket: OFF

Fields: OFF

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 340 Hz

Run 1043

Gas pocket: OFF

Fields: OFF

Threshold: 250 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 290 Hz

Run 1044

Gas pocket: OFF

Fields: OFF

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 240 Hz

Run 1045

Gas pocket: OFF

Fields: OFF

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 170 Hz

Run 1046

Gas pocket: OFF

Fields: OFF

Threshold: 800 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  140 Hz

Run 1047

Gas pocket: OFF

Fields: OFF

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  110 Hz

Run 1048

Gas pocket: OFF

Fields: OFF

Threshold: 150 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  400 Hz

Run 1049

Gas pocket: OFF

Fields: OFF

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  400 Hz

Run 1032

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 100 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 800 Hz

Run 1033

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 350 Hz

Run 1034

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 250 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 260 Hz

Run 1035

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 400 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 200 Hz

Run 1036

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 600 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate: 130 Hz

"ERROR INFO: 17:19:23.176 2019/11/10 [feov1725MTI00,ERROR] [feoV1725.cxx:1048:read_trigger_event,ERROR] Error: did not receive a ZMQ bank after 100.000000 ms. Stopping run."

Run 1037

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 800 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  110 Hz

Run 1038

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  80 Hz

Run 1039

Gas pocket: OFF

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 150 ADCc below baseline

Trace length: 30us total, 5 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  340 mbarg

SiPM HV: 65 V

Number of events: 50k

Rate:  400 Hz

Run 1022

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  159 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1024

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  186 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1025

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  207 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1026

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  225 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1028 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1029 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1016

Gas pocket: ON (thinkness 7mm)

Fields: Off

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 201 mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1017

Gas pocket: ON (thinkness 7mm)

Fields:  drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 201 mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1018

Gas pocket: ON (thinkness 7mm)

Fields:  drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 217 mbarg

SiPM HV: 65 V

Number of Events: 10K

Run 1019

Gas pocket: ON (thinkness 7mm)

Fields:  drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 240

SiPM HV: 65 V

Number of Events: 10K

Run 1020 Laser Run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Comment: Recieved this error message but run seemed to continue ok and data appeared to be written: 19:56:04.839 2019/11/09 [feov1725MTI00,ERROR] [feoV1725.cxx:1048:read_trigger_event,ERROR] Error: did not receive a ZMQ bank after 100.000000 ms. Stopping run.   â•³

Run 1021 Laser Run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

frontend process still alive but slow, causing lot of rejected triggers.

stopped and restarted the feov1725 process to fix this

"ERROR INFO: 10:38:10.924 2019/11/09 [feov1725MTI00,ERROR] [feoV1725.cxx:1048:read_trigger_event,ERROR] Error: did not receive a ZMQ bank after 100.000000 ms. Stopping run."

Emailed to Ben.

Run 999

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  233 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1000

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  235 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1001

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  238 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1002

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  238 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 1003 Laser run (manually stopped)

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Comment: Triger rate too low (~8 Hz), manually stopped

Run 1004 Laser run (manually stopped)

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Comment: Triger rate too low (~8 Hz), manually stopped

Wrong MAC address fixed in CERN network setting, now the machine is visible from external network too:

From CERN network (even laptop) can connect by 

ssh -XY dsproto@ds-proto-daq2.cern.ch.  (same password as local account in ds-proto-daq)

From ds-proto-daq can also connect by:

ssh darkside01

That is to say that the two network card are now associated to two different names.. but the "real" name is still darkside01.

(will fix).

 enjoy

Run 995

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  230 mbarg

SiPM HV: 65 V

Number of events: 10k

Rate  ~9 Hz

Comment: TPC pressure increased up to 290 mbarg

Run 996

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  225 mbarg

SiPM HV: 65 V

Number of events: 10k

Rate  ~9 Hz

Comment: TPC pressure increased up to 240 mbarg

Run 997

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 10k

Rate  ~9 Hz

Run 998

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  240 mbarg

SiPM HV: 65 V

Number of events: 10k

Rate  ~9 Hz

The cable has been fixed. Socket 0001/007 is still not active.

I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

The analysis server from Roma (darkside01) is now connected to the local network. It is not yet connected to CERN.

There are 2 network ports on the rear of the machine:
* The port labelled "1":
 * MAC: 9c:71:3a:22:d5:62
 * Name: enp2s0f0
 * Configured for DHCP
 * Connected to local switch
 * Assigned IP 192.168.1.10 by ds-proto-daq
* The port labelled "2":
 * MAC: 9c:71:3a:22:d5:63
 * Name: enp2s0f1
 * Configured for DHCP
 * Should be connected to CERN network

I have set up password-less ssh between ds-proto-daq and darkside01. 

There are 3 problems connecting to the CERN network:
1) The long cable that was created does not work. I tried using it to connect ds-proto-daq to socket 0001/06 (i.e. the socket ds-proto-daq is normally connected to) and there is no link light - the cable is broken.
2) Socket 0001/07 does not seem to be operational. I tried connecting ds-proto-daq to socket 0001/07 using the cable we know works, and there was no link light. Either CERN have disabled the connection intentionally, or it is broken.
3) When I used the good cable to connect to the good socket (0001/06), CERN claimed the machine was unregistered. Marco told me that the interfaces have MACs 9c:71:3a:22:d6:4[ab] (rather than 9c:71:3a:22:d5:6[23]), so presumably the wrong MAC addresses were registered with CERN.

So, we need to register the correct MACs, fix the cable, and either get 0001/07 working or connect to a different socket.

I have changed `/Logger/Channels/0/Settings/Filename` to start with a dot, so files will be hidden until they are complete. 
The current value is `.run%05dsub%03d.mid`.

This will help Pablo with his script for transferring files to EOS.

I made a new page that makes it easier to switch between laser/noise/physics runs. 

It handles setting the most common chronobox trigger settings, and the V1725 self-trigger thresholds. 
For the latter, it assumes that the baselines are at 15500. A suggested workflow would be to run
"equalize_baselines.py" in the morning to ensure all channels are at 15500, then you can use this
page to set the self-trigger thresholds for the rest of the day/week without using the python scripts.

The page is available at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type

I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).

Run 973

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 974

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 975

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 976

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  222 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 978 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 979 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

After laser run:

Switch Laser OFF

In Chronobox:

DIsable "External Triger Enable [ext_trig_ena]0\[0]

Input "0x3F3F3F7F" in "Enable Channel [ch_enable] 

Laser Intensity: MIN 

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 30us, 5us pre-trigger

SIPM HV: 65V

MISSING FILE (E.Sanchez)

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 30us, 5us pre-trigger

SIPM HV: 65V

Run 956

Gas pocket: ON (thickness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 957

Gas pocket: ON (thickness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 958

Gas pocket: ON (thickness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 960

Gas pocket: ON (thickness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 200 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 961

Gas pocket: ON (thickness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

> Not sure if the trigger threshold is correctly set, there is an error:

Sorry, I made a change to the baseline calculation and forgot to update the threshold tool. It's updated now.

The thresholds currently seem to be set to 12801 - this is 2699 below the nominal baseline of 15500.

Not sure if the trigger threshold is correctly set, there is an error:

Traceback (most recent call last):
  File "/home/dsproto/online/dsproto_analyzer/threshold_tool.py", line 71, in <module>
    main()
  File "/home/dsproto/online/dsproto_analyzer/threshold_tool.py", line 47, in main
    (mean, rms) = fit_res
ValueError: too many values to unpack (expected 2)

the trigger threshold for all the runs in this Elog is 2699 ADCc below the baseline (see Elog 89)
 

Run 943

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 945

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 946

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 948

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 949

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 951

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 952

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 954

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

The CAEN VME crate now gets its IP address from the ds-proto-daq DHCP server. To do this, I set the IP address of the crate to 0.0.0.0 (using the front panel interface) then reset the crate (not just a power cycle - pressing the reset button is required).

I am now able to ping the crate, but am NOT able to load the webpage it is supposed to serve (I just get a "connection refused"). I can't see any configuration settings that would disable the webpage or would cause it to be served on a non-standard port. I may have missed something though.

Run 917

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 918

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 919

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 920

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar) [Edgar\Yi: correction: OFF (during recirculation tpc pressure about 400 mbar)]

Fields: drift 500 V/cm, extraction 2.8 kV/cm (wire grid 3780 V, 1st ring 2780 V, cathode 7780 V) [Yi: 1st ring setting is incorrect, inverted fied between wire grid and field cage ring]

Threshold: 400 ADC below baseline (baselines equalized to 15500)

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment:

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar) [Edgar\Yi: correction: OFF (during recirculation tpc pressure about 400 mbar)]

Fields: drift 100 V/cm, extraction 2.8 kV/cm (wire grid 3780 V, 1st ring 3580 V, cathode 3780 V) [Yi: 1st ring setting is incorrect, inverted fied between wire grid and field cage ring]

Threshold: 400 ADC below baseline (baselines equalized to 15500)

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment:

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar) [Edgar\Yi: correction: OFF (during recirculation tpc pressure about 400 mbar)]

Fields: drift 200 V/cm, extraction 2.8 kV/cm (wire grid 3780 V, 1st ring 4180 V, cathode 6180 V) 

Threshold: 400 ADC below baseline (baselines equalized to 15500)

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment:

Laser intensity: MAX

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock (laser NIM), 500Hz (laser set to 1kHz)

Trace length: 30us total, 5us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 67 V

Comment:

Laser intensity: MIN

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock (laser NIM), 500Hz (laser set to 1kHz)

Trace length: 12us total, 5us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 65 V

Comment:

Laser intensity: MAX

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock (laser NIM), 500Hz (laser set to 1kHz)

Trace length: 30us total, 5us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 65 V

Comment:

Laser intensity: MIN

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock (laser NIM), 500Hz (laser set to 1kHz)

Trace length: 12us total, 5us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 67 V

Comment:

Laser intensity: MAX

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock (laser NIM), 500Hz (laser set to 1kHz)

Trace length: 12us total, 5us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 65 V

Comment:

Laser intensity: MAX

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock (laser NIM), 500Hz (laser set to 1kHz)

Trace length: 500us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 65 V

Comment:

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock, 50Hz

Trace length: 500us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 50 V

Comment: data rate capped at 25Hz due to limited speed of writing on disk. 1ms traces are not possible yet.

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: external clock, 50Hz

Trace length: 500us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >460 mbarg

SiPM HV: 65 V

Comment: DAQ rate ~25Hz

I've written a new script that will set all the V1725 baselines to the same level, by adjusting the DAC values. All the baselines are currently set to 15500.

The script is documented in the "Adjusting V1725 baselines" section of https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation.

I also added a new option to the "adjust self trigger threshold" script that allows the user to specify an exact ADC threshold (rather than doing it in relation to baseline). E.g. if you know that the baselines are all currently 15500, and you want to set a threshold ~200mV below, you could set a self-trigger threshold of 13900 (1mV is ~ 8ADC).

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment: compare to Run 873, 875

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: OFF

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment: compare to Run 873

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar)

Fields: drift 500V/cm extraction 3.78kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment: try to trigger on S2

The V1725s have been changed to have board config 0x50 rather than 0x10. This means that they will now trigger on the leading edge of the pulse rather than the tailing edge. There is now much less jitter in the location of pulses in the digitized waveforms.

Gas pocket: ON

Fields: wire gate 3780 V, 1st ring: 4500 V, cathode: 7000 V

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 300 mbarg

SiPM HV: 65 V (noise run)

Comment: noise conditions good. Rate ~30Hz. Look for S2s!

I have created a new script that simplifies the adjustment of self-trigger thresholds (e.g. to change between "noise" runs with a low threshold and "normal" runs with a higher threshold).

It supports either setting the threshold to a certain number of ADC beneath baseline, or just shifting all the current thresholds by a given amount. It can also be used to just print the current baseline of each channel. Baseline calculation is done "live" by reading a few events.

The code is available in dsproto_analyzer, and is documented at https://bitbucket.org/ttriumfdaq/dsproto_daq/wiki/Midas%20Software%20Operation (scroll to "Adjusting self-trigger thresholds"). We used it recently to change to a noise run, and it worked well.

/home was getting very full on ds-proto-daq, so I moved the history files onto the /data disk. There is now a symlink from ~/online/history to /data/dsproto/history/.

This morning I installed the xrootd-client and python3-devel packages from yum. For the latter to work, I needed to do a yum update. This was long overdue, and updated 1200+ packages.

I have compiled a new version of ROOT 6 that links to Python 3. This will allow us to use ROOT and midas from the same python scripts.

Increased drift and extraction fields.

Current settings: wire gate 5100 V, 1st ring: 6100 V, cathode: 11100 V

P.S. Increased by in the morning by Yi.

Gas pocket: ON

Fields: drift field OFF (1st ring 0 V, cathode 0 V), extraction field OFF (wire gate 0 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 244 mbarg

SiPM HV: 50 V (noise run)

Comment: noise conditions good. Thresholds are too high, rate below 1 event/sec, only few events saved. Postpone to next day.

Gas pocket: ON

Fields: drift field OFF (1st ring 0 V, cathode 0 V), extraction field OFF (wire gate 0 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 244 mbarg

SiPM HV: 65 V

Comment: noise conditions good, 1e5 evts

Gas pocket: ON

Fields: drift field ON (1st ring 4180 V, cathode 6180 V), extraction field OFF (wire gate 0 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 244 mbarg

SiPM HV: 65 V

Comment: noise conditions good, 1e5 evts

Gas pocket: ON

Fields: ON (wire gate 3780 V, 1st ring 4180 V, cathode 6180 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 242 mbarg

SiPM HV: 65V

Comment: noise conditions good, 1e5 evts

Gas pocket: ON

Fields: ON

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

Comment: Noise appeared on event 48500

Ramp UP the fields, in 100V steps.

Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Gas pocket: ON

Fields: OFF

Threshold: 400 ADCc below baseline

Threshold extend:  5x16 = 80ns

Comment: Noise appeared on event 4200

Ramp down the fields, in 200V steps.
Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

After booting, the elog was only showing logs from before July, and was missing the "For Shifters" category. It appears we were loading the incorrect config file. The running command was:

/usr/sbin/elogd -D -c /etc/elogd.cfg -s /usr/share/elog/ -d /var/lib/elog/logbooks/

while it should have been:

/usr/sbin/elogd -D -x -c /home/dsproto/online/elog/elogd.cfg -s /usr/share/elog


rc.local was set up to run the latter command, but rc.local~ had the prior version. Perhaps that was being picked up. I have deleted the rc.local~ version, and will see if elog starts correctly next time we reboot.

For a few days we were unable to access ds-proto-daq remotely. We were also unable to reach the outside world from ds-proto-daq.

To gain more control over the network settings, I have disabled the automatic configuration by dracut. I created the file /etc/dracut.conf.d/no-ifcfg.conf with the single line `omit_dracutmodules+="ifcfg"`.

I then set the following contents in /etc/sysconfig/network-scripts:

# cat ifcfg-enp0s31f6
NAME=enp0s31f6
DEVICE=enp0s31f6
ONBOOT=yes
NETBOOT=yes
UUID=4887c207-13da-4424-8307-116ef2163fd8
IPV6INIT=no
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
HWADDR=18:31:BF:2E:00:12
BOOTPROTO=dhcp
IPV6_PRIVACY=no
AUTOCONNECT_PRIORITY=1
DNS1=137.138.16.5
DNS2=137.138.17.5
PEERDNS=no


# cat ifcfg-enp5s0
TYPE=Ethernet
PROXY_METHOD=None
BROWSER_ONLY=no
BOOTPROTO=static
DHCPCLASS=
IPADDR=192.168.1.1
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
HWADDR=68:05:CA:8E:66:5C
ONBOOT=yes
DEVICE=enp5s0


Network settings are now correct, and have survived a reboot. We are able to talk to both the local network and the outside world. We are also able to connect to ds-proto-daq from lxplus, and via the web proxy.

The HV board from Naples did not turn on: Yury gave it to the CAEN guys to check and/or bring back for replacement.

Yi and Luigi rented a new HV module (A1519). The HV module (A1520P) we used for tests of the I-V script is also present in the Mainframe (see below).
The first 24 channels (slots 0 and 1) are therefore HV. If the new HV does not show up on time we'll try and adapt the cables to work with A1519.

The A2517A module is LV. This is currently operated from the DAQ pc using CAEN_HVPSS_ChannelsController.jnlp (located in the Desktop/SteeringModule folder).
The three low voltage channels (0,1,2) should be turned on at the same time by setting Pw = ON.

Settings: 

Channel 0 and 1: I0Set = 2.0 A , V0Set = 2.5 V ,  UNVThr = 0 V, OVVThr = 3.0 V, Intck = Disabled
IMon = 1.44 (this depends on the illumination) , VMon = 2.48 V , VCon = 2.79 V

Channel 2: I0Set = 1.0 A , V0Set = 5.0 V ,  UNVThr = 0 V, OVVThr = 5.5 V, Intck = Disabled
IMon = 0.07 , VMon = 4.998 V , VCon = 5.02 V

In case the channels trip they cannot be ramped back up unless the alarms are cleared.

We have tried to connect the steering module to the DAQ pc via Luigi's USB-Ethernet adapter. 

That does not appear to work. We'll receive another USB-Ethernet adapter tomorrow, and it should be configured to have a static IP address : 192.168.121.1 and NetMask 255.255.255.0
For the time being, we are using Tom's laptop. 

> V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) 

There was a problem with CMakeLists for the V1725 driver, and the frontend was only connecting to 1 board. This is now fixed, and all the boards should have the same (TTL) logic level.

I have installed the python3-tkinter package from yum. I checked that "import tkinter" works from a python3 prompt.

We need python3 package tkinter installed on ds-proto-daq in order to run the steering module GUI. 

I don't have root privileges.

People: Edgar, Simone

Installed differential to single-ended converters in VME crate, and turned crate back on. elog:47/2

Channel count in the converters is 0 to 15 starting from the low end. Wired according to:
left board, channels 0..15 = PDM slot 1..16
right board, channels 0..8 = PDM slot 17..25

N.B. V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) elog:47/3

In ODB / Equipment / V1725_Data00 / Settings /  set Channel Mask to 0x1555 for all V1725 boards (enables channels 0 2 4 6 8 10 12)
(board 0 receives 7 PDM inputs, boards 1,2,3 receive 6 PDM each)

In https://ds-proto-daq.cern.ch/chronobox/ , set Enable Channel [ch_enable] = 0x3F3F3F7F , Channel Assignment [ch_assign] = 0x00393340.
9 central PDM's assigned to "top" group, external PDM's assigned to "bottom" group elog:47/1

Wiki instructions with the script to get the CDM back into a sensible state seem outdated. The variables seem fine, though ... 
[dsproto@ds-proto-daq ~]$ esper-tool read 192.168.1.5 cdm ext_clk
[49999632]
[dsproto@ds-proto-daq ~]$ esper-tool read 192.168.1.5 lmk pll1_ld
[1]
[dsproto@ds-proto-daq ~]$ esper-tool read 192.168.1.5 lmk pll2_ld
[1]

People:  Simone, Sam, Tom, Edgar

Tested IV curve python script that runs using MIDAS. Measured resistance consistant with resistor specification. elog:46/1

The resistance is attached to ch11, and we called the script (channels go from 0 to 24):
chnum=11 ; python3 iv_curve.py --no_steering --caen_chan $chnum --stop_v 3 --step_v 0.001 --filename ivcurve_`date +%s`_$chnum.txt 

Issues:

No readout of trip condiotion therefore python script overides power supply trip and attemps to bring back voltage to next value. Potentially dangerous. Script only terminates with the final voltage value. See attached plot elog:46/1

Script terminated with error message:

Traceback (most recent call last):
  File "iv_curve.py", line 287, in <module>
    iv.run()
  File "iv_curve.py", line 269, in run
    self.set_caen_and_wait_for_readback(voltage)
  File "iv_curve.py", line 223, in set_caen_and_wait_for_readback
    raise RuntimeError("CAEN HV didn't respond to request to set voltage to %s; latest readback voltage is %s" % (voltage, rdb))
RuntimeError: CAEN HV didn't respond to request to set voltage to 2.976; latest readback voltage is 0.052000001072883606

Crew: Edgar Sanchez Garcia, Sam Hill, Simone Stracka, Tom Thorpe, Yi Wang

Morning: Set up equipment for the IV curve measurements. We enable the interlock system in the SMU in order to go higher than 42 V (handle with care).

Afternoon:  Measuring IV curves.  Note there is a voltage drop in the LV cable that needs to be compensated for.  

Chan 1 is set at +3.3V drawing 1.46A; sense line reads +2.39V
Chan 2 is set at -3.3V drawing 1.46A; sense line reads -2.39V
IV curves are scanning from 0-80V in 200mV steps w/ 100uA limit: elog:45/2 , elog:45/3
The two runs are visible ( elog:45/1 for color code and mapping )
Compare to elog:45/4 , obtained when probing individual tiles under more controlled light conditions

Crew: Edgar Sanchez Garcia, Sam Hill, Simone Stracka, Tom Thorpe, Yi Wang

Morning: Tested photocurrents for MB2 on tabletop using SMU and Keithley switching matrix.  All measured between 1-10 microAmps.

Afternoon: Tested PDU mounting procedure with TPC assembled and with the mockup.  No showstoppers. This will be the procedure for installation
With the mockup on the table we double checked the resistances.  Confirmed all values are as before cooling indicating no additional damage to the wire-bondings.

Majority of 8, threshold at 1000 ADC count, 24 channels enabled (16 trigger signals)  10 Hz, 200 us window, trigger at 9.5 microseconds.

10 k events recorded.

By direct inspection pileup seems at the 10 % level in the 200 us window.

Majority of 5, threshold at 1000 ADC count, 24 channels enabled (16 trigger signals)  100 Hz, 30 us window.

38 k events recorded.

Majority of 5, threshold at 1000 ADC count, 13 channels enabled (10 trigger signals)  50 Hz, 30 us window.

58 k events recorded.

This morning tried to get some data with Vbias=65 V, which seems better for the SiPM in MB1.

Busy handling still erratic.

At the beginning it seemed to work fine with busy_invert off and busy_enable on.

After a while (maybe after the first busy signal?) the only work around seemed to be to disable the busy_enable

Trigger setup with a threshold of 1500 ADC count wrt to baseline, on just two channels.

This is also to limit the trigger to 50 Hz, since the busy still did not work.

100 K events on disk.

Laser Run with Vbias=65 V , new recommended value from PE group

500 k events acquired

Run 747 , 250 k events

as requested by alessandro 40% post trigger, acquisition window 16 us.

Run 718 has 25 channel readout Board 0 ch 0-15 and Board 0 ch 0-7. 

Laser triggered, some noise, maybe not so bad as is look here.

RMS of baseline is 200 ADC count. Single PE peak is about 100-200 ADC count from baseline.

Looking carefully, some very nice scintillation event are found.

Implemented a simple charge integration on the midas display, most of the channels look almost as good as this one.

Nothing on the daq side changed, a part from the DAC value moved to 5000, allowing a slightly greater dynamic range.

Run 703 is being writte on disk.

16 channels of board 00. 

Laser trigger at 10 Hz

ended with

[feoV1725.cxx:1033:read_trigger_event,ERROR] Error: did not receive a ZMQ bank. Stopping run.   ]

Tried to get laser sync signal into chronobox (clkin1 input), apparently all triggers dropped by chronobox

Problem is that the same now happen also with the regular setup where the clkin1 signal is taken by the dual timer.

Changed custom size to 500 (20 us), tested ok run 681

The DAQ seems to be ready for tests with proto-0 tomorrow.  

I had to invert the first busy input in order to get chronobox to produce triggers.  I modified the setup script

setup_chronobox.sh

so this is now the default.  Not sure why necessary.

Another interesting fact; it seems the chronobox only asks for a DHCP IP when it first boots.  I think that
chronobox and ds-proto-daq were rebooted at the same time; ds-proto-daq dhcp server was probably not running
when chronobox asked for IP.  chronobox got IP fine when it was power cycled.

Darkside people seem to be doing some tests at CERN next week.  It looks like they aren't going to use our DAQ (they will I think just use CAEN tools).  But we took 
opportunity to make TRIUMF DAQ work again.  There was a couple issues

1) I needed to inver the busy signal for input 0 to the chronobox in order to get any triggers out of the system.  Not sure why; I don't think I had to do that before.  But it is 
working now.

2) The CERN web proxy was not initially working.  Somehow I 'fixed' it by changing the proxy configuration variable of SERVICE_NAME from 'midas' to 'dummy'.  I don't 
understand why that fixed it.  But now you can see the DAQ here:

https://m-darkside.web.cern.ch/chronobox/

I have done a couple longer tests of the DAQ setup.  The runs were done with a high trigger rate of ~60Hz, with
the V1725 asserting their busy to throttle the trigger.

I found that after a couple hours (~5hours) that we would no longer be getting ZMQ packets from the chronobox. 
 You can see this with error message like

15:29:04.092 2019/04/11 [feov1725MTI00,ERROR] [feoV1725.cxx:1033:read_trigger_event,ERROR] Error: did not
receive a ZMQ bank.  Stopping run.
10:26:37.158 2019/04/11 [mhttpd,INFO] Run #657 started

If I start a new run I am still missing ZMQ packets.  However, if I restart the frontend program (and hence
re-initialize the ZMQ link), then the chronobox does start sending triggers again.  So it may be more of a
problem with the ZMQ setup in the MIDAS frontend.  Needs more investigation.

Several points:

1) Gave a series of tutorial on DAQ to DS people yesterday and today.  Got a bunch of feedback, which I will pass on when I'm back at TRIUMF.

2) The computer ds-proto-daq was offline when I got in to lab this morning.  Hmm, not clear what is wrong with computer.  Didn't happen the 
first day.  Maybe another power blip?  Maybe we need a UPS for this DAQ machine, to protect it from power blips.

3) Using instructions from Luke, reconfigured the CDM to use the clock from the chronobox. 

4) Added scripts for putting the chronobox and the CDM into a sensible state.  Scripts are 

/home/dsproto/online/dsproto_daq/setup_chronobox.sh
/home/dsproto/online/dsproto_daq/setup_cdm.sh

The scripts need to be rerun whenever the chronobox or VME crate are power cycled.

5) Fix some bugs and added some new plots to online monitoring.  In particular, added a bunch of plots related to the chronobox data.

6) Found some problems with monitoring of chronobox trigger primitives, which I passed onto Bryerton.

Notes on day:

1) Fixed the problem with the network interfaces.  Now the computer boots with the correct network configuration; outside world visible 
and private network on.

2) fan tray on VME crate seemed to be broken.  Got another VME crate from pool and installed it.  This VME crate seems to be working 
well. 

3) Recommissioned the DAQ setup.  Found a couple small bugs related to the V1725 self-trigger logic.  Fixed those and the V1725 self-
triggers seem to be working correctly.

4) Tried to install new CDM from TRIUMF (with ssh access), but clocks didn't stay synchronized.  Will bring module back to TRIUMF.

5) Added some code to V1725 frontend for clearing out the ZMQ buffers of extra events at the end of the run.  This is to protect 
against the case where the chronobox is triggering too fast for the V1725s.

The last elog didn't go out cleanly.  Modified the elogd.cfg to point to the proxy.  Try again.

Pierre and I got the CERN proxy setup for the Darkside prototype.

Using your CERN single-sign-on identity, you should be able to login to this page

https://m-darkside.web.cern.ch/

and see our normal MIDAS webpage.

The CERN server is proxying the port 80 on ds-proto-daq.  You can also see all the other services through the
same page:

elog:
https://m-darkside.web.cern.ch/elog/DS+Prototype/

chronobox:
https://m-darkside.web.cern.ch/chronobox/

js-root:
https://m-darkside.web.cern.ch/rootana/


_________________________________

Technical details

1) We followed these instructions for creating a SSO-proxy:

https://cern.service-now.com/service-portal/article.do?n=KB0005442

We pointed the proxy to port 80 on ds-proto-daq

2) On ds-proto-daq, we needed to poke a hole through the firewall for port 80:


firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="188.184.28.139/32" port
protocol="tcp" port="80" accept"
firewall-cmd --reload

[root@ds-proto-daq ~]# firewall-cmd --list-all
public (active)
...
	rule family="ipv4" source address="188.184.28.139/32" port port="80" protocol="tcp" accept

This firewall rule is pointing to some particular IP that seems to be the proxy side of the server:

[root@ds-proto-daq ~]# host 188.184.28.139
139.28.184.188.in-addr.arpa domain name pointer oostandardprod-7b34bdf1f3.cern.ch.

It is not clear if this particular IP will be stable in long term.

3) We needed to modify mhttpd so it would serve content to hosts other than localhost.  So changed mhttpd
command from 

mhttpd -a localhost -D

to 

mhttpd -D

The ChronoBox latest FW is loaded. Let tme know if this is what the chronobox should look like in term of registers.

Are we monitoring the PLL Lock Loss  (odb: /DEAP Alarm ?)

Here is the dump of the 5 first sync triggers without any physics trigger behind.

1 0x830b577 0x17d7855 0x6b33d22 0x6b33d22 8.992791 
2 0x830b577 0x17d7855 0x6b33d22 0x6b33d22 8.992791 
3 0x830b577 0x17d7855 0x6b33d22 0x6b33d22 8.992791 
4 0x830b577 0x17d7855 0x6b33d22 0x6b33d22 8.992791 

1 0x830b581 0x2faf097 0x535c4ea 0x535c4ea 6.992792 
2 0x830b581 0x2faf097 0x535c4ea 0x535c4ea 6.992792 
3 0x830b581 0x2faf095 0x535c4ec 0x535c4ec 6.992792 
4 0x830b581 0x2faf095 0x535c4ec 0x535c4ec 6.992792 

1 0x830b587 0x47868d7 0x3b84cb0 0x3b84cb0 4.992792 
2 0x830b587 0x47868d7 0x3b84cb0 0x3b84cb0 4.992792 
3 0x830b587 0x47868d7 0x3b84cb0 0x3b84cb0 4.992792 
4 0x830b587 0x47868d7 0x3b84cb0 0x3b84cb0 4.992792 

1 0x830b5a1 0x5f5e119 0x23ad488 0x23ad488 2.992794 
2 0x830b5a1 0x5f5e119 0x23ad488 0x23ad488 2.992794 
3 0x830b5a1 0x5f5e117 0x23ad48a 0x23ad48a 2.992794 
4 0x830b5a1 0x5f5e117 0x23ad48a 0x23ad48a 2.992794 

1 0x830b5ab 0x7735959 0xbd5c52 0xbd5c52 0.992795 
2 0x830b5ab 0x7735959 0xbd5c52 0xbd5c52 0.992795 
3 0x830b5ab 0x7735959 0xbd5c52 0xbd5c52 0.992795 
4 0x830b5ab 0x7735959 0xbd5c52 0xbd5c52 0.992795 


With the physics triggers:
1 0x830b5c1 0x1834497f 0xeffc6c42 0x100393be 21.493591 
2 0x830b5c1 0x1834497f 0xeffc6c42 0x100393be 21.493591 
3 0x830b5c1 0x1834497f 0xeffc6c42 0x100393be 21.493591 
4 0x830b5c1 0x1834497f 0xeffc6c42 0x100393be 21.493591 
1 0x830b5d3 0x1834abb5 0xeffc0a1e 0x1003f5e2 21.495601 
2 0x830b5d3 0x1834abb5 0xeffc0a1e 0x1003f5e2 21.495601 
3 0x830b5d3 0x1834abb5 0xeffc0a1e 0x1003f5e2 21.495601 
4 0x830b5d3 0x1834abb5 0xeffc0a1e 0x1003f5e2 21.495601 
1 0x830b5dd 0x18350d7f 0xeffba85e 0x100457a2 21.497603 
2 0x830b5dd 0x18350d7f 0xeffba85e 0x100457a2 21.497603 
3 0x830b5dd 0x18350d7f 0xeffba85e 0x100457a2 21.497603 
4 0x830b5dd 0x18350d7f 0xeffba85e 0x100457a2 21.497603 
1 0x830b5e3 0x183585e7 0xeffb2ffc 0x1004d004 21.500068 
2 0x830b5e3 0x183585e7 0xeffb2ffc 0x1004d004 21.500068 
3 0x830b5e3 0x183585e7 0xeffb2ffc 0x1004d004 21.500068 
4 0x830b5e3 0x183585e7 0xeffb2ffc 0x1004d004 21.500068 
1 0x830b5ed 0x1a10bb9b 0xee1ffa52 0x11e005ae 23.991535 
2 0x830b5ed 0x1a10bb9b 0xee1ffa52 0x11e005ae 23.991535 
3 0x830b5ed 0x1a10bb9b 0xee1ffa52 0x11e005ae 23.991535 
4 0x830b5ed 0x1a10bb9b 0xee1ffa52 0x11e005ae 23.991535 
1 0x830b5fb 0x1a111f65 0xee1f9696 0x11e0696a 23.993578 
2 0x830b5fb 0x1a111f65 0xee1f9696 0x11e0696a 23.993578 
3 0x830b5fb 0x1a111f65 0xee1f9696 0x11e0696a 23.993578 
4 0x830b5fb 0x1a111f65 0xee1f9696 0x11e0696a 23.993578 
1 0x830b607 0x1a118115 0xee1f34f2 0x11e0cb0e 23.995577 
2 0x830b607 0x1a118115 0xee1f34f2 0x11e0cb0e 23.995577 
3 0x830b607 0x1a118115 0xee1f34f2 0x11e0cb0e 23.995577 
4 0x830b607 0x1a118115 0xee1f34f2 0x11e0cb0e 23.995577 
1 0x830b611 0x1a11e2c7 0xee1ed34a 0x11e12cb6 23.997577 
2 0x830b611 0x1a11e2c7 0xee1ed34a 0x11e12cb6 23.997577 
3 0x830b611 0x1a11e2c7 0xee1ed34a 0x11e12cb6 23.997577 
4 0x830b611 0x1a11e2c7 0xee1ed34a 0x11e12cb6 23.997577 


The ZMQ0 banks:
#banks:5 Bank list:-ZMQ0W200W201W202W203-
Bank:ZMQ0 Length: 40(I*1)/10(I*4)/10(Type) Type:Unsigned Integer*4
   1-> 0x000a5f1c 0x000000c4 0x00000001 0x0ebd5273 0x00000001 0x00010001 0xffffffff 0x00000000 
   9-> 0xffff0000 0x00000000 
------------------------ Event# 10 ------------------------
#banks:5 Bank list:-ZMQ0W200W201W202W203-
Bank:ZMQ0 Length: 40(I*1)/10(I*4)/10(Type) Type:Unsigned Integer*4
   1-> 0x000a5f1d 0x000000c5 0x00000001 0x0ebd5279 0x00000001 0x00010001 0xffffffff 0x00000000 
   9-> 0xffff0000 0x00000000 
[dsproto@ds-proto-daq dsproto_daq]$ 

Found that the Trigger out from the CB is on output1
Trigger / Not used
Clock   / Not used

Set frontend to use NIM in/out instead of TTL as there is a nice
NIM-TTL-NIM adaptor CAEN Nim module available

> f) However, I found that the frontend program still consistently failed with this error when the trigger rate
> was above the maximum sustainable:
> 
> Deferred transition.  First call of wait_buffer_empty. Stopping run
> [feov1725MTI00,ERROR] [v1725CONET2.cxx:685:ReadEvent,ERROR] Communication error: -2
> [feov1725MTI00,ERROR] [feoV1725.cxx:654:link_thread,ERROR] Readout routine error on thread 0 (module 0)
> [feov1725MTI00,ERROR] [feoV1725.cxx:655:link_thread,ERROR] Exiting thread 0 with error
> Stopped chronobox run; status = 0
> Segmentation fault

I sort of 'fix' this problem.  There is some sort of problems between the V1725 readout	thread and the system call
to esper-tools to stop the run.  Some collision between the system resources for these two calls causes the readout
thread ReadEvent call to fail.  I 'fix' the problem by adding in the end_of_run part a 500us pause of the readout
threads before I make the system call to esper-tool.  

Odd.  In principle I think that the system calls and the readout threads are running on different cores.  So not
clear what the problem was.  Should figure out better diagnosis and fix problem properly.

a) Bryerton implemented new version of the firmware.  New features:

1) run start/stop state
2) at run start 6 events are sent with 200ms separation
3) there is a greater set of counters about trigger, as well as configuring the trigger.

b) chronobox webpage can be seen here:

https://ds-proto-daq.triumf.ca/chronobox/

The mod_tdm page gives configuration of run state and trigger.  In particular

- button to start/stop run
- button to do manual trigger
- configure which channels are TOP and which are BOTTOM
- configure the number of TOP or BOTTOM channels that need to figure
- DecisionType: true= TOP AND BOTTOM groups must fire; false = TOP OR BOTTOM groups can fire.

c) Start and stop run can also be done on command line with esper tool:

esper-tool write -d true 192.168.1.3 mod_tdm run
esper-tool write -d false 192.168.1.3 mod_tdm run

d) I modified the V1725 frontend to integrate the chronobox start/stop.

At run start
- configure V1725s
- start chronobox with the command-line esper-tool call

At end run
- add deferred transition function which stops run (with esper-tool), then checks whether all the events have
cleared from ring buffers.
- once ring buffers are cleared, finish stopping the V1725s.

In the long run should somehow do the start/stop commands with some http post command, rather than command line
to esper-tool.

e) With the start/stop run could start to compare the timestamps of V1725s and confirm that they matched (ie,
all V1725s got reset at the same time).  Also, all the events are cleared from buffers.

f) However, I found that the frontend program still consistently failed with this error when the trigger rate
was above the maximum sustainable:

Deferred transition.  First call of wait_buffer_empty. Stopping run
[feov1725MTI00,ERROR] [v1725CONET2.cxx:685:ReadEvent,ERROR] Communication error: -2
[feov1725MTI00,ERROR] [feoV1725.cxx:654:link_thread,ERROR] Readout routine error on thread 0 (module 0)
[feov1725MTI00,ERROR] [feoV1725.cxx:655:link_thread,ERROR] Exiting thread 0 with error
Stopped chronobox run; status = 0
Segmentation fault


Next steps:

1) Fix the seg-fault for high rate running.
2) More detailed timestamp checking, with cm_msg(ERRORs)
3) Bryerton is now working on the event FIFO on chronobox.  That will be next thing to integrate.

The SDcard is currently required for operation of the device, the ext3/4 filesystem will immediately fail upon removal.


> Summary of setup of chronobox on network (mostly done by Pierre):
> 
> 1) Hook up USB connection from chronobox to ds-proto-daq.  Start serial-USB connection by doing
> 
> /home/dsproto/online/dsproto_daq/serialusb
> 
> can login through serial link.  Added the ssh key for dsproto@ds-proto-daq to the authorized_keys file on
> chronobox linux.
> 
> ctrl-a, ctrl-x to stop picocom
> 
> 2) On ds-proto-daq, setup dhcpd server.  Configuration in this file
> 
> /etc/dhcp/dhcpd.conf
> 
> dhpcd is bond to the second NIC only.  Configure DHCP to give IP 192.168.1.3 to chronobox.  In /etc/hosts set
> 192.168.1.3 to have hostname chronobox
> 
> Power cycle chronobox; it successfully gets IP:
> 
> [dsproto@ds-proto-daq dsproto_daq]$ ping chronobox
> PING chronobox (192.168.1.3) 56(84) bytes of data.
> 64 bytes from chronobox (192.168.1.3): icmp_seq=1 ttl=64 time=0.180 ms
> 
> 3) Can see the chronobox webpage locally as
> 
> http://chronobox:8080
> 
> 4) Some esper tool to access data:  did following to setup esper
> 
>   734  yum install python2-pip
>   735  pip install esper-tool
>   736  yum install ncurses
>   738  yum install ncurses-devel
>   739  pip install esper-tool
> 
> Can then connect to the chronobox by doing
> 
> esper-tool interactive http://chronobox:8080
> 
> This is as much as I understand at this point... more exploring now...
> 
> 5) Pierre unplugged the SD card to take it back to Bryerton's room.  But I guess this was bad.
> 
> lots of errors on serialUSB link now and the webpage doesn't work anymore:
> 
> [ 1246.665269] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.904770] EXT4-fs error: 339 callbacks suppressed
> [ 1246.679162] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.904780] EXT4-fs error (device mmcblk0p2):
> ext4_find_entry:1437: inode #2: com[ 1246.692884] blk_partition_remap: fail for partition 2
> m syslogd: reading directory lblock 0
> [ 1246.709020] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.929199] blk_partition_remap: fail for partition 2
> [ 1246.717394] blk_partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.942971] EXT4-fs error (device mmcblk0p2):
> ext4_find_entry:1437: inode #2: com[ 1246.731255] blk_partition_remap: fail for partition 2
> m syslogd: reading directory lblock 0
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.953760] blk_partition_remap: fail for [ 1246.747337]
> blk_partition_remap: fail for partition 2
> partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.967532] EXT4-fs error (device mmcblk0p2):
> ext4_find_entry:1437: [ 1246.763481] blk_partition_remap: fail for partition 2
> inode #2: comm syslogd: reading directory lblock 0
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.978320] blk_partition_rem[ 1246.779511] blk_partition_remap:
> fail for partition 2
> ap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.992091] EXT4-fs error (device mmcblk0p2): ext4_find[
> 1246.795586] blk_partition_remap: fail for partition 2
> _entry:1437: inode #2: comm syslogd: reading directory lblock 0
> Jan  1 00:20:46 buildroot user.warn kernel: [ 1243.002885] blk_[ 1246.811671] blk_partition_remap: fail for
> partition 2
> partition_remap: fail for partition 2
> Jan  1 00:20:46 buildroot user.crit kernel: [ 1243.016656] EXT4-fs error (device mmcblk0p[ 1246.827750]
> blk_partition_remap: fail for partition 2

Summary of setup of chronobox on network (mostly done by Pierre):

1) Hook up USB connection from chronobox to ds-proto-daq.  Start serial-USB connection by doing

/home/dsproto/online/dsproto_daq/serialusb

can login through serial link.  Added the ssh key for dsproto@ds-proto-daq to the authorized_keys file on
chronobox linux.

ctrl-a, ctrl-x to stop picocom

2) On ds-proto-daq, setup dhcpd server.  Configuration in this file

/etc/dhcp/dhcpd.conf

dhpcd is bond to the second NIC only.  Configure DHCP to give IP 192.168.1.3 to chronobox.  In /etc/hosts set
192.168.1.3 to have hostname chronobox

Power cycle chronobox; it successfully gets IP:

[dsproto@ds-proto-daq dsproto_daq]$ ping chronobox
PING chronobox (192.168.1.3) 56(84) bytes of data.
64 bytes from chronobox (192.168.1.3): icmp_seq=1 ttl=64 time=0.180 ms

3) Can see the chronobox webpage locally as

http://chronobox:8080

4) Some esper tool to access data:  did following to setup esper

  734  yum install python2-pip
  735  pip install esper-tool
  736  yum install ncurses
  738  yum install ncurses-devel
  739  pip install esper-tool

Can then connect to the chronobox by doing

esper-tool interactive http://chronobox:8080

This is as much as I understand at this point... more exploring now...

5) Pierre unplugged the SD card to take it back to Bryerton's room.  But I guess this was bad.

lots of errors on serialUSB link now and the webpage doesn't work anymore:

[ 1246.665269] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.904770] EXT4-fs error: 339 callbacks suppressed
[ 1246.679162] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.904780] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: inode #2: com[ 1246.692884] blk_partition_remap: fail for partition 2
m syslogd: reading directory lblock 0
[ 1246.709020] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.929199] blk_partition_remap: fail for partition 2
[ 1246.717394] blk_partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.942971] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: inode #2: com[ 1246.731255] blk_partition_remap: fail for partition 2
m syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.953760] blk_partition_remap: fail for [ 1246.747337]
blk_partition_remap: fail for partition 2
partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.967532] EXT4-fs error (device mmcblk0p2):
ext4_find_entry:1437: [ 1246.763481] blk_partition_remap: fail for partition 2
inode #2: comm syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1242.978320] blk_partition_rem[ 1246.779511] blk_partition_remap:
fail for partition 2
ap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1242.992091] EXT4-fs error (device mmcblk0p2): ext4_find[
1246.795586] blk_partition_remap: fail for partition 2
_entry:1437: inode #2: comm syslogd: reading directory lblock 0
Jan  1 00:20:46 buildroot user.warn kernel: [ 1243.002885] blk_[ 1246.811671] blk_partition_remap: fail for
partition 2
partition_remap: fail for partition 2
Jan  1 00:20:46 buildroot user.crit kernel: [ 1243.016656] EXT4-fs error (device mmcblk0p[ 1246.827750]
blk_partition_remap: fail for partition 2

I retested the chronobox trigger generation:

1) Inserting moderate sized pulse into channel 8 of V1725-0.
2) Configured threshold of V1725 so that channel triggers LVDS pulse into chronobox
3) Fan-out trigger out from chronobox to all V1725s.
4) busy signal from each V1725 fed into the chronobox
5) Start run, then push trigger from 20Hz up to 200Hz
6) System running stably!  Actual trigger rate about 60.2Hz.  The almost_full condition is set to 32 on the
V1725s and the estored on each board fluctuates below 32.

https://ds-proto-daq.triumf.ca/HS/Buffers/eStored?hscale=300&fgroup=Buffers&fpanel=eStored&scale=10m

The busy light on the V1725 never comes on, good.

The next thing we need is some way to start/stop the trigger generation on the chronobox, so that at
begin-of-run triggers do not get sent before the V1725s are finished configuration.

Other notes:

a) It turned out that the register to set the V1725 channel trigger threshold was different for RAW vs ZLE
firmware (0x1080 vs 0x1060); after fixing that the channel threshold seemed to work as expected.
b) Map for chronobox NIM cables:

channel 1-4: busy IN from V1725
channel 5,7,8: trigger OUT from chronobox
channel 6: clock OUT from chronobox

I installed a PCIe 1Gbps network card and configured it as a private network.  The PC (ds-proto-daq) is
192.168.1.1.  I guess we can make the chronobox 192.168.1.2.

[root@ds-proto-daq ~]# ifconfig enp5s0
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::bb27:5db:f778:d584  prefixlen 64  scopeid 0x20<link>
        ether 68:05:ca:8e:66:5c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 4145 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf72c0000-f72e0000  

It turns out that the ZLE V1725 firmware we are using only supports reading out up to 4000 samples per channel.
 We need 80000 samples to readout 200us, which is requirement.

So we switch the V1725s to use the backup firmware on the board, which is the standard waveform firmware. 
Firmware version is 17200410.

Expected data size for event with 200us of data: 200000ns * 1/4 ns/sample * 16 ch * 4 boards * 2 bytes/sample =
 6.4MB per event

Measured max event rate of 60Hz with 385MB/s with 200us readout.

Needed to increase max event size, set buffer organization to 6 and set almost full to 32 in order to
accommodate the larger event size.

Changed some registers for different firmware:
- V1725_SELFTRIGGER_LOGIC

More tests needed

For Reference,

I put a simple schematic for the Trigger/Run control.

Bryerton, please have a look. Let's try to issue 3..5 SW trigger before opening the HW trigger.

Checking again Marco's A3818:
- Port #2 (third from top of card) is acting up.
- change the SFP makes no difference.
- Symptoms: Get stuck on Rx/Tx.
- Need more investigation...

I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.

First check the maximum trigger rate and maximum data rate for different sample lengths (for each channels):

sample length      Maximum rate   MB/s    CPU % (per thread)
16us               0.44kHz        231     20
6.4us              1.04kHz        214     20  
3.2us              1.95kHz        198     22   
1.6us              3.12kHz        160     26   
0.8us              4.88kHz        127     30   (what are these threads doing?)

Look at the code more.  See that there is a maximum size for the block transfer of 10kB.  Increase this to 130kB
(which is the maximum amount that this board can make per event).  Now find

sample length     Maximum rate       MB/s     CPU % (per thread)
16us              0.68kHz            346      7
6.4us             1.46kHz            300      10 
1.6us             3.6kHz             190      25

Good.  So for long samples we are actually slightly above the maximum transfer rate of 85MB/s*4 = 340MB/s

Tried writing out the data to disk at the maximum 345MB/s rate; the DAQ can't keep up.  Maximum rate was more
like 270MB/s.
I think the mlogger was actually fine.  But I think the write-to-disk speed of the harddrive could not keep up.
 So I think we are limited 
by hardware in that case.  We would need a large raid array to be able to write faster.

Confirmed this ETTT configuration is working.

                     ETTT Enabled [22..21] = b10
                      |
                      v
Data [0x811C] = 0x 00 4 D 013C

                ETTT Time [47..32]
                 |  Ch Mask[16..0]  Time[31..0]
                 |    TTTTTT           |
Header 1         v    v    v           v
0xa0001914 0x00 0025 ff 0xff1ca598 0xe9c6e8a1   < event 1
0xa0001914 0x00 0025 ff 0xff1ca599 0xe9c82e25   < event 2

dTime :  0x25e9c82e25 &#8722; 0x25e9c6e8a1 = 0x14584 => 83332
Time interval: 8ns  => 666.7e-6s => Freq: 1500Hz corresponding the current trigger rate

PAA

Somehow the trigger rate was not matching the trigger source.
Find out that Link 3 was not collecting and possibly holding the fragment assembly in the main thread.
Swap Link3 <-> Link0 on the V1725, restarted.
Needs further investigation!

Date rate is fine now! CPU load is balanced on all 4 threads (~25%)
- irqbalance disabled
- change affinity for A3818 to cpu9: /etc/rc.local add: echo 0200 > /proc/irq/136/smp_affinity
  Check : watch -n 0.1 'cat /proc/interrupts'

Maximum Trigger rate (HW buffer not rising) 1950 Evt/s => 200MB/s 
for event size of 100KB composed of 4 banks with 32us per channel.

I added javascript webdisplay of the V1725 waveforms to the Darkside setup.  You can see the waveform display here:

https://ds-proto-daq.triumf.ca/CS/webdisplay

You can also see the webdisplay that comes built into the JSROOT webserver here:

https://ds-proto-daq.triumf.ca/rootana/

For Pierre: I went through and reset the baselines for all the channels.

A little progress on a couple fronts:

1) We got the baseline DAC setting working correctly from both the CAEN command line tool and from the MIDAS
frontend.  Needed to add some extra sleep before the DAC settings were modified.

2) We figured out how to set the threshold for self-triggering on each channel. It is actually register 0x1n60
for this firmware.  The threshold is defined relative to the online calculated baseline. The threshold is
settable from the ODB.  However, the polarity of the triggering is currently hard-coded to be negative (bit 6 of
register 0x1n64).

3) We could confirm that the LVDS outputs from the V1725 appeared and disappeared as expected when the
self-trigger threshold was changed.

4) We added some basic documentation in Markdown here:

https://bitbucket.org/ttriumfdaq/dsproto_daq/src/master/

Added pair of 10TB hard-drives in raid-1 to ds-proto-daq.  MIDAS data files will get written to this raid volume.

[dsproto@ds-proto-daq tmp]$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        9.1T  180M  8.6T   1% /data
[dsproto@ds-proto-daq tmp]$ cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdc1[1] sdb1[0]
      9766435648 blocks super 1.0 [2/2] [UU]
      [>....................]  resync =  0.9% (88214784/9766435648) finish=4372.8min speed=36886K/sec
      bitmap: 73/73 pages [292KB], 65536KB chunk

unused devices: <none>

I modified the analyzer so that it shows data from all four modules.

I find that there is evidence of corruption of the V1725 ADC data on a couple channels.   You can see an example of a waveform with corruption in the attachment.  You see that there seems to be a bunch of fluctuations of exactly 16 
ADC counts, which seems unphysical.  So far I see this problem in these channels:

module 0, ch 0
module 0, ch 1
module 0, ch 13
module 1, ch 4
module 3, ch 6

I saw corruption similar to this on the V1730 readout.  CAEN does have a scheme for ADC calibration (by poking a register), which worked well for the V1730.  I thought that I implemented this ADC calibration for the V1725, but I might 
have messed it.  I'll look at this again.

Pierre figured out that NIM crate not working.  We now see LVDS outputs from the individual channels firing. 
Each set of two different channels is ganged together into a single self-trigger output.  By setting the
register 0x1n84 to 3 we enable so that if either input channel fires then the self-trigger for that group fires.

Bryerton provided CDM outputting 50MHz clock; all V1725s now running with external clock.

Modified frontend to readout 4 modules and 16 channels per module.

Still need to modify the analyzer to show data for all 4 modules.

Couple weeks of work documented in one elog...

V1725 serial numbers: 455, 392, 460, 462, 474

firmware: DPP-ZLE+

The 3818 kernel module not getting loaded on start-up... it is in /etc/rc.local, but not getting called

[root@ds-proto-daq ~]# grep 3818 /etc/rc.local 
# Load A3818 driver...
/sbin/insmod  /home/dsproto/packages/A3818Drv-1.6.0/src/a3818.ko

Called manually for now...

Reworked and cleaned up DEAP version of V1725 code
- Updated register map
- Removed smartQT code

Quick test: the V1725 seems to be getting busy at a maximum rate of 10kHz (with no samples being saved though).

Got preliminary documentation from CAEN for ZLE-plus firmware... posted here:

/home/dsproto/packages/CAEN_ZLE_Info

In particular, manual shows how the data structure is different for ZLE-plus data banks (as compared to V1720 ZLE data banks).

Fixed the /etc/rc.local setup so that the A3818 driver is loaded and mhttpd/mlogger is started on reboot.

Looking at V1725-ZLE register list.  I don't entirely understand how we are going to do the trigger outputs and the busy.  The manual makes it clear that the triggers from each pair of channels are combined together and send to the trigger logic.  So there should be 8 trigger primitives from the board.  The LVDS IO connector allows to configure groups of 4 outputs as being for either the trigger outputs or the busy/veto outputs.  So I guess I will configure the first two groups to output the trigger primitives and the third group to output the BUSY information.

Added code to do the ADC calibration; added extra equipment for reading out ADC temperatures periodically.

Renewed the lets-encrypt SSL certificate... but haven't set up automatic renewal yet...

[root@ds-proto-daq ~]# certbot renew --apache
[root@ds-proto-daq ~]# systemctl restart httpd

Analyzer program basically working.  Pushed to bitbucket:

https://bitbucket.org/ttriumfdaq/dsproto_analyzer/src

1) Install and tweak elog:

[root@ds-proto-daq ~]# cat /etc/elogd.cfg 
[global]
port = 8084
MTP host = trmail.triumf.ca
Use Email Subject = {$logbook} $Subject
Remove on reply = Author
Quote on reply = 1
URL = https://ds-proto-daq/elog/

[DS Prototype]
Theme = default
Comment = ELOG for DS Prototype MIDAS DAQ
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem, Problem Fixed, Configuration, Other
Options Category = General, Hardware, Digitizer, Trigger, MIDAS, Software, Other
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type
Email all = lindner@triumf.ca

[root@ds-proto-daq ~]# systemctl start elogd.service
[root@ds-proto-daq ~]# systemctl is-active elogd.service
active
[root@ds-proto-daq ~]# systemctl enable elogd.service

2) Tweak apache and restart 

[root@ds-proto-daq ~]# grep elog /etc/httpd/conf.d/ssl-ds-proto-daq.conf 
ProxyPass /elog/ http://localhost:8084/ retry=1
[root@ds-proto-daq ~]# systemctl restart httpd

3) change MIDAS to use elog

[dsproto@ds-proto-daq bin]$ odbedit 
[local:dsproto:R]/>cd Elog/
[local:dsproto:R]/Elog>create STRING URL
String length [32]: 256
[local:dsproto:R]/Elog>set URL https://ds-proto-daq.triumf.ca/elog/DS+Prototype/

1) Install and tweak elog:

[root@ds-proto-daq ~]# cat /etc/elogd.cfg 
[global]
port = 8084
MTP host = trmail.triumf.ca
Use Email Subject = {$logbook} $Subject
Remove on reply = Author
Quote on reply = 1
URL = https://ds-proto-daq/elog/

[DS Prototype]
Theme = default
Comment = ELOG for DS Prototype MIDAS DAQ
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem, Problem Fixed, Configuration, Other
Options Category = General, Hardware, Digitizer, Trigger, MIDAS, Software, Other
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type
Email all = lindner@triumf.ca

[root@ds-proto-daq ~]# systemctl start elogd.service
[root@ds-proto-daq ~]# systemctl is-active elogd.service
active
[root@ds-proto-daq ~]# systemctl enable elogd.service

2) Tweak apache and restart 

[root@ds-proto-daq ~]# grep elog /etc/httpd/conf.d/ssl-ds-proto-daq.conf 
ProxyPass /elog/ http://localhost:8084/ retry=1
[root@ds-proto-daq ~]# systemctl restart httpd

3) change MIDAS to use elog

[dsproto@ds-proto-daq bin]$ odbedit 
[local:dsproto:R]/>cd Elog/
[local:dsproto:R]/Elog>create STRING URL
String length [32]: 256
[local:dsproto:R]/Elog>set URL https://ds-proto-daq.triumf.ca/elog/DS+Prototype/