It looks like there's a memory leak in the high voltage driver, and we were running out of memory (using a lot of swap). That *may* be related to the recent instability, but I'm not sure.

If things are more stable now, then the memory leak was probably the cause. If not, then we should try stopping the V1725 program, power-cycling the VME crate, then starting the V1725 program.


Yi: restarted the VME crate twice, now it seems like smooth.

When we start the HV program (CAEN_SY4527), it spews error messages of the form:
"16:58:30.035 2019/11/13 [CAEN_SY4527,ERROR] [dd_sy4527.cxx:280:fParam_get,ERROR] Param Not Found Type : 0"

This is because the LV module in the SY5527 crate doesn't support some of the parameters that the driver wants to read 
(e.g. it support "RUpTime" rather than "RUp"). For our current usage, the errors are benign. But it does add to the list 
of deficiencies of the driver provided by core midas:
* No automatic discovery of total number of channels
* Confusing ODB structure
* No discovery of which features a module supports
* Odd detection of whether a channel is on/off
* ODB status doesn't update if module settings are changed with a different interface (e.g. SSH or Caen's Java-based GUI).

All these limitations are imposed by the core HV driver "class" in midas. Assuming we will be using Caen HV modules for the
foreseeable future, I think I should write a new slow control frontend that is less generic, but fixes all the above issues.

The chronobox isn't serving the human-interactive webpage that should be visible at https://m-darkside.web.cern.ch/chronobox/ - it responds with "File not found". Oddly, it is still responding to API calls at /read_var, /write_var etc.

So shifters can still change between laser/noise/physics runs using the "Run type" page at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type (which uses the API), but can't as easily monitor / sanity check the chronobox behaviour.

In the first instance, I think it would be useful to power-cycle the chronobox. If that doesn't work, I'm not sure how to proceed.

I have set up automatic copying of data from ds-proto-daq to darkside01, using the lazy logger.

There are 2 lazy logger channels set up - one for lz4 files and one for gz files. It will take some time for all the files to copied over. Note that lazy logger writes a message into the midas log for every file that it copies. The lazy logger channels can be started from the Program page like all the other midas-related programs.

Once files have been copied to darkside01 (by lazy logger) and to EOS (by Pablo/Edgar) we can delete files from ds-proto-daq. We will probably not be able to do this until tomorrow (copying is quite slow).

Lazy logger has caught up with the backlog, and continues to transfer new files to darkside01 as they are written.
EOS currently contains data for runs 822-1201.

I have deleted the following runs from ds-proto-daq:
* runs < 700 that were taken with gzip compression
* runs 800-1100 that are available on EOS and darkside01

The following files remain:
* runs 700-800 that haven't been copied to EOS (but are on darkside01). If they're of interest, they should be copied to EOS.
* runs 1200+ that haven't been copied to EOS yet (but will be soon).

Disk usage has improved from 85% to 33%.

A few months ago I created a new midas frontend for CAEN HV crates. It behaves more like what users expect - changes made through other interfaces (e.g. ssh or the Java-based GUI) will be reflected in the ODB. The ODB structure is created dynamically based on which modules are present in the crate, and the parameters that each module supports (e.g. whether you specify a ramp-up rate in V/s or a ramp-up time in s).

When I first wrote the frontend, ds-proto-daq was offline. Pierre reminded me today that I should actually deploy the new code.

The new program code is at ~/online/dsproto_sy4527/caen_hv*. The executable is at ~/online/bin/caenhv_fe. The frontend is available on the "Programs" page in mhttpd. If we want to run the old HV frontend, it is still available at ~/online/bin/sy4527.

Connection parameters for the frontend are found in the ODB at /Equipment/CAEN_HV/Settings/Global. Currently the crate appears to be off, so the frontend will refuse to start properly as it cannot connect to 192.168.1.9. If the frontend is started with the crate on, it will create more directories in /Equipment/CAEN_HV/Settings (one for each module). The "Voltage" page on the experiment now points to a generic webpage that will work with arbitrary combinations of modules.

I have added a backup of the CERN proto-0 ELOG at TRIUMF, in case ds-proto-daq is unavailable in future.

The backup ELOG is at https://ladd00.triumf.ca/elog-ds/. There are other logbooks hosted in the same place (requiring a username/password), but the CERN backup logbooks (at the bottom of the list) are available to all. I have configured things so that the TRIUMF replicas are read-only. To add new entries one must still use the CERN logbooks.

The logbooks are rsynced every hour via cron.

Gas pocket: ON

Fields: wire gate 3780 V, 1st ring: 4500 V, cathode: 7000 V

Threshold: 1000 ADCc below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: 300 mbarg

SiPM HV: 65 V (noise run)

Comment: noise conditions good. Rate ~30Hz. Look for S2s!

Ramp down the fields, in 200V steps.
Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Gas pocket: ON

Fields: OFF

Threshold: 400 ADCc below baseline

Threshold extend:  5x16 = 80ns

Comment: Noise appeared on event 4200

Ramp UP the fields, in 100V steps.

Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Gas pocket: ON

Fields: ON (wire gate 3780 V, 1st ring 4180 V, cathode 6180 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 242 mbarg

SiPM HV: 65V

Comment: noise conditions good, 1e5 evts

Gas pocket: ON

Fields: drift field ON (1st ring 4180 V, cathode 6180 V), extraction field OFF (wire gate 0 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 244 mbarg

SiPM HV: 65 V

Comment: noise conditions good, 1e5 evts

Gas pocket: ON

Fields: drift field OFF (1st ring 0 V, cathode 0 V), extraction field OFF (wire gate 0 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 244 mbarg

SiPM HV: 65 V

Comment: noise conditions good, 1e5 evts

Gas pocket: ON

Fields: drift field OFF (1st ring 0 V, cathode 0 V), extraction field OFF (wire gate 0 V)

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

TPC Pressure: 244 mbarg

SiPM HV: 50 V (noise run)

Comment: noise conditions good. Thresholds are too high, rate below 1 event/sec, only few events saved. Postpone to next day.

Increased drift and extraction fields.

Current settings: wire gate 5100 V, 1st ring: 6100 V, cathode: 11100 V

P.S. Increased by in the morning by Yi.

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar) [Edgar\Yi: correction: OFF (during recirculation tpc pressure about 400 mbar)]

Fields: drift 200 V/cm, extraction 2.8 kV/cm (wire grid 3780 V, 1st ring 4180 V, cathode 6180 V) 

Threshold: 400 ADC below baseline (baselines equalized to 15500)

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment:

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar) [Edgar\Yi: correction: OFF (during recirculation tpc pressure about 400 mbar)]

Fields: drift 100 V/cm, extraction 2.8 kV/cm (wire grid 3780 V, 1st ring 3580 V, cathode 3780 V) [Yi: 1st ring setting is incorrect, inverted fied between wire grid and field cage ring]

Threshold: 400 ADC below baseline (baselines equalized to 15500)

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment:

Gas pocket: ON (thinkness unknown, since the TPC pressure is >200 mbar) [Edgar\Yi: correction: OFF (during recirculation tpc pressure about 400 mbar)]

Fields: drift 500 V/cm, extraction 2.8 kV/cm (wire grid 3780 V, 1st ring 2780 V, cathode 7780 V) [Yi: 1st ring setting is incorrect, inverted fied between wire grid and field cage ring]

Threshold: 400 ADC below baseline (baselines equalized to 15500)

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Comment:

Gas pocket: ON

Fields: ON

Threshold: 200 ADCc below baseline

Threshold extend:  5x16 = 80ns

Comment: Noise appeared on event 48500