Run 747 , 250 k events

as requested by alessandro 40% post trigger, acquisition window 16 us.

Drive folder with data files and some pictures:  https://drive.google.com/drive/folders/1Sr27kQTOO2kciBY7v-wYBP06tWLVEjiP

Lab notes @ CERN

11/10/2020

11:00 – First look at MB2.  Noticed on some of the PDMs that some of the tile PCB are more delaminated than others.  PDM 25 also had the FEB connectors more inserted than any other PDMs.  No other differences noticed.

11:30 – Connected SMU and switching matrix to MB2.  Power w/o steering module.  Checked that all channels match by running photo current tests with and w/o a flashlight on one corner.  Using the computer form Pisa with Labview scripts from Matteo.  Everything is ok.

12:00 – Replace plastic shields around Proto-0 for IV curve measurements in dark conditions.  PDM 1 breakdown is around 63V.  Looks ok.  PDM 2 is over 65V…. Seems high.

12:30 - PDM 6 People walking, disrupted the light conditions.  IV curve will be jagged.  Will go for lunch.  Closing door and turning off exterior lights.  Current will decrease

15:30 – IV curves look fine.  …/MB2_retriveval_CERN/data_files/iv_warm_1

16:00 – Took forward IV curves.  Saved as …/MB2_retriveval_CERN/data_files/forward_warm_1.  They look fine.  Checked old pictures of the lamination of the tile PCB and they have always varied in thickness, from before the LNGS LN test.

16:45 – Power FEBs on to look at noise spectra with 20V bias on the tiles.  

18:00 – PDM 2, 4, 6, and 25 have very odd power spectra.  George said they look like they’re not connected.

18:30 – Saving FFT to disk for analysis.  Saved as …/MB2_retriveval_CERN/data_files/noise_warm_1

19:00 – PDM 2,4,6 and 25 have one of the differential lines disconnected.  Using a light and looking at the differential outputs on the scope, this is obvious.  20V bias and FEBs powered at 5V.  Saved as …/MB2_retriveval_CERN/data_files/waveform_light_1

19:30 – Started another round of reverse IV curves in dark conditions.  Saved as …/MB2_retriveval_CERN/data_files/iv_warm_2

21:15 – IV still going.  EP will come back to shut off the LV supplies

12/12/2020

10:00 – Checking connectivity of all the differential lines.  One side of PDM 6 is not connected.  2, 4, and 25 are connected.  Doesn’t explain what we saw yesterday.  Channel 2 and 4 now are responding to light on both differential lines.  The only thing that has changed is unplugging the external signal cable and plugging it back in.  PDM 6 is the only one now not responding on both lines.

10:30 – Checking the power spectra.  PDM 4 and 6 are bad on both differential lines.  PDM 2 looks fine.  Seems to be non-repeatable.

11:00 – Disconnected the signal patch board from the finger strip.  Holding the patch and reconnecting the entire signal path and pinging now shows PDM 6 and 25 have one differential line disconnected.

11:30 – Checking the power spectra on every PDM channel directly from the fingerstrip with a flat cable.

12:30 – PDMs 2, 4, 6 power spectra from the fingerstrip look normal.  Both connections are there.  Light response is normal.  Channel 13, however, now has one differential line disconnected and the other looks abnormal.  Saving waveform data with 60Hz light on as waveform_light_2.

All of this is pointing to multiple connection issues…

15:00 – It looks like PDM 13 is disconnected from the fingerstrip, or at least it is pulled further away than the others.  This is likely also due to the removal of the patch.  PDM 2 also looks pulled a bit further.

15:30 – Unmounted MB2 and put on the stand to do an optical inspection.  The microscope didn’t allow a close enough look at the wirebonds to search for any broken ones.  A close look by eye didn’t reveal any obvious damage.

16:00 – Packed everything up.  MB2 needs a closer look, likely needs to be disassembled.

Run 1096 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1097 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock (laser NIM), laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Problem: The access to the Chronobox page is not possible.  Rate 400 Hz 

Run 1166 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Trigger rate went to 0, run was stopped manually.

Run 1169 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1170 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1232 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1233 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1252 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Low trigger rate - must restart manually 

Run 1253 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MAX

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

Run 1254 Laser run

Chronobox: External trigger Enable

Chronobox: Enable Channel [ch_enable]: 0x00000000

Laser Intensity: MIN

Threshold: external clock, laser set to 1kHz

Trace length: 12us, 5us pre-trigger

SIPM HV: 65V

Number of events: 100k

The HV board from Naples did not turn on: Yury gave it to the CAEN guys to check and/or bring back for replacement.

Yi and Luigi rented a new HV module (A1519). The HV module (A1520P) we used for tests of the I-V script is also present in the Mainframe (see below).
The first 24 channels (slots 0 and 1) are therefore HV. If the new HV does not show up on time we'll try and adapt the cables to work with A1519.

The A2517A module is LV. This is currently operated from the DAQ pc using CAEN_HVPSS_ChannelsController.jnlp (located in the Desktop/SteeringModule folder).
The three low voltage channels (0,1,2) should be turned on at the same time by setting Pw = ON.

Settings: 

Channel 0 and 1: I0Set = 2.0 A , V0Set = 2.5 V ,  UNVThr = 0 V, OVVThr = 3.0 V, Intck = Disabled
IMon = 1.44 (this depends on the illumination) , VMon = 2.48 V , VCon = 2.79 V

Channel 2: I0Set = 1.0 A , V0Set = 5.0 V ,  UNVThr = 0 V, OVVThr = 5.5 V, Intck = Disabled
IMon = 0.07 , VMon = 4.998 V , VCon = 5.02 V

In case the channels trip they cannot be ramped back up unless the alarms are cleared.

We have tried to connect the steering module to the DAQ pc via Luigi's USB-Ethernet adapter. 

That does not appear to work. We'll receive another USB-Ethernet adapter tomorrow, and it should be configured to have a static IP address : 192.168.121.1 and NetMask 255.255.255.0
For the time being, we are using Tom's laptop. 

I installed a PCIe 1Gbps network card and configured it as a private network.  The PC (ds-proto-daq) is
192.168.1.1.  I guess we can make the chronobox 192.168.1.2.

[root@ds-proto-daq ~]# ifconfig enp5s0
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::bb27:5db:f778:d584  prefixlen 64  scopeid 0x20<link>
        ether 68:05:ca:8e:66:5c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 4145 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf72c0000-f72e0000  

I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.

I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).

Increased drift and extraction fields.

Current settings: wire gate 5100 V, 1st ring: 6100 V, cathode: 11100 V

P.S. Increased by in the morning by Yi.

IV of 25 SiPM put in ivdata_191119 directory

Temp at start: 

A: 242.7K    B: 278.16K    C: 86.4K    D: 240.89K

Temp at End:

A: 248.87K    B: 278.24K    C: 86.37K    D: 247.75K

Took iv curves, data is on DAQ PC here: /home/dsproto/online/dsproto_sy4527/ivdata_191120

The problem reported in ID 117 is somehow fixed this morning.

Several points:

1) Gave a series of tutorial on DAQ to DS people yesterday and today.  Got a bunch of feedback, which I will pass on when I'm back at TRIUMF.

2) The computer ds-proto-daq was offline when I got in to lab this morning.  Hmm, not clear what is wrong with computer.  Didn't happen the 
first day.  Maybe another power blip?  Maybe we need a UPS for this DAQ machine, to protect it from power blips.

3) Using instructions from Luke, reconfigured the CDM to use the clock from the chronobox. 

4) Added scripts for putting the chronobox and the CDM into a sensible state.  Scripts are 

/home/dsproto/online/dsproto_daq/setup_chronobox.sh
/home/dsproto/online/dsproto_daq/setup_cdm.sh

The scripts need to be rerun whenever the chronobox or VME crate are power cycled.

5) Fix some bugs and added some new plots to online monitoring.  In particular, added a bunch of plots related to the chronobox data.

6) Found some problems with monitoring of chronobox trigger primitives, which I passed onto Bryerton.

Notes on day:

1) Fixed the problem with the network interfaces.  Now the computer boots with the correct network configuration; outside world visible 
and private network on.

2) fan tray on VME crate seemed to be broken.  Got another VME crate from pool and installed it.  This VME crate seems to be working 
well. 

3) Recommissioned the DAQ setup.  Found a couple small bugs related to the V1725 self-trigger logic.  Fixed those and the V1725 self-
triggers seem to be working correctly.

4) Tried to install new CDM from TRIUMF (with ssh access), but clocks didn't stay synchronized.  Will bring module back to TRIUMF.

5) Added some code to V1725 frontend for clearing out the ZMQ buffers of extra events at the end of the run.  This is to protect 
against the case where the chronobox is triggering too fast for the V1725s.

I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

Confirmed this ETTT configuration is working.

                     ETTT Enabled [22..21] = b10
                      |
                      v
Data [0x811C] = 0x 00 4 D 013C

                ETTT Time [47..32]
                 |  Ch Mask[16..0]  Time[31..0]
                 |    TTTTTT           |
Header 1         v    v    v           v
0xa0001914 0x00 0025 ff 0xff1ca598 0xe9c6e8a1   < event 1
0xa0001914 0x00 0025 ff 0xff1ca599 0xe9c82e25   < event 2

dTime :  0x25e9c82e25 &#8722; 0x25e9c6e8a1 = 0x14584 => 83332
Time interval: 8ns  => 666.7e-6s => Freq: 1500Hz corresponding the current trigger rate

PAA

Not sure if the trigger threshold is correctly set, there is an error:

Traceback (most recent call last):
  File "/home/dsproto/online/dsproto_analyzer/threshold_tool.py", line 71, in <module>
    main()
  File "/home/dsproto/online/dsproto_analyzer/threshold_tool.py", line 47, in main
    (mean, rms) = fit_res
ValueError: too many values to unpack (expected 2)

the trigger threshold for all the runs in this Elog is 2699 ADCc below the baseline (see Elog 89)
 

Run 943

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 945

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 946

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 948

Gas pocket: ON (thinkness 7mm)

Fields: drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 949

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 951

Gas pocket: ON (thinkness 7mm)

Fields: drift 500 V/cm, extraction 3.78 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 952

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k

Run 954

Gas pocket: ON (thinkness 7mm)

Fields: OFF

Threshold: 1600 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

TPC Pressure: >200 mbarg

SiPM HV: 65 V

Number of events: 10k