HW-Time-Slice Firmware-Filter Software vx2740 Steering Module Reports MVM MVM Vexos MVM-Bug listing MVM TRIUMF Local DS Prototype DS Cryogenic For Shifters BCIT-31 ChronoBox Run Operation DS-DAQ
  CERN DS-Proto0 read-only backup, Page 6 of 8  Not logged in ELOG logo
ID Date Author Typedown Category Subject
  151   14 Oct 2020 08:39 Tom ThorpeRoutineGeneralLog regarding MB2 removal from Proto-0

Drive folder with data files and some pictures:  https://drive.google.com/drive/folders/1Sr27kQTOO2kciBY7v-wYBP06tWLVEjiP

Lab notes @ CERN

11/10/2020

11:00 – First look at MB2.  Noticed on some of the PDMs that some of the tile PCB are more delaminated than others.  PDM 25 also had the FEB connectors more inserted than any other PDMs.  No other differences noticed.

11:30 – Connected SMU and switching matrix to MB2.  Power w/o steering module.  Checked that all channels match by running photo current tests with and w/o a flashlight on one corner.  Using the computer form Pisa with Labview scripts from Matteo.  Everything is ok.

12:00 – Replace plastic shields around Proto-0 for IV curve measurements in dark conditions.  PDM 1 breakdown is around 63V.  Looks ok.  PDM 2 is over 65V…. Seems high.

12:30 - PDM 6 People walking, disrupted the light conditions.  IV curve will be jagged.  Will go for lunch.  Closing door and turning off exterior lights.  Current will decrease

15:30 – IV curves look fine.  …/MB2_retriveval_CERN/data_files/iv_warm_1

16:00 – Took forward IV curves.  Saved as …/MB2_retriveval_CERN/data_files/forward_warm_1.  They look fine.  Checked old pictures of the lamination of the tile PCB and they have always varied in thickness, from before the LNGS LN test.

16:45 – Power FEBs on to look at noise spectra with 20V bias on the tiles.  

18:00 – PDM 2, 4, 6, and 25 have very odd power spectra.  George said they look like they’re not connected.

18:30 – Saving FFT to disk for analysis.  Saved as …/MB2_retriveval_CERN/data_files/noise_warm_1

19:00 – PDM 2,4,6 and 25 have one of the differential lines disconnected.  Using a light and looking at the differential outputs on the scope, this is obvious.  20V bias and FEBs powered at 5V.  Saved as …/MB2_retriveval_CERN/data_files/waveform_light_1

19:30 – Started another round of reverse IV curves in dark conditions.  Saved as …/MB2_retriveval_CERN/data_files/iv_warm_2

21:15 – IV still going.  EP will come back to shut off the LV supplies

 

12/12/2020

10:00 – Checking connectivity of all the differential lines.  One side of PDM 6 is not connected.  2, 4, and 25 are connected.  Doesn’t explain what we saw yesterday.  Channel 2 and 4 now are responding to light on both differential lines.  The only thing that has changed is unplugging the external signal cable and plugging it back in.  PDM 6 is the only one now not responding on both lines.

10:30 – Checking the power spectra.  PDM 4 and 6 are bad on both differential lines.  PDM 2 looks fine.  Seems to be non-repeatable.

11:00 – Disconnected the signal patch board from the finger strip.  Holding the patch and reconnecting the entire signal path and pinging now shows PDM 6 and 25 have one differential line disconnected.

11:30 – Checking the power spectra on every PDM channel directly from the fingerstrip with a flat cable.

12:30 – PDMs 2, 4, 6 power spectra from the fingerstrip look normal.  Both connections are there.  Light response is normal.  Channel 13, however, now has one differential line disconnected and the other looks abnormal.  Saving waveform data with 60Hz light on as waveform_light_2.

All of this is pointing to multiple connection issues…

15:00 – It looks like PDM 13 is disconnected from the fingerstrip, or at least it is pulled further away than the others.  This is likely also due to the removal of the patch.  PDM 2 also looks pulled a bit further.

15:30 – Unmounted MB2 and put on the stand to do an optical inspection.  The microscope didn’t allow a close enough look at the wirebonds to search for any broken ones.  A close look by eye didn’t reveal any obvious damage.

16:00 – Packed everything up.  MB2 needs a closer look, likely needs to be disassembled.

Attachment 1: forward_warm_1.png
forward_warm_1.png
Attachment 2: iv_warm_1.png
iv_warm_1.png
Attachment 3: iv_warm_2.png
iv_warm_2.png
  49   28 Oct 2019 08:33 Ben SmithProblem FixedSoftwareNeed python3 package tkinter on DAQ pc
I have installed the python3-tkinter package from yum. I checked that "import tkinter" works from a python3 prompt.
  50   28 Oct 2019 14:10 Ben SmithProblem FixedSoftwareConverters installed in VME crate
> V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) 

There was a problem with CMakeLists for the V1725 driver, and the frontend was only connecting to 1 board. This is now fixed, and all the boards should have the same (TTL) logic level.
  53   01 Nov 2019 06:07 Ben SmithProblem FixedSoftwareRestoration of network settings
For a few days we were unable to access ds-proto-daq remotely. We were also unable to reach the outside world from ds-proto-daq.

To gain more control over the network settings, I have disabled the automatic configuration by dracut. I created the file /etc/dracut.conf.d/no-ifcfg.conf with the single line `omit_dracutmodules+="ifcfg"`.

I then set the following contents in /etc/sysconfig/network-scripts:

# cat ifcfg-enp0s31f6
NAME=enp0s31f6
DEVICE=enp0s31f6
ONBOOT=yes
NETBOOT=yes
UUID=4887c207-13da-4424-8307-116ef2163fd8
IPV6INIT=no
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
HWADDR=18:31:BF:2E:00:12
BOOTPROTO=dhcp
IPV6_PRIVACY=no
AUTOCONNECT_PRIORITY=1
DNS1=137.138.16.5
DNS2=137.138.17.5
PEERDNS=no


# cat ifcfg-enp5s0
TYPE=Ethernet
PROXY_METHOD=None
BROWSER_ONLY=no
BOOTPROTO=static
DHCPCLASS=
IPADDR=192.168.1.1
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=
DEFROUTE=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
HWADDR=68:05:CA:8E:66:5C
ONBOOT=yes
DEVICE=enp5s0


Network settings are now correct, and have survived a reboot. We are able to talk to both the local network and the outside world. We are also able to connect to ds-proto-daq from lxplus, and via the web proxy.
  54   01 Nov 2019 06:15 Ben SmithProblem FixedSoftwareRestoration of elog
After booting, the elog was only showing logs from before July, and was missing the "For Shifters" category. It appears we were loading the incorrect config file. The running command was:

/usr/sbin/elogd -D -c /etc/elogd.cfg -s /usr/share/elog/ -d /var/lib/elog/logbooks/

while it should have been:

/usr/sbin/elogd -D -x -c /home/dsproto/online/elog/elogd.cfg -s /usr/share/elog


rc.local was set up to run the latter command, but rc.local~ had the prior version. Perhaps that was being picked up. I have deleted the rc.local~ version, and will see if elog starts correctly next time we reboot.
  89   06 Nov 2019 12:28 Ben SmithProblem FixedGeneralEvening data taking, purity check
> Not sure if the trigger threshold is correctly set, there is an error:

Sorry, I made a change to the baseline calculation and forgot to update the threshold tool. It's updated now.

The thresholds currently seem to be set to 12801 - this is 2699 below the nominal baseline of 15500.
  95   08 Nov 2019 02:21 Ben SmithProblem FixedSoftwareIncreased max waveform length that we can take
I've increased various thresholds so that we can now take 3000us (3ms) waveforms.

The changes are:
* /Experiment/MAX_EVENT_SIZE in ODB is now 50,000,000
* /Experiment/Buffer sizes/SYSTEM in ODB is now 500,000,000
* max_event_size for V1725 frontend is now 45,000,000 (we use this in several places, so I now have `#define V1725_MAX_EVENT_SIZE 45000000` in v1725CONET2.hxx)
* event_buffer_size for V1725 frontend is now big enough for 10 of these largest events

The main data-taking works smoothly with 3ms traces, albeit at a low event rate. 

However protoDisplay segfaults after a few events. The backtrace is random each time, so we're clearly overflowing a buffer somewhere. I will try to look into it, but it will probably be painful and I don't think it's the highest priority issue to fix (it works "fine" for the usual 200us waveforms).
  105   09 Nov 2019 07:29 Marco RescignoProblem FixedMIDASthis morning problem

frontend process still alive but slow, causing lot of rejected triggers.

stopped and restarted the feov1725 process to fix this

  114   11 Nov 2019 00:47 Yi / EdgarProblem FixedDigitizerProblem fixed by restarting the VME crate
  118   11 Nov 2019 23:57 Yi WangProblem FixedTriggerID 117 problem is fixed

The problem reported in ID 117 is somehow fixed this morning.

  12   08 Feb 2019 12:08 ThomasProblemHardwareInstalled Marco's A3818; didn't work
I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.
  13   13 Feb 2019 16:45 PierreProblemHardwareA3818 from Marco
Checking again Marco's A3818:
- Port #2 (third from top of card) is acting up.
- change the SFP makes no difference.
- Symptoms: Get stuck on Rx/Tx.
- Need more investigation...
  33   12 Jul 2019 02:05 Marco RescignoProblemTriggerMB1 test in proto-0 setup/1

Tried to get laser sync signal into chronobox (clkin1 input), apparently all triggers dropped by chronobox

Problem is that the same now happen also with the regular setup where the clkin1 signal is taken by the dual timer.

 

 

  40   16 Jul 2019 02:41 Marco RescignoProblemTriggerBusy handling

This morning tried to get some data with Vbias=65 V, which seems better for the SiPM in MB1.

Busy handling still erratic.

At the beginning it seemed to work fine with busy_invert off and busy_enable on.

After a while (maybe after the first busy signal?) the only work around seemed to be to disable the busy_enable

  46   26 Oct 2019 07:33 Sam HillProblemSoftwareTest of IV Script

People:  Simone, Sam, Tom, Edgar

Tested IV curve python script that runs using MIDAS. Measured resistance consistant with resistor specification. elog:46/1

The resistance is attached to ch11, and we called the script (channels go from 0 to 24):
chnum=11 ; python3 iv_curve.py --no_steering --caen_chan $chnum --stop_v 3 --step_v 0.001 --filename ivcurve_`date +%s`_$chnum.txt 

 

Issues:

No readout of trip condiotion therefore python script overides power supply trip and attemps to bring back voltage to next value. Potentially dangerous. Script only terminates with the final voltage value. See attached plot elog:46/1

Script terminated with error message:

Traceback (most recent call last):
  File "iv_curve.py", line 287, in <module>
    iv.run()
  File "iv_curve.py", line 269, in run
    self.set_caen_and_wait_for_readback(voltage)
  File "iv_curve.py", line 223, in set_caen_and_wait_for_readback
    raise RuntimeError("CAEN HV didn't respond to request to set voltage to %s; latest readback voltage is %s" % (voltage, rdb))
RuntimeError: CAEN HV didn't respond to request to set voltage to 2.976; latest readback voltage is 0.052000001072883606

 

Attachment 1: IV_curve_29K.png
IV_curve_29K.png
  48   28 Oct 2019 06:28 Simone StrackaProblemSoftwareNeed python3 package tkinter on DAQ pc

We need python3 package tkinter installed on ds-proto-daq in order to run the steering module GUI. 

I don't have root privileges.

 

  51   28 Oct 2019 14:56 Simone StrackaProblemSoftwareIssues with network configuration for steering module

We have tried to connect the steering module to the DAQ pc via Luigi's USB-Ethernet adapter. 

That does not appear to work. We'll receive another USB-Ethernet adapter tomorrow, and it should be configured to have a static IP address : 192.168.121.1 and NetMask 255.255.255.0
For the time being, we are using Tom's laptop. 

 

  112   10 Nov 2019 12:15 E. SanchezProblemMIDASProblem DAQ

Error during data taking  1573416708 21:11:48.151 2019/11/10 [feov1725MTI00,INFO] V1725 PLL loss lock Board:0 (vmeAcq=0x0)

It stop saving event but run in process. Impossible to stop the run. I tried to reset feov1725MTl00. After that error: 

1573416857 21:14:17.792 2019/11/10 [feChronoEsper,INFO] Client 'feov1725MTI00' on database 'ODB' removed by db_cleanup called by cm_periodic_tasks because pid 9724 does not exist

Run stopped after some time. Error 21:21:19.121 2019/11/10 [mhttpd,ERROR] [midas.cxx:3951:cm_transition_call,ERROR] cannot connect to client "feov1725MTI00" on host localhost, port 40944, status 503   ╳

Trying to restart feov1725MTI00. Status initializing

It is not possible to initialize 

 

 

  113   11 Nov 2019 00:37 Yi WangProblemMIDAScannot connect to rootana

The connection to rootana is lost.

No data taking can be started.

  117   11 Nov 2019 10:05 Sam HillProblemGeneralEvening data taking

Attemted the following run, and it started but the rate was shoing 0 and it got stuck on 5 events

Was unable to connect to Chronobox webpage even after turning VME crate off and on.

Abandoned run and will attempt to fix in the morning.

Run 1073

Gas pocket: ON (thickness unknown)

Fields:  drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252mbarg

SiPM HV: 65 V

Number of Events: 10K

 

ELOG V3.1.4-cb3afcd8