I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

The cable has been fixed. Socket 0001/007 is still not active.

Wrong MAC address fixed in CERN network setting, now the machine is visible from external network too:

From CERN network (even laptop) can connect by 

ssh -XY dsproto@ds-proto-daq2.cern.ch.  (same password as local account in ds-proto-daq)

From ds-proto-daq can also connect by:

ssh darkside01

That is to say that the two network card are now associated to two different names.. but the "real" name is still darkside01.

(will fix).

 enjoy

A few months ago I created a new midas frontend for CAEN HV crates. It behaves more like what users expect - changes made through other interfaces (e.g. ssh or the Java-based GUI) will be reflected in the ODB. The ODB structure is created dynamically based on which modules are present in the crate, and the parameters that each module supports (e.g. whether you specify a ramp-up rate in V/s or a ramp-up time in s).

When I first wrote the frontend, ds-proto-daq was offline. Pierre reminded me today that I should actually deploy the new code.

The new program code is at ~/online/dsproto_sy4527/caen_hv*. The executable is at ~/online/bin/caenhv_fe. The frontend is available on the "Programs" page in mhttpd. If we want to run the old HV frontend, it is still available at ~/online/bin/sy4527.

Connection parameters for the frontend are found in the ODB at /Equipment/CAEN_HV/Settings/Global. Currently the crate appears to be off, so the frontend will refuse to start properly as it cannot connect to 192.168.1.9. If the frontend is started with the crate on, it will create more directories in /Equipment/CAEN_HV/Settings (one for each module). The "Voltage" page on the experiment now points to a generic webpage that will work with arbitrary combinations of modules.

I have added a backup of the CERN proto-0 ELOG at TRIUMF, in case ds-proto-daq is unavailable in future.

The backup ELOG is at https://ladd00.triumf.ca/elog-ds/. There are other logbooks hosted in the same place (requiring a username/password), but the CERN backup logbooks (at the bottom of the list) are available to all. I have configured things so that the TRIUMF replicas are read-only. To add new entries one must still use the CERN logbooks.

The logbooks are rsynced every hour via cron.

I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.

Checking again Marco's A3818:
- Port #2 (third from top of card) is acting up.
- change the SFP makes no difference.
- Symptoms: Get stuck on Rx/Tx.
- Need more investigation...

Tried to get laser sync signal into chronobox (clkin1 input), apparently all triggers dropped by chronobox

Problem is that the same now happen also with the regular setup where the clkin1 signal is taken by the dual timer.

This morning tried to get some data with Vbias=65 V, which seems better for the SiPM in MB1.

Busy handling still erratic.

At the beginning it seemed to work fine with busy_invert off and busy_enable on.

After a while (maybe after the first busy signal?) the only work around seemed to be to disable the busy_enable

People:  Simone, Sam, Tom, Edgar

Tested IV curve python script that runs using MIDAS. Measured resistance consistant with resistor specification. elog:46/1

The resistance is attached to ch11, and we called the script (channels go from 0 to 24):
chnum=11 ; python3 iv_curve.py --no_steering --caen_chan $chnum --stop_v 3 --step_v 0.001 --filename ivcurve_`date +%s`_$chnum.txt 

Issues:

No readout of trip condiotion therefore python script overides power supply trip and attemps to bring back voltage to next value. Potentially dangerous. Script only terminates with the final voltage value. See attached plot elog:46/1

Script terminated with error message:

Traceback (most recent call last):
  File "iv_curve.py", line 287, in <module>
    iv.run()
  File "iv_curve.py", line 269, in run
    self.set_caen_and_wait_for_readback(voltage)
  File "iv_curve.py", line 223, in set_caen_and_wait_for_readback
    raise RuntimeError("CAEN HV didn't respond to request to set voltage to %s; latest readback voltage is %s" % (voltage, rdb))
RuntimeError: CAEN HV didn't respond to request to set voltage to 2.976; latest readback voltage is 0.052000001072883606

We need python3 package tkinter installed on ds-proto-daq in order to run the steering module GUI. 

I don't have root privileges.

We have tried to connect the steering module to the DAQ pc via Luigi's USB-Ethernet adapter. 

That does not appear to work. We'll receive another USB-Ethernet adapter tomorrow, and it should be configured to have a static IP address : 192.168.121.1 and NetMask 255.255.255.0
For the time being, we are using Tom's laptop. 

Error during data taking  1573416708 21:11:48.151 2019/11/10 [feov1725MTI00,INFO] V1725 PLL loss lock Board:0 (vmeAcq=0x0)

It stop saving event but run in process. Impossible to stop the run. I tried to reset feov1725MTl00. After that error: 

1573416857 21:14:17.792 2019/11/10 [feChronoEsper,INFO] Client 'feov1725MTI00' on database 'ODB' removed by db_cleanup called by cm_periodic_tasks because pid 9724 does not exist

Run stopped after some time. Error 21:21:19.121 2019/11/10 [mhttpd,ERROR] [midas.cxx:3951:cm_transition_call,ERROR] cannot connect to client "feov1725MTI00" on host localhost, port 40944, status 503   ╳

Trying to restart feov1725MTI00. Status initializing

It is not possible to initialize 

The connection to rootana is lost.

No data taking can be started.

Attemted the following run, and it started but the rate was shoing 0 and it got stuck on 5 events

Was unable to connect to Chronobox webpage even after turning VME crate off and on.

Abandoned run and will attempt to fix in the morning.

Run 1073

Gas pocket: ON (thickness unknown)

Fields:  drift 200 V/cm, extraction 2.8 kV/cm

Threshold: 1000 ADC below baseline

Trace length: 200us total, 100 us pre-trigger

Threshold extend:  5x16 = 80ns

TPC Pressure:  252mbarg

SiPM HV: 65 V

Number of Events: 10K

09:15:33.955 2019/11/12 [feov1725MTI00,ERROR] [feoV1725.cxx:663:link_thread,ERROR] Exiting thread 3 with error

It looks like there's a memory leak in the high voltage driver, and we were running out of memory (using a lot of swap). That *may* be related to the recent instability, but I'm not sure.

If things are more stable now, then the memory leak was probably the cause. If not, then we should try stopping the V1725 program, power-cycling the VME crate, then starting the V1725 program.


Yi: restarted the VME crate twice, now it seems like smooth.

The chronobox isn't serving the human-interactive webpage that should be visible at https://m-darkside.web.cern.ch/chronobox/ - it responds with "File not found". Oddly, it is still responding to API calls at /read_var, /write_var etc.

So shifters can still change between laser/noise/physics runs using the "Run type" page at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type (which uses the API), but can't as easily monitor / sanity check the chronobox behaviour.

In the first instance, I think it would be useful to power-cycle the chronobox. If that doesn't work, I'm not sure how to proceed.

I have installed the python3-tkinter package from yum. I checked that "import tkinter" works from a python3 prompt.

> V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) 

There was a problem with CMakeLists for the V1725 driver, and the frontend was only connecting to 1 board. This is now fixed, and all the boards should have the same (TTL) logic level.