Couple weeks of work documented in one elog...

V1725 serial numbers: 455, 392, 460, 462, 474

firmware: DPP-ZLE+

The 3818 kernel module not getting loaded on start-up... it is in /etc/rc.local, but not getting called

[root@ds-proto-daq ~]# grep 3818 /etc/rc.local 
# Load A3818 driver...
/sbin/insmod  /home/dsproto/packages/A3818Drv-1.6.0/src/a3818.ko

Called manually for now...

Reworked and cleaned up DEAP version of V1725 code
- Updated register map
- Removed smartQT code

Quick test: the V1725 seems to be getting busy at a maximum rate of 10kHz (with no samples being saved though).

Got preliminary documentation from CAEN for ZLE-plus firmware... posted here:

/home/dsproto/packages/CAEN_ZLE_Info

In particular, manual shows how the data structure is different for ZLE-plus data banks (as compared to V1720 ZLE data banks).

Fixed the /etc/rc.local setup so that the A3818 driver is loaded and mhttpd/mlogger is started on reboot.

Looking at V1725-ZLE register list.  I don't entirely understand how we are going to do the trigger outputs and the busy.  The manual makes it clear that the triggers from each pair of channels are combined together and send to the trigger logic.  So there should be 8 trigger primitives from the board.  The LVDS IO connector allows to configure groups of 4 outputs as being for either the trigger outputs or the busy/veto outputs.  So I guess I will configure the first two groups to output the trigger primitives and the third group to output the BUSY information.

Added code to do the ADC calibration; added extra equipment for reading out ADC temperatures periodically.

Renewed the lets-encrypt SSL certificate... but haven't set up automatic renewal yet...

[root@ds-proto-daq ~]# certbot renew --apache
[root@ds-proto-daq ~]# systemctl restart httpd

Analyzer program basically working.  Pushed to bitbucket:

https://bitbucket.org/ttriumfdaq/dsproto_analyzer/src

Added pair of 10TB hard-drives in raid-1 to ds-proto-daq.  MIDAS data files will get written to this raid volume.

[dsproto@ds-proto-daq tmp]$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        9.1T  180M  8.6T   1% /data
[dsproto@ds-proto-daq tmp]$ cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdc1[1] sdb1[0]
      9766435648 blocks super 1.0 [2/2] [UU]
      [>....................]  resync =  0.9% (88214784/9766435648) finish=4372.8min speed=36886K/sec
      bitmap: 73/73 pages [292KB], 65536KB chunk

unused devices: <none>

Confirmed this ETTT configuration is working.

                     ETTT Enabled [22..21] = b10
                      |
                      v
Data [0x811C] = 0x 00 4 D 013C

                ETTT Time [47..32]
                 |  Ch Mask[16..0]  Time[31..0]
                 |    TTTTTT           |
Header 1         v    v    v           v
0xa0001914 0x00 0025 ff 0xff1ca598 0xe9c6e8a1   < event 1
0xa0001914 0x00 0025 ff 0xff1ca599 0xe9c82e25   < event 2

dTime :  0x25e9c82e25 &#8722; 0x25e9c6e8a1 = 0x14584 => 83332
Time interval: 8ns  => 666.7e-6s => Freq: 1500Hz corresponding the current trigger rate

PAA

I installed Marco's A3818 PCIe card.  Didn't seem to work.  I got communication errors talking to link 2.  The
communication problems didn't happen right away, but happened once the run started.

I swapped the fibres going to port 2 and port 3 on the A3818.  The problem stayed with port 2.  So I conclude
that this A3818 module is no good.

Checking again Marco's A3818:
- Port #2 (third from top of card) is acting up.
- change the SFP makes no difference.
- Symptoms: Get stuck on Rx/Tx.
- Need more investigation...

I installed a PCIe 1Gbps network card and configured it as a private network.  The PC (ds-proto-daq) is
192.168.1.1.  I guess we can make the chronobox 192.168.1.2.

[root@ds-proto-daq ~]# ifconfig enp5s0
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::bb27:5db:f778:d584  prefixlen 64  scopeid 0x20<link>
        ether 68:05:ca:8e:66:5c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 4145 (4.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf72c0000-f72e0000  

Found that the Trigger out from the CB is on output1
Trigger / Not used
Clock   / Not used

Set frontend to use NIM in/out instead of TTL as there is a nice
NIM-TTL-NIM adaptor CAEN Nim module available

Crew: Edgar Sanchez Garcia, Sam Hill, Simone Stracka, Tom Thorpe, Yi Wang

Morning: Tested photocurrents for MB2 on tabletop using SMU and Keithley switching matrix.  All measured between 1-10 microAmps.

Afternoon: Tested PDU mounting procedure with TPC assembled and with the mockup.  No showstoppers. This will be the procedure for installation
With the mockup on the table we double checked the resistances.  Confirmed all values are as before cooling indicating no additional damage to the wire-bondings.

Crew: Edgar Sanchez Garcia, Sam Hill, Simone Stracka, Tom Thorpe, Yi Wang

Morning: Set up equipment for the IV curve measurements. We enable the interlock system in the SMU in order to go higher than 42 V (handle with care).

Afternoon:  Measuring IV curves.  Note there is a voltage drop in the LV cable that needs to be compensated for.  

Chan 1 is set at +3.3V drawing 1.46A; sense line reads +2.39V
Chan 2 is set at -3.3V drawing 1.46A; sense line reads -2.39V
IV curves are scanning from 0-80V in 200mV steps w/ 100uA limit: elog:45/2 , elog:45/3
The two runs are visible ( elog:45/1 for color code and mapping )
Compare to elog:45/4 , obtained when probing individual tiles under more controlled light conditions

People: Edgar, Simone

Installed differential to single-ended converters in VME crate, and turned crate back on. elog:47/2

Channel count in the converters is 0 to 15 starting from the low end. Wired according to:
left board, channels 0..15 = PDM slot 1..16
right board, channels 0..8 = PDM slot 17..25

N.B. V1725 board #0 logic level is set to TTL (boards #1, #2, #3 to NIM) elog:47/3

In ODB / Equipment / V1725_Data00 / Settings /  set Channel Mask to 0x1555 for all V1725 boards (enables channels 0 2 4 6 8 10 12)
(board 0 receives 7 PDM inputs, boards 1,2,3 receive 6 PDM each)

In https://ds-proto-daq.cern.ch/chronobox/ , set Enable Channel [ch_enable] = 0x3F3F3F7F , Channel Assignment [ch_assign] = 0x00393340.
9 central PDM's assigned to "top" group, external PDM's assigned to "bottom" group elog:47/1

Wiki instructions with the script to get the CDM back into a sensible state seem outdated. The variables seem fine, though ... 
[dsproto@ds-proto-daq ~]$ esper-tool read 192.168.1.5 cdm ext_clk
[49999632]
[dsproto@ds-proto-daq ~]$ esper-tool read 192.168.1.5 lmk pll1_ld
[1]
[dsproto@ds-proto-daq ~]$ esper-tool read 192.168.1.5 lmk pll2_ld
[1]

The HV board from Naples did not turn on: Yury gave it to the CAEN guys to check and/or bring back for replacement.

Yi and Luigi rented a new HV module (A1519). The HV module (A1520P) we used for tests of the I-V script is also present in the Mainframe (see below).
The first 24 channels (slots 0 and 1) are therefore HV. If the new HV does not show up on time we'll try and adapt the cables to work with A1519.

The A2517A module is LV. This is currently operated from the DAQ pc using CAEN_HVPSS_ChannelsController.jnlp (located in the Desktop/SteeringModule folder).
The three low voltage channels (0,1,2) should be turned on at the same time by setting Pw = ON.

Settings: 

Channel 0 and 1: I0Set = 2.0 A , V0Set = 2.5 V ,  UNVThr = 0 V, OVVThr = 3.0 V, Intck = Disabled
IMon = 1.44 (this depends on the illumination) , VMon = 2.48 V , VCon = 2.79 V

Channel 2: I0Set = 1.0 A , V0Set = 5.0 V ,  UNVThr = 0 V, OVVThr = 5.5 V, Intck = Disabled
IMon = 0.07 , VMon = 4.998 V , VCon = 5.02 V

In case the channels trip they cannot be ramped back up unless the alarms are cleared.

Ramp down the fields, in 200V steps.
Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Ramp UP the fields, in 100V steps.

Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Increased drift and extraction fields.

Current settings: wire gate 5100 V, 1st ring: 6100 V, cathode: 11100 V

P.S. Increased by in the morning by Yi.

The CAEN VME crate now gets its IP address from the ds-proto-daq DHCP server. To do this, I set the IP address of the crate to 0.0.0.0 (using the front panel interface) then reset the crate (not just a power cycle - pressing the reset button is required).

I am now able to ping the crate, but am NOT able to load the webpage it is supposed to serve (I just get a "connection refused"). I can't see any configuration settings that would disable the webpage or would cause it to be served on a non-standard port. I may have missed something though.

The analysis server from Roma (darkside01) is now connected to the local network. It is not yet connected to CERN.

There are 2 network ports on the rear of the machine:
* The port labelled "1":
 * MAC: 9c:71:3a:22:d5:62
 * Name: enp2s0f0
 * Configured for DHCP
 * Connected to local switch
 * Assigned IP 192.168.1.10 by ds-proto-daq
* The port labelled "2":
 * MAC: 9c:71:3a:22:d5:63
 * Name: enp2s0f1
 * Configured for DHCP
 * Should be connected to CERN network

I have set up password-less ssh between ds-proto-daq and darkside01. 

There are 3 problems connecting to the CERN network:
1) The long cable that was created does not work. I tried using it to connect ds-proto-daq to socket 0001/06 (i.e. the socket ds-proto-daq is normally connected to) and there is no link light - the cable is broken.
2) Socket 0001/07 does not seem to be operational. I tried connecting ds-proto-daq to socket 0001/07 using the cable we know works, and there was no link light. Either CERN have disabled the connection intentionally, or it is broken.
3) When I used the good cable to connect to the good socket (0001/06), CERN claimed the machine was unregistered. Marco told me that the interfaces have MACs 9c:71:3a:22:d6:4[ab] (rather than 9c:71:3a:22:d5:6[23]), so presumably the wrong MAC addresses were registered with CERN.

So, we need to register the correct MACs, fix the cable, and either get 0001/07 working or connect to a different socket.

I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

The cable has been fixed. Socket 0001/007 is still not active.

Wrong MAC address fixed in CERN network setting, now the machine is visible from external network too:

From CERN network (even laptop) can connect by 

ssh -XY dsproto@ds-proto-daq2.cern.ch.  (same password as local account in ds-proto-daq)

From ds-proto-daq can also connect by:

ssh darkside01

That is to say that the two network card are now associated to two different names.. but the "real" name is still darkside01.

(will fix).

 enjoy

The chronobox isn't serving the human-interactive webpage that should be visible at https://m-darkside.web.cern.ch/chronobox/ - it responds with "File not found". Oddly, it is still responding to API calls at /read_var, /write_var etc.

So shifters can still change between laser/noise/physics runs using the "Run type" page at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type (which uses the API), but can't as easily monitor / sanity check the chronobox behaviour.

In the first instance, I think it would be useful to power-cycle the chronobox. If that doesn't work, I'm not sure how to proceed.