Ramp UP the fields, in 100V steps.

Nominal values:
1st ring: 4180 V
wire gate: 3780 V
cathode: 6180 V

Increased drift and extraction fields.

Current settings: wire gate 5100 V, 1st ring: 6100 V, cathode: 11100 V

P.S. Increased by in the morning by Yi.

The CAEN VME crate now gets its IP address from the ds-proto-daq DHCP server. To do this, I set the IP address of the crate to 0.0.0.0 (using the front panel interface) then reset the crate (not just a power cycle - pressing the reset button is required).

I am now able to ping the crate, but am NOT able to load the webpage it is supposed to serve (I just get a "connection refused"). I can't see any configuration settings that would disable the webpage or would cause it to be served on a non-standard port. I may have missed something though.

The analysis server from Roma (darkside01) is now connected to the local network. It is not yet connected to CERN.

There are 2 network ports on the rear of the machine:
* The port labelled "1":
 * MAC: 9c:71:3a:22:d5:62
 * Name: enp2s0f0
 * Configured for DHCP
 * Connected to local switch
 * Assigned IP 192.168.1.10 by ds-proto-daq
* The port labelled "2":
 * MAC: 9c:71:3a:22:d5:63
 * Name: enp2s0f1
 * Configured for DHCP
 * Should be connected to CERN network

I have set up password-less ssh between ds-proto-daq and darkside01. 

There are 3 problems connecting to the CERN network:
1) The long cable that was created does not work. I tried using it to connect ds-proto-daq to socket 0001/06 (i.e. the socket ds-proto-daq is normally connected to) and there is no link light - the cable is broken.
2) Socket 0001/07 does not seem to be operational. I tried connecting ds-proto-daq to socket 0001/07 using the cable we know works, and there was no link light. Either CERN have disabled the connection intentionally, or it is broken.
3) When I used the good cable to connect to the good socket (0001/06), CERN claimed the machine was unregistered. Marco told me that the interfaces have MACs 9c:71:3a:22:d6:4[ab] (rather than 9c:71:3a:22:d5:6[23]), so presumably the wrong MAC addresses were registered with CERN.

So, we need to register the correct MACs, fix the cable, and either get 0001/07 working or connect to a different socket.

I have failed to communicate with the steering module arduino from the DAQ PC.

The steering module has the most bizarre communication protocol of any device I have used:
* it has a static IP address (192.168.121.2)
* if something connects and sends a command, it doesn't send a response to that device - it sends responses to a different static IP address on a fixed port
* before a device connects, it's ethernet port is not open all the time - it turns it on for a brief period, and if something connects during that period, it remains on; if not, it turns off for a while. I'd estimate something like on for 500ms, off for 500ms.

We are currently using a macbook to connect to the steering module. The macbook is statically assigned IP address 192.168.121.1. When pinging the steering module, then first few pings fail with "no route to host". Eventually one of the pings coincides with the steering module having its ethernet on, and then the steering module keeps its ethernet on and the rest of the pings succeed.

We bought a new NIC for DAQ machine. I assigned it same static IP address (192.168.121.1), added a hole in the firewall (and even tried turning the firewall off for a bit), ensured the correct interface was used for connections to 192.168.121.0/24, but we never get a route to the steering module.

I have run out of things to try (but I may have missed something). My hypothesis is that linux isn't able to negotiate a connection quickly enough, but I may be wrong. It is my last evening at CERN, so I have returned the steering module to be connected to the macbook. Shifters will continue to have to manually change channels rather than having it automated by midas.

I hope that future versions of the steering module will have a more normal network protocol - open a server socket (static if must be, but preferably on DHCP), wait for connections, reply to commands on that same connection.

The cable has been fixed. Socket 0001/007 is still not active.

Wrong MAC address fixed in CERN network setting, now the machine is visible from external network too:

From CERN network (even laptop) can connect by 

ssh -XY dsproto@ds-proto-daq2.cern.ch.  (same password as local account in ds-proto-daq)

From ds-proto-daq can also connect by:

ssh darkside01

That is to say that the two network card are now associated to two different names.. but the "real" name is still darkside01.

(will fix).

 enjoy

The chronobox isn't serving the human-interactive webpage that should be visible at https://m-darkside.web.cern.ch/chronobox/ - it responds with "File not found". Oddly, it is still responding to API calls at /read_var, /write_var etc.

So shifters can still change between laser/noise/physics runs using the "Run type" page at https://m-darkside.web.cern.ch/?cmd=custom&page=Run%20type (which uses the API), but can't as easily monitor / sanity check the chronobox behaviour.

In the first instance, I think it would be useful to power-cycle the chronobox. If that doesn't work, I'm not sure how to proceed.

/home was getting very full on ds-proto-daq, so I moved the history files onto the /data disk. There is now a symlink from ~/online/history to /data/dsproto/history/.

I have changed `/Logger/Channels/0/Settings/Filename` to start with a dot, so files will be hidden until they are complete. 
The current value is `.run%05dsub%03d.mid`.

This will help Pablo with his script for transferring files to EOS.

frontend process still alive but slow, causing lot of rejected triggers.

stopped and restarted the feov1725 process to fix this

Error during data taking  1573416708 21:11:48.151 2019/11/10 [feov1725MTI00,INFO] V1725 PLL loss lock Board:0 (vmeAcq=0x0)

It stop saving event but run in process. Impossible to stop the run. I tried to reset feov1725MTl00. After that error: 

1573416857 21:14:17.792 2019/11/10 [feChronoEsper,INFO] Client 'feov1725MTI00' on database 'ODB' removed by db_cleanup called by cm_periodic_tasks because pid 9724 does not exist

Run stopped after some time. Error 21:21:19.121 2019/11/10 [mhttpd,ERROR] [midas.cxx:3951:cm_transition_call,ERROR] cannot connect to client "feov1725MTI00" on host localhost, port 40944, status 503   ╳

Trying to restart feov1725MTI00. Status initializing

It is not possible to initialize 

The connection to rootana is lost.

No data taking can be started.

IV of 25 SiPM put in ivdata_191119 directory

I have added a backup of the CERN proto-0 ELOG at TRIUMF, in case ds-proto-daq is unavailable in future.

The backup ELOG is at https://ladd00.triumf.ca/elog-ds/. There are other logbooks hosted in the same place (requiring a username/password), but the CERN backup logbooks (at the bottom of the list) are available to all. I have configured things so that the TRIUMF replicas are read-only. To add new entries one must still use the CERN logbooks.

The logbooks are rsynced every hour via cron.

1) Install and tweak elog:

[root@ds-proto-daq ~]# cat /etc/elogd.cfg 
[global]
port = 8084
MTP host = trmail.triumf.ca
Use Email Subject = {$logbook} $Subject
Remove on reply = Author
Quote on reply = 1
URL = https://ds-proto-daq/elog/

[DS Prototype]
Theme = default
Comment = ELOG for DS Prototype MIDAS DAQ
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem, Problem Fixed, Configuration, Other
Options Category = General, Hardware, Digitizer, Trigger, MIDAS, Software, Other
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type
Email all = lindner@triumf.ca

[root@ds-proto-daq ~]# systemctl start elogd.service
[root@ds-proto-daq ~]# systemctl is-active elogd.service
active
[root@ds-proto-daq ~]# systemctl enable elogd.service

2) Tweak apache and restart 

[root@ds-proto-daq ~]# grep elog /etc/httpd/conf.d/ssl-ds-proto-daq.conf 
ProxyPass /elog/ http://localhost:8084/ retry=1
[root@ds-proto-daq ~]# systemctl restart httpd

3) change MIDAS to use elog

[dsproto@ds-proto-daq bin]$ odbedit 
[local:dsproto:R]/>cd Elog/
[local:dsproto:R]/Elog>create STRING URL
String length [32]: 256
[local:dsproto:R]/Elog>set URL https://ds-proto-daq.triumf.ca/elog/DS+Prototype/

1) Install and tweak elog:

[root@ds-proto-daq ~]# cat /etc/elogd.cfg 
[global]
port = 8084
MTP host = trmail.triumf.ca
Use Email Subject = {$logbook} $Subject
Remove on reply = Author
Quote on reply = 1
URL = https://ds-proto-daq/elog/

[DS Prototype]
Theme = default
Comment = ELOG for DS Prototype MIDAS DAQ
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem, Problem Fixed, Configuration, Other
Options Category = General, Hardware, Digitizer, Trigger, MIDAS, Software, Other
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type
Email all = lindner@triumf.ca

[root@ds-proto-daq ~]# systemctl start elogd.service
[root@ds-proto-daq ~]# systemctl is-active elogd.service
active
[root@ds-proto-daq ~]# systemctl enable elogd.service

2) Tweak apache and restart 

[root@ds-proto-daq ~]# grep elog /etc/httpd/conf.d/ssl-ds-proto-daq.conf 
ProxyPass /elog/ http://localhost:8084/ retry=1
[root@ds-proto-daq ~]# systemctl restart httpd

3) change MIDAS to use elog

[dsproto@ds-proto-daq bin]$ odbedit 
[local:dsproto:R]/>cd Elog/
[local:dsproto:R]/Elog>create STRING URL
String length [32]: 256
[local:dsproto:R]/Elog>set URL https://ds-proto-daq.triumf.ca/elog/DS+Prototype/

I added javascript webdisplay of the V1725 waveforms to the Darkside setup.  You can see the waveform display here:

https://ds-proto-daq.triumf.ca/CS/webdisplay

You can also see the webdisplay that comes built into the JSROOT webserver here:

https://ds-proto-daq.triumf.ca/rootana/

For Pierre: I went through and reset the baselines for all the channels.

First check the maximum trigger rate and maximum data rate for different sample lengths (for each channels):

sample length      Maximum rate   MB/s    CPU % (per thread)
16us               0.44kHz        231     20
6.4us              1.04kHz        214     20  
3.2us              1.95kHz        198     22   
1.6us              3.12kHz        160     26   
0.8us              4.88kHz        127     30   (what are these threads doing?)

Look at the code more.  See that there is a maximum size for the block transfer of 10kB.  Increase this to 130kB
(which is the maximum amount that this board can make per event).  Now find

sample length     Maximum rate       MB/s     CPU % (per thread)
16us              0.68kHz            346      7
6.4us             1.46kHz            300      10 
1.6us             3.6kHz             190      25

Good.  So for long samples we are actually slightly above the maximum transfer rate of 85MB/s*4 = 340MB/s

Tried writing out the data to disk at the maximum 345MB/s rate; the DAQ can't keep up.  Maximum rate was more
like 270MB/s.
I think the mlogger was actually fine.  But I think the write-to-disk speed of the harddrive could not keep up.
 So I think we are limited 
by hardware in that case.  We would need a large raid array to be able to write faster.

Pierre and I got the CERN proxy setup for the Darkside prototype.

Using your CERN single-sign-on identity, you should be able to login to this page

https://m-darkside.web.cern.ch/

and see our normal MIDAS webpage.

The CERN server is proxying the port 80 on ds-proto-daq.  You can also see all the other services through the
same page:

elog:
https://m-darkside.web.cern.ch/elog/DS+Prototype/

chronobox:
https://m-darkside.web.cern.ch/chronobox/

js-root:
https://m-darkside.web.cern.ch/rootana/


_________________________________

Technical details

1) We followed these instructions for creating a SSO-proxy:

https://cern.service-now.com/service-portal/article.do?n=KB0005442

We pointed the proxy to port 80 on ds-proto-daq

2) On ds-proto-daq, we needed to poke a hole through the firewall for port 80:


firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="188.184.28.139/32" port
protocol="tcp" port="80" accept"
firewall-cmd --reload

[root@ds-proto-daq ~]# firewall-cmd --list-all
public (active)
...
	rule family="ipv4" source address="188.184.28.139/32" port port="80" protocol="tcp" accept

This firewall rule is pointing to some particular IP that seems to be the proxy side of the server:

[root@ds-proto-daq ~]# host 188.184.28.139
139.28.184.188.in-addr.arpa domain name pointer oostandardprod-7b34bdf1f3.cern.ch.

It is not clear if this particular IP will be stable in long term.

3) We needed to modify mhttpd so it would serve content to hosts other than localhost.  So changed mhttpd
command from 

mhttpd -a localhost -D

to 

mhttpd -D