********************************************
** Setup of DAQ x86l-131 (outside pocket) **
********************************************

Used FEBEX GUI to set thresholds for all channels.
Using ~65 mV pulser input, find 3000/0xbb8 (decimal/hex) to be a robust level where pulser is always seen and the trigger level is well above noise.
We expect much higher real detector signals so this should be an appropriate trigger.

Set channel trigger mask to:
0x0000, 0x,0000, 0x0000, 0xcf57

This will trigger readout on n-sides 1-3, 5, 7, the DSSD and the crystal PD.
(pad in slot #7 = pad 6)
Triggers are turned off for all p-strips, unconnected channels and the n-side of pad 4 since that cannot be properly biased.

See some high frequency noise on all channels so extended integration time of hit finder to integrate this out.
Hit finder params set to:
TRIG_SUM_A 20
TRIG_GAP   10
TRIG_SUM_B 21

Trace length (FEB_TRIG_LEN) is set at 3000 and the trigger delay at (FEB_TRIG_DELAY) 500

Sparsifying turned on so if the hit finder doesn't find at least one hit on a given channel, the trace for that channel will not be saved.

*******************************************
** Setup of DAQ x86l-124 (inside pocket) **
*******************************************

The inner pocket n-sides seem to see more noise and the trigger rate is not as stable -> additional component due to fluctuating noise.
Set up similarly to x86l-131 in terms of hit finder parameters, thresholds etc.

Trigger mask is set to:
0x0000, 0x,0000, 0x0000, 0xcf5b

This will trigger readout on n-sides 1,2,4,5,7, the DSSD and the crystal PD.
Triggers are turned off for all p-strips, unconnected channels and the n-side of pad 3 since that cannot be properly biased.

Data files written to /mnt/raw.data/e0203_e0018.

From MBS computers:
cd exp_mnt/data.ESR/e0203_e0018/run***.lmd

Everything called run*** is from timesorter.
Everything with a number (either 124 or 131) before run is from one of the independent DAQs.

Sample file taken with pulser to inside DAQs (124) and written to 124_run0001.lmd.

The MHV modules are set up for remote control. This can be accessed via the mesycontrol software running on x86l-128.

A configuration file has been save to: esr/usr/litv-exp/DAQ-control/mesycontrol/pleiades_2025_v2.xml

DAQ OPERATION

The files of each MBS node or the timesorter are written to the ESR NAS by the RFIO server.
The server is running on the virtual machine atprfio.gsi.de. 
The NAS is mounted there and we write data to the subfolder: /mnt/raw.data/e0203_e0018/

This folder can be mounted to any x86l-XXX pc using litv-exp user with the shell command
> mountESR
The NAS is then mounted at /esr/usr/litv-exp/exp_mnt/data.ESR 
Which is where you will find the folder e0203_e0018 and the data.

We started inserting the inside and outside detector, see
https://elog.gsi.de/esr/E0203_E0018/46

Outside detector (131): started beam scraping at -40mm, at -35mm no beam left
Inside detector (124): drove it in up to 2mm, no beam scraping seen.

MBS files: run0010.lmd (outside detector, ~14:25)
run0011.lmd (16:50): first inside detector, then also testing outside detector again
run0012.lmd: running now with both detectors in:

inner detector: BGO
device: GE01DD2IG 
position: +20mm
scraping beam: not possible

outer detector: GAGG
device: GE01DD2AG
position: -55mm
scraping beam at: -45mm

The experiment has been running stably during the afternoon.
DAQ rates have remained constant and the frequency of hits in the Go4 has remained 1 hit every ~few mins.

However, leakage currents have increased significantly in most detectors, by up to a factor of 20x times in some.

I find it hard to see how radiation damage could cause this, but using the rule of thumb of 2x current for 8C rise in temperature, this would imply a >35C rise. This also doesn't seem realistic.

I don't currently have any other ideas.

Attached is the current leakage current plot - it is clear they have been increasing linearly since this afternoon.

Due to the continued rise in leakage currents, I have stopped the run, moved the detectors completely back and turned off the 
bias to all detectors.

-X86L-128:transport  :-I- remote output file /mnt/raw.data/e0203_e0018/run0015.lmd closed
-X86L-128:transport  :Rfio server atprfio: closed file after    999.981 MB, open next:
-X86L-128:transport  :/mnt/raw.data/e0203_e0018/run0015.lmd
-X86L-128:transport  :/mnt/raw.data/e0203_e0018/run0016.lmd
-X86L-128:to         :error (no timeout) getting event from node X86L-124,
-X86L-128:to         :exiting..
close file
-X86L-128:transport  :closed file after    338.199 MB written:
-X86L-128:transport  :-I- remote output file /mnt/raw.data/e0203_e0018/run0016.lmd closed
-X86L-128:transport  :/mnt/raw.data/e0203_e0018/run0016.lmd

The settings of the MBS daq during the first data taking with fragments around 194Pb (W.Korten experiment) was not entirely correct:

1. both FEBEX nodes had the very same sub-sytem identifier 0x200. The timesorter did merge the two data streams into the lmd files. Now the events cannot be easily separated/identified, 
because both have the same WR header. The only option to identify which FEBEX was the source for an event might be to look for the actual channels receiving hits, because they are slightly 
different for both arrays/DAQs. This might only work for "good" events though...

2. The trigger signals transmitted to the scaler DAQs (VME & NTCAP) had a different input for both DAQs. The GAGG trigger (outside) was using the FEBEX self-trigger as input only, while 
the BGO trigger (inside) was using the FEBEX self-trigger OR the 10Hz sync trigger as input.

The runs with run number lower than 0016 are affected by the above points.

Now the setup has been changed, for run numbers 0017 and higher the following should be valid:

1. WR identifiers
The BGO DAQ on x86l-124 (inside) does use WR sub-system identifier 0x100
The GAGG DAQ on x86l-131 (outside) does use WR sub-system identifier 0x200
The VME DAQ on x86l-204 does use 0x400

2. transmitted triggers
Both DAQs now transmit only the FEBEX self-trigger to the scaler DAQs.

3. Gate Generator for PEXARIA input
We now use a Gate Generator to provide 200ns TTL signals for the PEXARIA, with the goal of avoiding double trigger/pulses (a known issue of the old EXPLODERs we are using)

4. SPARSIFYING
We turned off SPARSIFYING for both FEBEX DAQs. So 
in f_user.c 
#define DATA_FILT_CONTROL_DAT 0x80
was changed t0
#define DATA_FILT_CONTROL_DAT 0x82

During an access break to the ESR, I made a few tests with the bias scheme to see if we could better understand some of the weird quirks.

Two detectors showed likely shorts during initial setup - these detectors were still cabled to the bias network though. 
If there is a short to ground, we wondered if that could then impact the other detectors on the same MHV4 module.
These detectors were uncabled at the pre-amp end.
No change was noticed we bias was applied again.

On the outside pocket, I also noticed that when one detector is turned off, the leakage currents of the others remain the same amplitude but flip polarity as the other biases down.
I disconnected all detectors from that MHV module and connected them one-by-one and this effect was immediately seen with two detectors connected.
I double-checked the cabling and cannot see anything wrong there.

Please enter the leakage currents from the mesycontrol every ~1h into this Google document:

https://docs.google.com/spreadsheets/d/1Gt1tFHwn4XFrJ5_-nQvTi635UD0aHgN6gCdXDTHQdkI/edit?gid=299353113#gid=299353113

The inner detector was moved into position.
The device control for the outer detector is not responding and I am not an expert/don't know how to make it work.

See ELog: https://elog.gsi.de/esr/E0203_E0018/126

Inner detector (x86l-124 BGO)
Device: GE01DD2IG
Position: +20 mm
Scraping beam: not possible

At 19:00 the timesorter slowed down and stopped responding. at 19:56 the the inner detector was retracted (pulled back further from the beam)

Outer detector (x86l-131 GAGG)
Device: GE01DD2AG
Position: -127 mm
Scraping beam: n/a

This could be an interesting test of the detector leakage current response to being in place (i.e. only the inner detector is in place, do all leakage currents increase or 
only that detector?)

Started taking data with file run0017.lmd @ 17:35.

Will continue to run for a few hours this evening and closely monitor leakage currents.

Go4 is running on x86l-184 connected to stream server.

186Hf now stored stably in the ring.

Will move both detectors in to position.

run0021 taking data during moving process and start a new file once they're in place.

Inner PLEIADES detector moved into position at +2mm.
Again, the outside pocket cannot be moved.

run0022 started 16:26:50.

Run stopped at 16:47:20.

Going to attempt a bias curve on one detector. With the present leakage currents, the voltage drop induced is close to the applied bias voltage.

After talking to T. Davinson this afternoon a few things became clearer.

Because of the huge increase in leakage currents, and the 20 MOhm bias resistors uses across most of the pre-amp cards, it is likely that the induced voltage drop has been great enough to not fully deplete 
the detectors.

Maeve and I carries out extended bias curves for most of the detectors and found the plateaus to be reached at much higher voltages than expected.
See detector health spreadsheet and figures in attachment #1.

Bias voltages have now been increased on most detectors to put them in the plateau region.
Some also seem to be shorting to ground.

SiPad #1 on the inside detector was found to like have a short so is now unbiased and its n-side has been removed from the trigger mask.
The trigger mask for FEBEX#3 on x86l-124 is now 0xcf5a (n-side 2, 4, 5, 7, DSSD, crystal PD).

In both detectors:
    The FPGA energy filter SUM_A, GAP, and SUM_B were changed to 100, 200, 101, respectively to better account for the long rise times of our detectors.

    Sparsifying was also turned off for all channels. (How does this interact with data filter control 0x82?)

Inside pocket moved in to 2mm

run0024 started @ 12:27:50

Given that I hope we are now moving towards stable operation, here is the current status of detectors for run0024 on...

Inside pocket (x86l-124, 0x200)
SiPad #1: Suspected short. Biased but linear IV response. Very long charge collection time. n-side removed from tigger mask.
SiPad #2: Bias at +150V. High leakage current, but operating normally.
SiPad #3: Suspected short. Bias disconnected at MSI-8. Removed from trigger mask.
SiPad #4: Bias at +150V. High leakage current, but operating normally.
SiPad #5: Bias at +150V. High leakage current, but operating normally.
SiPad #6: Bias at +150V. High leakage current, but operating normally.
DSSD    : Bias at -100V. High leakage current, but operating normally. Traces in ch2 often look quite strange.
Crystal : Bias at -140V. High leakage current, but operating normally.

Outside pocket (x68l-131, 0x100)
 - Pocket actuator not responsive in device control. Currently in parked position.
SiPad #1: Bias at +100V. High leakage current, but operating normally.
SiPad #2: Bias at +100V. High leakage current, but operating normally.
SiPad #3: Bias at +100V. High leakage current, but operating normally.
SiPad #4: Suspected short. Bias disconnected at MSI-8. Removed from trigger mask.
SiPad #5: Bias at +100V. High leakage current, but operating normally.
SiPad #6: Bias at +100V. High leakage current, but operating normally.
DSSD    : Bias at -100V. High leakage current, but operating normally. Traces in ch2 often look quite strange.
Crystal : Bias at -140V. High leakage current, but operating normally. Leakage current steadily increasing.

There is something we still don't understand on the outside pocket because when one detector is biased off, the other leakage currents flip polarity...

Bias curves were completed for most detectors yesterday evening.

General news is that, due to the massively increased leakage currents, we were likely not fully depleting the detectors up until yesterday.
Generally, all detectors require much higher voltages to reach plateau than was expected.

For the inside pocket, this is +150V for the SiPads, -100V for the DSSD and -140V for the crystal PD.
The outside pocket may have different bias resistors in the MSI-8 (need to check at end of exp!) so they hit plateau at lower bias, but still higher than expected: +100V for SiPads, -100V for DSSD, -140V for 
crystal PD.

Bias curves are attached and can be found in the detector health spreadsheet at https://docs.google.com/spreadsheets/d/1Gt1tFHwn4XFrJ5_-nQvTi635UD0aHgN6gCdXDTHQdkI/edit?usp=sharing

Attachment 1: Bus 0, Mod 0. Inside pocket. Ch2 is uncabled at the MSI-8 so a bias curve couldn't be done for this detector, but was previously identified as likely having a short in Beavertail Elog 21.
Attachment 2: Bus 0, Mod 1. Inside pocket.
Attachment 3: Bus 1, Mod 0. Outside pocket. Ch3 is uncabled at the MSI-8 so a bias curve couldn't be done for this detector, but was previously identified as likely having a short in Beavertail Elog 21.
Attachment 4: Bus 1, Mod 1. Outside pocket. Ch3 is the crystal PD. Leakage current on this detector is slowly creeping up despite not being in the beam making a bias curve hard to do.

The leakage currents are 6-7 micro-amps for the Si pads at the moment. The max current for the MHV-4 is 20 microamps PER CHANNEL, so we should be fine.
https://www.mesytec.com/products/nuclear-physics/MHV-4.html