run0034 stopped at 14:11

Leakage currents are increasing and once again we are not fully depleting the pad detectors at 150V.
Noticed that the strip vs energy histogram began with a well-defined band across all strips and the n-side for a given pad, but by the time we stopped event energies were spread up to a max value across all 
strips and n-side. Also saw *incredibly* long rise times and a rollover of the pre-amp pulse rather than a peak and decay.
  -> This implies loss of bias to the detector.

We increased the bias voltage to 300V on Bus 0, Mod 1, Ch 2 (6th SiPad on the inside pocket) and saw the leakage current rise in tandem with the voltage (up to 15uA) such that with a 20 MOhm bias resistor, 
the induced voltage drop is almost equal to the applied voltage.

In short, up 300V we cannot apply high enough bias to overcome the equivalent increase in leakage current.

Explanation?
This could be an effect of ions on eccentric orbits hitting something and showering the detector with electrons, leading to surface charging and the increased leakage currents.
Either we are receiving huge numbers of ions in the detector but not seeing them, or something else like above is going on.

Surface charging could/would dissipate over time and potentially bring the detectors back into a useful range. Radiation damage likely would not.

15:14 Started run #35: inside pocket at 20 mm.
Trigger mask changed. Gated only in ch8 in MSI-8 #2 (one side of photodiode)
No events hitting the detector yet.

15:37: run stopped.

No sign of any events.

After some Sergey-magic the outside pocket moves again. We slowly moved the outside detector in, we are clipping the beam at -20mm. Moved the detector to -40mm and take some data as of 17:48 (run0036.lmd)

The leakage currents of the SiPad detector jump back and forth 2 microamps (btw 25 and 4.5 microamps), the DSSD seems stable, the crystal too

https://elog.gsi.de/esr/E0203_E0018/207

At 10:50am today, we connected a remote Picoscope to the first 4 n-sides, one end of both resistive chains, and both crystal channels to directly measure the output of the preamps without the ADC clipping at 2V.

In the parked position, we see a ~1 Hz rate on the detector with small signals in the crystal and very small signals (potentially just electronics) on the n-sides. 

At 17:15, we moved the detector into the beam (-100 mm from the beam). With each injection, the preamps output massive signals, roughly 10 V each for a 1.8 V max input. Attached is a characteristic screenshot, with more in the ZIP file.

The flash appears to be correlated with the injection timing. The signal seems to reach a cap at 10 V, which I believe from talking to Roman is the HV bias reaching its leakage current cap. After that, the RC discharge comes back into 
range. Roman suggested that roughly, this RC discharge is representative of the 
signal height. The 1/e constant is around 35 us, so a saturation time of 560 us implies an signal peak of e^16 higher, ie 8.9e7 V or 1.8e7 GeV or roughly 1e7 particles. There may be some other saturation effects that mean we do not get 
quite this many particles, especially given only 10^5 particles are stored, but 
either way, our detectors were seeing at least 10^5 particles ever injection.

Thus, we can clearly conclude that the observed leakage currents come from radiation damage and that our detectors are fried. This is definitely a hard learnt lesson that we cannot have our detectors in for the isochronous mode...
Comment from Iris: In the isochronous mode via the TE line when we inject too many ions in the ring. To be tested with the IM from the FRS when we can tailor the beam injected into the ESR much better (more slits, better cleaning).

At 9:53 we moved the outside detector in to take a short data run. We went -110 mm compared to a fully retracted position -126 mm. Data is collected in run 38 starting at 
9:56:25. Data acquisition was stopped and then moved the detector back at 10:16.

We start recording data in run 39 and the outside detector is moved to -100 mm at 11:36. At 11:59 the detector was moved back to the fully retracted position, -126 mm, and the 
data acquisition was stopped.

Final injection stored in ring around 10:00 and will be stored for ~1h.
Both detectors are in the parked position but I started a run just as "background"/to see if we get anything.
We have previously seen some events while in this position so who know what might show up.
run0040

11:01 All voltages turned off. Remote control disabled.

BGO detector assembly originally on flange A.
GAGG  "                        "    flange B.

BGO detector, SiPad#3:
+ Showed linear bias response during experiment and was thought to have a short.
  Upon conducting another bias curve post-run, the same behaviour was observed.
  However, when I checked the detector visually, all bond wires seem to be in tact and when I tested the diode response with the multimeter, it behaves as a diode should.
  ^ (potential difference of ~0.4 - 0.8 V when forward-biased, open line--OL--when reverse-biased) ^
  We swapped it onto the other flange/arm to see if the flange is the problem but now cannot apply bias at all to this detector.

BGO detector, crystal:
+ During initial bias curve measurement, the crystal PD would not bias.


+ In trying to bias the BGO detector on the other flange (B) to see if the problem follows the detector or the flange, we were not unable to bias either Mod0Ch2 or Mod1Ch0 (SiPad#3 and SiPad#5).
  The channels were both inconsistently problematic for the GAGG originally on this flange this morning while Maeve did bias curves.
  Perhaps there is an intermittent poor connection on this flange or with the Samtec ribbon cables/end cap PCB/etc.

+ There seems to also be some interdependency between the leakage current on BGO detector SiPad#3 and SiPad#5 - they share a D-Sub.
  The leakage currents of each detector and equal and opposite. We changed the D-Sub to see if that was the problem, but the problem persisted.
  When both are biased simultaneously, leakage currents on each goes to zero. A short between the two might then have them at the same potential in this scenario meaning no applied voltage and no leakage 
  current.
  Visual inspection of these two pads doesn't reveal any obvious problems and both behave like a diode in multimeter tests.

+ We also swapped out the Samtec ribbon cables to investigate any issues with those. When the new spare cable was connected to X2, the crystal PD biased just fine
  This may imply a lose/broken connection on that cable that prevented the crystal being biased. However, this had no impact on SiPad#3 and #5.


GAGG detector assembly
+ Cannot bias SiPad#4 on either flange. This detector also shows open line in both directions when doing a diode test with a multimeter. (MSPAD 16A)
  Upon further investigation, the corner of the chip near Ch1 is no longer bonded to the PCB and lifting. This will likely have broken the bond wire on the rear.
  Will be boxed and left for Jan to take to the detector workshop for further investigation.

+ Crystal couldn't be biased originally on flange B, but biased "normally" on flange A. Potentially indicated some connection issue with either flange B or the Samtec cables.
  Considering the previously noted test replacing the Samtec cable X2 in flange B, this suggests the problem may lie with that cable.

All detector components are now in safe storage in Jan's container.

One SiPad (MSPAD 16A) shows lifting of the chip from the PCB in one corner. This has been boxed, labelled and left visible in the lab and should be taken to 
the detector workshop for repair.

Attached is the final detector health spreadsheet as of today.
Also available at: https://docs.google.com/spreadsheets/d/1Gt1tFHwn4XFrJ5_-nQvTi635UD0aHgN6gCdXDTHQdkI/edit?gid=299353113#gid=299353113

In summary, all detectors in the BGO assembly show signs of significant radiation damage and increased leakage currents. Most of these detectors do not reach 
plateau until >200V, but may still be usable in future at bias voltages of ~250V.

The detectors in the GAGG assembly also show signs of radiation damage, but less so due to this pocket being stuck in the parked position for some time. The 
detectors are still very usable.

Some bias problems were observed during and after the experiment, but all detectors (except for MSPAD 16A) show an expected diode response, so this may be 
indicative of shorts/etc elsewhere in the Samtec cables/flange/end-cap PCBs/etc. Future tests should be done to identify the source of these irregularities.

We are missing one MHV4 module. I will check at TRIUMF but I think it *should* be at GSI.

Data are backed up on HDDs to go to TRIUMF, as well as MBS configuration, the Go4 build, and the screenshots taken while we were using the remote scope.