Open Heart Surgery

I’m back in the US now, catching up on homework, and everything else I’ve missed. My last month at Pole was extremely busy, so I didn’t have time to write. I won’t be able to write a lot about what happened either, as I was pretty sleep-deprived, and the memories are already fading. That being said, I’ve got a few pictures to help jog my memory of some of the work we did. Here’s one example.

Something’s Wrong

This story starts with our attempt at what is known as “first light” in the astronomy community. First light refers to the first time a new telescope (or camera) sees an astrophysical object. First light is a pretty big deal because it’s the first demonstration that all the parts of the telescope (optics, detectors, drives, etc.) are more or less working. Usually, all of those parts have been tested individually, but you never really know if they’ll all work together. Unfortunately, the first light for SPT-3G didn’t quite go as planned.

Let me set the scene up a little bit. We’d spent almost a week getting everything installed in the telescope. Everyone was excited to actually see something with the telescope. Furthermore, we’d asked for some IT maintenance to be delayed so it didn’t mess up any of our communications that day, so the whole station new we were getting close. By the time we got around to starting observations, it was nearly midnight. The entire SPT crew (25 people) and a few folks from the station were trying to crowd around about 5 different laptop screens to watch things happen. After a false start or two (at least one of which was my fault), we finally got our observation going.

When we observe astrophysical sources, we start with the telescope looking below the source, and scan back and forth, slowly stepping up each time. This ensures that every detector has an opportunity to see the source (which is important for calibrating their response). It also means there’s some delay between starting an observation and when we should start to see the source. In fact, due to some imperfections in our pointing model, we don’t know quite when we’ll actually start to point detectors at anything visible.

To make a long, agonizing story short, we spent over an hour sitting around, wondering when we’d see anything, before concluding we wouldn’t be able to. That’s when we found out our superconducting detectors were heating up every time the telescope accelerated, which caused them to become normal metal (and very poor detectors). The exact cause is still unknown, but most of us have come to the conclusion that the acceleration was causing vibrations inside our camera that eventually dissipated as heat.

The Fix

After looking carefully at the telescope movements and the detector temperatures, we concluded that the telescope was accelerating much faster than it should have. Unfortunately, the maximum acceleration is a hard parameter to control. It’s set in hardware, deep in the heart of the telescope. So, we had no choice but to open up the telescope’s heart, and change the hardware.

the heart
The telescope drive cabinet.

The hardware in question is a pair of RC filters that control the acceleration of the two axes (azimuth and elevation). To decrease the acceleration, we had to change the resistors and capacitors inside one of the telescope drive cabinets. These particular components are on a circuit board mounted inside a computer which is, in turn, embedded in the drive cabinet.

The circuit board in question.

You’ll notice that board has a lot of unlabeled white wires coming out of it, which made detaching the board entirely a daunting prospect. Instead, we rolled a cart over to the drive cabinet and did the modification with only a few cables disconnected (we did turn off the power first, though).

Removing the board from its mount. Picture by Daniel Michalik

From there, we were able to do all the necessary soldering. It was a fairly complicated operation, since the cables were barely long enough for use to put the board on the cart. The whole operation required two people (one to operate the soldering iron, and one to manage the components we were changing.

resistor removal
Removing the resistor. Picture by Daniel Michalik
The end result. The large red component is a capacitor we added. Picture by Daniel Michalik

In the end, the operation was a great success. We reduced the acceleration on both axes to the point that any heating we get is acceptable. The next night we were able to get a proper first light, and had a grand old celebration afterwards.

Thanks to one of our winterovers, Daniel Michalik, for allowing me to use his pictures of the repairs. I had my hands full of soldering iron at the time!

2 thoughts on “Open Heart Surgery

  1. All I can say is, omg, omg, omg. That was a scary operation, I’m glad it was a success.

    Did you have to draw straws on who held the cool handle of the soldering iron and who had to hold the components up to the 700F tip of the iron?

    (My soldering iron is better than yours)

    • We had pliers for the hot side of things. They are, of course, hidden behind the board (on the side where the components live).

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s