Open Heart Surgery

I’m back in the US now, catching up on homework, and everything else I’ve missed. My last month at Pole was extremely busy, so I didn’t have time to write. I won’t be able to write a lot about what happened either, as I was pretty sleep-deprived, and the memories are already fading. That being said, I’ve got a few pictures to help jog my memory of some of the work we did. Here’s one example.

Something’s Wrong

This story starts with our attempt at what is known as “first light” in the astronomy community. First light refers to the first time a new telescope (or camera) sees an astrophysical object. First light is a pretty big deal because it’s the first demonstration that all the parts of the telescope (optics, detectors, drives, etc.) are more or less working. Usually, all of those parts have been tested individually, but you never really know if they’ll all work together. Unfortunately, the first light for SPT-3G didn’t quite go as planned.

Let me set the scene up a little bit. We’d spent almost a week getting everything installed in the telescope. Everyone was excited to actually see something with the telescope. Furthermore, we’d asked for some IT maintenance to be delayed so it didn’t mess up any of our communications that day, so the whole station new we were getting close. By the time we got around to starting observations, it was nearly midnight. The entire SPT crew (25 people) and a few folks from the station were trying to crowd around about 5 different laptop screens to watch things happen. After a false start or two (at least one of which was my fault), we finally got our observation going.

When we observe astrophysical sources, we start with the telescope looking below the source, and scan back and forth, slowly stepping up each time. This ensures that every detector has an opportunity to see the source (which is important for calibrating their response). It also means there’s some delay between starting an observation and when we should start to see the source. In fact, due to some imperfections in our pointing model, we don’t know quite when we’ll actually start to point detectors at anything visible.

To make a long, agonizing story short, we spent over an hour sitting around, wondering when we’d see anything, before concluding we wouldn’t be able to. That’s when we found out our superconducting detectors were heating up every time the telescope accelerated, which caused them to become normal metal (and very poor detectors). The exact cause is still unknown, but most of us have come to the conclusion that the acceleration was causing vibrations inside our camera that eventually dissipated as heat.

The Fix

After looking carefully at the telescope movements and the detector temperatures, we concluded that the telescope was accelerating much faster than it should have. Unfortunately, the maximum acceleration is a hard parameter to control. It’s set in hardware, deep in the heart of the telescope. So, we had no choice but to open up the telescope’s heart, and change the hardware.

the heart
The telescope drive cabinet.

The hardware in question is a pair of RC filters that control the acceleration of the two axes (azimuth and elevation). To decrease the acceleration, we had to change the resistors and capacitors inside one of the telescope drive cabinets. These particular components are on a circuit board mounted inside a computer which is, in turn, embedded in the drive cabinet.

The circuit board in question.

You’ll notice that board has a lot of unlabeled white wires coming out of it, which made detaching the board entirely a daunting prospect. Instead, we rolled a cart over to the drive cabinet and did the modification with only a few cables disconnected (we did turn off the power first, though).

Removing the board from its mount. Picture by Daniel Michalik

From there, we were able to do all the necessary soldering. It was a fairly complicated operation, since the cables were barely long enough for use to put the board on the cart. The whole operation required two people (one to operate the soldering iron, and one to manage the components we were changing.

resistor removal
Removing the resistor. Picture by Daniel Michalik
The end result. The large red component is a capacitor we added. Picture by Daniel Michalik

In the end, the operation was a great success. We reduced the acceleration on both axes to the point that any heating we get is acceptable. The next night we were able to get a proper first light, and had a grand old celebration afterwards.

Thanks to one of our winterovers, Daniel Michalik, for allowing me to use his pictures of the repairs. I had my hands full of soldering iron at the time!

A Slice of Work

Happy cable-wrap fixers. Happy because the job is almost done.

I haven’t written very much about the kind of work I actually do down here, especially this year. My posts have been a lot more focused on the photography, because I’ve been taking that more seriously than I have in the past.
My typical day actually turns out to be pretty boring. I typically have some minor IT issues to deal with (software updates, or a hung process), or a minor maintenance task.
I usually have a lot of free time. I’ve been using most of it to work on analyzing SPTpol data from the last few years. However, we did have one major repair this year, and I’m going to try to describe the process we went through. This is largely adapted from an internal memo I wrote for future SPT winterovers, who might have to deal with something similar.

The Symptoms

The primary mirror with snow covering the bottom third

The first thing we noticed was that the primary mirror of the telescope had a lot of snow buildup. Fortunately, a little bit of snow on the mirror is not a particular bother for us. Since the wavelengths we look at are so long, the microwaves mostly pass through the snow. However, it does cause some diffraction and reflection, which essentially blurs the image we see. Some snow build up is normal during bad weather, but this amount of snow is problematic (and very unusual). We have a large heating system that keeps the mirror just a few degrees above the ambient temperature. Most of the time, this keeps the mirror completely clear.

When we looked a little further, we found that the breaker for the heaters had tripped. This meant that something had drawn too much current, and it was shut off to prevent damage (like starting a fire). We reset the breaker, and the heaters came back on. However, we found them tripped again a few days later. This pattern continued for a while, before we found the problem.

At the same time, we were having some problems with the telescope’s drive system. SPT uses a regenerative braking system (the same basic idea that is used in many hybrid cars). It turns the motion of the telescope back into power when we want to slow down. We noticed that the regeneration subsystem was occasionally failing. It was a strange failure, though. Some event would trigger the failure, and the control system would note it. However, everything was working normally afterwards.

In hindsight it seems obvious that the two problems were related, but real life is never that simple. In the interest of brevity, I’m going to skip over a couple weeks of debugging and attempted fixes. It turns out that the cables powering the heating system had been cut, and they were periodically shorting out against the telescope structure. Every time the cables shorted, they caused a large spike of current before the breaker cut it off. This was enough to make the regeneration circuitry think it was broken.

To understand how the cables were cut, you need to know how the telescope is connected to stationary utilities on the ground. The problem is not a simple one, as we have a large variety of things that need to be connected between the moving part of the telescope and the building. Besides electricity, we need fiber optics to transfer data back and forth, as well as large lines for compressed helium gas. Altogether, they create a bundle of cables that is 10-12 inches in diameter. That bundle is wrapped in a spiral at the base of the telescope. The bottom is attached to the building, and the top moves with the telescope. Each layer in the spiral moves a little bit more than the layer below it.

The heater power cables

In principle, this is a very nice system. However, our implementation is somewhat flawed. It requires constant maintenance to keep all of the cable supports in place, and we were not watching it closely enough. After an attempt to improve the arrangement of cables this summer, they had all slipped downwards in the spiral wrap. The topmost section got so tight that it made contact with the central pillar that runs through the spiral. Over a period of several weeks, the cables that provided power to the mirror heaters were scraped back and forth against the pillar. Eventually, they were cut entirely.

The Fix

The patched cables sitting in a junction box

Fixing an electrical issue of this magnitude is well beyond my own experience. Fortunately, we have people who know how to do these things. The station employs an electrician year-round to fix problems like this, but we ended up getting help from one of the IceCube winterovers who works on power distribution systems in the real world. We also got a lot of help from the winter machinist, both in doing the repairs, as well as building tools and new pieces.

It turns out that patching a cable like that is not very hard. However, we could not just leave the patch in the middle of the cable wrap, we had to move the broken part of the cables to the top of the wrap, where we could mount a junction box. To get enough slack in the cables we had to remove them entirely, and then replace them. In order to get them back in, we had to disassemble a lot of the brackets that hold the cable wrap together. When we tried to put the whole thing back together, we found that many of the brackets had been twisted by the weight of the cables, and would not go back together. This was after two days of non-stop work (most of us were eating ramen at the telescope to avoid spending the time walking back to the station).

One of the brackets, after our fixes.

To the left, you can see a picture of one of the brackets with our method of fixing them. Originally, they each had two threaded rods going through their tops (you can still see the holes for the rods). To get the broken cables out, we removed the threaded rods. In several of the brackets, we found they would not come out at all, so we cut them.

After removing those rods, the brackets twisted so that the holes no longer lined up. We were able to force some of them back in by bending the rods. However, that meant that different layers of the cable wrap were unevenly spaced. In certain positions, the brackets from a lower layer, would scrape against the cables in another layer. Over time, that would have led to exactly the same problem we were trying to fix. Our solution was to make pairs of aluminum bars that could be squeezed together by screws. These are much stronger, and serve the same purpose.

Unfortunately, that did not entirely solve the layer-spacing problem. The spacing is partly controlled by a set of brackets that are mounted to bearings on the central column. These brackets hold their portion of the cable wrap at a fixed elevation. However, the sections between the fixed brackets can sag, allowing them to contact the next layer down. The only additional control we have over the spacing is the angle at which the cable wrap enters and leaves each fixed bracket. That angle is determined by a long flat spring that forms the core of the wrap. The cables are all mounted to the spring, and they follow it quite tightly. By adjusting the angle of the spring at each bracket, we were able to get to the point where we could spin the telescope over an entire circle without any layers touching.

The new bracket (left) side-by-side with the bracket it replaces

This last part has been the problem that keeps on giving. In order to solve this problem for good, we designed a new style of bracket that has almost not vertical head room. Our machinist made a small set of them, which we have been testing for several months. Just this week, the design was approved for manufacture. We will be getting several dozen this summer to install on the telescope.

As I said at the beginning, most of the time I don’t have a lot of work to do. But I am here for instances like this: when something goes catastrophically wrong. This fix took us four days, which is the longest unscheduled down time SPTpol has ever had. It was quite frustrating. For those four days, there were three or four of us spending 8-10 hours a day inside a cylinder that is roughly 10 feet in diameter. No matter who you are working with, that is bound to create some friction. I consider it a testament to everyone’s patience that we never got into a shouting match.

I forgot this one…


A time lapse showing the docking process from inside.

I made a time lapse of Cynthia and me docking the telescope. This allows us to get up into the receiver cabin without going outside. From there, we can access all the electronics responsible for running the detectors, as well as the cryostats.

FYI, this is an absurdly large file (66 MB), so you may not want to look at it on a slow connection. Also, you may have to click on the picture to get the time lapse part.