I’ve just arrived at the South Pole for the sixth time (as in 4 hours ago, so I’m pretty oxygen deprived. Please excuse any hypoxia-induced typos). The set of things in my life that have just become normal (I have a favorite breakfast on a continent I don’t even live on) is strange.
Anyhow, not much to say right now, but I wanted to document my trip down from Christchurch. We fly from Christchurch to McMurdo Station, just off the coast of Antarctica, and then onward to the South Pole. The flights are all on military aircraft. At the beginning and the end of the summer season, the runways in McMurdo are hard enough to land wheeled aircraft, so we get to fly in C-17s. The C-17 is a large, jet-engined cargo plane. It makes the flight in about 5 hours. At this time of the year, we have to use LC-130s, since they have skis. The LC-130 is a smaller, propeller driven plane. Similar to the small commuter planes that hop between nearby airports in the US. They take 8 hours of noise and discomfort to get to McMurdo. You may be able to guess which plane I prefer to fly.
It’s not unusual for flights to be delayed, or outright canceled. Predicting the weather is extremely difficult, and the planes are old. This time down was fairly smooth, as opposed to last year when I was “stuck” for 5 days in Christchurch. We were only an hour late departing for McMurdo, and the flight was reasonably smooth. Upon our arrival, we were informed that we’d be heading out for the flight to Pole at 15:45 the next day. However, we discovered at 9:15 that our departure time had been bumped up to 9:55. Of course, we all scrambled to leave, and track down anyone else who was supposed to go (not an easy feat without cell phones). Some time between 9:20 and 9:35, the departure was pushed back to 10:45. About an hour later, it was back to 15:45, so I went on a hike. 15:30 rolls around, and I check to make sure we’re still going, then go to pack up a few last minute items. 3 minutes later, we find out there’s a 2 hour delay. In my mind, this isn’t the worst thing in the world, since it means we’ll get dinner in McMurdo, rather than whatever we can shove in our pockets to eat on the plane. Finally, 17:37 rolls around, so I head out to meet the bus that will take us to the plane. In the 4 minutes it takes me to walk from my room to the check-in, the flight has been canceled entirely.
The next day, we did go out to the plane on time, get in and take off, but we had to turn around about 10 minutes later (the navigation system had failed). Fortunately, there was another plane on the ground, and the mechanics were fairly certain they’d fixed the landing gear problem, so we got on that one. As luck would have it, it made it all the way to Pole.
I would say that trip is about average for delays and turnarounds (“boomerangs” as we call them). I’ve had worse, and I’ve had better. This time around, I at least have a C-17 flight to look forward to on the Northbound side.
I’m back in the US now, catching up on homework, and everything else I’ve missed. My last month at Pole was extremely busy, so I didn’t have time to write. I won’t be able to write a lot about what happened either, as I was pretty sleep-deprived, and the memories are already fading. That being said, I’ve got a few pictures to help jog my memory of some of the work we did. Here’s one example.
This story starts with our attempt at what is known as “first light” in the astronomy community. First light refers to the first time a new telescope (or camera) sees an astrophysical object. First light is a pretty big deal because it’s the first demonstration that all the parts of the telescope (optics, detectors, drives, etc.) are more or less working. Usually, all of those parts have been tested individually, but you never really know if they’ll all work together. Unfortunately, the first light for SPT-3G didn’t quite go as planned.
Let me set the scene up a little bit. We’d spent almost a week getting everything installed in the telescope. Everyone was excited to actually see something with the telescope. Furthermore, we’d asked for some IT maintenance to be delayed so it didn’t mess up any of our communications that day, so the whole station new we were getting close. By the time we got around to starting observations, it was nearly midnight. The entire SPT crew (25 people) and a few folks from the station were trying to crowd around about 5 different laptop screens to watch things happen. After a false start or two (at least one of which was my fault), we finally got our observation going.
When we observe astrophysical sources, we start with the telescope looking below the source, and scan back and forth, slowly stepping up each time. This ensures that every detector has an opportunity to see the source (which is important for calibrating their response). It also means there’s some delay between starting an observation and when we should start to see the source. In fact, due to some imperfections in our pointing model, we don’t know quite when we’ll actually start to point detectors at anything visible.
To make a long, agonizing story short, we spent over an hour sitting around, wondering when we’d see anything, before concluding we wouldn’t be able to. That’s when we found out our superconducting detectors were heating up every time the telescope accelerated, which caused them to become normal metal (and very poor detectors). The exact cause is still unknown, but most of us have come to the conclusion that the acceleration was causing vibrations inside our camera that eventually dissipated as heat.
After looking carefully at the telescope movements and the detector temperatures, we concluded that the telescope was accelerating much faster than it should have. Unfortunately, the maximum acceleration is a hard parameter to control. It’s set in hardware, deep in the heart of the telescope. So, we had no choice but to open up the telescope’s heart, and change the hardware.
The hardware in question is a pair of RC filters that control the acceleration of the two axes (azimuth and elevation). To decrease the acceleration, we had to change the resistors and capacitors inside one of the telescope drive cabinets. These particular components are on a circuit board mounted inside a computer which is, in turn, embedded in the drive cabinet.
You’ll notice that board has a lot of unlabeled white wires coming out of it, which made detaching the board entirely a daunting prospect. Instead, we rolled a cart over to the drive cabinet and did the modification with only a few cables disconnected (we did turn off the power first, though).
From there, we were able to do all the necessary soldering. It was a fairly complicated operation, since the cables were barely long enough for use to put the board on the cart. The whole operation required two people (one to operate the soldering iron, and one to manage the components we were changing.
In the end, the operation was a great success. We reduced the acceleration on both axes to the point that any heating we get is acceptable. The next night we were able to get a proper first light, and had a grand old celebration afterwards.
Thanks to one of our winterovers, Daniel Michalik, for allowing me to use his pictures of the repairs. I had my hands full of soldering iron at the time!
So, I’m back at the South Pole again. It’s been just under two years since
my last trip, and a lot has changed. First of all, I’m no longer a
freelancing scientist-for-hire. In the Fall of 2015 I started a
PhD. program at UC Berkeley. As such, I’ve had less time to run around and
do fun stuff (hence the two year gap between visits to Pole). However, I
think I spent the time pretty well. I’ve been helping to build parts of the
SPT-3G receiver that we are currently deploying.
SPT-3G (for 3rd generation) is a brand new camera we are currently preparing
to mount on the telescope. The instrument capabilities and science goals
are largely similar to the SPTpol instrument I worked on for 4 years.
However, SPT-3G has 10 times as many detectors (about 16,000, in total). By
the end of the survey, the increased detector count should allow us to do
some very interesting science. I don’t have time to go into all of it now,
but I want to mention one topic I’m very excited about.
One of the primary goals for SPT has always been to find massive clusters of
galaxies. The superheated gas in these clusters is hot enough to interact
with the CMB photons. This leaves a distinct imprint at the location of
each cluster when we look at the sky. Massive galaxy clusters are of
particular interest, because they can tell us quite a lot about how
structure formed in the universe. However, to get to that point, we need
the mass of the galaxy clusters.
Current methods (based on a X-ray observations) lead to a 10-20% error in
the mass estimate, which is pretty limiting to our ability to make
definitive statements about the growth of structure. However, SPT-3G has the
potential to improve those measurements. Galaxy clusters are so heavy that
they bend light (much like a black hole), including the CMB. By looking for
these small shifts of the background radiation, it is possible to tell how
much a foreground object weighs. This measurement is very hard to do, since
the bending is very subtle, and we don’t know exactly what the background
should look like before it is distorted. But, by looking at hundreds or
thousands of galaxy clusters, we can determine the average mass, and use
additional observables to back out the mass of each cluster. In the next 10
years this technique should give us 2-3% errors on cluster mass.
So, there you have it, a quick introduction to SPT-3G. This has been a huge
project for the entire group, and everyone has been working way too many
hours. But, things are getting close. If we’re lucky, we’ll be seeing
first light in the next 36 hours.
Stories from the Southbound Voyage
I’m running out of satellite and brain power now, so I’m just going to dump
a couple pictures with brief stories here.
We took a very pleasant walk through the botanical gardens on our first day in Christchurch. Nathan thought that the Ponderosa Pines didn’t smell right.
On the way through McMurdo, I climbed the small hill right outside of town
for the first time. It is called Observation Hill (shortened to Ob Hill),
as it gives you a really good vantage point over the surrounding area. The
top has a small memorial to the Scott expedition that failed to return from
the South Pole in 1912.
As I was finishing up this post, we got some data coming live from the
camera. Check it out!
This is just a quick post to show off some of my recent pictures. The sun is almost up (the official sunrise is in 5 days). So far, most of the colors have been obscured by clouds. We’ve only had a few days of clear weather, and even those had clouds on the horizon.
I haven’t written very much about the kind of work I actually do down here, especially this year. My posts have been a lot more focused on the photography, because I’ve been taking that more seriously than I have in the past.
My typical day actually turns out to be pretty boring. I typically have some minor IT issues to deal with (software updates, or a hung process), or a minor maintenance task.
I usually have a lot of free time. I’ve been using most of it to work on analyzing SPTpol data from the last few years. However, we did have one major repair this year, and I’m going to try to describe the process we went through. This is largely adapted from an internal memo I wrote for future SPT winterovers, who might have to deal with something similar.
The first thing we noticed was that the primary mirror of the telescope had a lot of snow buildup. Fortunately, a little bit of snow on the mirror is not a particular bother for us. Since the wavelengths we look at are so long, the microwaves mostly pass through the snow. However, it does cause some diffraction and reflection, which essentially blurs the image we see. Some snow build up is normal during bad weather, but this amount of snow is problematic (and very unusual). We have a large heating system that keeps the mirror just a few degrees above the ambient temperature. Most of the time, this keeps the mirror completely clear.
When we looked a little further, we found that the breaker for the heaters had tripped. This meant that something had drawn too much current, and it was shut off to prevent damage (like starting a fire). We reset the breaker, and the heaters came back on. However, we found them tripped again a few days later. This pattern continued for a while, before we found the problem.
At the same time, we were having some problems with the telescope’s drive system. SPT uses a regenerative braking system (the same basic idea that is used in many hybrid cars). It turns the motion of the telescope back into power when we want to slow down. We noticed that the regeneration subsystem was occasionally failing. It was a strange failure, though. Some event would trigger the failure, and the control system would note it. However, everything was working normally afterwards.
In hindsight it seems obvious that the two problems were related, but real life is never that simple. In the interest of brevity, I’m going to skip over a couple weeks of debugging and attempted fixes. It turns out that the cables powering the heating system had been cut, and they were periodically shorting out against the telescope structure. Every time the cables shorted, they caused a large spike of current before the breaker cut it off. This was enough to make the regeneration circuitry think it was broken.
To understand how the cables were cut, you need to know how the telescope is connected to stationary utilities on the ground. The problem is not a simple one, as we have a large variety of things that need to be connected between the moving part of the telescope and the building. Besides electricity, we need fiber optics to transfer data back and forth, as well as large lines for compressed helium gas. Altogether, they create a bundle of cables that is 10-12 inches in diameter. That bundle is wrapped in a spiral at the base of the telescope. The bottom is attached to the building, and the top moves with the telescope. Each layer in the spiral moves a little bit more than the layer below it.
In principle, this is a very nice system. However, our implementation is somewhat flawed. It requires constant maintenance to keep all of the cable supports in place, and we were not watching it closely enough. After an attempt to improve the arrangement of cables this summer, they had all slipped downwards in the spiral wrap. The topmost section got so tight that it made contact with the central pillar that runs through the spiral. Over a period of several weeks, the cables that provided power to the mirror heaters were scraped back and forth against the pillar. Eventually, they were cut entirely.
Fixing an electrical issue of this magnitude is well beyond my own experience. Fortunately, we have people who know how to do these things. The station employs an electrician year-round to fix problems like this, but we ended up getting help from one of the IceCube winterovers who works on power distribution systems in the real world. We also got a lot of help from the winter machinist, both in doing the repairs, as well as building tools and new pieces.
It turns out that patching a cable like that is not very hard. However, we could not just leave the patch in the middle of the cable wrap, we had to move the broken part of the cables to the top of the wrap, where we could mount a junction box. To get enough slack in the cables we had to remove them entirely, and then replace them. In order to get them back in, we had to disassemble a lot of the brackets that hold the cable wrap together. When we tried to put the whole thing back together, we found that many of the brackets had been twisted by the weight of the cables, and would not go back together. This was after two days of non-stop work (most of us were eating ramen at the telescope to avoid spending the time walking back to the station).
To the left, you can see a picture of one of the brackets with our method of fixing them. Originally, they each had two threaded rods going through their tops (you can still see the holes for the rods). To get the broken cables out, we removed the threaded rods. In several of the brackets, we found they would not come out at all, so we cut them.
After removing those rods, the brackets twisted so that the holes no longer lined up. We were able to force some of them back in by bending the rods. However, that meant that different layers of the cable wrap were unevenly spaced. In certain positions, the brackets from a lower layer, would scrape against the cables in another layer. Over time, that would have led to exactly the same problem we were trying to fix. Our solution was to make pairs of aluminum bars that could be squeezed together by screws. These are much stronger, and serve the same purpose.
Unfortunately, that did not entirely solve the layer-spacing problem. The spacing is partly controlled by a set of brackets that are mounted to bearings on the central column. These brackets hold their portion of the cable wrap at a fixed elevation. However, the sections between the fixed brackets can sag, allowing them to contact the next layer down. The only additional control we have over the spacing is the angle at which the cable wrap enters and leaves each fixed bracket. That angle is determined by a long flat spring that forms the core of the wrap. The cables are all mounted to the spring, and they follow it quite tightly. By adjusting the angle of the spring at each bracket, we were able to get to the point where we could spin the telescope over an entire circle without any layers touching.
This last part has been the problem that keeps on giving. In order to solve this problem for good, we designed a new style of bracket that has almost not vertical head room. Our machinist made a small set of them, which we have been testing for several months. Just this week, the design was approved for manufacture. We will be getting several dozen this summer to install on the telescope.
As I said at the beginning, most of the time I don’t have a lot of work to do. But I am here for instances like this: when something goes catastrophically wrong. This fix took us four days, which is the longest unscheduled down time SPTpol has ever had. It was quite frustrating. For those four days, there were three or four of us spending 8-10 hours a day inside a cylinder that is roughly 10 feet in diameter. No matter who you are working with, that is bound to create some friction. I consider it a testament to everyone’s patience that we never got into a shouting match.
This year, I’ve been trying to wait several days between taking a photo and editing it. I usually try to wait a few more days before doing a final review and uploading them. The wait time allows me to separate myself from the photos a little bit, and view them more critically.
However, I took some this afternoon that I just couldn’t wait on. The sky is distinctly brighter now, and it shows up as this beautiful purple in the pictures. Combine that with a bright and active aurora, and it makes for an excellent scene. Here’s a quick sample of the pictures. You can see the full collection, which I am calling Purple Skies. I’ll probably add more in the next week or two.
I highly recommend looking at these in higher resolution. The blog-sized photos don’t show the smaller, fainter stars. This is the curse of high-resolution images.
Continuing with my efforts to create more cohesive photos, I started a set of photos I am calling Foregrounds in the Background. This is a play on the jargon we use in the CMB community. Since we are looking at the oldest light in the world (literally), it is behind everthing. Therefore, anything else is in the foreground. By that definition, I could post photos of anything above a telescope, and it would technically fit in with the theme. However, I was a little more stringent than that. One of the most common foregrounds we worry about is our own galaxy. Yet, when I take a picture of the Milky Way from the ground, it always appears as the background of my picture. Hence “Foregrounds in the Background”.
This set of photos was taken over a period of about 6 weeks beginning on July 1st. During that time, the South Pole went from full dark to the early stages of twilight. That is why the sky appears more purple in the later photos. We are going to enter what is known in most of the world as the “blue hour” very soon. Since we only get one sunrise and one sunset per year at the South Pole, the blue hour is more like the blue week (see this post for a discussion of lighting at the South Pole).
Here are a few photos that I think particularly stand out.
One of the most common camera accessories at the South Pole is the remote trigger. Remote triggers provide an alternative to using the shutter button mounted on the camera which gives the operator a few advantages:
For long exposures (longer than one-tenth of a second, but shorter than two seconds) or very long telephoto lenses, moving the trigger off the camera prevents any camera movement from pushing the button.
Many remote triggers have can be locked in the “on” position, meaning the camera will continually take pictures as long as the trigger remains plugged in.
The buttons are often larger, making them easier to use with gloves (or many layers of gloves and mittens, as the case may be).
For me, the first point is not very important. Over a 20-30 second exposure, some small movements in the first second are unimportant. However, the ease of use and the lock are very useful. Unfortunately, the standard insulation used on the vast majority of cables will freeze and crack at standard South Pole temperatures, so I replaced it with a cold-resistant cable.
The wire I used is a coaxial cable with two conductors. In the picture above, you can see one in the center, and one spread around the outside (this configuration is very useful when you need to transmit high-speed signals over long distances, but it is overkill for my use). The outside is a silicone insulator. The insulator between the conductors is silicone as well, but much softer, so the cable remains flexible. Silicone maintains its flexibility to temperatures well below anything we see at the South Pole (or any natural environment in the world). I got a small section of this cable from Robert Schwarz, the Keck Array winterover.
My initial intention was to just replace most of the original black cable with a short section of silicone cable. However, I decided that was too boring, and that I was going to be ambitious. I replaced the entire cable. That meant I had to open up the trigger itself to detach the original cable from the contacts, and solder mine in place. That was relatively easy. The hard part was removing the original cable from the plug and replacing it with my own.
Nikon uses a very complicated plug for its remote triggers (in fact, it uses about seven different plugs, depending on which camera you have). This plug is not only capable of triggering the shutter, but it can also communicate with GPS devices and other complicated electronics. So, instead of a simple three-contact plug, I had to work with something the size of a mini-usb plug that had eight contacts. In order to even access the contacts, I had to dig through several layers of plastic. After that, it was easy to solder on the new cable.
However, I was then left with a plug that had almost unprotected wires. It would have broken in the course of one walk to or from DSL. I needed some sort of strain relief. Again, I got ambitious.
I could have done something as simple as wrap the base of the plug in electrical tape, but I made an epoxy covering instead. Before it hardens, epoxy is quite runny, so I had to make a mold for it (see the picture to the right).
Unfortunately, the mold was a bit leaky, so I ended up with only a partial covering (the picture to the right shows the plug after sanding the epoxy). So, I had a make a second mold that didn’t leak.
The second mold was much easier to make, as I had already laid down the base. I simply used some aluminum tape.
The final step was to sand down the epoxy to create a more finished look.
The finished product (bottom) and the original (top).
You might have noticed that the site looks a little different. I changed the wordpress theme because the old one was too narrow for my tastes. I like to share my pictures in a relatively large resolution (800 pixels wide, for landscape shots), which was wider than the column of text in the old theme. This is part of a larger effort to improve the aesthetics of my corner of the internet, and perhaps unify my two sites. Over the last two days, I also put some pretty significant effort into revamping my photo website.
The remainder of this post is going to be rather off my usual topic, but it may foreshadow the direction I intend to go in the future. On the other hand, it might just be a few ideas that are itching to get out that will subside after I write this.
First, I’m going to say a few words about copyright, and the particular licensing I use on my photos. First of all, any photo that I take is my property, and, by default, I reserve all rights to it. That means that almost any use of it is prohibited without my written permission. I find that restrictive and stifling. If I was getting paid (or intended to sell) my work, I might feel differently. However, as a complete amateur, I’d rather allow some use of my photos, by default. So, I license them under one of the Creative Commons licenses. The particular license I use allows for non-commercial use (including derivatives) as long as you give me credit. The main thing I’m trying to block is other people making money off of my work. That seems unfair, especially since I’m not getting any money for it myself. Otherwise, I don’t much care what people do.
You may have noticed that I’ve been much less prolific with my photos this year. Which is odd, given that I came down with two to three times as much photo gear. However, I’ve been relatively frustrated with my photography recently.
At some point, I decided that my photos were missing something. I was getting the technical aspects right, in the way I wanted them, but the content was lacking. In short, I was displeased with my composition skills.
There certainly have been photos that I’ve been very pleased with (the one to the right, for example). Granted the auroras are pretty, but the overall image is relatively boring. This particular picture would have benefited from an interesting foreground, possibly something that ran parallel to the central aurora stripe (as it turns out, the flag pole that was about 10 feet behind me would have worked quite nicely). Most of the pictures I was particularly happy with came about through sheer luck. Recently, I’ve been trying to think about the composition before pressing the shutter release, and I’m finally starting to get some decent results. Here are a few examples:
This photo was taken at the open mic night after our midwinter celebration. I particularly like this one because the lights have nicely highlighted the people. The background had the decency to remain either dark or unlit. Getting that focus on my subjects has been hard for me, because real life tends to be cluttered. If I remember correctly, I actually darkened the background in post-processing to help the effect along.
This is another shot where I got lucky (although the version that is exactly what I intended worked out pretty well, too). In order to take this self-portrait, I had to set up my camera on a tripod, and then set it to take a long series of pictures. I then walked into the frame. For this picture, I walked into the frame about halfway through the exposure. The railing in front of me was partially exposed, before my body covered it.
On this one, I was slightly less lucky, but the fore ground is interesting enough to hold the image together (barely). I wish the auroras had been slightly higher, and in a place where they would show through the structure better.
My satellite connection is about to go down, so I have to conclude quickly. I’m beginning to see composition elements in my photos before I take them. This is a good thing, but it does slow me down. It’s particularly difficult in this environment- carefully positioning a camera on a tripod can be quite difficult with three layers of mittens. Nonetheless, you’re going to start seeing more carefully thought out photos. You’ll probably see more posts that are not South Pole-centric, or do not involve the South Pole at all. 30 seconds to post!
This past weekend marked our halfway point through the winter. On June 21st, 22:51, the sun reached its farthest point from us. For the next 8-12 weeks, we will continue to be in the dark. However, some time in August, the light from the sun will start to diffract through the atmosphere to us. Around that time it will look like a hazy, vague bright spot on the horizon. From there it will only get brighter, until sunrise on or around September 21st.
As usual, we celebrated the solstice with a two-day weekend, and a special dinner. This year also featured an odd collection of gifts from some of the summer people. We got 5 large boxes containing chocolate, socks, and toy models from Star Trek and Star Wars. We also had an open mic night after dinner. As expected, the quality of performance was mixed, but it was fun nonetheless. I’ve uploaded many pictures to SmugMug, but here is a collection of my favorites.