Mark Loveless, aka Simple Nomad, is a researcher and hacker. He frequently speaks at security conferences around the globe, gets quoted in the press, and has a somewhat odd perspective on security in general.

The AT&T Fiber Upgrade Pt. 1

The AT&T Fiber Upgrade Pt. 1

AT&T DSL Speed

AT&T DSL Speed

I’ve had DSL for ages. This DSL line had to be a “business” line as I wanted 5 static IPs, and I've had servers up and operational on the Internet for years. Having the business line is nice, as I get slightly better service such as prioritization during outages and so on. It is cheap, all things considered. It’s not a very fast plan at 50m down 5m up (actual speeds are typically maybe 55m down 12m up), so while not great we can still stream movies during a Zoom meeting and everything is just fine. I've stayed on DSL because I was told no static IPs on fiber in a residential neighborhood even if one has a business account. That seems to be the party line (no phone company pun intended), so no fiber.

So on March 10, 2021 out of the blue AT&T called me and said “free upgrade to fiber” and I could keep my 5 static IPs. They even stated I could keep the same 5 static IPs - they would handle it on their back end. The lowest priced fiber package was 100m down and up and it’s nearly the same price I was paying for DSL, so I jumped at the chance. I still expected everything to go wrong, but they said they would keep both services up until the cutover was complete, and the technician would not leave until the cutover was successful. And so began the adventure.

The First Visit

I had Friday the 19th of March off from work, and I scheduled the install (4 hours anticipated) for that day. The original window was 8am until 4pm, the dispatcher called and arranged for noon until 4, and the tech arrived at 8:35am, so 3 and half hours early. Fine, I was already up and waiting on an early morning grocery delivery.

The tech couldn’t find the fiber terminal. The old DSL line’s terminal was next door, but not the fiber terminal. He looked it up on the super-secret map of AT&T stuff and it was across the street. Fortunately they’d already run line under the street with a junction in our front yard. Granted it was at edge of the yard on the opposite side from where the existing phone drop is. So a line would need to run off of that junction, it would need to be buried under the edge of the front yard, run under the driveway, up toward the side of the house, under a sidewalk, and finally up the side of the house.

Well this would mean another work order. I was given the choice of the AT&T tech installing the fiber run - running the cable across the yard, driveway, and sidewalk - and then in a couple weeks or so they’d have someone else come out and bury it. That’s right, fiber across the top of my driveway with our cars driving over it (plus deliveries, yard people with mowers, etc). I opted to wait until they buried the line before cutting the service over.

The Next Visit

I received an email from AT&T and was told the next visit was going to be March 25th. Well, that was far less than two weeks the on-site tech mentioned so that made me happy. However much to my surprise, the crew of two cable contractors (non-AT&T) showed up at 5:30pm on March 22nd. Unannounced. This was weird since before the previous visit I received at least a dozen emails and texts. But this pair didn't even ring the doorbell or anything, just started working. I only knew they were there thanks to the security cameras and the dog going nuts.

I ran out and voiced my concerns over the sprinkler system, and they said since they were digging "by hand" that they would not harm anything. And they really didn't "dig" as much as just open up a seam across the yard and slide the cable in. It seemed to go quick, even putting in the conduit under the driveway and under a sidewalk with some specialized machinery. The cable-burying job from start to finish took them about an hour.

The Tech Returns

After the cable contractors sent in their update, AT&T quickly arranged for a tech to return, and a different AT&T tech was back on March 26th. After showing the new guy where I wanted the cable to enter the house and where I wanted all the equipment installed, he went to work.

Unfortunately the cable guys had done something wrong, and there was an error showing up on the tech’s gear 190 feet down the cable. This put the problem squarely at where the cable crew attached things. This also meant I was going to have to expect yet another visit from the cable guys. Fortunately the tech did add another junction box out there to make their job slightly easier, but he didn’t have the rest of the parts to redo their work, especially the cable itself.

The good news was that I did find out that the AT&T router, while slightly outdated, would not have to be changed out. In theory there should be no configuration changes on any of the equipment, and while I again raised my concerns over having static IP addresses, I was told there should be no changes needed on my end. This sounded like a chunk of routing would be handled upstream, and as long as AT&T managed their routes properly then this should work.

After the tech left, I received a call from AT&T customer support apologizing for the problems. The tech had apparently relayed that the cable contractors had arrived unannounced, so I was told that within two business days they would call to make the arrangements.

The Cable Contractors Return

On April 5th at 6:45pm, the cable contractors came back just like the last time - unannounced. The exact same two guys. I assumed that 1) they knew what they were doing, and 2) they knew why they were there. However they were asking me all kinds of things, with these exact questions being standouts - “What is wrong?” and “Why are we here?” They looked in the box where the junction was and said “it looks installed, what’s the problem?” I told them I had no idea what the exact problem was, that the AT&T tech said the fiber drop didn't work and was either not installed correctly or broken. It needed to be replaced. This went back and forth for at least 10 minutes. I made the shocking suggestion that this main person on this two-person crew contact whoever dispatched him out to get the details, because I had told him everything I knew. After he dramatically paused while thoughtfully staring at the ground, he looked up and said, "We'll be back in the morning to redo the cable."

They did not return the next morning, or at any point that day. Or the day after that.

Buried Wire Center

On the afternoon on April 8th, I called AT&T to find out what was going on. They were unaware that the contractors had been out on the 5th, and after I told them the story of what had happened so far, and it was "escalated". They seemed surprised that all of this was happening. Yeah well, that made two of us. It was escalated to the "Buried Wire Center" who were now going to be involved in the repair. This is a nation-wide department of AT&T whose job it is to get cabled installed or repaired that is buried in the ground. I figured they were responsible for major cable cuts like with a backhoe or whatever. Now they are going to help fix 190 feet of fiber in my yard. This was a big deal for the person on the phone, as he was "chatting" as in web chat via some app with the Buried Wire Center. Based on this person’s description, it seemed to be that same shitty web chat app thing on the AT&T website.

The Cable Contractors Return Yet Again

April 13th. It's 5:50pm. Those same two contractors show up again. In usual style they arrive and start working in the yard, unannounced. I was out running an errand and when I came back, my wife is in the yard asking them who the hell they are and why they were digging. A neighbor also joined in to yell at them.

I tell them that the install was escalated with the Buried Wire Center, I was told they (those two) were not supposed to come back and that the Buried Wire Center was going to handle it. I reminded the main contractor guy that he told me on the 5th they would return in the morning and did not. He looked down like a sheepish child that was scolded, and said very quietly "....sorry...."

I asked if the Buried Wire Center was going to come out and undo everything that they might do today. I get a shrug, and a look like this person has no idea who the Buried Wire Center is. I gave up. "I guess go ahead, you're here." So they did the exact opposite, and said they would let the Buried Wire Center do it and left. That's right, they left.

Conclusion Part One

It's April 13th. A Tuesday. It has been over a month since I was first contacted by AT&T about the upgrade. This blog post is way too long considering I am still trying to get past the first hurdle - installation of the cable. Let's hope this is only going to be a two-part blog, and not a series that lasts until next year.

Server Error

Server Error

The Home Network Overhaul

The Home Network Overhaul