The AT&T Fiber Upgrade Pt. Three

Double the speed of DSL for the exact same price. But there was still a cost - in anguish.

As stated in part two, got another cold call on the phone from AT&T. Jay called on June 21, asking if I wanted an upgrade to fiber. Why of course, Jay. Here is my business account number, here is a ticket number, I need someone to come out and simply fix the broken fiber from the previous three attempts. He thought that was odd and wanted the entire story. I gave it to him quickly, which still took a few minutes because of what had happened.

“I will look into this and call you tomorrow.”

It took a few days, but Jay did call back. And the entire process kicked off anew. This time things were quite different, as I was receiving frequent phone calls, text messages, and email updates, and repeatedly asked to fill out forms and confirm dates and times.

The new install date was June 30. I was told the technician would arrive between 8-5, but probably would be there first thing in the morning. I got up earlier than usual in the off chance the tech showed up straight up at 8am. So naturally, I received a call at 3:22pm stating they would arrive sometime after 4pm.

"So have they been out before?"

Why yes, you'll be the sixth visit. Perhaps you can fix the previous installation attempts.

Old School

Julius arrived at 5pm. He had a bit of grey hair, seemed to move a bit slower than some of the younger techs, but appeared to possess a deep understanding of the tech he was dealing with. I explained the previous situation, and after he mulled it over he came up with a plan. He’d run a new cable, set up a proper junction box on the outside of the house, I’d have DSL and Fiber until the cable techs buried the new cable. Less for them to screw up. He promised that while it would be above ground, he would work it into a seam in the driveway for now.

The 5 static IPs are going to be a pain, but I can live with this for the time being. At least there is a plan - a plan I understand - instead of pushing it off onto the next guy. The last technician had wanted to just run the cable straight across the driveway - no suggestion of using the seam or anything.

Julius got everything up and running with a much thicker fiber cable (“I prefer the thicker cable for outside runs, this is what you should have got the first time anyway”) and added a new router/gateway, an Arris BGW210-700. Since I had the static IPs he recommended not to cut over until the IPs were set up for the new service AND they’d buried the cable outside. I wasn't thrilled that some subcontractors from the Buried Wire Center were coming out to bury it, and I had a cable going across my driveway (in a seam, but still). However with the junction box on the outside of the house, there is far less that can go wrong, and it will be a much easier repair if they screw it up.

He made sure I had access to all of the administrative interfaces for the new gear, and stepped me through some of the trickier bits. He even gave me his personal cellphone number in case there were any problems when they buried the cable. He said “You’ve been through enough.” The preliminary speed tests were awesome. I’m paying for 100mb up and down, I’m getting over 120mb up and down.

The next day, as promised Julius texted me the details for the new /29 of static IPs. Great! At this point all I could do was hope that the cable burying goes well and I could cut things over at my leisure.

He mentioned that the crew that will bury the cable should be out within a couple of weeks.

Guess Who's Back

Two weeks to the day, I called AT&T to get install status, and was told the fiber cable burying guys should be on their way out. And yes, those same two guys (the idiots from part one of our saga) arrived for their fourth visit in as many months, and for the fourth time did not call first like AT&T assured me they would. Fine, at least they showed up. I tried speaking with them but since only one of them could speak any English, I managed to get the word "technician" out of him. I had no idea what he said, had him repeat it several times, and eventually gave up. I said "Are you going to bury the cable?" He answered yes, and I went back to work, glancing at them on the security camera now and then.

Their job was to bury the existing fiber cable that Julius installed on June 30th. They did not. Instead they buried yet another cable. Neither end is attached to anything, and each end of the new cable is loose sitting next to where Julius' cable is.

More Woes

I called and after dealing with the automated front end, I ended up with a call taker named Mike. After hearing what had happened, he said he would put me on hold, and then the call disconnected. Fortunately he called back and said I had to be transferred to a technician at UVerse. He did so and an automated service called the Exception Resolution Center answered, and after I tried to put in my four digit passcode (that I had never assigned as this was a new account) I was told I needed to call a different number. It was the number I called originally. Good god.

I called back and went through the same steps as before, and was transferred to an agent but it rang and rang maybe a dozen times, and finally something answered but it was static. Fearful that they had closed, I hung up and called back. After the automated dance, I reached yet another automated announcement that they were closed. Perfect.

Not 30 minutes after I got off the home, my wife came up and said “I hate to bring this up, but…” and she showed me two AT&T bills - one for the DSL and one for the fiber. Well that’s interesting, they were trying to bill me for three months of fiber, and technically I’d only had it for two weeks. Apparently the meter started running about the time of the very first visit in March.

This is insane.

Slowly Getting Things Right

The next morning I called and got the billing straightened out. The person (Diane, IIRC) assisting me was extremely helpful - she managed to not only straighten out the billing issue, but she also got me the Buried Wire Center ticket number for the botched install from the previous night. Apparently the status of the ticket was still open, I guess since the ends of the newly buried cable were still uninstalled.

I considered calling the Buried Wire Center and yelling, but remembered Julius’ earlier offer. I felt bad, but Julius did state that if I had an issue with the installers to go ahead and text him. He was such a nice guy, that I hated to bother him, but obviously I was still pissed off enough to not proceed with the favor. So I texted him for some assistance.

He responded fairly quickly that he would look into it, but told me it could be a bit. And then later that day, Julius called and stated that someone should be out here in a few days and will hook things up with the freshly-buried cable - assuming it is a good clean run. He stated that it probably would not be him, but if nothing had happened by the following Wednesday, to call him back and he’d do it.

Close to the End

Finally. On July 17th AT&T comes out, and connects the cable the Buried Wire Center contractors buried on July 14th. I have fiber up and operational on an actual buried fiber cable. After 8 visits between various technicians starting in March, the hardware part is done.

The main hardware concerns - having a buried fiber cable and having it actually functional with Internet access - only took four months. But they were completed on July 17th and things seems really good. I only wish I had started some of the configuration steps in advance, and instead did not start one particular key element until July 16th.

The Day Before

So what happened on July 16th? Since I have an email server that is going to have one of the new static IP addresses I was paying for, I needed reverse DNS lookups to work right, mainly to prevent rejected email. So I called AT&T Internet Tech Support to ask if could I set up reverse lookups, and maintain the old reverse lookups from the DSL setup. That would greatly ease the transition process.

As expected, the first person I spoke to at AT&T Tech Support could not help me, but it got worse. I spoke with people from AT&T Tech 360 support, and was even transferred to AT&T Fiber Support briefly, whose automated system transferred me back to the first call center.

I spoke to a total of five people, and I have to say that this chunk of wasted time infuriated me more than anything else that had happened so far. I told each person repeatedly how DNS works, how everything on my end was configured, and how this requires a change on their end. One person said “we don’t do that” and I stated that they do and had done it on my DSL account which was currently still up. Two people got all excited at one point, then proudly read off the recommended DNS server IP addresses used by business customers, thinking that since I said “static IP” and “DNS” this must be what I wanted.

I finally hung up on the last person, saying that I was sorry but I had a dentist appointment and needed to run. I even decided to bitch publicly about in a somewhat shallow but self gratifying way.

Maybe Progress?

To their credit, AT&T did respond to my tweet and ask for a DM and details. A nice woman named Rose called me, and while I tried to explain things I also told her I was in the car in rush hour traffic, and we’d have to continue the conversation later. She understood and stated she would arrange this. Besides, asking me technical questions when my gear was across town wasn’t going to cut it. The next day I even had a couple of calls from the Executive Complaint team (thank you Shelly) to help me resolve it, but by that time I think I had stumbled into a fix.

A Major Hurdle

In the DM conversation with AT&T they mentioned they wanted to escalate so I had given them the info they asked for to help that along. I did not wait on any response though, the following Monday on July 19th I tried calling the AT&T Fiber Support again. This time I suffering through all of the automated diagnostics, and even rebooted the new router (although I certainly did not need to). After the automated tests were complete, I was advised to hang up and call back, so I did. The automated diagnostics could see that I had rebooted, and asked if I were still having issues. Why yes, yes I am. I was transferred to a technician.

I reached Joseph, who asked what the issue was. When I mentioned I needed PTR records updated so reverse DNS would work, he said he was going to be able to help as he knew what I needed. Sure Joseph, sure. If you say so. I had no faith in this process whatsoever, as I had been told numerous times by a call taker that they knew what I was talking about and knew what was needed.

Instead, Joseph stated I needed to visit att.net/dns and I could fill out the online forms to get the PTR records updated. I nearly fell over. Joseph even stayed on the line with me, and made sure I got the forms filled out. Additionally at the bottom of the form was all of contact information for the provisioning help desk for DNS, including an email address. Holy shit. I hit submit on the form and thanked Joseph profusely.

Final Steps and Retrospect

It took about four or five hours for the PTR records to get updated and propagated out to the rest of the Internet’s DNS servers. And since I had plenty of time (4 months) to prepare for the cutover, I had detailed a migration plan and had pre-configured a number of files on the various servers. I implemented the plan, and with the changes made at my registrar Gandi the entire migration took about three hours. I will admit that the bulk of this was me getting all nit-picky with configuration files (fully commented, verifying backups were labeled properly, etc) but it is up and running.

My main mistake? Well, I cannot fault myself on the actual fiber cable install, this was painful and I could not think of another way to have done this that would have sped up the process. Julius was an absolute rockstar, and is going to get a letter or email or something sent to his higher-ups.

On the reverse DNS issue, well, all I can say is that I highly recommend one does not use any term at all that might point a non-technical (or non-deep-technical) person to the wrong information. I should have stuck with “I have a business account and need you to update PTR records” and I might have reached my destination quicker. Also, once I reached the AT&T Fiber Support call center, when I went through the pointless automated testing, router reboot, and “yes I still have the problem” dance with the robo-tech, I reached a person who knew exactly what a PTR record was. Live and learn - maybe someone will read this and save a few steps and some anguish.

But the good thing is I now have five machines with individual IP addresses doing their thing on the live Internet at more than double their old speed.