Square blames last week's outage on DNS screw-up

Square says the widespread outage that hit its payment terminals last week was caused by a DNS failure and not a cyberattack nor an intrusion.

During that 14-hour downtime, businesses across the US, UK, and beyond that relied on Square's point-of-sale systems were unable to process customers' credit and debit cards, hitting sales significantly. Restaurants, cafes, and shops were left asking punters to pay by cash or use another method like Venmo to transfer money to cover bills.

Some owners complained of losing thousands and thousands of dollars as the IT meltdown continued from Thursday into Friday. CashApp was also affected by the outage.

Shares in Block, which operates Square and CashApp, fell more than five percent during the fiasco. The San Francisco-based biz today blamed updates to its network infrastructure for breaking its DNS, bringing down services.

• If your DNS queries LoOk liKE tHIs, it's not a ransom note, it's a security improvement
• Microsoft DNS boo-boo breaks Hotmail for users around the globe
• If you can't log into Azure, Teams or Xbox Live right now: Microsoft cloud services in worldwide outage
• ICANN responds to Ukraine demand to delete all Russian domains

"The outage impacted an important part of our infrastructure, known as a Domain Name System, or DNS," Square explained Monday in a brief postmortem report.

"While making several standard changes to our internal network software, the combination of updates prevented our systems from properly communicating with each other, and ultimately caused the disruption.

"The issue also affected many of our internal tools for troubleshooting and support, making them temporarily unavailable. There is no evidence that this was a cybersecurity event or that any seller or buyer data was compromised by the outage."

Here's our timeline of Square's IT woes last week:

• Wednesday, September 6: A half-hour outage affecting Square's payroll payment service. There was also an hour-long issue preventing sellers from changing some of their settings. Likely unrelated to the above retail payment issues but we'll mention it anyway.
• Thursday, September 7: Multiple failures in Square's backend systems affecting payment processing, transfers, and other services, starting at midday PT, and lasting until the early hours of the following day. At midnight on the 7th, Square admitted: "We do not have a solution for the disruption." Then about an hour later, it had figured it out: "Our engineering team has implemented a fix and services are beginning to recover." At 0200 PT, 14 hours after everything started to go wrong, Square said it was monitoring the situation as more of its backend righted itself, and payment services became available again.
• Friday, September 8: While Square was battling to fix the above outage, its Time Cards system, used by customers' workers to clock in and out, was also down for most of Thursday and into Friday.

  By 0700 PT on Friday, Square said it had fixed its broken IT systems that had brought down its services since midday the day before, and apologized.

Square also outlined how it hopes to avoid this sort of meltdown again:

1. It claimed it has made changes to its DNS and firewall servers to "protect against the issue we saw," and has taken other defensive steps.
2. Square is working on expanding the availability of what's called Offline Mode to all new Square payment terminals as well as most ones out in the field already. As the name suggests, this mode allows a terminal to queue up transactions for processing later while backend systems are down or can't otherwise be reached. Square claimed many sellers used Offline Mode during the outage; we note the biz warns "you are responsible for any expired, declined, or disputed payments accepted while offline."
3. The payments giant promised to improve the way it communications updates about downtime.

Finally, you know how the haiku goes. It's not DNS. There's no way it's DNS.

It was DNS. ®