Can you provide a link or reference to this path? I can't find much with basic searches and have given up several times on the gpg-connect-agent path while using Windows or Ubuntu. The path on Mac seems okay but on other platforms is way too difficult.
The false Seelbach Cocktail story is repeated here:
> Max “Scoopie” Allen was making an Old Fashioned in the Seelbach Hotel Rathskeller in 1917. Whether someone bumped him or was walking by with an open bottle, somehow champagne splashed into his mixing glass.
NYTimes wrote about this created history:
> Shortly after being put in charge of the hotel’s bar and restaurant operations in 1995, Mr. Seger declared that he had discovered a recipe for a pre-Prohibition cocktail that was once the hotel’s signature drink.
For the many of us who have built businesses dependent on S3, is anyone else surprised at a few assumptions embedded here?
* "authorized S3 team member" -- how did this team member acquire these elevated privs?
* Running playbooks is done by one member without a second set of eyes or approval?
* "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years"
The good news:
* "The S3 team had planned further partitioning of the index subsystem later this year. We are reprioritizing that work to begin immediately."
The truly embarrassing that everyone has known about for years is the status page:
* "we were unable to update the individual services’ status on the AWS Service Health Dashboard "
When there is a wildly-popular Chrome plugin to fix your page ("Real AWS Status") you would think a company as responsive as AWS would have fixed this years ago.
Precisely right. It will be fascinating to see if Lightsail creation can be automated via Terraform or Cloudformation, and the $0.09 per GB for outgoing bandwidth can be easily subverted with this new pricing structure.
It's probably temporary? Move to R53, then figure out what's needed to manage records on two providers (if only for internal processes). Top engineering teams aren't going to knee-jerk this, right? Or did Dyn show some unfixable incompetence?
Is it? I thought GitHub, at least, was split Dyn/Route53 shortly after the Dyn outage started as a means of getting back online. Now, they've removed their Dyn, and are now exclusively Route53.
I don't think Dyn showed any incompetence; the parent-poster was merely remarking on relying entirely on a single provider, who, if they get DDoS'd, causes your site goes down. (There was some previous discussion about splitting between providers, but some commenters noted that it was difficult, or at least non-trivial, to replicate records between two providers.)
The problem is that you need to find a DNS provider that allows master and slave configurations of your DNS information. For example, Dyn can act as a master and UltraDNS can act as a slave, however, Route53, you can't be either. With Route53, you are all in.
Lucky for Route53 users, Route53 DNS surface is really large and there is a really good chance that not even is attack could hurt it.
AXFR isn't the only way to sync records between providers. You just need a tool that speaks to the APIs of each provider and can sync between them that way. Heck, I had syncing in place at a startup between Route 53, DNS Made Easy, a pair of TinyDNS servers, and a git repo (which was our historical backup of changes) years ago. It was 300 lines of Python and 100 lines of shell. Albeit, we only had a few dozen or so records to manage, but this isn't rocket science.
Aside: I came out of college as a sys admin with a CS degree and writing tools like this was par for the course. If devops folks aren't writing tools like this today, what are they doing?
Honestly: I think they are spending most of their time moving existing working infrastructure into containerized infrastructure and figuring out how to deploy their blog on k8s. They are working on learning libraries that abstract abstractions.
To be fair Route 53 will split over other providers so you should be good in theory. But yeah if something more specifically targets Route 53 then that could be the same problem.
So we now run a split view with DYN and AWS. My biggest issue with AWS is again they are a large attack surface, but also they don’t really play super nice with others and no DNSSec.
We are currently evaluating the Netflix denominator tools to spread DNS and sync our alternate providers.
My biggest problem during this outage was that I could not login to my registrar and make change to DNS directly - I had to login to DYN and ADD Route 53, it was impossible to remove DYN completely. And that's how we landed up with a split view.
NOW if anyone can tell me of a competitor for the Traffic Director product that works on Port 25 I'll be happy to consider a migration. Cloudflare has something in the works, but I’d really just like a DNS provider with a virtual load balancer that can handle my 250qps at a reasonable price.
You could build your own DNS set up for Traffic Director. Based upon the recursive server IP hitting you, send back responses that are closest to the user.
Which Google DNS uses to tell your name servers more information about where the client is located. This allows you then direct them to the nearest server.
We use CloudFlare and after this my boss said "set us up with secondary DNS somewhere." Unfortunately, CloudFlare doesn't support being a primary DNS provider with NOTIFY messages. They are designed to handle the DDoS for us by proxying content. It's an interesting problem and I don't know whether to push back to CloudFlare or my boss. Anybody else running secondary DNS after this with CloudFlare?
What you should do depends on your setup and threat model. Do you fear DNS auth going down? Do you think your DNS will be a target? Do you use Cloudflare to hide your HTTP origin IP addresses?
For example, if you fear DNS auth going down, but you must use Cloudflare for HTTPS (say: for caching and SSL certs), then changing DNS off CF makes little sense. You already assume stability by expecting it to work HTTP layer.
If you think you can be a target of DNS attack, I'd say having multiple auth is unlikely to give you more mileage.
If you can afford disabling CF on HTTP layer, exposing your HTTP origin IP and want to have two different DNS auth providers, fine, you can do CNAME. But then you have three vendors to worry about, and problems with each can lead to trouble.
By the way, slightly out of topic but I was very frustrated with a Cloudflare sales guy who reached out to my customer during the outage and told him that we should switch to Cloudflare to be protected from DDOS.
It comes a bit as gloating in the face of the attack on Dyn and there's no reason to believe that Cloudflare's DNS would fare any better.
From the numbers that were published, it seems that Cloudflare would've probably handled the attack without outages. They have significantly more PoPs, especially in the regions that were attacked (Dyn has 2 in US-East and 8 in US, Cloudflare has 6 US-East and ~20 in US overall). I think it's unlikely that an attack of 1-2Tbps would've brought them down.
Answering DNS is not very costly, so if you have enough capacity to the servers, answering shouldn't be the bottleneck.
I agree that it's very bold to do that, but I'd trust them with handling DDOS more than most other providers.
I don't know much about running nameservers but moving to all internally hosted seems like an odd choice to me, can anyone explain whey that's a good move?
With only a modest simplification you can view security as ultimately just being a figure measured in dollars: "it costs an adversary $X to beat these countermeasures." Your goal in securing a system is not to push X to infinity, though that might be a reasonable goal (e.g. if you're a security researcher designing new crypto primitives). Instead your goal in engineering your company's security consists in evaluating the value $V of what you're securing, and then raising X until X > V. There are uncertainties in measuring X and V and in how attackers will view these tradeoffs and so forth, but it's nothing you can't account for by building in an engineering tolerance like X > 2V. The basic story remains.
Spotify simultaneously has large resources and offers a non-essential infrastructure service (music to listen to while you're doing something else). The V gained in DoSing them is very small. They got attacked anyway because they shared infrastructure with other companies, which pools the V together to create something much larger. Some attacker saw a case where V >> X and attacked it to great success until Dyn was able to bring up X again. During the interim, Spotify was down despite having V << X.
In short: Spotify probably can't do DNS better than Dyn, but they can do DNS better than the sort of people who have reason to attack them (presumably trolls, maybe some future hacktivist who doesn't like some business decisions they make, unscrupulous competitors). This attack was a wake-up call for them, "oh, if we're pooling with these other folks then we'll become targets of larger hacktivist attacks and state actors, who are not directly targeting us per se." Those attackers could presumably still take out Spotify's home-rolled DNS, but they have no real motivation to target Spotify in particular any more.
It lower surface attack. With companies like Dyn, they are affected even when someone is targeting other sites, while with internal DNS servers that are only used by themselves they will be down only if someone is attacking them directly.
If someone is targeting them directly it doesn't matter much that DNS is up and running, their site is still down.
So they don't waste cycles on something not part of their core business or competency? Pretty standard reasons to pay someone to solve a problem. I think what this really showed is Dyn was not as competent in mitigating as what people thought.
The implication of incompetence isn't really fair here. This attack was fairly unique, in that it had a sufficient quantity to be a quality of its own. It's unclear whether any DNS provider could have survived it, except by luck of not being chosen as the target.
Yeah, that's exactly why I asked. Seems like one of those things where it makes sense to me to outsource, but I don't really know if I'm right on that.
[I'll try to make it simple, ignoring edge cases and real world complexity]
You can't outsource DNS. It's one of the critical piece of networking that must be in every infrastructure.
The common DNS server is BIND. It's been there for 30 years, it's well known, well manageable and well understood. Sysadmins have to know it and manage it. It's especially critical for worldwide multi-site tech organizations.
There is no need for anything else. BIND can do everything and is the most flexible. Some of the alternatives lack some or most of the features (e.g. some type of DNS records).
You should assume that any organization is running it's own DNS servers. (ignore the edge cases).
---
In practise for large scale operations, the DNS tree will get very complex.
What the websites changed was only the public DNS server for reddit.com or airbnb.com. It's only the top of the iceberg. There is likely a very complex DNS setup underneath including public domains, private domains, special internal domains, CDN, per datacenter, per continent, etc... which could imply 10 different DNS services.
Who serves the top level public domain is a details. We should assume that the companies put whatever they could in little time to fix the ongoing issue.
> You can't outsource DNS. It's one of the critical piece of networking that must be in every infrastructure.
This is simply not true. For resolvers, you can use your ISPs DNS servers or use a public resolver like Google DNS, OpenDNS, etc. For authoritative DNS there are plenty of hosted (outsourced) offerings like Route53, Dyn, Google Cloud DNS, etc.
This may not work for sufficiently complex organizations, but in my ~20 person SaaS company we have zero DNS servers and it works just fine. We use our ISP's resolvers for client lookups, and Google Cloud DNS for authoritative DNS.
As I said. It's a simplification. I really don't (and can't) get into a long explanation here about how to run a complex DNS infrastructure spanning multiple continents and datacenters ^^
Thing is. You gotta to run your own DNS since the moment you want your own DNS names. Good for you if a simple external DNS service is enough for you, a single 20 people office is not comparable to what the websites mentioned are operating.
If you think nobody will have much motive to run a very sophisticated / expensive attack on you specifically (e.g. Spotify), then self-hosted is great. You won't be taken out as collateral damage when they're targeting someone else.
> If Spotify's networks are all down, what good would a functioning DNS do?
Email would still work. You can't receive email if the sending server can't look up your MX records. Since spotify.com uses Google Apps, their email would survive a total network outage if they used third-party DNS.
"This outage exposed a critical weakness in our DNS hosting configuration. We are taking immediate steps to add additional DNS providers. This should allow us to avoid impact in the future, provided that at least one of our DNS providers is operational."
What happens if you have a DDOS on Route53? I'm sure they can handle the attack, but do you have to pay for the requests? Or are there clauses that they drop the fees if the requests were malicious? If not, the financial risk could easily outweigh the benefits of availability for smaller companies.
I'm in the process of looking for a secondary DNS server for a client but because they rely heavily on geolocation load balancing it's not simple... I wonder if anyone has other recommendation beside UltraDNS for a good slave?
Short version: banker provides liquidity to the markets and buys up junk bonds from panicked sellers at 40 cents on the dollar. Banker then sells the bonds for more money as the panic dissipates. Some people wonder if important banks like GS should still be making volatile bets like this.
Goldman Sachs will pay $5.06bn for its role in the 2008 financial crisis, the US Department of Justice said on Monday. The settlement, over the sale of mortgage-backed securities from 2005 to 2007, was first announced in January.
“This resolution holds Goldman Sachs accountable for its serious misconduct in falsely assuring investors that securities it sold were backed by sound mortgages, when it knew that they were full of mortgages that were likely to fail,” acting associate attorney general Stuart Delery said in a statement.
Actually the comment I was responding to was about "running a tight shop" -- which is about operational integrity in general (not just on the matter of unhedged risk).
In the, you know, "would you buy a used car from these guys?" sense.
Perhaps a difference of definition but I don't consider a tight shop to have anything to do with morals or integrity. Tight shops I always considered efficient and optimized. Car dealerships are shady, but they can run a tight shop.
The only reason GS is still around is because they did what they did. They protected their shareholders the fine is trivial (less than 10%) compared to the losses they would have taken if they followed the course of action that everyone else did.
Yep and? Every IB had to secure a deal GS got money from
Berkshire Hathaway, everyone else got acquired Lehman could not secure a deal. Unlike in every other deal government was not providing downside guarantees for Berkshire Hathaway investment in GS. Government did provide huge guarantees to JPM and every other bank that acquired an IB.
Operational integrity for investment bank is to a very significant degree risk management. Integrity as in having control over something not in a moral sense. In the case you referenced I'd rather see GS's counterparties taking more blame. The whole we are poor guys running multi-billion dollar funds and charging millions in fees didn't do due diligence and want to blame someone else thing is pure BS.
The point is that the assertion you attempted to make in favor of the idea that GS just of course wouldn't take large unhedged positions -- "they run a tight shop" -- just doesn't have a great deal of solid backing.
You can believe whatever you want. Based on my own research over the years -- and personal dealings with people who have worked there -- I tend not to trust that "shop".
The most important question: how do I find an apartment to rent in Stockholm city center without Swedish connections and without spending more than 1/3 net pay?
It's the same all over. In what capital city of a major democracy are there jobs and affordable apartments?
I initially meant that rhetorically, but now I'm actually interested to know. Maybe there are good non-govt jobs and low prices in smaller capital cities like Ottawa (<0.8M) and Canberra (<0.4M)? Educate me!
So if they sold at their true price via the Google-style reverse auction, they would have $9 * 10 million more to use for business growth. Instead lots of hedge funds and high net-worth individuals that are big clients of GS and JPM made an easy $100k this morning.
Even with the auction, Google IPO'd at $85, and the stock opened at $100. If you think that the investment banks gave away $90M of Twilio's money to their friends, do you believe that Google did the same thing with $300M of their own money?
Or they would have pissed everyone off by having an auction instead of a regular IPO and they wouldn't have had nearly as much interest in their shares. Google was a very special company and even then nearly managed to screw up their IPO.
No need for an auction. I don't know about this specific IPO, but often there are predictions of what the true stock price is (i.e. the "first trading day" stock price) floating around before the stock starts trading. E.g. for Facebook, the expected share price was about 37, the IPO was at 37, and the trading opened at 37. Facebook got the "premium", instead of the hedge funds and banks.
It absolutely has to do with the discussion. The reason for the tightness of the IPO and first day closing price of Facebook was because of those technical issues. Trading was disastrous and had Facebook not been a household name, it's easily believable that it would have traded even lower.
Actually, the news prior to the IPO about FB's advertising not being quite as effective made people lose a lot of confidence in FB and want to pull out.
This exposed a race condition within the IPO process that bungled it up. The post mortem talks about "order modifications" which are actually order cancels right up to the point of the cross was supposed to take place.
Pricing the IPO lower would have had people not feeling that the IPO was overpriced. The overpriced feeling people got caused them to want to tap out and brought out the particular race condition that Nasdaq hit. Normally, IPOs get lots of buying interest. The all-in, then folding behavior was unprecedented as people who didn't believe in the company IPO would never have put in bids in the first place.
How are you tying efficacy of advertising with busted exchange? I don't see it.
This deal was upsized very close to the IPO date and completely oversubscribed with huge retail interest. Everyone wanted this deal to succeed and news behind it was extremely positive. Don't see how random chatter about advertising causes a stock exchange to blow up.
To piggyback on the weird comparisons, why didn't Square encounter any issues on their IPO? News was extremely negative and it was said that they were forced to IPO to cash out investors.
But that's good for the GGP's argument. Going down means they extracted maximum value from the market. It's a very successful public offering by GGP's metric.
What do you mean by "going down"? The IPO price was the IPO price. If people are unable to trade during the day and the price is unchanged, it just means that the stock didn't have an opportunity to trade up.
But Facebook didn't trade at a higher price when there was no longer any trouble. Nothing material changed about Facebook in those few days. So either the price would have gone down (meaning Facebook did the right thing), it would have stayed flat (same here), or if it would go up, it would go up when the exchange fixed their trouble (it didn't).
Nothing indicates that this went poorly for Facebook. It seems to indicate the opposite.
Think about it this way: 9 * 10m = $90 million. But probably they couldn't have sold the whole 10M slug at 24. Let's guess and say they left more like $45M on the table.
They are currently at a ~2B valuation.
45M / 2B = 2.25%
Twilio is (in theory) a high growth tech company. Say they want to grow at 15% a year. 2.25% / 15% = .15 or about 2 months worth of growth.
2 months is certainly something, but in the grand scheme of things isn't the hugest deal in the world.
Sentiment behind tech IPOs is already negative and complicating the IPO process would only further that sentiment. Hard to imagine the bookrunners would ever agree to that.
