I really don't understand why the industry seems to have lost sight of this. It's really common to see super complicated, incredibly expensive, but highly scalable cloud deployments for problems that can be trivially solved with one or two dedicated servers. Even the mere suggestion of renting a dedicated server provokes scorn from devops teams. The overall difference in cost when taking into account all of the complexity, feature-lag and general ceremony must be at least 10x and maybe even closer to 100x. It's madness.
The allure of our industry to mime FAANG leaders might be one of the reasons we're at this stage. Most products could be safely run on simpler setups.
I suspect that VendorOps and complex tools like kubernetes are favored by complexity merchants which have arisen in the past decade. It looks great on resume and gives tech leaders a false sense of achievement.
Meanwhile Stackoverflow, which is arguably more important than most startups, is chugging along on dedicated machines.
Software development and related things are known to have fantastic often free online resources for autodidacts.
It seems like the trend in this space is to jump directly to the highest layer of abstraction. Skipping the fundamentals and point to $buzzword tools, libraries and products.
You see glimpses of this in different forms. One is social media threads which ask stuff like “How much of X do I need to know to learn Y?” Where X is a fundamental, possibly very stable technology or field and Y is some tool du jour or straight up a product or service.
"complexity merchants" - I love this term. It aptly describes beneficiaries of earlier IT fads. Remember UML? Remember WS-*? One could come up with a long list of technologies that were a bonanza for billing consultant hours but no so great for getting to solutions.
To be fair - J2EE (and its associated technologies like SOAP, XML) is a bit like alcohol. It is great when used in moderation, but gives you headaches when used excessively.
XML is great - if you don't believe it, then you might be interested in the pains of parsing JSON: http://seriot.ch/projects/parsing_json.html
And don't get me started on YAML...
The dedicated server deployments I worked on at smallish software companies 10+ years ago wound up being really annoying. I enjoy sysadmin type stuff and this idea has tempted me, but I think it’s a false economy in most cases.
The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc.
I want to worry about whether our product is good, not about middle of the night hardware failures, spinning up a new server and having it not work right because there was no incentive to use configuration management with a single box, having a single runaway task cause OOM or full disk and break everything instead of just that task’s VM, fear of restarting a machine that has been up 1000 days etc.
100%. There are definitely cases where managing your own dedicated server infra (not hardware, just the software part) makes sense.
For me, the allure of the cloud is not some FAANG scalability fad when its not required, but automatic OS patching, automatic load balancing, NAT services, centralized, analyzable logs, database services that I don't have to manage, out of the box rolling upgrades to services, managing crypto-keys and secrets, and of course, object storage. (These are all the things we use in our startup)
I'd go on a limb and say dedicated servers may be very viable/cost effective when we reach a certain scale, instead of the other way round. For the moment, the cloud expenses are worth it for my startup.
The key is to use some kind of config management, even for a single box. The problem you need to solve is basically to separate your configurations from the system default and make it easy to bring the system into a known good state.
I do think there's something in there about the immediacy of a replacement if a currently-live box falls over. Rough rule of thumb says it's hours for a physical box, minutes for a VM, seconds for a k8s container, a hopefully smaller number of seconds for a serverless cold start. Each of those has ergonomics different enough that it might matter even with config management that rhymes across the options.
How so? You should still be doing those things with cloud deployment anyway. The alternative is (the pretty terribly named) "click ops" where devops folks have to fiddle with some cloud UI settings to make production changes or go through wizards to attempt to replicate an environment.
If you are doing your environment config as code it ultimately shouldn't matter if your target is a dedicated server, a vm or some cloud specific setup configured via an api.
But most cloud deployments have config management too. Using Ansible/Terraform/Kubernetes on bare metal isn't much different than using Ansible/Terraform/Kubernetes in the cloud.
yes, or Puppet, Aviary, Shell, CFEngine, Salt, Capistrano, etc...
Doesn't really matter which one, the point is that you have one that can build you a new box or bring a misbehaving box back to a know good state by simply running a command.
Aaand that's why k8s seems like a good step in the right direction. It has all - okay, absolutely not all, but a good number of the important ones - of these concepts "built-in".
Sure, the actual pile of Go(o) that drives it can be improved (and it is indeed improving).
That said, the hard truth is that your approach is the correct one, almost all businesses (startups!) overbuild instead of focusing on providing value, identifying the actual niche, etc. (Yes, there's also the demand side of this, as the VC money inflated startups overbuild they want to depend on 3rd parties that can take the load, scale with them, SLAs and whatnot must be advertised.)
> k8s seems like a good step in the right direction
Kubernetes is fantastic! Whenever I see a potentially competing business start using Kubernetes, I am immediately relieved, as I know I don't have to worry about them anymore. They are about to disappear in a pile of unsolvable technical problems, trying to fix issues that can't be traced down in a tech stack of unbelievable complexity and working around limitations imposed by a system designed for businesses with traffic two-three orders of magnitude larger. Also, their COGS will be way higher than mine, which means they will need to price their product higher, unless they are just burning through VC money (in which case they are a flash in the pan).
k8s is almost the polar opposite of this idea for most companies. At minimum, a half dozen daemons spread across multiple hosts for.. what? Making the easy things built-in and everything else much more complex than it needs to be?
(I haven't used it in a few years, when I tried to deploy OpenStack with it, so if this is outdated, please, correct me!)
Ansible is imperative, it can work toward a static state, and that's it. If it runs into a problem it throws up its SSH connection, and cries tons of python errors, and gives up forever. Even with its inventory it's far from declarative.
k8s is a bunch of control loops that try to make progress toward their declared state. Failure is not a problem, it'll retry. It'll constantly check its own state, it has a nice API to report it, and thanks to a lot of reified concepts it's harder to have strange clashes between deployed components. (Whereas integrating multiple playbooks with Ansible is ... not trivial.)
Yes, ... and? Those are much more "standardized" than whatever else any team cooks up. (And k8s along with Go is steadily improving, so I don't see this as "let's use WordPress because its the platform that has the most answers on StackOverflow".)
And even if k8s puts on too many legacy-ness, there are upcoming slimmer manifestations of the core ideas. (Eg. https://github.com/aurae-runtime/aurae )
You need to solve a bunch of SRE/DevOps problem, fearless deploys, rollbacks, canary rollout, blue/green deployment, HA/LB, backups, dev environments, supporting multiple teams, yadda-yadda.
So of course you can implement a minimal complexity solution or you can use something "off the shelf".
k8s is complexity and some of it is definitely unneeded for your situation, but it's also rather general, flexible, well supported, etc.
> you’re suggesting that people use a “simpler” non “standard” implementation?
What I suggested is that if k8s the project/software gets too big to fail, then folks can switch to an alternative. Luckily the k8s concepts and API is open source, so they can be implemented in other projects, and I gave such an example to illustrate that picking k8s is not an Oracle-like vendor lock-in.
I think there's a lot of room to bring the best of both worlds together. Some are moving on this already, for example Hetzner Cloud load balancers can also have dedicated servers as targets.
If you've got your configuration properly automated, nothing prevent you from redeploying your bare-metal nodes every weeks. The opposite can actually also be true, it may become risky at some point to reboot a VM with a high uptime.
I'm completely clueless about any server administration beyond a LAMP stack on a single machine serving a few dozen people
I always hear about all the great stuff you get practically for free from cloud providers. Is that stuff actually that easy to set up and use? Any time I tried to set up my LAMP stack on a cloud service it was such a confusing and frightening process that I ended up giving up. I'm wondering if I just need to push a little harder and I'll get to Cloud Heaven
It's a mixed bag. It's more standardised, so once you know how to set up one app, you can mostly just repeat it multiple times as needed. Even for custom things, I can't remember the last time I started a CloudFormation stack from scratch - it's mostly copy&customise.
Being able to say "I want a clustered MySQL here with this kind of specs" is much better (time-wise) than doing it on your own. The updates are also nicely wrapped up in the system, so I'm just saying "apply the latest update" rather than manually ensuring failovers/restarts happen in the right order.
So you pay to simplify things, but the gain from that simplification only kicks in with larger projects. If you have something in a single server that you can restart while your users get an error page, it may not be worth it.
The big problem for server operations is not the damned servers but all of the infrastructure you need to keep them running, like power cooling network/storage and internet connectivity.
The cloud is not easy but damn trying to get cooling and power efficiency of an small server room anywhere near the efficiency levels most big data-center publish is next to impossible as is multi vendor internet connectivity.
With the cloud all of that kind of goes away as it's managed by whatever data-center operator that cloud is running on but what people forget is that that is also true for old fashioned colocation services which is often offering a better cost/value then cloud.
And while it's definitely harder to manage stuff like AWS or Azure because it bleeds a lot of abstractions small scall vpc providers hide from you or that you dont really get with a single home server, it's not hard on the scale of having to run a couple of racks worth of vmware servers with SAN based storage.
I think the problem is configuration. Every thing that has its own configuration is basically a new language you must be able to speak perfectly. Configuration languages are not statically typed so when you make an error it typically does not tell you you made a mistake, and where, and thy it is a mistake.
With Cloud stuff you have more configuration to do because it is about configuring virtual servers etc. Instead of carrying the PC in a box to your room you must "configure it" to make it available.
Laravel Forge + Digital Ocean (or Hetzner, your own machine, Vultr ...) is a pretty good solution. You can up & downgrade as you see fit or spin up a new server in a few min (any specs you need, from scratch or using an image of one of your existing machines, it installs what you need and you can of course add whatever is still missing).
DO databases have backups you can configure to your liking, store them on DO Spaces (like S3). DB user management is easy. There's also cache servers for Redis.
You can add a load balancer and connect it to your various web servers.
I think it took me about 30 min to setup 2x web servers, a DB server, cache server, load balancer, a storage server and connect them all as needed using a few simple forms. Can't really beat that.
By your own estimate -- how confident do you feel that these servers and services are secure? "Setting up" web servers to perform their function is rather easy, in my experience. Ensuring those servers can withstand standard-issue hacking attempts, not so much, especially within just 30 minutes.
The added value of these types of service, I think, is that they're fairly well set up with their provisioning script. You're paying for this service after all and if it appears they can be easily compromised "by default" then ... there would be many problems.
If you have any more info or opinions then please do share.
I'm no expert on Linux security but what I've seen when provisioning a VPS from anybody is a system with the base OS installed and that's it. SSH on port 22. Root account active. Accounts not needing a private key to login. No firewall. Known vulnerabilities in base packages unpatched.
Lots of articles around the internet about hardening a Linux server, the ones I've tried take a bit more than 30 min to follow the steps, a lot longer if I'm trying to actually learn and understand what each thing is doing, why it's important, what the underlying vulnerability is, and how I might need to customize some settings for my particular use case.
As someone who also runs a Laravel site on DO (but doesn't use Forge so can't comment on whether it handles any additional security configurations) buying a cheap DO droplet is very much handling your own infrastructure, it comes with reasonable enough defaults for most people out of the box but if you're expecting any decent amount of traffic it definitely needs to be hardened a little more. Also other administrative tasks like adding swap space if you're like me and trying to get as much performance as possible from as little spend as possible.
a bash script that auto installs everything will do. It has to be maintained though. You also lock down and harden the server. Running a stable OS helps.
I've run a few debian servers for years with little issues. Set up auto-update (very easy) for security updates. There's really little to go wrong.
I'm sure you can find example setup scripts online (configure autoupdates, firewall, applications, etc.), should be a matter of running 'curl $URL' and then 'chmod +x $FILE' and 'bash $FILE'. I didn't need configuration management (I do use my provider's backup service which is important I guess).
Totally, though I think in a fast paced/under-resourced environment incentives matter, and ephemeral cloud vms incentive admins to use configuration management, whereas it's easy to take a shortcut in the moment and quickly manually install something/edit config on a rarely-changing dedicated host.
Obviously the same can be said for long running VMs, and this can be solved by having a disciplined team, but I think it's generally more likely in an environment with a single long running dedicated machine.
This here is the trick. I like to do blue/green deployments. This workflow forces you to script (and implicitly test ) the full server+application deployment. With everything scripted, it just becomes a matter of deciding to rebuild a new machine every week/month/n-deploys which makes you comfortable. Plus, you get a hot spare for free.
> The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc.
> With booked daily backup or the backup included in the type of server, all data is backed up daily and retained for a maximum of 14 days. Recovery of backups (Restore) is possible via the konsoleH administration interface.
But i get the impression that the databases on managed servers are intended for use by apps running on that server, so there isn't really a concept of failover.
EC2 or on-premise you need monitoring to detect issues before they become issues.
A single drive on a single server failing should never cause a production outage.
A lot of configuration issues can be tracked down to self-contained deployments. Does that font file or JRE really need to be installed on the whole server or can you bundle it into your deployment.
Our deployments use on-premise and EC2 targets. The deployment script isn't different, only the IP for the host is.
Now, I will say if I can use S3 for something I 100% will. There is not an on-premise alternative for it with the same feature set.
HackerNews went down because two SSDs in a mirror went down at the same time. This was due to a hours on SSD bug. While Raid-Z2 works, and provides great reliability, two-servers replicated, both with raid-z2 and a nightly offsite backup and as long as you are testing your backups it's a solid plan.
What about simply running k8s on metal? You wouldn’t be able to auto-scale (unless you automated buying and selling the hardware provision from the DS) but it would be nice.
What PaaS providers / platforms do you like? I want to move some personal shell scripted boxes to PaaS, but dropped the idea with the decline of Heroku. Tried Dokku and struggled, but it seems like I should give it another try.
I'd love if you jumped into our Discord/Slack and brought up some of the issues you were seeing so we can at least make the experience better for others using Dokku. Feel free to hit me up there (my nick is `savant`).
Thank you for the kind words and I'll check out your Discord when I next take a pass at it.
Let me preface and say that I'm an application dev with only a working knowledge of Docker. I'm not super skilled at infra and the application I struggled with has peculiar deployment parameters: It's a Python app that at build-time parses several gigs of static HTML crawls to populate a Postgres database that's static after being built. A Flask web app then serves against that database. The HTML parsing evolves fast and so the populated DB data should be bundled as part of the application image(s).
IIRC, I struggled with structuring Dockerfiles when the DB wasn't persistent but instead just another transient part of the app, but it seemed surmountable. The bigger issue seemed to be how to avoid pulling gigs of rarely changed data from S3 for each build when ideally it'd be cached, especially in a way that behaved sanely across DigitalOcean and my local environment. I presume the right Docker image layer caching would address the issue, but I pretty rapidly reached the end of my knowledge and patience.
Dokku's DX does seem great for people doing normal things. :)
I’m in a weird position where I work with both extremely large enterprises and very small startups and not much in between. The enterprises in my area lean towards Azure so it’s still my cloud provider of choice. I haven’t had occasion to look for PaaS providers outside of the major cloud vendors yet. On the startup side, most of these companies want to be cloud first so it’s much easier to start them the right way and avoid the legacy mess that most enterprises deal with today.
Anything of note between the two which you found favorable? I am early-days into designing a Dokku backed system. Dokku has a lot going for it that makes it appealing to me, but happy to learn what I am missing.
The only problem with this is I don’t want something running on my own server that I barely understand. I’d rather use say fly.io that way if there is a problem I can get support from them.
In addition to what has already been said in the other comments, devs have an incentive to push cloud tech against the interest of their employers so they can put the experience in their resume, since cloud tech is seen as more specialized and in higher regard and as such a dev experienced in it commands a higher salary.
Plus who doesn't want to play with the newest, coolest toy on another's dime?
In my experience it's often the other way around. CTO/mgmt/HR would rather swallow the perf/cost hit of Cloud because it makes auditing, compliance, support, dr, security, training and hiring easier. Not to mention it usually provides much more flexibility during the PMF stage of a startup (where product roadmap, team talent/bandwidth and execution speed are often much more important drivers than the raw perf/cost ratio of your infra).
There's definitely an argument to move (some) stuff off cloud later in the journey when flexibility (or dealing with flux/pivoting) becomes less of a primary driver and scale/cost start dominating.
"It's insane not to start your startup on the Cloud and even more so to stay on it" paraphrasing something I think I've read on HN some time ago.
But I agree, it's best to treat the Cloud as an intermediate solution and once you stabilize and a clearer picture about requirements is available move at least partially to on-prem (especially the more expensive stuff like compute, storage is I think fine)
I wonder if there is any honest research into the TCO of cloud vs on-prem. In my team we pay ~100k/month for gcp, management doesn't seem to care, since the whole company is on gcp. But they also didn't care when it was on prem and was using hundreds of servers.
Realistically RDD is a very sane strategy for employees. When I read my company's job ads a few years ago I had serious doubts if I would get hired by them because my resume didn't have the buzzwords they wanted. Since then I always sprinkle in some new tech with each new project, not because it makes sense for the company, but for my resume and future employability.
If you're ever trying to get big enterprise contracts, the kind of scalability, compliance and disaster recovery that providers like AWS/Azure enable are now just table stakes.
Sure, you can get something working on Hetzner but be prepared to answer a lot more questions.
Yes fair point though I doubt some of these super distributed, scalable apps are able to properly execute a disaster recovery... And just because all the serverless components scale "infinitely" doesn't mean the patchwork stitching them together is.
Agreed on your last point of enterprise asking for this which again is just sad that these business "requirements" dictate how to architect and host your software when another way might be the much better one
I run a self-funded SaaS business, so I only deal with real problems, not imaginary ones.
Disaster recovery is a very real problem. Which is why I test it regularly. In a scorched-earth scenario, I can be up and running on a secondary (staging) system or a terraformed cloud system in less than an hour. For my business, that is enough.
We went through a growth boom, and like all of them before, it meant there were lots of inexperienced people being handed lots of money and urgent expectations. It’s a recipe for cargo culting and exploitative marketing.
But growth is slowing and money is getting more expensive, so we’ll slow down and start to re-learn the old lessons with exciting new variations. (Like Here: managing bare metal scaling and with containers and orchestration)
And the whole cycle will repeat in the next boom. That’s our industry for now.
A lot of its the buzz. 'Cloud Hosting' is something everyone thinks they need, despite not grasping why, or that the chances are they arent getting cloud hosting at all, just a VPS.
A solid dedicated server is 99% of the time far more useful than a crippled VPS on shared hardware but it obviously comes at an increased cost if you dont need all the resources they provide.
I think it might be due to the incentives - the costs are borne by the company (thus irrelevant to the employee), however putting new cool technologies on their CV is very relevant to the employee.
Yes - but stupid yearnings to "do what all the cool kids are doing now" are at least as strong in those who would normally be referred to as "managers", vs. "employees".
Good point - and in the case of managers there's also the "nobody got fired for buying IBM" effect:
outcome A) huge microservices/cloud spend goes wrong, well at least you were following best practices, these things happen, what can you do.
outcome B) you went with a Hetzner server and something went wrong, well you are a fool and should have gone with microservices, enjoy looking for a new job.
Thus encouraging managers to choose microservices/cloud. It might not be the right decision for the company, but it's the right decision for the manager, and it's the manager making the decision.
Yeah that's the problem. It's not just a "devs are idiots, they should think of their company and not be obsessed with their CV" - not having worked with the latest trendy tech has a very real cost for developers.
(As does being a consultant wanting an extension and writing software that works, as I found out the hard way.)
> I really don't understand why the industry seems to have lost sight of this.
There's a disconnect between founders and everyone else.
Founders believe they're going to >10x every year.
Reality is that they're 90% likely to fail.
90% of the time - you're fine failing on whatever.
Some % of the 10% of the time you succeed you're still fine without the cloud - at least for several years of success, plenty of time to switch if ever necessary.
Even if you 10x every year, going from year 1 to year 2 you switch from renting two 6-core servers to renting two 64 core servers, in year 3 you rent 20 servers, in year 4 you are renting a small room of racks in a colo space and add a server every other day (or you continue renting servers, if that's your style). Out of all the problems that arise from 10x-ing your organisation every year, this is a really minor one.
would prefer to spend 20-30-50 USD and get a managed deployment infrastructure with rollback, blue/green deployment, github integration and so. this setup can last a long time.
I think one of the reasons is that people confuse physical servers with manual administration. As I said, I do not do manual administration. Nothing ever gets configured on any server by hand. All administration is through ansible.
I only have one ansible setup, and it can work both for virtualized servers and physical ones. No difference. The only difference is that virtualized servers need to be set up with terraform first, and physical ones need to be ordered first and their IPs entered into a configuration file (inventory).
Of course, I am also careful to avoid becoming dependent on many other cloud services. For example, I use VpnCloud (https://github.com/dswd/vpncloud) for communication between the servers. As a side benefit, this also gives me the flexibility to switch to any infrastructure provider at any time.
My main point was that while virtualized offerings do have their uses, there is a (huge) gap between a $10/month hobby VPS and a company with exploding-growth B2C business. Most new businesses actually fall into that gap: you do not expect hockey-stick exponential growth in a profitable B2B SaaS. That's where you should question the usual default choice of "use AWS". I care about my COGS and my margins, so I look at this choice very carefully.
I'll be the counter opinion for this. Let's say you're Sprocket Masters, and you have to run two front ends (a Sprocket Sales site and a Sprocket Repair site) and a backend connecting them. But ultimately your sprockets are made in your factory on-site, and the majority of your staff is related to the manufacture of sprockets.
You're not a software company, fundamentally you make and sell Sprockets.
The opinions here would be hire a big eng/IT staff to "buy and maintain servers" (PaaS is bad) and then likely "write a bunch of code yourself" (SaaS is bad) or whatever is currently popular here (last thread here it was "Pushing PHP over SFTP without Git, and you're silly if you need more" lol)
But I believe businesses should do One Thing Well and avoid trying to compete (doing it manually) for things outside of their core competency. In this case, I would definitely think Sprocket Masters should not attempt to manage their own hardware and should rely on a provider to handle scaling, security, uptime, compliance, and all the little details. I also think their software should be bog-standard with as little in-house as possible. They're not a software shop and should be writing as little code as possible.
Realistically Widget Masters could run these sites with a rather small staff unless they decided to do it all manually, in which case they'd probably need a lot larger staff.
The core business for Sprocket Masters isn't really tech stuff: it's making sprockets. For those kind of businesses it makes a lot of sense; the tech stuff is required, but only in the sane sense that a phone line or electricity is.
However, what I also see – and what I think the previous poster was talking about – are businesses where tech is at the core of the business, and there it often makes less sense, and instead of saving time it seems to cost time. There's a reason there are AWS experts: it's not trivial. "Real" servers also aren't trivial, but also not necessarily harder than cloud services.
"Tech" isn't a thing. If I ran a company that produced CAD software, I would work in "tech", but I would have no reason to believe that my competency in writing CAD software should translate to competency in maintaining web servers or network infrastructure.
You still have to do all that with cloud though - instead of firewalls and IP tables it’s VPCs, IAM, security groups, auto scaling groups, etc etc. The reason I think it’s still probably better for Sprocket Masters to go cloud is purely because those skills are easier to hire for.
Sounds like Sprocket Masters should really outsource their two sites to a an agency that specialises in maintaining such solutions for SMEs, then they wouldn't need to concern themselves with the nitty gritty of hosting/backups/security/updates etc…
> . In this case, I would definitely think Sprocket Masters should not attempt to manage their own hardware and should rely on a provider to handle scaling, security, uptime, compliance, and all the little details.
But Sprocket Masters still has to have an expensive Cloud Consultant on retainer simply to respond to emergencies.
If you're going to have someone on staff to deal with the cloud issues, you may as well rent a server instead.
When the web was starting, that was the hardest sell to companies. They don't want to worry about those technical staff they just want a website/ecommerce working and not putting too much resource on it. Then social media happens and those sites just became status symbols or just to retain the email address.
it's not all or nothing. There's plenty of cases where using AWS or similar makes sense, but using the cloud for absolutely everything has a bunch of costs, both visible and invisible. Nuance and taste are the hardest things to teach, and these are in short supply in many engineering orgs.
Because cloud is the new "No one gets fired for using IBM". If you are using dedicated servers somewhere, you cannot have the same reputation of say Using AWS for cloud. Everyone knows the AWSs, Google Cloud etc and no one gets fired for selecting/using them.
I personally have run Dedicated Servers for our business in the earlier days but as we expanded and scaled, it became a lot easier to go with the Cloud providers to provision various services quickly even though the costs went up. Not to mention it is a lot easier to tell a prospective customer that you use "cloud like AWS" than "Oh we rent these machines in a data center run by some company" (which can actually be better but customers mostly wont get that). Audit, Compliance and others.
> trivially solved with one or two dedicated servers...
For many even lot less than that. I run a small side project[1] that went viral a few times (30K views in 24 hours or so) and it is running on a single core CPU web server and a managed Postgres likewise on a single CPU core. It hasn't even been close to full utilization.
I feel like many people use the cloud "wrong". If you're just standing up VMs then yeah, it's going to be slower and more expensive than renting a physical server. If you're just going to setup VMs and pretend you're still on-prem you're better off not switching. But I've got projects setup on PaaS services that literally cost a few bucks a month because it's only charging me for actual use and not idle time. I've got some small customers (tens of thousands of users) who I've helped move into the cloud who see the same thing. They have SPA websites backed by lambda functions and on-demand databases. If their website isn't being hit, it's not costing them anything at all. The only lock-in is the boiler plate code around the lambda activation which could be switched fairly easily to target any Functions as a Service platform.
I'm curious to know what on-demand databases do you use with AWS Lambdas? Two options that I've evaluated are: Aurora Serverless v2, which isn't quite serverless in practice and DynamoDB. And the mental model for DynamoDB very different from that of traditional SQL databases.
I’m actually primarily building on Azure with Cosmos DB. The mental model is very different, but I think important if you want to build cheap cloud services. I think that’s the larger point. You have to think about doing things differently in the cloud otherwise you’re just deploying and managing VMs as if you were on-prem albeit with more scaling control. It feels like everyone who is complaining about how expensive cloud hosting is go into it thinking it’s just someone else hosting the VMs where as a VM is the option of last resort for me.
I am responsible for about 10-12 t3.medium-class servers (number includes redundant servers) and the services running on them. All of them are fairly overprovisioned for what they are, excepting that the monitoring system we use on them for some reason likes to consume about half-a-gig of RAM so I can't run on even smaller servers. I write my services in Go, so the CPU isn't sitting there spending 98% of its time chasing pointers and can use multiple CPUs as needed for its heavy-hitting tasks. It's a rough day for those servers when they pass 5% average CPU for the day.
Could I switch some of them to lambda functions? Or switch to ECS? Or switch to some other cloud service du jour? Maybe. But the amount of time I spent writing this comment is already about six month's worth of savings for such a switch. If it's any more difficult than "push button, receive 100% reliably-translated service", it's hard to justify.
Some of this is also because the cloud does provide some other services now that enable this sort of thing. I don't need to run a Kafka cluster for basic messaging, they all come with a message bus. I use that. I use the hosted DB options. I use S3 or equivalent, etc. I find what's left over is almost hardly worth the hassle of trying to jam myself into some other paradigm when I'm paying single-digit dollars a month to just run on an EC2 instance.
It is absolutely the case that not everyone or every project can do this. I'm not stuck to this as an end-goal. When it doesn't work I immediately take appropriate scaling action. I'm not suggesting that you go rearchitect anything based on this. I'm just saying, it's not an option to be despised. It has a lot of flexibility in it, and the billing is quite consistent (or, to put it another way, the fact that if I suddenly have 50x the traffic, my system starts choking and sputtering noticeably rather than simply charging me hundreds of dollars more is a feature to me rather than a bug), and you are generally not stretching yourself to contort into some paradigm that is convenient for some cloud service but may not be convenient for you.
> Even the mere suggestion of renting a dedicated server provokes scorn from devops teams.
Have you ever had to manage one of those environments?
The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor (possibly voiding your savings).
You're inevitably going to end up with a bespoke solution, which means new joiners will have a harder time getting mastery of the system and significant people leaving the company will bring their intimate knowledge of your infrastructure with them. You're back to pets instead of cattles. Some servers are special, after a while automation means "a lot of glue shell scripts here and there" and an OS upgrade means either half infra is KO for a while or you don't do OS upgrades at all.
And in the fortunate case you need to scale up... You might find unpleasant surprises.
And don't ever get me started on the networking side. Unless you're renting your whole rack and placing your own networking hardware, you get what you get. Which could be very poor in either functionalities or performances... Assuming you're not doing anything fancy.
> The thing is, if you want to get some basic more-than-one-person scalability and proper devops then you have to overprovision by a significant factor
If you want 100.0000% uptime, sure. But you don't usually. The companies that want that kind of uptime normally has teams dedicated to it anyway.
And scaling works well on bare-metal too if you scale vertically - have you any idea the amount of power and throughput you can get from a single server?
It's concerning to keep hearing about "scaling" when the speaker means "horizontal scaling".
If you requirements are "scaling", then vertical scaling will take you far.
If your requirements are "horizontal scaling on demand", then, sure, cloud providers will help there. But, TBH, few places need that sort of scaling.
Just getting a highly available storage with any kind of performance requires you to overprovision by something like 12x (3 boxes with raid 1+0) which is frankly insane. And then you have to manage it and fix it when it inevitably breaks. The team who’s supposed to do that will be swamped by trying to keep the other 15 infrastructure services running and upgraded to a version which is only barely out of vendor support.
> Just getting a highly available storage with any kind of performance requires you to overprovision by something like 12x (3 boxes with raid 1+0) which is frankly insane.
I'm not saying 100% uptime on bare metal is cheap, I'm saying 100% uptime is frequently not needed.
Its big companies - their priorities are different. Saving a couple million dollars isn't worth the tradeoff of not being industry standrd (AWS, Google, MS). They hire tech consulting companies who sell their product, large deployments.
IME aiming for scalability is exceedingly wrong for most services/applications/whatever. And usually you pay so much overhead for the "scalable" solutions, that you then need to scale to make up for it.
Having talked to AWS evangelists, it seems that they truly believe it cuts costs. In their reality this is true, because an alternative with the same guarantees and risk costs more. The implicit disagreement is that you often don’t need all those guarantees.
> because an alternative with the same guarantees and risk costs more
I really doubt this. The one theme you see on almost every AWS proponent is some high amount of delusion about what guarantees AWS actually provides you.
Hmm. I guess there's guarantees that it won't happen, and guarantees that if it happens then it wasn't supposed to and can't be your fault, and the latter may still be pretty attractive to some decision-makers.
> if it happens then it wasn't supposed to and can't be your fault
Yeah, sure. Nobody gets fired for buying from IBM.
Well, if large companies had any competence in decision-making, they would be unbeatable and it would be hopeless to work on anything else at all. So, yeah, that's a public good.
Anybody with experience on the ops side of that is absolutely in favor of simple and easy to understand -- and that often gets you cheap as a side effect.
In my experience those devops people where hired and empowered by senior managers to make things expensive and complicate because the company is 'enterprise' and has to follow "industry best practices"
if you need durability in the cloud you just pick any object storage service from the 3 bigs, and a db with multiple deployment with automatic replication and managed failover.
getting the equivalent reliability with irons is a lot more expensive than renting "two dedicated servers" - now you might be fine with one server and a backup solution, and that's fair. but a sysad to create all that, even on a short contract for the initial setup and no maintenance, is going to go well beyond the cloud price difference, especially if there's a database in the mix and you care about that data.
I'm not saying there's no place for cloud, but that many workloads benefit from using dedicated hardware. It is often much simpler to deploy to a dedicated server rather than spending months rearchitecting your system (or even worse, cutting customer facing features!) to efficiently make use of many small cloud instances.
There are several reasons for that:
- lack of knowledge how to maintain own servers on premises
- MASSIVE PR by a handful of top tier cloud providers to instill the notion that cloud is the only solution, the best one and most advanced technologically and attractive silk set wise
- hockey stick growth expectations of (past days) startups from investors, forming the mindset that it must be met from day one in deployment
- “no one got fired for buying IBM” managers/tech leaders mentality, where IBM is replaced with “cloud”
- difficulty obtaining true costs information of cloud vs on premises deployment. For example - a team of ~16 people already pays off the costs of dedicated sysadmin to maintain on premises servers
- etc.
As a developer I appreciate the compartmentalization, being able to make truly isolated changes. This is something that's not trivial for lots of folks, me included. I guess if one has the expertise in house to manage a K8s cluster on a raw machine, that'd be an option but I, personally, am not gonna touch that with a ten foot pole. So yeah with a big and capable enough infrastructure team that complexity could be abstracted away but in reality I (where I've worked) haven't seen that yet.
Growth hides sins. I ran a large self-funded enterprise service at a loss for 4-5 years, which is possible if you are growing 30%.
Today, cloud is similar - the time to market is quicker as there are less moving parts. When the economy tanks and growth slows, the beancounters come in and do their thing.
the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies.
This only makes AWS richer at the expense of companies and cloud teams
> the most funny thing is that cloud engineering teams have no problem of using EC2 hosts, but somehow don't want to rent dedicated servers/VPC from regular datacenter companies.
It is trivial to provision and unprovision an EC2 instance automatically, within seconds if your deployment needs to scale up or scale down. That's what makes it fundamentally different from a bare metal server.
Now, I'm not denying that it might still be more cost effective when compared to AWS to provision a few more dedicated servers than you'll need, but when you have really unpredictable workloads it's not easy to keep up.
the whole concept of a need to spin up more VM instances to meet increased load is flawed and is driven by cloud vendors to make money.
if you are spinning up and shutting down VMs to meet demand curve - something is seriously wrong with your architecture.
Did it ever occur to you, how come stackoverflow uses ~8 dedicated servers to serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand?
--
When planning compute infrastructure, it is important to go back to basics and not fall for cloud vendor's propaganda
> serve entire world and doesn't need to spin up and shutdown VMs to meet global customer demand?
You said it yourself. Not all applications need to serve entire world, therefore demand will be lower when people go to sleep.
Even with global applications there are regulations that require you to host applications and data in specific regions. Imagine an application that is used by customers in Europe, Australia, and the US. They need to be served from regional data centres, and one cluster will be mostly sleeping when the other is running (because of timezones). With dedicated servers you would waste 60-70% of your resources, while using something like EC2/Fargate you can scale down to almost 0 when your region is sleeping.
Because the question needs to be revised. It is not about what you can do with an append only database vs. blockchain. That's not reflecting its purpose.
* Insurances (e.g. your travel insurance) are contracts. They should be digital, standardized, on a public ledger, so that I can travel anywhere and prove that I have one instead of having to bring, a fakeable piece of paper.
* If I resell my phone, transfering the remaining guarantee and/or insurance should be nothing else then a transaction. On the blockchain that's 10x more efficient than the status quo.
* If an artist sells his art the first time for $10, and the person resells it for 10000$, then the artist should be compensated too. Royalty mechanisms can solve for that and avoid secondary markets or at least create a more beneficial secondary.
There are an endless amount of examples which will make a lot of things more efficient, transparent and enforce standardization on a global level. Long way to go, but absolutely worth it.
As usual, all of these "use cases" suffer from the problem at the boundaries (see also Oracle problem), namely that you can automate everything on the blockchain you're using, but once you're back in the real world you no longer have any guarantee. Your new layer of abstraction just leaves you with a more brittle system.
* insurances: you still need somebody outside the smart contract that needs to connect it to what happens. You've just moved the trust from one place to another, and you're not better off.
* transferring money is faster in many countries than it is with (e.g.) BTC when you include the wait for the confirmation blocks. It's also bound to become faster and cheaper, while many blockchains (especially BTC) seem stuck with their processing time. Even countries with slow bank transfers now tend to have a fast solution, like Zelle in the US. Add problems like volatility of the cryptocurrency with the respect to the currency that actually matters, high (and volatile) fees, and transfers via cryptocurrencies is only attractive in a few countries and will only remain so as long as these countries do not improve.
* with art you again have the same boundary problem: the blockchain you use doesn't know what's happening, and only the law will resolve your conflicts.
> efficient, transparent and enforce standardization
So far the blockchain systems have been less efficient than non-blockchain ones they purported to replace, see the usual comparison with the Visa network for example.
Transparency and standardization are orthogonal to the use of a blockchain: using one doesn't mean you can't be opaque or follow any standard.
But this is only because businesses have yet to be built around this technology. These things will change. We are super early. We should not build these use cases everyone is asking for right now. The tech is still early and has to figure out many things. There is a lot of innovation happening on the base- and interoperability level, and it needs a lot more abstraction (e.g., network and account abstraction) to make things more usable.
Standardization also plays a huge role, and yes, in many ways, it's a technological chicken-and-egg problem whereby systems have to work with each other and the chain in a way that makes intermediaries as obsolete as possible.
The transfer speed of money or the general transaction speed - well, that's something that is being solved as we speak. These systems (e.g. Ethereum) have to mature, and many ways are being explored, using known principles like separation of concerns, e.g., separating settlement, execution, and data layers - but in a decentralized manner.
The public and many entrepreneurs need to understand how early this tech is and rush use cases that don't add value as of now. You must go deep and understand that this has to grow from the bottom up.
> this is only because businesses have yet to be built around this technology
No, these are fundamental problems that time can't solve. Businesses aren't built around this technology because it's not advantageous to use it in a legally compliant environment.
Companies in this field keep adding new layers of abstraction pretending they're solving a problem; they're just moving it to somewhere else, but the fundamentals remain the same. Basically running in circle.
It is hard to comprehend what it'd mean to have the world connected via distributed ledgers that provide transparency and would be able to automate a lot of what we do to a much higher degree.
We're not running in circles.
> Businesses aren't built around this technology because it's not advantageous to use it in a legally compliant environment.
This is just one of the reasons. The technology is also not scalable and abstracted enough yet. No one is pretending anything here.
> It is hard to comprehend what it'd mean to have the world connected via distributed ledgers that provide transparency and would be able to automate a lot of what we do to a much higher degree.
idk, take your phone thing as an example. Blockchain doesn't solve anything, because nothing stops me from reselling the phone without transferring the insurance, or transferring the insurance and keeping the phone. The whole problem is that it's not linked to the phone itself.
instead of going on about benefits, what is an industry that currently suffers from an actual problem that a blockchain could solve? because right now I'm not aware of any problems in society where blockchain is the solution
All blockchain proves is that you have access to someone's insurance key. While it would be a bit more difficult or costly than faking a card, it's not impossible. And most places I've been to don't even care about the card or the insurance number, they just ask for my regular ID and insurance provider to verify with.
> resell my phone
Two things:
1) The status quo could be that the new owner has the serial number of the phone. The fact that it isn't this simple (if it really isn't this simple) is because the seller doesn't want to make it that simple.
2) Insurance is particular to a person and their activities. For things such as aftermarket phone insurance the price you pay for the insurance will be based in part on your individual risks. This is obviously not transferable.
I just gave some examples. Tech is early, and the businesses around this new environment still have to be built. It's kind of a chicken-and-egg problem.
But what sets append-only databases apart from blockchains for any of these examples? Why can't I symmetrically say, "It's very early for append-only DBs, and new businesses around them still have to be built"?
Easy. Unlike pen and paper, the operator of an append-only database can publicly distribute it in real-time so that interested parties can read it and audit it for consistency. I don't see what the distributed consensus required by a blockchain adds on top of that, unless you're saying that all these applications require a zero-trust source of truth.
> unless you're saying that all these applications require a zero-trust source of truth.
Not all of them, of course. It's not a wonder child that aims to solve all of humanity's problems. It'll bring humanity closer, though and make it more efficient in many ways. People always try to look so narrow at it.
There are plenty of use cases and attributes (permissionlessness, immutability, zero-trust sources, etc.), and of course, not always all of these count all the time or are even desired. But it allows us to have and apply those as needed in an interoperable way.
But why do you think that its permissionless and zero-trust properties allow blockchain technology in particular to "bring humanity closer and make it more efficient in many ways", better than other technologies can? In other words, why are these properties so important that a blockchain is to an append-only DB as an append-only DB is to pen and paper, by your own analogy?
It may be early, but it's also actively competing with other systems. Those other systems aren't going to just stay stagnant until blockchain is ready.
> If an artist sells his art the first time for $10, and the person resells it for 10000$, then the artist should be compensated too.
This is far from axiomatic[1].
Also, presumably you can move up a level of abstraction and sell the identity that bought the artwork for $10000. The blockchain doesn't need to know that the keys have changed hands.
"Never" is a tiny bit harsh, IMHO; arguably, there have been several of them they just (evidently?) don't offer content producers as much value as it offers to consumers (which includes search engines), and thus the incentives play out in exactly that way
* JSON-LD markup exists in plenty of modern sites
* schema.org markup exists in some
* microformats.org briefly raised its head
The BBC website is the biggest example I know of which has a lot of semantic markup/annotations. IIRC they used to actually have RDF attributes in a lot of the BBC Radio listings
The solution is to use a virtualised scrollbar as well, so you have a div whose only purpose is to show a scroll bar, and then when that scrolls you scale the scrollTop proportionally. E.g. dragging the scroll bar by 1 pixel might equate to moving the grid by 20 items, but this is fine because in grids of that size it's impossible to precisely scroll using a scroll bar anyway. Scrolling via the scroll wheel is unaffected so the user doesn't notice.
This idea gets brought up in every thread about Insight's dust issues - there is not enough air pressure on Mars for the downwash to clean the panels. For Ingenuity specifically the problem is that it is also far too far away - it would take over ten years for the chopper to reach Insight.
The idea that Ghislaine Maxwell was one of the most powerful mods on Reddit, with close connections to the founders and admins, was widely derided at the time, but is probably true.
There's a mountain of circumstantial evidence including confirmation from Ellen Pao in a since-deleted tweet. The only rebuttals came from a newly appointed mod on reddit who claimed to have spoken to the user shortly after the arrest (but provided no evidence, the story was inconsistent and they have never spoken of this again), and a scathing article in Vice mocking the idea as a conspiracy theory, but again providing no evidence and it turns out that Vice clearly benefited from a business relationship with /u/maxwellhill
If you're looking for something cast-iron then I don't have it, but this is really a numbers game given that we only need 33 bits of information to deanonymise people on the internet. So let's count them and I'll let you decide how many bits to award each nugget.
1. The account's name is maxwellhill and Ghislaine Maxwell's surname is Maxwell and used to live at a place she referred to as Maxwell Hill.
2. The account was the first to reach 1m reddit karma and in the congratulatory post Alexis Ohanian revealed that he knew the poster in real life and that they'd been there since almost the beginning.
3. Ghislaine Maxwell is known to have attended events with Reddit staff in the early days, as confirmed by former CEO of reddit Ellen Pao.
4. maxwellhill has a December birthday, Ghislaine Maxwell's birthday is 25th December.
5. maxwellhill stopped posting just 2 days before GM's arrest, previously they'd posted almost every day for 14 years.
6. The few gaps when maxwellhill was not posting align with dates GM is known to have attended events including her mother's funeral.
7. GM's father was a known and prolific media manipulator, maxwellhill was repeatedly accused of manipulating reddit before the arrest.
8. maxwellhill posted prolifically about ocean conservation, GM had a startup called TerraMar which was focussed on ocean conservation.
9. maxwellhill mostly uses British English, GM is from Britain.
10. maxwellhill repeatedly complained about over-zealous child protection laws, posted articles suggesting CSAM should be legalised.
11. maxwellhill was careful to stress the difference between pedophilia and ephebophilia - a distinction important to Epstein and his apologists but not to most normal people.
12. maxwellhill has still not posted to reddit since the arrest, despite it having significant political and financial value.
13. After all of the noise of the accusations, no one at Reddit has actually denied that GM was maxwellhill.
I'm not saying this is conclusive, hence saying "probably", but that's a lot of bits.
I think the main thing people were pointing to was that account was extremely active then suddenly went inactive at the same time of Ghislaine's arrest. Been a while so there was probably other stuff too but that's the main one I recall.
Not particularly convincing, they seem just more like the kind of person you'd expect to be a reddit admin, i.e. a dodgy pedo incel male somewhat in the same area as /u/violentacrez (summary here for those unfamiliar: https://www.gawker.com/5950981/unmasking-reddits-violentacre...).
> I befriended a Korean pastor and his wife (long story there) where I live and we would meet occassionally for dinner.
> One day he asked if I could help teach some of his "students" conversational English. I didn't realise that he meant adults undergoing intensive training in Korea and abroad as missionaries for posting to other parts of the world. I thought he meant local teenagers who wanted to be more conversant in English - pronunciation, diction etc
> So I volunteered.
So, if people start looking for reddit accounts that mention maxwell and dodgy stuff, I can see why they'd stop posting.
I think part of the reason this gets so easily dismissed is that people still see reddit as a quaint little internet message board rather than an influential mainstream juggernaut that reaches half a billion people every month.
That account is/was enormously powerful, with the ability to determine which stories reached the frontpage and which didn't. That makes it politically and financially very valuable, and I don't think someone would just give up that because of association with Maxwell when they could have just posted a single comment saying "I am not Ghislaine Maxwell" and everyone would have forgotten about it.
The best thing you can do to minimise corruption is to make it necessary to bribe so many people for the corrupt to achieve their goals that doing so would be uneconomical. The reason Qatar is hosting the world cup right now is because FIFA has a very small selectorate, and therefore it's only necessary to bribe a relatively small number of people to swing the vote.
There's really no reason to blame this on the foreigners installing corrupt politicians, corruption is far more frequently organic and homegrown. Corrupt people work to further their own goals, if that happens to be using their power to take on e.g. debt on unfavourable terms then so be it, but that's not because they're beholden to foreigners, it's because that is their best route to enriching themselves - they can just steal the borrowed funds!
Keep in mind that corrupt societies are the most likely to have their debts forgiven, i.e. they are not likely to pay back the money borrowed.
I really recommend reading The Dictator's Handbook to learn more about how corruption works.
South Korea and Ghana were at the exact same place 50 years ago. South Korea’s leadership were largely competent, so they’ve industrialized and used trade to become one of the wealthiest countries in the world. Ghana has not had nearly as good leadership and institutions, so they haven’t.
Everyone constantly blames the West for all the problems of the world. If trade was really that deleterious to developing countries, you wouldn’t see the rise of China happening at all
Ha-Joon Chang explains the case of South Korea in details in his book "Kicking away the ladder": Korea developed because it was extremely protectionist, while receiving substantial financial support from the US (because of North Korea, China and USSR).
While Ghana has been under the rule of the IMF the whole time.
IMF policies are arguably extremely toxic; they basically guarantee that countries can't develop by keeping them open to a flood of subsidized products from more powerful countries, killing their ability to develop their agriculture and industry.
China developed similarly by stringently protecting its interior markets. The US did the same in the 19th century. Basically, all countries that successfully developed were staunchly protectionist, then became "free traders" once they become powerful enough to defend themselves on open markets.
Correlation, not causation. The whole point of IMF intervention is to rescue countries that are already in the toilet, and need to consolidate foreign debt.
and you would have a situation like recently in Sri Lanka where they defaulted on loans, and cannot obtain necessary resources like oil to run their economy.
IMF loans are bandaids to stop the bleeding, but the cure depends on the disease. If the country does not cure the disease, the bandaid will not help after a while.
If by country you mean the people who live there, yes.
If you mean whatever oligarchy is nominally in charge, well, the bankers can make sure they’re taken care of as long as the interest payments keep rolling in.
This was the policy of the USA— see Henry Charles Clay, the chief economic advisor of Abraham Lincoln. Started “the American School” of economics. It is all about healthy markets, but still allows for some protectionism. A middle way.
South Korea had a propped up dictatorship that the United States strengthened and funded because they were scared of commies.
Meanwhile Ghana had a dictator which was saddled with US loans who was overthrown. But the country still had to pay the loans back, despite them being will outside of their means.
This is extremely disrespectful of the economic savvy and work ethic of South Korean people.
Korean state had an extremely focused industrial policy to improve the added value created by the country's economic output.
The thing that needs critique here is that free trade would be somehow always beneficial. South Koreans state aggressively supported those industries it wanted to uplift.
Countries do rise up from third world countries to advanced industries. It just does not happen very often - sadly.
Another example of a country that rose from total poverty to industrialized advanced economy is my home country Finland which was in the 18th century dirt poor, started to industrialize, and only after second world war started to get some resemblance of wealth. However, the long history of poverty is still reflected to this day in lack of capital (are there other reasons for lack of capital I'm not sure).
Your implication is that Ghana failed where South Korea succeeded because they didn't work hard enough.
Of course people in South Korea worked hard to get where they are. So did people in Ghana. But in South Korea, the US protected them and funded them. Meanwhile in Ghana, the US stripped them of their wealth.
How well off do you think South Korea would be if the US didn't intervene in the Korean War?
US military intervention and economic uplifting are partly two different things.
I would limit this discussion to "how can a once poor country become an industrialized economy".
Since not all countries do this, the question is incredibly interesting. Perhaps I were too harsh but "US helped them" is not a satisfactory answer.
The fact that north korea is poor and south korea is not, is an interesting part of this puzzle as rare empirical sample of an area with roughly equivalent starting position well into modern history, and it should not be explained away by "because US duh".
I found "Why Nations Fail" an extremely interesting book pondering the puzzle of apparent differences in economic outcomes between countries and warmly suggest it to anyone interested in the topic. It's not really about state failure but about the bigger picture of what makes some states successfull (using the metrics of the author of course).
IMO more accurate to insinuate whether countries succeed or fail to develop is ultimately at the discretion of the (wealthy) hegemon. Especially more so in recent history with the viability of export led growth models and US / western bloc constituting huge % of global GDP, who can choose who to "uplift" via absorbing exports. Which is zero sum process, since demand is still finite, ergo Hegemon + co's policies / cooperation / coercion / indifference picks winners and losers, frequently explicitly via trade agreements / sanctions.
Also The moat and wall around modern tech is so high that industrializing / modernizing / moving up value chain can only be expediently done with active facilitation from other advanced economies. Even primary resource states that on paper should be able to extractive their way to wealth (not equality) still depends on increasingly advanced tech (and now increasingly export restricted) to even get goods to market in first place.
SK / NKR good example. NKR initially more developed than SKR via USSR until US with more resources and consumption seriously invested in SKR while absorbing their industrial policies' surplus. Meanwhile US kept NKR down via sanctions. Even post Korean war, NKR had fairly competent residual industrial prowess. Once country has the industrial institutional memory, it's able to quickly rebuild base to technically flourish but they still need permission to grow unhindered. Most of destroyed european powers postwar got back on feet with push from Marshall plan. Except NKR got pushed down.
"Why Nations Fail" comes from the Jared Diamond school of "My Model Fits My Data So It's Causal". It's a series of just-so stories, which discounts outliers (ex: Singapore, India, China). They take the scientific method (find examples to disprove your hypothesis) and invert it (find examples that match your hypothesis), which does nothing except provide people with the feeling of confidence that they've finally come across the Truth.
Yes, but like Guns, germs and steel it contains data even though the model is flawed, so is an interesting read in any case :)
I think the key point of the book is that unless there is a massive transfer of people from place to place the power of historical institutions can carry over generations and generations.
Is quality of institutions the one truth to explain it all? No, but I find it pretty hard to argue against that it is a relevant factor in historical evolution of polities regarding their quality of administration and general expectations of life outcomes of their populations.
My intent was not to sell the book as "the one truth", sorry.
But we're not talking about long lived, historic institutions; we're talking about South Korea, a republic formed out of the ashes of Japan's colonial project, and Ghana, a new country formed from the rapid retreat of Britain from its colonial empire.
You can talk about the legacy of the colonial institutions in the respective countries, but that has little to do with how hard South Korea worked in the 1950s, and more to do with how Japan and England administered their colonies, and how the transition from colony to state happened.
They're interesting books for sure, I remember enjoying reading them. But they're very selective in the facts they tell and how they tell them, and so they're almost worse than fiction in that respect.
Perhaps I should have not used the term "working hard".
I agree with what you wrote in general.
Why I find other explanations more worthwhile - which you opened up above - than "US helped them" is that is quite critical for countries to be able to advance economically. Based on several examples I don't think it can be explained simply by having a wealthy external sponsor. Hence it would be more important to focus on the things that made this thing happen, in order to understand if something could be replicated.
As an example of potentially replicable factors, for instance land reform seems to be a thing that has been at the roots of economic advancement of several countries. (Finland, South Korea). Of course, it can be screwed up as well (Zimbabwe).
I agree books that try to simplify things are dangerous, but really entertaining when they structure their thesis around solid facts. I don't think anyone has "the final truth" so an author that has at least tried to put forward an explicit thesis is always a worthwhile read - a clearly stated thesis is something the reader is then able to clearly refuse.
As for South Korea, did you look into the impact of US economic aid as well as industrial advisors, and compare it to the economic drag that the US induced via the collection of loan payments on Ghana?
Yes, similar to the assumption that the differences between North and South Korea can't be put down to "lazy people" and "hard working people" being magically seperated on opposite sides of the border.
seems im responding to the wrong user, my apologies.
the person who made the claim was thorougly roasted and edited their comment but you can ctrl + f 'ghana' and see the downstream context.
> Yes, similar to the assumption that the differences between North and South Korea can't be put down to "lazy people" and "hard working people" being magically seperated on opposite sides of the border.
> Your implication is that Ghana failed where South Korea succeeded because they didn't work hard enough.
It's not as simple as that. Building domestic industries takes time and you can never be sure about the outcome (especially if you believe everyone around you is corrupt as was believed in the countries behind the Iron Curtain). The easy money from foreign investors is hard to resist.
They have their debts forgiven on the basis of adopting certain policies (free market reform) that allows corporations from wealthy countries to set up shop and extract raw materials from the counter while avoiding giving back to the country via taxes. The profits get booked in Singapore or the Bahamas, and the stock price keeps going up.
Free market reforms do not forbid taxes on natural resource extraction. The U.S. has plenty of such taxes, and they score very highly on measures of economic freedom.
Of course. But we aren't talking about the States, we're talking about countries that are forced into return as a condition of loan forgiveness.
Some of those conditions involves avoiding any sort of export tariffs and using the declared value of goods to calculate profits (ex: if I spent 20$ extracting copper, and sold it for 20$ to an overseas company which totally isn't related to me, then the taxable profit in your country is 0$).
