I'm not so sure that it was node.js that revolutionized paypal... seems like the gains were from switching to a more agile process (and adding some supporting tools). A lot of the fixes seem pretty agnostic to the actual platform that they switched to.
Note that all those open source modules that they talk about aren't released yet. krakenjs.com doesn't even resolve (for me at least).
Its will be a little saying too much, to say node.js revolutionized paypal. But speaking as a matter of fact programming languages do play significant role in the overall productivity of the teams.
I wouldn't be too surprised if they are saying using Java was making it difficult to turnaround issues. The problem isn't Java the language itself, but the processes involved in taking changes to production with these languages are generally so much ceremony large turnaround times become the norm.
Add to this issues like fixing build failures, test failures, code reviews that span meta aspects of code(config changes, xmls etc), running around to get libraries standardized per your company standards(getting it into your local maven repo etc); and much more. These sort of things really slow you down. With these if you are tending to multiples issues, or if you have dependencies on your colleagues- I wouldn't be surprised with the 6-month delays mentioned in the article are actually true.
Better said, since there is no compilation step, so you can push non working js through and it will look "green".
Java isn't defined by a process, that's sad most people don't seem to understand that. Look at projects like dropwizard, happily throwing away what they dont like.
I think you are close to the mark. There are ways to work in Java/JSP environment that is pretty quick cycle time. But I will say that our experience at PayPal is the code is more terse, less files, npm so much better than maven for managing dependencies, coupled with the fact that our javascript templating allows portability between server & client.
The Shift-Refresh style of programming is hard to overstate how much that creates a different mindset.
And to my point in the talk, the blending of prototyping and production tech stacks is also huge for being able to quickly iterate with users.
There is some brief lip-service paid to using the same language on the front-end and the back-end, but mostly you're right - they fixed processes, not technologies.
I can assure you that I had to attack multiple fronts. Process is a big part of it (I brought in LeanUX methodology), but I can tell you (and painfully so) that if we had not changed the technology we would not be moving at anywhere near the speed we can now.
Mind you that what you see on the PayPal sites is not all updated. This is a testament to just how deep the technical stack has been.
Other colleagues have been attacking the services layer and they are doing just as much change in that layer as my team has done on the app/frontend layer.
a new programming language is a perfect excuse to drop old ways of doing things, that way there's no legacy code pulling you back into old ways of doing things.
Unfortunately in practice that's not always the case. People are often so set in their ways they'll still try to do direct ports of "the old ways" even if the new language/platform offer easier/more elegant solutions. It can be very frustrating.
You raise a great point. Another big challenge we have had is re-educating developers. This has been helped by finding the really sharp engineers internally and hiring new engineers that bring in the right DNA.
I think we are getting to a tipping point, but it is a lot of work obviously.
Technology is not a silver bullet. Nor process. At the end of the day it is about your people. Technology & process can set them free.
I am doing a similar thing in a organization that has been at least as dis-functional, I know exactly what you're talking about. In fact, everything that you have presented and chatted about here resonates so well with me that I'd be interested in getting in touch with you. Case studies like yours are preciously rare on HN. My address is in my profile if you're open for it.
That's seriously going to reduce adoption of their open source modules. They're getting a lot of hits from this PR-piece, it's too bad they can't use any of that momentum for kraken or lusca...
PayPal is still running on something they call XPT and their admin tools are built with something called Maxcode (Max Levchin wrote it).
As of 2 years ago they revolutionized CSS by having it being generated from Java because their fellowship of the ring wanted control over which properties you can use and how. Everything they do from a tech perspective has been to allow them to hire cheap labor while restricting which HTML/CSS they have access to. It's probably a good business strategy (like making sure they don't get classified as a bank) but it fosters an abysmal tech culture.
I have no way to prove the points above you'd just have to trust that I worked there as a UI dev.
I worked in PayPal's mobile team for a few months last year (acqui-hire). I can confirm that as of two years ago, everything you said is true, but as of last year the groundwork was laid for everything in this article being true today.
I used the PayPal Node stuff in its infancy (i.e., before anything was working). At that point it was a thirdhand tarball being passed around. Even in its preliminary state it was about 500% less shitty than using the old stuff (Sparta, JSP, etc).
I am glad to see that the (small, relatively insulated) team could deliver. I expect this migration will make many PP engineers obsolete, but I don't have a problem with that.
Interesting article, but I'm in the UK and Paypal still looks and works exactly the same for me as it has for years. It's very slow, rather ugly, and very inconsistent - the splash page seems styled differently to the login page seems styled differently to the help page... I could go on.
I know this is mostly superficial stuff, but it doesn't contribute to a good experience at all.
Slightly OT, but why is it suddenly all the rage for sites to have persistent nav-bars that follow you down the page? The one on this blog is obnoxiously huge, and obscures a lot of content. What's wrong with scrolling to the top of the page?
I put a persistent nav bar at the top of my latest site (http://bosscheck.me) so that potential customers are always looking at a "book now" option. Shallow and obvious, perhaps, but my assumption is that this is optimal for conversion.
Your navbar is much nicer and less intrusive though, I like it. Have you thought about differentiating the booking button from the rest of your navbar elements?
Yes, I've thought about that exact idea probably more than it deserves, but I'm trying to err on the side of not appearing spammy while simultaneously keeping the "book now" button always in easy reach.
I genuinely wonder how it is possible for paypal to have such terrible design for so long? Everyone knows its an issue, and it seems like a couple 100k in design work would clear up the issue for the next 4-5 years.
The move toward Node as a platform will make things like cosmetic redesigns much easier to accomplish. I don't think many design firms could, or would want to, work within PayPal's older stack, which requires knowledge of not only Java but dozens of internal services.
Keep in mind that PayPal is a gut-wrenching edge case of internationalization and localization. If you're an Israeli person transferring Thai baht while you're in Chile, the website must account for that while respecting your user preferences and all local, national, and international law. It's a gross problem no matter how you handle it, and it utterly prevents drive-by redesigns.
How come? Wouldn't the company that acquired you want you to succeed so their acquisition wasn't wasted? If you're point is that the founder has 'succeeded' and won't try as hard his new boss will fire him.
I would imagine main objective would be not taking risks because you already earned your big time money. Redesigning GUI and switching technology are risks.
"Head count: Node.js. Typically they are seeing anywhere from ⅓ to 1/10 the number of developers needed on the new stack vs the old stack."
That's ridiculously implausible. I know node is more concise, but 300-1000% productivity increase doesn't come from less typing. Even if we assume they had a lengthy build/restart cycle on some J2EE web container, that could only plausibly increase output by 10-20%. Sounds like they had an absolutely shitty application architecture and he convinced them to do something both reasonable and modern with the former being the biggest improvement.
First of all, J2EE web container lengthy restart is a huge overhead alone, and in case of iterating over a feature, I can easily see a 30-50% difference.
But second of all, from the article: "You didn’t write html, css or javascript – you wrote css in java, you wrote html in java and you wrote javascript in java." Can you imagine developing that?
I've been involved with writing Javascript in Java using GWT and it has poor developer productivity, especially as the project gets to anything beyond a medium size. At a medium size, compilation times get well above 5 minutes, and I can't even imagine a large app the size of a PayPal.
Imagine having to figure out the right XML/XSLT to transform into the right Javascript.
So it's not at all ridiculously implausible. At previous jobs, I worked with some really arcane stacks and you can't imagine the number of unnecessary layers that get in the way of good old GSD.
Servlet container restarts don't have to have a huge overhead for development (not sure about using a full j2ee container). If you use SBT[1] plus the the xsbt-web-plugin[2] then restarts are just as fast as they are for node projects.
J2EE-containers can take a while to start indeed (for large projects), but in most cases they can be tuned to restart in less than 10sec - at least during development.
Using GWT you don't "write Javascript in Java", the java-code is written in java and then compiled to optimized, cross-browser, javascript. Compile-times can be quite long - but again, during development you will use the hosted-mode (or even the new super-dev-mode) version, which compiles on the fly. Full java-to-javascript compilation is typically only needed when new versions are deployed to stage/prod.
Poor separation of concerns can be easily solved in J2EE and build/restart can be solved with something like JRebel or just using a lighter contain like Jetty for dev. I'm just saying Java to Node is not going to give everyone a huge increase in productivity. Programming in itself doesn't even solve most of the worst bottlenecks.
I agree that Java to Node is not a magic solution. Clearly, like posters below have mention, it's the old architecture/stack that's the problem. J2EE containers have come a long way now, I can restart my Tomcat/Jetty containers in a flash, unlike Weblogic/Websphere of the days past (shudder).
I mean, I still think Java and the JVM are great for many many a use case, not trying to pine for node here.
I agree. I am actually not a Java hater :-) In fact I have built 4 distinct Java/JSP frameworks since 1996.
You can really create some very nice frameworks with Java using some of what you say below. The Java framework at PayPal made a few mistakes (in my opinion) that made the environment harder to work in than it should have been.
But I started with needed to rapidly iterate, move between prototype & production easily, take advantage of asynchronous nature of node for our web apps to scale better, simplify the engineering team makeup (not have multiple types of engineers). These all led to deciding to move forward with node.
Oh, and let's face it, recruiting is a little easier with nodejs.
No, it's also about a ridiculously bad old architecture to a better new architecture.
There is nothing about Java that stops you writing your CSS in CSS and your JavaScript in JavaScript. In fact, that's the standard way to do it. It seems that at PayPal, something went wrong early on that led them to build some byzantine homebrew web framework as a rod for their own backs. I have no idea why anyone would do that, but it seems to be a remarkably common thing to do.
Paypal is ancient. It's likely they developed their web framework before any open source java web frameworks became popular. Paypal existed around 1998, Java Struts 1.0 came out in 1999. Struts 1.0 is probably one of the big reasons many people flocked to PHP.
They probably made their own web framework because nothing good existed at the time.
I read that another way. The whole article sounded like they replaced engineering with hacking/prototyping and measuring the productivity of a prototype vs the productivity of battle-hardened code. There is an admission that they have no specs and i also got the impression that they have no tests. I seriously doubt that at the claimed turnaround times they have a proper code review process. It sounded to me like they were getting false productivity and eventually would program their way back into a corner.
Having been on a team where the pendulum has swung both ways I know firsthand that you can be too agile and end up regretting it later.
I wondered about that too--I didn't see any mention of any software testing. Definitely user/outcome testing, but no unit tests or functional tests were mentioned.
A little off topic but that giant fixed header that obscures 30% of the vertical space I have available on my 15" macbook pro retina so I can see a lot of whitespace. This is terrible and made me not want to read the post.
PayPal has so much legacy UI cruft even their own employees can't figure it out. I was talking to a support rep on the phone last week who was telling me to click something on my home screen that wasn't there. It wasn't there because she couldn't figure out which of the 4 home screen interfaces my account was on -- having a business account with PP Pro subscription means I have extra reporting menus but none of the updated design of the past few years. Going from their public site to my logged-in profile page exposes me to 3 totally different page layouts.
In most cases position: fixed is unnecessary. But it's especially unnecessary when your navbar takes up a quarter of the height of the screen. Please reconsider this approach, it's really not needed for users to see your sitemap at all times while reading a technical article.
Who cares about Paypal? they have screwed so many people and companies one has to wonder how they get away with it and still are in business
Bitcoin (or bitcoinlike replacement) is the future and the most interesting thing to come out (Technologywise) since email, bittorent or social networks
PayPal is to worldwide online payments what Microsoft was to OSes in the nineties. Bitcoin is Linux. It's free, and by some measures better, yet nobody wants to use it.
They're essentially a monopolist.
Nobody else has what they have: access to nearly every country, worldwide.
Except that the analogy is off, given that paypal deals in established currency that you don't first have to generate or buy (via a convoluted process that involves handing over your ID etc to trading websites, or selling post-order drugs).
PayPal processed $44 billion in 2013 Q3 alone. It's miles ahead of everyone else right now. That can obviously change, but when a the slow giant gets a little quicker on it's feet, everyone notices.
I don't know much about bitcoin. If I pay you with that, and you fail to ship me the product, can I get a chargeback? If not, I think that answers your question.
See the comment above comparing Paypal vs Bitcoin to Microsoft vs Linux
in the long run (10 years) either bitcoin or bitcoinlike replacement is going to trash Paypal in same manner that Linux, android etc is trashing Microsoft now, yes Microsoft still make billions but bitcoin is growing rapidly, many people wrote off Linux years ago too for very similar sounding reasons.
Yes, but your question wasn't a long run question, was it? It was a teenage-angst filled question: "Who cares about Paypal?" $44,000,000,000 in payments processed in three months - I think that is another way of saying, "Most of the people who pay for things online care about PayPal."
We've considered doing the same thing at our startup. I'd love to hear more about your experience. Did you simply build a node.js UI server that made calls to a C# service that did the database accessing and business logic? Or did you cut out C# all together and access the database and perform business logic directly in node? We're worried that re-writing major components in node would slow us down too much at the moment, but I'd love to hear your thoughts on the productivity boosts you found.
Another "What - how does that work?" question here. What you've said just isn't making sense - you're replacing a server side programming language with a client-side JS framework. There's obviously other "things" going on - I'd be interested to hear much more. C# is such a great language but ASP.NET is so slow...
"Lines of code: In general they have seen code size shrink by factor of 3-5 by moving from the Java stack to the Node stack. Even the number of files shrunk by a similar factor."
How do they went to 3-5 times less files without recurring to several classes, controllers, etc in the same file?
I'm guessing that a good chunk of that came from either reducing the actual complexity of their UI, or from simplifying horribly-fragmented old code. And I would guess more of the former than the latter.
We actually had the luxury of comparing parallel efforts. While Node was being readied for production we were writing our new consumer/wallet app with the PayPal Java framework (based on Spring). Meanwhile we started writing the Node version. Once we got to a certain point the Java team insisted that they stop the Java development and start on Node with the node team. At this point we had two versions of the app built in two stacks and thus could compare code size, developer productivity & scale.
Of course, if we compare the new node apps vs the old PayPal code... well that would be a poor comparison as they really were old and monolithic.
Cool - it's great to hear that you guys actually got to put it to a head-to-head comparison with the old ways.
How heavily have you customized Spring, versus how much did you use various Node modules out of the box (versus having to adapt things to your backend)? Do you think part of the productivity gap comes from the large learning curve of your PayPal framework or just from Spring in general?
Not from Spring. Spring is a solid reasonable framework. Unfortunately the modifications were enough to cause confusion, the Windows-biased support (not good support for Mac) for the dev environment, forcing people to use Eclipse, having lots of server-side solutions for stuff that is already readily available on github as a simple JS library and on and on.
I still feel that even set against Spring or GWT or any Java server-side UI framework, Node plays much nicer and really gets you to the Shift-Refresh type of programming that is perfect for app development.
We used (and use) node modules completely out of the box. We add additional modules to augment. We are really adamant not to follow the roll your own mindset. We have had numerous times that we threw away our work in lieu of a new npm module that did what we were doing as good or better than ours. Why spend time doing that when we have so much technical debt to overcome to get to the state of really innovating.
It could also means the functionalities of those files had been moved over to libraries. NodeJS does have an impressive amount of libraries, and dare I say probably more than Java.
If they had that many developers I am not sure why they invented such a complex alternative to a private npm repo with the full replication. I mean it only took half a day to set up and then two days to load. Heh.
But I glad they did it. I really feel like someone should take some code like the core if Kappa and integrate it into npm so you don't have to wait two days or whatever to replicate a giant database just so you can have private modules. I know that will make a few peoples business model go away but I hope that's not what is stopping it from happening.
This is interesting. We researched Node.js a bit and found many arguments on StackOverflow against using it for apps that require heavy DB interaction and/or non-real-time apps. Also, there seemed to be many people pointing to the single-process model as risky. Is this no longer the case or have these issues been addressed through configuration? I would love to give Node.js a try again.
Hmmm... That is kind of humorous, given that Nicholas was a consultant for us early last year as we started this effort. His team was also at PayPal today (from Box). That article was influenced to some degree by me giving a talk at Box HQ on this very topic.
Also, maybe you misunderstand what we are doing. We are using nodejs for the web app layer. But it could be used for services (like Walmart Labs is doing). You really have to determine the context, the people, the technology, the amount of risk, and so on to decide.
But I do agree, no one should see nodejs as a silver bullet.
PayPayl is licensed and regulated as a money transmitter in multiple US states and as a bank in the EU¹. The idea that states would allow it to transfer the amounts it does without being regulated is funny, but not realistic (and not true).
I read through the article, hard to believe they will be processing billions of dollars on top of JavaScript. Maybe I've been wrong and now it's time to dump Java and go all script.
Article really confirms my suspicion that node.js is pretty much the new perl. (Although modern perl is still an excellent choice for new projects if you need small team high impact stuff).
it's interesting how some established startups (if paypal could even be referred to as that anymore) have been getting gains by leveraging node.js into existing stacks.
To me, it's more interesting to see the willingness of large companies like PayPal to leverage "new" technologies in any way. The common narrative of large companies suggests otherwise.
Note that all those open source modules that they talk about aren't released yet. krakenjs.com doesn't even resolve (for me at least).