I agree, as a long time Business Intelligence developer I‘m still confused and astounded with all the tooling and bits and pieces seemingly necessary to create analytics/dashboards with open source tools.
For years I used a proprietary solution like Qlik Sense for the whole journey from data extraction to a finished dashboard (mostly on-prem). Going from raw data to a finished dashboard is a matter of days (not weeks/month) with one single tool (and maybe some scripts for supporting tasks). There is some „scripting“ involved for loading and transforming data, but if you already understand data models (and maybe have some sql experience) it is very easy. The Dashboard creation itself does not need any coding at all.just drag and drop and some formulas like sum(amount).
But this a standalone tool and it is hard to integrate it into your own piece of software. From my experience, software developers have a much more complicated view on data handling. Often this is just the complexity of their use cases, sometimes it is just a lack of knowledge of data preparation for analytics use cases.
Another part which complicates stuff greatly is the focus on use-cases involving cloud storage and doing all the transformations on distributed systems.
And it is often not clear what amount of data we are talking about and if it is realtime (streaming) data or not. There is a big difference in the possible approaches, if you have 6h hours to prepare data or if it has to be refreshed every second (or when new data arrives etc).
Long story short: Yes it is complicated to grasp. There is also a big difference if you use the data for normal analytics use cases in a company (mostly read only data models) or if you use the data in a (big tech) product.
I would suggest to start simple by looking into a „query engine“ to extract some data from somewhere and then doing some transformations with pandas/polars/cubejs for basic understanding. You will need some schedulers and orchestration on the way forward. But this will be dependent on the real use cases and environment you are in.
I would argue that stuff like Iceberg is really aimed at Data Platform Engineers, not BI analysts. Companies I've worked with in the past have 10-15 people on a Platform team that work directly with stuff like this, to offer analysts and data scientists a view into the company's data.
The picture you are painting is way too dark. And does not give a realistic picture.
A lot of what you say is true for doctors in their first 5-10 years into their career, when employed in a hospital.
This not true for doctors which reached a certain level like „oberarzt“ and above.
This is especially not true for doctors with their own „office“ (business).
Yeah people may cry, but normally it is very hard to bring a doctor to justice even when there are quite obvious mistakes or misconduct. They are very well protected, suing a doctor not seldom takes 10 years from start to verdict, with a lot of legal costs involved.
And last but not least, it is a very secure profession. You must be really really stupid to end up jobless. So you have 5-10 years with a „ok“ salary compared to the power you invest. And 20-30 Years with a very good to exceptional salary, especially when compared to the broader population.
Surgeon here, in private practice. Agree with the article - all the stressors he mentions are typical of both residents and staff physicians. The hour crunch for me is better post residency but overall the stress is unchanged. Probably higher after training with the added responsibilities & risks.
My sense is that the field developed in the era of independent/private practice, where the grueling hours worked was justified by high pay and minimal bureaucratic/administrative burden. Add decades of stagnant/falling pay plus death by a thousand administrative cuts and the profession no longer justifies the difficult working conditions as convincingly. Some practices are still good, others terrible. Look at the rate of physician turnover to see which is which.
Oh and the “provider” discussion is worth paying attention to. Your doctor has this calculus worked out - years & energy invested, work environment & income expected, then the only viable option in your city is to be employed by a large hospital system (because hospitals get paid at least double for the same work, outcome is as expected.) But wait there’s more: you are now called “provider” by your large hospital employer who hires 2x NP employees to do the “same” work as you and pay half. Guess what direction the pricing pressure is going. In the future expect few MDs to stay in primary care because the system does not support that path. Specialty training is the future for MDs who invest time, energy, & money to excel in their field.
I’m so glad I’m not alone in noticing this “provider” bs. Peel back the creepy Orwellian doublespeak and all you find is cynical ploy to save money by creating a false equivalence of doctors’ work with non-doctors. The health care industry is just the latest home of the money-grubbing vampire squid of finance. Sickens me.
Not sure how much you are actually in the business here, but almost everything you write is incorrect for literally every single doctor we know (cca 50, variously close, everything from GP to heart/neurosurgeons). You are clearly talking about German part, I talk about French, but still same general rules do apply.
Its trivial to sue a doctor, my wife, on her effin' first night shift in the country here got involved death of a patient and got into court case that took 6 months of court hearings to resolve. Not her fault, wasn't her patient even, but she still had to spent ridiculous amount of time for it outside work to get finally cleared.
Her colleague at this moment is getting sued, almost immediately after situation, for overlooking a cancer, when markers from test twice were non-conclusive (I don't/can't go into details, its a very complex case). Suing is very common here, its just that in case error can't be proved on their side, they have cca decent (and expensive) legal insurance. If they don't, license revocation, life-destroying fines, or even jail are on the table. Cases like this are common. This is very common for GPs with their own practice too, since they see more patients than some specialists.
Also not sure why you degrade other's people mental issues when under semi-constant decades of pressure from all sides. "Yeah people may cry" - this ain't how mental issues and burnout should be acknowledged. Please show some respect they properly deserve, you clearly are an outsider to profession and I sense some envy in between your lines. If its that bad with your life, go and start medicine studies, schools are open for anybody of any age and public schools here are free.
Last part - yes unemployment isn't generally high among doctors willing to work, but ie check canton Geneve now - no new GP licenses are granted (as = 0), and old folks are retiring fast. People are desperate to get a GP, I have colleagues begging me to find somebody via my wife for them, new doctors need to travel 2-3h every day to other cantons to find work, and some are properly desperate. As IT guy, I don't know a single capable colleague who has even similar employment issues, companies are always hiring good seniors, and there are tons of companies needing good IT folks left and right.
Jup, and if your employer doesn't stop you or even expects such behavior that is called a bad work culture.
Sure, I would also do long programming sessions on my own things every now and then, but those happen because I am in the flow and it feels like more work to come back into it the other day. But those are my own projects. I don't owe this to my employer. If they want that they could offer me double my current salary and even then I would insist on only doing it when it makes sense to me.
In my case my superior comes by in my office or sees I made a commit and tells me I should stop working (I tend to forget to look at the time if I am in a flow).
But I also live in Europe and my superiors want me to come in with a fresh mind, instead of burning myself for things that should be normal work.
But in reality there is not an equal distribution between these 3 groups. And there is a high probability that the user base is not as limited as in your pseudo factual simplification. (journalists come to mind for example etc. pp)
Don’t you think it is strange to post this here, when the expected content about „data engineering“ is not (yet) there?
The only content i could find is an introduction to rust which is in my opinion not necessary, you could just reference the existing resources. When I saw that a category like „First data pipeline“ is empty, i closed the page.
Maybe I’m a bit old school, but it seems to me, that you care more about subscribers than delivering useful content.
I am not the person who created the content, I just stumbled upon it while doing research. Personally, I think the idea in the title alone is even enough to warrant posting this for the commentary.
The article saysthat they had 8000 before layoffs started... What are all these people doing at an exchange? Maybe they work with orders on paper in the background? :D
They operate in 100 countries, have 30m users, support trading in 360 coins PLUS a load of derivatives on top and process $15bn of trades per day. And they are not just "an exchange". They are also custody, clearing and settlements. Frankly I am amazed they are only 8,000.
Idk...throwing around big numbers, implying confidence does not convince me. Also your numbers are mostly not neccessarily related to the amount of workforce needed.
For example my blog/website operates theoretically in all countries on the world ...still it's only me needed. I think you get the point.
I think a measure like active users is more related to the size than number of countries and coins/tokens ...
But it is true I focused a too much on the platform itself etc. They have a lot stuff going on around it for legal, kyc, regulations, Marketing etc ... The operation of the platform itself is probably not the biggest part of their business anymore.
But coinbase for example, does it with less than 5000 ... But probably not so easy to compare
