Gergely Orosz’s Post

Whoa: we're in the middle of what is likely the largest global software-inflicted outage: Airports, hospitals, pharmacy services, flight operators, train services, TV broadcasters, supermarkets and more are down. What we know: (The below photo is from Sydney airport where most screens are the Windows Blue Screen of Death, and travel plans are disrupted thanks to this outage.) The outage has hit Windows machines that Crowdstrike for endpoint protection (antivirus, firewall, intrusion detection, encryption and application control.) Crowdstrike is a cybersecurity company valued at $80B, and the market leader for Windows endpoint protection, with around 22% market share. So 1 out of 5 businesses operating Windows machines use them. What seems to have happened is that Crowdstrike pushed an innocent-enough software update... to all Windows machines, globally, pretty much a the same time. Crowdstrike's software operates at the kernel level: and this update crashes Windows. Normally, when buggy code is pushed to production: you'd "just" revert this change, and push the previous version (or code that works correctly,) and when clients get this patch: they are restored. But not in this case: because these machines are non-functional. The fix - as advised by Crowdstrike - is manual and time consuming, and needs to be repeated for every single Windows machine impacted. The machine needs to be booted to safe mode, a file needs to be deleted, then the machine rebooted. What is unexplainable about this outage is how (and why?) did Crodstrike do a global release of a software change with no staged rollout (aka a rollout with canarying)? This makes no sense and no cybersecurity vendor with reasonable deployment practices would ever consider this. For all we know so far, this "global deploy" is more of a "YOLO deploy" (we covered approaches to ship to production in The Pragmatic Engineer, including YOLO deploys at https://lnkd.in/dsQzhQ7 ). YOLO deploys are fine when you don’t care much if a deploy goes wrong, and it’s easy enough to revert. A deploy that could take down the majority of your customers is not one to experiment with this approach with. To me, it's incomprehensible how staged rollout could have been sidestepped: how it’s not an enforced process for all deploys, large or small. The fallout of this outage will surely be noticeable at the global GDP level - and it will be very bad news for Crowdstrike's business, looking ahead (who wants a security vendor that crashes 100% of Windows machines their software runs on, when it should keep them secure?) My main takeaway is that staged rollouts/canaries should never be skipped, when your software is used to operate important or ciritcal infrastrucutre. (Image via Reddit: https://lnkd.in/ecfeP2Ys ) --- Follow me for insights on how the software engineering industry is evolving. Subscribe to The Pragmatic Engineer at https://lnkd.in/grXSBkVw for more in-depth analysis.

More than 200 developers shared their views on AI-assisted software development in a The Pragmatic Engineer survey. Findings on genuinely helpful use cases: 1. Massive help in completing specific projects The most-cited positive experience is that AI tooling is a huge help for completing projects, such as converting existing bash scripts that only worked on Linux, to cross-platform Python3 scripts that also work on Windows 2. Better testing leading to better code quality “We are seeing better use of test cases, better code quality, and better documentation. This is true both at the individual, and team-level.” – from a web developer with 7 YOE 3. Educational Respondents shared many cases of this, and LLMs do indeed seem well suited to helping tech workers learn new technologies and frameworks faster, in a more interactive way. 4. Easier to prototype and experiment Building a prototype version of a product is much easier, share several respondents. 5. Better documentation = better communication Examples mentioned included Boosting the quality of documentation, reviewing documents with AI tools, or finding relevant information with tools like Notion AI. And there are a few more (see in the screenshot below.) It's pretty clear that there a lot of areas where this tooling is already genuinely helpful. Of course, there are downsides (e.g. poor output, hallucinations, or devs over-trusting these tools.) But the upsides are hard to improve for anyone who does software engineering for a living. (All these findings were from the report "AI Tooling for Software Engineers in 2024: Reality Check" from The Pragmatic Engineer. The full report is available for subscribers, here: https://lnkd.in/eQwYbtbx --- Follow me for regular insights on how the software engineering industry is evolving. Subscribe to The Pragmatic Engineer at https://lnkd.in/grXSBkVw for more in-depth analysis of these.

This was such an interesting research we did with Elin Nilsson. The reality of AI-assisted software development is much more grounded than the hype would suggest.

History doesn't repeat: but sometimes it rhymes. An example of building a social network from the ground up, with a small team, with limited funding: Now, and a decade ago. Slide from my talk "What's Old is New Again." See the full talk here: https://lnkd.in/e5rSHtZw If you're interested in a deepdive on how Bluesky was built, read, the full (free) article in The Pragmatic Engineer at https://lnkd.in/eSgb79Mu

I do one conference talk per year. This year, it was at Craft Conference, on 30 May. If you missed it, you can rewatch the full recording and the Q&A, here: https://lnkd.in/e5rSHtZw In this talk, I aimed to figure out the biggest reasons for all the sudden changes that are happening across the tech industry. The drying up VC funding, Big Tech cutting jobs even as they reach record profits, fewer tech IPOs than before, and a tougher job market than we've seen in a good time. This is my best attempt to summarize months of analysis and research on this topic. I hope you find it useful! (And if you were at the conference, I hope you enjoyed it as much as I did. Thanks to the Craft Conference team for the superb event on this 10th, anniversary year of the conference!)

What a milestone: today, more than 700,000 of you read The Pragmatic Engineer!! This is such a hard-to-grasp number: it's already larger than the population of Boston / Copenhagen / Edinburgh (I used to live here!) Thank you for those of you reading. While I mostly write the twice-weekly articles, Elin Nilsson also helps with deeper and broader research (if you're a reader, you've likely seen even more detailed deep dives and more ambitious research than before). Occasionally, we also have guest writers sharing their hard-earned, first-hand experience. Details for those of you interested in potentially writing a guest post: https://lnkd.in/eKiDsB5k One interesting observation is how, in the beginning a few widely-shared articles used to result a large spike in signups (like How Big Tech runs project and the curious absence of Scrum: https://lnkd.in/eU_nwVbb) However, over time, these "spikes" became increasingly rare. Or, I can say that every article keeps bringing new readers through these regularly being shared. The last time I could find a smaller - but still visible - spike was for the deepdive on how OpenAI works, as shared by the ChatGPT team at https://lnkd.in/eBFTp-fu Finally, some interesting stats on how subscriber numbers changed over time:

How Intuit is doing layoffs maddening and inexplainable. One sentence after Intuit's CEO claims to treat people let go with "the compassion and respect they deserve," he states that most people let go are actually low performers. With a single sentence, Intuit calls the people losing their jobs “low performers” in full view of the public, in a message labeled “taking care of our people.” If that’s what taking care of people looks like at Intuit, then I fear for its current employees! This fully public message is incredible for a few reasons: 1. It could harm affected folks’ employment prospects. This is because most of them (more than half) have been publicly labeled “low performers” by their ex-employer. This means anyone considering hiring someone who just left Intuit can assume that there's a ~60% chance this person was a low performer, by Intuit standards. The 750 people let go for non-performance reasons likely have to live with this unfortunate public association as well! 2. What does Intuit’s CEO, Sasan Goodarzi gain from it? It makes Intuit’s leadership team look incompetent for having employed so many people who are apparently “not meeting expectations” (his words.) Shareholders could argue this was working against shareholders’ interest. 3. Let go for performance reasons, yet still offered generous severance? Intuit claims 1,050 people (of the 1,800 let go) did not meet performance expectations, but it’s still paying them all very generous severance of 16 weeks pay, plus two weeks for every year served, plus bonuses paid. Actual low performers rarely get such a package. Basically, the statement seems incoherent and baseless upon close inspection. By comparison, Meta let go of around 25% of staff in 6 months, but didn’t feel the need to say publicly that a single one of them was not meeting expectations. Anyone who has conducted layoffs knows that the group affected contains a mix of low performers, and those in the wrong team or organization at the wrong time. Companies do not comment on who is who, so hiring managers give people the benefit of the doubt and the chance of a fresh start elsewhere. This clumsy statement by Intuit’s CEO strips everyone of the benefit of that doubt, tarring them all as incompetent. Who’s being incompetent here? We previously did a deep dive on how to execute layoffs humanely in The Pragmatic Engineer (free article: https://lnkd.in/e9NKSMP3 ). Intuit seems to have used several of the practices mentioned in the article, like generous severance, additional insurance, and job placement services. It makes the needless lack of compassion in the CEO’s email stick out even more. -- This was an excerpt from today's The Pragmatic Engineer. Read the full analysis on these tone-deaf cuts at Intuit here: https://lnkd.in/eTrpnD3G

How can we expect software engineering practices to change, looking ahead? Here is my full keynote from Craft Conference, from a month ago: We've been seeing a lot of change in the software engineering industry. In the end of 2021, it was the hottest tech jobs market we've (probably) ever seen. And just a few months later, VC funding slowed down, companies did cuts, and there were no tech IPOs for 18 months. What happened, and why? I've been analyzing these events for a few months, and summarized my findings on Craft Conference, in a 45 minute talk. Here is the full recording: https://lnkd.in/eQ7Rh3BW And if you're interested in reading a summarized version, here is the article in today's The Pragmatic Engineer, at: https://lnkd.in/e5rSHtZw Now, answering the question on how I expect software engineering to change, looking ahead, observations from my talk: 1. More push to choose “boring” technology 2. Monoliths becoming more popular (at least at early-stage) 3. “Fullstack” to be more popular (+Typescript) 4. More responsibilities “shifting left” onto developers 5. Less reinventing the wheel Jump to this section in the video for my commentary on why these are likely to play out: https://lnkd.in/eMyCzbi5 What other changes would you expect to see, and why?

Look at what is out: the new book, Leading Effective Engineering Teams by Addy Osmani. I got to read this book before it came out, and wrote the foreword: The background on how this happened is I met Addy at the end of last year, when I was in the Bay Area. In-between him giving a tour of the Google offices, Addy shared that he's working on this book, which is a collection of many, many years of notes he's taken on helpful approaches. Here is an excerpt of my foreword - which also summarized what to expect from the book: "In the first part of this book, Addy walks through the "modern theory" of engineering management: research and mental model on this space that have stood the test of time for useful management practices - ones that work for tech teams. Many of these models are complementary, and some slightly contradict one another. In your own journey, you'll need to pick and choose the ones that are most helpful in your specific context - and you will likely find yourself changing philosophies as you change teams or companies. The craft of leadership is far more practical than theoretical, though. The second part of his book is where Addy shares his practical approach to leading engineering teams, and this is the part where Addy shares a condensed history of a decade of leading tech teams. (...) In addition to sharing his own, distilled experience, Addy allows us to peek at how effective managers at Google operate, as well as what effective leadership looks like in other environments like startups or large enterprises." I recommend this book for those in tech lead, or engineering leadership positions, looking for mental models, or examples of approaches that have worked for others. Congratulations on publishing, Addy!!

Are you a tech professional who does paid talks, and would do them in Amsterdam or the Netherlands? On topics relevant to software engineers or mobile engineers? Then comment or message me: (Update: a v1 of this page is live here: https://lnkd.in/edgc4MJ3 ) I am lucky enough to get regular enough requests to speak at company events, and (sometimes) conferences. Unfortunately, I rarely do such events (I speak at one conference per year.) However, I'd love to redirect such inquiries to those that do this. If this is you: please message me (you can send direct messages thanks to my open profile - this is preferred) or just comment on what topics you usually speak at, and at with at least one past talk recording at an event or conference (without a recording I cannot add you unfortunately - as companies / conferences unlikely to book a paid speaker without this reference). I'm building a list to share with organizers reaching out for speaking engagements, looking for paid speakers: which will hopefully help these organizers, as well as those looking for such speaking opportunities. Thank you! (I did something similar with coaching and mentoring for engineering managers and leaders. If you're looking for pointers there: I no longer do it, but here are plenty of people who do: https://lnkd.in/efus-zPe )