• ClumsyTomato@lemmy.sdf.org
    0·
    6 months ago

    I worked for a loooong time in a medium size development company (about 200 developers, mostly doing large web portals). My team was some kind of central DevOps in charge of architectures, cloud, technology stacks… we were ALWAYS involved in EVERY deployment, and we were directly in full charge of the big ones.

    After many years of constant work alongside the DEV/QA teams my team had gotten REALLY good doing deployments (we mostly sailed on each of them, since all was well tested, prepared and automated), and the project leaders simply trusted us. In the scarce occasions we said “sorry, this is not ready for prod” they knew it was true and didn’t pressured us. And our customers were happy, since needing a rollback was EXTREMELY rare.

    One of the most important things we managed to agreed with all the team leaders:

    1. Fridays are read only.
    2. No, that doesn’t means we all can go home: Friday is now “Documentation Day”.
    3. Of course, if shit hits the fan, we are ALWAYS ready to deploy fixes.

    I think in about 10 years I only had one call on a weekend.

      • ClumsyTomato@lemmy.sdf.orgEnglish
        0·
        6 months ago

        That “wet dream” was only possible after several years of hard work of dozens of people, and only because many other small pieces fitted in their proper place.

        Hi, manager! You said you want environments for your developers without needing my intervention in every step? Of course, here you have infra and config automation: this is how you can create (and backup! and restore!! and destroy!!!) DEV, TEST, QA, PRE and DEMO environments, all ready with your specific stack and versions and code and data. And this is how much will it cost to you, and this is how you define a budget limit in case something gets out of control. Everything is repeatable and 100% reproducible in seconds, so please do not hesitate to test and test and test. (And no, sorry, I won’t let you touch PRO on your own, because that can cost a lot of money and we need to keep proper security).

        So, you are asking me if we have heard about code versioning? Yes, of course! Here is a proper git structure, with predefined branches, segregated groups and permissions, and strict (and automated) revision requirements for every PR. I own the organization, you own the repo, QA owns the tests, and your developers own their branches and are self sufficient. Oh, and please remember we freeze the main branch 48h before the deployment, and time only begins counting after all the automated tests have passed and QA has given their final approval! No cheating!!

        Oh, you have a picky customer who wants a guaranteed instant recovery in case THE WORST happens? Here you are, a highly available blue/green deployment, so you can deploy the new version without touching the old and only switch when everyone gives the final OK. And please remember to warn them it does cost DOUBLE the money!

        Believe me, is not a wet dream, is just a lot of initial effort and A LOT of trust and confidence in the work of those around you.

        And you have no idea how satisfying was begin work a Tuesday at 9AM sending a message “Hi, we are starting deployment in PRO” and then less than 5 minutes later reply saying “Hi, all is finished and checked OK from all parts, thanks to everyone and see you next week”.

    • The Giant Korean@lemmy.worldEnglish
      0·
      6 months ago

      Yes. Tuesday is the superior day.

      You’ve gotten over the jarring shock of Monday, and nothing is happening in your life on a Tuesday night after COB.

  • fluckx@lemmy.world
    0·
    6 months ago

    Lets get one thing straight.

    This is rarely ever the developer and more a business stakeholder forcing you to push the Friday deploy button.

    I’ve had somebody in the business escalating to my team lead, head of development and CIO because i flat out refused to deploy something on Friday at 16h.

    So no. This is not the developer making a hard choice. There should be somebody coercing or forcing him to push the deploy button.

  • Jo Miran@lemmy.ml
    0·
    6 months ago

    We never make minor production changes on Fridays or right before holidays. It is always a bad move because if you get into trouble your odds of not being able to teach the necessary resources are greatly increased.

    Major production changes are only done during a scheduled downtime which is planned well in advance to make sure everyone is available, including third party vendors.

    • sabreW4K3@lazysoci.al
      0·
      6 months ago

      Shout-out to you because every Friday/Saturday when I check the Play Store, it seems like every app has an update.

    • PapstJL4U@lemmy.worldEnglish
      0·
      6 months ago

      and on the other side is game dev, where it is kind of expected to get content before weekend…and the reason they are paid well, right? right? >_>