My current job.
Many SQL servers use scripts that run as domain administrator. With the password hard coded in.
Several of the various servers are very old. W2K, 2003, 2008. SQL server, too.
Several of the users run reports via rdp to the SQL server - logging in as domain admin.
Codebase is a mashup of various dev tools: .net, asp, Java, etc.
Fax server software vendor has been out of business for a decade. Server hardware is 20 years old. Telecom for fax is a channelized PRI carrying POTS - and multiport modem cards.
About a 3rd of the ethernet runs in the office have failed.
Office pcs are static IP. Boss says that’s more secure.
We process money to/from the Fed.
The last line was the least surprising after hearing all that.
Ah the sweet security that is a static IP lol
A company making signage and signal lights for road construction, with 15 employees. Their former IT guy had switched all of their PC’s to Linux for ideological reasons and to save money.
Then they found out that they had a long term contract for an accounting software that housed all their customer and billing data, only ran on Windows and required a server-client model.So they hauled in the boss’s private laptop which ran Windows 7, and installed both the server role, database and client software on it. When his employees needed to access the accounting software, the boss had to stop what he was doing and grant them full access to his laptop via teamviewer. When the boss’s laptop was off or he was on vacation, there was no way to access any price info, customer contact info, or financial data (This was during Covid when everyone was working from home).
The laptop was set up to back up (using Windows 7’s integrated backup tool) to an external drive which wasn’t attached and no one remembered ever existing.
The Linux server (which was actually a gaming PC) was running and attached to an MCU when my company surveyed their infrastructure, but no one (including the former IT guy) knew the correct root password, and we never found out what it was even doing.
This is surely the worst of all.
I had another customer who wrote down all passwords to everything in an unprotected Excel sheet and uploaded it to OneDrive, with the comany’s single, shared Microsoft login being admin@companyname.onmicrosoft.com . The password was companyname in lower case letters with no 2FA.
And another one who had their server in a shared office that was inside the owner’s privately owned apartment building. During the Christmas holidays, the owner turned off the heating for the office to save money, which crashed the server when temperatures dropped below freezing inside the room.
Small business IT is wild. It’s one of the main reasons I quit my job at that small MSP and switched to a larger company’s internal IT.
Oh God please stop.
Wow. Just wow
I know it’s a bit of a silly example, but in the public school in Korea where I taught for a while, teachers would write their Windows passwords on post-its and stick them to the monitors. Haha!
I can vouch for this. My coworker has his password post it noted to his monitor now
Office Depot. They are still using IBM machines from the 90s with receipt printers the size of a shoebox.
My current company has a script that runs and deletes files that haven’t been modified for two years. It doesn’t take into account any other factors, just modification date. It doesn’t aks for confirmation and doesn’t even inform the end user about.
That’s the worst foresight I think I’ve ever heard of, you might as well make that 3 months if you’re just going to trash thousands of labor hours on those files.
You should write a script to touch all the files before their script runs.
Thought about it but I use modification date for sorting to have the stuff I’ve recently worked on on top. I instead keep the files where the script isn’t looking. The downside is they are not backed up so I might potentially lose them but if I don’t do that, then I’ll lose them for sure…
Have a script that makes a copy of all files that are 1.9 years old into a separate folder.
Create a series of folders labeled with dates. Every day copy the useful stuff to the new folder. Every night change modified dates on all files to current date.
Have you…called attention to this at all?
You don’t actually have to set all the modification dates to now, you can pick any other timestamp you want. So to preserve the order of the files, you could just have the script sort the list of files by date, then update the modification date of the oldest file to some fixed time ago, the second-oldest to a bit later, and so on.
You could even exclude recently-edited files because the real modification dates are probably more relevant for those. For example, if you only process files older than 3 months, and update those starting from "6 months old"1, that just leaves remembering to run that script at least once a year or so. Just pick a date and put a recurring reminder in your calendar.
1: I picked 6 months there to leave some slack, in case you procrastinate your next run or it’s otherwise delayed because you’re out sick or on vacation or something.
Change the date on all the files by scaling to fit the oldest file. Scale to 1 year as a safe maximum age. So if the oldest file is 1.5 years old, scale all files to be t/1.5 duration prior to now.
That sounds like a lawyers dream… “can’t provide it if it doesn’t exist” … now granted, if they got a subpoena they’d have to save it going forward, but before then, if their not bound by something that forces data retention, the less random data laying around the better.
Put all your files in a single zip file. No compression. Since Windows handles zip files like folders, you can work like normal. And the zip file will always have a recent time stamp.
What industry are you in. This could be compliance for different reasons. Retention is a very specific thing that should be documented in policies.
I know financial institutions that specifically do not want data just hanging around. This limits liability and exposure if there is a breach, and makes any litigation much easier if the data doesn’t exist by policy.
Should they be more choosy on what gets deleted, yea probably. But I understand why it’s there.
This want their fault and I feel for them.
Working in telecommunications and we get a call from a customer that they are moving premises in 3 months time. They want us to check the place out of we can get a service in there. All goes well, time table for install goes out and we rock up the day after they get the keys, only to find the previous tenant took ALL the copper cables with them. Now when I say all, I mean ALL. Data, telephony, and power were stripped out all the way back to the dmarc point of the building.
They were fucking pissed. The lease for the existing place ran out about 3 weeks after the got the keys to the new place.
I was hired to implement a CRM for an insurance company to replace their current system.
Of course no documentation or functional requirements where provided, so part of the task was to reverse engineer the current CRM.
After a couple of hours trying to find some type of backend code on the server, I discovered the bizarre truth: every bit of business logic was implemented in Stored Procedures and Triggers on a MSSQL database. There were no frontend code either on the server, users have some ActiveX controls installed locally that accessed the DB.
every bit of business logic was implemented in Stored Procedures and Triggers on a MSSQL database.
Provided the SP’s are managed in a CVS and pushed to the DB via migrations (similar to Entity Framework), this is simply laborious to the devs. Provided the business rules are simple to express in SQL, this can actually be more performant than doing it in code (although it rarely ever is that simple).
There were no frontend code either on the server, users have some ActiveX controls installed locally that accessed the DB.
This is the actual WTF for me.
There was no version control at all. The company that provided the software was really shady, and the implementation was so bad that the (only) developer was there full time fixing the code and data directly in production when the users had any issue (which was several times a day).
This was 5 years ago at a usd200mil multinational…
The email system was pop3. There were no document backups. There was no collaboration tools. There was no IT security. You could basically copy company data out and no one would ever find out. The MS Office license was bought singly. Ahem!
I have worked as a lead developer for a major print shop with about 100 employees. The entire order workflow for all branches was shoehorned into one order management system that was initially hacked together for one or two users. It was built on a then already ancient OpenERP system and it had a PHP and smarty frontend for the actual order management. All was hosted on one old debian box which was a VM on a Windows server.
At some point in time, MT decided to slap a web shop onto this system, which was part of the main code base. User data were saved into the same database with plain text passwords. That was convenient for the support people: if somebody forgot their password, you could call support and they would read you your password over the phone.
Another thing that made my hair raise in fear, was that for every single order, any working file was retained indefinitely, even in the light of the then-looming GDPR laws. This amounted of terabytes of data, much of it very private.
I worked at the main branch. When a person walked in, there was a desktop computer at the counter. No password protection, an order management screen open by default. People could just walk in and start viewing orders at will. I am not sure whether they did, but we did push MT to at least have manadatory password protection on their PCs.
Probably not as bad as some of the other examples here, but the company I currently work for has its 10tb shared drives backing up to a server that’s right next to it in the same cabinet. Those two servers, plus all of the networking hardware and a variety of ancillary devices are all plugged in to one socket via a bunch of extension cords.
Yes, the boss has been told to get it sorted, but he’s the kind of older guy who doesn’t give a shit.
TLDR - Boomer problems
Yeah, exactly that.
So I back up my own shit on my own external drive, just in case.
Happened to us, they put the backups on a different device, away from the servers, but still in the same premise. Cryptolocker locked everything on the network, including the backups. No off-site backups.
Source control relying on 2 folders: dev/test and production. Git was prohibited due to the possibility of seeing the history of who did what. Which made sense in a twisted way since a previous boss used to single out people who made mistakes and harras them
Just share a git user, come on. Have everyone check in under the same name “development” or whatever, but no version control whatsoever?
When you lift up the red flag and there are more red flags underneath.
In every small problem are many large problems that want to come out.
The recent Falcon cock up?
I actually disagree. I only know a little of Crowdstrike internals but they’re a company that is trying to do the whole DevOps/agile bullshit the right way. Unfortunately they’ve undermined the practice for the rest of us working for dinosaurs trying to catch up.
Crowdstrike’s problem wasn’t a quality escape; that’ll always happen eventually. Their problem was with their rollout processes.
There shouldn’t have been a circumstance where the same code got delivered worldwide in the course of a day. If you were sane you’d canary it at first and exponentially increase rollout from thereon. Any initial error should have meant a halt in further deployments.
Canary isn’t the only way to solve it, by the way. Just an easy fix in this case.
Unfortunately what is likely to happen is that they’ll find the poor engineer that made the commit that led to this and fire them as a scapegoat, instead of inspecting the culture and processes that allowed it to happen and fixing those.
People fuck up and make mistakes. If you don’t expect that in your business you’re doing it wrong. This is not to say you shouldn’t trust people; if they work at your company you should assume they are competent and have good intent. The guard rails are there to prevent mistakes, not bad/incompetent actors. It just so happens they often catch the latter.
Freight shipping company still running on a custom AS400 application for dispatch. Time is stored as a 4-digit number, which means the nightside dispachers have their own mini Y2K bug to deal with every midnight.
On one hand, hooray for computer-enforced fucking-off every night. On the other hand, the only people who could fix an entry stuck in the system because of this were on dayside.
Apparently, this actually isn’t uncommon in the industry, which I think is probably the worst part to me.
Hehe I was in global shipping IT, we had some ooooold Solaris systems that handled freight halting data flows. Windows Server 98 servers that handled data for very large shippers. Every daylight savings time change something would break.
IBM
/thread
One of my friends quit IBM not too long ago. From the stories he’s told me, it sounded like almost everyone there spends all of their time and energy blamimg others for failed projects and unhappy clients.
Exactly this. I don’t know anyone in the IT industry that would willingly buy IBM. They’re either locked in due to legacy reasons or government projects where most of them are incompetent.
Thankfully it’s changing, but slowly.
A behavioral health company with 25 iPads deployed to field employees as patient data collection devices all signed into the same iCloud account instead of using MDM or anything.
They all had the same screen lock PINs and though most of the data was stored in a cloud based service protected by a login, that app’s password was saved by default.
