Several years ago, when I was more just the unofficial office geek, our email was acting up. Though we had Internet access as normal. At the time, email (Exchange) was hosted on-prem on our server. Anything server related, I’d contact our MSP to handle it. Which usually meant they’d simply reboot the server. Easy enough, but I was kinda afraid and hesitant to touch the server unless the MSP explicitly asked/told me to do something.

I reported it to our MSP, expecting a quick response, but nothing. Not even acknowledgment of the issue. This was already going on for like an hour, so I decided to take matters into my own hands. I went to the server, turned on the monitor…and it was black. Well, shit. Couldn’t even do a proper shutdown. So I emailed again, waited a bit, and again no response.

Well, if the server was being unresponsive, I figured a hard shutdown and reboot would be fine. I knew that’s what the MSP would (ask me to) do. What difference was them telling me to do it versus just me doing it on my own? I was going to fix email! I was going to be the hero! So I did it.

Server booted up, but after getting past the BIOS and other checks…it went back to black screen again. No Windows login. That’s not so terrible, since that was the status quo. Except now, people were also saying Internet all of a sudden stopped working. Oh shit.

Little did I know that the sever was acting as our DNS. So I essentially took down everything: email, Internet, even some server access (network drives, DBs). I was in a cold sweat now since we were pretty much dead in the water. I of course reached out AGAIN to the MSP, but AGAIN nothing. Wtf…

So I told my co-workers and bosses, expecting to get in some trouble for making things worse. Surprisingly, no one cared. A couple people decided to go home and work. Some people took super long lunches or chitchatted. Our receptionist was playing games on her computer. Our CEO had his feet up on his desk and was scrolling Facebook on his phone. Another C-suite decided to call it an early day.

Eventually, at basically the end of the day, the MSP reached out. They sent some remote commands to the server and it all started working again. Apparently, they were dealing with an actual catastrophe elsewhere: one of their clients’ offices had burned down so they were focused on BCDR over there all day.

So yeah, I took down our server for half a day. And no one cared, except me.

Older gentleman walked into the lobby of our office. None of us knew who he was or had seen him before. He looked confused and lost. Someone went over to ask if they could help him. He tried to but didn’t respond. Then fell over. Hit his head on a table on the way down. Was dead before the pandemics arrived.

We were all in shock. Poor guy was starting into a stroke when he walked in. Maybe even walked into our office to try getting help. But it was already too late.

I spent a decade working in insolvency.

When we were going into a business that had failed the question was “Are the idiots, criminals or both?”

One highlight:

A boat sales / marine business goes bust. When we arrive with the paper work and seize the place there are about a dozen new boats on the lot worth several million. We change the locks on the gates.

Arrive the next day, the gates have been busted open and several million in boats are now missing. We look up the addresses of the owners (one of them lives on acreage) and drive to their property…from the road we can see the boats stashed there. Really smart guys.

So we call the police. Someone inside notices use there and decides to flee with one of the boats, it is huge but they think they can get away.

We then have the slowest car chase in history as we calmly follow this guy towing a boat on a trailer down the road while talking to the cops to meet us.

Not my oh shit moment but certainly someone’s. Working in a call centre they sent out an example of a fraud email that was being sent out with our logo. It asked for all your personal information and credit card information.

Several individuals replied with all their details filled in. 3 of them replied all (entire call centre) with their details filled in.

I have a small PC I use for exposing a private PC to the wider web via nginx proxy. It had two accounts on it: mine, and one I called “remote” with some basic password I set up to forward the proxy connection.

One day, this machine started making 100% CPU noises, for several hours. Wtf? I check the processes and a Tor node had been setup and was transmitting gigabytes to some Russian IP.

My brain goes into panic mode, I kill the process, wipe the remote user, and eventually pull the Ethernet plug.

I wish I hadn’t wiped the user directory as I wanted to know what was being sent and where. Nonetheless the logs showed that several Russian IPs had been attempting an SSH brute force for literally months and one finally guessed “remote” and weak password I set for it.

I have decades of experience on Unix system, and I cringe having made such a rookie mistake.

Lesson learned: change the default SSH port to a transient port, have one dedicated SSH user with a non-standard username, and use auth-key entry only.

I still wonder what was being sent over that Tor node, and why it required all the CPU cores. My best guess is crypto mining, or it was used for a DDOS attack net somewhere.