So... about that downtime


#1

So, per usual, we wanted to be completely transparent with our community about what happens around these parts (i.e. our website) and give you a little bit of context of why The Pub™️ had been down for so long.

Our first notification that something was wrong was just after midnight, PDT time. We were given a pretty useless notice from Digital Ocean:

We then proceeded to not hear from them, at all, for almost 24 hours, as you can see from our repeated attempts to just get a living, breathing response from them:

The most frustrating thing is that their only suggestion was to console in and “fix” the issue but they had shut down our networking and droplet so we couldn’t actually do any of that! We couldn’t even ssh into our console to even take a look around!

:angry:

I eventually went to Twitter to publicly share my outrage because I had no other options, about 46 minutes ago:

… to which they replied very quickly and then gave us this response:

… zero help here either. I then gave them this response:

… in an attempt to document our “good historical behavior” and make sure (maybe…?) that they won’t do this shit again.

I’ve literally spent my entire day sitting in front of my computer waiting for a response… so that we could get our site back up online. Needless to say, our team is going to start thinking about alternatives because our disappointment is at an all-time high.

Sorry to have the site down for so long… what a complete and utter clusterfuck.


If you see any “phishing” attempts or any content that is “out of bounds” for our community guidelines please report it as SPAM immediately so we can assess it and make sure that no one gets hurt.

We all need to do our part around here… although, sometimes, that’s not even enough.


#2

Thanks for your hard work on this my man.


#3

Thanks John for keeping this awesome community running!


#4

fuck administering all this , a thought jon has daily


#5

Key take away is, companies tend to become a lot more responsive if you publicly (twitter) throw shit their way.


#6

in my opinion worthwhile to hire a part time dev ops that keeps your servers up and running, your time is better spent on other things and have the person be on 24/7 call, if something goes down he/she has to fix it

the incentive being that the person puts such a good infrastructure in place to avoid any break ups as nobody wants to wake up at 3 AM to fix an issue

this position is definitely one you should delegate fast at this point


#7

Glad it’s all sorted @john. Now time for you to enjoy your well earned holiday!