While every DBA knows they need to backup all their databases, not all may realize the importance of testing those backups. Performing backups is pointless, if you’re unable to restore them. I wanted to restore our backups on a regular basis, so I set up a process to test them automatically, on a nightly basis. Background Early on when I started working on the SQL Servers at Stack Overflow,…
If you follow me on Twitter , then you might have noticed that I’ve been fighting a lot of fires lately. Between high CPU (several times 1 , 2 ), blocking queries , a slow failover , and deadlocks there have been a ton of things that needed attention. Not all of these issues are interesting to people, but one of them might be…the deadlocks. I’m going to go into details about what…
I mentioned this in a few previous posts, but for for those who may have missed it or forgotten, here’s a quick refresher - we use Always On Availability Groups at Stack Overflow on all of our main production servers running the network of public Q&A sites, Jobs, and Stack Overflow for Teams . It’s a great way to implement disaster recovery for a SQL Server environment. Always On…
In my previous post , I listed out the tools I use with SQL Server. Some of the tools are SQL scripts that need to be deployed to each server. If you have 1-2 SQL Servers, manually deploying scripts might not be bad, but ideally you don’t want to manually deploy anything, so I wrote a little script that allow me to install the SQL scripts in my toolbox to any environment. Background When I…
I get asked a lot about the tools I use at Stack Overflow to monitor and work with our SQL Servers. I figured it might be helpful to others to make the list public. I’ll also do my best to keep it updated as things change. The current list of both free and 3rd party paid tools is below: Free Tools Opserver - Stack Exchange’s Monitoring System - I pretty much live in our instances of…
Initially, I wasn’t sure whether to write about this migration project, but when I randomly asked if people would be interested , the response was overwhelming. This was a long, kind of boring, very repetitive, and at times incredibly frustrating project, but I learned a lot, and maybe someone else will learn from this too. There may be far better ways to move this amount of data. In the path I…
This is a follow-up to my post about an issue with clustered columnstore , when upgrading from SQL Server 2017 to SQL Server 2019. After extensive testing and working with support, I wanted to share some information about a change in SQL Server 2019 that might impact others. Overview I suggest reading my other post first , it’ll only take a few minutes. I’ll wait… However, if…
In late March 2020, we upgraded our production SQL Servers to SQL Server 2019 with CU3. After finishing the upgrade, we hit an issue with clustered columnstore that we hadn’t experienced in the previous version of SQL, SQL Server 2017. The issue also wasn’t something we encountered during our extensive testing on various servers in development, which dated back to September 2019. The…
In my last post , I mentioned that I started the process of upgrading Stack Overflow to SQL Server 2019 . This week I tackled our first production servers and after upgrading, we hit a small issue aka a gotcha because we were using an old system view. Below is a recap of what I encountered. A Little Background The servers I upgraded were the three SQL Servers that run Stack Overflow for Teams .…
Recently, I kicked off a project to start moving us to SQL Server 2019 . During my initial review of our servers, I found quite a few (9 total) that were still running on Windows Server 2012 R2. This meant that I would need to upgrade the operating system and move us to SQL Server 2019. Having completed plenty of SQL Server upgrades, as well as operating system upgrades , I couldn’t possibly…
Last week at Stack Overflow we had an internal hack-a-thon, or as we call it, a make-a-thon. I was on the bug-bashing team, which is the team that attempts to fix smallish bugs we haven’t gotten around to fixing, due to other time-constraints. I was tagged to investigate a bug about duplicate badges being awarded because it looked to possibly be an easy fix in SQL. At first glance it looked simple…
This month’s T-SQL Tuesday is hosted by Jon Shaulis , who has asked that we talk about impostor syndrome . This subject is close to my heart, as I have dealt with it throughout my career. I didn’t study computer science or engineering in college. I never even took a computer class. I sort of just fell into working as a web developer because I had someone tell me they wouldn’t hire me…
In late June 2019, June 26th to be exact, we experienced an outage on Stack Overflow for about 11 minutes. It’s not unusual that we had an outage. They happen. Not often, but they do still happen. This one, however, was a little different because it was caused by a maintenance job that was running on our primary SQL Server for Stack Overflow. The job that caused it was something I’d…
I’ve been working with SQL Server for a long time, and have always wanted to attend PASS. For one reason of another, with numerous employers, I hadn’t been able to go. The answer was always no. <insert sad face> The biggest blocker was always the cost. The conference is expensive. Seattle is expensive. It has always been impossible to get my employer to foot the bill for it. This year I was…
Warning: This post is long. While working through this massive server upgrade/migration process, tears were shed, many cuss words were said, along with a general feeling of frustration, which ultimately culminated into extreme happiness once the migration was completed. The scale and complexity of the implementation factor into the length of this post, and I’ll share my thought process on how this…
This post has been rattling around in my head for months, and with SQL Server 2019 on the horizon, I figured I’d finally put my thoughts down, especially since I know we might hit some of the same issues when we upgrade. A quick background, when I became the DBA at Stack Overflow, we had various versions of SQL Server in place — the majority were at some level of SQL Server 2016, each with a…
Almost a year ago, in November 2017, I moved back into a more technical role at Stack Overflow and became a Database Administrator. During this time, I have wanted to relaunch my blog, as it’s been over three years since I really wrote anything (yes, I’m terrible). Well, that time has finally come. After spending a few weeks, looking at various options to replace Wordpress, I finally…
Well it’s been an interesting couple months and there are quite a few changes going on in my life, but before I get into what’s coming up, I feel that I need to give a bit of an overview of my journey thus far. I never thought about becoming a developer. When I was growing up, I loved school (yeah, I’m weird), I loved math and science (even weirder), I was going to be a doctor,…
A few weeks ago, we were running into severe disk space and memory issues on our development servers at work. Our set-up is a bit odd, we have 3 servers - one for the transactions, one for the web interface, and the final one for reporting. Using transactional replication we have databases that can exist on all 3 servers. Yes, it’s can be a real nightmare to maintain, but anyone who works…
A few weeks bit over a month ago, I took the plunge and attended my very first SQL Saturday, SQL Saturday #370 in Phoenix . I’ve never had a chance to go to any of the local user group events or another SQL Saturday due to scheduling issues, etc. but this year I reshuffled things so I could take part. I’m really glad that I did, I truly learned a lot of stuff and I also realized I know…