ERROR 3546 (HY000) at line 24: @@GLOBAL.GTID_PURGED cannot be changed: the added gtid set must not overlap with @@GLOBAL.GTID_EXECUTED As a MySQL 8.0 user, you may have encountered the following error message when trying to dump data from one database server and add that data to another server: "ERROR 3546 (HY000) at line 24: @@GLOBAL.GTID_PURGED cannot be changed: the added gtid set must not…
Currently, I'm doing a lot of work in Kubernetes, especially around operators. One operator, in particular, I am working on is aws-service-operator from awslabs . We ran into a bug with the default behavior around the dynamodb CR . There is a bug in this cloudformation template that defaults RangeAttributeTypes into Strings, when the operator supports strings, number, bytes. I know this is a bug,…
Working with Aurora MySQL I thought would be a breeze, but its subtle differences make me scratch my head. Thus I need to find out more about this and write a post :) What is Aurora? It's a mySQL wire protocol compatible storage management system that sits on top of mySQL and modifies some innodb internals. You can read about more of the architecture here : I think of it as a Proxy Storage Engine…
I like hackathons. Hackathons provide the freedom to build outside the process. The forced speed to deliver something to demo and the fun self-deprecation of "ooh this is really ugly/bad TODO don't do this." in the source/commit logs which tells a great story. Also, a great side effect; people are really interested in refactoring and fixing the code especially if the demo went well. So, I started…
I do a lot of data analysis lately, and I try to find answers to questions through data for my companies pressing questions. Let's look at the past year of 2017 and answer questions for people who like music. artist is the artist name track is the artist's track name list_date is which chart date the artist show up on the top200 streams is the number of plays following spotify specific rules Let's…
Programming in golang is fantastic. I find it fun, expressive and simple to build concurrent programs. Deploying a golang app from laptop to production is as hard now as when it was back when I was building Monolithic services. A great way to deploy nowadays is to deploy microservices in containers. Containers keep the environment between laptop and AWS Linux instance in sync since in essence the…
Building tables initially should not really require a lot of thought. What? I'm suggesting that when designing a table think of the Table as a spreadsheet. Yes. For instance, let's create a table that combines all social scores of a users' media in a single table. We will call this table platform_resources. What do we need to record the social score total of a single person? Who is this person?…
I have not been regular in blog posts as I've just been focused on everything. I got lazy. Well, that is over. At Shots Studios, a teen social network consisting of nearly 2M lines of code is no more. Shots is now a one-stop shop for select Creators. We are a Production Studio, Ad/Talent Agency, Talent Management Media company focused on creating timeless content. A 21st-century answer to getting…
I've been using Go for some time now (3 years) and I am constantly impressed with the language's ease of use. I originally started my career in C-Unix System Programming, then Java, then PHP and now I am rather language agnostic. Out of all the languages I know, go is the most fun and there is a strong community behind it. BoltDB is yet another NoSQL Key-Value store, designed to be embedded and I…
I just uploaded a quick tool that I think you will find useful if you need to consolidate, expand innodb databases if tablespaces are in use. This golang application will copy an entire innodb database from one server to another server via scp. innodb-tablespace-copy follows the algorithm described here . This golang application copies 4 tables in parallel after setting up the remote environment.…
The Shots APP we use the following technology to serve many millions of Photos, Videos and Cached Links. LAMP RedHat Enterprise 6 on the Front ends and DBs. Amazon Linux (Centos) on Elastic Search and Go servers Apache 2+ Percona 5.6 XTRADB with some minor custom stuff (sharded) PHP we have a little bit of Python, JAVA and a lot of GO! One of the current features which has wildly been successful…
I believe the future of SQL is to take a lot of primitives that are Computer Science fundamentals and add them as datatypes to expand on the allowed columns today. The idea is of the ilk of a merging of noSQL and SQL for solving problems to make it easier for a new person to develop. For instance, what would be awesome is a LIST type, where the list contains a distinct number of string items…
Golang is awesome. My 1st Language back in 1994 was C. Then the following year my Computer Science Dept. switched from C/Pascal to C++. I even like C++ but I like C more mainly because of nostalgia. Enter Go. The Syntax for me is a mix between JSON, and C. I love it. I've created 3 new servers all doing a ton of TPS. What I would like to share with you is some GO code to that handels websockets If…
Shots Architecture and how we handle extreme load spikes I invite you to come out and join me in a talk about the above heading. I will describe many things and walk through the cases of what technology is used, where, why and how. The event information is located here . I'll also touch on, how cost is reduced, how we handle celebrity's load when they promote and what's next to make the system…
If you have chosen to split your data across boxes, and architected your app to not query across boxes there is still a case where you will need to. Data mining, reports and data health checks require hitting all servers at some point. The case I am going over is sessions and figuring out the Session Length without taking averages of averages which is wrong. Let's assume you have a session table…
A friends table is the cornerstone of social applications. Its purpose is to define relationships and help answer the question what are my friends doing. Here is an example friend’s table: CREATE TABLE `friends` ( `user_id` bigint(20) unsigned NOT NULL, `friend_id` bigint(20) unsigned NOT NULL, `auto_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`user_id`,`friend_id`), KEY…
So, its been a long time since I contributed anything to my blog. That will end very soon. Things coming up is writing about the architecture of Shots, Shard optimizations, Data Organization and Grouping, Java, Golang and some cool other stuff. Also how to handle Justin Biebers traffic, which is INSANE. In the meantime if you live in the San Francisco California Bay Area, you want to work with the…
I'm really excited about Fabric which was recently announced. Everything it does has been a variety of scripts for me or manual tasks, but before I can integrate Fabric into my system I must know more about it. When dealing with live-data and moving servers around I still do things manually just because it makes me feel better to know that if data is lost, I was the cause for doing something dumb.…
MySQL has opened a lot of avenues and opportunities for me. I am the CTO of Shots and I got here because I was a mySQL DBA, who can code in a variety of languages, understand data access, layout and design fast backends that scales to well over 100 million users, manage a team, give back to the community and prove myself through constant good work. Plus I've made every single mistake, so I know…
Every company needs to see stats to understand how the application is performing, and how users are using the application(s). Typically a stat for most basic questions and even some advance questions can be summarized as "What is said event over time?". We call this EventTracker. To add to the complexity of generating stats, how do you get stat events from a DataCenter (DC) in Singapore, a DC in…
I will be in Budapest talking about Scale and Rapid Growth. I will start off with Flickr's Five minute conversation to take a direction on how to scale the backend to getting 90 million users in 3 weeks after going Viral. http://rampconf.com/main.html#schedule RAMP will also have live streaming broadcasted at TNW , HWSW and on USTREAM .
The PHP code that I've experienced in China so far is pretty good. I have been in some environments where the Code is horrendous-where variables are set in one file yet used in another file via a require_once. If that magic variable is not set everything would break with side-effect galore. This is not the case here for the China Team. This team is really good not to imply the other-one wasn't…
I see my role as enabling others. When I was a pure awesome DBA in the early 2000s I enabled developers and customers of a companies product by making mySQL fault-tolerant and fast. As I moved up the stack as an Architect while still holding onto my roots as a DBA-I kept my DBA discipline by enabling my team and company through all the knowledge I garnered. The first thing I identified in China…
I am currently in Beijing for a month as the VP of Technology for Fun+, a US/China based gaming company, spreading the joys of open-source I have an entire team to do benchmarks, study INNODB flushing, build new technologies, which I hope to open-source I will also post the results here. Our Stack is mostly on AWS with the following. HA Proxy Load Balances the Web Tier Web-Tier runs nginX and…
I recently went to MongoDB Days, a conference about everything MongoDB in SF. Starting my career as a Systems Programmer then Web Developer, MySQL DB[Admin|Architect], to Software|System Architecture I like to keep an open mind about new technology and trends. When you work with a lot of different languages, and technology you find out that it’s basically the same Science from about 40 years ago.…