Recent content on SkiToy Blog
One of blogs that I enjoy reading is The Daily WTF. Recently there was this article that highlighted some weird Geoinformatics WTF. What was interesting was that the comments on this post revolved around the performance of counting digits. Should somebody use Math.Log10 or do some basic counting. 1var digits = 0; 2while (num >= 1) { 3 digits++; 4 num /= 10; 5} I had to wonder, how many digits…
Software development and deployment is very much like an adventure game. Where you're faced with puzzles that might take you a day to solve, by picking up pieces and stumbling around in a dark room. Once you've figured it out, you could go back and solve the puzzle in 10 minutes. One of the manage challenges with serverless based architectures is that when you introduce a relational database…
Short post -- ended up switching from sqlx to db.v3 since I felt that the sqlx layer was more likely to be error prone. db.v3 example 1func (u *UserService) UpdateUser(user *model.User, args struct { 2 Name *string 3 Email *string 4 Password *string 5}) (*model.User, error) { 6 update := make(map[string]interface{}) 7 8 if args.Password != nil { 9 user.HashPassword(*args.Password) 10…
Performance of JavaScript object reduce Just a short post to look at assumptions. A few weeks ago was having one of those classic debates with a co-worker about the performance of object construction. Since with ES6 it is very easy to write: 1const obj = data.reduce((acc, [k, v]) => ({ ...acc, [k]: v }), {}); I had internally always made the assumption that JavaScript's runtime was making some…
I recently just fought my way through getting OpenVPN community edition running on our AWS VPC environment and wanted to share so that other can learn. There are a few key take aways and I'm just going to focus on the key elements. Past experience has shown that you don't want to use 192.168.X.X or 10.0.X.X as your VPN networks. They are frequently used by home routers and having people configure…
The classic problem in cloud engineering is that you have a bunch of API keys, secrets and passwords you need to have available to your code but not available to others. The ideal solution looks like: You have your secrets in Git so you can merge them into config files They're not availble to anybody who gets a copy of your repo Your machines can easily decrypt them You can edit them without a lot…
TL;DR version - Use UUIDs, but make them save space. Generating Unique IDs Do you key your whole system by database generated IDs that are autoincrementing integers. As you probably know that means that your APIs and endpoints are ready targets for somebody who wants to test lots of differt IDs against them. After all customer ID # 1000 demonstrates there is probably customer # 1001 and 1002.…
Writing a react and redux appliction is a great experience, we all have seen the vision of great applications. However every time you sit down to work on your application there are times that the boilerplate is killing you. In most Redux applications you start seeing it very quickly, since there are great initial demos of "Todo" or other classic applications. The question is how to create great…
This is really just a short post saying I discovered a really handy MacOS/Linux tool I hadn't seen before. Autoenv is a very handy tool for managing environments. Why would you need this? If you've ever done any quantity of Python or Go work you're probably using virtualenvwrapper or gvm to manage all of the different environemnts that you have. The challenge is that you want to quickly switch…
GraphQL DateTime Scalar Value Strings are not your friend when it comes to date time types. The problem is that with the naive definition you'll get the wrong value. For starters the naive definition looks like this: 1// GraphQL 2// type Object { 3// updated_at String 4// } 5// 6 7// The Go structure definition 8struct { 9 // .... 10 UpdatedAt time.Time `json:"updated_at" sql:"NOT NULL"` 11} 12…
This is some useful inisights I've gained, that are worth noting. The code examples are from a gist clone that I'm playing around with, it's easier to have a "known" problem and target to experiment with software than trying to do both at the same time. React What's clear is that with appropriate use of PropTypes, you really have a well structured "strict" environment for building components. It's…
These are really cliff notes for the next person, but quite useful. Was working with Spark in a local Docker using the very useful jupyter/docker-stacks, these are really nice to get a fully working Spark installation on your Mac without messing with lots of packages on your local machine. I’m now a big convert of working from Docker since I don’t have to keep on installing and installing things.…
If you ever looked at my github – public or private you can see that I have settled on a personal template for Angular applications. While I’m sure that there are things that are against some blog post of conventions, it works and gets lots of jobs done. After digging into the performance of our SPA at Tubular I found that we were spending 1.5 seconds rendering the results of an XHR (Ajax)…
One of the great challenges in software engineering is that you don’t know what the problem is until you’ve worked through it one. Sometimes this is the idea that you need to top down design a program which is a great start, but what’s interesting is when it comes to the language itself and what it makes easy and hard. My latest project was a making a Go version of the python s3cmd, there is now…
Like many pieces of code, there is always a story behind it. Your FizzBuzz solutions of the day – using operator precedence to get a result 1for num in xrange(1,101): 2 print 'Fizz' if not num % 3 else '' + 'Buzz' if not num % 5 else '' or num Or on one line. 1print '\n'.join('Fizz' if not i % 3 else '' + 'Buzz' if not i % 5 else '' or str(i) for i in range(1,101))
I do like delight, I really do. I was wondering last night about why saved searches have sucked for so long. Got thinking about some slides I’m putting together about quality. Part of it is that “move fast and break things” is counter productive to where we need to be, we need to move fast, but letting people internalize “break things” is part of why I think we’ve not put the attention into…
This post really documents the checkpoint in my experiments with GoLang and Frontend development. As some people might know that I’ve been playing around with application structure, languages and approaches for a while. My GoLang AngularJS and ES6 Todo App over at GitHub – it’s the reference for the rest of this post. File Layout Some quick to points, general directory structure: 1go-angular-es6/…
Was at a wonderful conference hosted by one of Tubular’s investors (FirstMark) where VPE’s, CTOs and key technical leads got together from a bunch of their investments to listen to speakers and have some good round table discussions. I spent my time participating in a round table on development processes, which evolved quickly from “life is good” to “here is a rough spot we’re having’. It’s really…
Some principles that should be followed when designing micro services. Accuracy – Fail hard or show full accurate results, never in between. Our goal is to provide precise numbers customers trust. Sending incomplete numbers because our systems aren’t healthy can’t be an excuse, we should follow the rule of “all or nothing”. Scalability – Predictable way to increase capacity. Growing only the…
Yesterday’s post where I figured out what it took to build thrift interfaces to attach to Cloudera Impala got a big improvement today. I combined my work with the hivething project that Derek Greentree wrote. It’s of course called impalathing over on github, it really does clean up the API. The big thing is that since this behind the scenes uses the ImpalaServer Thrift API everything is marshaled…
This started out as a quick project to see about taking a component of our service and migrating it from Python to Go. We’ve been talking about migrating services from the semi-monolithic version to more loosely coupled – the general idea is to move to Thrift oriented services. We have a core component of our system that uses Impala as a key backend, it’s a very stable service that could be…
In the begininning there was main() and that was good. But under the surface that has changed, it’s still main() but what really happens is dynamic linking, exit handling resources… We’re even throwing garbage collection in for good luck. But, it’s still main(). Why? If you do anything that’s isn’t linear programming you see that ioloop() is really main and you have boilerplate to set up…
As seen on a thread on Hacker News about Fizz Buzz and “interesting” functional ways to solve it. Realized that there are many ways to boil the ocean, but this feels like a nice compromise between data/program separation and language. Note this is using a “Bazz” variant of the FizzBuzz problem where Bazz is printed every 7 numbers. 1cases = [(3, "Fizz"), (5, "Buzz"), (7, "Bazz")] 2 3for i in…
One could say I’ve got too many projects with too much free time, but another way to look at things is that constant exploration can put a smile on your face. I’ve been working on a few projects: GearTracker (http://geartracker.com**)** This is my big project, which I **really need **some product marketing help on. I’ve gotten most of the infrastructure in place, but need somebody to come in an…
One of the big challenges is getting backbonejs and requirejs working together without jumping through lots of hoops. After reading quite a few blog posts, stackoverflow answers I finally came up with a simple canonical solution to the problem. The assumed directory layout is something like: 1static/ 2 app/ 3 config.js 4 main.js 5 vendor/ 6 ...third party stuff like backbone, jquery, etc.etc.…
It’s all about CoffeeScript (and quietly about Ruby). Why do people feel the need to remove punctuation? I came across the following code snippet: 1class TenFarms.View extends Backbone.View 2 constructor: -> 3 functions = _.difference _.functions(this), _.functions(TenFarms.View.prototype) 4 _.bindAll.apply _, [this, "render"].concat(functions) 5 super What I like and dislike – in line # order…
Was going to post this to stackoverflow, but the question was deleted before I posted. Turned out it was a fun exercise in writing a short program. 1def head(f_in, f_out, count=20): 2 all([not f_out.write(l2) for l2 in [line for line in f_in][:count]]) 3 4head(open('/etc/passwd'), open('/tmp/p', 'w'), count=2)
Sitting on hold with AT&T and what I’ve really noticed is besides being on the phone with them for an hour is how customer service is handled. They’ve done everything “right” however what’s interesting is to think about how the dialog has gone as it compares to all of my training via Vail/Northstar Ski School. The key words I keep on hearing are “I’m Sorry”, “I apologize for that”, “It won’t let…
Was looking around for a mail handler for Tornado, found it pretty amazing that it didn’t exist. Since I’ve only written on commercial SMTP server and have been playing with just about everything in Tornado Web at this point I figured it wasn’t too much work to whip one out. So, instead of using annoying little threads, here’s a fully async smtp client for Tornado. Nothing fancy, just gets the job…
I’m sure there’s a more general title for this kind of object, but the challenge came up and here’s what I put together. The basic idea is to have a python class that only recomputes something hard when one of the attributes is modified. It doesn’t make sense to recompute on every modification, nor does it make sense to recompute if it’s never read. In all cases, here’s the code sample.
We were working on some customer issues the other day and it another customer was having problems, you know double whammy. What was interesting is that a more senior person started talking about the customer about “they’re such a problem” and then talking about strange things they did in the past. As we dug into their problem, yes they were using an API in a very strange way (20 seconds of runtime…
Quick bit of background, I’ve used many programming languages over the years. Though in general I would bias myself to being a procedural/object guy rather than a functional programmer. Though I can say that I’ve done big projects in most of the popular languages at this point [C++, Java, JavaScript, Perl, PHP, Python, Ruby]. So, this post is more for trying to formalize a few of my thoughts,…
Looking at event IO frameworks and found some really good comments about the “ev” library, the challenge is that I would rather work in C++ than pure C – I like methods.. Found that the documentation is lacking for the C++ side of the library and needed to build a test harness. This is a very basic TCP Echo server using libev in c++. The C++ side is only using std::list and the ev side is playing…
Over the last year I’ve had an authentication library that I’ve used to slice and dice public services and like most things it’s collected more than it’s share of dust, cruft and other ugly appendages that you wonder if it’ll work then next time you use it. I’ve been hot and heavy over django (even if it’s embedded inside of Tornado) as a general framework for a while, it’s not broke don’t fix it…
Brief bio background – I’ve been doing email in some way shape or form since my address was: …!uoregon!chemstor!koblas or …{ames,decwrl,pyramid,wyse}! mips!koblas If that means something to you, then you probably have a clue that I’ve done this for a while. If it doesn’t here’s some other tidbits, I can write a sendmail.cf file from $* rules, and written SMTP servers, managed 200M++ mailboxes for…
Continue to poke away at looking at twitter data and what it means. One of the things I think about, because everybody else is doing it, is the idea of sentiment analysis. What’s interesting is that this posting reminded me of a Facebook “problem” — Sombody posts a note on facebook like “Broke my arm” and you end up clicking “Like” if you want to follow the conversation around this, but of course…
Code review time… In a conversation about URL shorteners and “Coke Rewards” realized that there was a case where I needed to be able to generate safe character strings that had high reliability for input back by human beings. The typical Base62 systems where there is ambiguity between (O, o and 0) make things hard (along with all of those upper vs. lower case cases). Here’s the quick module I put…
The project of the week, is something that I’ve been putting off for a very long time. Which is to get something running on Extra that’s more than just a nothing site. Part of the problem is that it’s a good domain name that I’ve had parked for a very long time, and it makes real $$ in parking revenue, which I would rather not endanger. FYI — The real purpose of this post is to document the code…
Part of this post is the gratuitous, gosh that was easy to integrate! Of course part is a small point that I would like developers to think about. First off here’s the code snippet, which owes it’s history to: Zendesk Remote Authentication with Django and Unicode Names and Zendesk remote authenticatin with Djnago (the original posting) 1def authorize(request): 2 if not request.user.is_active : 3…
I just read a post talking about the “Does your startup pass The Sleep Test“, in principal the idea is sound it’s a little simplistic. Fifteen years ago I worked as a consultant, the money was good but the problem I quickly realized is that fundamentally I was just trading time for money. We can all see how a construction worker trades time for money — $15/hour here I come I’ll hammer and pour…
If you have a few minutes and speak python & twisted, it would be useful to have an extra set of eyes on this section of code. The basic idea of this is to be a reconnecting thrift client, such that I can just write simple client.function(a,b,c) calls without having to worry about if there is or isn’t a client and it will queue reconnect as needed. 1from thrift.transport import TTwisted 2from…
One of those moments where an interview question turns into a research project, or is it really a bake off? The simple problem is demonstrate an algorithm to intersect two lists of numbers, fundamentally it’s a question about using modern interpreted languages and their associative array bits to make a simple intersection routine. However many languages support many different ways to do things.…
This is not a scientific nor rigorious test… But, here’s some interesting data for people to chew on. I’ve got a production site built using the Zend Framework and a beta site built using django. What’s interesting is the “Time spent downloading a page” graphs from Google Webmaster Tools. The Zend Framwork Graph — average speed is 458 ms with a min of 246 ms The django graph — you can see where I…
I’ve been working on I’ve been working on and still having the love hate with python and django. Areas that I would like see improved: Template Variables : I’m currently doing strange things like: 1<body id="{% block tmpl_id %}{% endblock %}" class="{% block tmpl_class %}{% endblock %}"> and then the included template is setting those variables. Which when you think about it is a bit off.. Footer…
I hate sessions, they’re evil. PHP is the worst offender since it’s built into the language and you end up with effective scalability limitations and turds in your temp file system. django isn’t much better since all of the cool admin functionality is built using the contrib.auth module which depends on sessions as well. Long ago in a galaxy far away I learned that such assumptions are bad, you…
I got a new laptop.. Yeah! Part of the idea of this new laptop is to run a bunch of vmware sessions to do some multi-server development. Lots of ram, fast CPU… now the challenge is getting the unix enviornment working. I’ve been using gentoo at work for a while — also on my own personall server — it’s nice, but it’s a totally pain to setup. Love: I can install just about any package and any…
Once again I'm faced with the challenge of how to make good decissions. It's not Life vs. Death, not anything meaningful in the scope of the world, but something amazingly simple — where to eat! Two easy examples where this has been a problem: Last Night — I'm "stuck" in a hotel in Washington DC with my timezones totally messed up, so at 8pm at night I want a quick dinner.
I hate silos… A long time ago a very smart engineer who worked for me pointed out that you can never under communicate. I’ve worked in many organizations where under communications was the norm, the classic example is somewhere between micro managment and lack of awareness. Example #1: You’re working on a project, there might be three or four groups of people involved… You’ve got Engineers,…
You don’t really want to know, but I’ve spent my morning fighting spam. It’s a periodic activity, makes me almost want to go back to the MailFrontier days when I could focus on these topics as a full time thing. Though on the advantage side I can use a lot more off the shelf components in my battles. If you recall I’ve been running sqlgrey on my box for a while, but over time more and more spam is…
RThe WSJ just did an article about the pointlessness of performance reviews it’s interesting to see that the posting I wronge a long time ago about Performance Reviews. Shared many of the same points: Performance doesn’t determine pay Objectivity is subjective It disrupts team work Managers should go coach some kids sports and really understand what teamwork is about… It’s both useful to the…