Last week I sent all Yupdates customers the following email: While subscriptions have grown, Yupdates hasn't seen the interest and growth rate needed to sustain a bootstrapped business. I tried many avenues to raise awareness and adjusted to great feedback (thank you). But the slow growth rate combined with personal financial surprises this year have made it unwise to proceed. Yupdates will…
There are many options for periodic work, and they're often stack specific. If you're using Sidekiq, there's a plugin . If you're running on Heroku, there's an add-on . If you're running on Kubernetes, there's a feature . There are many tutorials about hooking into localhost cron daemons. These might be good options for your system. But as the number of tasks…
There's a lot of discussion about what IDs to use in your databases, but the human/developer aspect of IDs often falls by the wayside. Your ID format choice is quite important for database performance. I'm not arguing against that. These days, the contenders are usually integers, UUID v4, ULID , or the new UUID v7 . There are more options and aspects of the decision beyond…
Staying on top of a lot of projects on GitHub is frustrating. I've had my inbox blow up with low-information notifications, sifted through endless issues and PRs, and felt dumb after discovering something happened days ago that I needed to know or weigh in on. One of the issues is context. As a senior developer or manager, you acquire hundreds of contexts you're operating under, each…
This post will show you how to implement rate limiting with Redis using two techniques: fixed time windows and sliding time windows. I provide code samples, explain how they work, and discuss the advantages and disadvantages of each technique. The examples use Python, but the real discussion is around how to use Redis, and the methods are straightforward to transfer; there are good Redis clients…
When designing a product or subsystem, you need to look at the cost model. You ask and answer questions like: Where will this be deployed? What are the fixed costs of deploying there? What controls do we have, especially for max costs? What are the predicted, marginal costs as usage increases? Does the value we produce (money or other types of value) scale with that usage enough to justify those…
You can end up doing a lot of legwork ensuring DynamoDB (DDB) queries will work for a use case. But it can give you huge advantages. I'm going to use an example from Yupdates. One of the design tenets is to be fast for all feeds at all timestamps . It needs to feel instant no matter what feed you're looking through and no matter how many items in the history you want to explore. DDB is…
In this Reddit post today, Christian Selig, the developer of the popular Reddit client Apollo , predicts his monthly API fees will soon be $20 million (US) per year. A death sentence. The new Twitter API rates are equally punishing. Even the lower-cost "API Pro" tier prices most of us out at 1M tweets per month for $5000 (US) per month. Ironically, many of the apps that could afford that tier shut…
RSS feed parsing is pretty messy, and that's well-known. People/libraries regularly insert random tags, bizarre timestamp formats, and various 'illegal' things in their feeds. And they're constantly changing item publication timestamps (personal pet peeve). You're pretty much dead in the water if you don't embrace Postel's law : "Be conservative in what you…
Today I'm moving Yupdates into public preview (!). The 9+ month private preview period was exciting and rewarding (and, at times, frustrating). I received great feedback that changed the course of the service, and I'm grateful for everyone and their generosity in dealing with rough edges. Yupdates is a new take on feeds. It's usable by anyone but geared toward people in the software…
This question crops up from time to time, especially when there's a big AWS anniversary or around re:Invent. It's that time again, and the new AWS CEO (Adam Selipsky) said in his keynote that first came S3 and then EC2. And people responded with comments about that being wrong, etc. What is that all about? And why do I care enough to write a blog post? Great question! No idea 😏.…
As a beginner to deploying Rust code on AWS Lambda, I hit a speed bump using Rusoto to call AWS APIs during the function run. Calling AWS APIs from Lambda functions is of course not required, just typical. Why is it taking so long to call anything? Especially with both the client and server in AWS datacenters, it should be very fast. Calls to DynamoDB (DDB) were taking well over 100ms, not the…
Fast integration tests are the best. Being in the flow with unit tests is good but I love having that same rapid iteration cycle with integration tests. And I think it's better to exercise as much real code as possible in them as possible. Why rely on mocking, especially for your own services, if you can run the real thing quickly? So I built a mechanism for launching groups of arbitrary…
In the system I'm building there are a significant amount of SQS messages that need to be processed asynchronously and Lambda is a really good fit for that. It is perfect for spiky workloads. You can control max parallelism easily. And it now has a native integration where Lambda functions are invoked with pending SQS messages without any extra work on your part. Lambda's great. I think…
I resigned from AWS to start a business. Exhilarating. No, I didn't hate it. While managing multiple engineering teams at AWS is certainly demanding, the good outweighed the bad. An L7 SDM in Seattle is a great job. I was fortunate and privileged. Each year, I learned more and more about distributed systems, operational excellence, and the business side of software. My management and coding…
There's a pattern in many software development blogs, talks, tweets, etc. They often lack context and they're received poorly and discussions end up with people talking clear past each other. Understanding situational factors makes it far easier to have a meaningful discussion about technology choices. The answers to every one of these questions can affect your designs and technology…
There are always things to learn about distributed systems, especially how they can turn against you. Companies that publish postmortems are doing us a great favor, there is a jackpot of system design and operations knowledge to be gleaned from studying as many of these as you can get your hands on. I’ve collected [now over 350] outage and security related postmortems in this Pinboard feed .…
I raised my eyebrows at one statement Google is making about Google Compute Engine : Deploy your applications on an infrastructure that provides consistent performance. Benefit from a system designed from the ground up to provide strong isolation of users’ actions. Use our consistently fast and dependable core technologies, such as our persistent block device, to store and host your data. While…
Web application developers, in particular ones on a small team, are usually focused on the next feature or getting MVPs out, not security. When security does come up, the focus is usually mitigating direct webapp attacks. We rely on Django or RoR ‘s mechanisms for XSS/CSRF protection and password hashing. We turn to App Engine , Heroku , or traditional hosts for DDoS protection. And so on.…
I am no longer adding anything to gridvm.org , that was probably obvious at some point last year. A young child and increasing work responsibilities will do that to you. But a bigger issue with that site than lack of free time was that the topic did not feel right anymore. One reason was that the term “cloud computing” solidified and it was hard to tell what “virtualization and grid computing”…
Cool to see the mainstream press mention Nimbus! Number Crunching Made Easy .
Just ran across an “interesting” definition of Nimbus: “Client-side cloud-computing interface to Globus-enabled TeraPort cluster at U of C” … in these slides: http://eucalyptus.cs.ucsb.edu/documents/eucalyptus-slides-wolski-cloud_expo_apr08.ppt There is a Nimbus client (which can be replaced by an Amazon’s EC2 client), true. But most of Nimbus is server side software. Teraport…
Press Release: Nimbus and Cloud Computing Meet STAR Production Demands We’ve been running self-configuring 100+ node clusters on EC2 since 2007, but I would be remiss if I did not link to this announcement.
Announcement : Perspectives on Distributed Computing: Thirty People, Four User Types, and the Distributed Computing User Experience is available for download . “This report chronicles and analyzes the responses of thirty users to questions about using the Globus Toolkit – starting with summaries of results and conclusions but also including very detailed appendices and even transcripts of the…
The Nimbus TP2.2 release provides a standalone context broker that can be used across Nimbus and EC2 clouds and continues our work on EC2 compatibility with the introduction of EC2 metadata server. In addition, the release contains new documentation and bug fixes. See the changelog for all the details.
Quoting from http://www.phreedom.org/research/rogue-ca/ “As a proof of concept we executed a practical attack scenario and successfully created a rogue Certification Authority (CA) certificate trusted by all common web browsers. This certificate allows us to impersonate any website on the Internet, including banking and e-commerce sites secured using the HTTPS protocol.” I…
It was a good feeling to pause for a moment and put a user quotes page together for Nimbus. We’ve worked hard to make Nimbus usable and useful — but the best is yet to come! (Update: that page is offline sorry)
Some interesting things in the new Xen road map . [via Stephen Spector ]
Besides the good stuff added to Nimbus, this release also introduces something called the AutoContainer which allows you to set up a Globus Java web services environment, from scratch and with security working, within about a minute (requires Linux/OSX and Java 1.5+). The main new features provided in this release are tools facilitating the deployment, configuration and management of clouds.…
All this Windows cloud news afoot. Make sure to read this post on the GoGrid blog: Windows in the Cloud? Been there, done that! GoGrid has already almost a year of proven experience providing Windows Server 2003/2008 to end users – we are also a Microsoft Gold Certified Partner.
Tom Scavo writes on gridshib-user : We are pleased to announce GridShib for Globus Toolkit v0.6.1: http://gridshib.globus.org/downloads/gridshib-gt-0_6_1-src.tar.gz http://gridshib.globus.org/downloads/gridshib-gt-0_6_1-src.zip Please visit the GridShib for GT home page for an introduction and links to software and documentation:…
For those who haven’t heard about the Xen 0wning Trilogy , make sure to check that out here and here . In a followup post to some apparent misinformation being spread ( Microsoft executive “rebuts” our research! ), I was surprised by this comment: Interestingly, if Mr. Riley only attended our Xen 0wning Trilogy at Black Hat, then he would notice that we were actually very positive about Hyper-V.…
EDIT: see http://gist.github.com/7263 (tip via HubbuH)
An interesting project I ran across, it started in 2004. From http://vcl.ncsu.edu/ : The Virtual Computing Lab (VCL) is a remote access service that allows you to reserve a computer with a desired set of applications for yourself, and remotely access it over the Internet. You can use all your favorite applications such as Matlab, Maple, SAS, Solidworks, and many others. Linux,…
I arrived in Las Palmas de Gran Canaria yesterday. Here to give a presentation of the pilot paper at Euro-Par and represent Nimbus on a panel at VHPC . I was jetlagged, so I tried to read a little bit, learning about some of the deeper corners of the fantastic Spring framework: Here you can see the northern end of this beautiful city.
It would be interesting to have a lot of input on this survey if you have some time, it’s only a few questions. From: Stephen Childs Subject: Survey on use of virtualization in production grids Hi all, Apologies for the slightly off-topic post. I am giving presentations soon on the use of virtualization in production grids and was thinking it might be nice to have some data to present! I have set…
It’s thrilling to organize things better. From the Nimbus features page: There are currently two supported remote protocol sets: WSRF based: protocol implementation in longstanding use by previous workspace services and clients including the cloud-client. EC2 based: clients written for EC2 can be used with Nimbus installations. For more information, see What is the EC2 frontend? These protocols…
See the announcement : new strong internal interfaces and a new remote protocol implementation (compatible with EC2 clients) that can run alongside the WSRF based ones.
Go Ask ALICE , the iSGTW image of the week. (Funny headline, see Go_Ask_Alice ). Check out some screenshots here (invalid now sorry) of Nimbus resources invovled in supporting this experiment. It’s a small part of things as you can see from the scope of the grid but exciting nonetheless. The AliEn based virtual cluster is now “ one-click ” and can be launched anywhere running a workspace cloud…
Chris Samuel posted this job opportunity for a job with VPAC in Australia. If you know anyone that may be interested in the job, please pass this announcement along. It seems like people reading here would have a good probability of knowing someone that may be interested and who also has the experience they are looking for. Thanks From…
I like the end of the recent S3 downtime explanation : “Though we’re proud of our operational performance in operating Amazon S3 for almost 2.5 years, we know that any downtime is unacceptable and we won’t be satisfied until performance is statistically indistinguishable from perfect.”
VWS RSS feed: http://www.nimbusproject.org/news/rss/
A lot of developments with the workspace service and science clouds recently! The cluster technology lets you bootstrap generic images into new network and security contexts on the fly. We built a sample cluster on top of the technology that lets you create the cluster and be immediately ready to submit jobs to a Torque cluster fronted by GRAM and GridFTP that use a newly created self-signed…
Congratulations everyone! On behalf of the Globus Toolkit development team I am pleased to announce that a new stable release of the Globus Toolkit is now available. GT4.2.0 contains an upgrade to the web services specifications used by the toolkit as well as new features in all services. New users are encouraged to use the 4.2.0 release. Existing users may wish to evaluate the new software while…
Cloud Computing and Its Applications Also see Ian Foster’s blog entry
From http://www.gridtoday.com/grid/2370981.html rPath Enables Cloud Computing for DoE, CERN RALEIGH, N.C., June 4 — rPath, whose unique technology simplifies application distribution and management through virtual appliances, today announced that the U.S. Department of Energy (DOE) and the European Organization for Nuclear Research (CERN) have been using rBuilder to deliver…
EUCALYPTUS – Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems – is an open-source software infrastructure for implementing “cloud computing” on clusters. The current interface to EUCALYPTUS is compatible with Amazon’s EC2 interface, but the infrastructure is designed to support multiple client-side interfaces. May 14th, 2008: EUCALYPTUS is publically demonstrated…
I am happy to announce the TP 1.3.2 release — the “cloudkit release” of the Workspace Service. You can download the new release from: http://workspace.globus.org/downloads/index.html As many of you have probably noticed we have recently been sending announcements about the availability of compute clouds for scientific communities:…
From workspace-announce : I am happy to announce the availability of a science cloud (codenamed “Stratus” 😉 at the University of Florida. This cloud introduces a new feature: the use of virtual networks with virtual machines for cloud computing. The cloud is available for members of the scientific community: to obtain access you will need to provide a justification (a few sentences explaining…
If you’re feeling adventurous, there’s a workspace service pre-release out (click on the pic):