Articles from The Swamp
I looked forward to Dataform becoming part of Google Cloud. Merging into GCP took longer than I hoped and expected. In the second half of 2022, Dataform was (finally) integrated as a part of BigQuery. This whole article is about Dataform as a part of Google Cloud, not the initial (now legacy) Dataform. Dataform offers UI to initialize a project, write queries, verify syntax, compile, and execute.…
I have a few small websites hosted on the Digital Ocean server. Each of them has a MySQL database and I wanted to regularly set uploading full database export to some independent location outside of Digital Ocean to sleep more calmly. Since my life revolves around Google Cloud, my preference for storage was clear. The goal is to use crontab to periodically run a bash script to execute MySQL…
Jupyter Notebooks are a great tool for data science & analysis. They are developed on Jupyter Notebook servers which can be installed and run locally for development. It usually starts with exploring, developing, making prototypes, playing with the data. For example, exploring Covid data loaded as a CSV file from Our World in Data . You see there are the whole World stats and you would like to…
In one of my projects I am working on, I'm using AWS Elastic Container Service to run Fargate tasks. Why? I have jobs that I need to run, so it's quite easy to pack into a Docker container and deploy it on Fargate and pay just for used resources. I'm using Airflow to schedule those tasks. Since Airflow runs tasks on Fargate, all job logs are created within the Fargate task (Docker container).…
Firebase is a popular platform for web and mobile app development. The main database option when developing applications on Firebase platform is the NoSQL database Cloud Firestore which is also part of Google Cloud Platform. Since Firestore is a NoSQL database, there are different considerations regarding the modeling and design of relationships in comparison with SQL databases. The weaker part of…
Google Cloud Functions is a serverless compute platform on Google Cloud Platform. Its main principle is: "Write code in NodeJS, Python or Go and deploy it and don't worry about anything else". This flow is very well suited for small code, a simple function that responds to HTTP requests or various events from PubSub, Cloud Storage, Firestore, etc. Although there is no restriction to install any…
Of course, the title is an overstatement. For the first time, I joined the hackathon in Prague, the Czech Republic this weekend, called "hack the state" ( Hackuj stát ) and it was about using government's Open Data to do some interesting projects and demonstrate that it's worth publishing government's data. The Czech Republic already have open data initiative with many datasets from many…
Serverless products on Google Cloud Platform (GCP) such as Cloud Functions and App Engine due to their serverless nature (hidden server infrastructure) can connect to some of the GCP products (Compute Engine, Memory Store, Cloud SQL) only through their public IP addresses which of course is adding security risks by exposing them to the whole internet as well as adding latency, since traffic goes…
On May 21st, 2017 it was (somewhat) quietly announced that for Cloud Task there is Beta support for HTTP targets. I want to explain what does it mean in this article since I think it deserves more attention, but first let's go back in history to understand the whole context. Cloud Tasks is a product on Google Cloud Platform which manages dispatching of asynchronous tasks. What that means is that…
In my previous article , I described the process of deploying private Cloud Run service and then using Service Account to make requests to it. With these properties plays nicely use of Cloud Run as an asynchronous task worker which I want to describe in this article. I'll use as an example of service from the previous article, Microsoft Word Docx conversion to PDF using Libreoffice but instead of…
Cloud Run is a new compute serverless solution on Google Cloud Platform. It can run any web app deployed as Docker image. One of the nice features it has is built in automatic authentication, i.e. you can hide service from public internet and control access via IAM. With this, you grant access to concrete users or groups. Since one of the primary uses of Cloud Run are microservices and with access…
With Google Cloud Run introduction during the Cloud Next 2019, Serverless/Managed Compute family got a new member. There are now 3 major products: App Engine (I wrote in more details about it here ) Cloud Functions Cloud Run. How does Cloud Run fit here and what are its properties and use cases? I created a comparison table so it's easier to compare with other serverless compute products on Google…
Journey on Google Cloud for most people starts with registering, setting credit card info and then diving into Cloud Console to use awesome products and services. Google is gracious that it gives 300$ of credit for start, so you can freely explore and don't worry that you will be billed from your own pocket from the first minute. The first thing you should do however after getting to Cloud Console…
Firestore is next generation of NoSQL databases on Google Cloud Platform adding various improvements over Cloud Datastore like real time updates, strongly consistent queries etc. It was recently promoted to General Availability, so it's about time to dive more into it. Firestore exists in 2 flavours. One is Native mode which resembles all features of new Firestore, whereas second one, Datastore…
In one of my previous articles I wrote about creating data pipeline to get regularly information about BigQuery public datasets and inserting them into one BigQuery table. In this article I want to describe how it is designed. It's not rocket science, but rather example how to create serverless pipeline. Everything runs on Google Cloud Platform and costs 0$. Complete code is here…
I created public BigQuery table which contains metadata about BigQuery public datasets. As it well known, BigQuery has public datasets containing data with various nature and size. For example, there are Bitcoin and Ethereum transactions, data from World Bank, data about patents, varios (mostly USA) agencies like Bereau of Labor or Forest statistics etc. Since they are public, that means that…
This years second generation of standard runtimes for Google App Engine was released. So now there is First and Second generations of Standard and also Flexible. For people who are coming to Google Cloud Platform and are thinking about using Google App Engine this can be confusing, so with this article I wanted to bring maybe bit more clarification. It can be also nice nostalgia reading for long…
Forth article regarding topic Search on Google Cloud is devoted to Cloud Spanner. Idea is to explore various different products on Google Cloud Platform and how they could be used to implement search functionality for fictious eshop. So far, articles in serie are devoted to: App Engine and Search API (contains most of information and details which are not repeated in other articles) Cloud…
Continuing series on topic "Search on Google Cloud Platform", in this article I'll explore Cloud SQL as storage, namely PostgreSQL built in functionality for text search. Previous articles / solutions in this series are: Search on Google Cloud Platform - App Engine and Search API Search on Google Cloud Platform - Cloud Datastore I'll repeat task description from previous articles: "Create…
The other day I was answering question on Quora " How your company is using Google cloud and what for? ". Of course since my business is related to consulting for other companies and it usually involves Google Cloud Platform (GCP), I though to write some concrete cases what can be done with GCP (of course a lot). As I was thinking and then writing, one point especially stood out to me: using…
In one of my previous articles , I was uploading 1.2 million of records from csv file to Cloud Datastore. I was reading file and making list of 200 parsed lines and post that as json sequentially to webserver ( here is full script). Of course that's rather slow, i.e. process took some 2.5 hours to complete. Only afterwards I remembered that there is a faster way and that is by using Cloud…
In previous article I described simplified solution of how to implement search functionality (for eshop or similar) using Google Cloud Platform Products, namely using Search API which is part of Google App Engine Standard. In this article I will describe solution using Cloud Datastore as storage. Most of introductory explanation is written in previous article, thus I will focus here only on…
I came around few times on topic "Search on Google Cloud Platform", i.e. people asking question of different ways how to implement search. Not sure if it's not part of some Google Cloud test or exam, but it's interesting topic and GCP offers different ways how to do it so I decided to do series on this topic. So in each article I want to describe one way of implementation as well as code…
Wordpress is really not my are of interest but I'm writing this because of several reasons: 1. In comments of my youtube video regarding Google Cloud DNS people had different problems and with one way or another, it was always related to setting up WordPress website (mostly connecting domain). 2. With my article Service Accounts on Google Cloud Platform I started unintentionally series of articles…
It often happens when I work with clients that they don't have much experience with Google Cloud Platform, i.e. they generally register and create project, add developers to project, but for example setting service accounts in not so straightforward in case they want to give developer granular access. So in this article I want to give an overview about Service Accounts which is often used feature…
Since the beginning (2008) Google App Engine (GAE) had integrated Datastore API which could be used also for local development. Few years later Datastore was created as separate product on Google Cloud Platform and could be accessed either with HTTP requests or with client libraries for most popular languages. To use Datastore locally for development, Datastore emulator was developed as part of…
SSH from web browser to Google Compute Engine instance is one of those cool features which simplifies every day life. With one click of a button, you can connect to Compute Eninge VM and do stuff on command line. Since main purpose is of course to connect to your GCE instance and use command line, some people don't notice or care that in top right corner of ssh window there is "option" icon…
Webapp2 is Python framework for creating web applications and is primarily used for development on Google App Engine. In fact it's included in Google App Engine SDK, which simplifies initial development, because when starting developing for Google App Engine Standard, most important things are included in SDK (webapp2 as framework, jinja2 as template engine). It supports WSGI interface so it can…
As I wrote in previous article Google Cloud Storage has interesting features which doesn't have to be immediately obvious. Today I want to continue this topic with focus on command gsutil . gsutil is a command line tool to work with Google Cloud Storage. It's part of Google Cloud SDK but it can be installed as standalone program. Code is open sourced on Github . If you look at gsutil commands…
When somebody tells you Google Cloud Storage, probably first thing that comes to your mind is bucket, blob, Nearline, Coldline, you can set ACL for buckets and objects in there... but lets have a look at some cool features which are not so often mentioned. All code examples are in this github repository . Transcoding Transcoding is feature that allows you to upload files compressed in gzip but…
From time to time, there is need in web application for users to upload file or multiple files to server, whether it is image or video or some text file. Starting point is usually HTML form with select button: <form action="/" method="POST" enctype="multipart/form-data"> Select files: <input type="file" name="files" multiple> <input type="submit" value="Upload"> </form> When button is pressed,…
gcloud is program which provides command line interface to Google Cloud Platform products and services. So basically everything that is possible to do via Web at console.cloud.google.com it's more or less possible to do with gcloud: deploying Google App Engine applications, create Google Compute Engine instances, hard disks, containers etc. gcloud is part of Google Cloud SDK which consists of…
So finally at Cloud Next 2017 (March 2017), Google App Engine Flexible was announced as generally available. You can read about history and my views of GAE Flex in these posts: Google App Engine Flexible Environment status Reflections on Google App Engine Flexible going Beta Since functionalities should be stable now, lets move forward and see how can we use this. First I want to write about some…
Google Cloud Platform (GCP) offers for some of it's products and services daily/monthly free tiers. This can be very useful if you are student or living in a third world country and you want to spent as less money as possible whether you are just starting, trying things out, hacking some personal project or prototyping your idea into application which will make from you next billionaire or as a…
When project is created on Google Cloud Platform, my-project-name.appspot.com domain is automatically created and deployed Google App Engine (GEA) application handles request there, but if web application is production ready, we want that it handles requests for our own (custom, more name friendly) domain. Further we want that website has only one unique domain where visitors will go. People often…
In November 2016 GAE Flex went Beta, and since some changes were introduced I want to write my thoughts about it, but first I want to rewind and go to the beginning. Note April 2017: latest article about Google App Engine Flexible is here Google App Engine Flexible (with example) and it reflects status where GAE Flexible is generally available. Back to the future In March 2014 at Google Cloud…
On August 25th I’ve got email from Google Cloud Platform inviting me to fill out survey regarding Google Cloud Endpoints. It took me approximately one hour, since I used that time to dig deeper and collect all my wisdom and experience, frustration and kindly fill out survey :) Current situation Then I look at Endpoints issues and noticed there are 63 issues all in status Acknowledged or Accepted…
Google Cloud DNS is one of products of Google Cloud Platform (it's placed under Network products) that allows you to manage DNS records for your domain. Just to explain briefly, DNS stands for Domain Name System. To take excerpt from Wikipedia , article starts with "DNS is a hierarchical decentralized naming system for computers, services, or an resource connected to the Internet or a private…
I am writing this article out of my own personal experience working with GAE Flexible environment (more precisely one case where it didn't work out so well). Note April 2017: I've written two more articles about Google App Engine Flexible since this one became kinda obsolete and some things changed during period: Reflections on Google App Engine Flexible going Beta and latest Google App Engine…
Have your thought to create some course and show it to the world? Of course you can create video and post it on the Youtube but how about to do it more sophisticated and improve quality of course? You can use online platforms to host courses with extra functionalities like tests, analytics etc. or you can host on your own Open Source edX . I created short presentation in which I walk through…
In May this year, Google announced availability of Big Table database for public use through Google Cloud Platform. What that means is that we mortals, can use the same database that Google uses for it’s products like Gmail, Google Analytics or Google Earth and many others. It’s praiseworthy that Google released paper about Big table in 2006 which was inspiration for other opensource NoSql…
Google App Engine is great platform for building web applications. I created presentation where I pointed out some thoughs from a perspective of person (or startup) who has idea for some web application and looks where to host it. Google App Engine - platform for your Startup from Zdenko Hrcek
Small step for humanity but big step for one man as saying goes. This week I published my first Android app to Google Play Store. It's not rocket science, since I made application which opens website and then you can normally navigate as in you were on website in app. But I think it's still cool, that I don't have to go to browser on my mobile, type url and then do something but I can click on app…
One aspect of working with database (doesn't relates only to GAE) is retrieving stored data. It can be single entity or several entities. Quite often, need is to look at large number of data whether is to displaying for user or some background batch processing. To retrieve large number of data is slower and leads to increasing latency so often pagination is used. Pagination (in context of web…
When Udacity launched in summer 2014 course Developing Scalable Apps (with Java), my review here: http://www.the-swamp.info/blog/mooc-udacity-course-review-developing-scalable-apps/ there were requests for same course in Python, fortunately Udacity heard plea and few days ago I accidentally look at courses listing and I noticed Developing Scalable Apps in Python among new courses. So I was curious…
The other day I wanted to put good and inspiring videos regarding Google Cloud Platform (Google App Engine) all together on one place but when I searched Google first link was to google page where they have collection of videos from youtube. Anyway for the sake of invested time in this I putting links here also: Google Cloud Platform - Videos from Google developers regarding Google Cloud Platform…
In the end of 2014 I started little more to show to the world my side project whatmademehappytoday.net and I wrote today a blog post where I described what made me happy in 2014. It's more like a general analysis than concrete things, but I am glad for it. Happy 2015 !!!
Few weeks ago I was doing backup of various folders, which I zipped into one file and uploaded to Google Drive (22.8 GB totally). I downloaded it today and when I wanted to unpack it on my Linux computer with: unzip backup.zip I've got: Archive: backup.zip warning [backup.zip]: 20207777058 extra bytes at beginning or within zipfile (attempting to process anyway) error [backup.zip]: start of…
I am doing courses on Udacity since their beginning, so I was quite curious when they recently introduced new courses in cooperation with Google. Those are: Developing Scalable Apps UX Design for Mobile Developers Developing Android Apps Website Performance Optimization All sounds interesting of course, and the one which got attention was the first one, which is about developing web applications…
Few weeks ago I was solving problem which consisted of calling url in Java program through proxy. It took me a while to figure out correct combination :) so I am sharing sample code that worked for me. I was doing request through Microsoft proxy NTLMv2. Here is the main code: import java.io.IOException; import org.apache.http.HttpEntity; import org.apache.http.HttpHost; import…