Recent content on stdout by Arnab
It has been a year since my Japan trip, and I am finally summarizing my thoughts. I put “architecture” in the title because, for me, architecture is not just some lines on a piece of blue paper or diagrams on a whiteboard. It is about the people, the connections, the feelings, and the tiny details that make up the whole.
When building data workflows, you might need to poll a RESTful API at regular intervals until a specific condition is met. Azure Data Factory (ADF) makes it easy to implement this using Until, Web, and Wait activities. In this article, we will walk through setting up short polling in ADF. Let’s get started!
When working with different teams and using a database from another team, it’s essential to ensure that your application has the necessary permissions to utilize the database functionalities.In this article, we explored how to use Python with SQL Alchemy to check permissions for an application in a database without creating an entry in the database.
When you are using a functional mailbox which gets too many emails in Outlook (older Outlook Application in Windows), it can be distracting. In this article, we are going to see how to disable notifications for shared mailbox. Table of contents Disable All Notifications Add Rules to Mailbox In Conclusion Disable All Notifications In first step, we need to disable all the notifications for the…
We assume you already have a working KongHQ setup and running gRPC services. Otherwise we recommend looking at these following resources: Dev.to: Setup Kong with Docker GitHub: gRPC services example Dev.to: Docker with gRPC API Gateways like KongHQ are essential for exposing internal services outside a firewall, private network, or internally.
Even if your have monospace fonts like Operator Mono is installed in your Windows machine, it will not be listed in Settings. In this article I will show how you can add monospace fonts which are not listed in Intellij IDEA settings. Table of contents First option: install fonts in Administrator mode Second option: copy the fonts to IDEA’s font directory In conclusion References: First…
Stack Overflow (SO) is the biggest platform for developers’ questions and answers, yet one-third of the questions remain unanswered. Most often, the feedback I get from people who are answering for the first time is that they are afraid to ask questions or get a lot of downvotes. As a contributor at SO, I feel biased toward the contributors as I see many questions on the platform which are…
The TFRecord (Tensor Flow Records) format is a simple format for storing a sequence of binary records. It is much more efficient storage because it can take up less space than the original data. Aside from that, you can partition TFRecords into multiple files. In this article, we are going to see how you can convert Pandas Dataframe (essentially from a CSV) to TFRecords or vice versa.
If you have a CS background, then you are bound to have encounters with professional projects. Before starting one, you should keep a few things in mind; otherwise, working could be a bit difficult. Here are some tips for a jump start in your professional project development. Table of contents Knowing your stakeholders All IDEs are good IDEs Git is your friend Keeping only one copy of the project…
When I was working with data science project running in Docker, I found that working with huge data can be problematic. Even tools like PyArrow is not fast enough in Docker. But when I debugged the logs, I saw that if I could give it access to NVIDIA Cuda, then it would be much faster.
Adding search functionality make your website more accessible and usable to the stakeholders. In this article, I am going to share on how you can add this in a static site generated by hugo. Table of contents Expose JSON output Create layout Build the search page Add lunr.js file Add search functionality Add HTML Add JavaScript In conclusion Expose JSON output In the first step, we need to expose…
If you are already using Cloudflare, you know that Cloudflare caches fonts (and other contents). But it won’t work if you are using Google font’s CDN. Cloudflare has a detailed guide on how to improve Google fonts performance, but it heavily relies on workers, which might cost a lot of money if your website has a lot of users.
Even if you use Vim, it does not mean that you have to code in a plain, stuttering, and boring terminal. In this article, we are going to see how to set up a beautiful interface for coding, which is highly configurable yet highly automated. Without further aduo, let’s go!! Table of contents Alacritty Fish Shell tmux Vim Save configuration In conclusion Alacritty Alacritty is a GPU…
If you are comfortable with your current IDE but still want to try something interesting which might increase your ✌️ productivity ✌️, then you can look into Vim. This article is not setting up your Vim, you will find thousands of articles on Google. Rather, I want to share my thoughts which might motivate(or demotivate) you to use Vim.
Suppose you are using Github free plan, you are running actions in your private repo and keep exceeding the monthly minutes limit, then this article is going to help you to make optimized usage of the actions minutes in step by step manner. Table of contents Step one: run workflow on schedule instead of push Step two: run workflow manually only if you push In conclusion Step one: run workflow on…
In JAMStack(aka static sites), we use different paid/free services for things like using Disqus for commenting, MailChimp for newsletters etc. These services come with a tradeoff, for example lack of privacy, bloated JavaScript and many more. What if we can build them on our own using a serverless stack, with low to no cost and almost zero maintenance?
When you run brew upgrade or brew upgrade <package>, all the packages are upgraded or packages linked to that particular package you want to upgrade gets updated. For Python, if the version upgrade is minor, then it should be fine. But major updates break a lot of development setup like, if you are used to develop using virtual environment, then you might have re-install every package.
When it comes to web scraping, you have to keep in mind that it is not the coding skill rather cunning observations will get you results. Sometimes the data you get comes in a straight forward way, other times the website does not want you to crawl itself. Either way you will get the data, but it takes time depending on how many puzzles do you have to solve before getting it.
You will see many Hugo themes boast perfect scores on Google PageSpeed Insights(PSI). To be honest, most of them are not as pretty as others and lack a significant amount of JavaScript(or no JavaScript at all). But, it does not matter anymore. Because you can use any theme(or develop on your own) and get an almost perfect score in PSI if you implement the following steps.
We all know or heard of Progressive Web Apps. I wanted to implement it because it enables web applications workable in offline, like a native app. Also it provides content faster by caching them. With help of github actions and gulp, we can easily add it to a static site deployed in Github pages.
Hugo provides a built-in module named {{ .TableOfContents }} to generate table of contents. It is pretty handy because it generates the table of content without help of any JavaScript or CSS code. But there is a problem with it, you need to put it inside templates. You can’t render it inside the contents of the markdown file.
If you want to use data science packages like numpy, scipy, pandas etc with your docker container and using pip to install them, then it will take forever to build the images. Because pip compiles the C extended code and that compilation needs a lot of time. For numpy it took me around 4 minutes.
Let us say, you need an authentication service in a rush, which needs to be cutting edge and uses JWT based authentication. Or you need a stand alone authentication service to be plugged in with other microservices. Do not worry, you can create one in 10 minutes. All you need to have is Python and Docker installed in your machine.
AWS CodeBuild has nice integration with different Git repository hosting service providers(like GitHub, BitBucket, even amazon’s own CodeCommit etc). Even using WebHook is pretty easy. You can start a build in CodeBuild for every push, pull, PR created, PR merged etc. But it can be bit tricky when it comes to trigger for every tag push only, as it us not a default event type provided by…
Hugo is a fantastic framework to generate static site from markdown and serve them. Using environment variables in templates is a breeze but using them in markdown files can be a bit tricky. You need to use custom shortcodes as workaround to resolve this issue. Table of contents What is shortcodes Custom shortcodes Use environment variable in shortcodes By getenv variable By site variable In…
Disclaimer: this article has been generated as part of IBM Data Science Professional Certificate course’s final submission. This report consists of three parts: business problem and data preparation, methodology, visualization and results. In this part, we are going visualize the data, discuss about the results and draw a conclusion.
Disclaimer: this article has been generated as part of IBM Data Science Professional Certificate course’s final submission. This report consists of three parts: business problem and data preparation, methodology, visualization and results. In this article we are going to describe the methodology we used in step by step: Table of contents Step one: New York city data with latitude and…
Disclaimer: this report has been generated as part of IBM Data Science Professional Certificate course’s final submission. This report consists of three parts: business problem and data preparation, methodology, visualization and results. In this part, we will discuss about business problem and data preparation steps. Table of contents Business problem Data Approach Data preparation Source…
If you are using Hugo to generate static pages, you are familiar with CLI commands which are to build the static pages in your local machine and make push to your <username>.github.io repository. When it comes to using Hugo for blogging, compared to platforms like Medium or WordPress, it is very painful because you do not have any web interface to make changes whenever you want or wherever you…
DISCLAIMER: This is not the recommended process. Ideally you should have the database running in a separate container and use network to interact with that container. Usually, we don’t ship application with the database inside the same container. But sometimes we are forced to do that. In this article we are going to see a minimalist way of shipping application with database built-in inside docker…
Maintaining authentication layer in ReactJS can be painful. Back in the old days(even now-a-days), you needed to override every componentDidMount method in class based components to see if the user is authenticated. In the latest ReactJS, there are hooks(specially useEffect) where you need to write these checks. Either way, some codes will be redundant.
AWS CodeBuild is an extraordinary tool for building your code. Recently I have been using it to build projects and store docker images in AWS ECR. During those CodeBuild processes, I needed Database for running tests. As the need of the Database was for a limited time and I didn’t want to pay extra for it(that is why I did not use AWS RDS), I decided to create database using docker inside…
We already know that Docker is used for containerization. But networking in docker makes it even better tool for deployment. You can make standalone deployments in containers and communicate between them, or make containers act as host, or even assign mac addresses to containers and act them as devices. It does not matter which host machine you are in, you can make dockers communicate in-between…
Collecting data in static sites or serverless sites(with POST request) is a hassle because you don’t have a server to receive POST requests and store data. But Staticman is here to help. It will enable users to make POST requests, that will allow you to gather these data. In this article, I am going to tweak Staticman a little to implement newsletter subscription.
Have you worked with Docker before? Do you think you have trouble with it, like when you are developing an application, you need to build every time to see the results, or thinking of the best way to load data in DB, or may be you are annoyed because there are lots of markdown files, screenshots etc that are taking a lot of space in your container.
Imagine, you have some microservices and you want to use API from one service to another, how can you do that if all of them are running in different docker container? Docker Compose has a solution for you. Suppose you want to deploy a project in test, stage and production environment with same Dockerfile for all of them through a single Dockerfile.
For the past couple of years, I have been creating Dockerfile for different projects. based on those experiences I am going to share some tips on writing docker files to communication between containers. _FYI: before reading this article, please read the article from official docker blog site to learn about best practices for creating Dockerfile.
For the past couple of days, I am testing out Manjaro linux. Most of the time, I was working with KDE Plasma Edition. In my opinion, it is the most beautiful desktop environment from Manjaro official distributions. I am a fan of Ubuntu Unity workspaces and wanted to use similar features in KDE plasma as well.
Recently I tried out Manjaro operating system, and did some experiment with XFCE and KDE desktop environments. Here I am going to share some tips and tricks to setup Manjaro in you machine. Now, let us dive in. Table of contents Copy bash history to ZSH Backlight On using terminal Install visual studio code(VS Code) Wireless headphone muddy sound fix GPG verify key error fix Package managers AUR…
Lets say you have an API which is being used by many stores. Now for a specific store ABC, they want to have a different response format. So how would you make this implementation generic without making different views for store ABC? Simple, use middleware. Table of contents How it works Usage Another usage scenario How it works First update the url like this:
Adding a plugin to Solr is relatively simple process. In this article, we are going to see how to add a plugin by creating a jar file using a docker container, then add that to a Solr which also runs inside docker container. Table of contents Building the plugin Workflow for building the plugin Example of solr image Explanation Run solr Without docker-compose With docker-compose Configure plugin…
VS Code is the most popular IDE at the moment. You can use it for developing applications in almost any programming language. In addition to that, you can do remote development in Docker,VM etc. In this post, we are going to discuss about how to use Docker Environment in VS Code.
Django’s serialization framework provides a mechanism for “translating” Django models into other formats. Usually they are in json output, yaml output, XML output, GeoJSON output etc text based formats. Here, we are going to supercharge these serializers to do more things, even try to render properties method and instance method.
When you are using OpenShift, you will be using routes to expose a route from a service. Let’s say you want to expose a path in example.com/dummy. When you do that for a django application(without any reverse proxy server), it usually becomes a problem, because the sub directory does not work well with Django’s urls.
Connecting to MySQL from Python in MacOS is a very problematic and painful process. In this post, we are going to see how to install MySQL and connect a Python application to it using mysqlclient. Table of contents Step one: install Homebrew Step two: install MySQL Step three: install MySQL-Connector-C Step four: install XCode-Select Step five: install OpenSSL Step six: unlink MySQL and link…
Lets say you have been using a class named ClassA, objects created from that class has been used in many places. Now you have a requirement that you need to make a simillar class ClassB, which has almost simillar functionality. Objects of ClassA and ClassB will have simillar method signatures, and could be swapable.
Today, I am going to write about few useful snippets/functionalities which I have used for Amazon S3 or any S3 compatible storage using Boto3 and Django Storage. FYI, this post focuses on using S3 with Django. So without further ado, let us begin. Table of contents Configuring S3 Using S3 which it does not belong to AWS Serve S3 in different domain(through cloudFront or varnish) Create new bucket…
Implementing the procedures mentioned here in a production system may cause security issues in the future. I wrote this article as a hack. Suppose you are dealing with a poorly written API, which returns different type of objects at different scenarios. For example, sometimes it returns a string(ie. abcd), sometimes it returns a list(ie.
Linked list is a data structure where each object has points to next. In django, linked list can provide an interesting solution when it comes to custom ordering/grouping issues. Lets think of an example like this= you are making a blog site, and you have made a Post Model like this:
Every now and then I see some questions pop up in Stack Overflow with title: ValueError: Related model ‘app.User’ cannot be resolved This is a very common issue and pops up when the user tries to run the migration. So, lets talk more about what is this problem and how can we solve it.