Testing FastAPI applications with async resources? If you're using Starlette's TestClient, you're mixing sync and async contexts in ways that might break your tests. Here's why httpx.AsyncClient should be your go-to choice for keeping everything properly async. The Hidden Problem with TestClient: Sync/Async Context Mixing TestClient looks convenient …
In modern web applications, we often use async fire and forget patterns to improve user experience, for example, when a user clicks a item on a page, or an ML model makes a real-time prediction and want to log the event. These events are often not super critical and we …
Introduction Online inference is a technique used to deploy machine learning models in production. However, there are several considerations to keep in mind when deploying models, especially when using FastAPI/gunicorn along with libraries such as NumPy and PyTorch. This article highlights a few of these caveats. Model Loading When …
Prior to spark 3.0+, to optimize for performance and utilize vectorized operations, you'd generally have to repartition the dataset and invoke mapPartitions . This had the major drawback of performance impact that was incurred from repartitioning (caused by shuffle) the DataFrame . With spark 3.0+, if your underlying function is …
I had been revisiting concurrent libraries that I had worked upon earlier and just wanted to highlight the importance of using separate wait sets and condition queues for your library implementations. The performance of these has been benchmarked using JMH . Let me just list down the advantages of using separate …
For structured (bounding box based) text extraction, it becomes imperative that the received image and target image are aligned properly and to scale. OpenCV is a great image processing library that has a ton of features. To align source and template images, following steps are required. First convert images to …
As you all are aware that Google Plus is shutting down in March 2019 and so are all its services. I have had a legacy android app on play store that was using the GoogleApiClient for authentication with Google Plus services, alas, I had to upgrade the application to use …
In this post, I will cover a tutorial that involves different moving pieces. It covers the following: Java WatchService Spring Boot Initialization-on-demand holder idiom Managing concurrency RXJava Lombok (because why type more?) The example will expose a Spring Boot REST service that exposes csv file records from a directory. In …
RXJava is an extremely useful streaming framework (here is an example application using it for parallel processing of restful calls to both uber and lyft ( RT_UBER_NYC_TAXI )). However, In this post, I will cover how you can reactively stream and process a CSV file. Firstly, you can create a Flowable of …
In this post, I will share a few quick tips about scaling your Spark applications to larger datasets without having large executor memory. Increase Shuffle partitions : The default shuffle partitions is 200, for larger datasets, you are better off with larger number of shuffle partitions. This helps in many ways …
I have been working on ML projects that require image preprocessing and text extraction. To improve the quality of text extraction, there are many preprocessing steps that we need to do, they are elicited below. We use OpenCV for doing the preprocessing and tesseract-ocr for text extraction. Image preprocessing Rescaling …
Column ambiguity is quite common when you join two tables. Now this poses a unnecessary hassle when you want to select all the columns from both the tables whilst discarding the duplicate columns. The aforementioned problem is difficult to handle especially, if you have wide tables, where you would want …
If you are working with Spark, you will most likely have to write transforms on dataframes. Dataframe exposes the obvious method df.withColumn(col_name,col_expression) for adding a column with a specified expression. Now, as we know that the dataframes are immutable in nature, so we are getting a newly …
Apache Spark offers the ability to write Generic UDFs . However, for an idiomatic implementation, there are a couple of things that one needs to keep in mind. You should return a subtype of Option because Spark treats None subtype automatically as null and is able to extract value from Some …
Testing Spark Dataframe transforms is essential and can be accomplished in a more reusable manner. The way, I generally accomplish that is to Read the expected and test Dataframe, and Invoke the desired transform, and Calculate the difference between dataframes. The only caveat in calculating the difference is that in …
I have been working a lot on Spark and Scala . I have really like scala as a language, due to its numerous advantages over Java, the foremost being that for a simpler API having Type classes and Default Method Arguments does wonders. Also, idiomatic scala code uses higher order functions …
Apache Zeppelin provides a Web-UI where you can iteratively build spark scripts in Scala, Python, etc. (It also provides autocomplete support), run Sparkql queries against Hive or other store and visualize the results from the query or spark dataframes. This is somewhat akin to what Ipython notebooks do for python …
Apache spark has an advanced DAG execution engine and supports in memory computation . In memory computation combined with DAG execution leads to a far better performance than running map reduce jobs. In this post, I will show an example of using Linear regression with Apache Spark. The dataset is NYC-Yellow …
Google recently has deprecated the Google+ Sign in and process of obtaining oauth access tokens via GoogleAuthUtil.getToken API. Now, they reccomend a single entry point via new Google Sign-In API. The major reasons for doing so are 1. It enhances user experience and 2. It improves security, more here …
Hive or Impala ? Hive and Impala both support SQL operation, but the performance of Impala is far superior than that of Hive . Although now with Spark SQL engine and use of HiveContext the performance of hive queries is also significantly fast, impala still has a better performance. The reason that …
The Idea Java 8 introduced functional programming support, this is a powerful feature which was missing from earlier versions. One of the benefits of functional programming is that it can be used to implement decorator pattern without the use of inheritance. One common requirement is to implement some kind of …
You might run into a scenario where you might require conditional authentication with Retrofit 2.0. This post provides an example of integration with the Lyft API . In case of the Lyft API, first we need to authenticate with and query the oauth/token endpoint to obtain the OAUTH token …
In this post I will explain why you should use square root of Gini index while building decision tree classification models. In decision tress, We know that at every node we need to choose a feature that provides the best split i.e. the feature that reduces the child nodes' …
Earlier, I had covered an example here , which showed how to dynamically create users and map the application roles to enterprise groups. In this post, the sample application is extended to show how you can query the application roles from the application stripe(application specific policies). To query the application …
You might be aware that there are different content encoding formats for encoding the text. Generally, it is safe to use UTF encoding, but at least you would expect that the websites would specify the encoding format in the response. Alas, you might find certain sites , which just send the …