Installing Python 3.11.9 on Amazon Linux 2 (AL2) If you try to install Python3 on AL2, you might hit this particular error. WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. Here s how you can install it: Reference:1. https://alas.aws.amazon.com/AL2/ALAS-2023-1934.html2. https://github.com/actions/setup-python/issues/93
My career had always been that of an individual contributor (IC). For the last decade, I have always worked as an IC role, from DBA to SRE to Solution Architect. When I joined InfoWiz, I stepped into the role of a Solution Architect, yet another IC position. But then, just three months in, my manager Continue reading Transitioning from IC role to Managerial role
The tech industry is in a constant state of flux, driven by relentless innovation and evolving market needs. This dynamic nature has profoundly influenced career paths, especially in the data industry. Let s take a journey back a decade ago to understand how careers in this field have transformed and what the future holds with the Continue reading The Ever Changing World of Tech Industry Data…
Loaded data using the database processing resources. You might wonder, why was there a shift from ETL to ELT? Back in 2006, Oracle was the undisputed ruler of the database kingdom. Oracle 10gR2 was all the rage, and Oracle RAC + DataGuard was the shiny armor of the premium database tech stack. Every Fortune 500 Continue reading Beware of Vendors and Market Trends
sql_mode in managed MySQL 8.0 across Cloud Providers In the official MySQL 8.0, the default values for sql_mode is the following: ONLY_FULL_GROUP_BY STRICT_TRANS_TABLES NO_ZERO_IN_DATE NO_ZERO_DATE ERROR_FOR_DIVISION_BY_ZERO NO_ENGINE_SUBSTITUTION However, most of the Cloud providers have it differently. Official MySQL 8.0 Alibaba Cloud AWS GCP ONLY_FULL_GROUP_BY STRICT_TRANS_TABLES…
AWS Lambda /lib64/libc.so.6: version `GLIBC_2.28 not found (required by /var/task/cryptography/hazmat/bindings/_rust.abi3.so) , errorType : Runtime.ImportModuleError AWS Lambda has many nuances and one of the most annoying issue is that the system that you are building your packages is incompatible with the Lambda runtime configuration. Typically, when we run our CI process, the images we used are…
AWS Glue An error occurred while calling o199.pyWriteDataFrame. Cannot find catalog plugin class for catalog glue_catalog : org.apache.iceberg.spark.SparkCatalog To resolve this issue please subscribe to an Iceberg Connector by doing the following: After doing this, navigate to your Glue job and do the following: Reference:
An error occurred while calling o177.pyWriteDynamicFrame. Schema cannot be updated: data type has been changed, new data type: [column] optional int32 column (DECIMAL(4,2)) does not match old data type: [column] optional int32 column (DECIMAL(3,2)) The AWS ‘glueparquet writer class has a validation function under the hood which throws exception when it detects datatype changes in Continue reading…
Partition Data from DateTime in Spark Imagine if you have the following table with a timestamp field. You would like to store the above table in a hive-style partitioning to reduce data scanned by using partition pruning. Since the queries are often time ranged by last_play_date, you want to partition by the last_play_date column. If Continue reading Partition Data from DateTime in Spark
AWS Glue An error occurred while calling o93.getDynamicFrame. No such file or directory s3 parquet Ensure that your Glue IAM role has the required permission, not just on the S3 bucket itself, but the objects within the bucket. A silly example for the IAM role permission as below: It s not recommended for production!