Unique and New Ways to Create Spark Dataframe | Scholarnest
Spark Dataframe Apache Spark is a powerful open-source distributed computing framework that provides efficient and scalable processing of large datasets. One of the key features of Spark is its ability to handle structured data using a powerful data abstraction called Spark Dataframe. Spark Dataframe are similar to tables in a relational database, and they provide a high-level API for processing …
Continue Reading
How to Remove Duplicate Rows in a Spark Data Frame
Apache Spark Data Frame API allows you to read data from various sources and creates a Spark Data Frame out of the source data. However, you may have duplicate rows in your Spark Data Frame. Duplicate rows may show up in your Spark Data Frames for various reasons. Your ETL tool that moves data from one place to another place …
Continue Reading
Unlock Apache Spark Memory Allocation – All you want to know
In this blog post, I will explain the memory allocation for the Spark driver and executor. If you are here, I assume you are already familiar with Apache Spark, its architecture, and why Spark needs memory. However, I will quickly revise a few concepts to bring all readers to the same page. Let’s start with the following question. What is …
Continue Reading
Unlock Apache Kafka – All you want to know and focus
What is Apache Kafka? Apache Kafka is a distributed streaming platform that is built on the principles of a messaging system. Apache Kafka’s implementation started as a messaging system to create a robust data pipeline. However, over time, Kafka has evolved into a full-fledged streaming platform that offers all the core capabilities to implement stream processing applications over real-time data …
Continue Reading