Arc'teryx is proud of the quality and performance of all our products. Items covered under our Limited Warranty will be replaced or repaired at the discretion of Arc'teryx for the practical product lifespan.
Please include pictures of your product when filling out our online Product Service Request Form (up to 25 MB per file). Your pictures will help us identify the issue and solutions we can provide.
Please select a guaranteed delivery service that provides parcel tracking information, insurance and proof of delivery. Arc’teryx is not responsible for items that are lost or damaged during shipping and cannot replace parcels lost during return shipment if proof of tracking is unavailable.
Apache Spark Scala Interview Questions- Shyam Mallesh -
val df = spark.read.option("inferSchema", "true").json("data.json")
val rdd = sc.parallelize(1 to 4) rdd.map(x => x * 2) // 2,4,6,8 rdd.flatMap(x => 1 to x) // 1,1,2,1,2,3,1,2,3,4 rdd.mapPartitions(iter => iter.map(_ * 2)) // same as map but per partition Spark uses lineage (RDD dependency graph). Each RDD remembers how it was built from other datasets. If a partition is lost, Spark recomputes it using the lineage, not replication. However, you can also cache/persist with replication (e.g., StorageLevel.MEMORY_AND_DISK_2 ). Apache Spark Scala Interview Questions- Shyam Mallesh
Here’s a curated set of , structured in the style of Shyam Mallesh (known for clear, practical, and depth-driven technical content). These range from beginner to advanced, covering RDDs, DataFrames, Spark SQL, optimizations, and internals. 🚀 Apache Spark Scala Interview Questions – By Shyam Mallesh ✅ 1. What are the differences between map , flatMap , and mapPartitions in Spark? | Transformation | Description | |----------------|-------------| | map | Applies a function to each element of an RDD/DataFrame and returns a new collection of same size. | | flatMap | Applies a function that returns a sequence (or Option) and flattens the result. Useful for one-to-many transformations. | | mapPartitions | Applies a function to each partition as an iterator. Avoids per-element function call overhead. Good for initialization (e.g., DB connections). | val df = spark
✅ ✅ 6. How do you handle skewed data in Spark? Skewed keys cause a few partitions to receive most of the data → slow tasks. However, you can also cache/persist with replication (e
val rdd = sc.textFile("data.txt") // nothing read yet val words = rdd.flatMap(_.split(" ")) // transformation val counts = words.map(w => (w, 1)).reduceByKey(_ + _) // transformation counts.saveAsTextFile("output") // 🔥 Action triggers job | Operation | Shuffle Behavior | Performance | |----------------|------------------|--------------| | groupByKey | Sends all values for a key across the network → high shuffle I/O | Slower, risks OOM | | reduceByKey | Combines values locally (map-side reduce) before shuffle → reduces data transfer | Faster, memory efficient |
breaks long lineages by saving RDD to reliable storage (HDFS/S3). ✅ 3. What is the difference between cache() , persist() , and checkpoint() ? | Method | Storage Level | Purpose | |--------------|------------------------------|---------| | cache() | MEMORY_ONLY (default) | Speed up repeated actions | | persist() | Choose level (MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY, etc.) | Fine-grained control over eviction | | checkpoint() | Saves to HDFS/S3 (reliable storage) | Break lineage, reduce driver memory, avoid recomputation chain | 💡 Use persist when memory is limited. Use checkpoint for long iterative algorithms (ML, GraphX). ✅ 4. Explain how Spark evaluates transformations and actions. Spark uses lazy evaluation – transformations build DAG but no data is processed until an action ( count , collect , save , show , etc.) is called.