Cfrgtkky

My Spark script is failing because the S3 bucket from which the df is drawn gets updated with new files while the script is running. I don't care about the newly arriving files, but apparently Spark does.

I've tried adding the REFRESH TABLE command per the error msg, but that doesn't work because it is impossible to know at execution time when the new files will arrive, and so no way to know where to put that command. I have tried putting that REFRESH command in 4 different places in the script (in other words, invoking it 4 times at different points in the script) - all with the same failure message

Caused by: java.io.FileNotFoundException: No such file or directory '<snipped for posting>.snappy.parquet'

It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.

I create the df with: df = spark.table('data_base.bal_daily_posts')

So what can I do to make sure that S3 files arriving at S3 post-script-kickoff are ignored and do not error out the script?

Move files you're going to process to a different folder(key) and point spark to work with this folder only

I'm not sure this will work or not, but give a try:

Once you read your input files from S3 immediately perform persist operation on that dataframe something like below

import org.apache.spark.storage.StorageLevel



val inputDataFrame = sparkSession.read.json("s3a://bucket_name/file_path/)

    .persist(StorageLevel.MEMORY_AND_DISK)

here even if your dataframe gets evicted from memory, as it is available in disk, it will load from disk instead of fetching from S3