We are reading athena data through jdbc connection from databricks notebook using spark.read.format(“jdbc”).
It is all working fine but not able to understand why it is automatically creating queries passed through spark.read.format(“jdbc”)..option(“query”, query) as preparedStatements and they are reamining there in athena workgroup forever. Hence making it to reach the max limit of 1000 preparedstatemnts a workgroup can hold and no longer it does allow any other quereis to run until we delete those queued up preparedstatemnts in the workgroup.
Note: Not all the queries are being stuck there as preparedStatements, only some and randomly.
Any idea how we set them not to keep the preparedstatement or any idea to work around unless clearing/deleteing these queued up PreparedStatements manually?
Tried to check in the documentation if we have any other options in the spark.read.format.
Tried deleting the queued up preparedstatements