I am trying to use the S3Hook in airflow to download a file from a bucket location on S3.
Below is my code
def s3_extract(key: str, bucket_name: str, local_path: str) -> str:
source_s3_key = key
source_s3_bucket = bucket_name
dest_file_path = local_path
source_s3 = S3Hook(aws_conn_id = "aws_conn_str")#This is my connection defined for S3Hooks
source_s3.download_file(source_s3_key, source_s3_bucket,f"{dest_file_path}/filename.txt")
file = open(f"{dest_file_path}/filename.txt","r")
text = file.read()
file.close()
return text
download_job = PythonOperator(
task_id = "s3_download_task",
python_callable = s3_extract,
op_kwargs = {
'key':'airflow/docs/filename.txt',
'bucket_name':'s3-dev-data-001',
'local_path':'usr/local/airflow'
}
)
When I try to run the above, I get the error
FileNotFoundError: [Errno 2] no such file or directory: 'usr/local/airflow/filename.txt/**airflow_tmp_90_6ogw5**'
Why is the bolded being added to the end of the destination path and what can I do in order to remove this?
Thanks