S3Hook download_file function looking for airflow_tmp folder that doesn’t exist

I am trying to use the S3Hook in airflow to download a file from a bucket location on S3.

Below is my code

def s3_extract(key: str, bucket_name: str, local_path: str) -> str:
    source_s3_key = key
    source_s3_bucket = bucket_name
    dest_file_path = local_path

    source_s3 = S3Hook(aws_conn_id = "aws_conn_str")#This is my connection defined for S3Hooks

    source_s3.download_file(source_s3_key, source_s3_bucket,f"{dest_file_path}/filename.txt")
    file = open(f"{dest_file_path}/filename.txt","r")
    text = file.read()
    file.close()

    return text

download_job = PythonOperator(
    task_id = "s3_download_task",
    python_callable = s3_extract,
    op_kwargs = {
        'key':'airflow/docs/filename.txt',
        'bucket_name':'s3-dev-data-001',
        'local_path':'usr/local/airflow'
    }
    )

When I try to run the above, I get the error

FileNotFoundError: [Errno 2] no such file or directory: 'usr/local/airflow/filename.txt/**airflow_tmp_90_6ogw5**'

Why is the bolded being added to the end of the destination path and what can I do in order to remove this?

Thanks

Leave a Comment