Flying unity AI shakes

I’ve got a problem with my flying unity AI. When it just chases a target it’s ok, but my system contains avoiding obstacles so when it starts to do it, it shakes weirdly. It’s been biting me for several days, but I still don’t know how to fix it, can somebody help?

Here’s my code:

using Cinemachine;
using System.Collections;
using UnityEngine;
using UnityEngine.UI;

public class EnemyNew : MonoBehaviour
{
    [Header("Settings")]
    public int HP;
    public float speed;
    public float detectionRadius;
    public int gunsCount;
    public Vector3 stoppingDistance;
    public LayerMask obstacleLayer;
    [HideInInspector]public bool alive;
    [HideInInspector]public float distance;
    [Header("Components")]
    public Rocket target;
    private AudioSource audioSource;
    RaycastHit hit;
    void Start()
    {
        alive = true;
        audioSource = GetComponent<AudioSource>();
    }
    void FixedUpdate()
    {
        distance = Vector3.Distance(transform.position, target.transform.position);

        if(distance <= 100)
        {
            if (target.isAlive)
            {
                Vector3 direction = target.transform.position - transform.position;
                Physics.SphereCast(transform.position, transform.localScale.magnitude, direction, out hit, detectionRadius, obstacleLayer);
                if (hit.collider != null)
                {
                    Debug.Log("Avoidence");
                    Vector3 avoidanceDirection = Vector3.Cross(hit.point, hit.normal);
                    transform.position = Vector3.MoveTowards(transform.position, avoidanceDirection, speed * Time.deltaTime);
                }
                else
                {
                    Debug.Log("Movement");
                    transform.position = Vector3.MoveTowards(transform.position, target.transform.position + stoppingDistance, speed * Time.deltaTime);
                }
            }
        }
    }

As I understood using Debug.Log that happens because of simultaneous switching between chasing and avoiding obstacles.