How to dampen a series of values in Pandas, with more dampening if further from the average?

I have a series of values that oscillate about the average, 1.0, with a minimum of 0 and no maximum, e.g.

s = pd.Series([1.0, 1.03, 1.05, 1.2, 0.8, 0.97, 0.99])

I wish to ‘damp’ this series about the average of 1, i.e. values close to 1, such as 1.01 and 0.99 should be changed very little, but values further away, such as 0.5 and 1.2, should be brought much closer to 1. My ideal result would look something like:

[1.0, 1.029, 1.047, 1.15, 0.87, 0.975, 0.991]

Applying a standard damping function, we have a reasonable result:

s.apply(lambda x: (1 + (x - 1) / x) if x >= 1 else (1 + (x - 1) * abs(x)))
0    1.000000
1    1.029126
2    1.047619
3    1.166667
4    0.840000
5    0.970900
6    0.990100

This is good, but I want to damp extreme values more, and values close to 1 less. This question suggests use of an exponential function -e^kx, and I can see that k applies a horizontal stretch, with smaller values (less than 1) widening the drop off. I attempted to implement this:

s.apply(lambda x: 1 + (x - 1) * exp(-0.5 * x))
0    1.000000
1    1.017925
2    1.029578
3    1.109762
4    0.865936
5    0.981529
6    0.993904

Unfortunately this also damps values close to 1 too much. I also tried using a standard x^n:

s.apply(lambda x: (1 + (x - 1) / x**10) if x >= 1 else (1 + (x - 1) * abs(x**10)))
0    1.000000
1    1.022323
2    1.030696
3    1.032301
4    0.978525
5    0.977877
6    0.990956

This is closer to what I want but the problem arises that values very far from 1.0 get damped to closer than 1 than close values, e.g. here 0.8 -> 0.978525 which is greater than 0.97 -> 0.977877.

How can I create a damping that reigns in more extreme values without this problem?

Think about the curve you want to draw, which is decreasing slowly from 0 and then quickly at some higher value, likewise the inverse for negatives. You also don’t wan’t more than one turning point in your curve. You will most likely want to work with a variant of y = -x^2:

    y
____|____ x
   *|*
 *  |  *
*   |   *

You can add 1 so the curve intersects the y-axis at 1, and then add a constant in front of x^2 to stretch the curve appropriately. In this case, small values widen the curve as you want. For example y = -0.2 x^2 + 1:

      y
      |
   *******
_*____|____*__ x 
*     |     *

You can apply this to your series, by multiplying the 1 - x component by it:

s.apply(lambda x: (1 + (x - 1) * (-0.1 * x**2 + 1)))
0    1.000000
1    1.026817
2    1.044488
3    1.171200
4    0.812800
5    0.972823
6    0.990980

This is now close to the desired result and you can vary the constant in front of x^2 to adjust the values.

Do note that when the curve intersects the x-axis, your values will become negative, so adjust your constant accordingly and set a cap on values at this point.

Leave a Comment