Notation: Equation numbers are placed in parentheses, (Eq. 1), at the end of the sentence preceding the mathematical presentation of the equation. They should not be read as if they are part of the sentence. When reference is made to an equation, the word "equation" will be spelled out and not abbreviated. Mathematical symbols are not in all cases precise. Greek symbols will show as such on a Windows operating system, but not a Unix or Mac system.
Taylor Series
Most numerical methods involve the development of an approximating function for the actual function and using this approximation in place of the actual function to develop a computational scheme. Thus the numerical solution is always an approximation to the exact solution. Even though the approximation may be sufficient for the purpose, it is nevertheless important to remember that one is dealing with an approximation.
A Taylor series, which is an infinite series, approximates a given function throughout a certain small interval about a given point. In order to develop a mathematical definition, let us first define an analytic function. A function f(x) is said to be analytic at x = xo if f(x) can be represented by a power series in powers of h = ( x - xo ) within a radius of convergence D > | x - xo | > 0. A necessary condition for a function to be analytic is that all of its derivatives must by continuous at x = xo and in the neighborhood of xo . A point at which a function f(x) is not analytic is called a singular point. Polynomials are analytic everywhere.
If f(x) is analytic about x = xo , then, in a small neighborhood of x = xo, f(x) can be exactly represented in a Taylor series of the form (Eq. 1)
where f '(xo) = (df/dx)xo and similarly for higher order derivatives, and where h = (x - xo) < 1. Suppose we write this one-dimensional Taylor series in the following fashion (Eq. 2)
where O(h2+...) represents all higher order terms whose magnitude declines with increasing order, since the multiplying coefficient is a power of the number, h, that is less than one. Thus the largest of these remainder terms is the first one in h2. Each successive term is smaller than the preceding term because powers of h are decreasing numbers. We can approximate the error we would make by truncating the series by neglecting all terms of order higher than h by evaluating the h2 term, since it is the largest term.
Approximating the Derivative
Note that if we rearrange our Taylor series as (Eq. 3)
we have an expression for evaluating the derivative of f(x) at the point xo . If we now truncate by dropping the higher order terms O(h2+...), we have an approximate expression for the derivative and a truncation error which is O(h2+...). Note again that we can approximate the value of the truncation error considering only the first term among the truncated terms since it is the largest
Thus we can symbolize the approximate truncation error we make in any approximation in which we truncate terms of order n and larger by O(hn).
Examples of Numerical Derivatives
Since a derivative is analytically a limiting process in which the interval between adjacent values of the function divided by the interval is allowed to go to zero, one might intuitively believe that one makes a better approximation to the limit process by making h = x - xo as small as possible. Let us consider two examples.
First, let f(x) = 3x2, whose derivative is f '(x) = 6x, where for x = 1, f(x=1) = 3, and f '(x=1) = 6. To calculate the derivative by our numerical method above, let us retain 6 significant figures and let h = 1.0 x 10-6. Thus,
xo = 1.000000, so that f(x=1.000000) = 3,
x = xo + h = 1.000001, so that f(x=1.000001) = 3.000006
Forming the numerical derivative, we have
f '(x = 1) = [f(x) - f(xo)]/h = (3.000006 - 3.000000)/0.000001 = 6.000000,
which is the same result as the analytical result.
Second, let f(x) = sin(x), whose derivative is f '(x) = cos(x), where for x = 1 radian, f(x=1) = 0.841471, and f '(x=1) = 0.540302. To calculate the derivative by our numerical method above, let us again retain 6 significant figures and let h = 1.0 x 10-6. Thus,
xo = 1.000000, so that f(x=1.000000) = 0.841471,
x = xo + h = 1.000001, so that f(x=1.000001) = 0.841472.
Forming the numerical derivative, we have
f '(x = 1) = [f(x) - f(xo)]/h = (0.841472 - 0.841471)/0.000001 = 0.000001/0.000001 = 1.00000,
which is clearly not the correct result. Let us repeat the calculation with h = 0.0001, so that f(x=1.0001) = 0.841525
f '(x = 1) = [f(x) - f(xo)]/h = (0.841525 - 0.841471)/0.0001 = 0.000054/0.000100 = 0.540108,
which differs from the analytical result by
Truncation Error = 0.000212 .
The lesson to learn from these two examples is first that the truncation error will depend on the mathematical nature of the function under consideration. The second point is that smaller values of h do not necessarily improved the accuracy of the computation.