Benchmarking
Evaluation of Electromagnetic Software
Revised: June 4, 1997
© Copyright 1994, 1999 Sonnet Software, Inc. All Rights Reserved
Table of Contents
Chapter 1 - Introduction
Chapter 2 - Technical Overview
Chapter 3 - The Stripline Standard
Chapter 4 - The Stripline Standard and Triangular Subsections
Chapter 5 - The Microstrip Standard
Chapter 6 - The Coupled Microstrip Standard
Chapter 7 - Limit Tests
Chapter 8 - Lossy Ground Plane and Lossy Dielectric Tests
GABMAC | Error Definition | Philosophy | Error and Accuracy | What Error Level Is Needed?
At Sonnet, we started working on microwave electromagnetic software in 1983. By 1989,
we considered the software ready for a production release to the general microwave design
community. Since then, Sonnet has rapidly climbed to dominance in the market niche we
describe as "3-D Planar Electromagnetic Microwave Software". We did this by
providing the fastest, most accurate electromagnetic software available anywhere in the
world. We wrote the software not only with serious attention to the underlying complex
electromagnetic theory, but also with significant effort devoted to the two additional
areas of software engineering (software testing, porting, maintenance, user interface,
etc.) and microwave engineering (obtaining and providing data in the format that the
microwave engineer finds most useful). But new customers, especially engineers, are not convinced by high sounding claims such
as those above. As the saying goes, "In God we trust, all others must have
proof." In achieving our high level of success, we have had to provide proof at every
step of every sale to every customer, one at a time. In this document, we assume that you are considering the purchase of electromagnetic
software. Our objective is to make you as informed as possible about the whole process, so
that you can decide exactly what it is that you want and whether or not a given product is
a match with your needs, whatever electromagnetic product that may be. It is common to evaluate an electromagnetic tool by applying it to an in-house circuit,
say a filter or an amplifier. We highly recommend this approach in conjunction with the
benchmarks provided in this document, however, the engineer should be aware of the
limitations of such an approach. For example, when the validation is complete, you have a measured versus calculated
plot. Normally we look at the agreement between measured and calculated data and announce
that we have "Good Agreement Between Measured And Calculated", so-called GABMAC.
Everyone walks away with a nice warm, fuzzy feeling. But what we really need to look at is the difference between measured and
calculated. This is the error. Now, how much of the error is due to measurement and how
much is due to analysis error? If the circuit is complicated, how much is due to that
capacitor and how much is due to this resistor, etc.? Also, in a large, complex circuit,
errors tend to cancel (Root Sum Squared, RSS error is what is seen). While desirable for a
given design, it does not allow us to quantitatively evaluate the intrinsic underlying
error of the analysis, which is the objective of the evaluation. Warm, fuzzy feelings are
important too, but we are engineers. We need numbers. If we are going to measure error quantitatively, we need to define it quantitatively. To calculate percent error, take the difference between the calculated value and the
correct value, divide by the correct value and multiply by 100: Percent Error = 100 × ( Calculated - Correct ) / Correct Note that percent error of S-Parameters is not useful, since the correct value of an
S-Parameter may be zero (as it is in the stripline standard to follow) and division by
zero is undefined. We have found that, for any given structure, it is best to define error in terms of an
underlying lumped model. For example if we loosely state that the S-Parameters of a step
discontinuity have 5% error, what we really mean is that the series inductance and shunt
capacitances of the equivalent underlying lumped model are in error by up to 5%. If we are dealing with a transmission line, then the equivalent infinite cascade of
series L - shunt C are in error by up to 5%. Or, more simply, the velocity of propagation
and characteristic impedance are each in error by up to 5%. No single benchmark can test all accuracy issues. In addition no set of benchmarks can
substitute for the confidence obtained from evaluating in-house designed and measured
circuits. The set of benchmarks that we provide here (and will be updating on a continuous
basis) are selected for 1) Simplicity, and 2) Accuracy. In other words, the benchmarks
must be simple to capture and the correct results must be known as accurately as possible,
preferably with exact results available. Simplicity allows the benchmarks to be quickly used on a wide variety of analyses.
Exact results allows us to eliminate measurement error. Thus, we may obtain noise-free
results on the error of an analysis and use this as a basis for comparison. One difficulty with simple benchmarks is that they do not test for circuit complexity.
Of course, in a properly performing electromagnetic analysis, accuracy does not depend on
circuit simplicity or complexity but only on subsection or mesh size. However, it would be
nice to have quantitative confirmation of this. Unfortunately, there are no complex
circuits for which we have exact results. Thus, testing for circuit complexity is left to
the GABMAC evaluation of existing in-house circuitry, and is outside the scope of this
document. In addition, as mentioned earlier, an appropriate comparison compares analysis speed at
a given level of error. It is inappropriate to compare the speed of two analyses when one
is being operated at a much higher level of accuracy. Thus, the result of a benchmark is a
plot of analysis time versus analysis error. Because the quantities can cover a wide
range, we use a log-log plot. When evaluations are performed using the same computer, such
an analysis time versus error plot is appropriate to use in a direct quantitative
comparison between two competing electromagnetic tools. Sometimes the terms "error" and "accuracy" are used
interchangeably. In reality, error is the opposite of accuracy. Small error means high
accuracy. If an analysis has a 5% error, the accuracy is 95%. A common (puffing) statement is, "Analysis accuracy is on the order of measurement
error." Note that this statements actually means the opposite of what is intended. It turns out that most electromagnetic analyses can achieve 5% to 10% error quite
easily. Also, coincidentally, this is about the same error level that a pure circuit
theory analysis can achieve on a modestly compacted circuit design. Thus, if a 5% to 10%
error level is acceptable (i.e., circuit theory gives a reasonable answer), by all means,
use circuit theory, don't bother with the time and expense of electromagnetics. The more common situation is that circuit theory works moderately well for the initial
design, but not quite well enough to achieve the highly desired success on first
fabrication. In this case you need to invoke electromagnetics on the tightly compacted
portions of the circuit, the parts of the circuit where circuit theory has problems. We
have found, through the experience of our customers, that a consistent level of about 1%
error in the critical portions of a circuit is required to realize a high probability of
success on first fabrication. When the error level creeps up to 5% in important areas of a
circuit, it can severely impact the probability of success on first fabrication. Note that a 5% error level can sneak through an un-critical GABMAC (see above)
validation. Such an error level has no chance of going undetected in the benchmarks which
follow. For these reasons, we generally recommend that when electromagnetics is invoked,
critical portions of the circuit should be analyzed as close to the 1% error level (i.e.,
use small subsection size) as possible. This does mean a slower analysis time, however, it
increases the probability of success on first fabrication dramatically, a very worthwhile
trade-off. On very rare occasions, accuracy on the order of 0.1% is needed. Electromagnetic
analyses which can achieve this level are very rare, and any analyses used at this level
need to be carefully and rigorously checked for error performance.Chapter 1 - Introduction
