This report gives the result of running the computer algebra independent integration test.
The download section in on the main webpage contains links to download the problems in plain text format used for all CAS systems.
The number of integrals in this report is [ 163 ]. This is test number [ 212 ].
The following are the CAS systems tested:
Maxima and Fricas and Giac are called using Sagemath. This was done using Sagemath integrate command by changing the name of the algorithm to use the different CAS systems.
Sympy was run directly in Python not via sagemath.
Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions. RootSum and RootOf are not allowed.
If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.
However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.
If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.
The results given in in the table below reflects the above.
| System | % solved | % Failed |
| Rubi | 100.00 ( 163 ) | 0.00 ( 0 ) |
| Mathematica | 100.00 ( 163 ) | 0.00 ( 0 ) |
| Fricas | 100.00 ( 163 ) | 0.00 ( 0 ) |
| Maple | 97.55 ( 159 ) | 2.45 ( 4 ) |
| Giac | 96.93 ( 158 ) | 3.07 ( 5 ) |
| Mupad | 92.64 ( 151 ) | 7.36 ( 12 ) |
| Maxima | 92.64 ( 151 ) | 7.36 ( 12 ) |
| Sympy | 90.18 ( 147 ) | 9.82 ( 16 ) |
The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.
| grade |
description |
| A |
Integral was solved and antiderivative is optimal in quality and leaf size. |
| B |
Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size. |
| C |
Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons
|
| F |
Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised. |
Grading is implemented for all CAS systems. Based on the above, the following table summarizes the grading for this test suite.
| System | % A grade | % B grade | % C grade | % F grade |
| Maple | 97.55 | 0.00 | 0.00 | 2.45 |
| Rubi | 96.32 | 3.68 | 0.00 | 0.00 |
| Mathematica | 91.41 | 8.59 | 0.00 | 0.00 |
| Giac | 81.60 | 15.34 | 0.00 | 3.07 |
| Maxima | 79.75 | 12.88 | 0.00 | 7.36 |
| Fricas | 69.94 | 30.06 | 0.00 | 0.00 |
| Sympy | 58.28 | 28.22 | 3.68 | 9.82 |
| Mupad | N/A | 92.64 | 0.00 | 7.36 |
The following is a Bar chart illustration of the data in the above table.
The figure below compares the CAS systems for each grade level.
The following table shows the distribution of the different types of failure for each CAS. There are 3 types of reasons why it can fail. The first is when CAS returns back the input within the time limit, which means it could not solve it. This the typical normal failure F .
The second is due to time out. CAS could not solve the integral within the 3 minutes time limit which is assigned F(-1).
The third is due to an exception generated. Assigned F(-2). This most likely indicates an interface problem between sagemath and the CAS (applicable only to FriCAS, Maxima and Giac) or it could be an indication of an internal error in CAS. This type of error requires more investigations to determine the cause.
| System |
Number failed |
Percentage normal failure |
Percentage timeout failure |
Percentage exception failure |
| Rubi |
0 |
0.00 % |
0.00 % |
0.00 % |
| Mathematica |
0 |
0.00 % |
0.00 % |
0.00 % |
| Maple | 4 | 100.00 % | 0.00 % | 0.00 % |
| Fricas | 0 | 0.00 % | 0.00 % | 0.00 % |
| Giac |
5 |
0.00 % |
0.00 % |
100.00 % |
| Maxima |
12 |
91.67 % |
0.00 % |
8.33 % |
| Sympy |
16 |
93.75 % |
6.25 % |
0.00 % |
| Mupad |
12 |
100.00 % |
0.00 % |
0.00 % |
The table below summarizes the performance of each CAS system in terms of time used and leaf size of results.
Mean size is the average leaf size produced by the CAS (before any normalization). The Normalized mean is relative to the mean size of the optimal anti-derivative given in the input files.
For example, if CAS has Normalized mean of \(3\), then the mean size of its leaf size is 3 times as large as the mean size of the optimal leaf size.
Median size is value of leaf size where half the values are larger than this and half are smaller (before any normalization). i.e. The Middle value.
Similarly the Normalized median is relative to the median leaf size of the optimal.
For example, if a CAS has Normalized median of \(1.2\), then its median is \(1.2\) as large as the median leaf size of the optimal.
| System |
Mean time (sec) |
Mean size |
Normalized mean |
Median size |
Normalized median |
| Rubi |
0.02 |
63.88 |
1.16 |
41.00 |
1.04 |
| Mathematica |
0.04 |
53.39 |
1.28 |
38.00 |
1.00 |
| Maple |
0.03 | 47.53 | 1.06 | 37.00 | 1.03 |
| Maxima | 0.38 | 58.46 | 1.29 | 36.00 | 1.01 |
| Fricas |
0.59 |
99.93 |
2.08 |
63.00 |
1.59 |
| Sympy |
3.96 |
134.24 |
2.33 |
39.00 |
1.09 |
| Giac |
0.45 |
57.59 |
1.49 |
36.50 |
1.09 |
| Mupad |
0.08 |
42.09 |
0.98 |
33.00 |
1.00 |
The following are bar charts for the normalized leafsize and time used from the above table.
|
|
|
{}
Rubi
{}
Mathematica
{}
Maple
{}
Maxima
{}
Fricas
{}
Sympy
{}
Giac
{}
Mupad
{}
The following are integrals solved by CAS but the verification phase failed to verify the anti-derivative produced is correct. This does not mean necessarily that the anti-derivative is wrong, as additional methods of verification might be needed, or more time is needed (3 minutes time limit was used). These integrals are listed here to make it easier to do further investigation to determine why it was not possible to verify the result produced.
Rubi
{}
Mathematica
{}
Maple
Verification phase not implemented yet.
Maxima
Verification phase not implemented yet.
Fricas
Verification phase not implemented yet.
Sympy
Verification phase not implemented yet.
Giac
Verification phase not implemented yet.
Mupad
Verification phase not implemented yet.
The command AbsoluteTiming[] was used in Mathematica to obtain the elapsed time for each integrate call. In Maple, the command Usage was used as in the following example
cpu_time := Usage(assign ('result_of_int',int(expr,x)),output='realtime'
For all other CAS systems, the elapsed time to complete each integral was found by taking the difference between the time after the call completed from the time before the call was made. This was done using Python’s time.time() call.
All elapsed times shown are in seconds. A time limit of 3 CPU minutes was used for each integral. If the integrate command did not complete within this time limit, the integral was aborted and considered to have failed and assigned an F grade. The time used by failed integrals due to time out was not counted in the final statistics.
A verification phase was applied on the result of integration for Rubi and Mathematica.
Future version of this report will implement verification for the other CAS systems. For the integrals whose result was not run through a verification phase, it is assumed that the antiderivative was correct.
Verification phase also had 3 minutes time out. An integral whose result was not verified could still be correct, but further investigation is needed on those integrals. These integrals were marked in the summary table below and also in each integral separate section so they are easy to identify and locate.
The following diagram gives a high level view of the current test build system.
