[next] [tail] [up]

Chapter 1
Introduction

1.1 Listing of CAS systems tested
1.2 Results
1.3 Time and leaf size Performance
1.4 Performance based on number of rules Rubi used
1.5 Performance based on number of steps Rubi used
1.6 Solved integrals histogram based on leaf size of result
1.7 Solved integrals histogram based on CPU time used
1.8 Leaf size vs. CPU time used
1.9 Performance per integrand type
1.10 Maximum leaf size ratio for each CAS against the optimal result
1.11 Pass/Fail per test file for each CAS system
1.12 Timing
1.13 Verification
1.14 Important notes about some of the results
1.15 Design of the test system

This report gives the result of running the computer algebra independent integration problems.

The listing of the problems used by this report are

  1. MIT_bee_integration_problems.zip
  2. handbook_integration_problems.zip
  3. CAS_integration_tests_2023_Mathematica_format.m
  4. CAS_integration_tests_2023_Maple_and_Mupad_format.zip
  5. CAS_integration_tests_2023_SAGE_format.zip
  6. CAS_integration_tests_2023_Sympy_format.zip

The Mathematica/Rubi format file above can be read into Mathematica using the following commands 

SetDirectory[NotebookDirectory[]] (*where the above .m file was save*) 
lst=First@ReadList["CAS_integration_tests_2023_Mathematica_format.m",Expression]; 
Length[lst]

lst[[1]] will be the first integrand,var and lst[[2]] will be the second one and so on.

The Rubi test suite files were downloaded from rulebasedintegration.org.

The current number of problems in this test suite is [3809].

1.1 Listing of CAS systems tested

The following are the CAS systems tested:

  1. Mathematica 13.3.1 (August 16, 2023) on windows 10.
  2. Rubi 4.17.3 (Sept 25, 2023) on Mathematica 13.3.1 on windows 10
  3. Maple 2023.1 (July, 12, 2023) on windows 10.
  4. Maxima 5.47 (June 1, 2023) using Lisp SBCL 2.3.0 on Linux via sagemath 10.1 (Aug 20, 2023).
  5. FriCAS 1.3.9 (July 8, 2023) based on sbcl 2.3.0 on Linux via sagemath 10.1 (Aug 20, 2023).
  6. Giac/Xcas 1.9.0-57 (June 26, 2023) on Linux via sagemath 10.1 (Aug 20, 2023).
  7. Sympy 1.12 (May 10, 2023) Using Python 3.11.3 on Linux.
  8. Mupad using Matlab 2021a with Symbolic Math Toolbox Version 8.7 on windows 10.
  9. Reduce CSL rev. 6657. December 10, 2023. On Linux.

Maxima and Fricas and Giac are called using Sagemath. This was done using Sagemath integrate command by changing the name of the algorithm to use the different CAS systems.

Sympy was run directly in Python not via sagemath.

Reduce was run directly.

1.2 Results

Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions.

If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.

However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.

If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.

The results given in in the table below reflects the above.

Table 1.1: Percentage solved for each CAS
System solved Failed
Mathematica % 99.606 ( 3794 ) % 0.394 ( 15 )
Rubi % 99.396 ( 3786 ) % 0.604 ( 23 )
Fricas % 89.525 ( 3410 ) % 10.475 ( 399 )
Maple % 87.897 ( 3348 ) % 12.103 ( 461 )
Giac % 79.312 ( 3021 ) % 20.688 ( 788 )
Maxima % 75.663 ( 2882 ) % 24.337 ( 927 )
Mupad % 73.326 ( 2793 ) % 26.674 ( 1016 )
Reduce % 72.828 ( 2774 ) % 27.172 ( 1035 )
Sympy % 69.782 ( 2658 ) % 30.218 ( 1151 )

The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.

Table 1.2: Description of grading applied to integration result

grade

description

A

Integral was solved and antiderivative is optimal in quality and leaf size.

B

Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size.

C

Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons

  1. antiderivative contains a hypergeometric function and the optimal antiderivative does not.
  2. antiderivative contains a special function and the optimal antiderivative does not.
  3. antiderivative contains the imaginary unit and the optimal antiderivative does not.

F

Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised.

Grading is implemented for all CAS systems in this version except for CAS Mupad where a grade of B is automatically assigned as a place holder for all integrals it completes on time.

The following table summarizes the grading results.

Table 1.3: Antiderivative Grade distribution for each CAS
System % A grade % B grade % C grade % F grade
Rubi 97.82 0.45 1.02 0.6
Mathematica 86.87 5.09 7.53 0.39
Maple 76.48 6.22 5.09 12.1
Fricas 67.26 20.16 2. 10.48
Maxima 64.93 10.16 0.5 24.34
Giac 61.46 16.91 0.84 20.69
Sympy 44.74 14.36 10.58 30.22
Reduce N/A 73.22 0. 26.67
Mupad N/A 73.22 0. 26.67

The following Bar chart is an illustration of the data in the above table.

pict

The figure below compares the CAS systems for each grade level.

pict

1.3 Time and leaf size Performance

The table below summarizes the performance of each CAS system in terms of time used and leaf size of results.

Mean size is the average leaf size produced by the CAS (before any normalization). The Normalized mean is relative to the mean size of the optimal anti-derivative given in the input files.

For example, if CAS has Normalized mean of \(3\), then the mean size of its leaf size is 3 times as large as the mean size of the optimal leaf size.

Median size is value of leaf size where half the values are larger than this and half are smaller (before any normalization). i.e. The Middle value.

Similarly the Normalized median is relative to the median leaf size of the optimal.

For example, if a CAS has Normalized median of \(1.2\), then its median is \(1.2\) as large as the median leaf size of the optimal.

Table 1.4: Time and leaf size performance for each CAS
System Mean time (sec) Mean size Normalized mean Median size Normalized median
Reduce 0.09 112.42 1.53 36. 1.
Rubi 0.21 71.88 1. 43. 1.
Maxima 0.23 76.7 1.38 32. 0.9
Mupad 0.27 76.27 1.27 28 0.87
Mathematica 0.29 66.49 1.1 39. 1.
Fricas 0.36 156.66 1.83 41. 1.12
Maple 0.5 114.53 1.3 30. 0.87
Giac 1.9 102.39 1.6 35. 0.99
Sympy 3.2 209.21 6.4 37. 1.06

1.4 Performance based on number of rules Rubi used

This section shows how each CAS performed based on the number of rules Rubi needed to solve the same integral. One diagram is given for each CAS.

On the \(y\) axis is the percentage solved which Rubi itself needed the number of rules given the \(x\) axis. These plots show that as more rules are needed then most CAS system percentage of solving decreases which indicates the integral is becoming more complicated to solve.

1.5 Performance based on number of steps Rubi used

This section shows how each CAS performed based on the number of steps Rubi needed to solve the same integral. Note that the number of steps Rubi needed can be much higher than the number of rules, as the same rule could be used more than once.

The above diagram show that the precentage of solved intergals decreases for most CAS systems as the number of steps increases. As expected, for integrals that required less steps by Rubi, CAS systems had more success which indicates the integral was not as hard to solve. As Rubi needed more steps to solve the integral, the solved percentage decreased for most CAS systems which indicates the integral is becoming harder to solve.

1.6 Solved integrals histogram based on leaf size of result

The following shows the distribution of solved integrals for each CAS system based on leaf size of the antiderivatives produced by each CAS. It shows that most integrals solved produced leaf size less than about 100 to 150. The bin size used is \(40\).

1.7 Solved integrals histogram based on CPU time used

The following shows the distribution of solved integrals for each CAS system based on CPU time used in seconds. The bin size used is \(0.1\) second.

1.8 Leaf size vs. CPU time used

The following gives the relation between the CPU time used to solve an integral and the leaf size of the antiderivative.

The result for Fricas, Maxima and Giac is shifted more to the right than the other CAS system due to the use of sagemath to call them, which causes an initial slight delay in the timing to start the integration due to overhead of starting a new process each time.

1.9 Performance per integrand type

The following are the different integrand types the test suite contains.

  1. Independent tests.
  2. Algebraic Binomial problems (products involving powers of binomials and monomials).
  3. Algebraic Trinomial problems (products involving powers of trinomials, binomials and monomials).
  4. Miscellaneous Algebraic functions.
  5. Exponentials.
  6. Logarithms.
  7. Trigonometric.
  8. Inverse Trigonometric.
  9. Hyperbolic functions.
  10. Inverse Hyperbolic functions.
  11. Special functions.
  12. Sam Blake input file.
  13. Waldek Hebisch input file.
  14. MIT Bee integration.
  15. Few problems from Ryzhik and Gradshteyn table of integrals handbook.

The following table gives percentage solved of each CAS per integrand type.

Table 1.5: Percentage solved per integrand type
Integrand type problems Rubi Mathematica Maple Maxima Fricas Sympy Giac Reduce Mupad
Independent tests 1892 98.78 99.21 94.24 82.14 95.51 75.74 86.73 76.27 82.03
Algebraic Binomial 1917 100. 100. 81.64 69.27 83.62 63.9 71.99 69.43 64.74
Algebraic Trinomial 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Algebraic Miscellaneous 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Exponentials 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Logarithms 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Trigonometric 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Inverse Trigonometric 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Hyperbolic 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Inverse Hyperbolic 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Special functions 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Sam Blake file 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Waldek Hebisch file 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
MIT Bee integration 0 0. 0. 0. 0. 0. 0. 0. 0. 0.
Table of integrals 0 0. 0. 0. 0. 0. 0. 0. 0. 0.

In addition to the above table, for each type of integrand listed above, 3D chart is made which shows how each CAS performed on that specific integrand type.

These charts and the table above can be used to show where each CAS relative strength or weakness in the area of integration.

pict

pict

1.10 Maximum leaf size ratio for each CAS against the optimal result

The following table gives the largest ratio found in each test file, between each CAS antiderivative and the optimal antiderivative.

For each test input file, the problem with the largest ratio \(\frac {\text {CAS leaf size}}{\text {Optimal leaf size}}\) is recorded with the corresponding problem number.

In each column in the table below, the first number is the maximum leaf size ratio, and the number that follows inside the parentheses is the problem number in that specific file where this maximum ratio was found. This ratio is determined only when CAS solved the the problem and also when an optimal antiderivative is known.

If it happens that a CAS was not able to solve all the integrals in the input test file, or if it was not possible to obtain leaf size for the CAS result for all the problems in the file, then a zero is used for the ratio and -1 is used for the problem number.

This makes it easier to locate the problem. In the future, a direct link will be added as well.

#

Rubi

Mathematica

Maple

Maxima

FriCAS

Sympy

Giac

Mupad

Reduce

1

1.8 (133)

3.9 (50)

4.5 (170)

3.8 (169)

4. (45)

4789.3 (145)

4.2 (164)

0. (-1)

5.7 (61)

2

1.9 (26)

5. (26)

3.6 (17)

113.1 (21)

14.3 (13)

16.8 (5)

4.6 (2)

0. (-1)

1.8 (2)

3

1.1 (4)

2. (7)

2. (6)

11.1 (7)

2. (8)

1.9 (5)

1.9 (5)

0. (-1)

2.4 (1)

4

6.8 (5)

14.3 (13)

11.7 (8)

29.7 (8)

5.5 (43)

4.8 (40)

5.3 (1)

0. (-1)

3.4 (30)

5

2. (225)

54.7 (278)

11.9 (280)

8.1 (280)

7.7 (280)

39.8 (123)

19.5 (141)

0. (-1)

8.4 (278)

6

1. (1)

1.4 (3)

2.2 (4)

1.9 (1)

1.4 (7)

0.8 (4)

2.3 (5)

0. (-1)

1.5 (7)

7

2.2 (3)

5.6 (7)

1.8 (3)

2.8 (3)

6.7 (9)

45.4 (9)

1.9 (3)

0. (-1)

1.8 (3)

8

2.9 (70)

5.3 (31)

4.5 (57)

6.5 (11)

5. (42)

26.4 (71)

5.8 (40)

0. (-1)

7.2 (31)

9

2.2 (112)

6.8 (316)

3.5 (323)

12.1 (328)

4.2 (341)

4789.3 (251)

15. (328)

0. (-1)

5.7 (14)

10

4. (604)

10.9 (446)

367.9 (417)

36.9 (399)

93.4 (137)

124.9 (217)

18.8 (537)

0. (-1)

9.9 (197)

11

7.7 (82)

2.8 (24)

24.7 (55)

2.7 (2)

14.9 (77)

43. (17)

6.6 (50)

0. (-1)

19.9 (49)

12

1.8 (6)

2.3 (4)

1.2 (8)

1.5 (2)

3.3 (3)

3.4 (3)

1.6 (2)

0. (-1)

1.8 (8)

13

7.1 (369)

23.8 (1323)

27.7 (1323)

32.9 (1323)

32.9 (1323)

136.1 (671)

34. (1323)

0. (-1)

34.6 (1323)

1.11 Pass/Fail per test file for each CAS system

The following table gives the number of passed integrals and number of failed integrals per test number. There are 210 tests. Each tests corresponds to one input file.

#
Rubi
MMA
Maple
Maxima
FriCAS
Sympy
Giac
Mupad
Reduce
Pass Fail Pass Fail Pass Fail Pass Fail Pass Fail Pass Fail Pass Fail Pass Fail Pass Fail
1 175 0 175 0 173 2 166 9 174 1 165 10 170 5 169 6 160 15
2 33 2 34 1 28 7 16 19 25 10 9 26 17 18 9 26 11 24
3 13 1 14 0 12 2 8 6 13 1 9 5 10 4 11 3 8 6
4 48 2 50 0 33 17 26 24 50 0 19 31 41 9 12 38 23 27
5 279 5 284 0 282 2 251 33 281 3 254 30 269 15 270 14 267 17
6 3 4 7 0 5 2 3 4 7 0 5 2 5 2 7 0 4 3
7 7 2 9 0 9 0 7 2 9 0 5 4 9 0 9 0 6 3
8 113 0 113 0 113 0 111 2 112 1 107 6 111 2 106 7 109 4
9 376 0 376 0 376 0 374 2 376 0 363 13 375 1 372 4 357 19
10 704 1 705 0 656 49 565 140 662 43 460 245 590 115 542 163 463 242
11 110 6 102 14 88 28 20 96 90 26 29 87 36 80 37 79 27 89
12 8 0 8 0 8 0 7 1 8 0 8 0 8 0 8 0 8 0
13 1917 0 1917 0 1565 352 1328 589 1603 314 1225 692 1380 537 1241 676 1331 586

1.12 Timing

The command AbsoluteTiming[] was used in Mathematica to obtain the elapsed time for each integrate call. In Maple, the command Usage was used as in the following example 

cpu_time := Usage(assign ('result_of_int',int(expr,x)),output='realtime'

For all other CAS systems, the elapsed time to complete each integral was found by taking the difference between the time after the call completed from the time before the call was made. This was done using Python’s time.time() call.

All elapsed times shown are in seconds. A time limit of 3 CPU minutes was used for each integral. If the integrate command did not complete within this time limit, the integral was aborted and considered to have failed and assigned an F grade. The time used by failed integrals due to time out was not counted in the final statistics.

For Reduce CAS, since it has no support for timelimit, there was no time limit used. But the time used was still recorded.

1.13 Verification

A verification phase was applied on the result of integration for Rubi and Mathematica and Maple.

Future version of this report will implement verification for the other CAS systems. For the integrals whose result was not run through a verification phase, it is assumed that the antiderivative was correct.

Verification phase also had 3 minutes time out. An integral whose result was not verified could still be correct, but further investigation is needed on those integrals. These integrals were marked in the summary table below and also in each integral separate section so they are easy to identify and locate.

1.14 Important notes about some of the results

1.14.1 Important note about Maxima results

Since tests were run in a batch mode, and using an automated script, then any integral where Maxima needed an interactive response from the user to answer a question during the evaluation of the integral will fail.

The exception raised is ValueError. Therefore Maxima results is lower than what would result if Maxima was run directly and each question was answered correctly.

The percentage of such failures were not counted for each test file, but for an example, for the Timofeev test file, there were about 14 such integrals out of total 705, or about 2 percent. This percentage can be higher or lower depending on the specific input test file.

Such integrals can be identified by looking at the output of the integration in each section for Maxima. The exception message will indicate the cause of error.

Maxima integrate was run using SageMath with the following settings set by default 

'besselexpand : true' 
'display2d : false' 
'domain : complex' 
'keepfloat : true' 
'load(to_poly_solve)' 
'load(simplify_sum)' 
'load(abs_integrate)' 'load(diag)'

SageMath automatic loading of Maxima abs_integrate was found to cause some problems. So the following code was added to disable this effect. 

 from sage.interfaces.maxima_lib import maxima_lib 
 maxima_lib.set('extra_definite_integration_methods', '[]') 
 maxima_lib.set('extra_integration_methods', '[]')

See https://ask.sagemath.org/question/43088/integrate-results-that-are-different-from-using-maxima/ for reference.

1.14.2 Important note about FriCAS result

There were few integrals which failed due to SageMath interface and not because FriCAS system could not do the integration.

These will fail With error Exception raised: NotImplementedError.

The number of such cases seems to be very small. About 1 or 2 percent of all integrals. These can be identified by looking at the exception message given in the result.

1.14.3 Important note about finding leaf size of antiderivative

For Mathematica, Rubi, and Maple, the builtin system function LeafSize was used to find the leaf size of each antiderivative.

The other CAS systems (SageMath and Sympy) do not have special builtin function for this purpose at this time. Therefore the leaf size for Fricas and Sympy antiderivative was determined using the following function, thanks to user slelievre at https://ask.sagemath.org/question/57123/could-we-have-a-leaf_count-function-in-base-sagemath/ 

def tree_size(expr): 
    r""" 
    Return the tree size of this expression. 
    """ 
    if expr not in SR: 
        # deal with lists, tuples, vectors 
        return 1 + sum(tree_size(a) for a in expr) 
    expr = SR(expr) 
    x, aa = expr.operator(), expr.operands() 
    if x is None: 
        return 1 
    else: 
        return 1 + sum(tree_size(a) for a in aa)

For Sympy, which was called directly from Python, the following code was used to obtain the leafsize of its result 

try: 
  # 1.7 is a fudge factor since it is low side from actual leaf count 
  leafCount = round(1.7*count_ops(anti)) 
 
  except Exception as ee: 
         leafCount =1

1.14.4 Important note about Mupad results

Matlab’s symbolic toolbox does not have a leaf count function to measure the size of the antiderivative. Maple was used to determine the leaf size of Mupad output by post processing Mupad result.

Currently no grading of the antiderivative for Mupad is implemented. If it can integrate the problem, it was assigned a B grade automatically as a placeholder. In the future, when grading function is implemented for Mupad, the tests will be rerun again.

The following is an example of using Matlab’s symbolic toolbox (Mupad) to solve an integral 

integrand = evalin(symengine,'cos(x)*sin(x)') 
the_variable = evalin(symengine,'x') 
anti = int(integrand,the_variable)

Which gives sin(x)^2/2

1.15 Design of the test system

The following diagram gives a high level view of the current test build system.

[next] [front] [up]