Chapter 1
Introduction

This report gives the result of running the computer algebra independent integration problems.

The listing of the problems used by this report are

1. MIT_bee_integration_problems.zip
2. CAS_integration_tests_2022_Mathematica_format.m
3. CAS_integration_tests_2022_Maple_and_Mupad_format.zip
4. CAS_integration_tests_2022_SAGE_format.zip
5. CAS_integration_tests_2022_Sympy_format.zip

The Mathematica/Rubi format file above can be read into Mathematica using the following commands

SetDirectory[NotebookDirectory[]] (*where the above .m file was save*) 
lst=First@ReadList["CAS_integration_tests_2022_Mathematica_format.m",Expression]; 
Length[lst]
 

lst[[1]] will be the first integrand,var and lst[[2]] will be the second one and so on.

The Rubi test suite files were downloaded from rulebasedintegration.org.

The current number of problems in this test suite is [85865].

1.1 Listing of CAS systems tested

The following are the CAS systems tested:

1. Mathematica 13.1 (June 29, 2022) on windows 10.
2. Rubi 4.16.1 (Dec 19, 2018) on Mathematica 13.0.1 on windows 10.
3. Maple 2022.1 (June 1, 2022) on windows 10.
4. Maxima 5.46 (April 13, 2022) using Lisp SBCL 2.1.11.debian on Linux via sagemath 9.6.
5. Fricas 1.3.8 (June 21, 2022) based on sbcl 2.1.11.debian on Linux via sagemath 9.6.
6. Giac/Xcas 1.9.0-13 (July 3, 2022) on Linux via sagemath 9.6.
7. Sympy 1.10.1 (March 20, 2022) Using Python 3.10.4 on Linux.
8. Mupad using Matlab 2021a with Symbolic Math Toolbox Version 8.7 on windows 10.

Maxima and Fricas and Giac are called using Sagemath. This was done using Sagemath integrate command by changing the name of the algorithm to use the different CAS systems.

Sympy was called directly from Python.

1.2 Results

Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions. RootSum and RootOf are not allowed.

If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.

However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.

If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.

The results given in in the table below reflects the above.

Table 1.1: Percentage solved for each CAS

System	solved	Failed
Mathematica	% 97.999 ( 84147 )	% 2.001 ( 1718 )
Rubi	% 94.208 ( 80892 )	% 5.792 ( 4973 )
Maple	% 84.582 ( 72626 )	% 15.418 ( 13239 )
Fricas	% 79.355 ( 68138 )	% 20.645 ( 17727 )
Giac	% 58.609 ( 50325 )	% 41.391 ( 35540 )
Maxima	% 57.048 ( 48984 )	% 42.952 ( 36881 )
Mupad	% 56.256 ( 48304 )	% 43.744 ( 37561 )
Sympy	% 42.09 ( 36141 )	% 57.91 ( 49724 )

The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.

Table 1.2: Description of grading applied to integration result

grade	description
A	Integral was solved and antiderivative is optimal in quality and leaf size.
B	Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size.
C	Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons antiderivative contains a hypergeometric function and the optimal antiderivative does not. antiderivative contains a special function and the optimal antiderivative does not. antiderivative contains the imaginary unit and the optimal antiderivative does not.
F	Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised.

Grading is implemented for all CAS systems in this version except for CAS Mupad where a grade of B is automatically assigned as a place holder for all integrals it completes on time.

The following table summarizes the grading results.

Table 1.3: Antiderivative Grade distribution for each CAS

System	% A grade	% B grade	% C grade	% F grade
Rubi	91.49	1.96	0.75	5.79
Mathematica	78.62	6.22	13.13	2.
Maple	56.45	18.49	9.64	15.42
Fricas	53.91	19.57	5.87	20.65
Maxima	43.72	11.7	1.63	42.95
Giac	42.55	15.	1.07	41.39
Sympy	28.64	9.85	3.6	57.91
Mupad	4.13	52.12	0.	43.74

The following Bar chart is an illustration of the data in the above table.

The figure below compares the CAS systems for each grade level.

1.2.1 Time and leaf size Performance

The table below summarizes the performance of each CAS system in terms of time used and leaf size of results.

Mean size is the average leaf size produced by the CAS (before any normalization). The Normalized mean is relative to the mean size of the optimal anti-derivative given in the input files.

For example, if CAS has Normalized mean of \(3\), then the mean size of its leaf size is 3 times as large as the mean size of the optimal leaf size.

Median size is value of leaf size where half the values are larger than this and half are smaller (before any normalization). i.e. The Middle value.

Similarly the Normalized median is relative to the median leaf size of the optimal.

For example, if a CAS has Normalized median of \(1.2\), then its median is \(1.2\) as large as the median leaf size of the optimal.

Table 1.4: Time and leaf size performance for each CAS

System	Mean time (sec)	Mean size	Normalized mean	Median size	Normalized median
Rubi	0.26	153.08	1.2	98.	1.
Maxima	0.52	548.38	3.89	62.	1.04
Giac	1.54	607.21	4.54	72.	1.14
Fricas	1.96	1055.34	6.03	96.	1.29
Mathematica	2.37	690.29	2.9	81.	0.97
Maple	3.32	100610.	916.3	100.	1.17
Mupad	2.72	219.68	1.99	54.	1.
Sympy	5.21	314.21	4.18	44.	1.06

1.3 Performance based on Rubi rules and leaf size histograms

1.3.1 Performance based on number of rules Rubi used

This section shows how each CAS performed based on the number of rules Rubi needed to solve the same integral.

The maximum number of rules Rubi used to solve an integral was 9.

The above diagrams shows that the precentage of solved intergals decreases as the number of rules increases.

As expected, for integrals that required less rules, CAS systems had more success which indicates the integral was not as hard to solve. As Rubi needed more rules to solve the integral, we see that the solved percentage decreased for most CAS systems which indicates the integral is becoming harder to solve.

Mathematica had the best performance where its percentage remained almost the same as Rubi’s followed by Maple, then Fricas then the other CAS systems followed.

1.3.2 Performance based on number of steps Rubi used

This section shows how each CAS performed based on the number of steps Rubi needed to solve the same integral.

The number of steps can be much higher than the number of rules, as the same rule could be used more than once.

The above diagrams shows that the precentage of solved intergals decreases as the number of steps increases.

As expected, for integrals that required less steps by Rubi, CAS systems had more success which indicates the integral was not as hard to solve. As Rubi needed more steps to solve the integral, the solved percentage decreased for most CAS systems which indicates the integral is becoming harder to solve.

Mathematica had the best performance where its solved percentage remained almost the same as Rubi’s followed by Maple, then Fricas then the other CAS systems followed.

1.3.3 Histograms based on leaf size

1.3.4 Mathematica source code used to generate the above plots

CAS.m

 
(*basic package to generate some states for Rubi Rules*) 
(*for summer 2022 CAS integration tests*) 
(*Nasser M. Abbasi, Version Oct 1, 2022. Written using Mathematica 13.1 version *) 
(*to use, do the following. *) 
(* 
<<CAS` 
db = CAS`openDB["cas_integration_tests.db"]; 
CAS`getNumberOfIntegrals[db] 
CAS`generateRubiStepsStats[db] 
CAS`generateRubiRulesStats[db] 
CAS`generateLeafStats[db] 
CAS`closeDB[db] 
*) 
 
BeginPackage["CAS`"] 
Unprotect @@ Names["CAS`*"]; 
ClearAll @@ Names["CAS`*"]; 
 
 
(* public API *) 
getNumberOfIntegrals::usage = "getNumberOfIntegrals[db]  gets number of integrals" 
openDB::usage = "openDB[folderName]  opens the database" 
closeDB::usage = "closeDB[db]  closes the database" 
generateRubiStepsStats::usage= "generateRubiStepsStats[db]  generate stats per rubi steps" 
generateRubiRulesStats::usage= "generateRubiRulesStats[db]  generate stats per rubi rules" 
generateLeafStats::usage= "generateLeafStats[db]  generate histograms for leaf size" 
 
 
(*----------------------------------*) 
 
Begin["`Private`"] 
Needs["DatabaseLink`"] 
 
(*----------------------------------*) 
 
openDB[fullPath_String]:= Module[{db},   
   db = Check[OpenSQLConnection[JDBC["SQLite", fullPath]],$Failed]; 
   db  
 
]; 
 
(*----------------------------------*) 
closeDB[db_]:= Module[{},Check[CloseSQLConnection[db],$Failed]]; 
 
(*----------------------------------*) 
   
getNumberOfIntegrals[db_] := Module[{numberOfIntegrals},   
    numberOfIntegrals = SQLExecute[db, "SELECT COUNT(*) FROM main;"][[1, 1]]; 
    numberOfIntegrals 
]; 
 
 
(*------------------------------------------*) 
(*Function to generate histogram of leaf size per cas*) 
getHistogramForLeafSize[casName_String, db_] :=  Module[{data, binWidth = 40}, 
  data = SQLExecute[db, "SELECT rubi_leafsize FROM main where " <> casName <> "_pass=1;"] // Flatten; 
 
  Histogram[data, {binWidth}, 
      Frame -> {True, True, False, False}, 
      FrameLabel -> {"Leaf size", "Number of integrals"}, 
      PlotLabel -> Style[casName, Bold], 
      ImageSize -> 300, BaseStyle -> 12] 
]; 
 
(*----------------------------------*) 
 
casSystems = {"rubi", "mma", "maple", "fricas", "giac", "maxima", "sympy", "mupad"}; 
 
(*------------------------------------------*) 
generateRubiStepsStats[db_]:=Module[ 
    {maxNumberOfSteps=20,dataPerStep,m,lbl,data,g,cas,nProblemsForTheseSteps,n}, 
      
    dataPerStep = Table[ 
        nProblemsForTheseSteps = SQLExecute[db, "SELECT COUNT(*) FROM main where rubi_number_of_steps=" <> ToString@n <> ";"][[1, 1]]; 
 
        m = SQLExecute[db, "SELECT COUNT(*) FROM main where rubi_number_of_steps=" <> ToString@n <> " and " <> cas <> "_pass=1;"][[1, 1]]; 
 
        m*100./nProblemsForTheseSteps 
        , 
        {n, 1, maxNumberOfSteps} 
        , 
        {cas, casSystems} 
    ]; 
 
 
    data = Transpose@dataPerStep; 
    lbl  = LabelStyle -> Directive[Italic, Small, Black]; 
    data = Callout[data[[#]], casSystems[[#]], {1.1*#, Above}, lbl] & /@  Range[Length@casSystems]; 
    g = ListLinePlot[data, 
           GridLines -> {Range[maxNumberOfSteps], Automatic}, 
           GridLinesStyle -> LightGray, 
           ImageSize -> 600, 
           Frame -> {True, True, False, False}, 
           FrameLabel -> {"Rubi number of steps", "% solved problems"}, 
           Axes -> False, 
           (*PlotLegends -> casSystems,*) 
           PlotRange -> {{1, maxNumberOfSteps}, Automatic}, 
           FrameTicks -> {{Automatic, None}, {Range[maxNumberOfSteps], None}}, 
           BaseStyle -> 16 
    ]; 
 
    g 
]; 
 
 
(*------------------------------------------*) 
generateRubiRulesStats[db_]:=Module[{maxNumberOfRules=9,m,lbl,data,g,cas,nProblemsForTheseRules,n}, 
 
data = Table[ 
   nProblemsForTheseRules = SQLExecute[db, "SELECT COUNT(*) FROM main where rubi_number_of_rules=" <> ToString@n <> ";"][[1, 1]]; 
   m = SQLExecute[db, "SELECT COUNT(*) FROM main where rubi_number_of_rules=" <> ToString@n <> " and " <> cas <> "_pass=1;"][[1, 1]]; 
   m*100./nProblemsForTheseRules, 
    {n, 1, maxNumberOfRules}, {cas, casSystems} 
   ]; 
 
data = Transpose@data; 
lbl   = LabelStyle -> Directive[Italic, Small, Black]; 
data = Callout[data[[#]], casSystems[[#]], {1.1*#, Above}, lbl] & /@  Range[Length@casSystems]; 
g = ListLinePlot[data, 
  GridLines -> {Range[maxNumberOfRules], Automatic}, 
  GridLinesStyle -> LightGray, 
  ImageSize -> 600, 
  Frame -> {True, True, False, False}, 
  FrameLabel -> {"Rubi number of steps", "% solved problems"}, 
  Axes -> False, 
  (*PlotLegends -> casSystems,*) 
  PlotRange -> {{1, maxNumberOfRules}, Automatic}, 
  FrameTicks -> {{Automatic, None}, {Range[maxNumberOfRules], None}}, 
  BaseStyle -> 16]; 
 
  g 
]; 
 
(*----------------------------------*) 
generateLeafStats[db_]:=Module[{p,g,lbl}, 
 
p = getHistogramForLeafSize[#, db] & /@ casSystems; 
g = Grid[Partition[p, 2], Frame -> All, FrameStyle -> LightGray]; 
lbl = Style[Column[{"Histogram showing distribution of solved integrals", 
                    "based on leaf size using bin width of 40"}], Bold, 16]; 
g = Labeled[g, lbl, Top];  
g    
   
];   
(*----------------------------------*) 
 
End[]; 
Protect @@ Names["CAS`*"]; 
EndPackage[];

1.3.5 Maple source code used to generate same plots

This version uses Maple to access to the database. It is a small module. Below is the listing of the source code and link to the driver .mw and the module .mpl

CAS_plots.mpl

CAS_plots.mw

#small Maple module to generate some stats plots for CAS integration tests 
#by Nasser M. Abbasi,. Version OCt 1, 2022. 
#Written using Maple 2022 
# 
# To use do 
#read "CAS_plots.mpl"; 
#CAS_plots:-openDB("cas_integration_tests.db"); 
#CAS_plots:-make_number_of_steps_plot(); 
#CAS_plots:-make_number_of_rules_plot(); 
#CAS_plots:-closeDB(); 
# 
#see CAS_plots.mw driver for the above commands also. 
#make sure to set currentdir() to where the SQL database is 
# 
 
CAS_plots:=module() 
 
#global to the module only. Shared among all procs 
local db:=-1; 
local cas::list:=["mma","rubi","maple","fricas","giac","maxima","sympy","mupad"]; 
 
################# 
export openDB:=proc(full_name::string) 
 
    try 
       db:=Database[SQLite]:-Open(full_name): 
    catch: 
       error lastexception; 
    end try; 
 
end proc; 
################# 
export closeDB:=proc() 
 
    if db=-1 then 
        error "database not open"; 
    else 
        Database[SQLite]:-Close(db): 
        db :=-1; 
    fi; 
 
end proc; 
 
################# 
export make_number_of_steps_plot:=proc() 
 
    local A,number_of_integrals::posint,K,M::posint,N::posint; 
    local max_number_of_steps:=20; 
    local data; 
    local number_of_problems::integer; 
    local stmt,p1,p2; 
 
    if db=-1 then 
        error "database not open"; 
    fi; 
 
    data := Matrix(max_number_of_steps,8); 
    A := Database[SQLite]:-Prepare(db, cat("SELECT COUNT(*) FROM main;")); 
    A := Database[SQLite]:-FetchAll(A); 
    number_of_integrals:=A[1,1]; 
 
    for N from 1 to max_number_of_steps do 
        stmt := Database[SQLite]:-Prepare(db, 
                 cat("SELECT COUNT(*) FROM main where rubi_number_of_steps=",String(N),";")); 
        stmt := Database[SQLite]:-FetchAll(stmt); 
        number_of_problems:=stmt[1,1]; 
 
        for M from 1 to nops(cas) do 
            data[N,M]  := get_percent_by_steps(N,number_of_problems,cas[M]); 
        od; 
    od; 
 
    p1 := dataplot(data, style=line, gridlines=true, size=[600,"golden"], 
            view=[default,0..110], 
            labels=["Number of rubi steps used","% solved"], 
            font=[Times,default,12],labelfont=[14,14]): 
    p2 := plots:-textplot( [ 
           [  3 , 105 , "rubi"] , 
           [ 12,95, "mma" ], 
           [14,87,"maple"], 
           [12 ,69, "fricas"], 
           [8.8 ,48, "giac"], 
           [16 ,47, "mupad"], 
           [8.6,31, "maxima"], 
           [16,14, "sympy"] 
     ]): 
    return plots:-display( [p1 , p2]); 
 
 
 
end proc; 
################# 
export make_number_of_rules_plot:=proc() 
local A,number_of_integrals::posint,K,conn,M::posint,N::posint,p1,p2; 
 
    local max_number_of_rules:=9; 
    local data; 
    local number_of_problems::integer; 
    local stmt; 
 
    if db=-1 then 
        error "database not open"; 
    fi; 
 
 
    data:=Matrix(max_number_of_rules,8); 
 
    A := Database[SQLite]:-Prepare(db, cat("SELECT COUNT(*) FROM main;")); 
    A := Database[SQLite]:-FetchAll(A); 
    number_of_integrals:=A[1,1]; 
 
    for N from 1 to max_number_of_rules do 
        stmt := Database[SQLite]:-Prepare(db, 
                 cat("SELECT COUNT(*) FROM main where rubi_number_of_rules=",String(N),";")); 
        stmt := Database[SQLite]:-FetchAll(stmt); 
        number_of_problems:=stmt[1,1]; 
 
        for M from 1 to nops(cas) do 
            data[N,M]:=get_percent_per_rules(N,number_of_problems,cas[M]); 
        od; 
    od; 
 
    p1 := dataplot(data, style=line, gridlines=true, size=[600,"golden"], 
              view=[default,0..110], 
              labels=["Number of rubi rules used","% solved"], 
              font=[Times,default,12], 
              labelfont=[14,14]): 
 
    p2 := plots:-textplot( [[  3 , 105 , "rubi"] , 
           [ 6,95, "mma" ], 
           [8,88,"maple"], 
           [8,88,"maple"], 
           [8 ,73, "fricas"], 
           [6.5 ,52, "giac"], 
           [4.5 ,54, "mupad"], 
           [5.1,38, "maxima"], 
           [7,17, "sympy"] 
     ]): 
    return plots:-display( [p1 , p2]); 
 
end proc: 
 
################ 
# private procs below 
################# 
local get_percent_per_rules:=proc(number_of_rules::posint, 
                            number_of_problems::posint, 
                            cas_name::string) 
    local s,p; 
 
    s := Database[SQLite]:-Prepare(db, 
                             cat("SELECT COUNT(*) FROM main where rubi_number_of_rules=", 
                                   String(number_of_rules)," and ",cas_name,"_pass=1;")); 
    s := Database[SQLite]:-FetchAll(s); 
    p :=s[1,1]; 
    return p*100./number_of_problems; 
end proc: 
################# 
local get_percent_by_steps:=proc(number_of_steps::posint, 
                           number_of_problems::posint, 
                           cas_name::string) 
    local s,p; 
 
    s := Database[SQLite]:-Prepare(db, 
              cat("SELECT COUNT(*) FROM main where rubi_number_of_steps=", 
                 String(number_of_steps)," and ",cas_name,"_pass=1;")); 
    s := Database[SQLite]:-FetchAll(s); 
    p := s[1,1]; 
    return p*100./number_of_problems; 
end proc: 
 
end module;

1.4 Performance per integrand type

The following are the different integrand types the test suite contains.

1. Independent tests.
2. Algebraic Binomial problems (products involving powers of binomials and monomials).
3. Algebraic Trinomial problems (products involving powers of trinomials, binomials and monomials).
4. Miscellaneous Algebraic functions.
5. Exponentials.
6. Logarithms.
7. Trigonometric.
8. Inverse Trigonometric.
9. Hyperbolic functions.
10. Inverse Hyperbolic functions.
11. Special functions.
12. Sam Blake input file.
13. Waldek Hebisch input file.
14. MIT Bee integration
15. Ryzhik and Gradshteyn table of integrals handbook

The following table gives percentage solved of each CAS per integrand type.

Table 1.5: Percentage solved per integrand type

Integrand type	problems	Rubi	Mathematica	Maple	Maxima	Fricas	Sympy	Giac	Mupad
Independent tests	1892	99.	99.21	93.71	82.03	95.24	73.1	86.58	82.03
Algebraic Binomial	14276	99.99	99.8	83.03	60.65	80.86	61.73	64.67	60.65
Algebraic Trinomial	10187	99.99	99.34	90.73	52.98	84.97	40.18	72.23	56.66
Algebraic Miscellaneous	1519	99.41	98.49	87.62	52.27	81.96	46.08	61.55	61.88
Exponentials	961	99.79	96.67	81.48	66.81	90.74	45.68	49.53	71.49
Logarithms	3085	99.81	97.83	58.54	56.4	58.12	34.26	47.46	43.18
Trigonometric	22551	99.91	97.66	85.85	48.62	76.52	16.23	48.33	49.39
Inverse Trigonometric	4585	99.96	97.95	83.88	37.43	49.84	37.19	42.97	38.43
Hyperbolic	5166	100.	98.26	82.69	62.04	90.77	23.87	64.94	54.72
Inverse Hyperbolic	6626	99.97	98.25	79.79	47.83	63.25	28.4	36.84	39.6
Special functions	999	100.	95.7	70.37	47.65	71.27	46.85	31.23	40.24
Sam Blake file	3154	64.87	90.87	56.75	18.07	72.26	17.66	25.43	25.68
Waldek Hebisch file	10335	63.42	96.88	97.7	93.2	99.91	94.57	86.8	90.13
MIT Bee integration	316	93.35	98.73	94.62	92.41	96.84	82.28	91.77	89.56
Table of integrals	70	100.	100.	100.	98.57	100.	100.	100.	100.

In addition to the above table, for each type of integrand listed above, 3D chart is made which shows how each CAS performed on that specific integrand type.

These charts and the table above can be used to show where each CAS relative strength or weakness in the area of integration.

1.5 Maximum leaf size ratio for each CAS against the optimal result

The following table gives the largest ratio found in each test file, between each CAS antiderivative and the optimal antiderivative.

For each test input file, the problem with the largest ratio \(\frac {\text {CAS leaf size}}{\text {Optimal leaf size}}\) is recorded with the corresponding problem number.

In each column in the table below, the first number is the maximum leaf size ratio, and the number that follows inside the parentheses is the problem number in that specific file where this maximum ratio was found. This ratio is determined only when CAS solved the the problem and also when an optimal antiderivative is known.

If it happens that a CAS was not able to solve all the integrals in the input test file, or if it was not possible to obtain leaf size for the CAS result for all the problems in the file, then a zero is used for the ratio and -1 is used for the problem number.

This makes it easier to locate the problem. In the future, a direct link will be added as well.

#	Rubi	Mathematica	Maple	Maxima	FriCAS	Sympy	Giac	Mupad
1	1. (1)	3.9 (50)	16.9 (114)	3.8 (169)	4. (45)	4789.3 (145)	4.2 (164)	42.4 (169)
2	7.3 (21)	5. (20)	3.6 (17)	113.1 (21)	14.3 (13)	16.8 (5)	4.6 (2)	3.3 (26)
3	1. (1)	1.1 (14)	17. (6)	11.1 (7)	2. (8)	1.9 (5)	1.9 (5)	11.3 (5)
4	6.4 (5)	14.3 (13)	14.7 (46)	16.6 (43)	5.5 (43)	4.8 (40)	5.3 (1)	6.9 (4)
5	1. (1)	54.7 (278)	12737.8 (278)	8.1 (280)	7.7 (280)	39.8 (123)	19.5 (141)	14.1 (204)
6	1. (1)	1.4 (3)	2.2 (4)	1.9 (1)	1.4 (7)	0.8 (4)	2.3 (5)	1.3 (3)
7	2.2 (3)	5.6 (7)	1.8 (3)	2.8 (3)	6.7 (9)	5. (2)	1.9 (3)	1.7 (3)
8	1.6 (50)	5.3 (31)	5.1 (40)	6.5 (11)	5. (42)	26.4 (71)	5.2 (70)	22.5 (70)
9	1.2 (365)	7.2 (80)	3.7 (296)	12.1 (328)	4.2 (341)	4789.3 (251)	15. (328)	6. (9)
10	3.2 (335)	45.7 (446)	3343.5 (327)	36.9 (399)	261.1 (248)	124.9 (217)	18.8 (537)	12.8 (253)
11	529. (82)	127. (82)	317. (82)	2.7 (2)	70. (82)	41.3 (17)	6.6 (50)	207. (82)
12	1.8 (6)	2.3 (4)	1.2 (8)	1.5 (2)	3.3 (3)	3.4 (3)	1.6 (2)	0.9 (8)
13	7.1 (369)	23.8 (1323)	30.9 (1323)	32.9 (1323)	32.9 (1323)	136.1 (671)	34. (1323)	38.1 (1323)
14	2. (870)	16.5 (1101)	22.6 (1101)	24.4 (1716)	21.5 (1101)	441.1 (578)	46.7 (827)	29.4 (580)
15	3.3 (97)	9.3 (99)	29.2 (111)	2.9 (119)	10.7 (21)	49.2 (119)	10. (119)	23.5 (12)
16	1. (1)	2.2 (25)	8.6 (25)	4. (25)	9. (25)	143.6 (25)	19.9 (25)	1.7 (3)
17	2.6 (35)	10.1 (67)	38. (66)	1.7 (35)	9.5 (59)	12. (4)	37.2 (53)	20.5 (5)
18	1. (3)	27.5 (31)	100. (35)	0. (-1)	2.6 (3)	0. (-1)	0. (-1)	0. (-1)
19	8.2 (664)	8.7 (663)	7.9 (196)	10. (196)	10. (196)	55.3 (528)	8. (434)	10.1 (196)
20	1.6 (254)	8.9 (59)	136.7 (70)	4.4 (73)	26.1 (159)	10.2 (24)	5.9 (69)	23.2 (32)
21	1. (655)	12.6 (337)	33.5 (754)	3.1 (313)	17.2 (1016)	32.8 (324)	8.6 (553)	30.3 (244)
22	1.3 (64)	2.6 (63)	22.5 (55)	1.3 (15)	11.3 (62)	3. (21)	3. (98)	1.7 (21)
23	1. (1)	1.3 (37)	10.4 (15)	2.1 (15)	7. (15)	53. (15)	13.8 (15)	2.5 (1)
24	1.2 (173)	1.9 (45)	3.3 (163)	3.6 (161)	5.2 (26)	51.1 (57)	4. (157)	1.8 (133)
25	8.4 (2686)	13.4 (2913)	77.7 (1993)	13.2 (2285)	3754.9 (1276)	230.3 (2266)	28.4 (2813)	16.4 (2913)
26	1.4 (342)	10.3 (306)	17.9 (265)	4. (40)	15.1 (265)	232.7 (291)	6. (292)	20.4 (256)
27	1.3 (816)	13.6 (1007)	77.4 (546)	29.1 (1063)	57.9 (267)	36.6 (124)	9.8 (1052)	22.9 (873)
28	1.2 (46)	6. (14)	51.1 (15)	2.4 (15)	28.3 (15)	302. (9)	67.8 (15)	6.1 (15)
29	1.2 (552)	3.8 (45)	13.9 (215)	10. (43)	4003.6 (171)	14.9 (577)	8.1 (591)	16.9 (171)
30	1.3 (278)	10. (328)	51.5 (297)	11.2 (348)	16.3 (197)	10. (328)	10.6 (331)	12.5 (348)
31	1. (1)	1.6 (28)	4.9 (269)	3.2 (114)	4. (269)	21.6 (269)	6.3 (269)	3.4 (190)
32	2.8 (83)	6.2 (127)	5.8 (74)	2.2 (83)	7.2 (127)	16.4 (63)	6.9 (17)	3.5 (25)
33	2. (2419)	33.1 (1256)	96.2 (557)	27. (557)	66.3 (2300)	239.3 (2549)	43.7 (2354)	79.2 (1229)
34	1.3 (1471)	11.5 (1633)	73.8 (949)	50.9 (2170)	60.6 (1627)	1537.6 (1013)	78.9 (2229)	24.6 (2005)
35	2.1 (833)	46.1 (890)	116.1 (801)	5.5 (579)	62. (796)	141.7 (925)	20. (925)	24.8 (618)
36	1. (1)	3.4 (107)	425.1 (78)	2.7 (95)	29.8 (112)	1.2 (19)	92.3 (6)	3. (100)
37	1. (129)	4.1 (35)	14197.2 (12)	6.6 (27)	62.5 (74)	8.6 (14)	5.8 (37)	9.1 (4)
38	1.8 (76)	17.7 (261)	421. (278)	93.8 (278)	88.2 (278)	232.5 (367)	119.2 (278)	101.3 (278)
39	1.7 (636)	8.8 (109)	7. (109)	5.4 (515)	33.9 (1086)	27.8 (1105)	13.7 (885)	28.1 (910)
40	1.7 (212)	9.5 (88)	27.3 (220)	6.5 (88)	61.9 (268)	31.4 (284)	59.7 (275)	18.4 (218)
41	1.9 (327)	31.4 (371)	8.5 (55)	5.6 (70)	68.2 (308)	80.3 (55)	35.3 (309)	31.1 (105)
42	1. (59)	3.6 (103)	49.4 (108)	1.4 (111)	3428.4 (21)	43. (11)	42.2 (40)	11.9 (21)
43	1.6 (135)	1.8 (51)	13.8 (37)	1.6 (131)	4782.7 (22)	119.4 (37)	20.4 (60)	25.8 (51)
44	1.9 (1)	2.4 (22)	6.4 (29)	0. (-1)	4.2 (35)	0.8 (1)	2.5 (42)	3.8 (34)
45	1. (1)	4.9 (4)	0.9 (4)	0. (-1)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
46	2.1 (154)	20. (601)	54.7 (609)	7.4 (609)	46.4 (637)	103.6 (597)	141. (596)	42.1 (312)
47	1. (1)	25.5 (83)	1.6 (15)	1.8 (68)	51. (41)	42.2 (68)	15.3 (37)	12. (37)
48	1. (67)	25.1 (143)	88.8 (96)	88.7 (96)	77.1 (93)	82.9 (93)	73.6 (96)	75.2 (93)
49	1. (1)	11. (17)	1.7 (11)	2.1 (16)	2.2 (16)	3.2 (11)	3.3 (16)	1.8 (11)
50	1. (1)	1.7 (99)	4. (72)	1.1 (72)	9.5 (102)	18.1 (72)	12.1 (79)	22.8 (88)
51	6.2 (424)	11.6 (162)	294.5 (194)	42.3 (63)	15310.8 (134)	90.8 (255)	27.1 (202)	75.1 (192)
52	4.1 (1017)	172.1 (1010)	3059.3 (1010)	5.1 (612)	40.5 (871)	163.8 (182)	41.7 (717)	25.8 (404)
53	1. (1)	1.2 (82)	9.5 (87)	2.2 (2)	2. (81)	2.5 (2)	55.7 (2)	1.8 (2)
54	1. (1)	1.6 (4)	16. (46)	2.6 (46)	5. (58)	2.2 (32)	37.8 (39)	1.8 (20)
55	1.2 (655)	5.3 (636)	38.7 (267)	125.2 (267)	28.5 (292)	11.9 (563)	53.7 (563)	14.9 (268)
56	1. (1)	1.3 (133)	83.5 (150)	4.9 (149)	5. (150)	11. (150)	10.2 (81)	2.6 (61)
57	1.5 (120)	3.9 (363)	97.5 (440)	5.1 (348)	17.4 (440)	179.8 (441)	10.8 (392)	2.5 (65)
58	1.5 (176)	12.4 (64)	837. (189)	3. (166)	10. (237)	2.6 (5)	21.6 (186)	2.2 (166)
59	1.6 (178)	39.2 (308)	376.9 (168)	7. (10)	5.9 (269)	5.5 (119)	116.7 (239)	6.6 (239)
60	1. (1)	16.8 (81)	1428.6 (228)	79.4 (81)	7.1 (212)	8. (71)	26.2 (1)	10.7 (81)
61	1.4 (39)	30.8 (67)	11.2 (75)	14.2 (44)	4.5 (15)	11.7 (12)	12.3 (34)	7. (34)
62	1.2 (367)	9.6 (340)	161.9 (62)	9.1 (340)	9. (404)	38.1 (427)	35.8 (456)	3.6 (52)
63	1.5 (390)	4.3 (45)	1142.9 (92)	7.4 (390)	32.2 (197)	34.4 (183)	13.8 (45)	5.7 (197)
64	1.2 (284)	13.1 (44)	2190.9 (91)	10.6 (23)	11.4 (91)	15.9 (189)	15.3 (28)	7.6 (90)
65	1. (1)	114.1 (497)	33.3 (493)	3.9 (111)	7.8 (301)	137.4 (62)	5.7 (105)	6.2 (210)
66	1. (1)	8.6 (249)	7.6 (83)	21.4 (209)	13.1 (209)	29.5 (193)	633.3 (22)	12. (328)
67	1. (1)	9.2 (12)	4.3 (51)	2.4 (21)	5.9 (53)	17.1 (49)	2.4 (5)	1.8 (21)
68	1. (1)	1.8 (113)	7.6 (65)	21.3 (45)	2.3 (38)	2.2 (12)	69.6 (38)	1.7 (12)
69	1. (1)	3.3 (203)	7.8 (201)	168.3 (37)	4.7 (44)	8.4 (115)	14. (217)	2.7 (37)
70	2. (615)	510.4 (349)	397.6 (608)	9. (151)	23.4 (476)	68.3 (344)	14.1 (630)	17.8 (528)
71	1. (1)	1.1 (10)	1.4 (29)	8.1 (33)	1.1 (10)	3.9 (12)	2.3 (30)	1.1 (8)
72	1.6 (103)	56.7 (138)	3.1 (17)	4. (53)	7. (201)	2.6 (40)	762. (36)	14.1 (189)
73	1.9 (621)	1029.2 (406)	8115.7 (795)	33.4 (256)	16.2 (711)	79.3 (470)	20.9 (633)	29.5 (187)
74	1.6 (1108)	1478. (937)	150.5 (174)	9.3 (46)	15.3 (937)	223.9 (697)	443.5 (175)	22.1 (1306)
75	1.3 (12)	2212.1 (38)	686.6 (48)	7.7 (16)	29.9 (35)	3.4 (1)	4.1 (22)	13.8 (39)
76	1.2 (206)	83.1 (202)	7213.4 (353)	38.2 (48)	16.7 (327)	66.6 (265)	268.7 (222)	25.5 (248)
77	1. (1)	6.7 (10)	3.9 (2)	13.5 (1)	2.6 (2)	412.4 (8)	5.4 (12)	2.4 (3)
78	1.4 (32)	156.5 (4)	4.4 (33)	3.5 (20)	2.2 (18)	2.3 (32)	1.5 (16)	2.9 (1)
79	1.8 (236)	228.2 (240)	51483.2 (593)	19. (487)	3020.2 (254)	9937.7 (81)	19.3 (510)	30.3 (320)
80	1. (1)	2.2 (2)	1.8 (2)	1.3 (2)	4.6 (1)	11.7 (4)	2.3 (2)	35.6 (4)
81	1. (1)	1.5 (16)	1.4 (10)	1. (19)	51.7 (13)	2.8 (11)	1.8 (14)	13.3 (14)
82	1. (1)	3.7 (284)	8.3 (12)	16.5 (170)	4.1 (42)	2.5 (64)	1015. (141)	3.5 (41)
83	1. (1)	3.5 (187)	8.3 (76)	12.7 (133)	6.9 (33)	4.1 (9)	627.4 (22)	2.4 (1)
84	1. (1)	2.4 (61)	3.4 (50)	788.2 (7)	6.2 (52)	6. (41)	2. (5)	1.3 (4)
85	1. (1)	1.3 (94)	4.2 (26)	4.2 (86)	1.5 (35)	6. (61)	4.3 (35)	1.1 (87)
86	4.3 (11)	4.1 (60)	6.5 (78)	3.2 (3)	4.2 (32)	35.5 (25)	3.7 (11)	16.1 (24)
87	1. (1)	1. (10)	1.4 (29)	8.1 (32)	1.1 (10)	3.8 (12)	2.3 (30)	1.1 (8)
88	1. (1)	3.2 (1)	3. (3)	4.1 (3)	4.1 (20)	0. (-1)	3. (3)	14.7 (10)
89	1.4 (370)	35.3 (773)	9.3 (642)	6489.4 (123)	7.2 (484)	23. (452)	48.9 (782)	26.6 (462)
90	1. (1)	2.8 (2)	3. (2)	0. (-1)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
91	1. (1)	3. (1)	1.3 (1)	3.7 (1)	1.8 (1)	0. (-1)	1.6 (1)	1.2 (1)
92	1.1 (40)	36.7 (454)	14.3 (436)	7944.2 (100)	8.1 (279)	36.2 (252)	7.2 (259)	20.5 (260)
93	1. (1)	53.3 (393)	8. (29)	20.3 (115)	3.4 (319)	9. (35)	1080.9 (91)	2.8 (122)
94	1.4 (940)	84.8 (1350)	18. (1154)	7808.5 (402)	7.5 (1007)	29.3 (565)	7.5 (994)	28.4 (566)
95	1.2 (81)	4.9 (91)	5.9 (69)	9.4 (53)	3020.2 (79)	1914.1 (31)	3.7 (91)	20.3 (14)
96	1. (1)	2.1 (9)	2.3 (17)	1. (2)	15.4 (14)	13.9 (4)	7. (4)	38.3 (4)
97	1. (1)	1.9 (5)	1.4 (20)	0.8 (11)	51.4 (8)	3.2 (12)	66.7 (8)	13.4 (5)
98	1. (1)	106.8 (358)	402.6 (52)	1.3 (7)	9. (251)	3. (376)	24.8 (8)	26.5 (105)
99	1. (1)	4.5 (44)	8.6 (29)	10.5 (49)	5. (54)	2.5 (24)	6.5 (22)	2.8 (16)
100	1. (1)	3.8 (44)	1.2 (15)	7.9 (52)	4.5 (39)	16.9 (21)	1.7 (21)	2.3 (28)
101	1.5 (562)	75.5 (641)	173.9 (617)	19. (393)	8.6 (80)	40. (172)	718. (543)	10.6 (560)
102	1. (1)	7. (46)	2.8 (2)	2.9 (67)	7.5 (75)	1.3 (2)	352.2 (32)	6.3 (58)
103	1.4 (891)	200.5 (678)	7841827.2 (1268)	147.2 (1121)	295.8 (1257)	68.3 (1213)	27.4 (1203)	64.1 (1121)
104	1. (1)	941.7 (463)	15282.6 (454)	144. (373)	246.2 (354)	42.2 (280)	24.3 (257)	31.3 (373)
105	1. (130)	3975.5 (145)	43.6 (123)	3. (83)	11.4 (83)	145.8 (74)	39.8 (64)	18.8 (113)
106	1. (1)	44.6 (159)	2905.5 (351)	18.1 (272)	20.7 (379)	62.9 (245)	1941.1 (43)	48. (137)
107	1. (1)	777.6 (45)	37408.6 (26)	0. (-1)	16.5 (45)	0. (-1)	0. (-1)	0. (-1)
108	1. (1)	21.6 (47)	143.1 (43)	1.3 (4)	4.9 (20)	2.6 (1)	4.2 (3)	5.9 (7)
109	1. (1)	4.5 (42)	10.3 (27)	17.4 (47)	5.2 (59)	2.4 (22)	40.3 (8)	2.2 (16)
110	1. (1)	2.5 (11)	2.5 (2)	3.3 (11)	4. (7)	1.3 (2)	2.5 (7)	6.7 (17)
111	1. (1)	2.4 (5)	3.2 (2)	4.3 (7)	3.3 (7)	1.2 (2)	2.7 (6)	8.7 (15)
112	1. (1)	3.9 (15)	17.5 (103)	1.9 (94)	3.5 (90)	35.7 (93)	2.4 (94)	31.3 (90)
113	1. (1)	23.7 (22)	35. (29)	13.6 (8)	13. (57)	59.8 (7)	17.1 (37)	21.2 (57)
114	1. (1)	1997.4 (22)	31763.8 (3)	0. (-1)	26. (27)	0. (-1)	0. (-1)	0. (-1)
115	1. (1)	14.7 (42)	47. (288)	25.9 (47)	5.7 (42)	3.3 (1)	11.7 (42)	3.9 (223)
116	1. (1)	10. (40)	4.1 (29)	15.5 (16)	5.2 (6)	0. (-1)	6.1 (18)	1.8 (18)
117	1. (1)	3.2 (18)	5.9 (73)	120.4 (20)	4.5 (68)	2.2 (53)	4.2 (20)	5.5 (15)
118	1.4 (423)	249. (874)	14.7 (578)	1460.2 (263)	7. (515)	2.6 (5)	5.8 (513)	26.5 (502)
119	1. (1)	45.2 (153)	12.6 (284)	2.9 (65)	5.8 (227)	0. (-1)	7. (196)	10.9 (218)
120	1.7 (340)	55.8 (191)	46.5 (339)	3.7 (67)	34.3 (339)	13.1 (90)	7.1 (286)	19.8 (295)
121	1.3 (115)	2602.3 (169)	1151.1 (153)	40.5 (109)	9.1 (159)	0. (-1)	5.1 (159)	20.3 (190)
122	2.2 (197)	1877.2 (240)	7.1 (238)	46.4 (130)	15.6 (263)	3. (170)	4.3 (256)	24.3 (254)
123	1.3 (265)	350.5 (634)	15.8 (385)	1629.1 (267)	8.2 (336)	2.2 (47)	6.9 (335)	20.2 (324)
124	1. (1)	3.1 (36)	24.1 (25)	14.5 (25)	2.6 (58)	2.9 (33)	2.7 (41)	6. (58)
125	1.2 (870)	383.4 (1373)	19.8 (970)	2062.5 (624)	7.3 (808)	3. (930)	7.6 (489)	28.4 (679)
126	1. (1)	66.8 (138)	544.5 (433)	49.9 (379)	27.4 (461)	11.8 (363)	15. (389)	31.6 (116)
127	1. (1)	5.6 (42)	12.4 (21)	33.4 (39)	3.8 (42)	3.1 (1)	3.1 (41)	3.7 (61)
128	1. (1)	4. (25)	5. (74)	39.4 (15)	4.6 (69)	2.2 (53)	2.9 (61)	23. (27)
129	1. (1)	5.3 (36)	19.6 (18)	6.4 (13)	8.5 (20)	0. (-1)	13.6 (15)	24.8 (49)
130	1. (1)	2.5 (8)	2.6 (8)	4.9 (8)	3.7 (14)	0. (-1)	2.2 (8)	15.8 (9)
131	1.3 (20)	3.3 (10)	1.9 (5)	3.5 (1)	5. (22)	0. (-1)	2.2 (10)	27.9 (20)
132	1. (1)	2.7 (3)	2.2 (8)	2.5 (8)	2.3 (9)	4.9 (18)	3.3 (12)	3.9 (4)
133	1. (1)	1.2 (1)	1.8 (1)	0. (-1)	1.4 (1)	0. (-1)	0. (-1)	0. (-1)
134	1. (12)	3.1 (18)	26.8 (15)	26.6 (13)	16.6 (11)	0. (-1)	5.3 (16)	21.3 (6)
135	1. (1)	29.1 (187)	5299951.7 (81)	85. (57)	7.2 (231)	6948.3 (39)	558.9 (93)	8. (233)
136	3.3 (23)	25.3 (272)	6.3 (209)	9.5 (209)	8.5 (143)	19.3 (124)	406.5 (236)	38. (124)
137	1.1 (281)	8.4 (382)	14.9 (80)	58.8 (171)	13.9 (273)	10.3 (396)	6002.2 (153)	3.1 (81)
138	1. (1)	2.7 (1)	6.9 (9)	0.4 (5)	12. (4)	1.1 (5)	0.7 (5)	2.2 (5)
139	4.3 (259)	7.8 (318)	12.8 (259)	90.9 (225)	3.5 (111)	5.6 (18)	2097.9 (70)	4.1 (224)
140	19.2 (34)	9.1 (133)	9.1 (33)	79.9 (34)	4.2 (63)	33.8 (135)	264.4 (31)	8.6 (63)
141	10.8 (759)	718.9 (434)	651.2 (860)	418.5 (198)	27.5 (503)	4789.3 (480)	5737. (605)	42. (529)
142	1.4 (107)	2.5 (95)	4.8 (156)	1.7 (155)	1.8 (7)	2.3 (11)	9.9 (145)	3. (150)
143	1. (237)	9.5 (655)	19.9 (90)	3.3 (195)	6.1 (642)	2.9 (413)	56.7 (620)	3. (662)
144	1.9 (147)	20.7 (84)	14. (55)	12. (177)	8.2 (103)	8.1 (206)	15.5 (255)	3. (12)
145	1. (1)	4.9 (41)	2.8 (156)	1.8 (155)	3. (7)	2.3 (11)	26.4 (147)	2. (150)
146	1. (1)	1.9 (10)	2.8 (13)	2.4 (11)	5.1 (33)	2. (23)	36.3 (23)	1.1 (21)
147	1. (1)	7.2 (114)	3.3 (18)	2.4 (24)	5.7 (29)	2. (58)	3.9 (31)	2.4 (27)
148	1. (1)	4.6 (83)	27.4 (148)	1.5 (165)	3.4 (112)	7.5 (105)	1.9 (134)	1.8 (21)
149	1. (1)	4.1 (25)	44.1 (20)	1.7 (8)	18.6 (21)	44.7 (8)	1.2 (21)	4.1 (7)
150	1.3 (152)	36.6 (1229)	186. (1266)	4.8 (218)	10.2 (1214)	11.6 (1159)	2.4 (1279)	7.7 (1159)
151	1. (1)	3.3 (36)	47.6 (37)	24.5 (61)	3. (30)	9.5 (12)	1. (27)	5.2 (1)
152	1.9 (344)	2.7 (248)	15.1 (180)	9.5 (180)	10. (375)	11.6 (375)	7.9 (375)	4.2 (376)
153	1.1 (117)	8.3 (63)	27.1 (147)	5.4 (67)	9. (76)	15.1 (132)	5.8 (125)	5.5 (1)
154	1.3 (109)	8.3 (173)	55.9 (142)	13.3 (107)	9. (183)	5.9 (106)	27.1 (135)	5.5 (149)
155	1. (1)	1.2 (7)	1. (2)	1. (2)	1.1 (5)	2.7 (4)	1.1 (2)	0.9 (5)
156	1.2 (68)	3.3 (25)	11.7 (105)	3.4 (31)	6.5 (105)	2.5 (12)	84.8 (69)	2. (29)
157	1. (1)	9.1 (22)	5.4 (26)	1.7 (14)	4. (24)	2.7 (8)	2.6 (2)	1.2 (5)
158	1.4 (51)	2.8 (111)	11.7 (112)	1.9 (22)	6.5 (112)	2.6 (12)	27.3 (91)	1.9 (29)
159	1. (1)	9.3 (21)	7.1 (26)	1.6 (13)	4. (23)	2.7 (8)	3.5 (26)	1.3 (5)
160	1. (1)	21.7 (430)	7.5 (379)	3.7 (327)	33.5 (496)	24.6 (231)	8.7 (6)	16.6 (489)
161	1. (1)	5.4 (53)	3.4 (98)	12.9 (90)	6.6 (20)	1.9 (10)	6.9 (29)	1.7 (50)
162	1. (1)	1.5 (24)	1.9 (28)	6. (7)	6.1 (31)	0. (-1)	1.1 (7)	0. (-1)
163	1. (271)	8.7 (365)	7.3 (126)	21.3 (134)	32.5 (87)	22. (74)	25.9 (273)	22.4 (234)
164	1.3 (16)	9.9 (394)	8.5 (60)	23.2 (454)	2699.3 (269)	9185.8 (34)	23.5 (81)	30.4 (325)
165	1. (1)	13.6 (173)	6. (1)	3.6 (1)	16.2 (36)	4.1 (8)	8.7 (6)	2.4 (16)
166	1. (1)	1.8 (38)	3.6 (79)	3.5 (5)	4.3 (108)	2.3 (12)	18. (32)	1.7 (12)
167	1. (1)	2.1 (3)	3.4 (64)	12.9 (56)	6.6 (20)	1.9 (10)	5.2 (25)	1.9 (24)
168	1. (1)	1.5 (12)	1.9 (28)	6. (7)	6.1 (31)	0. (-1)	1.1 (7)	0. (-1)
169	1. (244)	8.7 (328)	6.8 (10)	11.4 (196)	44.9 (209)	22. (152)	25.9 (246)	21.4 (180)
170	1.3 (60)	2.5 (11)	9.3 (35)	7.4 (13)	2966.3 (69)	1913.7 (26)	5. (38)	36.5 (11)
171	1. (1)	3.7 (3)	5.5 (68)	3.4 (8)	39.5 (11)	1.7 (8)	3.2 (8)	1.6 (32)
172	1.2 (109)	3.5 (212)	6.4 (205)	12.6 (188)	65.9 (200)	129.1 (63)	4.2 (102)	11.1 (85)
173	1.3 (257)	10.1 (252)	7.9 (251)	23.3 (190)	178.8 (253)	36.3 (195)	13. (219)	27.4 (106)
174	1. (1)	4.7 (48)	5.8 (37)	3.7 (8)	30.5 (47)	11.7 (27)	3.2 (8)	1.6 (8)
175	1. (1)	6.3 (113)	6.4 (210)	13. (193)	66.3 (205)	11.6 (148)	5.9 (29)	11.1 (119)
176	1. (1)	10.3 (47)	7.9 (45)	6.3 (10)	177.8 (46)	5.5 (4)	8.7 (10)	19.2 (7)
177	1. (1)	2.6 (6)	2.9 (5)	2.5 (7)	16.2 (9)	0. (-1)	2.7 (7)	1.7 (7)
178	1. (1)	3.7 (83)	3.6 (79)	2. (15)	21.3 (15)	0. (-1)	2. (31)	4.7 (79)
179	3.5 (186)	3.4 (146)	12.7 (186)	8.6 (59)	168.4 (146)	2.2 (119)	5.4 (186)	7.5 (116)
180	1.4 (54)	14.4 (168)	5.6 (171)	21.9 (158)	170.3 (218)	5. (142)	4.3 (124)	18.2 (31)
181	1. (1)	6.3 (26)	4.6 (5)	3.1 (7)	17.5 (26)	0. (-1)	2.8 (7)	2.2 (25)
182	1. (1)	5.2 (18)	3.9 (78)	2.3 (15)	28.8 (15)	0. (-1)	1.9 (5)	9.7 (81)
183	3.3 (160)	6.7 (24)	22.3 (24)	6.3 (24)	33.8 (124)	0. (-1)	9. (24)	8.2 (120)
184	1.1 (12)	3.4 (24)	5.6 (17)	6.2 (1)	96.8 (15)	0. (-1)	8.8 (22)	9.5 (1)
185	1.9 (192)	515.8 (777)	142.2 (767)	26. (100)	105. (745)	138.5 (810)	12.2 (11)	24.9 (100)
186	1. (1)	1.9 (141)	2.7 (38)	1.4 (15)	3.3 (7)	1. (22)	2.3 (19)	0.9 (5)
187	1. (1)	3.5 (230)	10. (313)	3.5 (219)	11.1 (651)	3.5 (255)	2.6 (118)	1.9 (531)
188	1. (1)	6.7 (368)	5.7 (151)	12.2 (115)	26.7 (11)	8.2 (147)	8.4 (115)	2.7 (354)
189	1. (1)	3.2 (39)	2.3 (18)	1.2 (135)	2. (7)	1.1 (135)	2. (19)	0.9 (136)
190	1.4 (20)	5.5 (516)	17.2 (93)	2.6 (22)	19.2 (508)	1.7 (528)	2.2 (528)	1.6 (347)
191	1.3 (73)	6.9 (167)	26. (291)	9.1 (93)	49. (20)	7.6 (122)	6.3 (93)	25. (279)
192	1. (1)	1.9 (31)	73.1 (172)	5.3 (202)	9.9 (216)	30.9 (63)	7.5 (1)	3.3 (200)
193	1.6 (21)	8. (18)	56.2 (20)	2.3 (40)	236.8 (32)	62.6 (8)	10.6 (1)	8.9 (28)
194	1.6 (538)	5. (156)	71.1 (235)	16.1 (244)	7.1 (516)	4.5 (307)	6.7 (15)	6.9 (244)
195	1. (43)	11.2 (42)	61.2 (46)	4.9 (9)	14.6 (37)	119.2 (37)	15.4 (37)	16. (22)
196	2. (172)	3.7 (868)	51.8 (222)	18.3 (1152)	12.2 (1368)	30.7 (997)	9.7 (1368)	11.7 (652)
197	1.7 (81)	24. (319)	24.6 (312)	4.3 (72)	10.1 (312)	7. (276)	7.6 (133)	30.3 (131)
198	1.2 (78)	24. (238)	3055.9 (185)	3.9 (95)	14.6 (108)	119.2 (108)	15.3 (108)	9.4 (176)
199	1.9 (172)	4.9 (430)	11.6 (22)	3.5 (37)	4.3 (130)	15.8 (371)	5.1 (235)	2.8 (584)
200	1. (1)	16.3 (85)	19.4 (124)	1.4 (47)	21.8 (124)	1.5 (35)	0. (-1)	1.2 (27)
201	2.8 (38)	3.6 (29)	40.3 (80)	1. (34)	9.3 (6)	1.8 (80)	3.3 (47)	9.3 (74)
202	1.2 (75)	2.9 (111)	11.3 (112)	2.1 (10)	23.6 (112)	1.2 (9)	0. (-1)	1.2 (9)
203	1.6 (55)	8.2 (13)	5.2 (66)	2.7 (31)	7.3 (71)	2. (31)	3.1 (54)	1.5 (31)
204	1. (1)	1.7 (102)	2.5 (221)	1.1 (31)	2. (140)	2.7 (221)	1.6 (18)	4.4 (48)
205	1. (1)	2.5 (57)	1.5 (92)	3.3 (136)	2.4 (60)	2. (179)	0. (-1)	0. (-1)
206	1. (1)	2.6 (41)	3.4 (134)	4.4 (88)	5.7 (117)	7.3 (69)	557.1 (66)	0. (-1)
207	1. (1)	2.5 (131)	1.3 (7)	0. (-1)	0. (-1)	8.5 (69)	0. (-1)	0. (-1)
208	1. (1)	1.3 (195)	2.4 (41)	4. (155)	2.7 (28)	4.9 (30)	0. (-1)	1.6 (155)
209	380.1 (628)	2947.4 (1688)	183057. (2420)	6.9 (1028)	14782.7 (2646)	99.5 (821)	31. (1760)	166.7 (67)
210	643. (6841)	21182. (7738)	892063.8 (10190)	1205. (6099)	145.9 (2571)	363.2 (8409)	39084.8 (5246)	131.6 (3170)
211	66.8 (46)	527.7 (105)	439.4 (224)	439.4 (224)	35.8 (64)	522.8 (105)	573.4 (253)	208.3 (46)
212	1.2 (44)	9.5 (16)	2.5 (9)	6.5 (16)	10. (23)	7. (23)	12.5 (18)	0. (-1)

1.6 Pass/Fail per test file for each CAS system

The following table gives the number of passed integrals and number of failed integrals per test number. There are 210 tests. Each tests corresponds to one input file.

#	Rubi		MMA		Maple		Maxima		FriCAS		Sympy		Giac		Mupad
	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail
1	175	0	175	0	173	2	166	9	174	1	160	15	170	5	169	6
2	33	2	35	0	28	7	16	19	25	10	7	28	17	18	9	26
3	13	1	14	0	12	2	8	6	12	2	9	5	10	4	11	3
4	48	2	50	0	33	17	24	26	48	2	19	31	41	9	12	38
5	279	5	284	0	282	2	252	32	281	3	253	31	269	15	270	14
6	3	4	7	0	5	2	3	4	7	0	5	2	5	2	7	0
7	7	2	9	0	9	0	7	2	9	0	5	4	9	0	9	0
8	113	0	113	0	113	0	111	2	112	1	105	8	109	4	106	7
9	376	0	376	0	376	0	374	2	376	0	346	30	375	1	372	4
10	705	0	705	0	655	50	564	141	660	45	437	268	590	115	542	163
11	113	3	101	15	79	37	20	96	90	26	29	87	35	81	37	79
12	8	0	8	0	8	0	7	1	8	0	8	0	8	0	8	0
13	1917	0	1917	0	1565	352	1328	589	1601	316	1214	703	1301	616	1241	676
14	3201	0	3201	0	2870	331	2051	1150	2897	304	1608	1593	2414	787	1884	1317
15	158	1	155	4	128	31	39	120	66	93	33	126	42	117	49	110
16	34	0	33	1	28	6	16	18	28	6	19	15	28	6	4	30
17	78	0	78	0	78	0	27	51	64	14	5	73	46	32	40	38
18	35	0	35	0	35	0	0	35	9	26	0	35	0	35	0	35
19	1071	0	1071	0	767	304	632	439	735	336	1023	48	616	455	695	376
20	349	0	349	0	264	85	79	270	192	157	101	248	106	243	66	283
21	1156	0	1156	0	1041	115	704	452	952	204	631	525	822	334	730	426
22	115	0	114	1	107	8	27	88	35	80	26	89	31	84	27	88
23	51	0	51	0	14	37	14	37	14	37	29	22	14	37	14	37
24	174	0	174	0	170	4	170	4	170	4	155	19	170	4	129	45
25	3078	0	3059	19	2591	487	2196	882	2625	453	2743	335	2043	1035	2228	850
26	385	0	383	2	198	187	167	218	214	171	138	247	125	260	170	215
27	1081	0	1081	0	749	332	409	672	802	279	395	686	554	527	531	550
28	46	0	46	0	12	34	12	34	12	34	20	26	12	34	12	34
29	594	0	594	0	577	17	422	172	531	63	431	163	420	174	449	145
30	454	0	454	0	385	69	153	301	319	135	115	339	261	193	193	261
31	298	0	296	2	275	23	212	86	277	21	126	172	227	71	197	101
32	143	0	143	0	113	30	108	35	113	30	47	96	111	32	132	11
33	2590	0	2582	8	2325	265	1428	1162	2311	279	1035	1555	1969	621	1589	1001
34	2646	0	2646	0	2584	62	1720	926	2549	97	1247	1399	2279	367	1685	961
35	958	0	937	21	729	229	331	627	654	304	261	697	402	556	276	682
36	123	0	123	0	121	2	67	56	111	12	43	80	91	32	53	70
37	143	0	143	0	141	2	15	128	83	60	10	133	50	93	19	124
38	400	0	393	7	388	12	291	109	352	48	142	258	347	53	195	205
39	1126	0	1126	0	1062	64	688	438	1001	125	472	654	821	305	695	431
40	412	1	407	6	399	14	113	300	245	168	186	227	187	226	184	229
41	413	0	406	7	376	37	173	240	285	128	124	289	267	146	218	195
42	111	0	111	0	111	0	83	28	91	20	45	66	106	5	106	5
43	145	0	145	0	143	2	73	72	124	21	80	65	139	6	143	2
44	42	0	40	2	40	2	0	42	9	33	6	36	5	37	1	41
45	4	0	4	0	4	0	0	4	0	4	0	4	0	4	0	4
46	664	0	662	2	496	168	303	361	535	129	274	390	435	229	360	304
47	96	0	92	4	49	47	17	79	48	48	43	53	37	59	49	47
48	156	0	147	9	137	19	69	87	121	35	77	79	110	46	122	34
49	17	0	17	0	2	15	2	15	7	10	1	16	4	13	5	12
50	140	0	139	1	136	4	24	116	130	10	47	93	109	31	72	68
51	491	3	494	0	489	5	409	85	456	38	433	61	423	71	485	9
52	1019	6	1002	23	842	183	385	640	789	236	267	758	512	513	455	570
53	98	0	98	0	78	20	64	34	93	5	42	56	56	42	58	40
54	93	0	84	9	75	18	78	15	93	0	53	40	54	39	53	40
55	768	2	747	23	630	140	500	270	686	84	344	426	366	404	576	194
56	193	0	193	0	121	72	106	87	123	70	79	114	102	91	60	133
57	456	0	449	7	332	124	245	211	280	176	260	196	200	256	146	310
58	249	0	243	6	120	129	68	181	90	159	46	203	60	189	46	203
59	314	0	298	16	193	121	238	76	210	104	111	203	186	128	200	114
60	263	0	249	14	98	165	180	83	156	107	44	219	125	138	127	136
61	106	2	108	0	27	81	68	40	41	67	20	88	35	73	35	73
62	543	4	543	4	322	225	224	323	221	326	168	379	215	332	209	338
63	641	0	621	20	352	289	391	250	393	248	201	440	351	290	326	315
64	314	0	314	0	241	73	220	94	279	35	128	186	190	124	183	131
65	538	0	538	0	443	95	243	295	420	118	102	436	213	325	248	290
66	348	0	348	0	264	84	203	145	322	26	116	232	179	169	143	205
67	72	0	72	0	47	25	32	40	47	25	32	40	39	33	36	36
68	113	0	113	0	113	0	53	60	113	0	26	87	71	42	20	93
69	357	0	346	11	245	112	270	87	305	52	113	244	188	169	129	228
70	653	0	638	15	555	98	288	365	521	132	105	548	311	342	258	395
71	36	0	36	0	34	2	34	2	36	0	20	16	34	2	16	20
72	206	2	203	5	178	30	142	66	178	30	5	203	154	54	154	54
73	837	0	820	17	640	197	218	619	580	257	163	674	482	355	344	493
74	1560	3	1515	48	1380	183	983	580	1294	269	243	1320	1215	348	1131	432
75	51	0	51	0	50	1	16	35	31	20	4	47	21	30	13	38
76	358	0	348	10	290	68	133	225	275	83	102	256	286	72	178	180
77	19	0	15	4	12	7	13	6	13	6	8	11	12	7	13	6
78	34	0	34	0	5	29	7	27	9	25	1	33	3	31	9	25
79	592	2	583	11	522	72	332	262	474	120	74	520	359	235	334	260
80	9	0	9	0	9	0	2	7	9	0	5	4	9	0	9	0
81	19	0	19	0	19	0	5	14	17	2	6	13	9	10	19	0
82	294	0	294	0	196	98	92	202	197	97	18	276	34	260	80	214
83	189	0	189	0	135	54	140	49	137	52	55	134	112	77	74	115
84	62	0	62	0	45	17	39	23	45	17	32	30	39	23	35	27
85	99	0	99	0	87	12	81	18	91	8	33	66	52	47	30	69
86	88	0	88	0	88	0	27	61	57	31	23	65	32	56	34	54
87	34	0	34	0	32	2	32	2	34	0	18	16	32	2	15	19
88	22	0	22	0	22	0	17	5	21	1	1	21	21	1	18	4
89	932	0	924	8	854	78	303	629	675	257	100	832	273	659	310	622
90	4	0	4	0	4	0	0	4	0	4	0	4	0	4	0	4
91	1	0	1	0	1	0	1	0	1	0	0	1	1	0	1	0
92	644	0	635	9	634	10	204	440	470	174	65	579	208	436	231	413
93	393	0	389	4	236	157	119	274	238	155	15	378	18	375	75	318
94	1541	0	1535	6	1533	8	489	1052	1160	381	123	1418	536	1005	629	912
95	98	0	98	0	98	0	70	28	81	17	19	79	76	22	67	31
96	21	0	21	0	21	0	2	19	18	3	6	15	19	2	19	2
97	20	0	20	0	20	0	4	16	18	2	5	15	20	0	20	0
98	387	0	386	1	267	120	137	250	206	181	18	369	84	303	122	265
99	62	1	63	0	58	5	49	14	63	0	28	35	35	28	32	31
100	66	0	66	0	36	30	61	5	48	18	36	30	36	30	38	28
101	700	0	700	0	580	120	405	295	573	127	124	576	262	438	369	331
102	91	0	90	1	83	8	79	12	83	8	8	83	82	9	83	8
103	1328	0	1207	121	1116	212	577	751	950	378	295	1033	517	811	835	493
104	855	0	797	58	780	75	428	427	621	234	209	646	270	585	529	326
105	171	0	169	2	122	49	84	87	84	87	63	108	84	87	103	68
106	499	0	497	2	412	87	269	230	407	92	95	404	310	189	283	216
107	51	0	51	0	41	10	0	51	16	35	0	51	0	51	0	51
108	52	0	52	0	37	15	37	15	21	31	8	44	17	35	26	26
109	61	0	61	0	58	3	49	12	61	0	28	33	35	26	28	33
110	23	0	23	0	23	0	19	4	23	0	6	17	23	0	23	0
111	19	0	19	0	19	0	15	4	19	0	4	15	19	0	19	0
112	106	0	105	1	103	3	79	27	31	75	2	104	3	103	103	3
113	64	0	64	0	63	1	21	43	64	0	11	53	53	11	39	25
114	32	0	32	0	26	6	0	32	16	16	0	32	0	32	0	32
115	299	0	299	0	226	73	93	206	199	100	25	274	36	263	78	221
116	46	0	46	0	42	4	36	10	46	0	20	26	24	22	24	22
117	83	0	79	4	51	32	48	35	63	20	37	46	43	40	47	36
118	879	0	869	10	735	144	321	558	609	270	49	830	269	610	323	556
119	305	1	304	2	267	39	175	131	237	69	8	298	191	115	193	113
120	364	1	345	20	331	34	213	152	260	105	40	325	254	111	181	184
121	240	1	227	14	218	23	96	145	145	96	6	235	130	111	56	185
122	286	0	273	13	265	21	166	120	237	49	1	285	224	62	191	95
123	634	0	634	0	586	48	212	422	458	176	8	626	209	425	195	439
124	70	0	70	0	70	0	48	22	70	0	3	67	46	24	49	21
125	1373	0	1338	35	1263	110	510	863	1034	339	11	1362	545	828	552	821
126	468	2	424	46	431	39	286	184	402	68	25	445	275	195	243	227
127	70	0	70	0	53	17	28	42	53	17	9	61	31	39	16	54
128	84	0	80	4	52	32	51	33	64	20	37	47	44	40	47	37
129	59	0	53	6	41	18	25	34	41	18	3	56	41	18	33	26
130	16	0	16	0	16	0	12	4	16	0	0	16	16	0	16	0
131	23	0	23	0	23	0	18	5	23	0	0	23	23	0	23	0
132	24	0	24	0	24	0	24	0	24	0	9	15	24	0	24	0
133	1	0	1	0	1	0	0	1	1	0	0	1	0	1	0	1
134	27	0	27	0	27	0	17	10	27	0	0	27	20	7	8	19
135	254	0	252	2	215	39	159	95	214	40	70	184	161	93	169	85
136	294	0	294	0	289	5	271	23	290	4	67	227	281	13	290	4
137	397	0	396	1	359	38	341	56	365	32	126	271	246	151	155	242
138	9	0	9	0	9	0	1	8	9	0	1	8	1	8	1	8
139	330	0	305	25	107	223	141	189	171	159	69	261	90	240	149	181
140	140	2	142	0	114	28	114	28	115	27	41	101	63	79	50	92
141	944	6	938	12	908	42	655	295	910	40	423	527	725	225	700	250
142	227	0	226	1	217	10	75	152	85	142	100	127	163	64	75	152
143	703	0	694	9	554	149	265	438	265	438	208	495	239	464	146	557
144	472	2	464	10	377	97	123	351	204	270	164	310	246	228	89	385
145	227	0	226	1	215	12	75	152	85	142	100	127	163	64	73	154
146	33	0	33	0	30	3	12	21	15	18	11	22	15	18	3	30
147	118	0	114	4	78	40	30	88	51	67	34	84	51	67	22	96
148	166	0	163	3	145	21	93	73	92	74	96	70	85	81	108	58
149	31	0	26	5	30	1	14	17	11	20	11	20	6	25	14	17
150	1301	0	1279	22	1202	99	423	878	565	736	575	726	464	837	754	547
151	70	0	66	4	69	1	37	33	28	42	24	46	9	61	30	40
152	385	0	368	17	203	182	134	251	286	99	86	299	117	268	147	238
153	153	0	153	0	134	19	87	66	143	10	52	101	59	94	55	98
154	234	0	228	6	228	6	143	91	168	66	82	152	111	123	108	126
155	12	0	12	0	6	6	1	11	6	6	3	9	1	11	6	6
156	174	0	169	5	139	35	85	89	112	62	69	105	96	78	53	121
157	50	0	49	1	37	13	18	32	28	22	13	37	27	23	10	40
158	178	0	172	6	146	32	86	92	114	64	65	113	93	85	57	121
159	49	0	49	0	36	13	15	34	27	22	12	37	25	24	12	37
160	502	0	475	27	341	161	289	213	465	37	123	379	198	304	209	293
161	102	0	101	1	80	22	84	18	78	24	31	71	54	48	29	73
162	33	0	33	0	31	2	31	2	33	0	9	24	31	2	9	24
163	369	0	369	0	319	50	266	103	353	16	120	249	278	91	221	148
164	525	0	501	24	488	37	196	329	444	81	77	448	290	235	247	278
165	183	0	181	2	111	72	143	40	150	33	62	121	103	80	70	113
166	111	0	111	0	111	0	64	47	111	0	26	85	71	40	20	91
167	68	0	68	0	58	10	62	6	60	8	23	45	43	25	21	47
168	33	0	33	0	31	2	31	2	33	0	9	24	31	2	9	24
169	336	0	336	0	297	39	208	128	325	11	103	233	258	78	190	146
170	85	0	84	1	85	0	34	51	72	13	18	67	47	38	55	30
171	77	0	71	6	69	8	63	14	64	13	30	47	46	31	39	38
172	247	0	247	0	207	40	151	96	209	38	68	179	187	60	175	72
173	263	0	263	0	249	14	177	86	260	3	42	221	230	33	185	78
174	61	0	60	1	58	3	55	6	61	0	28	33	35	26	28	33
175	224	0	224	0	164	60	105	119	179	45	33	191	137	87	131	93
176	53	0	53	0	43	10	16	37	50	3	7	46	27	26	32	21
177	16	0	16	0	8	8	5	11	12	4	3	13	4	12	4	12
178	84	0	80	4	49	35	39	45	64	20	34	50	44	40	47	37
179	201	0	192	9	140	61	90	111	179	22	11	190	115	86	94	107
180	220	0	220	0	180	40	147	73	217	3	10	210	134	86	121	99
181	29	0	29	0	19	10	13	16	25	4	4	25	8	21	8	21
182	83	0	76	7	48	35	55	28	63	20	34	49	43	40	47	36
183	175	0	175	0	136	39	109	66	166	9	0	175	107	68	91	84
184	27	0	27	0	14	13	10	17	27	0	0	27	20	7	5	22
185	1059	0	1051	8	936	123	762	297	989	70	328	731	814	245	740	319
186	156	0	156	0	99	57	51	105	43	113	48	108	36	120	30	126
187	663	0	662	1	469	194	259	404	241	422	196	467	95	568	133	530
188	370	1	364	7	244	127	116	255	157	214	97	274	98	273	78	293
189	166	0	166	0	111	55	57	109	52	114	53	113	39	127	32	134
190	569	0	555	14	471	98	245	324	237	332	162	407	106	463	144	425
191	295	1	287	9	194	102	82	214	130	166	81	215	84	212	64	232
192	243	0	231	12	199	44	154	89	147	96	85	158	127	116	128	115
193	49	0	48	1	48	1	29	20	17	32	10	39	17	32	17	32
194	538	0	536	2	509	29	270	268	259	279	145	393	177	361	175	363
195	62	0	60	2	61	1	34	28	17	45	15	47	17	45	17	45
196	1378	0	1353	25	1099	279	620	758	1118	260	457	921	668	710	698	680
197	361	0	360	1	342	19	268	93	350	11	98	263	254	107	239	122
198	300	0	294	6	273	27	246	54	223	77	109	191	152	148	153	147
199	935	0	914	21	784	151	505	430	841	94	206	729	444	491	518	417
200	190	0	185	5	154	36	86	104	121	69	49	141	45	145	52	138
201	100	0	96	4	74	26	21	79	73	27	2	98	4	96	56	44
202	178	0	172	6	103	75	84	94	114	64	37	141	46	132	49	129
203	71	0	71	0	53	18	42	29	51	20	32	39	32	39	41	30
204	311	0	300	11	179	132	140	171	258	53	198	113	131	180	203	108
205	218	0	190	28	154	64	118	100	190	28	118	100	60	158	60	158
206	136	0	134	2	118	18	57	79	126	10	52	84	71	65	34	102
207	136	0	136	0	104	32	34	102	34	102	52	84	34	102	34	102
208	198	0	196	2	148	50	127	71	104	94	48	150	16	182	71	127
209	2046	1108	2866	288	1790	1364	570	2584	2279	875	557	2597	802	2352	810	2344
210	6554	3781	10013	322	10097	238	9632	703	10326	9	9774	561	8971	1364	9315	1020
211	295	21	312	4	299	17	292	24	306	10	260	56	290	26	283	33
212	70	0	70	0	70	0	69	1	70	0	70	0	70	0	70	0

1.7 Timing

The command AbsoluteTiming[] was used in Mathematica to obtain the elapsed time for each integrate call. In Maple, the command Usage was used as in the following example

cpu_time := Usage(assign ('result_of_int',int(expr,x)),output='realtime'

For all other CAS systems, the elapsed time to complete each integral was found by taking the difference between the time after the call completed from the time before the call was made. This was done using Python’s time.time() call.

All elapsed times shown are in seconds. A time limit of 3 CPU minutes was used for each integral. If the integrate command did not complete within this time limit, the integral was aborted and considered to have failed and assigned an F grade. The time used by failed integrals due to time out was not counted in the final statistics.

1.8 Verification

A verification phase was applied on the result of integration for Rubi and Mathematica.

Future version of this report will implement verification for the other CAS systems. For the integrals whose result was not run through a verification phase, it is assumed that the antiderivative was correct.

Verification phase also had 3 minutes time out. An integral whose result was not verified could still be correct, but further investigation is needed on those integrals. These integrals were marked in the summary table below and also in each integral separate section so they are easy to identify and locate.

1.9 Important notes about some of the results

1.9.1 Important note about Maxima results

Since tests were run in a batch mode, and using an automated script, then any integral where Maxima needed an interactive response from the user to answer a question during the evaluation of the integral will fail.

The exception raised is ValueError. Therefore Maxima results is lower than what would result if Maxima was run directly and each question was answered correctly.

The percentage of such failures were not counted for each test file, but for an example, for the Timofeev test file, there were about 14 such integrals out of total 705, or about 2 percent. This percentage can be higher or lower depending on the specific input test file.

Such integrals can be identified by looking at the output of the integration in each section for Maxima. The exception message will indicate the cause of error.

Maxima integrate was run using SageMath with the following settings set by default

'besselexpand : true' 
'display2d : false' 
'domain : complex' 
'keepfloat : true' 
'load(to_poly_solve)' 
'load(simplify_sum)' 
'load(abs_integrate)' 'load(diag)'
 

SageMath automatic loading of Maxima abs_integrate was found to cause some problems. So the following code was added to disable this effect.

 from sage.interfaces.maxima_lib import maxima_lib 
 maxima_lib.set('extra_definite_integration_methods', '[]') 
 maxima_lib.set('extra_integration_methods', '[]')
 

See https://ask.sagemath.org/question/43088/integrate-results-that-are-different-from-using-maxima/ for reference.

1.9.2 Important note about FriCAS result

There were few integrals which failed due to SageMath interface and not because FriCAS system could not do the integration.

These will fail With error Exception raised: NotImplementedError.

The number of such cases seems to be very small. About 1 or 2 percent of all integrals. These can be identified by looking at the exception message given in the result.

1.9.3 Important note about finding leaf size of antiderivative

For Mathematica, Rubi, and Maple, the builtin system function LeafSize was used to find the leaf size of each antiderivative.

The other CAS systems (SageMath and Sympy) do not have special builtin function for this purpose at this time. Therefore the leaf size for Fricas and Sympy antiderivative was determined using the following function, thanks to user slelievre at https://ask.sagemath.org/question/57123/could-we-have-a-leaf_count-function-in-base-sagemath/

def tree_size(expr): 
    r""" 
    Return the tree size of this expression. 
    """ 
    if expr not in SR: 
        # deal with lists, tuples, vectors 
        return 1 + sum(tree_size(a) for a in expr) 
    expr = SR(expr) 
    x, aa = expr.operator(), expr.operands() 
    if x is None: 
        return 1 
    else: 
        return 1 + sum(tree_size(a) for a in aa)
 

For Sympy, which was called directly from Python, the following code was used to obtain the leafsize of its result

try: 
  # 1.7 is a fudge factor since it is low side from actual leaf count 
  leafCount = round(1.7*count_ops(anti)) 
 
  except Exception as ee: 
         leafCount =1
 

1.9.4 Important note about Mupad results

Matlab’s symbolic toolbox does not have a leaf count function to measure the size of the antiderivative. Maple was used to determine the leaf size of Mupad output by post processing Mupad result.

Currently no grading of the antiderivative for Mupad is implemented. If it can integrate the problem, it was assigned a B grade automatically as a placeholder. In the future, when grading function is implemented for Mupad, the tests will be rerun again.

The following is an example of using Matlab’s symbolic toolbox (Mupad) to solve an integral

integrand = evalin(symengine,'cos(x)*sin(x)') 
the_variable = evalin(symengine,'x') 
anti = int(integrand,the_variable)
 

Which gives sin(x)^2/2

1.10 Design of the test system

The following diagram gives a high level view of the current test build system.