This PR attempts to measure the performance impact of #890 as discussed in #894 (comment). I just introduced a performance profiling test in #923 which we will use to compare.
- The stats for each benchmark are computed from 10,000 rounds, with 10 runs in each round. There are also 50 warm-up rounds.
- The profiling test job was retried multiple times for each referenced commit.
- The summary title of each job's expandable section refers to the median timings.
- All timings are in microseconds.
- Legend:
- Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
- OPS: Operations Per Second, computed as 1 / Mean
The current stats from most recent commit ( 0191e29 ) on main are as follows:
https://github.com/jcrist/msgspec/actions/runs/19278707083/job/55124831438
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 201.1243 (1.0) | 367.4259 (1.0) | 207.9662 (1.0) | 8.8061 (1.0) | 207.0729 (1.0) | 5.2049 (1.44) | 217;186 | 4.8085 (1.0) | 10000 | 10 |
| decode | 466.6362 (2.32) | 842.8725 (2.29) | 474.6967 (2.28) | 14.2482 (1.62) | 472.8765 (2.28) | 3.6026 (1.0) | 163;428 | 2.1066 (0.44) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19278707083/job/55151162634
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 203.7415 (1.0) | 464.0016 (1.0) | 209.6607 (1.0) | 8.7269 (1.0) | 208.4342 (1.0) | 2.5723 (1.0) | 179;548 | 4.7696 (1.0) | 10000 | 10 |
| decode | 467.9630 (2.30) | 962.4667 (2.07) | 475.4964 (2.27) | 16.5374 (1.89) | 473.5738 (2.27) | 2.9682 (1.15) | 128;633 | 2.1031 (0.44) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19278707083/job/55213730802
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 200.3558 (1.0) | 356.4271 (1.0) | 204.9474 (1.0) | 8.4551 (1.0) | 203.9945 (1.0) | 1.7164 (1.0) | 144;609 | 4.8793 (1.0) | 10000 | 10 |
| decode | 469.4412 (2.34) | 906.2031 (2.54) | 476.8766 (2.33) | 9.8599 (1.17) | 475.7701 (2.33) | 3.0983 (1.81) | 174;555 | 2.0970 (0.43) | 10000 | 10 |
The first commit of this PR (39679f2 (#927)) reverts the relevant code changes to the state before StructMeta was exposed ( 7acceff ). I manually removed the import in src/msgspec/__init__.py and then ran:
git show 7acceff82601a18f0bd5284da27eaf8279f753c8:msgspec/_core.c > src/msgspec/_core.c
git show 7acceff82601a18f0bd5284da27eaf8279f753c8:msgspec/__init__.pyi > src/msgspec/__init__.pyi
https://github.com/jcrist/msgspec/actions/runs/19279043046/job/55125796315
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 198.8778 (1.0) | 369.6624 (1.0) | 202.8351 (1.0) | 6.8731 (1.0) | 201.8113 (1.0) | 1.3997 (1.0) | 230;734 | 4.9301 (1.0) | 10000 | 10 |
| decode | 464.6508 (2.34) | 933.8461 (2.53) | 471.9030 (2.33) | 11.5599 (1.68) | 470.4537 (2.33) | 2.9133 (2.08) | 170;653 | 2.1191 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19279043046/job/55150123158
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 197.8401 (1.0) | 528.8221 (1.0) | 202.3317 (1.0) | 9.2803 (1.0) | 201.3001 (1.0) | 1.9451 (1.0) | 119;524 | 4.9424 (1.0) | 10000 | 10 |
| decode | 464.6451 (2.35) | 902.2993 (1.71) | 471.8639 (2.33) | 11.1040 (1.20) | 470.7170 (2.34) | 3.2227 (1.66) | 179;551 | 2.1193 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19279043046/job/55150356878
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 199.5605 (1.0) | 354.7804 (1.0) | 203.8711 (1.0) | 7.1735 (1.0) | 202.9358 (1.0) | 1.8154 (1.0) | 170;569 | 4.9051 (1.0) | 10000 | 10 |
| decode | 465.3009 (2.33) | 754.9604 (2.13) | 472.9567 (2.32) | 7.2304 (1.01) | 471.8717 (2.33) | 3.3296 (1.83) | 346;500 | 2.1144 (0.43) | 10000 | 10 |
The second commit of this PR (12ca746 (#927)) adds back the StructMeta changes but leaves the older conditional statements.
https://github.com/jcrist/msgspec/actions/runs/19279328219/job/55126685658
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 198.6341 (1.0) | 349.1181 (1.0) | 202.7256 (1.0) | 3.9781 (1.0) | 202.0596 (1.0) | 1.7832 (1.0) | 432;621 | 4.9328 (1.0) | 10000 | 10 |
| decode | 459.5138 (2.31) | 848.7020 (2.43) | 466.6248 (2.30) | 8.8354 (2.22) | 465.4165 (2.30) | 3.2405 (1.82) | 234;530 | 2.1430 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19279328219/job/55220713879
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 199.4497 (1.0) | 356.6277 (1.0) | 203.2230 (1.0) | 8.1591 (1.0) | 202.1670 (1.0) | 1.0973 (1.0) | 129;843 | 4.9207 (1.0) | 10000 | 10 |
| decode | 461.3140 (2.31) | 818.2023 (2.29) | 469.0633 (2.31) | 10.5852 (1.30) | 467.7030 (2.31) | 3.0546 (2.78) | 194;614 | 2.1319 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19279328219/job/55221019147
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 197.5292 (1.0) | 363.9527 (1.0) | 201.6201 (1.0) | 4.0040 (1.0) | 200.9772 (1.0) | 1.6488 (1.0) | 457;712 | 4.9598 (1.0) | 10000 | 10 |
| decode | 461.9943 (2.34) | 955.5254 (2.63) | 470.8668 (2.34) | 15.9028 (3.97) | 469.2514 (2.33) | 3.1925 (1.94) | 127;479 | 2.1237 (0.43) | 10000 | 10 |
The third commit of this PR (b0034bf (#927)) only adds back the Python import of StructMeta in src/msgspec/__init__.py, in case that for some reason makes a difference.
https://github.com/jcrist/msgspec/actions/runs/19279428342/job/55127014303
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 200.2113 (1.0) | 339.6793 (1.0) | 203.9622 (1.0) | 6.7081 (1.0) | 203.1143 (1.0) | 1.3481 (1.0) | 143;614 | 4.9029 (1.0) | 10000 | 10 |
| decode | 466.8242 (2.33) | 889.5582 (2.62) | 473.0216 (2.32) | 8.4826 (1.26) | 471.9352 (2.32) | 2.5813 (1.91) | 212;695 | 2.1141 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19279428342/job/55219914604
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 202.1548 (1.0) | 362.9893 (1.0) | 206.5176 (1.0) | 6.9384 (1.13) | 205.6086 (1.0) | 1.9924 (1.0) | 178;522 | 4.8422 (1.0) | 10000 | 10 |
| decode | 462.6665 (2.29) | 730.6224 (2.01) | 469.9820 (2.28) | 6.1526 (1.0) | 469.1426 (2.28) | 3.2699 (1.64) | 469;492 | 2.1277 (0.44) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19279428342/job/55220536395
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 200.6007 (1.0) | 358.5063 (1.0) | 204.8942 (1.0) | 8.1759 (1.0) | 203.7604 (1.0) | 2.0921 (1.0) | 163;553 | 4.8806 (1.0) | 10000 | 10 |
| decode | 466.5682 (2.33) | 820.0344 (2.29) | 473.9527 (2.31) | 10.8879 (1.33) | 472.6652 (2.32) | 3.2736 (1.56) | 188;605 | 2.1099 (0.43) | 10000 | 10 |
The fourth commit of this PR (90cfce6 (#927)) restored the final 2 code files to the way they are on main. Then, in src/msgspec/_core.c I tried refactoring the conditional logic a bit to be more maintainable/DRY.
https://github.com/jcrist/msgspec/actions/runs/19280475608/job/55130310717
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 201.2467 (1.0) | 359.8429 (1.0) | 205.6013 (1.0) | 9.8727 (1.0) | 204.3880 (1.0) | 1.6625 (1.0) | 139;639 | 4.8638 (1.0) | 10000 | 10 |
| decode | 482.1533 (2.40) | 921.6361 (2.56) | 489.7668 (2.38) | 12.7218 (1.29) | 487.9419 (2.39) | 3.6917 (2.22) | 199;595 | 2.0418 (0.42) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19280475608/job/55130695429
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 201.9173 (1.0) | 355.1093 (1.0) | 205.3439 (1.0) | 7.3993 (1.0) | 204.3767 (1.0) | 1.7447 (1.0) | 182;639 | 4.8699 (1.0) | 10000 | 10 |
| decode | 474.3272 (2.35) | 838.1676 (2.36) | 480.9436 (2.34) | 8.7424 (1.18) | 479.7496 (2.35) | 3.1857 (1.83) | 253;629 | 2.0792 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19280475608/job/55219760315
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 186.4530 (1.0) | 312.4932 (1.0) | 188.7505 (1.0) | 2.6518 (1.0) | 188.4990 (1.0) | 0.8344 (1.0) | 339;549 | 5.2980 (1.0) | 10000 | 10 |
| decode | 437.3303 (2.35) | 854.9992 (2.74) | 442.5566 (2.34) | 9.4223 (3.55) | 441.3936 (2.34) | 2.2717 (2.72) | 116;738 | 2.2596 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19280475608/job/55221584281
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 196.6335 (1.0) | 386.5246 (1.0) | 201.3585 (1.0) | 8.9834 (1.07) | 200.0785 (1.0) | 2.3201 (1.0) | 131;537 | 4.9663 (1.0) | 10000 | 10 |
| decode | 472.6448 (2.40) | 878.2723 (2.27) | 479.7974 (2.38) | 8.4194 (1.0) | 478.5824 (2.39) | 3.0813 (1.33) | 300;649 | 2.0842 (0.42) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19280475608/job/55222104467
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 198.6067 (1.0) | 350.5100 (1.0) | 202.7252 (1.0) | 3.4900 (1.0) | 202.2184 (1.0) | 1.8860 (1.0) | 472;483 | 4.9328 (1.0) | 10000 | 10 |
| decode | 476.2773 (2.40) | 832.8137 (2.38) | 483.1080 (2.38) | 8.0567 (2.31) | 481.9860 (2.38) | 2.9876 (1.58) | 243;535 | 2.0699 (0.42) | 10000 | 10 |
The fifth commit of this PR (2e79e6e (#927)) improves the previous refactor to be faster.
https://github.com/jcrist/msgspec/actions/runs/19284323563/job/55141735437
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 201.8770 (1.0) | 349.1732 (1.0) | 205.3730 (1.0) | 6.6212 (1.0) | 204.4297 (1.0) | 2.0717 (1.0) | 206;522 | 4.8692 (1.0) | 10000 | 10 |
| decode | 468.0291 (2.32) | 825.7441 (2.36) | 475.0507 (2.31) | 10.6493 (1.61) | 473.7804 (2.32) | 3.2670 (1.58) | 182;542 | 2.1050 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19284323563/job/55142261913
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 204.0911 (1.0) | 351.3634 (1.0) | 207.2974 (1.0) | 6.9403 (1.0) | 206.1753 (1.0) | 1.7064 (1.0) | 244;771 | 4.8240 (1.0) | 10000 | 10 |
| decode | 469.2734 (2.30) | 891.3440 (2.54) | 477.4993 (2.30) | 16.5942 (2.39) | 475.5152 (2.31) | 3.1734 (1.86) | 174;613 | 2.0942 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19284323563/job/55142778982
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 202.0598 (1.0) | 350.2796 (1.0) | 206.6133 (1.0) | 5.2972 (1.0) | 206.0142 (1.0) | 2.1084 (1.0) | 308;534 | 4.8400 (1.0) | 10000 | 10 |
| decode | 465.6591 (2.30) | 860.5129 (2.46) | 473.8062 (2.29) | 9.4256 (1.78) | 472.8009 (2.29) | 3.6421 (1.73) | 182;414 | 2.1106 (0.44) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19284323563/job/55221462586
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 188.8096 (1.0) | 353.0260 (1.0) | 192.6355 (1.0) | 4.1865 (1.0) | 192.2017 (1.0) | 1.9293 (1.0) | 281;362 | 5.1912 (1.0) | 10000 | 10 |
| decode | 438.9683 (2.32) | 1,469.8088 (4.16) | 444.1291 (2.31) | 14.7933 (3.53) | 442.4200 (2.30) | 2.7108 (1.41) | 93;1163 | 2.2516 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19284323563/job/55221890208
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 202.7853 (1.0) | 578.1251 (1.0) | 205.8727 (1.0) | 9.1838 (1.0) | 204.8113 (1.0) | 1.7907 (1.0) | 117;511 | 4.8574 (1.0) | 10000 | 10 |
| decode | 467.1764 (2.30) | 1,092.5360 (1.89) | 475.9442 (2.31) | 12.8802 (1.40) | 474.8899 (2.32) | 3.0806 (1.72) | 122;446 | 2.1011 (0.43) | 10000 | 10 |
Commits de51303 (#927) & 21da11e (#927) introduce a final implementation of the helpers that optimizes in the case of subclasses.
https://github.com/jcrist/msgspec/actions/runs/19320644266/job/55261304734
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 187.2177 (1.0) | 335.8345 (1.0) | 189.6079 (1.0) | 5.1489 (1.0) | 189.0633 (1.0) | 0.8365 (1.0) | 160;680 | 5.2740 (1.0) | 10000 | 10 |
| decode | 441.0673 (2.36) | 866.4667 (2.58) | 445.8347 (2.35) | 11.5915 (2.25) | 444.3397 (2.35) | 2.1955 (2.62) | 107;1405 | 2.2430 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19320644266/job/55261459898
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 198.4325 (1.0) | 353.4426 (1.0) | 203.5165 (1.0) | 6.3106 (1.0) | 202.6852 (1.0) | 2.0679 (1.0) | 286;610 | 4.9136 (1.0) | 10000 | 10 |
| decode | 466.2792 (2.35) | 798.7269 (2.26) | 474.2315 (2.33) | 9.3709 (1.48) | 472.9807 (2.33) | 3.5315 (1.71) | 246;527 | 2.1087 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19320644266/job/55261700792
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 200.1261 (1.0) | 354.6502 (1.0) | 204.7488 (1.0) | 6.5646 (1.0) | 203.8291 (1.0) | 2.0593 (1.0) | 186;576 | 4.8840 (1.0) | 10000 | 10 |
| decode | 463.3951 (2.32) | 1,195.7785 (3.37) | 471.8159 (2.30) | 16.2059 (2.47) | 470.4672 (2.31) | 3.6633 (1.78) | 117;404 | 2.1195 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19320644266/job/55261836412
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 198.3519 (1.0) | 438.4243 (1.0) | 204.4805 (1.0) | 11.8608 (1.0) | 202.9539 (1.0) | 2.4232 (1.0) | 137;517 | 4.8904 (1.0) | 10000 | 10 |
| decode | 465.4753 (2.35) | 1,163.0335 (2.65) | 474.5200 (2.32) | 17.4162 (1.47) | 472.7815 (2.33) | 3.5426 (1.46) | 144;531 | 2.1074 (0.43) | 10000 | 10 |
https://github.com/jcrist/msgspec/actions/runs/19320644266/job/55262300900
| Name (time in us) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS (Kops/s) | Rounds | Iterations |
|---|---|---|---|---|---|---|---|---|---|---|
| encode | 201.3492 (1.0) | 327.2208 (1.0) | 205.2371 (1.0) | 4.5759 (1.0) | 204.2903 (1.0) | 2.0764 (1.0) | 659;837 | 4.8724 (1.0) | 10000 | 10 |
| decode | 468.2807 (2.33) | 873.8386 (2.67) | 479.3891 (2.34) | 10.2809 (2.25) | 476.9618 (2.33) | 7.3895 (3.56) | 928;407 | 2.0860 (0.43) | 10000 | 10 |