JS web frameworks benchmark – Round 8

The last round was published more than half a year ago. Time for an update.

The js-framework-benchmark tries to compare the performance of web frameworks by measuring the duration for different operations on a large table like the duration for creating the table, updating, selecting or removing a row and appending rows.

The first round included 9 implementations. In Round 8 there are now 101 implementations. The project grew to more than 300 pull requests from contributors and 2,000 github stars. Thanks to everyone who contributed so far. It’s clear by looking at the number of implementations that keeping them up to date wouldn’t be possible without your contributions.

Keyed and non-keyed

A pretty significant milestone was the separation of keyed and non-keyed frameworks in round 5 (thanks to Leon Sorokin, author of domvm, for the guidance). As it’s an important concept to understand I’ll explain it here again in simple terms.
A framework can implement an update for a list of data in two ways. Either keep an association between a list element and a dom node (which is the keyed mode) or rearrange and reassign the DOM nodes as it wants to (non-keyed).
If you’re using CSS transitions or third-party frameworks non-keyed might cause problems, because e.g. removing an element might instead remove the last element from the dom node list and patch all others to display the correct values. This can cause problems when e.g. CSS transitions styles depend on removing the specific dom node.
Most (maybe al) keyed framework can behave non-keyed if one assigns the list’s array index as a key.
From the framework perspective handling only non-keyed updates makes dom reconciliation much easier. If you want my opinion: Choose a framework that supports keyed updates.

What has changed since round 7?

  • Added the first webassembly frameworks in this benchmark: stdweb and yew
  • Lots of small and fast frameworks were added: domc, stage0, solid, attodom
  • A webcomponent variant for vanilla-js
  • A fast alternative renderer for Knockout: ko-jsx
  • Significant Improvements for good ‘old’ Marionette.js
  • Updates for many others including angular 6 and a preview for the new angular ivy renderer
  • Update to the startup metrics. They now simulate a mobile device and use the latest lighthouse version.

Important results:

  • The fastest frameworks are very fast. I have an unresolved issue to “make vanillajs great again”. I can’t beat domc and stage0 and solid, surplus and ivy are very, very close.
  • Marionette made a significant jump. The first implementation for version 3.3.1 was about 248% slower than the then fastest pure javascript ‘vanillajs’ version. Version 4.0.0-beta.1 is only 24% slower than domc.
  • The (low level) webassembly framework stdweb is only 13% behind. Might be interesting to see how they continue.

Here’s the complete table:

Discuss this article on reddit.

JS web frameworks benchmark – Round 7

This is the 2017 Pre-Christmas edition of the JS web framework benchmark. Almost as usual a few new frameworks appear in this round:

Further the were new major releases for react and angular. All in all there are currently 76 framework variants in this round. A big thanks to all contributors to this round!

Continue reading

JS web frameworks benchmark – Round 6

Some people asked for an updated – and there were some interesting updates and contributions, so it’s a good time for a new round of the js web framework benchmark.

What’s changed

Continue reading

A first look at WebAssembly performance

WebAssembly gives us the promise to run high performance code in the browser in a standardized way. Now that there are a few WebAssembly previews available I decided it’s time to take a look at their performance. One source for benchmarks is the well known Computer Language Benchmarks Game and I decided to pick nbody (it’s almost four years ago since I did so last time…).

After playing a bit with the results I decided to put the code on github. I’m looking forward to your corrections, improvements and feedback. I’m already excited what the results will look like in a few months…

The following versions were compared:

  • webAssembly: A WebAssembly version compiled from the original c version, because this turned out to be faster than the other version I checked
  • object: The fastest javascript version from the Computer Language Benchmarks Game. It uses a javascript objects for each body to store the data.
  • arrayPerObject: Each body’s data is stored in a plain javascript array.
  • floatArrayPerObject: Each body’s data is stored in a typed array
  • oneTypedArray: All body’s data is stored in a single typed array and the advance function is programmatically unrolled (quite crazy, isn’t it).
  • To get a baseline the fastest java and the original c version were added.

Here are the results:


(click to enlarge)

Firefox does pretty well. The WebAssembly implementation is the fastest browser version and close to the java baseline, but the pure javascript implementation isn’t really much behind. Seems like Javascript VMs are already pretty good at simple numeric code.
For the other browsers WebAssembly couldn’t beat the javascript versions yet. And Safari has a completely different idea what Javascript version it can optimize best.

The fine print

Browser versions:

  • Chrome Canary, 58.0.3004.0, invoked with
    --js-flags="--turbo --trace-opt --trace-deopt --trace-bailout"

    for turbofan and

    --js-flags="--trace-opt --trace-deopt --trace-bailout"

    for crankshaft.

  • Firefox 53.0a2 (2017-02-06) (64-Bit)
  • Safari Technology Preview Release 22 (Safari 10.2, WebKit 12604.1.4.2)

WebAssembly setup:

  • Emscripten and Binaryen were installed as described on Compile Emscripten from Source.  (emcc (Emscripten gcc/clang-like replacement) 1.37.2 (commit 70d370296036cc5046083a3e215cb605c90e004b))
  • The c source code was compiled with that command:
    emcc nbody.c -O3 -s WASM=1 -s SIDE_MODULE=1 -o nbody.wasm

C compiler:

  • The c version was compiled with gcc -O3  nbody.c -o nbody (which is Apple LLVM version 8.0.0 (clang-800.0.42.1))
    This version took 4.4 seconds on my machine and was faster than the fastest C version from the shootout, compiled with gcc -O3 -fomit-frame-pointer -march=native -mfpmath=sse -msse3 nbody_fastest.c -o nbody_fastest, which took 4.9 seconds on my machine

Infrastructure:

All tests were performed on a 2015 MacBook Pro, 2.5 GHz Intel Core i7, 16 GB 1600 MHz DDR3. For all tests the best of three runs was selected for the result.

JS web frameworks benchmark – Round 5

Welcome to another round of the js web framework benchmark.
Once again a lot has happened:

  • Elm, Knockout, Nx, Binding.scala, Dio, Polymer, Simulacra and Svelete are new in this round.
  • Most other frameworks have been updated to their latest version (though it took a bit to write this blog post, so they might be outdated again by now…)
  • There are now two result tables: One for “keyed” implementations and one for “non-keyed”. I’ve written a separate blog post about that.  The classification as keyed or non-keyed is not complete yet and only based on the current benchmark implementation and does not indicate that the framework doesn’t support the other mode. Feel free to send me pull requests!
  • The “clear rows a 2nd time” benchmark has been removed. It showed a (maybe theoretical) issue in react that is fixed by now.

I’d really like to thank all contributors (at the time of writing there are 33 of them). Without you it would be impossible to make such a comparison.

The results are here:

Key observations (pun intended):

  • It’s still the case that frameworks are getting faster. Riot 3 is a big step in terms of performance in this benchmark.
  • Ember is included again in this round and 2.10-beta is a big improvement.
  • I’ll really have to rework my vanilla.js implementation again if it should become the fastest possible implementation again.
  • Apart from that many frameworks are in a performance range that should be fine for many requirements. Vue.js, kivi, vidom, plastiq, domvm and bobril in the keyed table show little weakness.
  • As for the non-keyed frameworks the results are much closer. Dio, inferno and svelte are incredibly fast, domvm and Vue.js are only a bit behind.

You can to click through the implementations in your browser here. Please keep in mind that all durations printed on the browser console are just approximations, but good enough to feel the effect between fast and slow implementations. The real results are gathered from the chrome timeline with the test driver in webdriver-ts.

JS web frameworks benchmark – Round 4

Here’s another update for the js web framework benchmark. This time the benchmark has seen lots of contributions:

  • Dominic Gannaway updated and optimized inferno
  • Boris Kaul added the kivi framework
  • Chris Reeves contributed the edge version of ractive
  • Michel Weststrate updated react-mobX
  • Gianluca Guarini updated the riot benchmark
  • Gyandeep Singh added mithril 1.0-alpha
  • Leon Sorokin contributed domvm
  • Boris Letocha added bobril
  • Jeremy Danyow rewrote the aurelia benchmark
  • Filatov Dmitry updated the vidom benchmark
  • Dan Abramov, Mark Struck, Baptiste Augrain and many more…

Thanks to all of you for contributing!

Continue reading

JS web frameworks benchmark – Round 3

It’s been a few months since I published Round 2 of my javascript web frameworks performance comparison. In Javascript land months translate to years in other ecosystems so it’s more than justified to introduce round 3.

Here’s what’s new:

  • Added a pure javascript version to have a baseline for the benchmarks (“vanillajs”).
  • Added cycle.js v6 and v7. What a difference the new version makes!
  • Added inferno.js. This small framework made writing a faster vanillajs version challenging.
  • Updated all frameworks to their current version.
  • New benchmarks: Append 10,000 rows, Remove all Rows, Swap two rows.
  • Added two benchmarks that measure memory consumption directly after loading the page and when 1,000 rows are added to the table.

The result table for chrome 51 on my MacBook Pro 15 reveals some surprising results (click to enlarge):

result_table

(Update: cycle.js v7 was reported with an average slowdown of 2, but I forgot a logging statement. Without it it improved further to a factor of 1,8. The table has been updated.)

  • Vanillajs is fastest, but not by much and that only by hard work.
  • Inferno.js is an incredibly fast framework and only 1.3 times slower than vanillajs, which is simply amazing.
  • plastiq comes in as  the second fastest framework being just 1.5 times slower, followed by vidom.
  • React.js appears to have a major performance regression for the clear rows benchmarks. (Clearing all rows for the third time is much faster. There are two benchmarks pointing to that regression which is a bit unfair for react.js)
  • Cycle.js v6 was slowest by far, v7 changes that completely and leaves the group of the slow frameworks consisting of ember, mithril and ractive.

All source code can be found on my github repository. Contributions are always welcome (ember and aurelia are looking for some loving care, but I to admin that I have no feelings for them. And I’d like to see a version of cycle.js with isolates) .

Thanks a lot to Baptiste Augrain for contributing additional tests and frameworks!

 

JS web frameworks benchmark – Round 2

Here’s a follow-up to my last blog post since there’s good reason for an update:

  • It turned out that ember performs better with ‘key=”identifier”‘ in the #each helper.
  • One of the vue.js creators contacted me and told me that they fixed vue.js in response to the my benchmark
  • Two other react-like libraries were added: Preact and react-lite

Here’s the new measurement showing the duration in milliseconds:

results2

(Click the image to see the graph in full size)

The duration is measured from the beginning of the dom click event to the end of repainting and thus includes javascript execution, rendering and layout duration. If you want to read more about it please check the according blog entry. All tests are run via a selenium test and multiple samples are taken and averaged (after removing the worst measurement). The full description for the benchmarks can be found in round 1.

And the new conclusions:

  • Ember got much faster for “partial update” (from 164 msecs to 58 msecs), but is still quite slow for create or update 1000 rows.
  • Vue.js improved a lot for the “update 1000 rows (hot)”  benchmark (from 435 msecs to 260 msecs)
  • Preact leaves quite a good impression. It’s a lot faster than react for create 1000 rows and update 1000 rows and not much slower for the rest.
  • Almost the same can be said about react-lite though the performance for “remove row” is rather weak.

All in all especially angular 2, vidom and preact impress with their performance. Still aurelia feels much faster in the browser than in the selenium tests (except for startup duration which might be my fault not using the bundler correctly). And yes, vue.js is now pretty close to the fastest frameworks.

You can view the benchmarks in the browser and find all source code on github.

JS web frameworks benchmark – Round 1

In this blog post I’ll try to compare the performance a few popular javascript frameworks. Measuring the performance of browser content can be challenging and I’m prepared for harsh replies.

One approach to measure the performance would be to use browser tools like the chrome timeline, which reveals exact timings, but has the disadvantage of being a manual and time consuming process and yielding only a single sample.

At first I tried automated benchmarking tools such as Benchpress or protractor-perf, but I didn’t really understand the results and thus decided to roll my own selenium webdriver benchmark. I wrote an additional blog entry to describe this approach. To put it short it measures the duration from the start of a dom click event to the end of the repainting of the dom by parsing the performance log. To reduce sampling artifacts it takes the average of runnig each benchmark 12 times ignoring the two worst results.

Screenshot_Angular2

Continue reading