One approach to measure the performance would be to use browser tools like the chrome timeline, which reveals exact timings, but has the disadvantage of being a manual and time consuming process and yielding only a single sample.
At first I tried automated benchmarking tools such as Benchpress or protractor-perf, but I didn’t really understand the results and thus decided to roll my own selenium webdriver benchmark. I wrote an additional blog entry to describe this approach. To put it short it measures the duration from the start of a dom click event to the end of the repainting of the dom by parsing the performance log. To reduce sampling artifacts it takes the average of runnig each benchmark 12 times ignoring the two worst results.
As for the benchmark I got caught on a gist called Performance Comparison for React, Angular and Knockout from Rich Ayotte which is based on a comparison by Chris Harrington. The gist measured the duration to create or update 1000 rows and the duration to highlight a row in response to a click. I decided to add a few more operations like removing a row, a partial update of every 10th row and adding 10 rows.
Further I wanted to add some other frameworks like Angular 2, Aurelia, Ember, Mithril, Ractive.js, Vidom and Vue.js. The post title is suffixed “Round 1” to acknowledge the possibility that faster implementations are possible and other frameworks may be included.
What is being measured?
- create 1000 rows: Measures the duration to add 1000 rows. This is executed after the page was loaded. Equivalent to clicking “create 1000 rows” once.
Update: I’ve published round 2.
- Angular is said to perform very poor. In this benchmark angular certainly isn’t the fastest especially for updating all rows, but it’s way better than expected.
- Angular 2 is an impressive improvement over Angular. Angular 2’s only slightly weak spot is the “remove row” benchmark.
- Aurelia‘s performance is very hard to grasp. Aurelia updates the model and fires a timer to update the dom. Sometime that delay can be pretty long:
- Ember forced a massive rewrite of the benchmark. I’m excited if someone can create a much faster version. Currently Ember is pretty slow for creating or updating rows.
- Ractive.js is quite extreme. It’s the fastest when all rows are updated and the slowest in the “remove row” benchmark.
- React is a bit slow for the cold “create 1000 rows” case and performs quite well for the other benchmarks.
- Vidom turned out to be my personal favorite framework. It is sometimes significantly faster than react and in the worst case only slightly slower.
- Vue.js does a bad job updating all rows, but is quite fast for the other benchmarks. Update: Vue.js got an update due to this benchmark. Please see round 2 for the very much improved results.
The benchmarks show that there’s no framework that is fastest for all benchmarks. Angular 2 and vidom excel at having virtually no worst case, while ember is the worst case. Angular is not nearly as bad as its reputation.
Please check also the round 2 of this benchmark.
If you don’t believe the results (I wouldn’t …) don’t hesitate to clone the git repository https://github.com/krausest/js-framework-benchmark and send me your pull requests. The build instructions have been tested on OSX. If you want to click trough the example app go here.
Response to Dmitry Filatov question below
Some of the frameworks contain a setTimeout call. This was used to approximate measuring the duration in the browser and logging it on the console. I’ve blogged about it here. Since the selenium test driver doesn’t need it it could be removed (and isn’t there for aurelia, since it doesn’t work there).
I hope it doesn’t influence the measurements. At least in the chrome timeline it looks ok:
Here with setTimeout(0) ~143 msecs should be reported. (And one can see that the timer fires after rendering and painting and thus would be a fine place to stop measuring).
Here without setTimeout(0) yields ~147.92 msecs.
I’ve reported 139 msecs for the selenium tests which looks plausible to me.