JS web frameworks benchmark – Round 1

In this blog post I’ll try to compare the performance a few popular javascript frameworks. Measuring the performance of browser content can be challenging and I’m prepared for harsh replies.

One approach to measure the performance would be to use browser tools like the chrome timeline, which reveals exact timings, but has the disadvantage of being a manual and time consuming process and yielding only a single sample.

At first I tried automated benchmarking tools such as Benchpress or protractor-perf, but I didn’t really understand the results and thus decided to roll my own selenium webdriver benchmark. I wrote an additional blog entry to describe this approach. To put it short it measures the duration from the start of a dom click event to the end of the repainting of the dom by parsing the performance log. To reduce sampling artifacts it takes the average of runnig each benchmark 12 times ignoring the two worst results.

Screenshot_Angular2

As for the benchmark I got caught on a gist called Performance Comparison for React, Angular and Knockout from Rich Ayotte which is based on a comparison by Chris Harrington. The gist measured the duration to create or update 1000 rows and the duration to highlight a row in response to a click. I decided to add a few more operations like removing a row, a partial update of every 10th row and adding 10 rows.

Further I wanted to add some other frameworks like Angular 2Aurelia, EmberMithril, Ractive.js, Vidom and Vue.js. The post title is suffixed “Round 1” to acknowledge the possibility that faster implementations are possible and other frameworks may be included.

What is being measured?

  • create 1000 rows: Measures the duration to add 1000 rows. This is executed after the page was loaded. Equivalent to  clicking “create 1000 rows” once.
  • update 1000 rows (hot): Time for updating all 1000 rows of the table. 5 iterations to warmup the javascript engine are performed before measuring. Equivalent to  clicking “create 1000 rows” when the table already exists.
  • partial update: Time to add a dot to the end of the text of every 10th row. 5 iterations to warmup the javascript engine are performed before measuring. Equivalent to  clicking “update every 10th row”.
  • select row: Duration to highlight a row in response to a click on the row. 5 iterations to warmup the javascript engine are performed before measuring. Frameworks that batch dom updates and perform the actual update in a request animation frame or a timer event may have a disadvantage in that benchmark. The duration to perform the update is almost negligible and the delay to wait for the rendering callback can determine the measured duration. Anything below 16 msecs should thus be considered fast enough.
  • remove row: Duration to remove a row. 5 iterations to warmup the javascript engine are performed before measuring.

Results

These are the measured durations in milliseconds on my MacBook Pro with Chrome 48:

Update: I’ve published round 2.

Conclusions

  • Angular is said to perform very poor. In this benchmark angular certainly isn’t the fastest especially for updating all rows, but it’s way better than expected.
  • Angular 2 is an impressive improvement over Angular. Angular 2’s only slightly weak spot is the “remove row” benchmark.
  • Aurelia‘s performance is very hard to grasp. Aurelia updates the model and fires a timer to update the dom. Sometime that delay can be pretty long:
    aurelia_long_delay
    To the left is the dom click event and almost 100 msecs later the rerendering starts. This makes aurelia pretty slow for the “select row” benchmark. Still aurelia feels quite fast when executing in the browser and the pure javascript execution duration looks very promising.
  • Ember forced a massive rewrite of the benchmark. I’m excited if someone can create a much faster version. Currently Ember is pretty slow for creating or updating rows.
  • Mithril is a direct competitor to react. It’s faster than react for the “create 1000 rows” benchmark, but a bit slower for the updates and particular slow for the “select row” benchmark – and the timeline shows that (in contrast to aurelia) javascript execution takes about 85 msecs in that case.
  • Ractive.js is quite extreme. It’s the fastest when all rows are updated and the slowest in the “remove row” benchmark.
  • React is a bit slow for the cold “create 1000 rows” case and performs quite well for the other benchmarks.
  • Vidom turned out to be my personal favorite framework. It is sometimes significantly faster than react and in the worst case only slightly slower.
  • Vue.js does a bad job updating all rows, but is quite fast for the other benchmarks. Update: Vue.js got an update due to this benchmark. Please see round 2 for the very much improved results.

The benchmarks show that there’s no framework that is fastest for all benchmarks. Angular 2 and vidom excel at having virtually no worst case, while ember is the worst case. Angular is not nearly as bad as its reputation.

Please check also the round 2 of this benchmark.

Check yourself

If you don’t believe the results (I wouldn’t …) don’t hesitate to clone the git repository https://github.com/krausest/js-framework-benchmark and send me your pull requests. The build instructions have been tested on OSX. If you want to click trough the example app go here.

Response to Dmitry Filatov question below

Some of the frameworks contain a setTimeout call. This was used to approximate measuring the duration in the browser and logging it on the console. I’ve blogged about it here. Since the selenium test driver doesn’t need it it could be removed (and isn’t there for aurelia, since it doesn’t work there).

I hope it doesn’t influence the measurements. At least in the chrome timeline it looks ok:

Here with setTimeout(0) ~143 msecs should be reported. (And one can see that the timer fires after rendering and painting and thus would be a fine place to stop measuring).

withTimeout

Here without setTimeout(0) yields ~147.92 msecs.

withoutTimeout

I’ve reported 139 msecs for the selenium tests which looks plausible to me.

10 thoughts on “JS web frameworks benchmark – Round 1

  1. Hi, Stefan! First of all, thank you for post and benchmark . I’m an author of vidom. I’ve overviewed benchmark code and I’m puzzled why you use “setTimeout(0)” inside your “stopMeasure” function. Without it I get quite a different result.

  2. Hi Dmitry, thanks for creating vidom – great job. I’ve put my reply above to include some images.
    What differences do you see?

  3. I see a huge difference in measurements (in console) between version with “setTimeout(0)” and without it if I try to run your benchmark directly inside browser, for instance chrome.

  4. Vue 1.0.16 fixed the row updating problem – now the “update 1000 rows (hot)” runs as fast as the initial render. Wouldn’t have caught this problem if it weren’t exposed in this benchmark, so thanks! Also would appreciate it if you can update the numbers ;)

  5. Agree that vanilla JS would be cool. I don’t think the React fanboys are going to be happy with this result ;) Very nice work.

Comments are closed.