Given most people’s heavy usage of the internet, browsers are increasingly one of the most important programs you have on your PC, and we wanted to test things which can be objectively measured but which are key when deciding which browser you should use. Most importantly, it’s considering whether a browser is both fast and light.
We’ve designed a methodology to measure the performance that, in addition to being as objective as possible, aims to be easily understood by anyone who wants to recreate it themselves.
The PC we used
The computer we used for all tests is a standard PC with 4 GB of memory. The operating system is 64-bit Windows 7 Ultimate with all the updates, and it has Windows 7 because it’s the most popular OS.
This is the PC we use for the tests; the hard drive is completely defragmented
During the test, we didn’t run other software (not even the antivirus) to avoid it interfering with the tests. All power-saving options were disabled so that the PC wouldn’t shut down due to inactivity.
We restarted the PC before each test, and we left it for 30 seconds before logging in, which we timed to make sure that all tests were done on an even playing field.
During the speed tests we don’t run any other program, not even an antivirus
Loading pages from the Internet implies that speed will always change, so the pages we used for the speed test were loaded from a PC running Nginx, a very fast and reliable web server.
A local server avoids errors due to the reliability of the connection
Browsers were always run in clean configuration with no history, cookies, or extensions, to make sure that there was no code external to the browser running at the same time.
Finally, the PC we used for the tests was connected to the Internet via a fiber optic connection. All tests are recorded with Adobe Premiere and an HDMI capture.
The tests we run for each browser
We chose to run only the tests that we considered to be accurate and as close of a representation to its daily use as possible, which included looking at the speed when loading many tabs, and performance measurements that use graphs and common operations.
Programs such as Peacekeeper test the page for common tasks.
The tests we carried out are:
- Cold start-up with one tab on the home page of each browser. We measured the time it takes the browser to run and open Google after starting the computer.
- Warm start-up with one tab. The same test was repeated half a minute after a cold start to see if the browser is faster after running it once.
- Cold start-up with 11 tabs. 11 pages were loaded from the local web server and the clock stopped when the browser responded to scrolling using the mouse wheel.
- Warm start-up with 11 tabs. Similar to the cold test with 11 tabs, we tested it 30 seconds after closing the browser from the cold start-up test.
- Memory usage in megabytes. Memory usage was measured with 1 tab loaded, with 20 tabs loaded, and again with 1 tab loaded after closing the others, to verify how the memory recovers.
- Speed tests with web technologies, which are used to create complex web apps. We chose to consider two independent tests which we believe are more reliable and closer to the real use:
Times were measured with Adobe Premiere: in each case, we took into consideration the time lapsed until the frame that marked the end of the test. The sites we used to measure speed include:
- Reddit.com home page
- The ‘Apple’ page on Wikipedia in English
- A search on Twitter
- A search on Google
- Softonic’s review of Ares
- The map of Barcelona on Bing
- Yahoo.com (USA) home page
- BuzzFeed.com home page
- The ‘90s Nostalgia Pinterest Board (complete)
- First page of Imgur
- The Softonic App page on Amazon
In the memory test, we opened Google’s homepage and Softonic Spain’s Top 20 reviews. We waited a minute before measuring the memory again after closing all 20 tabs.
Chrome’s memory monitor is very reliable when measuring usage
Measurements were taken three times for each situation and then averaged. We used Chrome’s memory monitor for measurement, a very reliable tool when it comes to checking the memory usage of modern browsers.
A method that’s evolving
We want this methodology to improve with time. It’s open to change and suggestions, which we’ll evaluate and apply in subsequent editions of our performance comparison.
What would you like us to measure?