End to End (e2e) – Angular Testing – Protractor vs Cypress

As the applications grows, the manual testing becomes more and more complex. End to end (e2e) testing is where we test our entire application from the start to the end. It is used to simulate a “real” user doing all kind of scenarios, functionalities and possibilities. This means we need the whole system working to assure that all pieces are integrated and working as it should, we need some e2e Tool like Protractor or Cypress.

As we are using Angular to build our applications, Protractor comes as default tool. Cypress is a new framework that are growing and gaining popularity quickly and can be used with any framework, not only Angular. To help us on this comparison, we are taking into consideration the following:

  1. Installation
  2. Code readability and maintenance
  3. Learning Curve
  4. Performance
  5. Multi platform
  6. Reporting
  7. Debugging
  8. Github

Preparation

I’m going to use a simple project to execute tests in both frameworks. It will allow us to test Forms, navigation, resolution and cache.

We have two screens, a Login page and a Home Page and we will execute exactly 7 tests in each framework:

App Tests:

  • It Should display welcome message
  • Should get title
  • It should hide menu (when mobile)

Login Tests:

  • Login form should be valid when type correct information
  • Login form should be invalid when type incorrect information
  • Should set email value to local storage (cache)
  • It should see the success message after login

Welcome screen:

Home Screen After login:

1. Installation

1.1 Protractor:

Protractor comes as default tool for Angular projects. So the command line takes care for us. Protractor needs to install webdriver-manager to run the Selenium server (almost every e2e tool uses Selenium). We install protractor executing the command:

npm install -g protractor

The webdriver-manager is a helper tool used by Protractor. Our tests will send requests to this Selenium server to simulate the user interaction on a local browser, this is how Protractor simulate the users doing something on the website, sending commands through the network.

Once the protractor is installed, you need to define a Test Framework, by default Protractor uses Jasmine. You can chose others like Mocha , Qunit, or Karma for instance. Than we need to choose an assertion library, that could be Chai or expect.js. To help us mocking we can use another library as Sinon or TestDouble. For this comparison, we are using Jasmine and Chai, so every code related to Protractor it will be Jasmine or Chai.

It’s not a good practice to Mock something while you are testing e2e. The more real environment the better.

Every configuration is done through the protractor.conf.js. We can start running the protractor with this initial configuration (We will need to add more afterwards):

exports.config = {
  framework: 'jasmine',
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['spec.js'],
  multiCapabilities: [{
    browserName: 'firefox'
  }, {
    browserName: 'chrome'
  }]
}

1.2 Cypress

Cypress does not use Selenium as base-code. It means the tests are really being executed inside the browser as the users do. The setting up of cypress is easier, all we need to do is, execute the command:

npm install cypress --save-dev

And that’s all, we don’t need to choose a framework or assertion library, it’s all included.

2. Code readability and maintenance

In this topic both solutions are really good, lets see both code for each test and compare some code side by side.

2.1 Page Objects

We going to use an optional design pattern in both frameworks called Page Object. This is good to organize the code storing our elements in a separated class, improving performance and re-usability.

Page Object is a pattern described by Martin Flower to store the “complexity” to retrieve elements from a HTML page.

It’s good to see the difference between the both solutions (not much actually):

	
 // ***** PROTRACTOR (This is a Jasmine code) ******
	

  getTitle(){
    return browser.getTitle();
  }
  
  navigateTo() {
    return browser.get('/');
  }

  getWelcomeText() {
    return element(by.css('app-root h1')).getText();
  }

  setWindowSize(w,h){
    return page.getWindow().setSize(w, h);
  }

			
	
 // ******  CYPRESS ******
			
  getTitle(){
    return cy.title();
  }
  
  navigateTo() {
    return cy.visit("/");
  }

  getWelcomeText() {
    return cy.get("app-root h1")
  }

  setWindowSize(w,h){
    return cy.viewport(w,h)
  }

2.2 Testing codes

Following we can see the both testing codes. Protractor is on the left and Cypress is on the Right.

App Tests:

Login Tests:

With Cypress I’ve written a little bit less code. Both frameworks have almost the same structure of code, which is good.

3. Learning curve

That is a important factor when we are deciding which framework to choose. The installation is the hardest part with Protractor if you need to do it by yourself. We have to choose a lot of frameworks, including that you need choose the right version of each one of them. Point for Cypress because it wraps everything together.

With this benefit of Cypress you can start programming faster because all the documentation is in the same place. If you are using Protractor you need to check different documentations every time, for instance, if you have framework questions you need to check Jasmine’s documentation, for assertion questions you need to check Chai’s documentation and so on. After a while you get used to it, but for a quick starter it’s really hard to find this. Point to Cypress.

4. Performance

According to benchmarks, Cypress seems to be faster. But, for this example where I’m doing 7 tests both frameworks had the same time of 7~8 seconds. For development, I personally found Cypress better thanks to its auto-reload and automatic waits.

5. Multi Platform

At the time of this article was written, Cypress only supports Canary, Chrome, Chromium and Electron. But they have plans to add support for Android, IOS using Webviews according to their Roadmap.

Protractor is a benefit in this case, you just need to install the engine and you can use Firefox, Safari, IE, Chrome. There are other plugins to allow debugging using android simulator etc. Point for Protractor.

6. Reporting

Another important factor, we need to check the report when the tests are finished.

5.1 Protractor

Protractor doesn’t have any report natively. We need to install some 3rd party plugin manually and configure in protractor.conf.js. I’ve found a plugin called protractor-beautiful-reporter (it’s not that much beauty, but it’s ok). After following mooooore installations, I could have it running, here is the delivered report:

It is a simple HTML page, with all executed tests and the screenshot. It’s a good helper.

5.2 Cypress

Cypress has natively an amazing reporter tool. It’s even hard to explain how good it is, you can check the exactly state on each step executed on the test, you can pin, debug, check the HTML DOM, execute instant scripts, etc.

It’s hard to show this with image, so I’m embedding this video to give the idea. Please, check it:

7. Debugging

We can debug both solutions. With protractor we need more configurations: we need to create a config debug file, add the reserved word “debugger” and configure the IDE to debug the code. Using Visual Studio Code, you need to create a debug task and point to protractor configuration (check more here), example:

	
{
	"type": "node",
	"request": "launch",
	"name": "Debug Protractor",
	"program": "${workspaceFolder}\\node_modules\\protractor\\bin\\protractor",
	"stopOnEntry": false,
	"args": ["${workspaceFolder}/protractor.debug.conf.js"],
}
	

With Cypress, you just need to add the reserved word “debugger” or using the function debug(). That’s all. You will be able to debug using the chrome tool. Another point to Cypress. Check more here.

8. Github

Both solutions are open source and at this time they have the following status:

  • Protractor: 1700 commits, 7800 stars, 2000 forks, 370 open issues, 3200 closed issues.
  • Cypress: 11300 commits, 7700 stars, 330 forks, 800 open issues, 1300 closed issues.

Conclusion

There are bunch of solutions for e2e. This article was written to give the idea about these two frameworks. Cypress as a brand new framework is becoming a very attractive and a good option. One one the huge benefits, in my opinion, is the possibility the easily remote debug and the report that allows us to inspect the exact state of the screen in the moment of the execution. In the other hand, we have Protractor that is a default framework for Angular Project and it comes with a huge community and a lot of options to use as library. Both are good solutions and specific requirements of the project can define which one to choose.

References:

  • https://docs.cypress.io/guides/guides/continuous-integration.html#What-is-supported
  • https://www.cypress.io/how-it-works/
  • https://docs.cypress.io/guides/overview/why-cypress.html#In-a-Nutshell
  • https://www.protractortest.org/#/
  • https://jasmine.github.io/
  • https://mochajs.org/
  • https://github.com/angular/protractor
  • https://github.com/cypress-io/cypress

 

Tulio Castro

Author: Tulio Castro

Fexco Senior Software Engineer

6 Replies to “End to End (e2e) – Angular Testing – Protractor vs Cypress”

  1. Great article indeed. It really assisted me in understanding them both.

    I am exploring new e2e solutions at the moment and these two frameworks got to the finals 🙂

    Cypress has lots of benefits but looking at their pricing page it looks as they charge extra for parallelization execution which is unfortunate.
    https://www.cypress.io/pricing/

    To sum it up, I think I’ll go for Cypress, just need to get the other developers vote 🙂

    Superb comparison and big kudos to you Tulio Castro

    1. Thanks a million Eliran, we’ve been using cypress for one of our projects that is starting, and we could do some tests. I can say that so far so good. Hope everything runs well for you too. Thanks

  2. Hi,

    thank you for your article. I arrived here because I have a doubt: what’s the best e2e testing “instrument” for Angular 7?

    Thank you

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.