UI test for PR #26979 by individual-it · Pull Request #27191 · owncloud/core

individual-it · 2017-02-21T09:59:05Z

Description

This is an UI test for the PR #26979 that fixes the issue #26975

This also includes the complete setup of the selenium infrastructure that can be used as the basis of further UI tests. After the discussion in #27057 behat+mink is used instead of the facebook webdriver.

For this to work travis starts behat/mink tests. And they connect to selenium running on https://saucelabs.com

Related Issue

PR #26979
issue #26975

List of PRs/issues that potentially would benefit of UI tests:

Can't set a user's quota back to Unlimited #26975
Fix long display names #27059
files_external: Keep checkbox and its label on one line #25120
Improve quota validation including a regexp which matches expected formats #25358
Rows are not updated after sharing via web ui. #27294 nextcloud/server#3688
Fixed disappearing of share info in file view #26534 nextcloud/server#2010

Motivation and Context

See issue #27055
Some bugs are hard to catch with karma & PHPUnit tests. Primarily this are front-end bugs. This PR implements end-to-end testing environment run on real browsers. Currently only Chrome browser is used, but it can be extended to multiple browsers/platforms including mobile browsers.
Currently my own sauce-labs account is used. Of course it would be good that a oC account would be used in future.

run this tests locally

download Selenium standalone server jar from http://docs.seleniumhq.org/download/
download the chromedriver from: https://sites.google.com/a/chromium.org/chromedriver/ and place it in the same folder as the selenium server
start selenium: java -jar selenium-server-standalone-3.0.1.jar -port 4445
set the 3 environment variables: SRV_HOST_NAME, SRV_HOST_URL and SRV_HOST_PORT e.g.:

export SRV_HOST_NAME=localhost
export SRV_HOST_URL=owncloud-core
export SRV_HOST_PORT=80

if no webserver is already running start php development server with: bash tests/travis/start_php_dev_server.sh (leave SRV_HOST_URL empty if you are using the php dev. server export SRV_HOST_URL="")
it will bind to: $SRV_HOST_NAME:$SRV_HOST_PORT
run tests: bash tests/travis/start_behat_tests.sh

limitations

Only PRs from local branches are tested. PRs that come from a fork cannot be tested as they do not have access to the saucelabs username+key.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

How Has This Been Tested?

@individual-it joined travis with their own account on sauce labs and has run multiple tests. Created bugs and run tests again to see if they are failing.

Screenshots (if appropriate):

travis job:
https://travis-ci.org/individual-it/owncloud-core/jobs/203733741
saucelabs job:
https://saucelabs.com/beta/tests/840fed2401854c50ab20109ac2812fb3/watch#305

This sets up UI tests. It uses behat/mink/selenium to run the tests on saucelabs.com Included are some basic tests 1. Login as admin 2. Tests to cover the fix made in PR #26979

mention-bot · 2017-02-21T09:59:08Z

@individual-it, thanks for your PR! By analyzing the history of the files in this pull request, we identified @DeepDiver1975, @PVince81 and @phisch to be potential reviewers.

SergioBertolinSG · 2017-02-22T13:07:19Z

start selenium:java -jar selenium-server-standalone-3.0.1.jar -port 4445

What version of java is good for this step? openjdk can be used there?

individual-it · 2017-02-22T15:20:15Z

According to the selenium github page you need Java 8 https://github.com/vvo/selenium-standalone/blob/master/README.md recent OpenJDK should be fine The java version on my Debian dev. machine is: openjdk version "1.8.0_111" OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14) OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

SergioBertolinSG · 2017-02-24T10:23:42Z

tests/ui/features/users.feature

+		|Unlimited  |1 GB        |
+		|1 GB       |5 GB        |
+		|5 GB       |Unlimited   |
+		|1 GB       |Unlimited   |


This feature cannot work, seems like a snippet.

Why shouldn't it work? At least it works fine for me. It runs 5 scenarios from that table. Check the bottom travis output:
https://travis-ci.org/individual-it/owncloud-core/jobs/203733741

Ahh right. It failed in my environment for different reasons.

SergioBertolinSG · 2017-02-24T10:27:22Z

tests/travis/start_behat_tests.sh

+#
+
+
+export BEHAT_PARAMS='{"extensions" : {"Behat\\MinkExtension" : {"base_url" : "http://'$SRV_HOST_NAME:$SRV_HOST_PORT/'","selenium2":{"wd_host":"http://'$SAUCE_USERNAME:$SAUCE_ACCESS_KEY'@localhost:4445/wd/hub"}}}}'


If you have localhost as hostname and a path to access your server this sentence doesn't work as it uses:

localhost/path:80 which doesn't work it has to be localhost:80/path

yes that is true, need to have $SRV_HOST_URL here too
always tested it with "start_behat_tests.sh" and that does not have any path
Also just found out that if you use port 80, the browser does not display it and so you get:
Expected to be on "http://localhost:80/index.php/login" but found "http://localhost/index.php/login" instead (SensioLabs\Behat\PageObjectExtension\PageObject\Exception\UnexpectedPageException)

both problems should be fixed now

SergioBertolinSG · 2017-02-24T11:34:37Z

Quota validation has this issue open #25358 .

DeepDiver1975 · 2017-03-01T10:30:26Z

I don't like that all travis tests are failing now - can we isolate this to only have selenium failing?

@individual-it can you please quide me to the docs on how to integrate travis and sauce? THX

individual-it · 2017-03-01T14:16:18Z

The travis tests are failing because the saucelab add-on is enabled but the sauce credentials are not set up in the owncloud travis account. All tests pass on my fork: https://travis-ci.org/individual-it/owncloud-core/builds/204940857

If you agree with the concept you should do this to integrate it and make all tests pass:

register to saucelabs and enrol in the open source plan https://saucelabs.com/open-source
in travis-ci.org set for the core repro the environment variables SAUCE_USERNAME SAUCE_ACCESS_KEY
generate the secure string with travis encrypt "your-access-key" https://docs.travis-ci.com/user/sauce-connect
get the other changes I've made in .travis.yml into the master .travis.yml. If you like to have as little changes as possible only get those from line: 17,18,34-37 (of course you would have to change line 35+37)
then I can make a new PR with all the other files included from a local branch and then all tests should pass

more information is available here: https://docs.travis-ci.com/user/sauce-connect

DeepDiver1975 · 2017-03-06T10:59:03Z

@individual-it I have added the credentials in the travis settings as environment variables.

Mind rebasing and removing your credentials? THX

individual-it · 2017-03-06T12:43:32Z

right, my instructions were far too complicated, most of it I can make in the PR itself.
So here is the PR from a local branch #27316
This PR is not needed any-more, so closing it

lock · 2019-08-03T20:01:32Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

individual-it added 4 commits February 16, 2017 15:13

build selenium branches with travis

c33a5a3

setup of UI tests

b6387c4

This sets up UI tests. It uses behat/mink/selenium to run the tests on saucelabs.com Included are some basic tests 1. Login as admin 2. Tests to cover the fix made in PR #26979

fixing a typo

09efe4b

Use clearer name setUpScenario to be consistent with PR #27189

a807580

individual-it requested review from DeepDiver1975 and SergioBertolinSG February 21, 2017 10:01

individual-it added the 3 - To Review label Feb 21, 2017

SergioBertolinSG suggested changes Feb 24, 2017

View reviewed changes

make test work when using port 80 and/or path in URL

aab0d72

individual-it mentioned this pull request Feb 28, 2017

fix for #25358 #27275

Merged

9 tasks

DeepDiver1975 self-assigned this Mar 2, 2017

SergioBertolinSG mentioned this pull request Mar 2, 2017

Run karma test on CI against several browsers owncloud/QA#385

Open

individual-it mentioned this pull request Mar 6, 2017

UI test for PR #26979 #27316

Merged

9 tasks

individual-it closed this Mar 6, 2017

lock bot locked as resolved and limited conversation to collaborators Aug 3, 2019

		#


		export BEHAT_PARAMS='{"extensions" : {"Behat\\MinkExtension" : {"base_url" : "http://'$SRV_HOST_NAME:$SRV_HOST_PORT/'","selenium2":{"wd_host":"http://'$SAUCE_USERNAME:$SAUCE_ACCESS_KEY'@localhost:4445/wd/hub"}}}}'

Conversation

individual-it commented Feb 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

List of PRs/issues that potentially would benefit of UI tests:

Motivation and Context

run this tests locally

limitations

Types of changes

Checklist:

How Has This Been Tested?

Screenshots (if appropriate):

Uh oh!

mention-bot commented Feb 21, 2017

Uh oh!

SergioBertolinSG commented Feb 22, 2017

Uh oh!

individual-it commented Feb 22, 2017 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergioBertolinSG Feb 24, 2017

Choose a reason for hiding this comment

Uh oh!

individual-it Feb 24, 2017

Choose a reason for hiding this comment

Uh oh!

SergioBertolinSG Feb 24, 2017

Choose a reason for hiding this comment

Uh oh!

SergioBertolinSG Feb 24, 2017

Choose a reason for hiding this comment

Uh oh!

individual-it Feb 24, 2017

Choose a reason for hiding this comment

Uh oh!

individual-it Feb 24, 2017

Choose a reason for hiding this comment

Uh oh!

SergioBertolinSG commented Feb 24, 2017

Uh oh!

DeepDiver1975 commented Mar 1, 2017

Uh oh!

individual-it commented Mar 1, 2017

Uh oh!

DeepDiver1975 commented Mar 6, 2017

Uh oh!

individual-it commented Mar 6, 2017

Uh oh!

lock bot commented Aug 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

individual-it commented Feb 21, 2017 •

edited

Loading

individual-it commented Feb 22, 2017 via email •

edited

Loading