wix / detox Goto Github PK
View Code? Open in Web Editor NEWGray box end-to-end testing and automation framework for mobile apps
Home Page: https://wix.github.io/Detox/
License: MIT License
Gray box end-to-end testing and automation framework for mobile apps
Home Page: https://wix.github.io/Detox/
License: MIT License
When a test fails, the terminal marks it as passed, and the afterEach hook fails.
Expected: Failing tests should fail.
Darwin 15.5.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "e2e"
npm ERR! node v5.8.0
npm ERR! npm v3.8.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] e2e: mocha e2e --opts ./e2e/mocha.opts
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the [email protected] e2e script 'mocha e2e --opts ./e2e/mocha.opts'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the rn-example package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! mocha e2e --opts ./e2e/mocha.opts
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs rn-example
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls rn-example
npm ERR! There is likely additional logging output above.
The symptom you see is that the "testee" does not connect. The underlying reason is that Detox dynamic framework tries to load and fails because one of the dependencies throws an exception in runtime.
This project depends on several "hardcore" projects that are tightly coupled to internals and therefore might break on major version changes of Xcode and the OS:
The solution is obvious, update these dependencies to latest version which solve all the issues. While we're at it, it's a good time to make the dependencies dynamic so they will update automatically by themselves from now on:
seems like an issue with fbsimctl
we need to fix that because it adds impossible delays to the tests
ideas:
Support CSS-like syntax.
For example: element(by.selector("#parentId @UIImage"))
Will get the first element of type UIImage
which is a descendent of an element with testID
parentId
the way to fix flakiness is to work on synchronization
EarlGrey synchronizes by waiting for the device to be idle:
https://github.com/google/EarlGrey/blob/master/docs/api.md#synchronization
we should probably add to EarlGrey the same ability to synchronize, but on RN instead:
this is not an easy research task
Since it takes a lot of space, we don't want to update it when it doesn't change
All simulator commands take forever, especially deleting an app
We need to improve its performance
One idea is to work with GUIDs directly and not with the models types, it probably enumerates all devices for every command and that takes longer
EarlGrey has a hard time dealing with them since they're external to the app
this thread offers some interesting workarounds.. we can add them directly into detox
For example, the navigator's Back
UIAButton
Need to see where it stopped :-/
Allow to write tests over the app log
For example, expect that we have no errors or warnings in log
maybe a new approach?
In order to select components in the tests we use the testID
property of course.
Right now you can't check the text by testID. It'll be great if we could do it.
Another great thing is to implement the toMatch
matcher so we won't break on every tiny text change.
Protractor syntax is:
expect($('#my-id').getText()).toMatch('some partial text');
P.S - if the toMatch
is too complicated having the ability to check the text by Id is also good enough.
Go team WIX!
Before starting a new session, the tester will ask isReady
and will wait for ready
.
When ready for a new session, or when asked isReady
, the testee will send ready
.
This will make sure the session only starts when the testee is ready
cd detox/detox
npm install
npm run build
npm run test-clean
npm run test-install
# don't run this now: npm run test
cd test
npm run e2e
The last line is how end users (not people developing detox itself) will eventually run detox
this means any parameters you want to send need to be designed to make sense there, like:
npm run e2e -- --scheme=ios-simulator.release
npm run e2e -- --verbose
If you want to improve the scripts for people developing detox, this must come as step 2 after the API for end users is finished and working
when a test fails we don't get the reason
Or in general, wait for a matcher/condition to be satisfied.
fbsimctl uninstall org.reactjs.native.example.example
vs
xcrun simctl uninstall booted org.reactjs.native.example.example
the apple version is still faster, we need to improve fbsimctl
All lines in a test scenario are naturally asynchronous since the device has to perform each interaction and that takes time. In order to avoid the boilerplate of writing await
before every line, we use a concept similar to what protractor does for its Control Flow.
Unlike protractor, our current implementation simply manages an invocation queue and then executes it until the test terminates.
This is not ideal. We need to switch to a pending promise queue like protractor.
In addition, we want to support a manual inline await
in the middle of a test. The reason is that you might want to wait on something external, for example restart the simulator in the middle of a test.
Need to be able to check various navigation scenarios, for example:
Also, need to be able to perform a 'back' navigation action.
if you get build problems, delete the following folders (since EarlGrey downloads them on build):
node_modules/detox/ios/EarlGrey/OCHamcrest.framework
node_modules/detox/ios/EarlGrey/fishhook
node_modules/detox/ios/EarlGrey/Tests/UnitTests/ocmock
Will make working with the project a breeze. Instead of defining stuff in the package.json, pass arguments to the cli, for example: detox build ios/MyApp.xcproj MyAppDev
, detox run e2e/
,
then the cli automatically figures out what and where to build.
Then, in the user's package.json, he can define those scripts once like so "e2e": "detox build ios/MyApp.xcproj MyAppDev && detox run e2e/"
, and forgo all other installation and configuration coupling to the build path
In particular, swipe up a ListView in order to trigger "fetch more items" action.
On react-native 0.25.1.
Sometimes, when detox starts the app, it freezes (showing video view screen), prints before all
in the console, then after a few seconds stops the app and exits with exception timeout exceeded
.
requires some research
what's the best way to match the different tab icons?
need to add this to the api
Fixed in #32
Fixed in #33
For example, I would like to be able to use async/await when calling simulator.deleteAndRelaunchApp()
Do you guys have this working remotely in some sort of continuous integration environment? (i.e. CircleCi)
Need a means to detect whether an alert is displayed, and if so - need to be able to verify the content (title, message, buttons).
Phase 1 - react-native-invoke for Java
Phase 2 - Integrate with Espresso
fixed in #40
EarlGrey on iOS synchronizes with async OS tasks such as network requests using objects that conform to GREYIdlingResource
. These objects listen on a resource like the network stack and are polled by EarlGrey to say whether this resource is busy or not (isIdleNow
).
Our current phase 1 sync mechanism has 2 new Idling Resources specific for React Native.
The current implementation uses heuristics (moving average of the time that takes a bridge command to travel the full cycle) and this isn't ideal.
We want to improve the sync mechanism, remove the heuristics and make it more robust.
each scheme needs its definitions in package.json because the app binary might be located in different places
we should specify which scheme we want using command line arguments to npm run e2e
the default scheme should be ios-simulator.debug
The thing that will really let us boost up our tests and give us much better coverage and usage testing is giving the option to set text and clear an input field.
Protractor syntax is:
$('#address-input').sendKeys(value);
$('#address-input').clear()
10x
I'm having the following logs on detox-server
login sessionId=MyApp role=tester
fw sessionId=MyApp action=isReady from role=tester
cannot fw sessionId=MyApp since other role=testee not connected
I'm also having timeout as this issue: #20
This code works:
it('starts with video view on first launch', (done) => {
global.simulator.deleteAndRelaunchApp(() => {
expect(element(by.id('VideoView'))).toBeVisible();
done();
});
});
it('does not display video view on second launch', (done) => {
global.simulator.reloadReactNativeApp(() => {
expect(element(by.id('VideoView'))).toNotExist();
done();
});
});
while this fails (prints error: _onTestResult is undefined on testFailed
):
it('starts with video screen on first launch only', (done) => {
global.simulator.deleteAndRelaunchApp(() => {
expect(element(by.id('VideoView'))).toBeVisible();
global.simulator.reloadReactNativeApp(() => {
expect(element(by.id('VideoView'))).toNotExist();
done();
});
});
});
I tracked the reason to this
Also, #57 would make this code much prettier
right now we instruct people to copy-paste the script into xcode as a post build step
cd ${SRCROOT}
if [ ! -d "build" ]; then
mkdir build
fi
cp -rf ${BUILT_PRODUCTS_DIR}/../ /${SRCROOT}/build/Products
we should instead implement is as a bash script that will be available inside detox, and then just reference the script from xcode
In order to determine if we are in debug or release mode, we need to add a key to the _currentScheme
in simulator.js
.
Something like:
{
app: 'ios/build/Build/Products/Debug-iphonesimulator/example.app',
mode: 'debug',
device: 'iPhone 6s, iOS 9.3'
}
We sometimes have sync problems with the keyboard not shown yet:
2016-10-08 01:47:39.567 example[38497:4275833] Detox Test Failed: Element matcher: (respondsToSelector(accessibilityIdentifier) && accessibilityID("UniqueId005")) Complete Error: Error Domain=com.google.earlgrey.ElementInteractionErrorDomain Code=1 "Failed to type string "�����������������", because keyboard was not shown on screen." UserInfo={NSLocalizedDescription=Failed to type string "�����������������", because keyboard was not shown on screen.}
I'm still having this this issue #20 with detox 3.2.0. ( Also tried on 3.1.0 )
06:12:37: server listening on localhost:8099...
06:12:41: role=tester login (sessionId=mytestapp)
06:12:47: role=tester action=isReady (sessionId=mytestapp)
06:12:47: role=testee not connected, cannot fw action (sessionId=mytestapp)
06:12:47: role=testee login (sessionId=mytestapp)
06:12:47: role=testee action=ready (sessionId=mytestapp)
06:12:47: role=tester action=reactNativeReload (sessionId=mytestapp)
06:13:07: role=tester action=cleanup (sessionId=mytestapp)
06:13:07: role=testee action=cleanupDone (sessionId=mytestapp)
06:13:07: role=tester disconnect (sessionId=mytestapp)
Simulators starts app but test isn't executed. I tried on iphone 6s, 6, 5s simulator as well and it was the same.
This seems intrusive to do that as a postinstall step, better ask that in the installation instructions, and throw exceptions on the postinstall step.
In ios I believe this essentially means supporting grey_typeText
we should have NSLog available for crashes.. if detox doesn't work, usually NSLog can explain what went wrong and help us debug
if it's saved on the CI, it could also help us a lot to investigate failed test runs
also, it will be very nice to add tests based on NSLog, for example verify that during the last test, there were no console.errors or console.warnings
For example:
"detox": {
**"beforeStart": "npm run fake-server",**
"session": {
"server": "ws://localhost:8099",
"sessionId": "abc"
},
"ios-simulator": {
"app": "ios/DerivedData/WixOneApp/Build/Products/Debug-iphonesimulator/ABC.app",
"device": "iPhone 6s, iOS 10.1"
}
}
Done in #45
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.