extending intents/steps elegantly...

[aisingapore]

extending intents/steps elegantly...

[dublindrupaller]

Hi Ken, It is fine for you to close this issue. I forked the repo at the weekend so I can work on it during this week...will probably be next weekend before I can get it into a demo for you. Will ping you when it is ready. Gus

[dublindrupaller]

@kensoh Hi Ken...just to let you know...I made really good progress this weekend. I am asking a collegue to review before it is ready for demoing for you. Will keep you posted.

[dublindrupaller]

@kensoh Hi Ken, I just added a branch called demo-for-ken to my fork on my dublindrupaller https://github.com/dublindrupaller/TagUI account. should be available for later this week. am about 70% of the way there at the moment and a collegue @exiled-hammer is helping.

Thought it best to comment in case you see that there is a branch demo-for-ken notice popping up on your TagUI main page.

[dublindrupaller]

Hi @kensoh ,

I have been busy over the last few weeks..but I managed to sneak in some spare time to bring the suggested approach to a working demo stage. If you have a few moments, can you have a quick look at the linked readme which explains how it works.

I have added a readme to explain the concepts a bit better

The demo-for-ken branch works for simple tests (casperjs only).

I have migrated all the steps from your tagui_parse.php file into an object orientated class file structure to improve efficiency and performance, among other benefits. Ditto for the live mode tagui_headerjs output. (the readme linked explains how it works)

A few things I would like to do next:

1. At the moment the first line in a TAGUI test file must be a valid URL. I would like to add an URI option when running TAGUI. e.g.

 ../tagui gus.ui firefox --uri my.localhost.net
 ../tagui gus.ui firefox --uri my.system-integration-host.net
 ../tagui gus.ui firefox --uri my.uat-host.net

So the same suite of TAGUI tests can be run against different instances.

2. I would like to proceed with the idea of having a language context....the default at the moment is casperjs, but, I would like to extend that for puppeteer, webdriver and maybe other script languages.
   The readme linked above explains how that works.

3. I would like to refactor your javascript so I can minify the tagui_header.js includes for live mode etc. You use line endings a lot, rather than a semi-colon, so minifying the js is tricky when there are comments in there as well.

I appreciate you may have your own plans for TAGUI moving forward. If you think this is too much of a change, please let me know. You mentioned before that you think it maybe better to fork what I am doing into a new project, which is fine, but, I am a big fan of open source and I think TAGUI could benefit from an open source approach and increasing the usage /contributions. I think the suggested approach in the demo-for-ken branch allows for that but there is no problem if you would prefer to extend TAGUI as it is today.

Gus

[dublindrupaller]

@klichukb Just saw your posts. Interesting stuff. FWIW, the main reason I like TAGUI is for the following reasons:

1. I joined a project that used selenium IDE (not webdriver) to build up a suite of a few hundred automated tests over a number of years to catch regression against new development in a continuous integration setup. Selenium IDE is currently not useable now with firefox making it extremely difficult to maintain and update those automated tests. A new selenium IDE maybe developed but that project is stuck in a bad place.

2. With TAGUI and the suggested approach I have put forward to @kensoh above in my previous comment, allows teams or individual engineers to "future-proof" their tests. In other words, if the project I joined used TAGUI, they would have a few hundred natural language tests they could simply switch to webdriver or puppeteer or other test script language framework. Instead of having to (a) choose a new framework that works...and (b) rewrite all those tests again in the new language.

3. In the same breath, the demo-for-ken branch I have created from TAGUI is only illustrative of the ability to switch languages. more work would be needed to allow TAGUI users to switch the language context from "casperjs", which works on that branch as the default, to "puppeteer" or selenium "webdriver".

As an aside...for me, puppeteer looks promising, but, is still in the early stages. My choice, after evaluating quite a number of other tools / frameworks is casperjs at the moment....the ideal scenario is to use TAGUI using casperjs now...with the ability to switch languages later if casperjs gets superceded by another framework.

hope that helps the discussion.

Gus

[kensoh]

Hi @dublindrupaller thanks for looking through and sharing your thoughts! The login step could be broken down into combinations of click or type steps to simulate the user actions on the GUI. For choose step there is already a select / choose step to handle drop down options which may already work out of the box.

For extending I get your idea, you are suggesting breaking up or at least supporting a separate file to allow custom steps defined by users. This seems tough now to separate. For example, adding a new step would need 4 changes in tagui_parse.php -

// 1 addition under parse_intent()
case "new": return new_intent($script_line); break;

// 2 additions under get_intent()
if (substr($lc_raw_intent,0,4)=="new ") return "new";
if ($lc_raw_intent=="new") return "new";

// below new function to do the actual translation
function new_intent($raw_intent)

Another 4 matching changes for above also needs to be added to tagui_header.js. tagui_parse.php is to generate the js code for automation. While tagui_header.js has these matching code in order to handle live mode, ie when user put in live step, it allows interpreting the natural language steps directly through live input and interact with chrome etc to see the effects live.

So with the current top-down structure, your above separation can't be implemented. Early on, I thought of putting all the translation logic into separate files so that in the future TagUI can support translating to other targets besides CasperJS. Maybe to WebdriverIO or others. But I drop that choice in favor of putting all the definition within the same file for the fastest iteration and development speed (easy to debug and change). Also sending to multiple targets may be confusing and impractical as it will introduce differences in outcomes due to fundamental differences in the target APIs.

Would like to hear more about your thoughts. I think it will be great to allow making custom definitions without having to re-edit each time there is a new TagUI release and the custom changes have to be reinserted. But I haven't thought of a way to do that without increasingly complexity of the main interpretation engine.

Oh I relook your sample function and understand why you want a separate login function. You want to access URLs dynamically. I think this can already be done now, just assign your variables according and call the URL in the flow -

username = "'jamesbond'"
http://full_url/?username='+username+'

or

https://full_url/?username='+username+'

[dublindrupaller]

Hi @kensoh,

Thanks for responding so quickly.

I had a quick look through the functions you mentioned and I think we can do this and fine-tune the php handling get_intent and parse_ intent so it scales up as well as allowing custom steps. Your idea of moving the translation logic into separate files is a good one...and it will avoid repetition.

The login example may have been a bad example to illustrate what I would like. I would tend to have jenkins jobs setup to run the casperjs tests against localhost, system_integration, UAT, PAT etc. a lot of different environments and there are a lot of different types of logins...editor, publisher etc. so the custom step of "login editor in german" would be very useful.

I also have scenarios where, on search, for example, I would like to switch languages and compare search results.

The chosen tagui step doesn't work out of the box for the chosen jquery unfortunately, hence the desire for a custom step.

The big "win" for me is the ability to write casperjs tests in natural language and keeping them "DRY" (don't repeat yourself). For a given STEP used across 100 tests...you only need to change the casperJS in one place. That's very useful.

Like I say, I think my query maybe possible...I have created a branch locally to play with and test. I will try and grab some time during this week to work on a branch based off your master and share with you to see what you think.

[kensoh]

@dublindrupaller I see.. thanks for sharing your thoughts and use cases.

Adding on, if the new custom steps uses CasperJS methods outside of those already used by existing steps there is a need to define custom functions to talk to Chrome browser, in order for the custom steps to work on visible / headless Chrome. What happens is I directly let TagUI talk to Chrome using DevTools Protocol through a websocket connection. Instead of using an intermediary such as chrome-remote-interface. That design choice was made in order to let TagUI unzip and run without having dependency on having a Node.js development environment.

I think the bulk of work for TagUI is already done, so I'm more open to introduce complexity than during pre-v2.0. In the longer time horizon, there may be a need for overhaul TagUI. PhantomJS development has officially stopped. CasperJS appears to have technical roadblocks integrating headless / visible Chrome. There may come a time when existing web standards change so much that current TagUI implementation base on CasperJS / PhantomJS is not ideal anymore.

If that reality starts to materialize, then there will be overhaul of the core engine to maybe Node.js + Puppeteer. There will be trade-off by introducing Node.js dependency, but may be a necessary path if needed. This overhaul is my major consideration when making significant changes to TagUI.

Lastly, you may have already heard of Puppeteer. It's a web automation tool from Chrome team. If I were to draw an analogy, Chrome + DevTools Protocol is something like what PhantomJS does. Puppeteer is something like what CasperJS does. I strongly recommend checking out Puppeteer. I think there are synergies which are unprecedented in this space, by having such vertical integration. To me, the launch of Puppeteer has ushered in a new era in this space, just like when PhantomJS appeared 7 years ago.

[kensoh]

No hurry, thanks Gus! Feel free to make multiple commits so that the development makes sense to you and minimize dev time. If this can be implemented, commits to fine-tune can be made later.

[kensoh]

Thanks Gus for your heads-up! Yes I saw some commits by your colleague John on the network graph. I looked through the code so far and it looks like the design is to do an OOP approach to split out tagui_parse.php into different files for each step definition, so that custom classes can be added easily by creating new PHP class files. Below are some points to check with you on your thoughts?

From the perspective of a single-maintainer project, it looks harder to maintain and to migrate to Node.js / Puppeteer later on. It may be best practice design for larger projects and multi-member team, but separating this way is harder to debug and iterate. It is like trying to split up the casperjs.js file into multiple files and classes instead of keeping it as one file amongst many other modules in the project.

The gain (easier custom steps) may not offset the cost (maintaining a more distributed code-base). Nevertheless this is something that you'll want to use and likely other PHP folks who would benefit from. If the final demo is something which I couldn't handle merging into master, would you consider forking or creating a new repo on your account? So that I can link to your project from the readme page.

[kensoh]

Did more comparisons on best practices. similar to CasperJS, other projects such as Puppeteer, Nightmare, Chromeless do not modularize the different web page actions into different modules / files.

The reason may be hard to maintain in exchange for ability for users to extend without modifications being overwritten by new releases. Though I can see you are heading towards a working solution (your newer commits to build the header js code during runtime).

[klichukb]

Hey @kensoh. I'm surprised by amount of quality research you do in this area. And the TagUI is awesome. Keep up the good work!!

I can't keep myself from asking your advice for my case. It has following requirements:

• I need an automation tool in headless mode, not for purpose of testing a flow of actions, but just getting some jobs done.
• I don't really care if it runs on multiple browsers, since i just need the job done, it can be a single environment, say Chrome, whichever offers best toolset.
• I'd need to be able to run multiple different scenarios with each taking up minimum system resources, so it would be cool if it wouldnt be too heavy... :)
• Probably i'd be willing to implement some basic decision making based on state of scenario execution, so i foresee it as a flow of instructions to the automation tool, maybe while getting some feedback.

What would be the best platform to use within mentioned? TagUI seems nice, as well Puppeteer and other guys are heavily popular right now, its hard to see which of them would best fit the case...

Sorry for overwhelming question, but i'm excitted by how experienced you are in this realm :)

[kensoh]

Thanks @klichukb !

From your requirements, it looks like Puppeteer is a good fit. In fact, in the TagUI developers reference section, I'm recommending potential TagUI users to check out Puppeteer first. I've also included there some other recommendations specifically for purpose of data-scraping and test automation.

The main reason for recommending Puppeteer is it has unprecedented synergy. The team creating the automation tool is the same team creating the Chrome browser. I have already seen issues raised which quickly gets patched upstream in Chromium. This tight level of integration is something no other web automation tool can match. The only thing to take note of is it is still in beta and I think it is going for initial release in Oct or Nov, by then the API would be more stable. Also, for process automation, there are now more options in terms of RPA (robotic process automation) tools. For example WorkFusion is coming up with a free version, UiPath has a community version.

So in general, I'll recommend potential users of TagUI to check out the other tools above depending on their specific use cases (process automation or data-scraping or test automation). If the other tools can't be used for some reason (due to limited developers on the team or limited budget for the commercial tools etc), then consider TagUI. Since it is designed to write the automation in natural language like syntax, and it doesn't need Node.js environment. Business users with some basic web or IT knowledge should be able to pick it up in a couple of days I guess.

Over the long term, if the underlying CasperJS/PhantomJS framework can't keep up with changing web standards, then TagUI has to switch its core to Puppeteer. Whether that can be done successfully is a question mark, and there is probably no 100% backward compatibility since a Node.js execution environment is very different from triggering CasperJS/PhantomJS processes from shell.

[klichukb]

Great feedback, thanks, @kensoh. Looks like Puppeteer is the most promising one since its backed up by Google Chrome team. I will also assess by how easily i can spawn such headless executors for multiple tasks.
But yeah, the fact that its beta gives me time hold off a bit, there are around 150 issues, on-going bugs and it might be hard to rely on one or another feature/glitch YET. Chromeless looks like very raw. In that sense Nightmare seems to be the mature one yet very popular, but I bet its worth waiting and contributing to Puppeteer.

I'm surprised as well by WorkFusion, they seem to be promising great features, especially if they're free.
Sorry if I have skipped documents that have the answer but what is the main problem for having Node.js in TagUI? Does it make the tool not that easy in installation for non-tech users?

[kensoh]

Yeah Puppeteer in my opinion, should be the first tool that developers consider when making a choice in web automation space. Unless it is very urgent and can't wait for v1.0.

NightmareJS in fact already has an integration with headless Chrome in the branch v3 for some time. Check it out if that works for your use cases. But it is not easy to merge that to master because there are many business-continuity considerations to be taken care of in order not to break the existing user scripts / assets that have already been developed based on the current Electron engine.

Oh I don't think I wrote that in doc. PhantomJS and CasperJS are not Node.js modules. CasperJS has a npm package to make it easier to install that's all. The official PhantomJS package is also done by someone from Medium for convenience, not released by PhantomJS maintainers.

So there is really no need to use Node.js for now, as it requires users to install Node.js development environment just to use TagUI. Versus now users can just unzip and run on the 3 OSes (whether done manually to download and unzip or using shell scripts). Developers with Node.js environment can use npm install tagui.

[klichukb]

Thanks @kensoh, for broader overview of the tools. I'm not acquianted though well with Selenium, but i've always wondered why, being almost a standard, there are so many alternatives? Isn't selenium something worth considering for similar situations?

[kensoh]

I'm not sure about this as well.. Personally I thought that Selenium is great, being an open-source and free tool it enabled a lot of use cases. However there are many issues that people have problems with, for example you can look at the Puppeteer Hacker News thread, just search selenium and you can see the different types of complaints about it - https://news.ycombinator.com/item?id=15028329

[kensoh]

Keeping issue open for now, there's a lot of useful info - Gus's inputs and contributions, the general discussion and my recommendation to try Puppeteer as the first approach for Node.js developers.

[kensoh]

(1/2 - I'll post my thoughts first on the demo branch, then on the next steps you are looking at)

Hi Gus, no probs.. I have looked through twice over your comments above, and the readme, as well as the demo-for-ken branch. Want to say, amazing work that you have done!

I added some comments few weeks ago, that separating every TagUI steps into separate files is not the best practice used by other tools such as CasperJS, Puppeteer, NightmareJS or Chromeless. I guess the main reason is much harder to maintain the code. Eg in the case of micro-services to break up when a service becomes too large to handle.

The other bigger factor, looking at the program structure in the branch, it is not a form which I can easily continue maintaining. In that form, the price to pay for gaining continuity and extensibility is too high.

I run into following issues when trying to run the demo branch, it is most probably due to differences between our environments, can you advise more? I tried with the positive test cases and a bare-minimal flow. I also tried copying the CasperJS / PhantomJS / SlimerJS binaries to be in the same src folder.

TA:src kensoh$ pwd
/Users/kensoh/Desktop/demo/src
TA:src kensoh$ ./tagui test/positive_test
<ERROR - [LINE 370] frame called consecutively frame mainframe | subframe
ERROR - [LINE 374] invalid after frame - popup url_keyword
ERROR - [LINE 376] invalid after frame - popup url_keyword
ERROR - [LINE 377] frame called consecutively frame mainframe
ERROR - [LINE 379] invalid after frame - popup url_keyword
ERROR - [LINE 380] frame called consecutively frame mainframe | subframe
ERROR - automation aborted due to above

TA:src kensoh$ cat sample
http://casperjs.org
TA:src kensoh$ ./tagui sample            
<ERROR - automation aborted due to above

To tackle extensibility problem, there should an easier to maintain way but I haven't figured out the best way yet. For example, having a global configuration .csv file for global TagUI translations / mappings and a local configuration .csv file for user-defined translations / mappings. So in each release, only global .csv file would be overwritten and upgraded, while the user-maintained copy is left untouched.

One overarching goal of TagUI is to let non-developers or people with little coding experience to benefit from the power of web automation. That's why the core is using natural-language-like syntax. Having some .csv config files to define such translations would be preferred than the object-oriented style of having different classes for each step, as that significantly raises the barrier for user customization to people who not only knowing coding, but PHP coding.

The continuity problem is a harder problem, because we know that over time, foundational structures can change so much that sometimes an overhaul is needed. In this case, the underlying engine is CasperJS, which runs within PhantomJS engine in a non-Node.js environment. Though it will be great for TagUI to concurrently able to run on CasperJS and Puppeteer, in practice, I believe it is easy to create separate branches to do that instead of handling both or more within the same code base. It exponentially increases the order of complexity for maintenance and iteration speed.

For example, the entry point for TagUI is actually a Linux shell script or a Windows batch script. In order to support both CasperJS and Puppeteer, the base scripts would be very different to trigger a run with casperjs or with node. This is not yet factoring differences in what can be supported within a Node.js based solution and CasperJS. Still, these are the smaller problems. The larger one is the translation is not straightforward in the sense step A = some string of code for step A. Some Node.js based solutions uses the chaining API, and their handling of frames or popups are very different in code layouts, compared to CasperJS. This may make it impossible to handle even when steps are separated into different classes.

For continuity, I'll think making a new branch when the time comes, and then writing a migration script to automatically convert existing flow assets to the new form (due to unadjustable API limitations in changes between CasperJS -> Puppeteer) would be the most efficient way. At the moment, I don't think it is ripe to start migrating to Puppeteer and a Node.js based engine. That immediately increases the barrier that users without a Node.js installation can't use TagUI, and cannot unzip and run it directly.

I appreciate the work that you did, what do you think of having a dublindrupaller-oop branch within TagUI project so that your work can be recognized and used by others? Or if you prefer, hosting your own fork and I can direct to it from TagUI readme homepage. I don't think I will be able to maintain the structure in demo-branch due to above considerations. At the same time, you have 100 over well-thought-out commits which will go to waste if not made available to other users.

[kensoh]

(2/2 - 2nd post to discuss my thoughts of the next steps you raised)

1. running with different URLs - at the moment TagUI can run different URLs by specifying the URL as an item in object repository. for example starting the flow file with `env_url` and having a csv with the same name as flow file to contain that definition. also by using datatables, you can save the urls to be used for different environments and automatically run for all the 3 environments. would that work for your use case? or is the limitation due to the error-checking for valid URL and http:// https:// string.

2. letting TagUI work concurrently for different targets such as CasperJS and Puppeteer - the issues are mentioned in above comment. mainly due to different code structures and execution environment. would think it is much harder in complexity than maintaining 2 separate branches.

3. minifying code base, for eg tagui_header.js - minifying / uglifying to reduce file size won't speed up execution time noticeably, this is because in a normal automation run, most of the time taken is due to network latency waiting for the server side to respond to some actions, for an element to appear or some change in UI to happen. also, the execution is already done locally so there shouldn't be extra time taken to transfer the .js file locally for execution. on a separate note, TagUI can run automation flows hosted online. just run specify the URL of the flow file as the 1st parameter instead of a local file. in this case, the natural language syntax is already minimal in file size, so optimization here won't be relevant.

I'm aware the recent issue with Selenium IDE being not able to be supported in newer versions of Firefox. From my limited knowledge, I think Kantu for Chrome is the spiritual successor to Selenium IDE. Although it is developed by a totally different team, I think it may be able to meet the needs for similar purpose. I strongly encourage everyone to check it out if you haven't!

UPDATE - On another topic, do you have some URLs or are you able to email me some samples of your Selenium IDE generated flow assets? You can email me at [email protected] or you can try dragging and dropping the zip file into this comment if there is no sensitive private info. I'm thinking there may an easy pattern and some level of similarity in structure between those scripts and TagUI scripts.

If there are, it is probably easy to write simple migration scripts for Linux and Windows to migrate those assets to TagUI flow format. If there are TagUI users interested in that I can look at it, to see if I can make a translation engine to migrate those assets.

[dublindrupaller]

hi Kensoh,
Thanks for the considered (and lengthy!) response. Appreciate it.
I have been swamped recently but will get back into this soon. unable to send you links or the selenium IDE tests I am afraid, I work in a bank here in switzerland and I'm working on internal intranet stuff. thanks for the offer though.
I think your idea of creating a new project is a good one. I can see the direction you are going in and I think you are correct that I am thinking of a slightly different objective and the two don't meet.
One thing I would disagree with is the multiple file thing....most contemporary libraries and frameworks are built the same way...lots of modular code separated out into logical patterns and files. When they are compiled, that is when they are merged together or "minified", so to speak. That said, I agree with the core thrust of your comments...and will follow up when I get some breathing space to create a new project and let you know when it is ready for consumption.

thanks again,

Gus

[kensoh]

Hi Gus, I see no probs. Yeah in that scenario, it would be a breach of discretion and confidentiality.

I guess probably for larger scale projects or enterprise projects. For similar web automation projects such as Puppeteer, or CasperJS, NightmareJS etc, the steps definitions are not separated into multiple files per step but kept within the same file.

But I definitely see your perspective. I think personally I use vi for development instead of IDE, so I try to keep structure simple to maintain. I'll close this issue for now and reply to your other comment. Keep in touch and I'll keep a lookout for your inputs!