unlsitemaster/site_master

site_master's Issues

"Verify Myself" is shown on users that are not me

But I'm already verified and this is not me:

Member Verification File Content

You ask the user to create a specifically named file on there server. It would be much easier if you gave them a file to download (clicking is easier than copy/paste).

I also suggest that a template verification file include some text about what the file is and who it's for:

This file {FILENAME}
belongs to {USER} {EMAIL}.

It's a verification key for SiteMaster.

I would also include some copy about if/when it is safe to delete the file.

issue scanning sites with https

If a site's base URL in webaudit starts with https, but the site contains links that are both http and https, only the https links are seen and the http links are ignored.

Add number of pages on to "My Sites" page to view which sites are larger than others

We had a request come in where it would be nice to see a page count on the My Sites page so you can easily compare the different sites

We would probably want to convert the panels on the My Site Page to a grid (/www/templates/html/SiteMaster/Core/Registry/Sites/All.tpl.php)

We would also need to use the getPageCount func from the site records (/src/SiteMaster/Core/Registry/Site.php)

301s

• Bring back 301 checking.
• Alter the points so that a 301 doesn't impact score, but we still get the list of links to update.

Improve 4.0 Theme integration

Suggestions:

• Remove padding from .site-list
• Add inner-wrappers to provide page content bottom padding.
• Move the .site-list .site styles to .panel

Mysterious "Pending Roles"

I should not have any pending roles on my site, but 4 (3 of the same type) appear. Also there is no way to "Deny" these roles.

Needs textual feedback for self reported progress slider

Without some sort of textual context, I have no idea what kind of value I am selecting and what value I have selected.

Add a status page

This state page would keep a record of all events which caused mass 0 scores.

These scores are due to a outside service that are used/linked on a majority of pages. When these services go down it causes each page to get a bad score and leads to the site getting a 0.

These 0's would also prompt a notice in the email notification that says that the issue may be related to an outside service and to check the status page for details.

Better help text for the metric_links

For 301 errors include the URL that the link should go to.

For all errors, include the link text to make it easier to find on the page.

Make the 'My Sites' list contain site navigation too

Fix erroring db tests

$ php vendor/bin/phpunit --bootstrap init.php tests
PHPUnit 3.7.29 by Sebastian Bergmann.

....EEE.E............

Time: 649 ms, Memory: 4.50Mb

There were 4 errors:

1) SiteMaster\Core\Registry\QueryDBTest::getByUser
Argument 1 passed to SiteMaster\Core\Registry\Site\Member\Role::createRoleForSiteMember() must be an instance of SiteMaster\Core\Registry\Site\Role, boolean given, called in /var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php on line 50 and defined

/var/www/html/src/SiteMaster/Core/Registry/Site/Member/Role.php:38
/var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php:50
/var/www/html/src/SiteMaster/Core/DBTests/DBTestCase.php:49
/var/www/html/tests/SiteMaster/Core/Registry/QueryDBTest.php:95
/var/www/html/tests/SiteMaster/Core/Registry/QueryDBTest.php:14

2) SiteMaster\Core\Registry\QueryDBTest::getByURL
Argument 1 passed to SiteMaster\Core\Registry\Site\Member\Role::createRoleForSiteMember() must be an instance of SiteMaster\Core\Registry\Site\Role, boolean given, called in /var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php on line 50 and defined

/var/www/html/src/SiteMaster/Core/Registry/Site/Member/Role.php:38
/var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php:50
/var/www/html/src/SiteMaster/Core/DBTests/DBTestCase.php:49
/var/www/html/tests/SiteMaster/Core/Registry/QueryDBTest.php:95
/var/www/html/tests/SiteMaster/Core/Registry/QueryDBTest.php:50

3) SiteMaster\Core\Registry\QueryDBTest::getByAll
Argument 1 passed to SiteMaster\Core\Registry\Site\Member\Role::createRoleForSiteMember() must be an instance of SiteMaster\Core\Registry\Site\Role, boolean given, called in /var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php on line 50 and defined

/var/www/html/src/SiteMaster/Core/Registry/Site/Member/Role.php:38
/var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php:50
/var/www/html/src/SiteMaster/Core/DBTests/DBTestCase.php:49
/var/www/html/tests/SiteMaster/Core/Registry/QueryDBTest.php:95
/var/www/html/tests/SiteMaster/Core/Registry/QueryDBTest.php:73

4) SiteMaster\Core\Registry\RegistryDBTest::getClosestSite
Argument 1 passed to SiteMaster\Core\Registry\Site\Member\Role::createRoleForSiteMember() must be an instance of SiteMaster\Core\Registry\Site\Role, boolean given, called in /var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php on line 50 and defined

/var/www/html/src/SiteMaster/Core/Registry/Site/Member/Role.php:38
/var/www/html/src/SiteMaster/Core/DBTests/BaseTestDataInstaller.php:50
/var/www/html/src/SiteMaster/Core/DBTests/DBTestCase.php:49
/var/www/html/tests/SiteMaster/Core/Registry/RegistryDBTest.php:16

FAILURES!
Tests: 21, Assertions: 52, Errors: 4.

Error + Success message after deleting member

I removed all roles for a user, but the success message said it was added and the page I was redirected to gave a "Could now find membership" error.

Change backslash to forward slash

Anywhere it says Add\Edit to Add/Edit

Add notices for social optimization, possibly under the SEO metric

This would help identify pages that would not be displayed well when shared on twitter, facebook, or other social platforms.

Site Verification is not working got colorado.edu sites

Not sure if this is the right issue queue. I was able to authenticate via InCommon, but I've tried both the file and the metatag verification using the options provided by https://www.drupal.org/project/site_verify

https://www.colorado.edu/webcentral/sitemaster_v_a188be672e9dd34ee5b75b56e5709cc7.txt

The verification step always fails.

Approve Pending Roles is missing a CSRF token

So submitting the form always fails.

https://github.com/UNLSiteMaster/site_master/blob/master/www/templates/html/SiteMaster/Core/Registry/Site/MembersForm.tpl.php should be edited to include something like

site_master/www/templates/html/SiteMaster/Core/Auditor/Site/Page/ScanForm.tpl.php

Line 6 in d141623

<?php $csrf_helper->insertToken(\SiteMaster\Core\Controller::urlToRequestURI($context->getEditURL())) ?>

Place PHPUnit Dependency in "require-dev"

Should https://github.com/UNLSiteMaster/site_master/blob/master/composer.json#L28 be in "require-dev"?

https://getcomposer.org/doc/04-schema.md#require-dev

Making that change allows other developers to exclude that package from production builds as well as makes a distinction between which packages are needed for development of the project and which packages are needed to run the project.

Track GPA over time

This would be for sites AND overall system

Look into why multiple users are being created with the same uid and provider

Explain where "Site Information" is used

How does the system use "Site Title" and "Support Email Address"?

Some links cause an endless loop

Take this scenario

A website has a badly coded relative URL link, we keeps appending the link's path to the end of the url. It would look something like this: http://www.example.com/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/contact-us/co

SiteMaster has logic to prevent such a URL from stalling the system. It does this by taking the md5 hash of the URL looking for another page in the scan that has the same hash.

This does not appear to be working correctly in some circumstances. I believe this is because of URI truncation in the database.

The script looks first for the full url, then gets inserted as the truncated url. because the truncated url has a different md5 hash than the full URL, it will result in duplicate pages being scanned.

place a limit on the number of links to check

There is currently no limit to the number of links that will be checked by the link checker metric. This is perhaps the main source of slow scans.

Currently, there are on average 80.73 links/page. To be generous, I propose implementing a limit of 150 links per page.

SELECT AVG (total) FROM (SELECT count(*) as total FROM `scanned_page_links` GROUP BY scanned_page_id) as query

Cannot Login if active UNL SSO session exists.

There is no login link or button in the navigation/home.

I don't particularly like auto login for services, so I would prefer to have to click a button than to have the system attempt a gateway auth.

Tree view for site pages

The list for site pages can easily become unwieldy for large (segmented) sites. Implementing a tree view would help better locate pages quicker.

strip URL `#fragments` from redirects while downloading

There is a bug in CURL versions < 7.20, which cases requests to a url with a fragment (index.php#fragment) to return a 400 HTTP response. Because redhat 6 and other distros only support 7.19, I am including a workaround

Remove the 'verify member' function for already verified members to verify someone.

This will be done by assigning the 'admin' role instead.

update libs on vagrant up

Some HEAD requests return false positives

Some HEAD requests return a false positive when checking links. For example, they will return a 404, 500, etc while the same GET request will return a 200.

Page remains a "hot spot" even after being fixed

If a page is a hot spot for a metric and that page is then fixed/changed by a single page scan, the "Current Report" will still show the page as a "hot spot" even though the GPA and pages all reflect "A+" status.

Admin verification error allows injecting content

The error messages encountered during verification (e.g. 404) are displayed to the end user, which allows XSS and content injection.

Consider using rel="canonical"

This <link> is used to tell robot agents the permalink for a page. This URL could be used to reduce the number of duplicate pages that appear in a scan. For example, a product page that has a URL for every category it's in.

Hide "Schedule" buttons for site/page if already queued

Or do you want to be able to queue something already queued?

Scans fail when a base_url is upgraded to https

When a site is upgraded to https, the scan fails. It should detect the change, update the base url and restart the scan.

Add unique id attribute to metric-grade-details sections so they can be linked via anchors

http://webaudit.unl.edu/sites/774/pages/101/

It'd be helpful to be able to jump to a specific metric section in the report with named anchors.

HREFs with spaces result in bad requests

The href attribute of an anchor element is allowed (by spec http://www.w3.org/TR/html5/links.html#attr-hyperlink-href) to be padded by spaces. So the hrefshould be trimmed before sent off to a metric for further work.

Add a list of "pages that linked to this page"

I often get the question "This page shouldn't be linked to, how did your tool find it?".

Don't allow the removal of member roles in GUI if they were set by another system

404s on pages where the baseurl redirects

Take http://www.unl.edu for example. On campus it redirects to http://news.unl.edu/newsrooms/unltoday/

Currently SiteMaster will follow the redirect, and scan, but it still thinks that the baseurl is www.unl.edu/ so any relative links that it finds are converted into absolute links incorrectly. That means there are a lot of 404s.

Perhaps it would be better to not scan baseurls that redirect...

Use puppeteer instead of nightmarejs

This project was created before puppeteer was released. Thus we used the best headless browser at the time, nightmarejs. Now that Google Chrome has an official headless browser, we should use that.

List of pages found on site (URIs) seems to be all lowercase

URIs which are case sensitive are converted to lowercase when presented in the list of pages found on site.

Move the verified status into its own role (admin). base isVerified logic on that role

Don't show epoc ( 12-31-69 6:00 pm ) date for queued scans

If a scan is queued the default date should not be shown and the GPA/info information should also be hidden.

update install, update and testing docs in the readme