matomo-org/plugin-visitorgenerator

Plugin to create fake visits, websites, users and goals to populate Matomo reports

Matomo VisitorGenerator Plugin

Description

Plugin to create fake visits, websites, users and goals. This can be used by Matomo users or developers as an easy way to generate fake data to populate Matomo reports.

You can overwrite the log files that are used to generate fake visits in plugins/VisitorGenerator/data or add new logs to the data directory. All files ending with *.log will be replayed.

Plugin developers can provide their own log files by placing '*.log' files within a 'data' directory of their plugin. This way plugin developers make sure there will be always useful test data.

Usage

UI

The plugin adds a new item to the Matomo admin UI visible only for users having Super User access under the section "Development". There you can select a site and for how many days in the past you want to generate new visits.

Note: you need to first enable the Development mode in Matomo. In the root directory of your Matomo install, run the following command to enable development mode: ./console development:enable

CLI

It also adds the following commands to the Matomo CLI tool:

• Generate visits

• Generate goals

• Generate users

• Generate websites

• Generate annotation

• Generate visits directly to database

• Shorten log file

• Anonymize log file

Examples

• ./console visitorgenerator:generate-annotation --idsite 5 generate one annotation for the current day for site with id 5
• ./console visitorgenerator:generate-goals --idsite 5 generates some predefined goals for site with id 5
• ./console visitorgenerator:generate-users --limit 100 generates 100 users
• ./console visitorgenerator:generate-websites --limit 100 generates 100 websites
• ./console visitorgenerator:generate-visits --idsite 5 generates many visits for site with id 5 for today
• ./console visitorgenerator:generate-visits --idsite 5 --days 2 generates many visits for site with id 5 for today and yesterday
• ./console visitorgenerator:anonymize-log /path/to/log takes an Apache log file, anonymizes it and places it in a data directory so it will be replayed the next time "generate-visits" is executed
• ./console visitorgenerator:shorten-log /path/to/file.log > file.short.log takes a large Apache log file and keeps only a small number of logs per day
• ./console visitorgenerator:generate-visits --idsite 5 --custom-matomo-url=http://example.com/ Uses 'http://example.com/' as Matomo-URL and generates many visits for site with id 5 for today
• • ./console visitorgenerator:generate-visits-db --idsite 5 --limit-visits=200 --days=3 --threads=8 generate 200 visits per day for the last three days directly in the database, using 8 threaded processes

Other notes

VisitorGenerator makes a lot of requests to the Matomo tracking API to send the visits, so if your server blocks requests based on rules (e.g. with mod_security), you might want to create an exception rule for VisitorGenerator.

Alternatively you can generate visit test data quickly without using the tracker at all using the generate-visits-db option.

Using it in Matomo for WordPress

It only works in Matomo for WordPress if the plugin is installed through git as it's only intended for development.

In Matomo for WordPress you can find the UI in the Matomo Admin under "System".

You can also use the command line to generate the data:

cd wp-content/plugins/matomo/app
./console  visitorgenerator:generate-visits --idsite=1

Legalnotice

This plugin is released under the GPLv3+ license.

This plugin uses the Faker library which is released under the MIT license.

plugin-visitorgenerator's Issues

visitorgenerator:generate-live-visits : Error in Matomo (tracker): Invalid idSite: '0'

Hello,

I'm trying to use visitorgenerator:generate-live-visits to test campaign tracking and Visits views on a Matomo dev instance.

Console command is :

./console visitorgenerator:generate-live-visits --idsite 1 --custom-matomo-url http://localhost:8000/ --log-file plugins/VisitorGenerator/data/domain.tld_access_log --token-auth aGeneratedTokenAuth --day-of-month 0 --stop-after 120 --verbose

Output log from above command :

Generating logs...
  tracked 0 actions.
  sleeping 5s.
DEBUG [2022-11-04 00:34:33] 446308  Tracking log on line 4548.
  tracked 1 actions.
Out of logs, exiting.

Above command throw those errors on Matomo server side run as a local instance :

[PHP-CGI    ] [04-Nov-2022 00:34:33 UTC] - Error in Matomo (tracker): Invalid idSite: '0'
[Web Server ] Nov  4 01:34:33 |WARN   | SERVER POST (400) //matomo.php host="127.0.0.1:8004" ip="127.0.0.1" scheme="https"

Have you some clue ?

Thanks,

Eric

hey, i think you are expert in piwik, do you know how to develop a new login plugin with openid?

i will be grateful if you give me some advice, thank you!

Can't find the UI

"The plugin adds a new item to the Piwik admin UI visible only for users having Super User access under the section "Development". There you can select a site and for how many days in the past you want to generate new visits."

I am the Super User, but there isn't a section called "Development" and I couldn't find this UI in the admin panel (I'm using PIWIK 3.1.0). Please let me know where is the UI!

[Matomo 4.0] Send requests to matomo.php instead of piwik.php

In matomo-org/matomo#13596 and matomo-org/matomo#12785 we renamed to endpoint to matomo.php . This endpoint might be blocked by some installations and we may need compatibility of log analytics to adjust visitor generator. Be probably best to do this in Matomo 4.0 or so.

Add Option to provide a different Piwik(tracker)-URL, e.g. if Piwik is behind a loadbalancer

CURL Timeout with fake visits generator

Here is the following command line I have used:

./console visitorgenerator:generate-visits --idsite 1 --days 1 --piwik-domain http://www.me.lo:8001 --limit-fake-visits 2 -vvv

And the results:

[Exception]
curl_exec: Operation timed out after 5001 milliseconds with 0 bytes received. Hostname requested was ww.me.lo

I am in my dev environment and I am using the built-in PHP server listening on port 8001.

I tried to look in the source code of the plugin but I did ot find anything for now.

Please help !

Fix broken link from User Interface --to--> Readme

Broken link

As shown in the screenshot, if you go to Matomo >> Development >> Visitor Generator >> [README] it is a broken link.

Please edit so it reaches the following URL
https://github.com/matomo-org/plugin-VisitorGenerator

Error when i generate visits

When i type this in the console ./console visitorgenerator:generate-visits --idsite 2.
I get this error

[Zend_Db_Adapter_Exception]
SQLSTATE[HY000] [2002] No connection could be made because the target machine actively refused it.

[PDOException]
SQLSTATE[HY000] [2002] No connection could be made because the target machine actively refused it.

PHP 7.4 compatibility

Related to matomo-org/matomo#14821
Currently the plugins shows the following deprecation warning in PHP 7.4:

WARNING [2019-08-28 11:52:26] 17841  /var/www/matomo-beta/plugins/VisitorGenerator/libs/Faker/Faker/Provider/Lorem.php(94): Deprecated - join(): Passing glue string after ar
ray is deprecated. Swap the parameters - Matomo 3.11.0 - Please report this message in the Matomo forums: https://forum.matomo.org (please do a search first as it might have
 been reported already)                                                                                                                                                      

This has been fixed in fzaninotto/Faker@167d805, so either the library should be updated or the patch applied to remove the warnings.

Rename to Matomo in the plugin description page, and Author names

This plugin, and a few others, still show Piwik as Author, and reference Piwik in the description.

• Let's rename to Matomo in this plugin,
• and ideally in other plugins as well 🔢

Possibility to generate fake visits with a specific Channel Type

Like the title said.

Is it possible to generate some visits with a specific Channel Type?
In my case I want some more fake visits with Campaigns as Channel Type to properly troubleshoot an issue I'm working on.

Br, Jorge

Generate visit data for content tracking

Update the doc: Enabling development mode breaks deployed install

From: https://forum.matomo.org/t/enabling-development-mode-breaks-deployed-install/44444

Appartently you do not need to enable dev mode to use the plugin. Instructions in the plugin itself should be amended, since the ones in the website are correctly not mentioning the dev mode.

For tracing purposes: the post:

Just installed 4.6.2 in a plain U20.04 VM, no issues until I tried to enable development mode throut CLI in order to use the VisitorGenerator Plugin.

Enabling the development mode returns error in dashboard visualization as per attached and makes left side menu unresponsive when entering the settings menu.

HTTP requests log:

Appartently you do not need to enable dev mode to use the plugin. Instructions in the plugin itself should be amended, since the ones in the website are correctly not mentioning the dev mode.

Tracking failure when using plugin in Matomo 4.1.1

When using the Visitor Generator plugin in Matomo 4.1.1 the following tracking error occurs:

Measurable	Problem	Solution	Date	Page URL	Tracking URL	Action

Hilpert, Flatley and Bashirian (Id 2) | Request was not authenticated but authentication was required. | Set or correct a "token_auth" in your tracking request. Learn more | Feb 22, 2021 00:26:24 | http://example.org/?utm_campaign=bsa&utm_term=buildinternet | action_name=ExampleCode+-+technology+with+passion&idsite=2&rec=1&r=299079&h=7&m=17&s=55&url=http%3A%...

No errors are returned when running the console command to generate the visit data

No data is generated for the reports in Matomo

Using the latest 4.0.3 plugin version.

Generates visits only today

./console visitorgenerator:generate-visits --start-date=2007-12-20 --idsite 1 --days 1

creates visits only on the day the command is run.

See https://github.com/matomo-org/matomo-php-tracker/blob/a7809441e5909d331c19338c718f8800a4cad5e5/MatomoTracker.php#L1817-L1907 and debug there using above command.

It may be a configuration issue but I could not find out what it might be. Maybe it is a regression as found with the team.

For WordPress Version ?

possible to use this plugin for the matomo Wordpress version ?

Add possibility for plugins to hook into fake visits generation

Plugins should be able to generate visits/actions/parsms as well. For example in Bandwidth plugin I'd like to generate some pages with Bandwidth.

You must specify a SKU for the Ecommerce item

Hi,

When generating some visitors, I randomly got this exception:

  [Exception]                                    
  You must specify a SKU for the Ecommerce item  

I guess this was just bad luck, but I know too little about Ecommerce to find the source and just wanted to document it here.

Support a direct database insertion option for fast visit generation

Objective

Enhance the visitor generator so it can also be used to quickly create large datasets for performance testing.

The Problem

Currently the visitor generator creates visits by sending tracker API requests. This is great for testing API functionality and closely simulates a real visit, but has the downside of being very slow when trying to generate a large number of visits.

For example, generating 1,000 fake visits for 1 day takes, 2min30sec or 7 requests per second.
So generating 100,000,000 visits at this rate would take ~4,000hrs or almost six months.

Some Background

As part of previous load testing work this standalone load testing script was created and can insert fake visits directly into a database at a rate of up to 3,000 visits per second.

Given adequate server resources this should be capable of generating a 100,000,000 visit dataset in just over 10hrs.

However this script was only intended for short term internal use and therefore lacks the quality of fake data provided by the visitor generator and doesn't have the ease of use required for general availability.

Proposed Solution

Add the direct data insertion functionality from the load testing script into the Visitor Generator plugin as an alternative option to sending requests via the tracking API.

• Including this functionality in the visitor generator plugin makes it available to anyone wanting to test large datasets.
• The existing high-quality fake data generation code can be reused for direct database inserts.
• The Console command interface means it can be used to automatically generate datasets to a required size in CI pipelines.
• Normal usage of the visitor generator plugin is unaffected.

This option would only generate fake visits, not visits loaded from logs.

Command Syntax and Examples

./console visitorgenerator:generate-visits-db
    --idsite=1                          Site for which to generate the visits
    
    --days=xx                           Create visits for this many days into the past
    --dates=yyyy-mm-dd,yyyy-mm-dd       OR create visits between these dates
    
    --total-visits=yy               Create this many total visits across all chosen days
    --daily-variance                    Percent variance in daily visits for total-visits
    
    --daily-visits                      OR number of visits to create each day, either an exact
                                        number or a random range

Create visits for a specific year, there will randomly be between 2466 - 3014 visits per day and a total of exactly 1 million visits.

./console visitorgenerator:generate-visits-db --idsite=1 --dates=2023-01-01,2023-12-31 --total-visits=1000000 --daily-variance=10

Create visits 90 days into the past from the current date with each day having somewhere between 200 and 600 visits, there will be somewhere between 18,000 and 54,000 total visits

./console visitorgenerator:generate-visits-db --idsite=1 --days=90 --daily-visits=200,600

Create visits for the previous 7 days with 1,000 visits per day, 7,000 visits total.

./console visitorgenerator:generate-visits-db --idsite=1 --days=7 --total-visits=7000