Display VirtualHosts for logs

[simark]

Display VirtualHosts for logs

[allinurl]

@simark there are plans to separate the data based on virtualhost. However, what I'm thinking from your comment above is to have a sub node underneath each request file that displays the virtualhost. i.e.:

11052 2.70%   63.93 MiB  45.10 ms /page.php
                                  ├─ www.mydomain.com
                                  ├─ www.example.com

Thoughts?

[simark]

Putting vhosts as subnodes of pages wouldn't really make sense, unless you know that both domain are aliases/point to the same documentroot.

I have for example a couple of users who have wordpress installed, so I get a lot of /wp-login.php, but they are different pages.

[abgit]

@simark, goaccess cannot group stats by virtual host or folder. maybe in a future version that will be possible.
Seems that the only way i can think to get independent stats based on virtual host is to generate a log file per virtual host. You should configure your http server to do that, or use a tool to split a log file based on a virtual host directive (eg: with awk)

best,

[simark]

@abgit, your comment made me remember about the command split-logfile, which is bundled with Apache and does just that.

For now, I hacked it by adding the parsing of vhost in parse_format. I then concatenate the vhost and the page. My pages look like "www.example.com/page.php", which produces the desired result. Would you be interested in a contribution that does that but cleaner (add a vhost field and column in the interface)? I suppose the hash tables that keep track of statistics would need to be augmented to use structures as keys instead of only strings, in order to aggregate by (vhost, request).

[simark]

For reference, here is what I did: simark@0ff83b8

[allinurl]

@simark, Contributions are welcome. Just share some thoughts on how the interface would look like.

[kepi]

+1000 to this future. Without it goaccess is totally unusable for most of our environments but with it it will be killer app :)

Split log files isn't way because we want see how our servers are behaving and what hosts are most demanding.

[allinurl]

@kepi, are you looking to concatenate the vhost to pages as @simark stated here?

[kepi]

I looked into it but I think that it is only quick hack to achieve at least something.

It will be better to design it to use vhosts.

One of approaches easy to add can be to add columns with vhost to views requested files and requested static files or better to switch very metric to take numbers from whole server or per virtual host. It will be great to add another metric with request per vhosts (not dependent on uri).

Another approach is simply to switch whole view asi i.e. in apachetop.

This wont' be small change probably but will be really useful.

[simark]

@kepi: What I had in mind, an option "group by virtualhost", which would add a column in the relevant views if selected. I started to work on code to group on arbitrary values, but I am not sure how far I went. This would allow quite flexible views. By grouping only by vhost, you could get your number of requests per vhost.

I'll try to get back into it soon and upload something useful, even if not completely finished.

[allinurl]

I like the idea of grouping vhosts. What I was thinking is to have some sort of interactive interface (perhaps a vertical tab interface for each vhost?) within both, request, and static requests which will essentially group requests by virtualhost. Any thoughts?

[kepi]

Grouping can be nice. Only keep in mind that one can have many virtual hosts (we have couple of hundreds i.e.) so show only top of them as in normal view for requests.

[ottok]

I think that the table "Requested files (Pages-URL)" should stay as it is. If you only want to have the results of a single vhost, you can either in Apache/Nginx settings very easily set you vhost to write a separate log per virtualhost or you can with simple grep/awk do some pre-processing of the access.log file before you pipe it to Goaccess.

However at the moment Goaccess seems to discard the vhost data completely. I think Goaccess should have an additional table titled "Top sever hostname" which would list all the different virtual hostnames found in the log like this:

Hits |  %   | Bandwith | Time served | Server hostname
3467 | 50 % | 55.2 MiB |      1810 s | example.com
2345 | 33 % | 22.1 MiB |       810 s | example.net
1234 | 12 % | 10.9 MiB |       210 s | www.example.com

If Goaccess had a table like this, admins could see what domains are used in the requests and maybe it whay make a mistake in grep/awk and wrong multiple unwanted hostnames slip in, they would notice them from this table.

[allinurl]

Added the ability to parse virtual hosts and a new panel to display metrics per virtual host. This was shipped in v0.9.5.

As @ottok pointed out, it displays data on a separate panel, e.g.,

 9 - Virtual Hosts                                                         Total: 230/230

 Hits    Vis.      %   Bandwidth Avg. T.S. Cum. T.S. Max. T.S. Data
 ------ ----- ------ ----------- --------- --------- --------- ----
 144909 10684 58.47%    1.99 GiB 325.35 ms  13.10 hr   4.36 mn www.example.com
 46849   9784 18.90%  738.80 MiB 307.65 ms   4.00 hr  56.18  s example.com
 19816   1455  8.00%  326.16 MiB 304.54 ms   1.68 hr  56.67  s www.domain.com
 11464     87  4.63%   54.70 MiB 364.63 ms   1.16 hr  40.06  s mail.googlex.net
 6129      54  2.47%  415.13 KiB  45.00 us 280.85 ms   9.41 ms box.yahoox.com
 4713     520  1.90%   55.66 MiB 208.47 ms  16.38 mn   8.16  s www.example.com.au
 3670     146  1.48%   13.24 MiB   1.08 ms   3.98  s  53.27 ms www2.example.com

#117 should cover the filtering by vhost, and other fields.