Giter Site home page Giter Site logo

redis_toolkit's Introduction

Redis Toolkit

Toolkit for actively monitoring, analyzing and reporting your Redis database.

The toolkit has 2 types of reporting:

  • hit rate - actively monitors a redis database using the redis-cli monitor command (https://redis.io/commands/monitor), stores the commands Redis is running locally and then generates a report.
  • memory - dumps the contents of the Redis database locally and analyzes the memory distribution per key.

Installation

A valid docker install is required.

Clone this repository, go to the clonned directory and run the commands below. The output should be displayed on screen.

$ chmod +x ./redis-toolkit
$ ./redis-toolkit install

Update

The tool is under constant development. To update it to the latest version, just run the commands below.

$ git pull
$ ./redis-toolkit update

Usage

Hit Rate report

First you'll have to start monitoring the target Redis server using the command below and following the instructions on screen.

$ ./redis-toolkit monitor
Please enter the redis-cli string for the Redis server you wish to monitor: redis-cli -h ... -p ...

Once you get enough data, you can run the report. You'll have to give it a name which will be used for storing the report locally in report/app/data/NAME.hitrate.gob. This is useful in case you want to see it again at a later time or if you want to play with the various filtering options - you won't need to regenerate the report again.

$ ./redis-toolkit report -name NAME -type hitrate -level 2
+----------------------------------------+----------+--------+-------+-------------+-----------+--------------------+----------------------+
|                  KEY                   | NR  KEYS |  GET   |  SET  | HIT RATE(%) | SIZE (KB) | LIFETIME (SECONDS) | NETWORK TRAFFIC (MB) |
+----------------------------------------+----------+--------+-------+-------------+-----------+--------------------+----------------------+
| wp_userslugs:*                         |       89 |     18 |   536 |           3 |      0.12 |               2.50 |                 0.06 |
| wp_useremail:*                         |       88 |     39 |   534 |           6 |      0.12 |               3.67 |                 0.07 |
| post_format_relationships:*            |       12 |     40 |   124 |          24 |      0.11 |              42.58 |                 0.02 |
| resource_tag_relationships:*           |       12 |     13 |    28 |          31 |      0.12 |              77.67 |                 0.00 |
| timeinfo:*                             |        3 |      3 |     3 |          50 |      0.07 | n/a                |                 0.00 |
| wpseo:*                                |       12 |     24 |    22 |          52 |      0.07 |             273.25 |                 0.00 |
| wp_userlogins:*                        |       75 |    673 |   534 |          55 |      0.12 |              68.32 |                 0.13 |

When you're done, you can stop the the monitoring process via the command below.

$ ./redis-toolkit stop

Memory report

The first is to dump the Redis database locally via the command below.

$ ./redis-toolkit dump
Please enter the redis-cli string for the Redis server you wish to monitor: redis-cli -h ... -p ...

Once the dump is done. you can run the report. You'll have to give it a name which will be used for storing the report locally in report/app/data/NAME.memoy.gob. This is useful in case you want to see it again at a later time or if you want to play with the various filtering options - you won't need to regenerate the report again.

$ ./redis-toolkit report -type memory -name NAME
+----------------------------------------+----------+-----------+----------+
|                     KEY                | NR  KEYS | SIZE (MB) | SIZE (%) |
+----------------------------------------+----------+-----------+----------+
| posts:*                                |      500 |      0.56 |     2.79 |
| post_meta:*                            |      440 |     18.48 |    92.78 |
| terms:*                                |      192 |      0.12 |     0.63 |
| options:*                              |      109 |      0.52 |     2.59 |
| term_meta:*                            |       67 |      0.02 |     0.08 |
| product_cat_relationships:*            |       45 |      0.00 |     0.01 |

Options

There are a few of options that can be passed to the both report generators. They are described below.

usage: ./redis-toolkit report [-h] -name NAME -type hitrate|memory [-regenerate] [-level LEVEL] [-prefix PREFIX]

Generates a hitrate or memory report from the Redis keys.

  -level int
    	How many levels deep the report should render. (default 3)
  -name string
    	The name of this report (e.g. -name clientname). This is going to be stored locally so that future reports take less time. (Required)
  -prefix string
    	Filter by prefix.
  -regenerate
    	Regenerate the report.
  -type string
    	The type of report you wish to generate. Possible values: memory, hitrate

Implementation details

Hit Rate

The output shows the key hitrate (calculated using the following formula hitrate = (gets / (gets + sets)) * 100), the number of keys in the group, the number of GET and SET operations, the average size of each key only for SET operations and the key lifetime (calculated as the time passed since the last SET operation - be careful, the only data we have is what we capture; not all keys will be SET during this interval). The result is ordered by hitrate asscending.

Memory

The memory analysis report shows how big the keys are and how much that represents compared to the total amount of space occupied by the entire data set. Please be aware that Redis has has optimization algorithms that store data in a compressed format. Thus, the actual size in memory will be smaller.

Running it in production

redis-cli MONITOR is a debugging command that streams back every command processed by the Redis server. Running this on a production database comes with a performance cost that's hard to estimate. Use it with caution on production servers.

redis_toolkit's People

Contributors

alexdicianu avatar fusionx1 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

redis_toolkit's Issues

Not working on macosx and docker 18.09.01

All the installation steps were followed as the docs describe them.


$ ./redis-toolkit report -type memory -name report
2019/02/18 15:18:42 dial tcp: lookup redis_toolkit_db on 127.0.0.11:53: no such host

report/Dockerfile not building

Step 3/12 : RUN apt-get update -y --fix-missing
 ---> Running in 0b93f718d57f
Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [7340 B]
Get:3 http://deb.debian.org jessie Release.gpg [2420 B]
Get:4 http://deb.debian.org jessie Release [148 kB]
Get:5 http://security.debian.org jessie/updates/main amd64 Packages [848 kB]
Get:6 http://deb.debian.org jessie/main amd64 Packages [9098 kB]
Fetched 10.1 MB in 10s (961 kB/s)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update -y --fix-missing' returned a non-zero code: 100
redis_toolkit

Solution Simply change report/Dockerfile to use the golang 1.11 image instead of .8

Speed up report generation by rewriting the reporting scripts in Go

More as a long term goal, but right now there's a complexity level of O(n) = n * log(n) for building the prefix tree. Often there are lots of keys in Redis which tends to make the tree generation quite a lengthy process for over 100K keys.

Doing a quick test python vs. go for a complexity level of O(n) = n * n, it's pretty clear who the winner is.

# test.go
package main
import "fmt"
func main() {
    sum := 0
    for i := 0; i < 100000; i++ {
        for j := 0; j < 100000; j++ {
            sum += j
        }
    }
    fmt.Println("Sum is:", sum)
}

# test.py
sum = 0
for i in range(100000):
    for j in range(100000):
        sum += j
print "Sum is:", sum

==============================================

# Go
$ time ./test
Sum is: 499995000000000

real	0m3.535s
user	0m3.501s
sys	0m0.014s

# Python
$ time python test.py
Sum is: 499995000000000

real	7m56.064s
user	7m30.009s
sys	0m3.786s

Add option to save the report by name

This would allow the report to be saved and looked at, at a later time.

To regenerate the report: ./redis-monitor report --name test --regenerate
To show the results from a previously generated report: ./redis-monitor report --name test. If no local copy is found, the report will be automatically generated.

Can't handle RDB format version 8

8:M 20 Sep 13:01:07.224 # Can't handle RDB format version 8
8:M 20 Sep 13:01:07.224 # Fatal error loading the DB: Invalid argument. Exiting.

This means you need a later version of redis than the one provided by apt-get install :)

Thats 2 hours of my life wasted :)

Add lifetime column for each key

The easiest way is to add a last_set column with the timestamp of the last SET operation and a second column lifetime in which lifetime = current_timestamp - last_set

Better prefixing based on cache bins

The current algorithm based on the Levenshtein distance isn't ideal for grouping this type of strings. A better solution is grouping the strings in a tree-like format based on cache bins.

E.g.

Level 1                                 pantheon-redis
                                    .            .                    .
                                  .                .                            .
                                .                    .                                    .
Level 2                    cache                       cache_page                               cache_menu
                             .                           .                                          .
                            .                             .                                          .
Level 3      facetapi, filter_formats               http://example.com, ...               user-menu, main-menu,...
Key                                                                                        Count      GET        SET        Hit Rate (%)    Avg Size (KB)   Lifetime (seconds)  
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
pantheon-redis:cache:facetapi:facet_info:search_api@default_node_index:en                  1          2          0          100             n/a             n/a                 
pantheon-redis:cache:filter_formats:en                                                     1          330        0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_sweepstakes                                              1          1          1          50              19.79           n/a                 
pantheon-redis:cache:gr8ist-instagram:170d36760f0580a37b2a1d234e8f9a78                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:ctools_plugin_files:ctools:export_ui                                  1          1          0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_default                                                  1          72         0          100             n/a             n/a                 
pantheon-redis:cache:ctools_plugin_files:entityreference:selection                         1          67         0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_collection_page                                          1          3          0          100             n/a             n/a                 
pantheon-redis:cache:plugins:bean:types                                                    1          339        0          100             n/a             n/a                 
pantheon-redis:cache:schema:runtime:1                                                      1          405        0          100             n/a             n/a                 
pantheon-redis:cache:ctools_plugin_files:facetapi:query_types                              1          2          0          100             n/a             n/a                 
pantheon-redis:cache:search_api:index-default_node_index--fields-1-0-en                    1          2          0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_info_page                                                1          3          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:30dbcb71818417b97395dd00e9804487                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:8891f91d109252de116e32934dc33154                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:ctools_plugin_type_info                                               1          384        0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:9a0b344a835b241fa3afa92f4bbb0c0f                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:filter_list_format                                                    1          329        0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:1c7a618ce092de9190b6fefea6ee9aea                     1          2          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:70d8076d8276415e24cc4c5370d8f77a                     1          7          0          100             n/a             n/a                 
pantheon-redis:cache:system_cache_tables                                                   1          1          0          100             n/a             n/a                 
pantheon-redis:cache:theme_registry:seven                                                  1          1          0          100             n/a             n/a                 
pantheon-redis:cache:theme_registry:greatist2x                                             1          32         0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:f408ca07289333ea9c4c7b55661279e0                     1          2          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:8efed7d09526e7b1e489cc143cfe3be1                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_collection                                               1          1          0          100             n/a             n/a                 
pantheon-redis:cache:image_styles                                                          1          390        0          100             n/a             n/a                 
pantheon-redis:cache:theme_registry:runtime:seven                                          1          5          0          100             n/a             n/a                 
pantheon-redis:cache:schema:runtime:                                                       1          22         0          100             n/a             n/a                 
pantheon-redis:cache:theme_registry:bootstrap                                              1          1          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:e1bb1d1e375b377f0093ce10224394fe                     1          2          0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_taxonomy                                                 1          28         0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:0afe6060bc27e87b9efc2fdecce421a7                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:theme_registry:adminimal                                              1          1          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:344a9ab42e1b1980092c7166e64e56c7                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:gr8ist-instagram:6d5ab92f7d3db2ab9e6c1abae0132265                     1          1          0          100             n/a             n/a                 
pantheon-redis:cache:bean_plugins                                                          1          339        0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_recipe                                                   1          8          0          100             n/a             n/a                 
pantheon-redis:cache:imageapi_optimize:methods                                             1          1029       0          100             n/a             n/a                 
pantheon-redis:cache:node_types:en                                                         1          435        0          100             n/a             n/a                 
pantheon-redis:cache:entity_info:en                                                        1          441        0          100             n/a             n/a                 
pantheon-redis:cache:theme_registry:runtime:greatist2x                                     1          377        1          99              51.89           n/a                 
pantheon-redis:cache:gr8ist-instagram:4b76d5a8f56969794face6ecb8573640                     1          5          0          100             n/a             n/a                 
pantheon-redis:cache:critical_css_article                                                  1          250        0          100             n/a             n/a                 
pantheon-redis:cache:entity_property_info:en                                               1          281        0          100             n/a             n/a                 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
pantheon-redis:cache*                                                                      45         5608       2          99              35.84           n/a            

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.