graphite-project/docker-graphite-statsd

Official Docker image for Graphite

docker-graphite-statsd's Issues

GRAPHITE_STORAGE_DIR is not present inside the container

Tried to use this image on Kubernetes.

There was an issue with persistent storage caused by lack of GRAPHITE_STORAGE_DIR environment variable.

Variable GRAPHITE_STORAGE_DIR should be set to /opt/graphite/storage by default

Disable nginx

Hi,

how can I disable the docker nginx and use my own installed nginx as proxy?

Regards,

Upgrade path and tags database

Coming from 1.0.2-2 and upgrading to 1.1.2-1, I see several errors related to tags:

curl -s "http://graphite/tags?pretty=1"

{
  "error": "no such table: tags_tag"
}

curl -X POST "http://graphite/tags/tagSeries" \
   --data-urlencode 'path=disk.used;rack=a1;datacenter=dc1;server=web01'
{"error": "no such table: tags_series"}

These are the tables I see in graphite.db. Since I did no configuring on my end I'm assuming this includes the local tags db.

account_mygraph             dashboard_dashboard_owners
account_profile             dashboard_template
account_variable            dashboard_template_owners
account_view                django_admin_log
account_window              django_content_type
auth_group                  django_migrations
auth_group_permissions      django_session
auth_permission             events_event
auth_user                   tagging_tag
auth_user_groups            tagging_taggeditem
auth_user_user_permissions  url_shortener_link
dashboard_dashboard

Is there a migration guide / am I missing something?

builtins.OSError: [Errno 95] fallocate with fresh container

Got this error today when trying this container, fresh, no volumes defined, ZFS as backing storage for docker. I remember getting this same years and years ago with my own install in a corporate setting (no docker) when the docker user wasn't able to write to the storage folder.

Jan 06 15:50:54 lanfear docker[25322]: Traceback (most recent call last):
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
Jan 06 15:50:54 lanfear docker[25322]:     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
Jan 06 15:50:54 lanfear docker[25322]:     return self.currentContext().callWithContext(ctx, func, *args, **kw)
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
Jan 06 15:50:54 lanfear docker[25322]:     return func(*args,**kw)
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/carbon/writer.py", line 189, in writeForever
Jan 06 15:50:54 lanfear docker[25322]:     writeCachedDataPoints()
Jan 06 15:50:54 lanfear docker[25322]: --- <exception caught here> ---
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/carbon/writer.py", line 142, in writeCachedDataPoints
Jan 06 15:50:54 lanfear docker[25322]:     state.database.create(metric, archiveConfig, xFilesFactor, aggregationMethod)
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/carbon/database.py", line 145, in create
Jan 06 15:50:54 lanfear docker[25322]:     self.sparse_create, self.fallocate_create)
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/python3.6/site-packages/whisper.py", line 528, in create
Jan 06 15:50:54 lanfear docker[25322]:     fallocate(fh, headerSize, remaining)
Jan 06 15:50:54 lanfear docker[25322]:   File "/opt/graphite/lib/python3.6/site-packages/whisper.py", line 93, in _py_fallocate
Jan 06 15:50:54 lanfear docker[25322]:     raise IOError(res, 'fallocate')
Jan 06 15:50:54 lanfear docker[25322]: builtins.OSError: [Errno 95] fallocate
Jan 06 15:50:54 lanfear docker[25322]: 06/01/2019 06:50:54 :: [console] Error creating stats.statsd.graphiteStats.flush_time: [Errno 95] fallocate
Jan 06 15:50:54 lanfear docker[25322]: 06/01/2019 06:50:54 :: [console] Unhandled Error

Running this as

/usr/bin/docker run --name graphite -p 8080:80 -p 2003:2003 graphiteapp/docker-graphite-statsd

I did a peek inside the actual container and looks like it's running as root inside, and folders do exists inside. Bit stumped why it wouldn't be able to create the .wsp files. It can create the folders in the storage/

TIMEZONE

Hi, I'm looking for a way to set the timezone so graphite displays the data relative to the server's location. It looks like local_settings.py is not mapped to a place where it is persisted in your build. And, it would be really nice if the startup read the TZ environment variable and patched the local_settyings.py. Unless you have a better idea, of course :)

I could fix this in my own build, but it seems that it would be worthwhile for everybody using your build.

Tags in graphite-web are broken

Tags in graphite-web are broken after 1.1.4-4, because TAGDB = '' by default, and it should be 'graphite.tags.localdatabase.LocalDatabaseTagDB' or absent.

1.1.0-pre2 image is failing

Using the latest image: 1.1.0-pre2

Here's the logs:

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_conf_init.sh...
*** Running /etc/rc.local...
*** Booting runit daemon...
*** Runit started as PID 14
Nov 29 05:26:01 4d5ab037c9ef syslog-ng[36]: syslog-ng starting up; version='3.5.6'
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-cache.py", line 32, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 111, in run_twistd_plugin
    config.parseOptions(twistd_options)
  File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 624, in parseOptions
    usage.Options.parseOptions(self, options)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/usage.py", line 277, in parseOptions
    self.postOptions()
  File "/usr/local/lib/python2.7/dist-packages/twisted/scripts/_twistd_unix.py", line 77, in postOptions
    app.ServerOptions.postOptions(self)
  File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 635, in postOptions
    % (self['logger'], e))
twisted.python.usage.UsageError: Logger 'None' could not be imported: No module named 'None'
Traceback (most recent call last):
  File "/opt/graphite/bin/carbon-aggregator.py", line 32, in <module>
    run_twistd_plugin(__file__)
  File "/opt/graphite/lib/carbon/util.py", line 111, in run_twistd_plugin
    config.parseOptions(twistd_options)
  File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 624, in parseOptions
    usage.Options.parseOptions(self, options)
  File "/usr/local/lib/python2.7/dist-packages/twisted/python/usage.py", line 277, in parseOptions
    self.postOptions()
  File "/usr/local/lib/python2.7/dist-packages/twisted/scripts/_twistd_unix.py", line 77, in postOptions
    app.ServerOptions.postOptions(self)
  File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 635, in postOptions
    % (self['logger'], e))
twisted.python.usage.UsageError: Logger 'None' could not be imported: No module named 'None'
...

Logrotate not working with 1.1.5-3

Good morning,

logrotate seems not to work with the Docker Image 1.1.5-3. I even can not start the logrotate manually because I cannot find the service.

Under /etc/logrotate.d there is a logrotate file present.

Support for aarch64?

With a number of images (node, nginx, etc) now being runnable on multiple architectures can we add support for ARM (specifically aarch64) in this image?

Issues when using mounted volumes.

The instructions mention that you can mount /opt/graphite/storage and /opt/graphite/conf to keep data and config outside the container. They also mention that it is ok to mount empty directories as they will be correctly populated.

On a first attempt this did not work.

I started the container like this:

docker run -d --name graphite --restart=always -p 8080:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 -v /opt/graphite/storage:/opt/graphite/storage -v /opt/graphite/conf:/opt/graphite/conf  graphiteapp/graphite-statsd

I expected to see /opt/graphite/conf populated, but the directory was empty.

I connected a shell to the running container and notice there that from within the container it was impossible to write to the mounted volumes. The culprit was selinux.

The solution was to start the container like this:

docker run -d --name graphite --restart=always -p 8080:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 -v /opt/graphite/storage:/opt/graphite/storage:Z -v /opt/graphite/conf:/opt/graphite/conf:Z  graphiteapp/graphite-statsd

Adding the "Z" option to the mount assures correct interaction with selinux, and now the processes in the container could write to the conf and storage dirs.

It might be useful to mention this in the docs.

But this was not sufficient to have everything working. Carbon did collect metrics, and files started to apprear in /opt/graphite/storage/whisper, but in the graphite browser the metrics remained empty.

On a hunch I just stopped and started the container again. And the metrics now appeared.

Could it be that when starting with an empty storage dir (no whisper subdir) the graphite browser doesn't scan this directory?

Gunicorn Startup command

Is there a way to change the startup of gunicorn? Such that I can change the number of workers or add different settings. Is there a script within the container that I am not aware of?

This is question is probably more indicative of my lack of knowledge more than anything.

Example shows weird graph

Hi, this is for information rather than an issue, you can close this if it's irrelevant;

I'm very new to all theses system, i followed your example (start docker and generate example data) and i ended up with this graph :

I modified the nc command to show what it was sending and it looks nothing like it it was juste random numbers with no particular spike.

Here's an exert of the graph's json

[{"datapoints": [[45.1, 1538162030], [0.0, 1538162040], [49.1, 1538162050], [47.7, 1538162060], [0.0, 1538162070], [55.0, 1538162080], [57.0, 1538162090], [0.0, 1538162100], [42.3, 1538162110], [44.4, 1538162120], [0.0, 1538162130], [53.1, 1538162140], [57.8, 1538162150], [0.0, 1538162160], [45.1, 1538162170], [32.5, 1538162180], [0.0, 1538162190], [50.9, 1538162200], [60.5, 1538162210], [0.0, 1538162220], [39.0, 1538162230], [50.9, 1538162240], [0.0, 1538162250], [53.3, 1538162260], [53.0, 1538162270], [0.0, 1538162280], [40.3, 1538162290], [52.1, 1538162300], [0.0, 1538162310], [null, 1538162320]], "target": "stats.example", "tags": {"name": "stats.example"}}]

there are clear 0s in there...

Is this normal? am i not understanding something properly? do you have links to understand what is happening?

TODO: Migrate to Python 3 and Bionic

I did that in separate branch, but Etsy statsd is dead on bionic.
Probably we should migrate to https://github.com/bitly/statsdaemon

Reduce Docker Image size from 864.8 MB to 465.3 MB

I have done a little experimentation and it appears it is possible to reduce the size of the Docker image by 50%.

The size reduction is mostly achieved by,

• Removing git and *-dev libraries from the first RUN(layer). Plus clearing the apt cache before the layer closes.
• Installing git, gcc , *-dev libraries, whisper, carbon, graphite in one RUN(layer). Plus purging git, gcc , *-dev libraries and the whisper, carbon, graphite sources from the layer post install, before the layer closes.
• Pip installing without caching.

The key thing to note with Docker is that once a RUN is complete nothing can be deleted from the resulting layer. Clean-up/deletion needs to happen in the RUN(layer) to be effective.

The Dockerfile below is less user friendly to read than the current published one. With the majority of the install work being done in one RUN. The size reduction might warrant that loss of readability.

I have only tested the resulting container briefly. I did nothing more than run the example events generator bash snippet and observe stats on a chart in graphite-web.

Might Docker Image size reduction following something similar to Dockerfile changes below be of interest to the project?

FROM phusion/baseimage:0.10.2
MAINTAINER XXXX YYYY <[email protected]>

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update --fix-missing && \
    apt-get install --yes --no-install-recommends \
    nginx \
    python-flup \
    python-pip \
    python-ldap \
    expect \
    memcached \
    sqlite3 \
    libcairo2 \
    python-cairo \
    python-rrdtool \
    nodejs && \
    apt-get clean && \
    apt-get autoremove --yes  && \
    rm -rf /var/lib/apt/lists/*

# choose a timezone at build-time
# use `--build-arg CONTAINER_TIMEZONE=Europe/Brussels` in `docker build`
ARG CONTAINER_TIMEZONE

RUN if [ ! -z "${CONTAINER_TIMEZONE}" ]; \
    then ln -sf /usr/share/zoneinfo/$CONTAINER_TIMEZONE /etc/localtime && \
    dpkg-reconfigure -f noninteractive tzdata; \
    fi

ARG version=1.1.4
ARG whisper_version=${version}
ARG carbon_version=${version}
ARG graphite_version=${version}

ARG whisper_repo=https://github.com/graphite-project/whisper.git
ARG carbon_repo=https://github.com/graphite-project/carbon.git
ARG graphite_repo=https://github.com/graphite-project/graphite-web.git

ARG statsd_version=v0.8.0
ARG statsd_repo=https://github.com/etsy/statsd.git

# Install compiler and dev libraries.
# Clone sources from statsd, whisper, carbon and graphite github repositories.
# Build and install whisper, carbon and graphite.
# Remove compiler and dev libraries.
# Delete source code directories.
# Clear apt cache.
RUN apt-get update --fix-missing && \
    apt-get install --yes --no-install-recommends gcc python-dev libffi-dev libcairo2-dev pkg-config git && \

    python -m pip install --no-cache-dir --upgrade pip setuptools && \
    pip install --no-cache-dir django==1.11.15 fadvise msgpack-python && \

    git clone -b ${statsd_version} ${statsd_repo} /opt/statsd && \
    git clone -b ${whisper_version} --depth 1 ${whisper_repo} /usr/local/src/whisper && \
    git clone -b ${carbon_version} --depth 1 ${carbon_repo} /usr/local/src/carbon && \
    git clone -b ${graphite_version} --depth 1 ${graphite_repo} /usr/local/src/graphite-web && \

    cd /usr/local/src/whisper && \
    python ./setup.py install && \

    cd /usr/local/src/carbon && \
    pip install --no-cache-dir --requirement requirements.txt && \
    python ./setup.py install && \

    cd /usr/local/src/graphite-web && \
    pip install --no-cache-dir --requirement requirements.txt && \
    python ./setup.py install && \

    apt-get purge --yes --auto-remove cpp gcc python-dev libffi-dev libcairo2-dev pkg-config git && \
    apt-get install --yes --no-install-recommends libpython2.7 && \
    apt-get clean && \
    apt-get autoremove --yes  && \
    rm -rf /var/lib/apt/lists/* && \

    cd ${HOME} && \
    rm -rf /usr/local/src/whisper /usr/local/src/carbon /usr/local/src/graphite-web /tmp/* /var/tmp/* /usr/share/python-wheels/*

# config graphite
ADD conf/opt/graphite/conf/*.conf /opt/graphite/conf/
ADD conf/opt/graphite/webapp/graphite/local_settings.py /opt/graphite/webapp/graphite/local_settings.py
# ADD conf/opt/graphite/webapp/graphite/app_settings.py /opt/graphite/webapp/graphite/app_settings.py
WORKDIR /opt/graphite/webapp
RUN mkdir -p /var/log/graphite && \
    PYTHONPATH=/opt/graphite/webapp django-admin.py collectstatic --noinput --settings=graphite.settings

# config statsd
ADD conf/opt/statsd/config_*.js /opt/statsd/

# config nginx
RUN rm /etc/nginx/sites-enabled/default
ADD conf/etc/nginx/nginx.conf /etc/nginx/nginx.conf
ADD conf/etc/nginx/sites-enabled/graphite-statsd.conf /etc/nginx/sites-enabled/graphite-statsd.conf

# init django admin
ADD conf/usr/local/bin/django_admin_init.exp /usr/local/bin/django_admin_init.exp
ADD conf/usr/local/bin/manage.sh /usr/local/bin/manage.sh
RUN chmod +x /usr/local/bin/manage.sh && /usr/local/bin/django_admin_init.exp

# logging support
RUN mkdir -p /var/log/carbon /var/log/graphite /var/log/nginx
ADD conf/etc/logrotate.d/graphite-statsd /etc/logrotate.d/graphite-statsd

# daemons
ADD conf/etc/service/carbon/run /etc/service/carbon/run
ADD conf/etc/service/carbon-aggregator/run /etc/service/carbon-aggregator/run
ADD conf/etc/service/graphite/run /etc/service/graphite/run
ADD conf/etc/service/statsd/run /etc/service/statsd/run
ADD conf/etc/service/nginx/run /etc/service/nginx/run

# default conf setup
ADD conf /etc/graphite-statsd/conf
ADD conf/etc/my_init.d/01_conf_init.sh /etc/my_init.d/01_conf_init.sh

# defaults
EXPOSE 80 2003-2004 2023-2024 8080 8125 8125/udp 8126
VOLUME ["/opt/graphite/conf", "/opt/graphite/storage", "/opt/graphite/webapp/graphite/functions/custom", "/etc/nginx", "/opt/statsd", "/etc/logrotate.d", "/var/log"]
WORKDIR /
ENV HOME /root
ENV STATSD_INTERFACE udp

CMD ["/sbin/my_init"]

wrong next page redirect after Django login on alternate port

I mapped the web port of the container to 8080 on the host (-p 8080:80), which breaks login.

When trying to login from the homepage

1. Login link points correctly to http://host:8080/account/login
2. POSTing the login data wrongly 302-redirects to http://host/
3. When browsing http://host:8080/ now, I am logged in.
4. Logout redirects again to http://host/ which is wrong.

This might bite people who want to Proxy via https, too. (Assuming host:80 --redirect--> host:443 --proxy--> container:80)

Setup CI/CD

To build latest release and upload to Docker Hub.

Document Nginx User Change

The release notes for https://github.com/graphite-project/docker-graphite-statsd/releases/tag/1.1.5-2 should also note that due to the switch to Alpine the user that the Nginx process runs as has changed.

For anyone upgrading from a previous version and using their previous nginx.conf file, this will cause Nginx to not run and cause problems. The code change is here 8d74450#diff-91e8cbe5e36ea5d88807739f99b79f71.

docker run fails when mounting empty volume to /opt/statsd

Currently the init script conf/etc/my_init.d/01_conf_init.sh performs this action:

if [[ -z $statsd_dir_contents ]]; then
  git clone -b v0.7.2 https://github.com/etsy/statsd.git /opt/statsd
  cp $conf_dir/opt/statsd/config.js /opt/statsd/config.js
fi

However, there is no such file in $conf_dir/opt/statsd/, there are two - config_tcp.js and config_udp.js.

I guess the solution would be to only copy one of them (I don't know statsd well enough, but I assume it requires exactly one config file).

Workaround is not mounting the volume there, but this is a bug.

Update local_settings.py for Tag Database configuration

Could you please add the section 'TAGDB' with comments to the Graphite webapp sample configuration
to make it more clear how to configure it and how it works for the new users?

carbon-relay is running when RELAY is not defined

/etc/service/carbon-relay/run runs unconditionally carbon-relay service.
There is a typo in file. The test has no effect
[[ -n ${RELAY} ]] || exit 1
it should be
[[ -n "${RELAY}" ]] || exit 1

How to establish graphite-statsd is ready?

Found that running this image, that it can take up to 2 minutes before dummy stats sent into the running container before querying for the metric returned usable results.

Run the image with:

docker run --rm -d \
    --name graphite-statsd \
    -p 8080:80 \
    -p 2003-2004:2003-2004 \
    -p 2023-2024:2023-2024 \
    -p 8125:8125/udp \
    -p 127.0.0.1:8126:8126 \
    graphiteapp/graphite-statsd:latest

This unfortunately makes it very difficult to determine when the services in the container are ready for use.

Some enquiries in #graphite on Freenode IRC suggested that it takes some time for a whisper file to be created as once about 90 seconds has past after the initial stat was submitted, any subsequent entries to the same stats are picked up almost immediately.

Is this the source of the delay? Is there anything here someone using the image could do?

If there is nothing to be done, seems like the delay from the initial stat being emitted and showing up in graphite should be mentioned in the documentation around verifying it is working to avoid any confusion.

Separate docker images

Sometimes using monolithic docker image for all components is not convenient. We need to create separate images for carbon and graphite-web.

Statsd repeater failed to start due to generic-pool lack

Given:

• Offical graphite docker repo
• Statsd-exporter in the same host + docker network
• My statsd_config:
  { "graphiteHost": "127.0.0.1", "graphitePort": 2003, "port": 8125, "flushInterval": 10000, "backends": [ "./backends/graphite", "./backends/repeater" ], "servers": [ { server: "./servers/udp", address: "0.0.0.0", port: 8125 } ], "repeater": [ { host: "127.0.0.1", port: 9125 }] }

When:

1. Docker run graphite docker run \ -d \ --name graphite \ --restart=always \ --network=graphite \ -p 80:80 \ -p 2003-2004:2003-2004 \ -p 2023-2024:2023-2024 \ -p 8125:8125/udp \ -p 8126:8126 \ -v /etc/graphite/conf:/opt/graphite/conf \ -v /data/graphite:/opt/graphite/storage \ -v /etc/statsd/config/config_udp.js:/opt/statsd/config_udp.js \ -v /var/log/graphite:/var/log \ graphiteapp/graphite-statsd

Then:

1. statsd.log - error in module.js
   missing "generic-pool" in repeater.js

Dashboard page shows error with graphiteapp/docker-graphite-statsd:latest image

Repro steps:

• Run docker run -d --name graphite --restart=always -p 80:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 graphiteapp/docker-graphite-statsd
• Open browser and launch http://localhost

Expected:

Actual:

Can't see example in dashboard

echo counters | nc localhost 8126 shows the random data is being sent and I can see the example.wsp file, but the dashboard is empty and there's no metrics in the main page. What am I missing?

Step 7/53 - Pip install & django install

Sending build context to Docker daemon  349.2kB
Step 1/53 : FROM phusion/baseimage:0.10.1
 ---> 2391dfad8777
Step 2/53 : MAINTAINER Denys Zhdanov <[email protected]>
 ---> Using cache
 ---> 3152d73b6dce
Step 3/53 : RUN apt-get -y update   && apt-get -y upgrade   && apt-get -y install vim   nginx   python-dev   python-flup   python-pip   python-ldap   expect   git   memcached   sqlite3   libffi-dev   libcairo2   libcairo2-dev   python-cairo   python-rrdtool   pkg-config   nodejs   && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 868784acd4e8
Step 4/53 : ARG CONTAINER_TIMEZONE
 ---> Using cache
 ---> 47bd1063221a
Step 5/53 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> bf45329bfea3
Step 6/53 : RUN if [ ! -z "${CONTAINER_TIMEZONE}" ];     then ln -sf /usr/share/zoneinfo/$CONTAINER_TIMEZONE /etc/localtime &&     dpkg-reconfigure -f noninteractive tzdata;     fi
 ---> Using cache
 ---> b10718a0ccc0
Step 7/53 : RUN pip install --upgrade pip &&   pip install django==1.11.12
 ---> Running in fc3fc2bc70a2
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-10.0.0
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
The command '/bin/sh -c pip install --upgrade pip &&   pip install django==1.11.12' returned a non-zero code: 1

This is the raw docker file returning this error.

It seems i django database has lost

Dear,
I have been following the readme to startup a instance, but got error when i access localhost: :

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/composer/views.py", line 35, in composer
profile = getProfile(request)
File "/opt/graphite/webapp/graphite/user_util.py", line 25, in getProfile
return default_profile()
File "/opt/graphite/webapp/graphite/user_util.py", line 41, in default_profile
'password': '!'})
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 405, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in len
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 965, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 318, in execute
return Database.Cursor.execute(self, query, params)
OperationalError: no such table: auth_user

My configuration is:

statsd:
container_name: graphite
image: graphiteapp/graphite-statsd
ports:
- "8088:80"
- "2003-2004:2003-2004"
- "2023-2024:2023-2024"
- "8125:8125/udp"
- "8126:8126"
volumes:
- "/Users/kevin/bigdata/graphite/conf:/opt/graphite/conf"
- "/Users/kevin/bigdata/graphite/storage:/opt/graphite/storage"
- "/Users/kevin/bigdata/graphite/log:/var/log"
- "/Users/kevin/bigdata/graphite/log:/var/graphite-log"
- "/Users/kevin/bigdata/graphite/log:/opt/statsd"
- "/Users/kevin/bigdata/graphite/logrotate.d:/etc/logrotate.d"

And i found "graphite.db" file under graphite/storage is zerobyte, seems the database initialized failure.

TODO: Include fadvise and msgpack

both are quite useful

Modifying graphite web's local_settings.py

Do you have any suggestions on the best way to make modifications to settings in graphite-web's local_settings.py? I thought about mapping the volume /opt/graphite/webapp/graphite/ but looking in the container there is a lot more in it than what is in github, so not sure how to get those locally first.

Any reason this directory isn't treated the same way as /opt/graphite/conf, where I can map to an empty volume and get the config files on the host that way?

We're just trying to change the FETCH_TIMEOUT value to something a bit higher because we have a few queries which return a lot of metrics.

Add an Alpine Linux variant

Alpine-based images are typically much smaller than Debian-based ones.

Make cross-origin URLs for django configurable

By default the nginx configuration shipped in the docker container returns "*" for theAccess-Control-Allowed-Origin header. This is a problem when using using include credentials as part of AJAX request since a specific host must be returned. The net effect is that authenticated cross-origin requests from grafana to graphite fail.

It looks like graphite already exposes this via ALLOWED_HOSTS (https://graphite.readthedocs.io/en/latest/config-local-settings.html#general-settings) but this isn't easy to access without doing a lot of work on the container itself.

I was able to work around this since I was using another nginx instance as we're running grafana/graphite on the same box, but I had to install nginx-extras and use more_set_headers to override the headers being returned to the correct value, but just based off what I saw from doing Google searches a lot of folks people will be prone to just echoing back the origin url and giving themselves XSS vulnerabilities .

If there is an easy way to do this would love a pointer to the docs :)

Does not work with empty /opt/graphite/conf

As per the README.md, "The container will initialize properly if you mount empty volumes at /opt/graphite/conf, /opt/graphite/storage, or /opt/statsd/config."

However, with empty /opt/graphite/conf directory, carbon continually prints this message to the logs:
Error: missing required config '/opt/graphite/conf/carbon.conf'

Graphite web starts, but has no data. No files are created in /opt/graphite/conf volume/directory

Current version of hopsoft/graphite-statsd does work with empty /opt/graphite/conf directory, and populates it with the following files:
aggregation-rules.conf
blacklist.conf
carbon.amqp.conf
carbon.conf
dashboard.conf
graphTemplates.conf
relay-rules.conf
rewrite-rules.conf
storage-aggregation.conf
storage-schemas.conf
whitelist.conf

how to migrate from hopsoft/graphite-statsd to graphiteapp/graphite-statsd

Hi,

I'm currently using docker-compose managing a grafana and hopsoft/graphite-statsd deployment, where a host volume is attached to the hopsoft/graphite-statsd container's /opt/graphite/storage. Everything is working fine, except hopsoft/graphite-statsd is still at 0.91 that a few functions I need is missing. So, when i'm trying to migrate my hopsoft/graphite-statsd to graphiteapp/graphite-statsd:1.0.2-3. I thought it's just simply replacing the image name. Looks like i'm totally wrong. I noticed that graphiteapp/graphite-statsd:1.0.2-3 can't initialize itself with the same host volume. Anyone can give me some advice how to do this migration smoothly? I would like to preserve all the collected data that currently stored on my local volume by hopsoft/graphite-statsd, and I would like Grafana to function normally after migration. Anyway to achieve this?

My docker-compose snippet is as follow. Thanks a lot!

my_graphite:
image: graphiteapp/graphite-statsd:1.0.2-3 #hopsoft/graphite-statsd #graphiteapp/graphite-statsd:1.0.2-3
volumes:
- my_data/graphite/storage:/opt/graphite/storage

Carbon-Relay Daemon Missing

I wonder why there is no carbon-relay daemon included?

For some tests I wanted to use docker for a simple graphite cluster and thought it would work out of the box if I modified the specific configuration files - but nope.

It would be neat if at least a docker environment variable could be set to start the carbon relay daemon.

carbon stats

Using this to setup graphite and trying to see carbon stats but nothing showing up.
Metrics I send via statsd show up.
metric is simple:
carbon.agents.*.metricsReceived

See image. Any idea on what I need to do

Docker run failing on the latest build

Error Log

prod_statsd.1@instance-1 | E: Unable to locate package git
prod_statsd.1@instance-1 | *** /etc/my_init.d/01_conf_init.sh failed with status 100
prod_statsd.1@instance-1 | *** Killing all processes...

add a new func in movingwindow

Hi,

I'm trying to add a p90 "feature" under graphite-web/graphite/render/functions.py movingWindow function, and then make it a docker image (in the Dockerfile by only changing the graphite git-repo to point to my fork, where i only added following lamda p90 funciton below).

aggFuncs = {
'average': safeAvg,
'median': safeMedian,
'sum': safeSum,
'min': safeMin,
'max': safeMax,
'diff': safeDiff,
'stddev': safeStdDev,
'count': safeLen,
'range': lambda row: safeSubtract(safeMax(row), safeMin(row)),
'multiply': lambda row: safeMul(*row),
'p90': lambda row: _getPercentile(row, 90), <<<<<<------------ my added line
'last': safeLast,
}

However, trying to call this function from Grafana gives me following error:

File "/opt/graphite/webapp/graphite/render/functions.py", line 1050, in movingWindow
raise Exception('Unsupported window function: %s' % (func))
Exception: Unsupported window function: p90

I then checked the mentioned functions.py in the docker but didn't see my p90 function.

---

docker exec d8a5149f4c8b cat /opt/graphite/webapp/graphite/render/functions.py > func_test.txt

So, am i doing this right? Or, I"m being too naive here? What's the right way to add a p90 function?

Thanks!

Init:CrashLoopBackOff

NAME                        READY     STATUS                  RESTARTS   AGE
graphite-649647f8d6-pn49t   0/1       Init:CrashLoopBackOff   6          7m

Jan 07 07:51:20 ip-172-20-60-166 kubelet[1249]: I0107 07:51:20.740085    1249 kuberuntime_manager.go:757] checking backoff for container "create-syncdb" in pod "graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"
Jan 07 07:51:20 ip-172-20-60-166 kubelet[1249]: I0107 07:51:20.740166    1249 kuberuntime_manager.go:767] Back-off 5m0s restarting failed container=create-syncdb pod=graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)

Jan 07 07:51:20 ip-172-20-60-166 kubelet[1249]: E0107 07:51:20.740190    1249 pod_workers.go:186] Error syncing pod 63be982e-124f-11e9-9860-02b6de547408 ("graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"), skipping: failed to "StartContainer" for "create-syncdb" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=create-syncdb pod=graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"
Jan 07 07:51:33 ip-172-20-60-166 kubelet[1249]: I0107 07:51:33.705362    1249 kuberuntime_manager.go:757] checking backoff for container "create-syncdb" in pod "graphite-649647f8d6-pn49t_graphite(63be982e-124f-11e9-9860-02b6de547408)"

I am trying to install with helm and not able to do the deployment. When I dig it saw the error for sync_db not able to create.

Getting an error when running on kubernetes with persistent volume (OperationalError: no such table: auth_user)

Hi,

*** I think I found the problem, the persistent volume is created with a lost+found dir, this cause:
this variable:
graphite_storage_dir_contents=$(find /opt/graphite/storage -mindepth 1 -print -quit)
in 01_conf_init.sh to not be empty, because of this django_admin_init.exp is not invoked:

if [[ -z $graphite_storage_dir_contents ]]; then
  /usr/local/bin/django_admin_init.exp
fi

---

original:
I'm trying to run the docker image on kubernetes when I try to enter the portal I get this error:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/composer/views.py", line 35, in composer
profile = getProfile(request)
File "/opt/graphite/webapp/graphite/user_util.py", line 25, in getProfile
return default_profile()
File "/opt/graphite/webapp/graphite/user_util.py", line 41, in default_profile
'password': '!'})
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 464, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 374, in get
num = len(clone)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 232, in len
self._fetch_all()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 900, in execute_sql
raise original_exception
OperationalError: no such table: auth_user

this is the configuration I'm using:

 ---
apiVersion: v1
kind: Service
metadata:
  name: graphite-svc
  labels:
    app: graphite
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: graphite
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: graphite-persistent-storage
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast
  resources:
    requests:
      storage: 300Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: graphite-deploy
  labels:
    app: graphite
spec:
  replicas: 1
  selector:
    matchLabels:
      app: graphite
  template:
    metadata:
      labels:
        app: graphite
    spec:
      containers:
      - name: graphite
        image: graphiteapp/graphite-statsd
        ports:
         - containerPort: 80
         - containerPort: 2003
           protocol: UDP
        volumeMounts:
          - name: graphite-data
            mountPath: /opt/graphite/storage
        env:
          - name: GRAPHITE_STORAGE_DIR
            value: /opt/graphite/storage
      volumes:
      - name: graphite-data
        persistentVolumeClaim:
          claimName: graphite-persistent-storage 

when running without persistent volume the image works as expected. I don't really know where to start looking. the only other different thing I see is that there is a lost+found dir in /opt/graphite/storage

Thanks.

RRD support is broken

Since graphite runs in /opt/graphite virtualenv, it is necessary to install rrdtool pyhton package in there. In the current state (109aba0) , the situation is the same as not having any rrd tool installed and Graphite ends up ignoring rrd files.

These lines in the dockerfile would suffice as far as I checked:

RUN apt-get update && apt-get install -y librrd-dev
RUN . /opt/graphite/bin/activate && pip install rrdtool

Missing /etc/nginx/mime.types file?

docker-graphite-statsd/conf/etc/nginx/nginx.conf

Line 27 in f0a22ba

include /etc/nginx/mime.types;

An example is available at Full Example Configuration | NGINX

Logrotate not working with 1.1.5-4

Good afternoon,

Logrotate seems still not to work even after the changes in 1.1.5-4.

Wrote a detailed report under the old post:
#81 (comment)

See issue #81

Greetings,
Stephan

RRD support still broken

This is a follow up of #61
After that one was fixed, rdd´s showed as folders but without any content.

Trying to open an rrd folder in the browser throws this in /var/log/graphite/exception.log:

Traceback (most recent call last):
  File "/opt/graphite/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/metrics/views.py", line 128, in find_view
    leaves_only=leaves_only,
  File "/opt/graphite/webapp/graphite/storage.py", line 264, in find
    for match in self._find(query):
  File "/opt/graphite/webapp/graphite/storage.py", line 295, in _find
    for node in result or []:
  File "/opt/graphite/webapp/graphite/finders/standard.py", line 101, in find_nodes
    for datasource_name in RRDReader.get_datasources(absolute_path):
  File "/opt/graphite/webapp/graphite/readers/rrd.py", line 61, in get_datasources
    #info = rrdtool.info(RRDReader._convert_fs_path(fs_path))

As far as I dug. it seems to be a bug in Graphite + Python3.
I created this workaround in the meantime by patching the image:

FROM graphiteapp/graphite-statsd
# At the moment, this does not work without this patch
RUN sed \
-i \
's/return os.path.realpath(fs_path)/return os.path.realpath(fs_path).decode("utf-8")/' \
/opt/graphite/webapp/graphite/readers/rrd.py

That allowed me to read the rrd files.

'RUN pip install --upgrade pip' causes ImportError: cannot import name "main"

In our build script we clone this repo and run the docker file on Ubuntu 16.04.
The command 'bin/sh -c pip install --upgrade pip returns a non-zero code:1

Should the way pip is called be modified??

Is there another way around this error other than the following steps??
    Manually clone repo
    Update DockerFile to hardcode older pip version
        RUN pip install --upgrade pip==9.0.3

Carbon crashing due to unhandled error

Scenario

I'm running this https://github.com/graphite-project/docker-graphite-statsd docker image. And sometimes I get this error, which causes carbon to crash. Could the reason be, that I take a backup of the whisper folder while running (maybe corrupting the data?)

EDIT: looks like a cache.py related issue?

Error log

01/12/2018 18:49:20 :: [console] Unhandled Error
Traceback (most recent call last):
 File “/opt/graphite/lib/python3.6/site-packages/twisted/python/threadpool.py”, line 250, in inContext
   result = inContext.theWork()
 File “/opt/graphite/lib/python3.6/site-packages/twisted/python/threadpool.py”, line 266, in <lambda>
   inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
 File “/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py”, line 122, in callWithContext
   return self.currentContext().callWithContext(ctx, func, *args, **kw)
 File “/opt/graphite/lib/python3.6/site-packages/twisted/python/context.py”, line 85, in callWithContext
   return func(*args,**kw)
--- <exception caught here> ---
 File “/opt/graphite/lib/carbon/writer.py”, line 189, in writeForever
   writeCachedDataPoints()
 File “/opt/graphite/lib/carbon/writer.py”, line 98, in writeCachedDataPoints
   (metric, datapoints) = cache.drain_metric()
 File “/opt/graphite/lib/carbon/cache.py”, line 187, in drain_metric
   metric = self.strategy.choose_item()
 File “/opt/graphite/lib/carbon/cache.py”, line 116, in choose_item
   return next(self.queue)
builtins.StopIteration:

Example stats script doesn't work on Mac OS X

I found that

while true; do echo -n "example:$((RANDOM % 100))|c" | nc -w 1 -u 127.0.0.1 8125; done

script from README doesn't work on my Mac OS laptop.

The problem is that nc doesn't respect -w option and hang up forever.

I was able to generate example stats using socat instead:

while true
do
  echo -n "example:$((RANDOM % 100))|c" | socat -t 0 - UDP:192.168.99.100:8125
  sleep 1
done

how to open cors in this docker container?

how to open cors in this docker container?

docker-compose for statsd

Add docker-compose file for starting dockerized version of statsd. Can somebody please take a look ?

#14

Dockerhub does not list this image as official

Hi all,
When I was searching for official release of graphite image I followed the dockerhub, entered graphite and then chose 'official' in the drop down menu there on dockerhub, and no image was shown if I did that.
It took some time to find and insure that graphiteapp/graphite-statsd image is official one.
I would recommend you to label that image as official so people can choose this image if they want an official image.
Thank you for having official image anyway.

[django admin] Can't access the default account

following the doc here, i setup graphite in docker with the latest tag

I tried to acces Django with the default root/root account on the mentioned linked (http://localhost/account/login). However it fails, with the following message:

Authentication attempt failed, please make sure you entered your login and password correctly

I found a few similar thread on launchpads but none of them were addressed. manage.py doesn't seem to exit on the container.

Is the doc wrong ? should i create a user myself ?