Giter Site home page Giter Site logo

wekan-cleanup's Introduction

wekan-cleanup

Tools to clean up Mongodb database and delete archived cards / list / boards (after few months)

Using Python3.

Local Installation

Need to have below Python module : pymongo

pip install pymongo

You need to have authentification setup on Mongo database and port opened (if docker) and create user :

use admin
db.createUser({user: "admin",pwd: "admin123",roles: [ "readWrite"]})

WekanCleanUp will manage below actions :

  • Set in lowecase email of users in database (alert if duplicated email)
  • Delete archived boards older that day_to_keep_board_arch
  • Delete live boards (with no cards into) older than day_to_keep_board_nocard
  • Removing from boards deleting users (if user deleted from database...)
  • Delete archived lists older that day_to_keep_list_arch
  • Delete archived cards older that day_to_keep_card_arch
  • Delete all the orphans objects at the end (lists,cards,checklists,cards_comments,attachements,files,chunks,activities)

You need to update parameters before executing it

# Parameters
day_to_keep_board_arch = 30
day_to_keep_list_arch = 30
day_to_keep_card_arch = 60
day_to_keep_board_nocard = 15
mongo_user = 'admin'
mongo_password = 'admin123'
mongo_server = 'localhost'
mongo_port = '27017'

Docker Installation

  1. Build Docker Image

Run the following command in the source root directory to build the docker image:

    docker build -t wekan-cleanup .

With DB authentication

  1. Create MongoDB User

You still need a MongoDB user with access to the wekan database. If MongoDB is running in a docker container you could enter the container via docker (e.g. docker exec -it wekan-db mongo) and create the user:

    use admin
    db.createUser({user: "admin",pwd: "admin123",roles: [ "readWrite"]})
  1. Save the password as a file

Save the password as the content of a file on your computer, for example:

    echo "admin123" > .MONGO_PASSWORD
  1. Run the docker container with your custom settings

Before you execute this command, please make sure that you have a backup of your MongoDB.

docker run \
    --rm \
    -v "$PWD/.MONGO_PASSWORD:/.MONGO_PASSWORD" \
    --env "MONGO_PASSWORD_PATH=/.MONGO_PASSWORD" \
    --env "MONGO_USER=admin" \
    --env "MONGO_SERVER=wekandb" \
    --env "MONGO_PORT=27017" \
    --env "MONGO_DATABASE=wekan" \
    --env "DAYS_TO_KEEP_BOARD_ARCHIVE=30" \
    --env "DAYS_TO_KEEP_LIST_ARCHIVE=30" \
    --env "DAYS_TO_KEEP_CARD_ARCHIVE=60" \
    --env "DAYS_TO_KEEP_CARD_NOCARD=15" \
    --network "wekan-mongodb_wekan-tier" \
    wekan-cleanup

Without DB authentication

NOT recommended in production environments

  1. Run the docker container with your custom settings

Before you execute this command, please make sure that you have a backup of your MongoDB.

docker run \
    --rm \
    --env "MONGO_USER_AUTHENTICATION=false" \
    --env "MONGO_SERVER=wekandb" \
    --env "MONGO_PORT=27017" \
    --env "MONGO_DATABASE=wekan" \
    --env "DAYS_TO_KEEP_BOARD_ARCHIVE=30" \
    --env "DAYS_TO_KEEP_LIST_ARCHIVE=30" \
    --env "DAYS_TO_KEEP_CARD_ARCHIVE=60" \
    --env "DAYS_TO_KEEP_CARD_NOCARD=15" \
    --network "wekan-mongodb_wekan-tier" \
    wekan-cleanup

Make sure the MongoDB is accessible If the MongoDB is running in a docker container, you could either use its network or expose the MongoDB port, the former is recommended. The example above uses the default network from the Wekan's docker-compose setup.

Other useful parameters

  • MONGO_DIRECT_CONNECTION: Don't try to connect to all replica nodes. Useful when you enable MongoDB replicaset to take advantage from Oplog, but the replica has been configured with an unreachable address (for example, localhost:27017). See pymongo docs

wekan-cleanup's People

Contributors

andreasgrill avatar buttermilchmensch avatar fmonthel avatar m1d1 avatar maulal avatar mgiacomoli avatar niklasdahlheimer avatar tleguern avatar xet7 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wekan-cleanup's Issues

cleanup.py deletes comments from cards

Do not use!
Since Wekan v2.90 cleanup.py deletes comments from cards, identifying them as orphaned activity.
[ACTIVITY] Deleting orphan activity xyz (listId not more in database)....

Username and Password

Hi there i try to use the cleanup script but i cannot handle it. I'm using WEKAN in snap on Ubuntu 18.04.4 LTS. I don't know the mongodb user or password to add this information to the script. I tried the admin user credential of WEKAN but without any positiv result! How to find the user credentials for mongodb in SNAP ?

Best Regards

Cleanup failes on removing deleted users from board

Wekan Version: 7.30
Installation method: Snap
Script run method: local with python 3.8.10

When I try to run the script, everthing seams fine, but the script failes on removing deleted users from board with following error message:
grafik

Cleanup attachments

From @fmonthel

Wekan-cleanup fix to delete (at least into cfs.attachments.filerecord) upload data.
Not sure how to proceed with other cfs.* collection, I need to look deeper
wekan-cleanup fix as well to delete checklists

Script is deleting as well board(s) with no cards into and created more than x days ago

Docker run command on README.md

Please fix docker run command top reflect MONGO_PASSWORD_PATH as seen in cleanup.py script:

docker run \
    --rm \
    -v "$PWD/.MONGO_PASSWORD:/.MONGO_PASSWORD" \
    --env "MONGO_PASSWORD_PATH=/.MONGO_PASSWORD" \
    --env "MONGO_USER=admin" \
    --env "MONGO_SERVER=wekandb" \
    --env "MONGO_PORT=27017" \
    --env "MONGO_DATABASE=wekan" \
    --env "DAYS_TO_KEEP_BOARD_ARCHIVE=30" \
    --env "DAYS_TO_KEEP_LIST_ARCHIVE=30" \
    --env "DAYS_TO_KEEP_CARD_ARCHIVE=60" \
    --env "DAYS_TO_KEEP_CARD_NOCARD=15" \
    --network "wekan-mongodb_wekan-tier" \
    wekan-cleanup

Script not working

Impacted version: v0.72
Server Setup Information: Local VM

  • Operating System: Debian 9
  • Deployment Method(tar/sandstorm/snap/docker/etc /http frontend (nginx, httpd, ...)): Docker quay.io Image
  • Node Version: v.8.9.3
  • MongoDB Version: 3.2.18
    *Python 2.7.13
    *python-pymongo 3.4.0-1

Hi,

we have tested the script in our wekan installation but there will no data cleaned up.
I have set all the variables to 7 days.

Here is the output from script:
[USERS] Checking users emails compliance (lowercase)...
[BOARDS] Deleting 0 archived board(s)....
[BOARDS] Checking boards with no card into....
[BOARDS] Checking deleted user into boards (userId not more in database)....
[LISTS] Deleting 0 archived list(s)....
[CARDS] Deleting 0 archived cards(s)....
[CARDS] Checking deleted user into cards (userId not more in database)....
[LISTS] Deleting 0 orphan list(s) (boardId = False)....
[LISTS] Checking orphan lists (boardId not more in database)....
[CARDS] Checking orphan cards (boardId or listId or userId not more in database)....
[CHECKLISTS] Checking orphan checklist (cardId or userId not more in database)....
[CARD_COMMENTS] Checking orphan card_comments (cardId or userId not more in database)....
[ATTACHMENTS] Checking orphan attachments (cardId or userId not more in database)....
[ATTACHMENT_FILES] Checking orphan attachment files (attachment not more in database)....
[ATTACHMENT_CHUNKS] Checking orphan attachment chunks (file_id not more in database)....
[ACTIVITIES] Checking orphan activities (cardId or userId or boardId or listId or oldListId or commentId not more in database)....

In Wekan i can see the archived boards which are older than 7 days too. (See attached screenshot)
bugwekancleanup

I already tried to set the vars to lower values with same result.

Script deletes template-container boards if empty

After running the cleanup script on an docker instance of the newest wekan version (4.03.0), existing users can not access the template page ("page not found"). Further investigations showed that the template-container boards have been deleted and can not be recreated without manual hands-on on the db.

I'm not really into python and mongoDb syntax, but it looks like this part of the script is deleting all boards without cards

# Clean ALL live boards with no cards into

So boards with type "template-container" should be excluded here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.