ail-project/lacus

Lacus is a capturing system using playwright, as a web service.

lacus's Issues

fix(README.md) Update installation process in README.md

Lacus

A capturing system using playwright, as a web service.

Install Guide

System dependencies

You need poetry installed, see the install guide.

Prerequisites

You need to have redis cloned and installed in the same directory you clone this template in:
this repository and redis must be in the same directory, and not redis cloned in this directory. See this guide.

Installation

Clone this repository:

git clone https://github.com/ail-project/lacus.git

From the directory you just cloned, run:

cd lacus
poetry install

Install the system dependencies required by playwright (will call sudo):

poetry shell 
playwright install-deps

Initialize the .env file:

echo LACUS_HOME="`pwd`" >> .env

Initialize the config and install playwright browsers:

poetry run update --init

* Restarting
Continue? (y/N) <-- N
Okay, quitting.

Clone the Redis server:

1. Navigate to the parent directory:

pushd ..

2. Clone the Redis repository:

git clone https://github.com/antirez/redis.git

3. Enter the Redis directory:

pushd redis

4. Checkout the version "7.0" of Redis:

git checkout 7.0

5. Compile Redis:

make

6. Return to the previous directory:

popd

7. Return to the initial directory:

popd

Lacus CLI(Command line Interface)

start

• poetry run start: This command is used to start a Lacus service

poetry run start 

stop

• poetry run stop: This command is used to stop a Lacus service

poetry run stop

update

• poetry run update: This command is used to update the project dependencies according to the configurations defined in the pyproject.toml file.

poetry run update

shutdown

• poetry run shutdown: This command is used to completely shut down a Lacus service

poetry run shutdown 

run_backend

• poetry run run_backend: This command is used to manager the backend redis

poetry run run_backend --help 
  -h, --help  show this help message and exit
  --start     Start all
  --stop      Stop all
  --status    Show status

start_website

• poetry run start_website: This command is used to start the web server

poetry run start_website

capture_manager

• poetry run capture_manager: This command is used to start the capture manager for data.

poetry run capture_manager

stop_capture_manager

• poetry run stop_capture_manager: This command is used to stop the running capture manager.

poetry run stop_capture_manager

@Terrtia @SteveClement @adulau @Rafiot @gallypette

Websites down - Where I can find logs?

Hello,

I am starting with AIL as well as with its crawler Lacus. However I am getting all errors after requesting/crawling websites? Do you know where could be problem or where I can find application logs to get more info? Thanks in advanced!

New Feature: Add statistics

Statistics by day:

• How many captures
• How many errors / list of errors and counter
• How many retries / successful retries

[Feature] Allow to restart the capture manager only

It is not super safe, but most of the time, the changes only apply to the process running the captures. Restarting this process only will most of the time be good enough and this way we won't trigger exceptions on the client side.

Problems installing

ail-project/ail-framework#164

Update installation process in README.md

[Feature] Clean handling of SIGINT

It would be nice to catch CTRL+C passed to the capture_manager in order to wait for the ongoing captures to finish but SIGINTs are directly passed to playwright and the browser is killed instantly: microsoft/playwright-python#1170

Alternative solution: catching SIGTERM instead.