meilisearch / cloud-providers Goto Github PK

☁ Meilisearch DevOps Tools for the Cloud ☁

License: MIT License

Shell 74.19% HCL 25.81%

cloud-providers's Introduction

Website | Roadmap | Meilisearch Cloud | Blog | Documentation | FAQ | Discord

⚡ A lightning-fast search engine that fits effortlessly into your apps, websites, and workflow 🔍

Meilisearch helps you shape a delightful search experience in a snap, offering features that work out-of-the-box to speed up your workflow.

🔥 Try it! 🔥

✨ Features

Search-as-you-type: find search results in less than 50 milliseconds
Typo tolerance: get relevant matches even when queries contain typos and misspellings
Filtering and faceted search: enhance your users' search experience with custom filters and build a faceted search interface in a few lines of code
Sorting: sort results based on price, date, or pretty much anything else your users need
Synonym support: configure synonyms to include more relevant content in your search results
Geosearch: filter and sort documents based on geographic data
Extensive language support: search datasets in any language, with optimized support for Chinese, Japanese, Hebrew, and languages using the Latin alphabet
Security management: control which users can access what data with API keys that allow fine-grained permissions handling
Multi-Tenancy: personalize search results for any number of application tenants
Highly Customizable: customize Meilisearch to your specific needs or use our out-of-the-box and hassle-free presets
RESTful API: integrate Meilisearch in your technical stack with our plugins and SDKs
Easy to install, deploy, and maintain

📖 Documentation

You can consult Meilisearch's documentation at https://www.meilisearch.com/docs.

🚀 Getting started

For basic instructions on how to set up Meilisearch, add documents to an index, and search for documents, take a look at our Quick Start guide.

⚡ Supercharge your Meilisearch experience

Say goodbye to server deployment and manual updates with Meilisearch Cloud. No credit card required.

🧰 SDKs & integration tools

Install one of our SDKs in your project for seamless integration between Meilisearch and your favorite language or framework!

Take a look at the complete Meilisearch integration list.

⚙️ Advanced usage

Experienced users will want to keep our API Reference close at hand.

We also offer a wide range of dedicated guides to all Meilisearch features, such as filtering, sorting, geosearch, API keys, and tenant tokens.

Finally, for more in-depth information, refer to our articles explaining fundamental Meilisearch concepts such as documents and indexes.

📊 Telemetry

Meilisearch collects anonymized data from users to help us improve our product. You can deactivate this whenever you want.

To request deletion of collected data, please write to us at [email protected]. Don't forget to include your Instance UID in the message, as this helps us quickly find and delete your data.

If you want to know more about the kind of data we collect and what we use it for, check the telemetry section of our documentation.

📫 Get in touch!

Meilisearch is a search engine created by Meili, a software development company based in France and with team members all over the world. Want to know more about us? Check out our blog!

🗞 Subscribe to our newsletter if you don't want to miss any updates! We promise we won't clutter your mailbox: we only send one edition every two months.

💌 Want to make a suggestion or give feedback? Here are some of the channels where you can reach us:

For feature requests, please visit our product repository
Found a bug? Open an issue!
Want to be part of our Discord community? Join us!

Thank you for your support!

👩‍💻 Contributing

Meilisearch is, and will always be, open-source! If you want to contribute to the project, please take a look at our contribution guidelines.

📦 Versioning

Meilisearch releases and their associated binaries are available in this GitHub page.

The binaries are versioned following SemVer conventions. To know more, read our versioning policy.

Differently from the binaries, crates in this repository are not currently available on crates.io and do not follow SemVer conventions.

cloud-providers's People

Contributors

Stargazers

Watchers

Forkers

chapa-et

cloud-providers's Issues

Make systemd logs persistent

in /etc/systemd/journald.conf add:

[Journal]
Storage=persistent
...

[GCP] Issue with importing Image to GCP

Description
Running into an error when importing the meilisearch image to GCP. It looks like the error is coming from the image (not gcp).

This happens about 12 minutes into importing the image from the URI provided by the meilisearch documentation: meilisearch-image/meilisearch-v1.0.2-debian-10.vmdk

Expected behavior
Successful import of the meilisearch image into GCP compute engine.

Current behavior
Error about 12 minutes in.

Screenshots or Logs
Last few lines of the logs:

[import-image]: 2023-07-14T14:47:57Z step "translate-disk" run error: step "wait-for-translator" run error: WaitForInstancesSignal FailureMatch found for "inst-translator-translate-translate-disk-r5qr1": "TranslateFailed: error: E: Sub-process /usr/bin/dpkg returned an error code (1)"
ERROR
ERROR: build step 0 "gcr.io/compute-image-tools/gce_vm_image_import:release" failed: step exited with non-zero status: 1

Environment (please complete the following information):

OS: [e.g. Debian GNU/Linux]
I believe the image is using Debian? When importing I set to "Detect operation system" since the meilisearch documentation doesn't say to specify.
Meilisearch version: [e.g. v.0.20.0]
v1.0.2
cloud-providers version: [e.g v0.1.0]
Not sure what version this is

Configure certbot: user email and auto-renew

Currently, certbot is used to create automatic SSL certificates if the user selects this option. This process needs to specify a user email address and creates a temporary certificate. The way it is handled now raises two main problems:

This script is using a MeiliSearch email address, so the user doesn't get notifications.
The auto-renew process is not handled

When these certificates expire, not only they probably won't be renewed automatically, but the user won't be notified about it.

"Request parse error: Invalid Header provided" error when trying to create a dump with cURL in a DO droplet

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to DO and create a droplet with the official Meilisearch image
Setup the Meilisearch instance for production with the setup script
Add some documents to Meilisearch:

curl \
  -X POST 'http://{address}/indexes/movies/documents?primaryKey=id' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer MASTER_KEY'  --data-binary @movies.json

Check task has succeeded:
curl 'http://{address}/tasks' -H 'Authorization: Bearer MASTER_KEY'

Send a create a dump request:

curl \
  -X POST 'http://{address}/dumps' \
  -H 'Authorization: Bearer MASTER_KEY'

No response received

Check logs:

[2022-12-01T12:44:00Z ERROR actix_http::h1::dispatcher] stream error: Request parse error: Invalid Header provided

Expected behavior
The request is enqueued and eventually processed. I receive a response with a taskUid

Meilisearch version:
The official Meilisearch image in DO is still using v0.29.0, I installed v0.30.0 manually. The behavior occurs with the 2 versions.

[AWS] Add every region

Meilisearch AWS AMI is only available in one region. We need to explicitly define all the regions we want it to be available.

[DO] Marketplace: Application Description and Getting started instructions in repo

Marketplace submission require to fill 2 fields in the form with full text: Application Description and Getting started instructions

This information is only stored in DO interface. It should be available in the repo, in order to modify it before submission, accept contributions and reviews, and store in our own repository.

In the release process, this should be copied from the repo to the DO interface

Create Disclamer on every devops-tools

The creation of this directory will make these 4 other repositories obsolete:

meilisearch-digitalocean
meilisearch-aws
meilisearch-gcp
cloud-scripts
A disclaimer must be written and added in each of these

[DO] Certificate expired error

Describe the bug

I hosted Meilisearch on digital ocean using the one-click app from the marketplace. I followed the instructions on the ssh console and quickly set up domain name and SSL in minutes. Thanks for that !

However within few days (around a week), some of my app users started running into the certificate expired issue. When I login into my droplet and check for renewal using certbot, I see no certificate pending renewal.

Not sure what causes this issue. PFA the screenshot attached for the same. Please also let me know if I need to provide any information or file contents from my droplet.

Regards,
Bhavya

[AWS] Guide to install Meilisearch using ECS

it will be great if we get clear steps to install Meilisearch using ECS

Create user documentation

Write the user documentation

You can inspire from the README of devops-tools repo and the README of cloud-scripts repo

README.md
CONTRIBUTING.md

Following this template to check everything as been well created

[AWS] Add cloudwatch agent to image to allow ssh-less access to logs

Description
In an usual deployment scenario you might be interested in the log files created by meilisearch. The current AMI does not include the cloudwatch agent & the documentation also does not include instruction on how to set-up the agent. As of now the only way to access those otherwise would be by ssh'ing into the machine which is not always encouraged.

Basic example
AWS provides a documentation that explains how the agent should be installed with valuable notes:
https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/include-cloudwatch-agent-amis.html

Create the core of the project

Initialize the project by adding the needed tools:

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

packer {
  required_plugins {
    amazon = {
      version = ">= 1.1.1"
      source = "github.com/hashicorp/amazon"
    }
  }
}

Rewrite the UFW commands in the cloud-config.yaml files

Rewrite the UFW commands in the cloud-config.yaml files :)

Add `CODEOWNERS` file to allow doc team to be notified of when GCP bucket is updated

Description
For each release on this repository, a new public bucket is published on GCP, and the address to this bucket must be updated in the documentation.
To ensure that the documentation team is aware of this change, a CODEOWNERSfile should be added.

Disadvantage
The granularity of the CODEOWNERS feature is at the file level, so, unfortunately, the @meilisearch/docs-team will be notified for every modification on the main file on this repo.

To do

Give write access to @meilisearch/docs-team
Create CODEOWNERS file at the root

Implement github-action workflow 2 - Build and Test the image

See packer-github-action

See build

Find a way to test the image (curl ? /health )

Create an instance from AMI
Test the instance
Remove the instance

Note: Remove AWS Snapshot / Script

Two solutions:

Shell script
Terraform

Update integration-action for pre-release-script

Delete the old script for these future obsolete repositories:

Add a new script for the pre-release automation for this repository

Check and fix the Nginx issue

Meilisearch images on different providers are currently configured via Nginx

See this issue

[AWS] Remove old AMI image after release

Description
This repo allows us to create EC2 images that will then be published as AMIs on AWS. However, when new releases are made, new AMIs are added, but the old versions remain on the platform. It would be good to keep only the last 2 or 3 versions not to overload our library of available AMI.

Possible solution
As we will probably already have terraform on this repo thanks to this PR #19, we could consider using this tool to make work

[AWS] Image private to public

Make the final image public automatically, instead of doing it manually.

Write the `provisioner` 1 - Provision the image

Configuration of the machine with the tools needed to install Meilisearch

See how to write the Provisioner section:

Software to install:
- wget to download Meilisearch binary
- cerbot (automatically using Let’s Encrypt certificates)
- python-certbot-nginx
- nginx
- ufw
File to download:
- 2 first-login scripts (meilisearch-setup)
- MOTD
- meilisearch.service (configuration file for Meilisearch)
- nginx-meilisearch (configuration file)
Variable to set:
- Retrieve provider name
- Set env variable for Meilisearch

[AWS] Install SSM Agent to AWS AMI image

Hello 👋

I'm transferring a feedback initially created on the product repository since it seems better suited here. Let me know if it's not the case @alallema 🙏

"Please install AWS's SSM Agent so it would not require to open SSH port and manage SSH key. Would be able to use Session Manager instead and access management would be inside IAM permissions instead, which is way more secure."

Add DigitalOcean provider

https://developer.hashicorp.com/packer/plugins/builders/digitalocean

Write `post-processor` - Create Artefact

Packer allows launching post-processor commands either after the build of an image.
The manifest post-processor writes a JSON file with a list of all of the artifacts packer produces during a run, see manifest

Move to read-only all the deprecated repositories

All of these repositories will be read-only after finishing #16 #7 #8

⚠️ Also, they should contain a disclaimer mentioning the new repository (this one).

[GCP] image bucket lacking permissions (storage.buckets.get)

It seems like the gs://meilisearch-images bucket has restricted their permissions since I last used it. I no longer can create images from the disks in the bucket due to permission issues.

tarpsvo@cloudshell:~ (my-app)$ gcloud compute images import meilisearch-v130 --source-file=gs://meilisearch-image/meilisearch-v1-3-0-debian-11.vmdk

ERROR: (gcloud.compute.images.import) User [[email protected]] does not have permission to access b instance [meilisearch-image] (or it may not exist): [email protected] does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).

I can run gcloud storage ls gs://meilisearch-image just fine, but GCP won't let me create images out of them.

Thanks!

Create a manual CI trigger for each supplier

Releases are currently launched when a tag is pushed to the main branch.
These releases are automatically published, and the script is launched in its entirety, allowing all providers to be published.
However, what will happen if some of them fail?
It would be nice to create events that can be run manually for each of them.

Fix the size of DO image

Add packages to updates info in MOTD

Based on previous discussions with @eskombro & @curquiza

Update every integration guide and tools

Add this new repo and remove the older ones (cloud-script, meilisearch-aws, meilisearch-gcp, meilisearch-digitalocean)

On the meilisearch/integration-guides:

Update REAMDE.md
Update pre-release-week.md
Update dependencies.md
Update the scopes
Update the issue.md

On the meilisearch/integration-automations: (internal repository)

Update README in SDK version bump
Update automatic response for github script
Update all the octopus and pre-release scripts

MVP

Project link on Notion
The MVP for this project might include the following features:

Base:
- #11
- #14
Reproduce the Meilisearch image build on at least the vendors already in the old DevOps tool:
- #25
- #26
- #27
- #28
  - Add AWS provider:
  - #15
  - #7
CI/CD features:
- #12
- #18 (Optional)

The following also has to be done (during the development process):

Error management
Testing
Update of the Readme

Transfert to DevOps-tools to cloud-provider

Possible addition (Optional)

[AWS] - installation docs use debian which is not cost efficient

please consider making AMI on graviton ec2, it's most cost efficient way to run memory intensive loads on aws:

https://aws.amazon.com/ec2/graviton/

https://aws.amazon.com/ec2/instance-types/x2g/

e.g. with curren AMI free tier on aws provide 1 cpu 1gb ram, with graviton it would provide double - 2cpu & 2gb ram.

[gcp, aws, digitalocean] Ensure the env var `MEILI_SERVER_PROVIDER` is being sent

We need to ensure that we are setting our environment variable correctly.

Like we used to do in the cloud-scripts: https://github.com/meilisearch/cloud-scripts/blob/56a7c2630c1a508e5ad0c0ba1d8cfeb8d2fa9ae0/scripts/providers/gcp/cloud-config.yaml#L33

Add GCP provider

https://developer.hashicorp.com/packer/plugins/builders/googlecompute

Implement github-action workflow 1 - Lint and Validate the template

See packer-github-action

Packer has a command to format HCL2 configuration files in a canonical format and style.
See fmt

Packer has a command to validate the syntax and configuration of a template.
See validate

[GCP] Export a custom image to Cloud Storage

Description
Create an Issue for GCP to transform the image to Bucket
Currently, the packer script will create an image of the release. The goal is to add the missing part who is convert it to a bucket to allow users to use it.

See the GCP doc on it: export a custom image to Cloud Storage

Add Azure provider

https://developer.hashicorp.com/packer/plugins/builders/azure

Write the `provisioner` 2 - Configuration of the image to launch Meilisearch

Configuration of the image to launch Meilisearch

Create a Meilisearch system user and give it the permission to the folder /var/opt/meilisearch
Move all provisioned files to the right place (including script and configuration files)
- meilisearch.service
- 000-set-meili-env.sh
- 001-setup-prod.sh
- 99-meilisearch-motd
Add MEILISEARCH_ENV_PATH to .bashrc
Add meilisearch-setup to aliases and auto-run
Change execution right of Meilisearch binary
Launch Meilisearch via systemctl

[AWS] How to add prefix url globally?

I have loadbalancer, currently we doing path based routing. I want to route the request from "api/search/*" path to ec2 instance. To match the pattern we need to add prefix url globally. So is any way available to add "api/search/" prefix url globally on meilisearch ec2?