glific/glific

The Main application that provides the core interface via the glific APIs

Glific - Two Way Open Source Communication Platform

Pre-requisites

Understanding of middle to advanced level required: It is assumed that you're comfortable using a terminal, installing tools and other dependencies, have git and curl for the backend, and yarn and react for the frontend.

1. Software dependency - Postgres server
2. Software dependency - Erlang / Elixir
3. Backend - Download
4. External service - Gupshup. <-- Get a Free trial to get API-key
5. External service - Oban. <-- Needs 100 Euro per month (patch available to work with free version)
6. Backend - Install certificate
7. Backend - Config
8. Frontend

1. Software dependency - Postgres server

• Download and start postgres server

For Postgres, for the development server, we default to using postgres/postgres/postgres as the username/password/machine name - this is configurable.

We tested and developed against the following versions:

    - postgres : v13.x, v14.x

2. Software dependency - Erlang / Elixir

• Install Elixir using asdf (check package versions below)

For Ubuntu users, you also need to install the inotify-tools package.

We tested and developed against the following versions (please check .tool-versions in the repository for the latest version we are using):

    - erlang : 26.1.2
    - elixir : 1.15.7-otp-26

After installing the asdf core, install the Erlang and Elixir plugins.

asdf plugin add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git

If you want to install the specific versions that were used for developing and testing:

asdf install erlang 26.1.2
asdf install elixir 1.15.7-otp-26
asdf global erlang 26.1.2
asdf global elixir 1.15.7-otp-26

If you get any warnings for missing packages, just install them using apt and try again.

Note: It is important to use asdf to install Erlang and Elixir.

3. Backend - Download

• Download the latest code from GitHub

git clone https://github.com/glific/glific

DO NOT run mix deps.get until the next steps are completed.

4. External service - Gupshup Create and link your Gupshup Account

Gupshup is an external service that connects to WhatsApp.

You will need to do the following:

• Create a Gupshup Account
• Create an app and select Access API
• You can name it NewNameHere "GlificTest <-- Bot Name is already in use, then use another one"
• Run the following command cp config/dev.secret.exs.txt config/dev.secret.exs
• Now, in Gupshup, find your API Key: check the top right corner and click the profile picture or inside the curl sample message
• Enter your APP name and API Key in the dev.secret.exs file using any text editor.

5. External service - Oban Pro

Oban is a cron-like library. Glific depends 100% on job processing. Oban is required before running mix for Glific to operate.

For contributors on social impact projects (including NGOs): Please get in touch with the team on Discord and get a limited-time key. Once they're provided to you, run:

For others, if you want to use the free Oban solution People have created and contributed versions of the code to allow Glific to work with the free version of Oban: #2391

mix hex.repo add oban https://getoban.pro/repo --fetch-public-key SHA256:4/abc/edf/gef+aIWPc --auth-key abcdefghi

with your keys

For production use: You must purchase a license. When purchasing, you must buy WEB+PRO. After purchasing, Go to account and run this command in glific_backend:

mix hex.repo add oban https://getoban.pro/repo --fetch-public-key SHA256:4/abc/edf/gef+aIWPc --auth-key abcdefghi

where public key "SHA256:4/abc/edf/gef+aIWPc" is replaced by your public key and auth key "abcdefghi" is replaced by your auth key.

Make sure your key is in the list:

mix hex.repo list

Name        URL                             Public key                                          Auth key
oban        https://getoban.pro/repo        SHA256:4/abc/edf/gef+aIWPc   abdedcqweasdj__KEY_AUTH__asdafasdf

If you see two key entries - caused by Oban moving from a public to a private repository - it will fail. This is what an example of failing looks like:

Name        URL                             Public key                                          Auth key
hexpm:oban  https://repo.hex.pm/repos/oban  SHA256:abc/edf/gef+aIWPc     abdedcqweasdj__KEY_AUTH__asdafasdf
oban        https://getoban.pro/repo        SHA256:4/abc/edf/gef+aIWPc   abdedcqweasdj__KEY_AUTH__asdafasdf

This is wrong. When you run mix deps.get as is, it will try to fetch from the public repository instead of the private one and fail. Simply follow the instructions below to remove the public key:

mix hex.repo remove hexpm:oban

Now, check again:

mix hex.repo list

Name        URL                             Public key                                          Auth key
oban        https://getoban.pro/repo        SHA256:4/abc/edf/gef+aIWPc   abdedcqweasdj__KEY_AUTH__asdafasdf

6. Install certificate - Use SSL for frontend and backend

Before completing the install, you need to create an SSL cert. Go to the glific_backend folder in the terminal console, and:

• Install mkcert (https://github.com/FiloSottile/mkcert)

• mkcert --install

• mkcert glific.test api.glific.test

• mkdir priv/cert

• mv glific.test* priv/cert

• cd priv/cert

• ls -1 Check that glific.test+1-key.pem and glific.test+1.pem exists.

• Check port 4001 sudo lsof -n -i:4001 | grep LISTEN should return nothing.

• Check hosts file grep glific /etc/hosts

  if it returns nothing, add these 3 lines to the hosts file:
  127.0.0.1 glific.test 
  127.0.0.1 api.glific.test
  127.0.0.1 postgres
  

For Windows the steps is as follows:

• Install mkcert (https://github.com/FiloSottile/mkcert)

• Run the following command to install the local CA certificates: mkcert --install

• mkcert glific.test api.glific.test

• mkdir priv/cert

• move glific.test* priv/cert

• cd priv/cert

• dir Check that glific.test+1-key.pem and glific.test+1.pem exists.

• Check port 4001 netstat -ano | findstr :4001 should return nothing.

• Check hosts file bytype %SystemRoot%\System32\drivers\etc\hosts | findstr glific

  if returns nothing
  add these three lines in your hosts file
  127.0.0.1 glific.test
  127.0.0.1 api.glific.test
  127.0.0.1 postgres
  

7. Backend - Config

• Run: cp config/.env.dev.txt config/.env.dev

• Run mix deps.get if this fails try mix local.hex --force followed by mix deps.get

  if you see the error below, then your Oban key is wrong or failing. Check step 5 or contact Oban.

  ❯ mix deps.get Failed to fetch record for 'hexpm:oban/oban_pro' from registry (using cache instead) This could be because the package does not exist, it was spelled incorrectly or you don't have permissions to it Failed to fetch record for 'hexpm:oban/oban_web' from registry (using cache instead) This could be because the package does not exist, it was spelled incorrectly or you don't have permissions to it ** (Mix) Unknown package oban_pro in lockfile

• Run mix setup At this point, you may get an error saying password authentication failed for user "postgres", in which case, you need to configure the postgres server properly:

createuser postgres -s # needed for more recent versions of postgres on MacOSgit
sudo -u postgres psql
ALTER USER postgres WITH PASSWORD 'postgres';

Exit the PostgreSQL terminal by typing \q and pressing Enter. Run mix setup again.

• Run iex -S mix phx.server
• Inside the iex (you might need to hit enter/return to see the prompt)
  • Update HSM templates by running the following command:
  • Glific.Templates.sync_hsms_from_bsp(1)

Now you can visit https://glific.test:4001 from your browser.

For Windows the steps is as follows:

• Copy the file: cp config/dev.secret.exs.txt config/dev.secret.exs

• Copy the file: cp config/.env.dev.txt config/.env.dev. You may not need to edit the default values for DB URL and hostnames in this file if they look suitable for your needs.

• Run this on the command prompt:

  cd <path-to-glific-backend>
  set /p=DUMMY < config\.env.dev
  

  Replace with the actual path to the glific_backend directory. This will load the environment variables from the .env.dev file.

• Run mix deps.get if this fails try mix local.hex --force followed by mix deps.get

  if you see the error below, then your Oban key is wrong or failing. Check step 5 or contact Oban.

  ❯ mix deps.get Failed to fetch record for 'hexpm:oban/oban_pro' from registry (using cache instead) This could be because the package does not exist, it was spelled incorrectly or you don't have permissions to it Failed to fetch record for 'hexpm:oban/oban_web' from registry (using cache instead) This could be because the package does not exist, it was spelled incorrectly or you don't have permissions to it ** (Mix) Unknown package oban_pro in lockfile

• Run mix setup

• Run iex -S mix phx.server

• Inside the iex (you might need to hit enter/return to see the prompt)

  • Update HSM templates by running the following command:
  • Glific.Templates.sync_hsms_from_bsp(1)

Now you can visit https://glific.test:4001 from your browser.

8. Front-end - Install glific front-end

You cannot do much from the glific backend unless you are an API developer. To see Glific in its glory, please install Glific Frontend

Front-end credentials

• Phone [phone]
• Password Secret1234!

Optional - Using NGROK

• Install ngrok
• Start ngrok to proxy port 4000:
  • Start the backend server: mix phx.server
  • $ ngrok http 4000 --host-header=glific.test:4000 (do this in a new window))
  • Remember the URL it assigns you, something like: https://9f6a7c7822d2.ngrok.io
• Goto the app settings section, Dashboard -> {{your_appname}} -> Settings.
• On that page, Search for Manage your Template messaging settings and enable it
• Goto app webhooks section, Dashboard -> {{your_appname}} -> Webhooks.
• Enter your callback URL that ngrok gave you, add: /gupshup to the end. Something like: https://9f6a7c7822d2.ngrok.io/gupshup/.
• Click Set. It should give you a Callback set successfully message. If not, check the above steps.
• Save the number [phone] on whatsapp and send a message PROXY {{your_appname}}.

Updating your instance

Run the following commands to update your codebase from the Glific repository.

• Ensure you are in the top-level directory of the Glific API code.
• Get the latest code from master: git switch master && git pull
• Ensure you have not modified any files in this directory, by running: git status
• Run the setup command: mix deps.get, compile, ecto.migrate

Documentation

• User Guide
• API docs (Postman)
• Code Documentation
• Recipes - Code samples for some common use cases in Glific.

Learn more

Glific

• Demo Video
• One Pager
• Product Features
• Glific Blogs
• Google Drive

Chat with us

• Chat on Discord

Funders

Thanks to our generous funders over the past few years who have funded this project:

• Project Tech4Dev
• The Agency Fund
• Cisco
• Omidyar Network India
• FOSS United

glific's Issues

Expose COUNT method on `Tags` table

Is your feature request related to a problem? Please describe.
I would love to be able to get a quick totalCount for a query to tags, or to the entire tag table itself, so that I can present to the user that we have X amount of tags without actually loading all the tags on the client-side (maybe there's 1000 objects, wouldn't want to load all 1000 objects into the table when I can just load the number 1000 and then load 10 at a time).

Describe the solution you'd like
If there could be a count parameter or a way to get the totalCount from a potential GraphQL query without having to necessarily run the entire query, that would be great!

Create and process optout tag for an user.

Is your feature request related to a problem? Please describe.

1. Create a tag called OptOut with keywords: stop, halt, unsubscribe (and their Hindi equivalents)
2. If the message body matches those keywords -> it gets tagged as optout

For messages tagged optout

1. Set the current timestamp to optout_time filed for the contact.
2. Set provider status as invalid for the contact
3. Send a session template message with shortcode "optout" if one exists

We can follow a similar pattern as New User Pattern and Welcome Message

When we start processing opt-outs from provider, we might not be able to send them back a message. we'll handle this case at that time

Migrate Tags to Glific

Add __meta__ to type for all schema definitions. Also fix delete operation

1. We have not added the meta field to the t() definition in our schemas. We need to add it to all the schema

2. Delete should respect FK constrains, Check language for an example which has an FK constraint from tags

%Language{}
%Glific.Settings.Language{
  _meta_: #Ecto.Schema.Metadata<:built, "languages">,
  description: nil,
  id: nil,
  inserted_at: nil,
  is_active: false,
  label: nil,
  locale: nil,
  tags: #Ecto.Association.NotLoaded<association :tags is not loaded>,
  updated_at: nil
}

Clean up Enums

Is your feature request related to a problem? Please describe.
Enums is polluting the top level namespace with one directory and two files.

Let's move it into its own directory called enums and all files under it. While we are at it, lets rename stuff to be consistent and call it Glific.Enums*

Migrate automated seeding of large number of messages and contacts

Migrate contact_tags to Glific

Message Tag context

Enable sobelow check when we get a better handle on it

We are ignoring it right now, so we'll disable it for now. Filing an issue so we enable it before the end of the month and fix the errors

Session template improvements ..

Is your feature request related to a problem? Please describe.

• Add :type to session_templates similar to messages

• Add a send_session_message(contact_id(s), session message id) - we need this for automation in API and graphQL

Restructure and migrate Communication + Gupshup to Glific

Make organization contact id non nullable

Describe the bug
There are some weird issues (Maybe Because of Ecto config) in the partner tests.

We will do it once we finalize the organization structure.

Add message, contacts, tags search to GQL

Given a search term we need to:

1. Find all the messages: body (and hence contacts) matching this term

2. Find all contacts whose name or phone matches this term

3. Find all contact and messages that have a tag: label, description attached to them that matches this term

Migrate BSP to glific

BSP will be all the services providers for our message communication. We will start with Gupshup and then add more BSP.

Add graphql subscriptions ...

We also need to think of all the places where we'll need to inform clients of state changes.

this is primarily in the message send/receive flow and any automated actions we take after receiving a message

Cleanups in message workflow

We need to make some changes in the code

• - Add custom validation - Media ID should not be null if the message type Is not text
• - Code cleanups in message controller
• - Add message timestamp when the queue worker will send the message.
• - Update status when we receive the message event callback.

Group by query is not consistent in mac and linux system

Describe the bug
In mac system server collation is not working with postgres and hence giving incorrect alphabetical order of strings (for case sensitive data)

1. If we write test cases for case insensitive data, it doesn't pass in local , but passes in the github.
2. This would happen for all the resources.

Define schemas with typespecs without all the boilerplate code.

Currently, we are adding writing type specs for every schema.
This is a bit problematic for some of the reasons, like code duplications and all.

https://github.com/bamorim/typed_ecto_schema looks a good solution but they did not provide some of the Ecto schema relationship functionality yet.

We created an issue on that pacakge bamorim/typed_ecto_schema#5.

Add parent_id and ancestors to messages

Add parent_id to messages
Cache ancestors (up to a limit) for efficient conversation retrieval

Migrate session_messages from two_way

Is your feature request related to a problem? Please describe.
Migrate session_messages from two_way to glific. This include: DB structure, ecto schema, graphql and tests

Describe the solution you'd like

1. Call the table: session_templates. The top level context should be templates (since we will add hsm_templates in v0.3)

2. add message_media_id to the table also

3. delete the is_translated field

4. Add a shortcode field called shortcode

5. Add a couple of sample messages in seeds and seeds_scale. Two examples could be a sample
   template for New User and a goodbye message

Conversations with filter returns empty value

Describe the bug

query conversations($nc: Int!, $sc: Int!, $filter: ConversationFilter) {
  conversations(numberOfConversations: $nc, sizeOfConversations: $sc, filter: $filter) {
    contact {
      id
      name
    }
    messages {
      id
      body
    }
  }
}

with variables

{
  "nc": 1,
  "sc": 1,
    "filter": { "id" : 359 }
}

returns

{
  "data": {
    "conversations": []
  }
}

To Reproduce
Steps to reproduce the behavior:

• Use the above queries and check the result.

Expected behavior

• It should return contact data with the messages
• If messages are empty then it should return contact data and empty messages object

Migrate database structure to glific

Let's keep all the tables in one migration file, makes for easier readability

seeds_scale should interleave sends and receives

Right now, all sent messages are first listed and then all received. This does not make it great for a real world scenario. we need to interleave them

also make sure the times are increasing and not the same time for everything. we need valid values to test things @ scale

also add the contact_id field for messages (I'm committing a fix for this soon)

Apply coveralls library for code coverage

Option to send message to multiple contacts

Organizations should be able to send messages to multiple contacts.

Check the contact optin status before sending a message.

Is your feature request related to a problem? Please describe.
We need to validate the contact provider status before sending the message.

If the status is invalid then we the function will raise an Error that (Contact is not valid or opt-out)

Add docs to github pages

Is your feature request related to a problem? Please describe.
We will expose all our code documentation to the Public, so we can link it

Describe the solution you'd like
Expose docs via github pages

Cleanups in the file name and migrations

Describe the bug
A clear and concise description of what the bug is.

• Rename BSP to Provider?
• Rename recipient_id to receiver_id? (sender <-> receiver)
• Rename anything with wa_ as provider_status, provider_message_id
• Rename message_media -> messages_media
• Quick check of file names, like: messages_tags.ex => message_tag.ex (context is plural, the datatype is singular)

Migrate Language to Glific

Adding OFFSET and LIMIT functionality

Is your feature request related to a problem? Please describe.
I would love to see both OFFSET and LIMIT functionality for different GraphQL requests we make in the front-end.

Describe the solution you'd like
If these parameters could be added to the different objects, that would be great!

Update mix dependancies, if they are outdated

Describe the bug
We need to have a look for the outdated version for all the deps and update them.

Please run a mix check after updating any dependency so that we can catch compatibility issues and fix them.

Make changes in conversation API based on the Ancestors

Currently, we are fetching all the parent message-ids by a query.
We need to change this based on the Ancestors fields.

Expose badges on glific repository

Is your feature request related to a problem? Please describe.
We need to expose: Build passing. code coverage, doc coverage and other badges to our github repository

This gives the project a lot of credibility and looks good :)

Tag Search for Glific

Is your feature request related to a problem? Please describe.
Allow user to search on all tags in glific

Once the above is done, add filter for language and parent tag

Migrate enums to glific (from two_way)

Enums form the basis of many constants in Glific, used across Postgres, Elixir/Ecto and Absinthe. Unfortunately, all three use slightly different variations of the same values. The advantage of this is it gives us strong type checking in GraphQL APIs

We adopted the enum standard from Elixir - Ecto / Absinthe Enums Recipe

Add GraphQL for Contacts

Collapse multiple related gql files in wormwood in one file

Right now for CRUD and Search operations on one object, we create multiple GraphQL files to exercise each operation. Might be easier to name each operation and have one file. Might make better reading and conciseness.

Created an issue in wormwood github also:

wormwood-elixir/wormwood#17

Incorporate Github Actions into CI Process for PR management

Migrate message and message media to GQL

Add support for keywords in Glific Tags

Is your feature request related to a problem? Please describe.

(revising this from my first idea)

Extends tags functionality to associate keywords that can be linked to a tag. All incoming texts, are automatically parsed and if there is a keyword match, we associate the tag with the message

Thus Hello/hi/hola/namaste/good morning could all be keywords for the tag "greeting"

numeric tags are automatically parsed by the system and have a value

We can drive automation both on the tag, and the value of the tag

I've started work on this in the genstage branch, and will merge into master by end of day

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Migrate message+contact tags to GQL

Add unread tag to message

Is your feature request related to a problem? Please describe.
We will tag every incoming message as unread.
The frontend team will make a query to remove this tag from the message.

Migrate BSP and Organization to GQL

Add user id field to message table

Describe the bug
We need to set a user id to the massage table so that we know when sent the message to the beneficiary on the behalf of an organization.

Since there are some automated messages which will don't have any user id so we will make this field nullable.

This change will also help us to audit the messages and all.

Cache contact_id in messages table

This avoids us writing conditional code in a bunch of places to see whether it was a sent or received message and makes some queries a lot simpler and easier to read

At some point, we might also want to add a counter to each message specific to a contact to optimize the queries even more

Feature: Search from messages

Give an option to users to search across messages based on message text and tags.

Add filter for tags

Migrate Contacts to Glific

Add a mock Whatsapp provider for testing and demo purposes

Let's add a mock provider so that we don't need to make some API calls in our testing and demos.

Migrate graphql absinthe to glific

Add conversations API to glific

Is your feature request related to a problem? Please describe.
Conversations are the bedrock of the application. The frontend displays conversations, and needs to be able to retrieve them quickly

Describe the solution you'd like
Create an API that we can expose via GraphQL

• Give me the conversations filtered and parameterized with
  • limit to N most recent conversations
  • Each conversation has a limit of M messages
  • Filter by tag on the message
  • Filter by tag on the contact
  • Filter by a set of contact IDs