innoraft/kabootr

An email marketking and newsletters solution by Innoraft.

kabootr's Issues

Quick research to check if deeper itegration is requried with Amazon SES for mailing statistics.

As evident from the sendy snapshot below there are various kind of statistics that are available for a mailing campaign.

We need to do a quick research (not implementation, just getting to know) to figure if deeper integration with Amazon SES is required (beyond just using the drupal SMTP module, or drupal php sendmail() to talk to SES) for sending mails in a way such that we have statistics avaiable.

Or may be sendy is getting all these stats some other way?

This issue can be successfully closed after we have a rough quick document of how we would need our kabootr drupal talk with Amazon SES. This issue does not deal with the implementation.

Sendy's code can be looked upon from here - https://github.com/AamAadmiParty/mass-emailer-sendy/ and I am sending the unlocked code (sendy has 2 locked files) over email.

Set batch operation, when newsletter list is created..

Currently when we create newsletter or subscription list, many user are subscribed to newsletter category. So there is chance of php time out. So I am setting batch for this operation.

Add Left navigation for doing administrator amazon ses configuration.

• Required left navigation bar, which will contain all the amazon ses configuration links.
• Create work-flow for getting access token from AWS account, so that kabootr can send mail.
• Use amazon SNS notification service for receiving mail feedback detail.

Create Amazon SNS end point in kabootr

Create Amazon SNS end point in kabootr which will receive notification from Amazon SNS.

Create Kabootr reports interface for bounce / spam / unsubscribe

Create Kabootr reports interface for bounce / spam / unsubscribe

reference : #38

Document the broad idea and the component details

• What are we building?
• What is special about it?
• How would be go about it?
• A functional idea of major components?

Kabootr's menu system, from which whole functionality is controlled..

Kabootr menu system discuses with sumeet.
https://docs.google.com/a/innoraft.com/document/d/15jCORZ9iT09JJxgzB17JmmlzCLsZFq5iD4Nw5BsXCOw/edit

Simple newsletter module research and create the dynamic list functionality

Need to study and understand all aspects of simple newsletter module.

And then create a functionality such that if we run a query that outputs N number of user ids, that should create a new list of simple newsletter subscribers on the fly.

1. To be able to do that we will have add some temp filed to the user profile (so that our queries can be tested.
2. Need to generated large number of dummy users
3. (Should the generated users be automatically added to subscribers table (not the subscriptions table))?
4. Thought should be given to how the various data points (opened the mail, bounced, spammed etc.) will be marked. This does not has to be implemented yet, but our dynamic lists should allow this in future.
5. Dev site should be ready on the server to be able to test this.

User login blocking for all non admin users

User login blocking for all non admin users

reference : #46

Setup an Amazon EC2 instance

Setup an Amazon EC2 micro instance for Kuldeep to put the Drupal application over there.

Create arch to support bounces / spam / unsusbcribe - against lists and nodes

Create arch to support bounces / spam / unsusbcribe - against lists and nodes

Gaurav and Kuldeep views

• When campaign unsubscribed or spam or bounces, in thrice cases user has to be unsubscribed from campaign.
• In simplenews letter contrib module's architecture user is unsubscribed from Newsletter category, not newsletter node. So if a newsletter category contains 5 node, if user is unsubscribed from 1 node, he will be automatically unsubscribed from all 5 nodes.

Past discussion with Mukesh

• If we want to create more than one campaign, then we will create more than one Newsletter category.
• Mean one newsletter category will contain only one node in kabootr.

So For supporting bounces / spam / unsusbcribe

• Add more three column in table simplenews_newsletter, contain the detail of newsletter node status.
  • complain_count
  • bounces_count
  • unsubscriber_count
• In thrice cases user will be unsubscribed from newsletter category.

Acceptance Criteria

• We will have inputs
  • Newsletter nid
  • User's mail
  • Action: Complain, bounces or unsubrcibed
• On the basis of these inputs user should see report. #43

(Stub) Do a complete dry run of how Sendy works for AAP

This needs to be expanded as per disucussion and new smaller issues referencing this need to be created.

Take full back of kabootr dev.

Export 100-200 records from AAP lists in Sendy.
Study them and create "data fields" on kabootr dev.

Edit the sample AAP data to use safe INNORAFT mails.
Import this data to kabootr

Create multiple dynamic lists with different types of filters.
Name these lists such that they describe the filter

Create and send campaign. Be 200% sure that all mails are only going to INNORAFT addresses.

From the entire process create notes on what we need to work on next for DCD release of Kabootr.

I was not able to mount EBS volume in amazon EC2 service.

I have created EBS volume of capacity 10 GB,

Then for mounting running this command sudo mount /dev/sdf /var
seeing this output mount: special device /dev/sdf does not exist
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

research how sandy knows that a mail sent is read by user or not

Roles management in Kabootr

create roles - system manager, campaign manager, subscribers
all the roles should be locked
no subscriber should be able to login unless he is in other roles too

Test if DKIM verification works

We need to check that is DKIM related changes are actually done on the server, do they really get picked up by the AWS API call in Kabootr and the domain or email shows as DKIM verified?

Make the module that allows drupal to send mails via Amazon AWS + SES contributable.

Make the module that allows drupal to send mails via Amazon AWS + SES contributable.

Details to be filled soon.

AWS SNS setup should be able to talk back to Kabootr

AWS SNS setup should be able to talk back to Kabootr

(stub) Exclude bounces and spam marked in future

(stub) Exclude bounces and spam marked in future

Repository and Server setup

1. There should be the master, develop and release branches in the repository.
2. All the drupal setup should be in the REPOROOT/docroot folder + there should be a REPOROOT/drush folder (we will add more folders later on)
3. The necessary drupal "getting started" setup should be done (typical development modules, module directory structures, controller feature, .gitignore file etc.)
4. The dev version of the site should be setup on a server that the team can access. (@mukeshagarwal to be the person to decide which machine, what server (apache or nginx), give access to @tkuldeep etc.)

Use of amazon SES service to send mails.

• For sending mails, we will use amazon SES service.
• Amazon SES provides useful statistics about our sending activities. With a simple query we can quickly obtain vital statistics such as volume sent, bounces and complaints.
• These statistics will be used in our project to show the status of sent mail.
• Spam users will be unsubscribed via this feedback service of amazon .

Research on best way to send efficient http request, and control over sending mail limit.

Way of sending Simplenews module email

• Triggered the sending mail by cron job, where it reads one by one
  • The recipient and mail content, send mail then update the status, that mail has been sent to that recipient or it is failed.

Email Sending in Amazon

• Send one http request to call Query API action SendEmail, so if we have 100 recipients, then Query API will be called 100 times.
• If we send mail single mail with multiple recipients, if mail content is same, then we can send mail to 100 recipient in two times. [Max 50 per request], But in this case all recipient will have the other recipient address, which is not desired [According to me].

So by studying above points, we have some conclusion.

• We have to send one request for each recipient even if mail content is same.
• So for controlling and sending maximum mail with controlled sending rate, we have to control http request.
• For sending request to web service, I am thinking to use Guzzle php http://guzzlephp.org/index.html.
  • It provides best way of sending request to web service client. http://guzzlephp.org/webservice-client/webservice-client.html
  • Request can be sent in batch. http://guzzlephp.org/batching/batching.html
  • I need some time to study guzzle and implement it.

Mail are not sending using aap ses secret keys

Kabootr must send html mails

Kabootr must be able to send formatted emails.

reference : #46

We should be able to test mails going out from kabootr via amazon ses

See the images below

1. when we create content
2. of type newsletter and goto newsletter tab
3. DO NOT select send when published
4. use the test option
5. enter an email where you can check the mails are coming

The incoming mail from kabootr should reach our inbox and we should be able to verify the from address and the 'via amazon ses' part as shown below.

Create subscription list during creation of newsletter category.

• During creation of newsletter category, we will add users filters so that subscription list can be created.

Settings navigation pane, links in it, and the pages/forms that these links point to

The order, name and paths of the links to be used can be picked from this google document - https://docs.google.com/a/innoraft.com/document/d/15jCORZ9iT09JJxgzB17JmmlzCLsZFq5iD4Nw5BsXCOw/edit

The AWS access key and secret entries form should have the right default value set.

After verification of the sender ID the page where the control goes to is wrong. There is a typo in the URL that needs to be fixed.

(stub) Basic reporting of sent count, opened, spam mark, bounce should be there

(stub) Basic reporting of sent count, opened, spam mark, bounce should be there

First simplenews category should be called MASTER LIST and not the site name

First simplenews category should be called MASTER LIST and not the site name

Work to be done before 7th nov 2013

• Install kabootr on AAP amazon and setup config + user import
• If you are out (bounce/spam) from cat A, you should not be included in any mails in any cat (B, C, D)
• Install PHPmyAdmin for quick kabootr DB access
• Report should have node link, list name, total count, bounce count, spam count,
• User login blocking for all non admin users
• HTML mails
• Need to talk to SNS

Results After Testing Of Kabootr

From the entire process of testing kabootr, create notes on what we need to work on next for DCD release of Kabootr.

reference : innoraft/kabootr #36

Initially all imported users must subscribed to Master Newsletter.

1. When we enable simplenewsletter module, it created Master newsletter category.
   This newsletter gets name from by default site name.
2. So when we enable our kabootar project. all users existing or newly imported must be subscribe by default to this Master newsletter category.

Install PHPmyAdmin for quick kabootr DB access

Url of Amazon SNS is static, but it should be dynamic.

@sumeetpareek Url of amaozn SNS end point that will be put into Amazon SNS topic.

[site-url]/kabootr_end_point.php.

In our new AAP server it would be
http://messenger2.aamaadmiparty.org/kabootr_end_point.php.

Delete all the users from the aap-kabootr except users having uid =0 & 1

After deleting all the users, create a list of volunteers from the aap sandy messanger, import them to app kabootr

Automatic subscribe and unsubscribe the user to master newsletter category [Low Priority]

• When a user get registered/unregistered in site, that use should be auto subscribed/unsubscribed to Master newsletter category.

Research mime mail, html mail and smtp modules to understand outgoing mail approach.

Right now we have modules with functions to be able to talk to Amazon AWS SES which call call functions and prepare and send mails via SES. Currently, for sending out simplenews module's content a separate custom module with it's cron implementation is created. Where on cron run things are done to see what simplenews needs to go out and then mails are sent via SES.

Now, a lot of this custom functionality is already present in simplenews module, which makes us wonder if there are better ways of sending out mails via SES such that existing code in simplenews module does what is already does, and only the outgoing mail part is handled by us.

A good stop for this is to research the below modules that add some kind of support for outgoing mails -

1. mime mail
2. html mail
3. smtp

Acceptance criteria = A google doc with useful observations of the module that are relevant to what we are trying to do. Questions and suggestions listed separately. Followed by a tech meeting to take the final decision.

(stub) We should be able to send a campaign to multiple lists.

(stub) We should be able to send a campaign to multiple lists.

Currently only 1 list per campaign is allowed, I think, but I am not sure.

If it is not feasible to finish before DCD release we should see if cloning node temporarily is a good option. In cloning and sending to multiple lists we need to check how same emails in multiple lists are handled?

Branch push to incorrect remote. Notes on how to push to the right one.

@gauravgoyal You seem to be pushing branches to Kuldeep's fork and not to the parent repository. See image below. To fix this, you can git remote rm KULDEEP REMOTE and then do a git remote add NAME(TYPICALLY=ORIGIN) PARENT-REPO

@gauravgoyal Can you also explain @tkuldeep that for kabootr we will stop using 'fork and pull' model and use the 'shared repository' model instead. Details of both can be found here - https://help.github.com/articles/using-pull-requests

Features and working of Kabootr version 1.0 before DCD-2013

On fresh installation of kabootr in new machine, following action will be taken by kaboot

1. Install Fresh drupal with

• Simplenews contrib module
• Simplenews User List cusotm module
• Simplenews Amazon SES custom module

1. Create Manually two rule

• When new user is added to drupal, he must be automatically subscribed to Master Newsletter Category created by simplenews contrib module.
• When user is deleted, user should be unsubscribed from Master Newsletter Category.

2. Import users to site.
3. Create User list.
   • In kabootr creation of campaign or creation of simplenews category have same meaning.
   • Alter user interface of creating campaign. Followed by Summet
4. Amazon SES setting
   • AWS Access keys
   • Verify Sender Identity
   • Identity List
   • DKIM Setting Has to be verified
5. Sending Mail
   • Sending mail through Amazon SES
   • Controll sending limit
6. Mail Statics
   • Complain, Bounces of mail
   • Mail is opened or not by client
   • Location form mail is opened

Add new role to kabootr, for secure access of kabootr

Add new role to kabootr, for secure access of kabootr

I will create two new role

1. Kabootr User
2. Kabootr Administrator

• Kabootr User: It can create mail recipient list or campaign.
• Kabootr Administrator: can preform Amazon SES configuration .

For creation of mail recipient list, need user feed importer

In kabootr for importing users from source data, we need feed importer..

Study the records of AAP lists in sandy, and create data fields on kabootr dev accordingly

Export 100-200 records from the AAP lists in sandy, study them, and create user account field list on kabootar dev.

Create User Account fields on the kabootr dev, and test the kabootr with real data.

Before sending the mail, be 100% sure that no mail is sent to the actual user.

reference : innoraft/kabootr #36

Error, when newsletter category is created after subscriber inactivated

On complain or bounces of mail, we inactivating the subscriber, So it showing error when we create new newsletter category or update existing newsletter category.

Dev server clean up

• change the admin password to something safe and share with team.
• more things to be added, keep the issue open

Setup real domains for kabootr development and testing instead of /etc/hosts entries

If we just use the /etc/hosts entries we will not be able to test Kabootr and SES talking back to each other. We need to setup real domain names as noted in this comment.

• real subdomain from IR account - dev-kabootr.innoraft.com and site should work from this.
• real subdomain from IR account - stage-kabootr.innoraft.com and site should work from this.
• no HTTP AUTH should be setup
• admin user/pass change to something safe and share with team@
• prevent google from indexing full site by editing robots.txt

The reports page is visible to ANON user. Should not be.

The reports page is visible to ANON user. Should not be.

This needs to be deployed on AAP too.

DCD Release - Placeholder issue for project management notes.

Placeholder issue for project management notes. I am not even sure at this point, what stuff I will push in here :-p

But I will keep this issue active only till the DCD Release.