The citizenos-api from citizenos

BUG: API - replying to deleted (hidden) argument throws 500 Internal Server Error

Reproduce

Create a new argument
Delete the argument
Reply to the argument

Result

500 Server Error on posting the reply

Expected

TODO

Disable "Reply/Report/Up-Downvote" buttons for hidden arguments
Add a "disable reason" message IF possible
Citizen OS API to return sensible HTTP code and not throw 500

Activity feed - filter

Enable filtering by:

activity type (vote etc)
topicId

The need - for example rahvaalgatus.ee wants User to see when and if he/she has voted.

Related tasks:

rahvaalgatus/rahvaalgatus#17

CLEANUP: Remove API routes that are not used

TODO

NOTE: Verify usage against application logs!

REMOVE API: GET /api/users/:userId/groups/:groupId/topics, which is replaced by GET /api/users/:userId/groups/:groupId/members/topics
REMOVE API ROUTE: GET/POST /api/users/:userId/groups/:groupId/members which is replaced by /api/users/:userId/groups/:groupId/members/users
REMOVE API ROUTE: PUT/DELETE /api/users/:userId/groups/:groupId/members/:userId which is replaced by /api/users/:userId/groups/:groupId/members/users/:userId

Logging - unique request ID to each log line wherever the line is logged

Overview

To ease debugging an unique ID should be added to each line logged in the request context no matter how deep in the stack.

The problem is that Node.JS being async, it is quite a difficult to carry on the id between context switches. It does not make sense to pass req object or the id to each function call either.

Possible solutions

https://nodejs.org/api/async_hooks.html

Outdated:

TODO

Each log line to contain unique request ID
Same unique request ID should be visible in the HTTP response headers

Report a Topic

There is a way to report an argument (comment) but not a whole Topic.

TODO

BDOC - Review BDOC generation, get rid of disk cache if possible.

Background

In the beginning of times when I had no idea what I was doing, I wrote the BDOC generation code. Among other things it streams zip to disk and then pipes to response. Which is an interesting idea, why not just pipe to response? File cache just makes maintenance and code more difficult.

TODO

Review code - https://github.com/citizenos/citizenos-api/blob/master/libs/cosBdoc.js
Is the file cache needed?
IF not, get rid of it.
Rewrite code to be more readable, reusable.

Vote delegation - enable delegation after User has voted

Overview

When an User has voted, the possibility to delegate the vote is disabled. As User can re-vote n+1 times, maybe it makes sense to enable delegating after voting?

TODO

Try to implement enabling delegation after voting, find out if it's even possible.

DOC: Define & describe citizenos-api versioning and upgrade/downgrade process

Make a Wiki post how the citizenos-api versioning works. Also how to upgrade and downgrade.

Don't limit Topic title to 100 chars

Parse and store whole title, let client decide what to do.

Related tasks:

rahvaalgatus/rahvaalgatus#58

TODO

Rise the limit 1000 chars in the DB and in the API code make the limit of title configurable.
Fix the character counting code so that it counts characters not bytes - https://app.citizenos.com/en/topics/f8f8bea2-362c-46c1-bae5-9d3bf8a83300

/api/topics throws 500 given an unrecognized status

This should respond with something in the order of 4xx.

http https://api.citizenos.com/api/topics limit==400 sourcePartnerId==b563ee8c-ba8e-4cd6-b592-fbcd4e8f22bb "statuses[]==closed" "statuses[]==voting" "statuses[]==discussion"

Missing 7z executable crashes CitizenOS API on container download

Seems that a missing 7z executable crashes the app on container download:

Error: spawn 7z ENOENT
    at exports._errnoException (util.js:1020:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:197:32)
    at onErrorNT (internal/child_process.js:376:16)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickDomainCallback (internal/process/next_tick.js:128:9)

While there is a 7z exe in the bin folder, using it requires remembering to set the PATH before starting the server. Crashing, however, is probably not a good outcome in any situation.

API: Topic attachment's original name becomes alphabet soup when downloaded

Current situation:
Downloaded topic attachment loses its original name, turns from descriptive to random characters and numbers.

Expected behavior:
Downloaded topic attachment retains its original file name.

Reproduce:

Give some file a proper name with words and all (Team meeting memo_November.docx in my case)
Upload the file as an attachment to a topic
Download the attached file from within the topic. Downloaded file's name becomes a collection of random characters (057ada2b-56c4-4798-8d0a-7371245eb3a5.docx in my case)

Voting - creates multiple VoteList lines and is not idempotent

Overview

Voting in general & mobiil-ID voting - GET /api/users/:userId/topics/:topicId/votes/:voteId/status, which is used for polling signature state, creates multiple VoteList lines and is not idempotent which is a bad design.

There are several things questionable about this voting endpoints:

By the REST standard GET should be safe (https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Safe_methods) and idempotent (https://en.wikipedia.org/wiki/Idempotence) BUT the GET /status is neither. It adds User entry on successful signing and also multiple VoteList lines if called several times AFTER success. NOTE: It does NOT affect the vote results.
/status used for polling the Mobiil-ID signing status. Why not do the polling server-to-server in the background which would enable the person to close the window after initiating the signing? That would make the calling /status optional. Right now it's required to register a vote, if something happens and the polling fails, the vote is never registered.
I used to create multiple VoteList lines to resolve disputes over "how many times I actually voted" and "I did not vote like this" cases. But after we created Activity Feed, those cases can be solved by looking at the feed.

Events - add support for type and createdAt

Events - add support for eventType and createdAt.

New fields:

createdAt (or similar) that can be modified via API
type - free text, partner classificator

GET /topics/:topicId - include=events - ideally also provide in what detail you want the event object.

Related tasks:

rahvaalgatus/rahvaalgatus#63

BUG: E-mails - images not visible in some e-mail clients due to incorrect Content-Type

Overview

Thunderbird/Postbox off the top of my head, but there could/should be others that will not show inline images when their content type is not image/*.

Currently the headers of an attachment are:

Content-Transfer-Encoding: base64
Content-Id: <logo>
Content-Disposition: inline; filename="logo"
Content-Type: application/octet-stream; name="logo"

Research

This one is strange:

We send image/png to Mailgun, but it ends up application/octet-stream in the e-mail.

POST /gmail.com/messages { from: 'CitizenOs Dev <[email protected]>',
  to: [ '[email protected]' ],
  subject: 'Activate your CitizenOS account - to: [email protected]',
  html: '\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">\n<html>\n    <head>\n        <meta content="text/html; charset=utf-8" http-equiv="Content-Type">\n    </head>\n\n    <body style="margin: 0;mso-line-height-rule: exactly;padding: 0;min-width: 100%;background-color: #fbfbfb">\n        <table border="0" cellpadding="0" cellspacing="0" id="backgroundTable" style="margin: 0;padding: 0;width: 100% !important;height: 100% !important;line-height: 100% !important;background-color: #f6f6f6;font-size: 14px;border-collapse: collapse;mso-table-lspace: 0;mso-table-rspace: 0">\n            <tbody>\n    <tr>\n        <td align="center" height="70" style="border-collapse: collapse" valign="middle">\n            <!--p style="font-family: arial, helvetica, sans-serif;font-size:12px;background-color: #f6f6f6; color:#828282;">Can\'t see this mail? <a href="#" target="_blank" style=" color:#828282; text-decoration:underline;font-family: arial, helvetica, sans-serif;font-size:12px;">Click here to view it in your browser</a></p-->\n        </td>\n    </tr>\n\n    <tr>\n        <td align="center" valign="top">\n            <table border="0" cellpadding="0" cellspacing="0" id="templateContainer" style="background-color: #FFFFFF;margin: 0;padding: 0;" width="600">\n                <tbody>\n                    <tr>\n                        <td align="center" valign="middle" style="background-color: #252525; border-bottom: 1px solid #e1e1e1; " height="100">\n                            <img width="237" height="43" src="cid:logo" alt="logo" style="display:block;" border="0">\n                        </td>\n                    </tr>\n\n                    <tr>\n                        <td align="center" valign="top">\n                            <table border="0" cellpadding="25" cellspacing="0" id="templateBody" width="600" style="background-color: #FFFFFF;margin: 0;padding: 0;">\n                                <tbody>\n                                    <tr>\n                                        <td class="bodyContent" style="background-color: #FFFFFF;" valign="top">\n                                            <table border="0" cellpadding="10" cellspacing="0" width="100%">\n                                                <tbody>\n\n                                                    <tr>\n                                                        <td align="center" valign="middle" style="background-color: #FFFFFF;border-bottom: 1px solid #f0f0f0;" height="80">\n                                                            <p style="font-family: georgia, serif;font-size:19px; font-weight:bold;background-color: #FFFFFF; color:#252525;">Hi Test 1496746632256!</p>\n                                                        </td>\n                                                    </tr>\n\n                                                     <tr>\n                                                        <td align="center" valign="middle" style="background-color: #FFFFFF;border-bottom: 1px solid #f0f0f0; border-top: 1px solid #fbfbfb; " height="130">\n                                                            <p style="font-family: arial, helvetica, sans-serif;font-size:14px;background-color: #FFFFFF; color:#252525;">To activate your account please verify your e-mail address by clicking the link below.</p>\n                                                            <p style="font-family: georgia, serif;font-size:18px; font-weight: bold; font-style: italic ;background-color: #FFFFFF; color:#0680fc;"><a href="https:&#x2F;&#x2F;dev.toru.ee:3001&#x2F;api&#x2F;auth&#x2F;verify&#x2F;bdbd054f-3d5e-43df-a310-6c1b8aed9cf5?token&#x3D;eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJyZWRpcmVjdFN1Y2Nlc3MiOiJodHRwczovL2Rldi50b3J1LmVlOjMwMDEiLCJpYXQiOjE0OTY3NDY2MzJ9.i3rQ3eeZXR8OOU2T8oMaITABKBd0DIrZFwS_G_PlFo30agJhZqFUpVxNOicUmP8T076OSUK8rOW3Jt7CmgvUc0W8uvksxtde86_joOtQGHMwtW6xp4c2uinkKE2Wj_BYMC-tcqFsKPywD5O0un5adbaP2SBbgan-5oOFG8n1r-7wb7g_KI8vtgI_gaLNtTD0DBnEaVcDeFi2CKUkF13wPaKZy-OmduWSXhVrr18lyaAvDCGj2xTknSUtyoygXuxtPJQbZ3O1gCxvM-TUNu__dZL0-0Xo6ScR4v4aD4BuHdzLBPIfHObe-6HKKEkLeBopUpos_BOT0ILnDUUJ_87eHQ" target="_blank">Verify my email</a></p>\n                                                        </td>\n                                                    </tr>\n\n                                                    <tr>\n                                                        <td align="left" valign="bottom" style="background-color: #FFFFFF;border-top: 1px solid #fbfbfb;" height="120">\n                                                            <p style="font-family: arial, helvetica, sans-serif;font-size:14px;background-color: #FFFFFF; color:#252525;">CitizenOS Team</p>\n                                                        </td>\n                                                    </tr>\n\n                                                </tbody>\n                                            </table>\n                                        </td>\n                                    </tr>\n                                </tbody>\n                            </table>\n                        </td>\n                    </tr>\n\n                </tbody>\n            </table>\n        </td>\n    </tr>\n\n     <tr>\n        <td align="center" valign="top" height="70">\n            <table border="0" cellpadding="0" cellspacing="0" id="templateFooter" style="background-color: #f6f6f6;margin: 0;padding: 0;" width="600">\n                <tbody>\n                    <tr>\n                        <td align="center" height="70" style="border-collapse: collapse" valign="middle">\n                            <p style="font-family: arial, helvetica, sans-serif;font-size:12px;background-color: #f6f6f6; color:#828282;">Learn more on <a href="https:&#x2F;&#x2F;dev.toru.ee:3001" target="_blank" style=" color:#828282; text-decoration:underline;">https:&#x2F;&#x2F;dev.toru.ee:3001</a></p>\n                        </td>\n                    </tr>\n                </tbody>\n            </table>\n        </td>\n    </tr>\n     <tr>\n         <td align="center" valign="top" height="100%">\n             <br>\n         </td>\n     </tr>\n\n</tbody>\n                <div style="margin-top: 20px; padding: 10px; border: 3px solid red; background-color: #ffd2d2; color: #384c53;">\n                    <div>\n                        <b>Recipients</b>\n                    </div>\n                    <pre>\n                        <code>{\n  &quot;to&quot;: [\n    &quot;[email protected]&quot;\n  ],\n  &quot;merge&quot;: {}\n}</code>\n                    </pre>\n                </div>\n        </table>\n    </body>\n</html>',
  text: 'logo [cid:logo]Hi Test 1496746632256!\n\nTo activate your account please verify your e-mail address by clicking the link below.\n\nVerify my email\n[https://dev.toru.ee:3001/api/auth/verify/bdbd054f-3d5e-43df-a310-6c1b8aed9cf5?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJyZWRpcmVjdFN1Y2Nlc3MiOiJodHRwczovL2Rldi50b3J1LmVlOjMwMDEiLCJpYXQiOjE0OTY3NDY2MzJ9.i3rQ3eeZXR8OOU2T8oMaITABKBd0DIrZFwS_G_PlFo30agJhZqFUpVxNOicUmP8T076OSUK8rOW3Jt7CmgvUc0W8uvksxtde86_joOtQGHMwtW6xp4c2uinkKE2Wj_BYMC-tcqFsKPywD5O0un5adbaP2SBbgan-5oOFG8n1r-7wb7g_KI8vtgI_gaLNtTD0DBnEaVcDeFi2CKUkF13wPaKZy-OmduWSXhVrr18lyaAvDCGj2xTknSUtyoygXuxtPJQbZ3O1gCxvM-TUNu__dZL0-0Xo6ScR4v4aD4BuHdzLBPIfHObe-6HKKEkLeBopUpos_BOT0ILnDUUJ_87eHQ]\n\nCitizenOS Team\n\nLearn more on https://dev.toru.ee:3001 [https://dev.toru.ee:3001]\n\n\nRecipients{\n  "to": [\n    "[email protected]"\n  ],\n  "merge": {}\n}',
  inline: 
   [ Attachment {
       data: <Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00 ed 00 00 00 2b 08 06 00 00 00 94 fc 0f 26 00 00 00 19 74 45 58 74 53 6f 66 74 77 61 72 65 00 ... >,
       filename: 'logo',
       contentType: 'image/png',
       knownLength: undefined } ],
  'o:tag': [ '(dynamic)' ],
  'o:tracking': true,
  'o:tracking-clicks': true,
  'o:tracking-opens': true,
  'recipient-variables': {} }

TODO

Make sure right content type is sent and images are shown correctly

E-mails - abandon the "fire and forget" method of sending e-mails

Overview

Somehow I thought it was a good idea to send emails in a form of "fire and forget", but that is a bad idea cause User needs feedback if e-mail was successfully sent. Specially password reset, account creation etc.

TODO

Go over places where e-mails are sent, remove "fire and forget" unless it actually makes sense where present.

API POST /api/auth/id - rewrite the code so that it's less likely to accidentially trust certificate when not supposed to

As per comments - 7b95c9e#diff-d12ae772f9826d4096efd4221a1874c9R572

Mobiil-ID auth - SPChallenge validation

Overview

When executing authentication, it's a best practice to generate a SPChallenge and validate it on successful authentication - http://sk-eid.github.io/dds-documentation/api/api_docs/#mobileauthenticate.

Related API routes:

POST /api/auth/mobile/init
GET /api/auth/mobile/status

TODO

Implement SPChallenge generation and validation for MID authentication
Review Mobiil-ID signing process, see if there is similar concept to SPChallenge. IF there is, make sure it's used.

DOCS: Update README - DB creation changed

Migrations introduced new way of creating the DB

#1

API: Unify restricted use token generation and validation

Overview

We issue restricted use JWT tokens for very specific authorization:

We issue a token to Topic Moderators, to moderate a Topic. This token is sent in e-mail.
We issue a token to User after singing a Vote so that signed container can be downloaded. Not all Users have to be logged in for signing thus enables unauthenticated download of their Vote.
We issue a token to Riigikogu (Parliament) so that they can add updates to Events (follow-up phase)
We issue a token to Riigikogu (Parliament) so that they can download signed Topic container.
...

Problem

The restricted use token format varies quite a bit in the system and creates significant amount of complexity/confusion/overhead in the code-base. It calls for a rewrite so that we can reuse the issuing and validation code.

What token formats are present, where are they issued?

Token usually contains JWT standard parts + Citizen OS extras. The variance is in the Citizen OS extras. For example path (string), paths (Array) where there may or may not be req.method prefix.

Different path/method format used:here that format is used.

path - string - Path, NO request method (GET, POST..) limitation. Example: "path": "/api/users/self/topics/:topicId/votes/:voteId/downloads/bdocs/final"
- getBdocUrl - https://github.com/citizenos/citizenos-api/blob/master/routes/api/topic.js#L461
- getZipURL - https://github.com/citizenos/citizenos-api/blob/master/routes/api/topic.js#L502
- _topicUpdate - https://github.com/citizenos/citizenos-api/blob/master/routes/api/topic.js#L1443
paths - Array - Array of paths with REST methods (GET, POST...). Example: "paths": ["POST_/api/...", "GET_/api/..."]
- sendCommentReport - https://github.com/citizenos/citizenos-api/blob/master/libs/email.js#L570

Proposed solution

Rewrite all restricted token issuing code to use audience claim (aud) to specify the scope of use:
- aud (was path/paths) - Array - with REST method everywhere. Separated by space instead of "_". For example: "aud": ["GET /api/users/self/topics/:topicId/votes/:voteId/downloads/bdocs/final"]
Create a new middleware to validate these tokens. Call it authTokenRestrictedUse.
- NOTE: MUST be backward compatible (path vs paths, vs scope) with formats above. Make a comment that other formats are deprecated so that if we know that it's unlikely for an old token to be used, we can delete the backward compatibility.

Subscribe to e-mails for specific activities

I as a User may want to get email notification for specific activities.
I as a User want to set these notifications on global and entity (Topic, Group) level.

For example:

New comment added
Topic was edited
Comment was reported

Related tasks:

rahvaalgatus/rahvaalgatus#8

BUG: Final BDOC generation is slow resulting in request timeouts

Overview

For in case of significant amount of signatures the final BDOC generation times out.

25 Oct 2017 15:39:22.5101048 <158>1 2017-10-25T12:39:21.942192+00:00 host heroku router - at=error code=H12 desc="Request timeout" method=GET path="/api/topics/78ba564c-9843-4dc1-8e79-dcad2941537c/votes/6f0ef16d-067c-4034-be36-9ab32dcda9f0/downloads/bdocs/final?token=x&accept=application%2Fx-7z-compressed" host=api.citizenos.com request_id=68252540-6ca7-4466-88ea-1c294c9ea8f3 fwd="x" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https

TODO

Speed up the generation, there is a test case to play around - https://github.com/citizenos/citizenos-api/blob/master/test/libs/cosBdoc.js#L164

Emails - Mailgun delivery is sometimes way too slow

Overview

Sometimes Mailgun delivers e-mails too slowly.
It took 8 min from accepting to delivery for an email below:

10/19/16 08:35 AM Delivered: [email protected] → [email protected] 'Rahvaalgatus.ee kasutaja John Smith kutsub Sind kooslooma arutelu/algatust' 
 10/19/16 08:27 AM Accepted: [email protected] → [email protected] 'Rahvaalgatus.ee kasutaja John Smith kutsub Sind kooslooma arutelu/algatust'

It is way too long for password reset and signup emails.

Reading

TODO

Pinpoint and fix the delivery lag.

Stronger password encryption - bcrypt instead of SHA256

Overview

SHA256-s time is over and a person with an access to Citizen OS database can figure out the actual password with a reasonable effort.

TODO

Use bcrypt OR something else. There is a whitepaper on the subject by RIA - https://www.ria.ee/sites/default/files/content-editors/publikatsioonid/cryptoreport2021.pdf
Figure out and implement a User friendly way to migrate the passwords from SHA256 to bcrypt.
- IF there are not many Users that have used user/password authentication, is to just random generate new passwords for them which will force them to "Forgot password" flow and thus generating a new bcrypt in the DB. IF allowed, we can e-mail Users that we did it. The solution is the easisest and does not clutter the code.
- Polite way, if many Users, is to swap the password on next login - when password is verified against SHA256 use bcrypt to hash the password and store it to DB. The problem with this solution is that we have to support parallel hashes for a while.

Social mentions - investigate possibility to move to Google/Bing search API

Overview

Our Social Mentions feature currently only shows results from Twitter. It is so because FB, Twitter and other social platforms protect their data - no or very limited search by hashtag.
I looked into using Google search and found out that if you use the advanced search, we can compile a search that returns results from all major platforms. But we dropped the investigation as the price of Google Search API usage seemed a bit high at the time.

TODO

Revive the reseach, see if we could get adequate results from Google/Bing search API searching by hashtag and maybe Topic title. Search result should cover Twitter and Facebook. Instagram is a bonus.
Record the research outcome here - what was tried, what worked, what did not work etc.
IF we find a way, implement it.

API: Activity feed - "Voting ends soon" activity

Overview

We want to raise vote participation by reminding Users that a vote is about to end.

TODO

Generate "Vote ends" activity.
Implement relevant FE parts.

GET /topics/:topicsId/members/user - return contact info

As a Topic author I would like to know what exact User is a member of a Topic.
For this I need to see the e-mail that was used for the invite.

Problems:

If another admin invites a User, that exposes the e-mail to all other admin users.

Related issues:

rahvaalgatus/rahvaalgatus#1

BUG: Base64 images crash DOCX generation

Base64 images in EP documents crash DOCX generation.

How do base64 images end up in EP documents?

Activity feed - add argument moderation activity

Overview

When a Moderator moderates an argument, there is no info about it in the activity feed.

You can read how moderation works in Citizen OS from here - https://github.com/citizenos/citizenos-api/wiki/Content-moderation

TODO

Record moderation activity
Verify that the activity is shown in the */activities outputs.
Verify that it's shown properly in the FE (https://github.com/citizenos/citizenos-fe). Make tasks needed for localisation.

Slug support - fetch Topic by slug

Human readable slug support needed in API because:

Emails need to have urls with slugs

API will still use the system ID, but way may need to move to BIGINT id-s from UUID because the "slug+ID" would look really long and awkward.

Related tasks:

citizenos/citizenos-fe#72 - FE work
rahvaalgatus/rahvaalgatus#15 - feature request

Authentication: Usage of FB / Google profile e-mail as a basis for logging in

Overview

Citizen OS API right now uses e-mail returned by FB and Google from their respective OAuth login APIs as a unique ID from the source and the basis of logging in to Citizen OS.
We assume, that FB and Google are good at verifying e-mails and we can trust the e-mail address. There is no extra verification of the address.
It may make sense to use Google / FB user ID instead as the unique.

TODO

Gather PROs and CONs of using either of the 2 as unique identificator - e-mail OR FB/Google User ID.

SECURITY: API PUT /users/self - verify e-mail on update - read the description please

Overview

When User updates e-mail in the profile, it does not get verified.

PANIC!? I don't think so, but definitely not nice.

What vector does it open up?

User A has never used the application, has no account with his e-mail.
User B registers with an e-mail belonging to him (B), doing the full verification flow.
User B updates his e-mail to e-mail belonging to A, NO verification.
User A:
4.1 Logs in with FB/Google that has the e-mail in the profile, User A sees whatever malicious user B has done in the account. If A changes the password, B is lock out and cannot access it any more.
4.2 Tries to create an account with e-mail/password and gets an error "e-mail already in use". He is confused and goes for the "Forgot password flow" after which A verifies the ownership of e-mail and gets the account locking malicious user B out.

Credits

@moll who brought this to our attention

Related code:

https://github.com/citizenos/citizenos-api/blame/37036b1e1f024d6d9fedb8aa00a010ae08892d39/routes/api/user.js#L16

TODO

Create a flow to verify every new e-mails ownership

E-mails: Drop usage of campaign-mailgun, use SMTP instead

Overview

campaign-mailgun uses Mailgun API, but Mailgun can be used over SMTP so it provides no extra value.
Rather, it provides extra problems - the Campaign switches on e-mail tracking which rewrites links in the e-mails.

TODO

Drop usage of campaign-mailgun
Configure servers to use campaign over SMTP

API: Activities - triple "upvote" events in activity feed

Overview

Activity fees shows 1 upvote 3 times.

Reproduce

Go to public Topic - https://app.citizenos.com/en/topics/3f3c38af-6e3b-471a-9d95-2030a632bcf6
Log in
Up/downvote an argument

Result

Activity feed contains several events per 1 up/downvote

TODO

Fix activities API to store and return upvote event once

API: POST /api/users/:userId/topics/:topicId/votes/:voteId - logs "Can't set headers after they are sent."

Overview

Calling POST /api/users/:userId/topics/:topicId/votes/:voteId logs Can't set headers after they are sent. on error. There is probably some error in the Promise chain.

NOTE: Does not break functionality, but shows bad coding.

TODO

Test the error conditions where the error is logged and fix them. Use case "User is not a Mobile Id client" should be able to reproduce the issue.

POST /api/auth/id - Support passing client certificate in the header

POST /api/auth/id - Support passing client certificate in the header

The need:

Deployments where you can configure client certificate in proxy would be saved from installing extra service https://github.com/citizenos/id-auth

Implementation notes:

Certificate in X-SSL-Client-Cert header
Add readme about Nginx client certificate configuration

DDSClient - Error subclassing done wrong resulting in non informative errors

Background

The way DDSClient subclasses error results in non informative errors.

The need

We want DDSClient errors to contain stack trace and other useful info.

TODO

There are 2 ways:

Use existing StandardError and subclass that - https://github.com/moll/js-standard-error
Investigate better way to do it and implement our own solution

BONUS POINTS

Update DDS documentation links in the code to relevant parts in http://sk-eid.github.io/dds-documentation/

API: GET /api/activities - DB out of memory errors on high load

Reproduce

ab -n 1000 -c 10 "http://citizenos-citizenos-api-prod.herokuapp.com/api/activities"

Result

05 Sep 2018 11:38:47.607226 <190>1 2018-09-05T08:38:46.917613+00:00 host app web.2 - �[31m[2018-09-05T08:38:46.912] [ERROR] production - �[39mEndpoint GET "/api/activities" failed miserably. Status: undefined Stack: SequelizeDatabaseError: out of memory
05 Sep 2018 11:38:47.60781 <190>1 2018-09-05T08:38:46.903537+00:00 host app web.2 - severity: 'ERROR',
05 Sep 2018 11:38:47.60681 <190>1 2018-09-05T08:38:46.903458+00:00 host app web.2 - severity: 'ERROR',
05 Sep 2018 11:38:47.476123 <132>1 2018-09-05T08:38:46+00:00 host app postgres.2097 - [DATABASE] [135-1] sql_error_code = 53200 ERROR: out of memory
05 Sep 2018 11:38:47.476122 <132>1 2018-09-05T08:38:46+00:00 host app postgres.2088 - [DATABASE] [75-1] sql_error_code = 53200 ERROR: out of memory
05 Sep 2018 11:38:47.273123 <132>1 2018-09-05T08:38:46+00:00 host app postgres.2112 - [DATABASE] [106-1] sql_error_code = 53200 ERROR: out of memor

Reading

Public note for Topic

Support public note for Topic so that people can insert extra public info.

Tech:

1 note per Topic.
Free text

Related tasks:

rahvaalgatus/rahvaalgatus#54

NPM shrinkwrap

Hey,

For both security and reproducible installations, please add a npm-shrinkwrap.json file with known-to-be-working dependency versions (generate it with npm shrinkwrap). Having versions in package.json is insufficient as nested dependencies still get upgraded. Having versions in package.json given npm-shrinkwrap.json is then redundant and unnecessary.

E-mails - review e-mail templating system

Overview

Partners want their own email layouts and texts, current system is quite bad in that.

Related to:

#26

TODO

Find a better e-mail templating solution taking into account:

Partner wants to be able to override the template designs
Partner wants to override the template texts
Possible need for "click here to view on the web"
~~Localisation with Crowdin~~ #41

E-mails: Configurable logos - update e-mail layout so that Citizen OS logo is always present even when custom logo is used

Overview

One can configure their own logo for the e-mails (https://github.com/citizenos/citizenos-api/blob/master/config/emails/README.md).

Problem

There is no Citizen OS branding in the e-mail, if a custom logo is used.
Why? Help spread spread the word of Citizen OS OSS existence.

TODO

Implement the e-mail layout as per https://projects.invisionapp.com/d/main#/console/9829159/318396373/preview

API: ID-card authentication and signing - support newly issued ID cards

Overview

Starting 2019 there will be new ID-cards issued by a new company.
It changes the information format returned by the cards.

Reading:

https://www.id.ee/38674

TODO:

Crunch through the changes, compile a list for additional TODO-s - https://www.id.ee/38674
Order new test ID-card

API: `POST /api/auth/password/reset` - passwordResetCode never expires and can be used until new code is generated using `POST /api/auth/password/reset/send`

Overview

POST /api/auth/password/reset - passwordResetCode never expires and can be used until new code is generated using POST /api/auth/password/reset/send

TODO:

passwordResetCode to have an expiry (for ex 1hr)
After successful reset delete the passwordResetCode

Permit configuring JWT with symmetric key

Using public and private keys for JWT is a bit overkill right now. One could just configure JWT to use symmetric methods, but app.js explicitly prevents that through https://github.com/citizenos/citizenos-api/blob/master/app.js#L205. I propose that limitation be removed.

MID/ID-card/SmartID authentication and signing - stop updating User name.

Related issue:

rahvaalgatus/rahvaalgatus#85

E-mails - localisation with Crowdin

Related to:

#6

TODO

Move templates to Citizen OS API Crowdin project - https://crowdin.com/project/citizen-os-api
While on it, also translate Etherpad templates
Make sure governmentNotification.mu is NOT translated and is in ET by default.
E-mail testing - think how the testing should be performed? Implement. Right after running all tests I just grep "e-mail" | grep -c "ERROR" and expect 0 which means e-mail sending code worked, but I dont check the payloads that were sent. The possible problems that this kind of testing does not catch are wrong language, wrong partner, wrong template, wrong receiver, wrong subject etc.

Tech dept: Remove `parentId` from Group model

Overview

Group model has parentId in the model definition which is not used - https://github.com/citizenos/citizenos-api/blob/master/db/models/Group.js#L32
This is probably a copy-paste mishap.

TODO

Remove parentId from Group model - https://github.com/citizenos/citizenos-api/blob/master/db/models/Group.js#L32

API: `PUT /topics/:topicId/members/users` and `PUT /groups/:groupId/members/users` to return HTTP 422 if any of the `userId`-s is not a valid email or UUID.

Overview

Initial implementation of PUT /topics/:topicId/members/users and PUT /groups/:groupId/members/users just ignore invalid UUID-s and e-mails. The shortcoming of this is that if you insert invalid e-mail/UUID, you will never know if the invites actually succeeded.
To provide adequate feedback, he input data should be validated and an error code returned.

TODO

PUT /topics/:topicId/members/users to return HTTP 422 IF any of the userId is invalid e-mail or UUID. Update tests.
PUT /groups/:groupId/members/users to return HTTP 422 IF any of the userId is invalid e-mail or UUID. Update tests.
Once done, create a task OR implement a fix in the Citizen OS FE (https://github.com/citizenos/citizenos-fe) project to update our UI to show the error.

Database [schema] migrations?

Hey!

Thought I'd open up your GitHub issues page with a question. Once the app's database is initialized (which I believe happened on the first run), how do you handle further database changes? Thanks!

citizenos / citizenos-api Goto Github PK

citizenos-api's People

Contributors

Stargazers

Watchers

Forkers

citizenos-api's Issues

Recommend Projects

Recommend Topics

Recommend Org