lusbenjamin/pymoji

✨📸🕶✨

pymoji's Issues

TODOOOOEW

✅ Done!

• replace default emoji with :|
  add levels of joy => :) :] :D
• turn linting back on
• Python3 or python2????
• think about removing google liscence and put MIT one in
• also seriously make a damn folder for your sloppy output
• make shit run on google server
• Ben: something about ENUMS in python
• what's up with max_results=5 ??1
  We need to be able to do groups with >10 people. This seems to have changed with us removing max_results=5 all over the test code
• externalize image files ...somewhere in the sky
• make image processing more efficient so that we don't open emoji images and convert them multiple times in each face loop
• hook up error reporting in production
• if face.headware_likelihood > 1, use 🤠
• exclude file types, handle malicious names
• exclude large file sizes, consider always resampling
• ensure unique filenames
• bugfix selfie MVP rotation problem by exploring EXIF
• fork dev config.py and Google config.py from app.py and constants.py
• 🤖 have the server ping slack when people be uploading their shit
• debounce rage-clickens on the upload button (userbug)
• expose custom run settings?
• turn on bounding box draws in haxxx mode, color code so we can see the score to the bounding box
• auto-resubmit individual crops of "boring" faces, then use Label Annotations for 🤓
• control Z-index based on y height and area of face

☑️ 📝 In Progress

• make 🤠 bigger
• MIDPOINT_BETWEEN_EYES would be a better centroid for the pasteing of the emoji
  (hardcoded: emoji faces are a little too low and eating into necks, hat emoji is too low, could pan/tilt to help)
• Basic auth of some kind:
  • non-anonymous user (HTTP?) OR
  • user OAuth logins from Google, Facebook, et al +fullstory
    • and SSL
    • make sure Stackdriver can get through (uptime checks are keeping it awake)
    • flask plugin would allow this based on regex flask-sslify????
• automatically throttle app to turn off after a certain number of calls
  • set up basic postgres / datastore on Google Cloud compute
  • establish a basic counter, and gate calls hourly and total

☑️ 🐛 Bugs

• diagnose PNG masking issue
• solve session management (race conditions), use anonymous cookies or Oauth (always on?), +fullstory
• handle timeouts (asynchronous, polling, etc)

☑️ 🦋 Features

• fix up the 500 page? add a custom 404
• lading page to show last 5 runs?
• ^re-run button on that page under each run
• add social media sharing links (requires on OAuth)
• add titles/descriptions/annotations (database)
• manually modify an emoji
• re-submit a specific face

☑️ 😎 Emojivision

• diff between faces internally in an image to get more variety
• Retry images where we find no faces by rotating them

☑️ 🤖 Tech

• do health checks properly instead of in a default way? (two-stage check of app)
  https://console.cloud.google.com/compute/healthChecksAdd?project=pymoji-176318
  #13 (comment)
• extend AnnotationsSchema so that we can capture run settings metadata in the JSON file
• Save original image even if it's rotated incorrectly(displaying our fixed image to the user)
• check to see if the exif data is the same as any other image recently uploaded, etc. Don't run images a billion times.
• worry about storage limits
• better uploading library
• betterest uploading library?
• Standardize image file names
• security (SSL, flask-login)???
• more functional tests
• add end-to-end browser testing tool
• move to Docker and Google Container Engine
• write slack integration for stackdriver error reporting
• write a generalized error-and-logging wrapper abstraction so that we can handle and make all exceptions unblocking
• fancy admin-only tools area
• Product discussion for Statefulness, security, database stuff
• React Native vs lots of flask modules? APP STRUCTURE
• create a SPA (React/Node/Webpack)
• python linting to require declaration of vars at top of function
  (because we're only sometimes doing bob = None??? *what are best-practices for python codestyle at larger shops)

☑️ 👩‍🔬👨‍🔬 Research

• hella easter egg: super crop and produce extra image if we find any emoji faces hidden in the background inanimate objects in the image
• get 1000 training images to start training our own
• Google video client

ML Project Structure

Initial Approach

• Continue to use Google Vision API to handle face detection
• Build a new model to handle classifying heads as emoji
  • use Vision API bounding boxes to crop and scale heads
  • input is 128x128x3 array of pixel values (unstructured data)
  • output is Softmax MECE classification

ML Goal

Single Evaluation Metric

• Start with basic accuracy on test set as optimizing metric
• Might add running time satisficing metrics if performance is an issue

Datasets

• Data augmentation likely to be a highly promising area for this project
• consider adding image noise, rotations, distortions (saturation, color-balance, etc)

Iteration

• Going to keep adding notes here
• Intuition suggests that emojivision is a problem with extremely good human-level performance.
• Manual error analysis will be relative quick and highly valuable.
• Getting to "super-human" performance may be really hard (humans ~ Bayes Error)

But What About <Bag O' Fancy Words>???

• Unsupervised learning: probably beyond the scope of the project. Requires more project time, compute resources, and lots of data to start tackling reinforcement learning
• Transfer learning: highly promising technique that might let us extend our initial model to handle new classes of emoji without "starting over from scratch"
• Multi-task Learning: probably beyond the scope of the project. Might be appropriate if we were simultaneously finding food emoji in the background. (This is not the same as a using a Softmax classifier to generalize beyond logistic!)
• CNN: usually a good idea to isolate the non-spatial information in an image. Might be a good idea if our first models are really inaccurate, but might not be necessary after our input processing step (cropping and scaling to night square image of head).
• RNN: stupidly cool sounding, but probably a terrible choice for this problem. Best at handling sequences where "short-term memory" is important. It might be useful if we ever got to optimizing the "end-to-end joke" across all faces in the image.
• End-to-end DL: probably beyond the scope of this project. Usually requires lots of data, and in our case, we'd need to invent a humor score to quantify "good" end-to-end emoji jokes.

Resources

• Coursera Deep Learning Specialization from deeplearning.ai
  • Course 3: Structuring Machine Learning Projects
  • Course 2: Hyperparameter tuning, Regularization and Optimization
• Udacity Deep Learning from Google
• Hotdog or Not-Hotdog Project