cashapp/backfila

Service that manages backfill state, calling into other services to do batched work

backfila's Issues

Add a configurable button to view logs of a backfill

Add service name to slack notifications.

May want regular file integrity checks during a backfill for the s3 client.

There are several assumptions around the files not changing out from under the s3 client. We might want to check this for a better customer experience.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

We probably need some kind of side channel data for this. The prepare can pull this information perhaps?

Create / prepare backfill should propagate exception from client service

We currently just display a "500" string instead of a useful exception. We should propagate the exception in the same way we do for RunBatchResponse by having a stack_trace string

set up circleci

blocked on getting a new misk release

Add option to create signalfx events when a backfill starts or stops.

Setting custom parameters to null vs empty is confusing

If a value is ever touched it will use empty string rather than null as the value. There is no way to unset it afaik either.

Not quite sure what we should do here. We could have a null checkbox that starts out checked. But that seems like overkill.

Another option is to treat empty as null but this isn't great either. Maybe someone else has a better idea.

Add way to add signalfx events for graph annotations

Group backfills by class for easier viewing

Dyanmo streams exposing sharding might help develop a better Dynamo Backfill

Maybe by leveraging Kinesis we can assume better long term sharding support:
https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

Fix navbar not showing staging environment banner

Sharded misk-client

UI improvements

• show yellow progress bar when backing off
• to service page, add "my backfills" section that shows your last 5 created backfills
  -show actual instantaneous # of active threads - can just read the VariableBuffer size
  -show runbatch latency, suggest smaller batch size if above 15s

Adding metrics to the Backfill page for easy analysis

fix development app injection errors

support pipelining data from one service to another

• web ui to select service/backfill to pipeline to, filters based on data types

Reflection for Java Parameter record classes is broken

class kotlin.reflect.jvm.internal.calls.CallerImpl$FieldGetter cannot access a member of class com.squareup.fidelius.server.backfill.backfila.dynamo.Parameters with modifiers "private final" 

To support record classes, we need to change the reflection strategy to use RecordComponents if the javaClass isRecord().

Better load sharing of backfill threads

Currently a backfills with a long wait between batches will hold onto processing power and reduce CPU so the autoscaler won't scale up.

Probably need to either come up with a way for the autoscaler to look at threads too or a way to process more evenly.

audit log

Change Instance(s) to cursor(s)

I'm finding several people are getting confused on what an instance is. It is a term that is already pretty well overloaded.

I think a better name might be cursor.

Make Backfila Guice make sure the admin console is installed.

Potentially investigate adding some copy around being careful for downstream dependencies.

Fix CircleCI MySql issue

Tests failing with invalid default value for 'lease_expires_at'

Terminated state for incomplete Backfills that should never be reattempted

Turn mysql strict mode back on for CircleCI

I had to turn it off so the tests pass in CI:
#21

Timing everywhere

Should add scan and batch run times to the UI and add times to various logs client and backfila side.

Be able to split a partition into multiple partitions over parts of the range.

This can only happen on a stopped backfill.

Add old backfill nuisance messenger

When an old backfill has been sitting around for a configured period. If it isn't in an allowlist then increasingly bother the creators to remove it.

do permission checks

Backfila times out and retries batches after 1 minute

Conversation here: https://square.slack.com/archives/CET1PJ8NP/p1600206743294600?thread_ts=1600192471.251500&cid=CET1PJ8NP

Backfila is unexpectedly retrying immediately on connection failure rather than respecting the backoff schedule. We are also currently seeing a repeatable connection failure after 1 minute

okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL

While backfill logic is largely idempotent, for large batches, a parallel batch on the same ID range can adversely performance since there'll be a nice bunch of extra lock waits.

We need to dig into what in Backfila's stack that's doing the retry on connection reset. Separately, why the reset is happening at all, though that's probably outside of backfila.

Ability to add notes to backfill runs

It'd be nice to be able to drop a note on a job to the effect of "Next time we run this cut the batch size by 50% because we're blowing up Thing X"

Similarly could use it to centralize findings, plans for future runs, sick recipes, whatever

normalize parameters into a table

Slack notifications should say what environment the backfill is running in

Encourage transaction per batch for the misk client.

Service list page should annotate somehow the services that are currently running backfills

Could custom parameters be mutable?

Requiring custom parameters to be immutable means creating a lot of extra backfills in some cases. Would be great if they were mutable like the batch size, scan size, etc.

Long waits don't hold onto the lease long enough

With a long enough wait between batches the lease will expire too early.

Automatically adjust the batch size based on actual latency and a target

Rather than batch size being set, have backfila adjust it over time.

show progress on all backfills on service page

Add misk client for dynamo

Totally doable with configurable number of instances:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html

validate() errors don't show up

Add a button to clone and tweak a backfill

Would be great for:

1. Going from dry run to wet run
2. Changing the range or other immutable properties

Support backfill parameters in the Misk Client

update db less often on completed batches

Automatic Backpressure

When registering on service startup also register ways to detect load so backfila can automatically adjust backfill speed based on this data.