cashapp / backfila Goto Github PK
View Code? Open in Web Editor NEWService that manages backfill state, calling into other services to do batched work
Home Page: https://cashapp.github.io/backfila/
License: Apache License 2.0
Service that manages backfill state, calling into other services to do batched work
Home Page: https://cashapp.github.io/backfila/
License: Apache License 2.0
There are several assumptions around the files not changing out from under the s3 client. We might want to check this for a better customer experience.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
We probably need some kind of side channel data for this. The prepare can pull this information perhaps?
We currently just display a "500" string instead of a useful exception. We should propagate the exception in the same way we do for RunBatchResponse by having a stack_trace string
blocked on getting a new misk release
If a value is ever touched it will use empty string rather than null as the value. There is no way to unset it afaik either.
Not quite sure what we should do here. We could have a null checkbox that starts out checked. But that seems like overkill.
Another option is to treat empty as null but this isn't great either. Maybe someone else has a better idea.
Maybe by leveraging Kinesis we can assume better long term sharding support:
https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
class kotlin.reflect.jvm.internal.calls.CallerImpl$FieldGetter cannot access a member of class com.squareup.fidelius.server.backfill.backfila.dynamo.Parameters with modifiers "private final"
To support record classes, we need to change the reflection strategy to use RecordComponents
if the javaClass
isRecord()
.
Currently a backfills with a long wait between batches will hold onto processing power and reduce CPU so the autoscaler won't scale up.
Probably need to either come up with a way for the autoscaler to look at threads too or a way to process more evenly.
I'm finding several people are getting confused on what an instance is. It is a term that is already pretty well overloaded.
I think a better name might be cursor.
Tests failing with invalid default value for 'lease_expires_at'
I had to turn it off so the tests pass in CI:
#21
Should add scan and batch run times to the UI and add times to various logs client and backfila side.
This can only happen on a stopped backfill.
When an old backfill has been sitting around for a configured period. If it isn't in an allowlist then increasingly bother the creators to remove it.
Conversation here: https://square.slack.com/archives/CET1PJ8NP/p1600206743294600?thread_ts=1600192471.251500&cid=CET1PJ8NP
Backfila is unexpectedly retrying immediately on connection failure rather than respecting the backoff schedule. We are also currently seeing a repeatable connection failure after 1 minute
okhttp3.internal.http2.StreamResetException: stream was reset: CANCEL
While backfill logic is largely idempotent, for large batches, a parallel batch on the same ID range can adversely performance since there'll be a nice bunch of extra lock waits.
We need to dig into what in Backfila's stack that's doing the retry on connection reset. Separately, why the reset is happening at all, though that's probably outside of backfila.
It'd be nice to be able to drop a note on a job to the effect of "Next time we run this cut the batch size by 50% because we're blowing up Thing X"
Similarly could use it to centralize findings, plans for future runs, sick recipes, whatever
Requiring custom parameters to be immutable means creating a lot of extra backfills in some cases. Would be great if they were mutable like the batch size, scan size, etc.
With a long enough wait between batches the lease will expire too early.
Rather than batch size being set, have backfila adjust it over time.
Totally doable with configurable number of instances:
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html
Would be great for:
When registering on service startup also register ways to detect load so backfila can automatically adjust backfill speed based on this data.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.