unownhash / golbat Goto Github PK
View Code? Open in Web Editor NEWLicense: The Unlicense
License: The Unlicense
some code like this needed (from RDM)
if self.cellId == nil {
let centerCoord = LocationCoordinate2D(latitude: encounter.pokemon.latitude, longitude: encounter.pokemon.longitude)
let cellID = S2CellId(latlng: S2LatLng(coord: centerCoord)).parent(level: 15)
self.cellId = cellID
}
Provide an api to allow dragonite to indicate that a quest area is being rescanned, and so should be cleared
If we have a scanner which always encounters everything; and it understands it needs to re-encounter pokemon on weather change then writing wilds to the DB is inefficient
Provide an option to ignore wilds and work on encounters only
This should not impact despawn time discovery
(nearbys unaffected)
When I added logic in RDM to support the same processing as Golbat I also "fixed" the ditto with weatherboosts.
it happened that 100% Dittos were stored in DB, that's wrong when those ditto was weatherboosted
Use real account names for updating account name fields
ERRO 2023-02-08 00:01:03 insert incident: Error 1062 (23000): Duplicate entry '5358044204288734400' for key 'PRIMARY'
Line 237 in cd93a82
if pokestop transitioned to gym this cache entry should expire if not raw processed anymore, so imo we could keep the whole cache in that case?! :)
Probably retrospectively categorising spawnpoints created during event times
-36.84929156444202,174.76839760477938
That's too accurate, we should cut them on pokemon
Clear weather
I have a gym with an empty id and lat/lon = 0 in my db, assuming the game sent that wrong for whatever reason. I think this happened on a GymGetInfoProto.
Probably query:
update pokestop join gym on gym.id=pokestop.id set pokestop.deleted = 1 where pokestop.deleted = 0 and gym.deleted = 0 and gym.updated > pokestop.updated;
update gym join pokestop on gym.id=pokestop.id set gym.deleted = 1 where pokestop.deleted = 0 and gym.deleted = 0 and gym.updated < pokestop.updated;
Pokemon 114 CP225"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa61c09]
goroutine 244420 [running]:
golbat/decoder.UpdateGymRecordWithGymInfoProto(0xc001c66050?, 0xc0000f3c20)
/home/dkmur/progs/Golbat/decoder/gym.go:444 +0x29
main.decodeGetGymInfo({0xc001c66050, 0x9, 0x9})
/home/dkmur/progs/Golbat/main.go:176 +0x12e
main.decode(0x9c, 0xc00059bef0)
/home/dkmur/progs/Golbat/main.go:108 +0x26a
created by main.Raw
/home/dkmur/progs/Golbat/main.go:358 +0x8f8
time="2022-07-24T00:04:02+02:00" level=info msg="Connected to database"
time="2022-07-24T00:04:02+02:00" level=info msg="Golbat started"
As description.
To support misleading characterisation of spawn-points
Poffins would be nice to have I guess.
We should define them somehow :)
Another possibility to save CPU time would be to optimize golbat/geo.MatchGeofences
It's worth checking the approach with indexing bbox of polygons inside RTree index. Depending on results, we might as well move MatchGeofences
higher to check all incoming locations.
Does old Pokémon still work?
This is a new message allowing to bulk request raid lobby counts.
message GetRaidLobbyCounterOutProto {
enum Result {
UNSET = 0;
SUCCESS = 1;
ERROR_PLAYER_BELOW_MINIMUM_LEVEL = 2;
}
Result result = 1;
repeated RaidLobbyPlayerCountProto raid_lobbies = 2; // name TBD
}
message RaidLobbyPlayerCountProto {
string gym_id = 2;
int32 player_count = 3;
int64 lobby_join_until_ms = 4;
}
GetRaidLobbyCounterOutProto {
result Result.SUCCESS:1
raid_lobbies { // no active lobby
gym_id "45a76c2b88ba4529a2f9379de6f88d30.16"
}
raid_lobbies {
gym_id "00f9208d76a86794b2178181c304e41d.11"
player_count 1
lobby_join_until_ms 1682090628489
}
}
This will require two new DB fields: lobby_count
and lobby_join_end
(names TBD) and two new webhook fields. If there's no active lobby, the fields should be null
.
As title. Move webhook settings from the config to the database and add an API endpoint to hot reload.
As we preserve IV values now on weather change when mon is still boosted we don't send pvp value in webhook anymore.
This is happening due to the pokemon processing rework / ditto detection improvement of @Mygod. As we don't store pvp value in Cache (to reduce memory usage) we should either recreate pvp value on the fly or find another way :)
they're super straight-forward and come in very handy. This must be done if we want to use AirScout in production.
message GetMapFortsOutProto {
enum Status {
UNSET = 0;
SUCCESS = 1;
ERROR = 2;
}
message FortProto {
string id = 1;
string name = 2;
double latitude = 3;
double longitude = 4;
repeated Image image = 5;
}
message Image {
string url = 1;
string id = 2;
}
repeated FortProto fort = 1;
Status status = 2;
}
For future peeps keep the in_memory
option set to disabled for now. MyGod added a bunch of ditto detection code and someone, cough Turtle cough, promises to get this added.
If you see the following error messages that's why.
ERRO 2023-03-30 22:42:02 Error pokemon [XXXXXXXXXXXX]: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 Error pokemon [XXXXXXXXXXXX]: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
ERRO 2023-03-30 22:42:02 getOrCreatePokemonRecord: no such column: iv_inactive
So in memory / db do not get out of sync
What do you think about adding Prometheus pull-type metrics to Golbat?
That would allow monitoring almost everything, store in TSDB, display using Grafana and even create alerts. Quite powerful stack used (mostly) by big companies.
Prometheus is open source (and written in Go). Here's very basic example of implementation in Go https://prometheus.io/docs/tutorials/instrumenting_http_server_in_go/
It's similar to @dkmur (great!) solution, except Golbat would handle counters inside and expose them to Prometheus server through a single HTTP endpoint. No need for queries or crons.
https://github.com/RealDeviceMap/RealDeviceMap/pull/410/files
spotted by fabio in his swift conversion
Clear incident table when archive mode is set
we removed capture_1, capture_2, capture_3 from processing because those values are not relevant for users. also worker can influence the capture rate with catch medals, so it's not always the base catch rate
we will need a way to keep track of the base catch rate, and then we can do some calculations in the frontends or poracle :)
Ability to switch off data processing based on context
sent along with data. Would allow us to select what type of data should be ignored in processing depending on what context worker sends and Golbat configuration.
Implementation #90
Never enough, I know...
Won't be as significant as others optimizations, but we might flatten database load a bit.
For a starter, we should randomize cache TTL and rise overall cache time (second open for discussion).
Load might be different, but describes a problem well.
there is no point. it's only clutter
my suggestion:
[pvp]
enabled = true
include_hundos_under_cap = false
level_caps = [50, 51]
process_ultra = true
process_great = true
process_little = true
old:
[pvp]
enabled = true
include_hundos_under_cap = false
level_caps = [50, 51]
[[pvp.leagues]]
name = "little"
cap = 500
little = false
[[pvp.leagues]]
name = "great"
cap = 1500
little = false
[[pvp.leagues]]
name = "ultra"
cap = 2500
little = false
eg. Voltdb
while playing with fortwatcher I notice missing webhooks for fort_update.
for me, the flow for new forts is often:
1 added
2 edit name+image
3 edit description
Example, added is send(step 1), the name edit (step 2) never received
eecf6e240f6831eeae2f165b363ea3a5.16
[20230516 21:08:36] Send added pokestop id: eecf6e240f6831eeae2f165b363ea3a5.16 location: 51.628082,4.926408 fence: Dongen name: "Unknown" time: 1.096 s
[20230516 23:12:24] Send edit pokestop description id: eecf6e240f6831eeae2f165b363ea3a5.16 fence: Dongen name: "Broederpad Links" time: 0.768 s
first_seen_timestamp for eecf6e240f6831eeae2f165b363ea3a5.16 2023-05-16 21:08:34
Example, added was never received but I do get an edit name(step2)
eec54b8f48d53bb98b008b720a43291e.16
[20230515 21:45:10] noHook edit pokestop name id: eec54b8f48d53bb98b008b720a43291e.16 fence: unfenced name: "Fietsroutenetwerk Drechtsteden" oldname: "Unknown" time: 0.355 s
[20230515 23:16:04] noHook edit pokestop description id: eec54b8f48d53bb98b008b720a43291e.16 fence: unfenced name: "Fietsroutenetwerk Drechtsteden" time: 0.456 s
first_seen_timestamp for eec54b8f48d53bb98b008b720a43291e.16 2023-05-15 20:50:05
I miss the MAD feature to only send e.g. quests to a specific endpoint rather than sending everything and creating unwanted traffic.
Use timezone database
https://github.com/evansiroky/timezone-boundary-builder
Golbat shouldn't write to disk to make stats, it should count into buckets by itself
Calculate quest end time based on lat/lon and store
I call a Ditto 0P if it is in partly cloudy weather whose disguise is unboosted. In this state, the seen IV is boosted but the caught IV is unboosted. Currently, the only way to know the unboosted IV is if we have encountered the same spawn before in a different state (having different PokemonDisplay and/or under different weather).
Another way to scan IV is to actually catch the Pokemon. Two things could happen:
CatchPokemonOutProto
.CatchPokemonLogEntry.CombatPoints
will have the correct CP set.For example, we have a 0P Ditto disguised as Snubbull as below, encountered as Lv32 and CP 1050.
Catching the Pokemon will give a Ditto at Lv27 14/4/13. However, on a different account where this spawn fled, we can see that the CP in the log was set to an anomalous 889, which is the CP of Snubbull at Lv27 14/4/13.
Using this correct CP + level, we can in principle reverse engineer the IV. While this in principle will not tell you a unique set of IVs, when both the level and the IV are high, it is in principle possible to identify the unique correct IV.
I would like to discuss how to process invasion teams & confirmed Giovanni
Refer to #12 for real invasion data examples.
This data is best extracted from a StartIncidentOutProto
.
StartIncidentOutProto.Incident.step
is an array of ClientIncidentStepProto
. In it, the confirmed grunt character is in both pokestop_dialogue.dialogue_line.character
(ClientPokestopNpcDialogueStepProto
) and invasion_battle.character
(ClientInvasionBattleStepProto
)
A confirmed Giovanni is type 44
. Male Decoy is 45
and Female Decoy is 46
(used by the game). Note that "unconfirmed" Giovanni is also 44
incident
table. I suggest calling it character_confirmed
character_confirmed
bool field._u
flag for grunt UICONs. This is part of the standard but afaik, no uicon repo supports it. Reference PR: UIcons/UIcons#34This data is provided in a OpenInvasionCombatSessionOutProto
.
OpenInvasionCombatSessionOutProto.combat.opponent.active_pokemon
is the first slot, OpenInvasionCombatSessionOutProto.combat.opponent.reserve_pokemon is an array containing the second and third slot (in order).
These Pokemon are of type CombatPokemonProto
. Refer to #12 for details on this.
You may want to abort processing if the array size != 2. This should never happen. Also only process OpenInvasionCombatSessionOutProtos that have teh SUCCESS status. It's not uncommon these can fail.
I'm not sure how to best tackle this.
incidents
table: slot_x_pokemon_id
and slot_x_form
where x is 0, 1, 2 (or 1, 2, 3)team: [
{
"pokemon_id": 0,
"form": 0,
...
},
...
]
Example grunt - showing line up and reward
OpenInvasionCombatSessionProto incident_lookup { incident_id: "5477255916644222015" fort_id: "d36fb83acc5f4516bcf2e3c083145f65.16" fort_lat: 54.090941 fort_lng: 13.397025 } step: 1 attacking_pokemon_id: 6253405486787881069 attacking_pokemon_id: 7051996254806144273 attacking_pokemon_id: 4049703279889415729 lobby_join_time_ms: 1659221762713
Response | OpenInvasionCombatSessionOutProto status: SUCCESS combat { combat_state: ACTIVE combat_id: "COMBAT_1659221782019_-5139414284709935119" player { public_profile { name: "Flux4218" level: 30 avatar { avatar: 1 avatar_hair: "AVATAR_f_hair_default_0" avatar_shirt: "AVATAR_f_shirt_default_5" avatar_pants: "AVATAR_f_pants_default_4" avatar_hat: "AVATAR_f_hat_default_A_5" avatar_shoes: "AVATAR_f_shoes_default_1" avatar_eyes: "AVATAR_f_eyes_0" avatar_backpack: "AVATAR_f_backpack_default_0" avatar_gloves: "AVATAR_f_gloves_default_0" avatar_socks: "AVATAR_f_socks_default_0" avatar_belt: "AVATAR_f_belt_default_4" avatar_necklace: "AVATAR_f_necklace_default_0" } caught_pokemon: 20 experience: 2104205 } active_pokemon { pokemon_id: 6253405486787881069 pokedex_id: ANORITH cp: 1094 cp_multiplier: 0.7068842 stamina: 96 max_stamina: 96 move1: SCRATCH_FAST move2: ANCIENT_POWER pokemon_display { gender: FEMALE form: ANORITH_NORMAL display_id: -2902952500726692630 } individual_attack: 10 individual_defense: 2 individual_stamina: 8 pokeball: ITEM_ULTRA_BALL } reserve_pokemon { pokemon_id: 7051996254806144273 pokedex_id: AIPOM cp: 995 cp_multiplier: 0.7068842 stamina: 108 max_stamina: 108 move1: SCRATCH_FAST move2: AERIAL_ACE pokemon_display { gender: FEMALE form: AIPOM_NORMAL display_id: 2480148140930035634 } individual_attack: 15 individual_defense: 1 individual_stamina: 8 pokeball: ITEM_ULTRA_BALL } reserve_pokemon { pokemon_id: 4049703279889415729 pokedex_id: RAMPARDOS cp: 2054 cp_multiplier: 0.65443563 stamina: 146 max_stamina: 146 move1: ZEN_HEADBUTT_FAST move2: FLAMETHROWER pokemon_display { gender: MALE display_id: 4961293038146386647 } individual_attack: 12 individual_stamina: 5 pokeball: ITEM_ULTRA_BALL } minigame_defense_chances_left: 2 lobby_join_time_ms: 1659221762713 } opponent { public_profile { name: "CHARACTER_FIGHTING_GRUNT_FEMALE" } active_pokemon { pokedex_id: MAKUHITA cp: 1754 cp_multiplier: 1.0204883 stamina: 116 max_stamina: 116 move1: ROCK_SMASH_FAST move2: CROSS_CHOP pokemon_display { gender: MALE alignment: SHADOW } individual_attack: 91 individual_defense: 15 individual_stamina: 9 } reserve_pokemon { pokedex_id: HITMONCHAN cp: 5004 cp_multiplier: 1.0204883 stamina: 92 max_stamina: 92 move1: BULLET_PUNCH_FAST move2: CLOSE_COMBAT pokemon_display { gender: MALE alignment: SHADOW } individual_attack: 153 individual_defense: 15 individual_stamina: 9 } reserve_pokemon { pokedex_id: MACHOKE cp: 4373 cp_multiplier: 1.0204883 stamina: 125 max_stamina: 125 move1: LOW_KICK_FAST move2: BRICK_BREAK pokemon_display { gender: MALE alignment: SHADOW } individual_attack: 143 individual_defense: 15 individual_stamina: 9 } minigame_defense_chances_left: 2 combat_npc_personality_id: "TRAINER_PERSONALITY_EASY" } combat_start_ms: 1659221792019 combat_end_ms: 1659222062019 server_ms: 1659221782019 }
Line 152 in 391a545
Golden lure is id 506
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.