Comments (4)
Thanks for the detailed bug report. We'll get a new release out asap to fix this.
from cassiopeia.
Hey - been planning on updating the caching stuff eventually to have configurable expiration periods for each type + some eviction policy. Haven't gotten around to it yet. For now you can just set the datastore to a new instance of Cache to clear it.
from cassiopeia.
After almost two years we finally get to close this issue. This was fixed / added over a series of recent commits.
Some details:
The data sinks now support expiration timeouts! I.e. you can specify how long you want data to "live" in the cache or diskstore. If a piece of data is accessed that has expired, it will be removed from the data sink and the data will be refreshed. Currently, data will not be removed from the data sink until it is accessed, and it's up to you as the user to remove "old" data if it's taking up too much space / memory. You can do this via one of the following two methods: The settings (available via cass.configuration.settings
) now has 1) .clear_sinks
and 2) .expire_sinks
which 1) remove things from all data sinks and 2) remove all expired data from all data sinks. You can specify how long you want data to live in a datasink in your settings.
from cassiopeia.
Hi @jjmaldonis, I have encountered the same issue but even after doing .clear_sinks
and .expire_sinks
the issue persists. After maybe 5 minutes the memory usage grows from 70-120MB.
Here is some code which I am using to reproduce the issue, I have removed all other parts which could cause memory leaks:
def crawl_region(region):
print('Starting crawling {}'.format(region))
last_index = 0
summoners = []
challenger_league = cass.get_challenger_league(queue=Queue.ranked_solo_fives, region=region)
master_league = cass.get_master_league(queue=Queue.ranked_solo_fives, region=region)
for s in challenger_league:
summoners.append(int(s.summoner.id))
for s in master_league:
summoners.append(int(s.summoner.id))
while last_index < len(summoners):
try:
summoner = Summoner(id=summoners[last_index], region=region)
last_index += 1
end = datetime.now()
start = end - timedelta(days=7)
match_history = cass.get_match_history(summoner, queues={Queue.ranked_solo_fives, Queue.blind_fives,
Queue.ranked_flex_fives, Queue.normal_draft_fives},
begin_time=start, end_time=end, region=region)
for match in match_history:
participants = [participant for team in match.teams for participant in team.participants]
for p in participants:
side = 0 if p.team.side == Side.red else 1
role = derive_position(p.timeline.lane, p.timeline.role)
cass.configuration.settings.clear_sinks()
cass.configuration.settings.expire_sinks()
except:
cass.configuration.settings.clear_sinks()
cass.configuration.settings.expire_sinks()
To get this code to work I had to wrap part of the expire_sinks
method using a try catch block, namely:
for sink in self.pipeline._sinks:
for type in types:
try:
sink.expire(type)
except:
pass
This was done because I got the following error:
File "C:\ProgramData\Anaconda3\lib\site-packages\cassiopeia\datastores\cache.py", line 137, in expire
self._cache.expire(type)
AttributeError: 'Cache' object has no attribute 'expire'
Even after I dug around the code in the library and set the default expiration in cache.py
to 0 the issue persists.
Versions used:
cassiopeia: 3.0.15
merakiommons: 1.0.2
python: 3.6 (anaconda)
from cassiopeia.
Related Issues (20)
- AttributeError: 'Match' object has no attribute '_lazy__timeline' HOT 1
- Add SEA region for OC1 HOT 1
- TypeError: '<' not supported between instances of 'Position' and 'Position' HOT 1
- IndexError: list index out of range when requesting the position too early in the game HOT 4
- Lane information
- Type-hinting of Match.participants is wrong? HOT 3
- How to get positions more than once every minute?
- Method __str__ of class SummonerSpell doesn't work as intended HOT 8
- CumulativeTimeline object doesn't have all the info available in the API HOT 2
- Missing platforms in Region Enum HOT 1
- Add `getByRiotId`, Add riot ID to summoner data HOT 10
- New release HOT 1
- New queue types need to be added to cassiopeia HOT 1
- Need update HOT 1
- Spectator-V4 API not available since Apr 8, 2024 HOT 3
- ACCOUNT-V1 seems to be using the wrong region for PH. HOT 1
- get_account does not allow you to set a region HOT 3
- AttributeError: 'Participant' object has no attribute '_lazy__champion' HOT 1
- Support RSO? HOT 2
- How i can catch the exceptions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cassiopeia.