How to generate a dataset for content popularity and run simulations with our own algorithm

[icarus-sim]

How to generate a dataset for content popularity and run simulations with our own algorithm

[lorenzosaino]

Hi Madhu,

I am not sure I understand your use case. Could you elaborate more what you'd like to do?

Best regards,
Lorenzo

[madhucharan]

Hi,
Let me explain to you a bit clear about our use case. we are writing a research paper on content popularity prediction and fog caching in IoT devices. We came across a research paper comparing 4 different algorithms(LRU etc) existing in Icarus already comparing their approach with the simulation with Icarus software. Our research paper is comparing our approach(here we use a deep neural network for content popularity prediction) with existing algorithms.

For this, we either need to simulate the environment or we need to generate data with some request rate, popularity range, etc which is done by Icarus as simulation. We in general don't know how to create such a dataset.

So my question is- is there any way to save the data generated with the help of Icarus simulation i.e the requests, popularity, and other parameters which are generated through some mathematical distributions, such that we will use our deep learning approach to predict the popular content through the parameters/dataset provided.

[lorenzosaino]

Hi Madhu, if I understand correctly you want to run simulations and produce a dataset with the log of all requests that were issued during the simulation that then you want to use to train your neural network. Am I correct?

The answer is yes, but you will need to do some implementation work. When you create a configuration for simulation you can specify what data collectors to use. Here is the list of all data collectors currently implemented: https://github.com/icarus-sim/icarus/blob/master/icarus/execution/collectors.py. You would need to implement a new collector that logs all requests and configure your simulation to use that new collector.

[madhucharan]

Yeah, I am asking the data simulation because I observed that icarus is mostly supporting general(custom as well) algorthms but not deep neural networks for content popularity prediction. So I want to extract the simulation and use it for DNN predictions and then compare against the extising algorithms.

would you please let me know implimenting a collector is doable or not?since I might need to learn few more things to do that without many errors. if there is a simple way to do it, could you please guide me implimenting it

[lorenzosaino]

It should not be too difficult if you are familiar with Python. You will need to create a class that inherits from DataCollector (see

icarus/icarus/execution/collectors.py

Lines 31 to 151 in 510aa62

	class DataCollector(object):
	"""Object collecting notifications about simulation events and measuring
	relevant metrics.
	"""
	def __init__(self, view, **params):
	"""Constructor
	Parameters
	----------
	view : NetworkView
	An instance of the network view
	params : keyworded parameters
	Collector parameters
	"""
	self.view = view
	def start_session(self, timestamp, receiver, content):
	"""Notifies the collector that a new network session started.
	A session refers to the retrieval of a content from a receiver, from
	the issuing of a content request to the delivery of the content.
	Parameters
	----------
	timestamp : int
	The timestamp of the event
	receiver : any hashable type
	The receiver node requesting a content
	content : any hashable type
	The content identifier requested by the receiver
	"""
	pass
	def cache_hit(self, node):
	"""Reports that the requested content has been served by the cache at
	node *node*.
	Parameters
	----------
	node : any hashable type
	The node whose cache served the content
	"""
	pass
	def cache_miss(self, node):
	"""Reports that the cache at node *node* has been looked up for
	requested content but there was a cache miss.
	Parameters
	----------
	node : any hashable type
	The node whose cache served the content
	"""
	pass
	def server_hit(self, node):
	"""Reports that the requested content has been served by the server at
	node *node*.
	Parameters
	----------
	node : any hashable type
	The server node which served the content
	"""
	pass
	def request_hop(self, u, v, main_path=True):
	"""Reports that a request has traversed the link *(u, v)*
	Parameters
	----------
	u : any hashable type
	Origin node
	v : any hashable type
	Destination node
	main_path : bool, optional
	If *True*, indicates that link link is on the main path that will
	lead to hit a content. It is normally used to calculate latency
	correctly in multicast cases. Default value is *True*
	"""
	pass
	def content_hop(self, u, v, main_path=True):
	"""Reports that a content has traversed the link *(u, v)*
	Parameters
	----------
	u : any hashable type
	Origin node
	v : any hashable type
	Destination node
	main_path : bool, optional
	If *True*, indicates that this link is being traversed by content
	that will be delivered to the receiver. This is needed to
	calculate latency correctly in multicast cases. Default value is
	*True*
	"""
	pass
	def end_session(self, success=True):
	"""Reports that the session is closed, i.e. the content has been
	successfully delivered to the receiver or a failure blocked the
	execution of the request
	Parameters
	----------
	success : bool, optional
	*True* if the session was completed successfully, *False* otherwise
	"""
	pass
	def results(self):
	"""Returns the aggregated results measured by the collector.
	Returns
	-------
	results : dict
	Dictionary mapping metric with results.
	"""
	pass

) and overrides all methods that you need. Each method is gets called during the experiment execution to notify an event to the data collector. You can see the documentation in the code for what each method does. Also, you may want to read this paper for an overview of Icarus architecture, including data collectors: https://lorenzosaino.github.io/publications/icarus-simutools14.pdf

An simple example implementation of a data collector that's used for testing is here:

icarus/icarus/execution/collectors.py

Lines 507 to 561 in 510aa62

	@register_data_collector('DUMMY')
	class DummyCollector(DataCollector):
	"""Dummy collector to be used for test cases only."""
	def __init__(self, view):
	"""Constructor
	Parameters
	----------
	view : NetworkView
	The network view instance
	output : stream
	Stream on which debug collector writes
	"""
	self.view = view
	@inheritdoc(DataCollector)
	def start_session(self, timestamp, receiver, content):
	self.session = dict(timestamp=timestamp, receiver=receiver,
	content=content, cache_misses=[],
	request_hops=[], content_hops=[])
	@inheritdoc(DataCollector)
	def cache_hit(self, node):
	self.session['serving_node'] = node
	@inheritdoc(DataCollector)
	def cache_miss(self, node):
	self.session['cache_misses'].append(node)
	@inheritdoc(DataCollector)
	def server_hit(self, node):
	self.session['serving_node'] = node
	@inheritdoc(DataCollector)
	def request_hop(self, u, v, main_path=True):
	self.session['request_hops'].append((u, v))
	@inheritdoc(DataCollector)
	def content_hop(self, u, v, main_path=True):
	self.session['content_hops'].append((u, v))
	@inheritdoc(DataCollector)
	def end_session(self, success=True):
	self.session['success'] = success
	def session_summary(self):
	"""Return a summary of latest session
	Returns
	-------
	session : dict
	Summary of session
	"""
	return self.session

Only thing to be aware of is that this line here (

icarus/icarus/execution/collectors.py

Line 507 in 510aa62

@register_data_collector('DUMMY')

) serves the purpose of assigning a name to your data collector that you can then refer in your configuration when running simulations.

Hope this helps. Best of luck with your research!

[lorenzosaino]

I am closing the issue for now. Feel free to reopen it if you still have further issues.