dpploy/cortix

Cortix is a Python library for network dynamics modeling and HPC simulation.

cortix's Issues

Verify MPI Implementation

After the wind is complete, test to make sure MPI implementation behaves as expected

add_module() design

cortix/cortix/src/cortix_main.py

Line 54 in dc1b401

def add_module(self, m):

This design has one issue. The ports of the module must all be set up before this function is called. Therefore the user cannot call the function before ports are wired. This is a reflection of having ports outside the Module class. I would like to be able to add a module to a Cortix object at any point in a program, and as the network is built, the module object is updated automatically. The consolidated run_droplet.py

cortix/cortix/examples/fluidflow/run_droplet.py

Line 82 in 979faa6

#cortix.add_module(data_plot)

run file in my branch is an example that does not work.

X server crashed with example running mutiprocessing

cortix/cortix/examples/fluidflow/run_droplet_multi_plot.py

Line 33 in dc1b401

cortix = Cortix(use_mpi=True)

When running this example with use_mpi=False, I get fatal IO errors on my X server. In fact at some point it crashed my machine and corrupted the filesystem.
Could someone verify this example is problematic when running with multiprocessing and why?
Thanks.

Output:
[snip]
Vortex::time[s] = 0.0
Vortex::time[s] = 10.0
Vortex::time[s] = 20.0
XIO: fatal IO error 25 (Inappropriate ioctl for device) on X server ":0"
after 426 requests (426 known processed) with 15 events remaining.
XIO: fatal IO error 25 (Inappropriate ioctl for device) on X server ":0"
after 426 requests (426 known processed) with 15 events remaining.
XIO: fatal IO error 25 (Inappropriate ioctl for device) on X server ":0"
after 434 requests (434 known processed) with 19 events remaining.
Destroyed Cortix object

connectivity of modules

This form of module.connect

prison.connect( 'parole', parole.get_port('prison') )

(see example notebook) could be greatly improved with a simple inspection of intent of usage. For example:

prison.connect(parole)

in this case the function would inspect the ports of both prison and parole and if the port names match, this will do the connection; making the code very expressive and simple.
See module doc for port names expected for this case.

Module self.rank

cortix/cortix/src/module.py

Line 16 in ddb5a10

self.rank = None

self.rank is not used anywhere in Module. Why is it needed? Probably it is not.

Large unpickle-able objects for multiprocessing data tranfer

It this relevant for our port implementation?

https://github.com/dRoje/pipe-proxy

"Because I needed my proxy to be able to handle big objects with attributes that can't be pickled."

Signal implementation for modules

Some modules like DataPlot simply listen for requests and serve them in a loop with no exit condition like in:

cortix/cortix/examples/dataplot.py

Lines 67 to 74 in f65005f

	while True:
	d = self.recv(port)
	if self.debug and i % self.print_freq == 0:
	self.log.info('DataPlot::'+port.name+' received: {}'.format(d))
	if isinstance(d, str) and d == "DONE":
	self.data[port.name] = data
	sys.exit(0) # kill thread
	i += 1

Such modules should ideally die whenever the rest of the modules in the network die. To implement this behavior, Cortix needs a way to cleanly terminate a module from the master process (rank) upon request.

Trim Logging

Console logging is becoming too verbose. Logs should output to the console only what is necessary.

pytest failures

screen.txt

Bug fix needed in Simulation:__setup_task()

The order of the network tag in the Cortix config file matters. It is a difficult bug to track but needs to fixed. Better yet, revamping the XML tag is probably needed.

All MPI processes have a copy of the entire network

cortix/cortix/src/cortix_main.py

Line 38 in dc1b401

self.modules = []

By doing this, every MPI process has a copy of the entire network. When running say a 1000-module network case, the total number of instances of modules is 1,000,000. Maybe this requires some thought...scaling becomes an issue.

Split up Droplet/Vortex and Cortix?

Droplet/Vortex is becoming a larger development. It might be a good idea from a design standpoint to make Droplet/Vortex a stand alone repo with its own README and docs and make Cortix a dependency. Thoughts?

Droplet DataFrame output

For the droplet example in console-run, can the output data be changed to CSV format? This would make parsing the file easier before moving on to true message passing.
I suggest incorporating these methods into the code:
pandas.to_csv()
pandas.read_csv()

Bug fix needed in PyPlot when plotting negative valued quantities

PyPlot error; y axis incorrect for negative valued quantities.

Cortix module has no attribute 'ports'

I'm Having trouble developing a Cortix module for dados. How should the ports be incorporated into the code?
Is it possible to have the option to omit port names and assigning ports?

Traceback (most recent call last):
File "dados.py", line 27, in <module>
app.run()
File "dados.py", line 19, in run
self.rs232.add_port(p1)                                                                                                                                               File "/home/pi/cortix/cortix/src/module.py", line 55, in add_port
if port not in self.ports:
AttributeError: 'RS_232' object has no attribute 'ports'

Add repo hot link to the gh-pages preferably on the side bar

mpirun info in the Cortix run file.

Is there a way to make a run file know whether it has been executed by the shell directly or by mpirun? This is to avoid running Cortix with a run file set to run in multiprocessing mode but using MPI. When testing and developing, switching back and forth between MPI and multiprocessing requires changing the value of use_mpi in the run file. It would be very helpful to have the run file say: hey you tried to run using mpirun but the use_mpi flag is set to False.

Incorrect network drawing for example with multiple plots of droplet.

cortix/cortix/src/cortix_main.py

Line 116 in dc1b401

def draw_network(self, file_name="network.png", dpi=220):

This function is not drawing the right graph of the run_droplet_multi_plot.py. Maybe the network created is incorrect too. See pic:

It should be one DataPlot per Droplet and only connected to a single Droplet. The example runs correctly with MPI but does not run correctly with multiprocessing (issue is open).

Headers maintenance

Please help add the same header to every file in Cortix. I have started but we need to keep an eye open and fix the files as we go. Use one of the files in the src/ directory as a model.

Document the cortix driver in examples/ and the cortix-config.xml file in examples/input. They need to appear in the docs gh-page too.

Having trouble figuring finding a good way to document cortix-config.xml

Unable to install Scipy on new machine

An additional library is needed to install the Scipy requirement:
sudo apt-get install libatlas-base-dev gfortran
Perhaps this step can be added to the Cortix README.

Deprecated matplotlib warning

I think this:

cortix/cortix/src/cortix_main.py

Line 156 in ddb5a10

def draw_network(self, file_name="network.png", dpi=220):

generates this:

/usr/lib64/python3.6/site-packages/matplotlib/cbook/deprecation.py:107: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance. In a future version, a new instance will always be created and returned. Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
warnings.warn(message, mplDeprecation, stacklevel=1)
Destroyed Cortix object on 30Jul19 18:52:24

on my machine.

Thread unsafe plotting

This loop seems not to work unless the loop is over one port. Since Matplotlib is not thread-safe once these plotting threads are created with plotting functions, the plots do not come out correct. Please verify or correct this. There is a run_droplet-2.py cortix run file in examples/ that runs a problematic case with one DataPlot connected to several Droplets. Thanks.

How to fix issue with cortix_driver files

How to communicate from module to main thread?

How do pass a message from a Cortix module to the main thread?
All the examples have modules only communicating with other modules.
I want to start an independent process using a Cortix object, then pass messages to and from that module while it runs indefinitely.

should Port do the send and receive directly?

This could be eliminated and module developers implement the call to the port directly. The resulting module derived code would be more expressive. For example:

""" Droplet module """
[snip]
port = self.get_port('flow-velocity')
port.send( (time,position) )
flow_velocity = port.recv()

Or maybe have a single call?
flow_velocity = port.send_recv( (time,position) )

Seems like the send and receive are jobs for Port objects.

Add an ID# to the end of file names

Would it be possible to add a unique identifier to the end of files? At the moment it is impossible to distinguish results from different runs. This change would also allow more than 1 results files to exist in the same directory.
Another advantage is improved continuity when having multiple instances running at the same time.
I like to get the ID string from the timestamp:
datetime.datetime.now()

Build a wrapper for configuration

It would be helpful to have a wrapper around the configuration system that enables us to configure cortix with python. The idea: python -> xml -> cortix

Generate documentation

Use sphinx to generate documentation for the project

Initial Thoughts

• This could either be in a separate branch or in a submodule
• Hosting should be done via github pages
• Document generation should be automatic (scripted), should be done in one step

Add Cortix repo hyperlink to the top of ghpages or to the sidebar

Port rank

Is the rank here

cortix/cortix/src/cortix_main.py

Lines 82 to 87 in ddb5a10

	if m not in self.modules:
	self.modules.append(m)
	if self.use_mpi:
	m.rank = len(self.modules)
	for port in m.ports:
	port.rank = m.rank

the same as the rank member variable in the constructor?

cortix/cortix/src/port.py

Lines 28 to 30 in ddb5a10

	if self.use_mpi:
	self.comm = MPI.COMM_WORLD
	self.rank = self.comm.Get_rank()

Or is it overriding it after construction?

Overflow of processes while using Cortix in a loop

Hi,
A Multiprocessing error is returned whenever more processes are open than the system can handle, which is:
OSError: [Errno 24] Too many open files
I'm not sure what triggers the exception. It could possibly be hardware dependent, but this behavior is expected.

However, this error also triggers while placing a Cortix simulation within a loop. It appears some processes carry over into the rest of the python program even though they are not needed. Is there anyway Cortix can handle ending processes better?

Use case:

n_list = [15,30,45,60,75,500,600,1200,2400,3200,4200,5400]
for i in n_list:
    procs = 15
    runtime=0.5
    balls = int(i/procs)
    cortix = Cortix(use_mpi=False)
    mod_list = []
    shape = geo.box(-30,0,30,50)
    
    plot = Plot(number=i,runtime=runtime)
    cortix.add_module(plot)

    for i in range(procs):
        time.sleep(0.01)
        app = BallHandler(shape, balls=balls,runtime = runtime)
        app.v0 = [40,40]
        app.r = 0.1
        mod_list.append(app)
        cortix.add_module(app)
        
    for c,i in enumerate(mod_list):
        i.connect('plot-send{}'.format(c),plot.get_port('plot-receive{}'.format(c)))
        for j in mod_list:
            if i == j:
                continue
            name = '{}{}'.format(i.timestamp,j.timestamp)
            name2 = '{}{}'.format(j.timestamp,i.timestamp)
            j.connect(name, i.get_port(name2))
            
    cortix.run()

After about 3 loops the program crashes. Each individual loop would be fine, but they can't be run subsequently.

network graph with arrows

It would be really insightful to have arrows on the network graph to denote whether an edge is used both ways. This helps understanding the flow of information. For instance see graph in

https://nbviewer.jupyter.org/github/dpploy/cortix-nb/blob/master/run_justice.ipynb#Verify-the-network-connectivity

not all edges have bidirectional information.

Specification of work directory

Add an option to specify a work directory in the task block of the xml config file. This will override any previously defined work directory in the xml file.
Add an option to specify a work directory in the argument list of the Cortix.run_simulations() method. This will override anything previously defined in the xml config file.

Bug with Droplet_run example

Traceback (most recent call last):
  File "/home/pi/cortix/cortix/src/network.py", line 115, in __init__
    assert use_port not in tmp.keys(), \
UnboundLocalError: local variable 'use_port' referenced before assignment

This traceback error appears sporadically. Sometimes the error appears in the Cortix application.

Initial test functions

Implement initial testing functionality for each module

• This should be done via the pytest module

Mods to simplify user creation of a run file

The user does not need to create ports or add a module to Cortix at a later time in the run file as currently it is done

cortix/cortix/examples/run_droplet.py

Lines 74 to 119 in ddb5a10

	# Vortex module (single).
	vortex = Vortex()
	#cortix.add_module(vortex)
	vortex.show_time = (True,100)
	vortex.end_time = end_time
	vortex.time_step = time_step
	vortex.plot_velocity()
	# DataPlot module (single).
	data_plot = DataPlot()
	#cortix.add_module(data_plot)
	data_plot.title = 'Droplet Trajectories'
	data_plot.same_axes = True
	data_plot.dpi = 300
	for i in range(n_droplets):
	# Droplet modules.
	droplet = Droplet()
	#cortix.add_module(droplet)
	droplet.end_time = end_time
	droplet.time_step = time_step
	droplet.bounce = False
	droplet.slip = False
	# Ports def.
	external_flow = Port('external-flow')
	droplet.add_port(external_flow)
	visualization = Port('visualization')
	droplet.add_port(visualization)
	# DataPlot module ports def.
	plot = Port('viz-data:{:05}'.format(i))
	data_plot.add_port(plot)
	# Vortex module ports def.
	fluid_flow = Port('fluid-flow:{}'.format(i))
	vortex.add_port(fluid_flow)
	# Network connectivity (connect ports)
	external_flow.connect(fluid_flow)
	visualization.connect(plot)
	cortix.add_module(droplet)
	cortix.add_module(vortex)
	cortix.add_module(data_plot)

Mods for simplifying this will be posted in a PR and linked here.

Script for generating .rst files

A script to loop through the subdirectories of src and generate .rst files for each one so that the project is fully documented.

run_droplet structure review

I am looking at the flow of a Cortix run file. Why 3 ports in a Droplet module? In the same vein. If a port is added to a different droplet object, why does it need an indexed name? why can't it be the same port name since it belongs to a different object? Ah. The indexed ports all go to Vortex. Disregard the indexing but this calls for clarity.

How to connect multiple ports to one module

How do you connect multiple ports to one module? There seems to be a bug with Port.connect(Port) where only the most recently called connection is saved.

Example of bug:
https://github.com/dpploy/dados/blob/seamus/src/dados.py
If you were to switch the order of line 22 and line 23, then rs232 can't communicate with plotting anymore but mcc118 can. Any idea what is causing the issue?

Droplet and wind example

Get modules working

modules list data lost when in multiprocessing mode

In this block:

cortix/cortix/src/cortix_main.py

Lines 131 to 138 in f65005f

	# Parallel run all modules in Python multiprocessing
	if not self.use_mpi:
	for mod in self.modules:
	processes = list()
	self.log.info('Launching Module {}'.format(mod))
	p = Process(target=mod.run)
	processes.append(p)
	p.start()

p = Process(target=mod.run) copies the mod object onto the child process. The data is frozen at that time and when the child process ends, the data is lost. The modules list in the parent process needs to get that data for all child process so the Cortix object can offer the caller access to the data in the mod objects as created by the child process. A chief use of that is when running a Jupyter notebook Cortix app. Once cortix.run() finishes the user can do cortix.modules and access all the data computed in parallel for plotting purposes (say); a very useful feature.

How to fix sphinx error when documenting pyplot.py and cortix_driver.py in the pyplot subdirectory.

Sphinx is not creating HTML for these files because it cannot find the imports due to a path/naming problem when using the current directory "." import. The names used in the imports can't be changed because it breaks the code.

What is the Cortix receive function?

There is a send function, where is its opposite function to receive? Has it been implemented yet?
Could it be added in with syntax to match existing socket language?
https://docs.python.org/2/library/socket.html#socket.socket.recv

How to make the modules list directly accessible from the Cortix front webpage?

vortex review

Vortex does not need to know about the existence of droplets. This points to a review of how ports are created.

Cortix Base Class

It would be useful to have a Cortix base class which implements functions for the driver to override. Similar to the design of threading in Python.

Cortix future changes experiment

I am experimenting with changes to accommodate future cases involving systems of systems or nested systems. An explicit introduction of a "top system" is showed below and introduction of a network as part of the system (same thing as module). Therefore modules turn into potential sub-system by inheriting a network from a Network class:

This is a simple look at a run file for a system without sub-systems; the usual case we have all been looking at in examples:

cortix/cortix/examples/example_single.py

Lines 22 to 46 in 52ac812

	def main():
	'''Cortix run file for a system.
	'''
	cortix = Cortix(use_mpi=True)
	# Create the top system
	usa = Module() # or Country()
	cortix.system(usa) # only allwed one top system
	# Create network
	usa.network = Network(n_modules=2)
	maine = State('Maine')
	vermont = State('Vermont')
	usa.network.connect(maine,vermont)
	usa.network.draw('network.png')
	# Run system (this will run any nested underlying networks)
	cortix.run(usa)
	# Properly shutdow cortix
	cortix.close()

This shows the same case with 2 levels of sub-systems:

cortix/cortix/examples/example_nested.py

Lines 26 to 90 in 52ac812

	def main():
	'''Cortix run file for a system.
	'''
	cortix = Cortix(use_mpi=True)
	# Create the top system
	usa = Module() # or Country()
	cortix.system(usa) # only allwed one top system
	# Create network
	usa.network = Network(n_modules=2)
	maine = State('Maine')
	vermont = State('Vermont')
	usa.network.connect(maine,vermont)
	usa.network.draw('network.png')
	# Maine sub-system
	maine.network = Network(n_modules=3)
	prison_A = Prison(style='A')
	maine.network.module(prison_A)
	parole_A = Parole(style='A')
	maine.network.module(parole_A)
	jail_A = Jail(style='A')
	maine.network.module(jail_A)
	maine.network.connect(prison_A,parole_A)
	maine.network.connect(jail_A,prison_A)
	maine.network.draw('network.png')
	# Vermont sub-system
	vermont.network = Network(n_modules=2)
	prison_B = Prison(style='B')
	vermont.network.module(prison_B)
	jail_B = Jail(style='B')
	vermont.network.module(jail_B)
	vermont.network.connect(jail_B,prison_B)
	vermont.network.draw('network.png')
	# Prison A-style sub-system
	prison_A.network = Network(n_modules=2)
	p1 = PrisonCampus('1')
	prison_A.network.module(p1)
	p2 = PrisonCampus('2')
	prison_A.network.module(p2)
	prison_A.network.connect(p1,p2)
	prison_A.network.draw()
	# Run system (this will run all nested underlying networks)
	cortix.run(usa)
	# Properly shutdow cortix
	cortix.close()

Dynamically generate configuration files

Implement functionality for dynamically generating the XML configuration files required to run Cortix

Build pylint config for Cortix

Need a pylint config for Cortix to avoid so many complaints and make code styling uniform.

Need a .pylintrc for Cortix.