Giter Site home page Giter Site logo

funk's Introduction

Funk

A Python tool to help you to find ressources on the Grid5000 platform. Requires execo 2.6.7

Developped by the Inria Hemera initiative 2010-2014

pip install execo==2.6.7

https://www.grid5000.fr/mediawiki/index.php/Hemera

usage: funk [-h] [--version] [-m {date,free,max}] [-y] [--quiet | -v] [-p PROGRAM] [--plots] [--json] [--json-file JSON_FILE]
			[-r RESOURCES] [-b BLACKLIST] [-R RATIO] [-o SUBMISSION_OPTS] [-k] [-n SUBNET] [-j JOB_NAME] [-q QUEUE] [--virtual]
			[--infiniband] [--green] [-w WALLTIME] [-s STARTDATE] [-e ENDDATE] [-c]

This tool helps you to find resources on Grid'5000 platform. It has three modes: 
- date = give you the number of nodes available at a given date, 
- free = find the next free slot for a combination of resources, 
- max  = find the time slot where the maximum number of nodes are available.

If no arguments is given, compile the planning of the whole platform and generate 
		a oarsub command line with all available resources for 1 hour, on every site.
		Based on execo 2.6.7, https://mimbert.gitlabpages.inria.fr/execo/oar 2.5, http://oar.imag.fr.

General options:
Define mode and controls I/O.

-h, --help            show this help message and exit
--version             show program's version number and exit
-m {date,free,max}, --mode {date,free,max}
						Setup the mode: date, free or max 
						default = date
-y, --yes             Perform the reservation automatically
--quiet               Run without printing anything
-v, --verbose         Run in verbose mode
-p PROGRAM, --program PROGRAM
						The program to be run when the reservation start
--plots               Draw a Gantt plot and the slots
--json                Output the computed data to standard output
--json-file JSON_FILE
						Output the computed data to a json file

Reservation:
Customize your Grid'5000 reservation.

-r RESOURCES, --resources RESOURCES
						Comma separated list of Grid'5000 elements  (grid5000, site or cluster)
						-r element1,element2 for date and max modes
						-r element1:n_nodes1,element2:n_nodes2 for free mode
						default = grid5000
-b BLACKLIST, --blacklist BLACKLIST
						Remove clusters from planning computation
-R RATIO, --ratio RATIO
						Apply a given ratio to the host resources
-o SUBMISSION_OPTS, --submission_opts SUBMISSION_OPTS
						Extra options to pass to the oarsub command line
-k, --kavlan          Ask for a KaVLAN
-n SUBNET, --subnet SUBNET
						Ask for subnets. slash_22=1 will retrieve a /22 subnet on every site of your requests, 
						but you can specify site1:slash_22=2,site2:slash_19=1
-j JOB_NAME, --job_name JOB_NAME
						The job name passed to the OAR subjobs
						default = FUNK
-q QUEUE, --queue QUEUE
						The OAR queue to use
						default = default
--virtual             Use only clusters with hardware virtualization
--infiniband          Use only clusters with infiniband or myrinet interfaces
--green               Use only clusters that have energetical power measurements

Time:
Define options related to date and time.

-w WALLTIME, --walltime WALLTIME
						Reservation walltime in OAR format
						default = 1:00:00
-s STARTDATE, --startdate STARTDATE
						Starting date in OAR format
						default = now
-e ENDDATE, --enddate ENDDATE
						End date in OAR format
						default = three weeks after the startdate
-c, --charter         Avoid charter periods

Examples:
Number of available nodes on stremi cluster from date to date + walltime 
funk -m date -s "2024-06-12 14:57:43" -r stremi
First free slots for a resource combination with deploy job type and a KaVLAN
funk -m free -w 2:00:00 -r grid5000:100,taurus:4 -o "-t deploy" -k
Maximum number of nodes available for the resources, avoiding charter periods
funk -m max -w 10:00:00 -r nancy,paradent,edel -c 
Issues/features requests can be reported to https://github.com/lpouillo/Funk

funk's People

Contributors

lpouillo avatar mimbert avatar simondelamare avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

funk's Issues

problem with rennes

Is this a bug or me? To the best of my knowledge and even thought there are no free slots available, the tool should not crash like that right?

agarciarecuero@frennes:~$ funk -m free -r rennes:15 -w 61:00:00 -s "2014-03-14 19:00:00" -o "-t deploy"
2014-03-10 15:24:03,892 INFO: -- Find yoUr Nodes on g5K --
2014-03-10 15:24:03,892 INFO: From 2014-03-14 19:00:00+01:00 to 2014-03-31 16:25:03+02:00
2014-03-10 15:24:03,892 INFO: Walltime: 61:00:00
2014-03-10 15:24:03,893 INFO: Mode: free
2014-03-10 15:24:05,843 INFO: Wanted resources
rennes       15


2014-03-10 15:24:05,844 INFO: Compiling planning
2014-03-10 15:24:05,865 WARNING: error with rennes, removing from computation
2014-03-10 15:24:05,866 INFO: Calculating slots of 61:00:00
Traceback (most recent call last):
  File "/grid5000/code/bin/funk", line 241, in <module>
    slots = compute_slots(planning, args.walltime, excluded_elements = blacklisted)
  File "/home/g5kcode/public/bin/funk_package/execo_g5k/planning.py", line 143, in compute_slots
    limits = _slots_limits(planning)
  File "/home/g5kcode/public/bin/funk_package/execo_g5k/planning.py", line 825, in _slots_limits
    limits.pop()
IndexError: pop from empty list
agarciarecuero@frennes:~$

minimal walltime

It would be great if the option -w was not mandatory and completed with another option to specify the minimal walltime wanted.

The maximum walltime could be computed by outofchart and the current available resources.

So depending of each site, the walltime could be different.

Improve final output

When doing a "oargridsub" reservation, it could be nice to output the "oarsub" id per site instead of just the "oargridsub" id.

For example:

execo INFO Grid reservation done, oargridjob_id = 46201
execo INFO sophia  oarjob_id = xxxxxx
execo INFO lyon     oarjob_id = xxxxxx
execo INFO reims   oarjob_id = xxxxxx
execo INFO Be smart, use oarsub -c ID ;)

besteffort reservation failed

Hello,
Here is a small improvement.

Consider the following funk command (reserve 2 nodes on sophia during 4 hours) :

funk -m free -r "sophia:2" -w "4:00:00" -o "-t deploy -t besteffort" -q "besteffort" -y

It will results in the following oarsub command

oarsub -t deploy -t besteffort -l "nodes=2,walltime=4:0:0" -q "besteffort" -r "2015-11-30 14:19:15" -n "FUNK" "sleep 31536000"

However because of the -r argument, it will be rejected

Admission Rule ERROR : [ADMISSION RULE] Error: a job cannot both be of type besteffort and be a reservation.

Could funk detects the queue and if it's besteffort remove the -r argument ?

Verify the -p flag validity before executing the command

The execution of "./funk -p" on grid5000 result in the following output:

[...]
Traceback (most recent call last):
  File "./funk.py", line 162, in <module>
    draw_gantt(planning.planning)
  File "/home/lpouilloux/.local/lib/python2.6/site-packages/execo_g5k/planning.py", line 644, in draw_gantt
    fig = PLT.figure(figsize=(15, 5 * n_row), dpi=80)
  File "/usr/lib/pymodules/python2.6/matplotlib/pyplot.py", line 254, in figure
    **kwargs)
  File "/usr/lib/pymodules/python2.6/matplotlib/backends/backend_tkagg.py", line 90, in new_figure_manager
    window = Tk.Tk()
  File "/usr/lib/python2.6/lib-tk/Tkinter.py", line 1646, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: no display name and no $DISPLAY environment variable

It could be useful to check then inform the user he can't use "-p" on grid5000 frontend.
Similarly it is not clear in the description if it is possible (or not) to execute the "-p flag" on a production node.

Add the program_to_run option

It could be interesting to add the "oargrid -p program_to_run" option as a "funk.py" option to execute a script instead of doing an interactive reservation.

add the maximum number of nodes to the list of resources for each cluster

feature intéressante pour funk: lors d'une réservation
rennes 79 parapluie: 7 paravance: 53 parapide: 0 paranoia: 1 parasilo: 18
ça serait intéressant de mettre le nombre de noeuds max qu'on peut réserver:
rennes 79/148 parapluie: 7/19 paravance: 53/72 parapide: 0/21 paranoia: 1/8 parasilo: 18/28

option -p seems not to work on multi site

When I try the following command :

funk -m max -p "/home/mzimmer/autorun" -w 10:00:00 -r nancy,lyon -c

The resulted reservation doesn't mention the -p option to oargridsub

oargridsub -v -s "2015-10-28 19:00:00" -w "10:0:0"  lyon:rdef="nodes=77":name="FUNK",nancy:rdef="nodes=204":name="FUNK"

What I am doing wrong?

If I remove nancy to reserve only on one site. It does work :

oarsub -l "nodes=77,walltime=10:0:0" -q "default" -r "2015-10-26 19:00:00" -n "FUNK" "/home/mzimmer/autorun"

Python 2 vs 3

Hello,
I noticed the code does not work with python3 (there is at least one print statement to replace).
And the shebang is #!/usr/bin/env python which may target python3 nowadays (if not the default on the OS, it is often the case when calling python from a virtualenv).
Updating the shebang to #!/usr/bin/env python2 would perhaps be a first workaround.
Thanks for your work on this
Etienne

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.