cgru/cgru

CGRU - AFANASY

cgru's Issues

user last job and last activity time

Afserver should store user last submitted job time and last activity time. This can help to reduce total jobs number as you can easy find "old" user and all his jobs.

keeper - version check, update message, auto update (mswin only)

It will be very good if keeper will get server version periodically. And indicate somehow to user that client is old, and new version is available.
ps
it will get version of a local afanasy server, of course, so whole company can use very old cgru version generally without update indication, and it all needed no notify user(admin) that some client is not updated

User names on nodes

Hi,

(starting a new tread here since it's a different topic.)

All my linux boxes are set up with the same user name for each artist. Is there a way to set a custom user in anafasy?
At the moment i get all the machines registering as camper (our main user acct for artist workstations).

I also get the "Warning, you don't exist" message in the bottom of the afwatch window... does this correlate with the above issue of the same user names?

Peter.

Blender cycles render? How ?

Working on windows 7 and with AE and Blender, AE is running fine, but i cant get blender to work properly ? And where do I select render engine ?
Anyone know ? ;) Thx in advance!

afcmd rlist - see online/offline

afcmd rlist - useful to show online/offline status.

afserver job solving - no recursion.

Now afserver solves only the first block, in a recursive function. It was created so to simplify system, and not to store whether a task is waiting some other or not (no depend state). Recursive function simple returns "nearest" ready task.
But if job is very complex (100blocks*1000tasks - very rare case) it can slow down server. Better to check dependence in some other function, and store task depend state. And on solve simple skip waiting dependence task (no recursive).

Ubuntu 12.0.4 Issue Pyside/PyQt

I don't know whats wrong here, i've got another machine just like this one and it works like a charm.

I'm getting this output.

CGRU_VERSION 1.6.8 : '/opt/cgru'
Traceback (most recent call last):
File "/opt/cgru/lib/python/cgrupyqt.py", line 9, in
PythonQt = import('PySide', globals(), locals(), List, -1)
ImportError: No module named PySide

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/cgru/utilities/keeper/keeper.py", line 14, in
from cgrupyqt import QtGui
File "/opt/cgru/lib/python/cgrupyqt.py", line 12, in
PythonQt = import('PyQt4', globals(), locals(), List, -1)
ImportError: No module named PyQt4

Any ideas?

Copy existing job with a new frame range.

Afserver should have an ability to copy specified job with a new frame range.

GUIs (json protocol) should have a command to copy a job and what parameters should be changed.
Or may be better, client can get job json data, change it and submit a new job - this case is flexible and needs no changes on server side.

Slow render an network but not in local (Windows)

Hello there,

I've an strange problem: When i send a mentalray job to Afanasy from afstarter, all tasks running on Windows are very very long to finish. I try the same command that afanasy send to render in local and the render is fine in term of speed.
I've try to modify the maya mentalray part of afjob.py to be less verbose: replace -v 5 to -v 4 and the problem gone. But if I do that, parser don't work anymore...

I don't have this problem on macosx.

Rendering from CentOS nodes

Hi,

When I kick off a render to the farm, the centos machines seem to hang on their first frame while the ubuntu machines work as they should.

I checked through the afrender logs, and the centos logs stop when they are about to go to "Available addresses:", basically the machines just hang there and it's not telling afserver something wrong because the job hangs indefinitely.

Any ideas?

Peter.

Parser finds out "activity" and it is a new task progress parameter.

Parser can find task "activity", and send that string to server and later to GUIs.
Activity should be some useful string to notify user current task state.
For example is it a shadow pass, or stereo right eye, or some other node being processed, or encoding starter and so on.
Already implemented in:
https://github.com/AlbertR/cgru
And should be merged in the main brunch.

One more task progress parameter means compatibility loose with previous versions, so it should be 1.7.0 (at least).

Postgres

Nwm, i just missed a package..

OS and enviroment requirments

Hi,
I got some issues that like these:
1.[root@blender2 Desktop]# blender --engine help
connect failed: No such file or directory
found bundled python: /usr/blender-2.65a-linux-glibc211-x86_64/2.65/python
Blender Engine Listing:
BLENDER_RENDER
BLENDER_GAME
CYCLES
[root@blender2 Desktop]#

2.[root@blender1 bin]# ./afrender
Creating folder:
/root/.cgru
Compilation date = 'Dec 21 2012 14:40:18'
CGRU version = '1.6.10'
Afanasy build revision = '62879bdf9b7f00967b839747d40bcbf088af9a3f'
Python version = '2.6.6'
GCC version = '4.4.6'
Config file does not exist:
/opt/cgru/config.json
Config file does not exist:
/root/.cgru/config.json
Network Interfaces:
eth0: fa:16:3e:42:5d:c9
IPv4 Address: 192.168.152.2
IPv6 Address: fe80::::f816:3eff:fe42:5dc9
unable to find statistics about 'sda' in '/proc/diskstats'.unable to find statistics about 'sda' in '/proc/diskstats'. ON
3. I can't find addon about blender in the folder /opt/cgru/plugin
[root@blender1 plugins]# ll
total 24
drwxrwxrwx 3 root root 4096 Jan 14 11:11 c4d
drwxrwxrwx 3 root root 4096 Jan 14 11:11 houdini
drwxrwxrwx 2 root root 4096 Jan 14 11:11 max
drwxrwxrwx 5 root root 4096 Jan 14 11:11 maya
drwxrwxrwx 4 root root 4096 Jan 14 11:11 nuke
drwxrwxrwx 2 root root 4096 Jan 14 11:11 xsi
My OS and Tools are like these :
OS : Centos6.3 x86_64
Gcc: gcc version 4.4.6 20120305 (Red Hat 4.4.6-4)
Pytond: [root@blender1 plugins]# python -V
Python 2.6.6

Blender : [root@blender2 Desktop]# blender -v
Blender 2.65 (sub 0)
build date: 2012-12-19
build time: 13:10:31
build revision: 53177
build platform: Linux:64bit
build type: Release
build c flags: -pipe -fPIC -funsigned-char -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -fopenmp -DNDEBUG -O2 -msse -msse2 -DWITH_MOD_FLUID -DWITH_MOD_OCEANSIM -D__LITTLE_ENDIAN__ -DWITH_AUDASPACE -DWITH_AVI
build c++ flags: -pipe -fPIC -funsigned-char -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -fopenmp -fpermissive -D__STDC_CONSTANT_MACROS -DNDEBUG -O2 -msse -msse2 -DWITH_MOD_FLUID -DWITH_MOD_OCEANSIM -D__LITTLE_ENDIAN__ -DWITH_AUDASPACE -DWITH_AVI
build link flags: -lrt
build system: SCons

Could you help me out those issues ? I wonder to finished Afansy test within this week, however, at starting, I encountered those issues. :(

Keeper hangs on new network protocol version.

Keeper does check message for protocol version. So it just hangs up on misunderstanding binary data.

afwatch - hide finished jobs

If user want to keep finished jobs for some store, he can have a huge list after some time. There should be an ability to hide them.

afwatch - sort&filder control should be more clear

There is a controls just above any nodes list to sort and filter nodes - people even does not see it. And it is very difficult to understand it.

afcmd render service on/off

To turn on or off render service via a command line.
"afcmd rsrvon|rsrvoff rendername|mask" - render service on or off, render name or regexp pattern

Non sequential tasks starting

Tasks block should have a parameter to start tasks not in a sequential manner, bit always in the "middle":
1-100: 1, 100, 50, 25, 75...
Can be very helpful to determine errors and estimate average run time.

Houdini path environment

Hi, I have a question regarding how afanasy deals with the houdini environment. In our studio some of the artists are still on H11 due to the unstable state of H12. If they send off a render through afanasy, it seems as if the nodes are picking up the houdini that was set locally in the "set houdini" script from the keeper. Would this be a correct assumption?

We have a site-wide install from a shared nfs folder, which holds both h11, h12-stable and h12-daily... any way we can make it so that the nodes fire up the correct version of houdini by sensing what version was used when it initially was sent to the farm?

Thanks,

Peter.

afanasy python parser finish a task with success

Afanasy python parser should be able to finish a task with success. It can decide to kill (terminate) a task and make afrender to send finished success status to server.

SQL: is it the right tool for the job ?

This is not an issue per say but some thoughts about SQL and why I think that in the long run it would be best not to use it! :)

Here is why I think SQL is not so good for Afanasy:

1- You need to install an SQL server to keep the state of the render farm manager! This is a big issue because each time you reply on something to provide some feature you loose some clients.

2- SQL is meant for heavy duty transactions. From what I can see, we have a very simple data base model :) Actually, many large companies are moving away from SQL (including google, notably to NoSQL solutions) because SQL is overkill for most cases.

If we want to try this, I can implement a very simple and efficient file-based system to store jobs/clients/machines.

-- aghiles

1.6.7 Afanasy server hangs ( there is no 1.6.7 release yet ).

( there is no 1.6.7 release yet )
1.6.7 Server testing failed.
It hangs every several days in our company.
So 1.6.7 release is delayed till a bug will be caught and fixed.
Do not use 1.6.7 from sources at work.

btw. We never saw 1.6.6 hung. Do not worry.

osx installer package needed

My farm is mostly built with 10.6.8 Snow Leopards, about 15 nodes. Would love to use CGRU's Afanasy as my renderfarm manager. Yet building sources by hand and paying for an older 'xcode' version just to get the AFrender going is what's keeping me from deploying the manager. Would it be possible to make a OSX package for easy OSX installation?

Blank Jobs database

Hi,

I'm currently setting up my render nodes, and I noticed that when I send a job, nobody can see it in the jobs area of afwatch. Only when you enter GOD mode, it shows up. I have tried afcmd db_resetall, but I'm getting some errors when i do that, not sure if they are of importance:

[root@blackhawk local]# afcmd db_resetall
DROP TABLE blocks CASCADE;
NOTICE: drop cascades to 2 other objects
DETAIL: drop cascades to constraint tasks_id_job_fkey on table tasks
drop cascades to constraint progress_id_job_fkey on table progress
DROP TABLE users CASCADE;
DROP TABLE progress CASCADE;
DROP TABLE renders CASCADE;
DROP TABLE statistics CASCADE;
DROP TABLE jobs CASCADE;
DROP TABLE tasks CASCADE;
AFERROR: SQL Drop table failed: ERROR: table "progress" does not exist

AFERROR: SQL Drop table failed: ERROR: table "tasks" does not exist

AFERROR: SQL Drop table failed: ERROR: table "blocks" does not exist

AFERROR: SQL Drop table failed: ERROR: table "jobs" does not exist

NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "jobs_pkey" for table "jobs"
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "blocks_pkey" for table "blocks"
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "tasks_pkey" for table "tasks"
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "progress_pkey" for table "progress"
AFERROR: SQL Drop table failed: ERROR: table "users" does not exist

NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "users_pkey" for table "users"
AFERROR: SQL Drop table failed: ERROR: table "renders" does not exist

NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "renders_pkey" for table "renders"
AFERROR: SQL Drop table failed: ERROR: table "statistics" does not exist

Peter.

Windows Issues

Hi again, I'm back ;)

This time I've really tried to look around in the documentation before I start pestering you with questions again.
I'm working on setting up the windows machines currently, and I'm having some problems starting keeper.

From a shell in windows, I have tried to run almost every setup_* file I can find, I have a feeling it must be due to the python environment, but not sure. When I run start.cmd I get "\cgru\lib\python was unexpected at this time."

I have no idea what that means...

Peter.

Studios & Projects

Lets add something to this page:

http://cgru.info/#about.html

(note, that projects and studios are not connected)

subframe task depend

If a block has "depend_sub_task" parameter on, and other block that depends on it has >1 frames_per_task, it does not work as expected. Seems that frames_per_task parameter is just ignored, it behaves as it always == 1.

aftalk

By historical reason, all users in our studio have the one, similar login name. Hence -> in users scroll list, they are visible as one .
So, it would be nice to separate users by host name (user@host), like in taskbar info. Or even better, allow to add nickname somewhere in aftalk inerface.

Afanasy: Support for thumbnails of frames

1. Create a framework for uploading thumbnails for the job. Either per task or per block.
2. Create file structure of uploaded files perhaps even on a seperate server to not stress the server too much. Suggest using WEBDAV for file upload.
3. Support for "film_strip" images playable within the browser. See https://github.com/shotgunsoftware/python-api/wiki/Reference%3A-Methods#wiki-upload_filmstrip_thumbnail for reference.

Problem connecting to PGSQL

Hi, first I'd like to thank the developers of cgru, it's an amazing software! Not sure why there is no press about it, it truly is a very handy tool in a CG department, with dailies etc. Fantastic work!

Now for the question/issue. I'm building a new afserver for our farm, I had it working pretty good on ubuntu before, but since I'm a die hard centos fan, I'd like to get it running on that instead. I can build all the targets without any problem, however, after I have setup my postgres database (followed the afanasy instructions to the dot), I try to run afcmd db_check, and I get:

AFERROR: afanasy.cmd.Check: Database connect failed: FATAL: Ident authentication failed for user "afadmin"

Database connection is NOT WORKING !

I have messed around with this now for 2 days, and I just can't seem to find a solution. I have tried the "trust" method, which I gather is just to replace the password line with trust in the pg_hba.conf file and restart pgsql? Didn't work.

I also tried the prebuilt centos package, but it's the same deal... can't connect.

Also, the afcmd db_drivers is not a recognized command according to afcmd... old documentation?

Cheers!

Peter.

Can afrender be compiled to run on a 32 bit version of Ubuntu?

First - CGRU is great! Thanks for creating this wonderful project!

Second - Can I use a 32 bit Ubuntu machine as an afrender client? I cannot get it to compile, is there something I need to set or is it just not compatible?

Thanks!

Job log - error hosts reset.

If somebody resets job error|avoid hosts (blacklist) it should be in stored in job log.

Wake-On-Lan: Automatic sleep and wake up.

Render should sleep after special time with no task (wol idle sleep time).
Server should wake sleeping renders when there are enough ready tasks.

mac os

mac os cgru setup file

Pathmap ?

Hate to be a pain in the ass but what am i doin wrong here ?

\server\projects\ /mnt/prj/

s:\ /mnt/s/

All my projects aso are in the samba share mounted as S: in windows.
I mounted the same share under /mnt/s/ on my ubuntu-slave.

Thx in advance!

Afanasy starter and negative frame values.

It should work with negative frame values.

afcmd.exe - Entry Point Not Found

The Procedure entry point inet_ntop could not be located in the dynamic link library WS2_32.dll .This Is Window XP 64bit OS.

afwatch renders list - show task user in normal and small size

Task user is an important information, better to show it somehow anyway (not to hide on small items size).

render single frame with multiple machines in afanasy

1.Question? is it possible to render single frame with multiple machines in Afanasy.
2.Question? is host means a machine.?
3.Question? I hv gone through the docs but please can anybody explain difference between a JOB, a TASK, a BLOCK, & a HOST.

All config moved to JSON.

All CGRU config moved to JSON. There is no XML in the project now.
It refers to paths map farm setup too.

http://cgru.github.com/#configuration.html

You should reconfigure the next version (>1.6.6)!

Isue a markup language instead of HTML for docs.

I suggest this one for afanasy:

http://sphinx.pocoo.org/

-- aghiles

afwatch preview task file(s) in a non-numeric block

Now afwatch has full info only for blocks, and tasks has only names in non-numeric blocks.
So if block is non-numeric it should request full info of a selected task from server to preview file(s).

enhancing CGRU

question 1 : Is it possible to add more feature in submit job for maya &
Aftereffects like getting cam list for maya from reading .ma file and any
such features like deadline has.

question 2 : Is it possible to shoot render directly from maya.

question 3 : Is it possible to submit job frames in non sequential form
like(1,5,12-30,59).

question 4 : Also how to change or edit the frame range of submitted job
from "watch".

afwatch renders panel and service with no icon

afwatch renders panel shows and empty line for task with no service icon.

For now, you can create some icon for any custom service.
( on the machine where afwatch runs, in cgru/icons/software folder should be some [service name].png )

OS and enviroment requirments

Hi,
I got some issues that like these:
1.[root@blender2 Desktop]# blender --engine help
connect failed: No such file or directory
found bundled python: /usr/blender-2.65a-linux-glibc211-x86_64/2.65/python
Blender Engine Listing:
BLENDER_RENDER
BLENDER_GAME
CYCLES
[root@blender2 Desktop]#

2.[root@blender1 bin]# ./afrender
Creating folder:
/root/.cgru
Compilation date = 'Dec 21 2012 14:40:18'
CGRU version = '1.6.10'
Afanasy build revision = '62879bdf9b7f00967b839747d40bcbf088af9a3f'
Python version = '2.6.6'
GCC version = '4.4.6'
Config file does not exist:
/opt/cgru/config.json
Config file does not exist:
/root/.cgru/config.json
Network Interfaces:
eth0: fa:16:3e:42:5d:c9
IPv4 Address: 192.168.152.2
IPv6 Address: fe80::::f816:3eff:fe42:5dc9
unable to find statistics about 'sda' in '/proc/diskstats'.unable to find statistics about 'sda' in '/proc/diskstats'. ON
3. I can't find addon about blender in the folder /opt/cgru/plugin
[root@blender1 plugins]# ll
total 24
drwxrwxrwx 3 root root 4096 Jan 14 11:11 c4d
drwxrwxrwx 3 root root 4096 Jan 14 11:11 houdini
drwxrwxrwx 2 root root 4096 Jan 14 11:11 max
drwxrwxrwx 5 root root 4096 Jan 14 11:11 maya
drwxrwxrwx 4 root root 4096 Jan 14 11:11 nuke
drwxrwxrwx 2 root root 4096 Jan 14 11:11 xsi
My OS and Tools are like these :
OS : Centos6.3 x86_64
Gcc: gcc version 4.4.6 20120305 (Red Hat 4.4.6-4)
Pytond: [root@blender1 plugins]# python -V
Python 2.6.6

Blender : [root@blender2 Desktop]# blender -v
Blender 2.65 (sub 0)
build date: 2012-12-19
build time: 13:10:31
build revision: 53177
build platform: Linux:64bit
build type: Release
build c flags: -pipe -fPIC -funsigned-char -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -fopenmp -DNDEBUG -O2 -msse -msse2 -DWITH_MOD_FLUID -DWITH_MOD_OCEANSIM -D__LITTLE_ENDIAN__ -DWITH_AUDASPACE -DWITH_AVI
build c++ flags: -pipe -fPIC -funsigned-char -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -fopenmp -fpermissive -D__STDC_CONSTANT_MACROS -DNDEBUG -O2 -msse -msse2 -DWITH_MOD_FLUID -DWITH_MOD_OCEANSIM -D__LITTLE_ENDIAN__ -DWITH_AUDASPACE -DWITH_AVI
build link flags: -lrt
build system: SCons

Could you help me out those issues ? I wonder to finish Afansy test within this week, however, at starting, I encountered those issues. :(
I don't know how I assign this issue to a right person yet. :(

Can't find render node

Sir,
I have a issue with afanasy in centos 6.2. Release version is 1.6.5. I installed afserver and afrender in a same server. I also installed afrender in the other server. Two servers can see eachother with ping test. However, I moniter all things with webvisor, I can only see one render that is installed with afserver together.
Afserver/afrender ip: 192.168.90.51
other afrender IP: 192.168.90.
I checked all respect of services. They are running well.
Could you tell me what problem is and how to do troubleshooting though.

Thanks

Some users jobs can't produce any tasks in 1.6.5.

Users added by "afcmd uadd" can't have jobs running. It happened in 1.6.5 after movement all solving code from libafanasy to afserver.
Do not add users by afcmd. It is more easy to added them in afwatch GUI in admin mode.
If you has such users, you can simple restart server, as users added from database works fine.

MS Windows: afserver can't create temp folder in non-existent path

It should try to create all parent folders.

Web Visor Dependencies

Hi, does the Web Visor need any dependencies for the chart creation?
If I go to statistics and press a "chart" link, it gives me a broken image icon.
Do I need a library installed for the proper creation?

Peter.

Documentation

New documentation is here http://cgru.info