dmwm / phedex Goto Github PK

CMS data-placement suite

Perl 63.20% Python 3.28% CartoCSS 0.30% Shell 3.54% CSS 0.67% HTML 4.49% JavaScript 22.43% PLSQL 0.87% SQLPL 0.42% Perl 6 0.27% Jupyter Notebook 0.08% Dockerfile 0.03% TSQL 0.41%

phedex's Introduction

			    PhEDEx README
				   
PhEDEx stands for Physics Experiment Data Export.  It consists of:

 * A distributed agent-based transfer management workflow engine
 * A central database containing workflow state and monitoring data
 * Web services to provide management and monitoring functions
 * Command-line tools for management, monitoring, and administration
   functions

This README is intentionally brief.  For up-to-date documentation on
PhEDEx, go to the project twiki.

Twiki Documentation:
  https://twiki.cern.ch/twiki/bin/view/CMS/PhedexDraftDocumentation

Web site:
  https://cmsweb.cern.ch/phedex

Admin mailing list:
  cms-phedex-admins at cern dot ch

Bug tracker:
  https://github.com/dmwm/PHEDEX/issues

Discussion forum:
  https://hypernews.cern.ch/HyperNews/CMS/get/phedex.html

Discussion mailing list:
  hn-cms-phedex at cern dot ch

Software repository at GitHub:
  https://github.com/dmwm/PHEDEX

phedex's People

Contributors

Stargazers

Watchers

Forkers

zhangxiaomei cboeser nataliaratnikova stormlv giffels geonmo sfink garrasmelos tswu nikmagini huangch-fnal kreczko geneguvo sartiran alberto-sanchez saltumdis cronot99 bbockelm

phedex's Issues

adapt scheduler script for V2

Original Savannah ticket 948 reported by None on Thu Jul 29 03:41:56 2004.

N/A

modify T0-Update-Catlog.pl- dump files it can't find

Original Savannah ticket 942 reported by None on Mon Jul 26 07:48:28 2004.

If it can't find a file mark the drop bad straight away- hopefully castor transient errors are rare.

Links to CVS browser from PhEDEx web page Documentation.

PhEDEx Documentation at
https://cmsweb.cern.ch/phedex/documents.html
contains several references to various README files from PhEDEX source tree via CVS browser, e.g.:
http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/PHEDEX/Documentation/README/README-Deployment.txt?revision=HEAD
As CVS access has been changed to read-only, documentation needs to be moved to some other place, where it could be updated if necessary.

define deploy dir structure for a node

Original Savannah ticket 917 reported by barrass on Thu Jul 22 13:56:42 2004.

N/A

eg03_tt_2e2mu for FNAL

Original Savannah ticket 954 reported by None on Fri Jul 30 11:03:57 2004.

eg03_tt_2e2mu

hg03_wbwb_2l_alpgen for FNAL

Original Savannah ticket 966 reported by None on Fri Jul 30 11:10:03 2004.

hg03_wbwb_2l_alpgen

allocation agent accesses V2 tmdb

Original Savannah ticket 913 reported by barrass on Thu Jul 22 13:35:16 2004.

update SQL according to schema doc

modify existing INFN transfer agent

Original Savannah ticket 918 reported by bonacor on Thu Jul 22 13:58:50 2004.

N/A

adapt scheduler web script for V2

Original Savannah ticket 925 reported by metson on Thu Jul 22 14:09:49 2004.

N/A

Link-creation should not be allowed between MSS and non-local nodes

the link creation tool currently doesn't check if it's creating a link between an MSS node and a non-local node. It should check and prompt the user (or abort) if the MSS node is linked to anything other than the local Buffer node.

deploy and test data sink (disk) node at FNAL

Original Savannah ticket 929 reported by None on Thu Jul 22 14:12:32 2004.

N/A

Data-service returns error strings with no regard to user-specified formats

The data-service currently returns error strings in a format which is not consistent with the user-specified 'Accept' header. It would be good to fix this

modify example transfer agent for RAL

Original Savannah ticket 921 reported by barrass on Thu Jul 22 14:01:12 2004.

N/A

modify existing FNAL transfer agent

Original Savannah ticket 920 reported by None on Thu Jul 22 13:59:57 2004.

N/A

hg03b_hww_2l_* for FNAL

Original Savannah ticket 958 reported by None on Fri Jul 30 11:06:55 2004.

hg03b_hww_2l_120
130
140
150
160
165
170
180
190
200

FileRemove or FileDownload fail badly if the 'delete' or 'validate' scripts do not exist

if the 'delete' script doesn't exist then FileRemove seems to spiral out of control, forking processes until the machine dies. This is not the desired behaviour!

At the very least, it should check if the file exists and is executable and die if not.

deploy single machine testbed for example agents on lxgate04

Original Savannah ticket 941 reported by barrass on Mon Jul 26 05:01:25 2004.

deploy and test in this order:

simple sink of data- must use cp, must successfully transfer a file manually advertised in dev tmdb.
simple allocation agent + simple sink: allocationa gent must successfully allocate manually entered data.
simple source + allocation + sink: full chain test.

allocation agent in cvs

Original Savannah ticket 912 reported by barrass on Thu Jul 22 13:34:32 2004.

Needs new Managers submodule

hg03_h2tau_2l_* for FNAL

Original Savannah ticket 968 reported by None on Fri Jul 30 11:11:01 2004.

hg03_h2tau_2l_140
200
250

project homepage

Original Savannah ticket 924 reported by metson on Thu Jul 22 14:09:08 2004.

get cern webspace
logo?
collect talks together
collect docs together
link to savannah
link to cvs
link to mailing list archive
link to admin
link to system state browser (when available!)

deploy and test data sink (disk) node at INFN

Original Savannah ticket 928 reported by bonacor on Thu Jul 22 14:12:06 2004.

N/A

adapt transfer state summary script for V2

Original Savannah ticket 926 reported by lat on Thu Jul 22 14:10:19 2004.

N/A

deploy and test data sink (disk) node at PIC

Original Savannah ticket 927 reported by hernan on Thu Jul 22 14:11:28 2004.

N/A

update schema doc

Original Savannah ticket 910 reported by barrass on Thu Jul 22 13:33:24 2004.

some small changes- subscriptions table, catalogue contact strings. check db schema, work around.

hg03_hzz_2e2mu_* for FNAL

Original Savannah ticket 951 reported by barrass on Fri Jul 30 11:01:33 2004.

hg03_hzz_2e2mu_115a
hg03_hzz_2e2mu_120a
hg03_hzz_2e2mu_130a
hg03_hzz_2e2mu_140a
hg03_hzz_2e2mu_150a

eg03_zz_2e2mu for FNAL

Original Savannah ticket 955 reported by barrass on Fri Jul 30 11:05:10 2004.

eg03_zz_2e2mu

eg03_hzz_2e2mu_* for FNAL

Original Savannah ticket 952 reported by barrass on Fri Jul 30 11:02:24 2004.

eg03_hzz_2e2mu_160
170
180
190
200
250
300
350
400
450
500
550
600

Agents API needs updating for the post-CVS era

The PHEDEX::Web::Agents API refers to the CVS version of files. It needs to be updated now, to allow for the GIT version instead, if that is even feasible.

Strictly, it should support the CVS version scheme as well as the GIT version scheme, but since CVS won't be with us forever we can skip that completely.

At the very least, the references to CVS should be expunged from the documentation.

modify existing IN2P3 transfer agent

Original Savannah ticket 922 reported by semenjuk on Thu Jul 22 14:01:41 2004.

N/A

hg03_wbwb_2l_madgr for FNAL

Original Savannah ticket 967 reported by None on Fri Jul 30 11:10:27 2004.

hg03_wbwb_2l_madgr

need a simple node manager

Original Savannah ticket 916 reported by None on Thu Jul 22 13:54:32 2004.

needs to read tmdb config messages to determine whether to start or stop agents- also needs to be deployed as a cron job to make sure agents come back online after machine reboot etc.

eg03_hzz_2e2mu_cpodd_* for FNAL

Original Savannah ticket 953 reported by barrass on Fri Jul 30 11:03:12 2004.

eg03_hzz_2e2mu_cpodd_200
300
400

mods to command line options on example agents

Original Savannah ticket 940 reported by lat on Mon Jul 26 04:57:53 2004.

We need to pass db username and password on command line so they don't appear in cvs.

Might be nice to specify db contact string on command line as well- maybe allow access to mysql tmdb, etc- but this is really nonessential.

eg03_wt_2l_toprex for FNAL

Original Savannah ticket 961 reported by None on Fri Jul 30 11:08:06 2004.

eg03_wt_2l_toprex

hg03_ww_2l for FNAL

Original Savannah ticket 959 reported by None on Fri Jul 30 11:07:29 2004.

hg03_ww_2l

develop autmated dropper between RefDB and PhEDEx

Original Savannah ticket 937 reported by lat on Fri Jul 23 09:31:53 2004.

GetAsId.py
RefDDrops
RefDBGenSmry
touch-go.pl

rough algo is

> 1.Know list of datasets/owners of interest
> 2.Run GetAsId.py for each dataset, owner
> 2a.Returns ids:
> [Means files are available at CERN?]
> Run RefDBDrop ... To generate drops for those ids
> Run ReDBGenSummary ... To generate summary files
> Copy drops to distribution inbox
> Touch file "go" in each drop
> 2b.Doesn't return (all) ids
> [Means files aren't ready at CERN yet?]
> Ignore 'til next time

Dumb, allocate from source
Less dumb, track back from dest and find nearest replica