Giter Site home page Giter Site logo

pulpito's Introduction

Ceph - a scalable distributed storage system

See https://ceph.com/ for current information about Ceph.

Status

Issue Backporting

Contributing Code

Most of Ceph is dual-licensed under the LGPL version 2.1 or 3.0. Some miscellaneous code is either public domain or licensed under a BSD-style license.

The Ceph documentation is licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

Some headers included in the ceph/ceph repository are licensed under the GPL. See the file COPYING for a full inventory of licenses by file.

All code contributions must include a valid "Signed-off-by" line. See the file SubmittingPatches.rst for details on this and instructions on how to generate and submit patches.

Assignment of copyright is not required to contribute code. Code is contributed under the terms of the applicable license.

Checking out the source

Clone the ceph/ceph repository from github by running the following command on a system that has git installed:

git clone [email protected]:ceph/ceph

Alternatively, if you are not a github user, you should run the following command on a system that has git installed:

git clone https://github.com/ceph/ceph.git

When the ceph/ceph repository has been cloned to your system, run the following commands to move into the cloned ceph/ceph repository and to check out the git submodules associated with it:

cd ceph
git submodule update --init --recursive --progress

Build Prerequisites

section last updated 27 Jul 2023

Make sure that curl is installed. The Debian and Ubuntu apt command is provided here, but if you use a system with a different package manager, then you must use whatever command is the proper counterpart of this one:

apt install curl

Install Debian or RPM package dependencies by running the following command:

./install-deps.sh

Install the python3-routes package:

apt install python3-routes

Building Ceph

These instructions are meant for developers who are compiling the code for development and testing. To build binaries that are suitable for installation we recommend that you build .deb or .rpm packages, or refer to ceph.spec.in or debian/rules to see which configuration options are specified for production builds.

To build Ceph, make sure that you are in the top-level ceph directory that contains do_cmake.sh and CONTRIBUTING.rst and run the following commands:

./do_cmake.sh
cd build
ninja

do_cmake.sh by default creates a "debug build" of Ceph, which can be up to five times slower than a non-debug build. Pass -DCMAKE_BUILD_TYPE=RelWithDebInfo to do_cmake.sh to create a non-debug build.

Ninja is the buildsystem used by the Ceph project to build test builds. The number of jobs used by ninja is derived from the number of CPU cores of the building host if unspecified. Use the -j option to limit the job number if the build jobs are running out of memory. If you attempt to run ninja and receive a message that reads g++: fatal error: Killed signal terminated program cc1plus, then you have run out of memory. Using the -j option with an argument appropriate to the hardware on which the ninja command is run is expected to result in a successful build. For example, to limit the job number to 3, run the command ninja -j 3. On average, each ninja job run in parallel needs approximately 2.5 GiB of RAM.

This documentation assumes that your build directory is a subdirectory of the ceph.git checkout. If the build directory is located elsewhere, point CEPH_GIT_DIR to the correct path of the checkout. Additional CMake args can be specified by setting ARGS before invoking do_cmake.sh. See cmake options for more details. For example:

ARGS="-DCMAKE_C_COMPILER=gcc-7" ./do_cmake.sh

To build only certain targets, run a command of the following form:

ninja [target name]

To install:

ninja install

CMake Options

The -D flag can be used with cmake to speed up the process of building Ceph and to customize the build.

Building without RADOS Gateway

The RADOS Gateway is built by default. To build Ceph without the RADOS Gateway, run a command of the following form:

cmake -DWITH_RADOSGW=OFF [path to top-level ceph directory]

Building with debugging and arbitrary dependency locations

Run a command of the following form to build Ceph with debugging and alternate locations for some external dependencies:

cmake -DCMAKE_INSTALL_PREFIX=/opt/ceph -DCMAKE_C_FLAGS="-Og -g3 -gdwarf-4" \
..

Ceph has several bundled dependencies such as Boost, RocksDB and Arrow. By default, cmake builds these bundled dependencies from source instead of using libraries that are already installed on the system. You can opt to use these system libraries, as long as they meet Ceph's version requirements. To use system libraries, use cmake options like WITH_SYSTEM_BOOST, as in the following example:

cmake -DWITH_SYSTEM_BOOST=ON [...]

To view an exhaustive list of -D options, invoke cmake -LH:

cmake -LH

Preserving diagnostic colors

If you pipe ninja to less and would like to preserve the diagnostic colors in the output in order to make errors and warnings more legible, run the following command:

cmake -DDIAGNOSTICS_COLOR=always ...

The above command works only with supported compilers.

The diagnostic colors will be visible when the following command is run:

ninja | less -R

Other available values for DIAGNOSTICS_COLOR are auto (default) and never.

Building a source tarball

To build a complete source tarball with everything needed to build from source and/or build a (deb or rpm) package, run

./make-dist

This will create a tarball like ceph-$version.tar.bz2 from git. (Ensure that any changes you want to include in your working directory are committed to git.)

Running a test cluster

From the ceph/ directory, run the following commands to launch a test Ceph cluster:

cd build
ninja vstart        # builds just enough to run vstart
../src/vstart.sh --debug --new -x --localhost --bluestore
./bin/ceph -s

Most Ceph commands are available in the bin/ directory. For example:

./bin/rbd create foo --size 1000
./bin/rados -p foo bench 30 write

To shut down the test cluster, run the following command from the build/ directory:

../src/stop.sh

Use the sysvinit script to start or stop individual daemons:

./bin/init-ceph restart osd.0
./bin/init-ceph stop

Running unit tests

To build and run all tests (in parallel using all processors), use ctest:

cd build
ninja
ctest -j$(nproc)

(Note: Many targets built from src/test are not run using ctest. Targets starting with "unittest" are run in ninja check and thus can be run with ctest. Targets starting with "ceph_test" can not, and should be run by hand.)

When failures occur, look in build/Testing/Temporary for logs.

To build and run all tests and their dependencies without other unnecessary targets in Ceph:

cd build
ninja check -j$(nproc)

To run an individual test manually, run ctest with -R (regex matching):

ctest -R [regex matching test name(s)]

(Note: ctest does not build the test it's running or the dependencies needed to run it)

To run an individual test manually and see all the tests output, run ctest with the -V (verbose) flag:

ctest -V -R [regex matching test name(s)]

To run tests manually and run the jobs in parallel, run ctest with the -j flag:

ctest -j [number of jobs]

There are many other flags you can give ctest for better control over manual test execution. To view these options run:

man ctest

Building the Documentation

Prerequisites

The list of package dependencies for building the documentation can be found in doc_deps.deb.txt:

sudo apt-get install `cat doc_deps.deb.txt`

Building the Documentation

To build the documentation, ensure that you are in the top-level /ceph directory, and execute the build script. For example:

admin/build-doc

Reporting Issues

To report an issue and view existing issues, please visit https://tracker.ceph.com/projects/ceph.

pulpito's People

Contributors

alfredodeza avatar amathuria avatar andrewschoen avatar badone avatar batrick avatar charpty avatar dmick avatar kamoltat avatar kshtsk avatar liewegas avatar nitzanmordhai avatar tchaikov avatar toabctl avatar vallariag avatar vasukulkarni avatar zmc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pulpito's Issues

add scrape.log at home screen

image

we could add it next to show the description for a batch run, can help in quicker failure analysis
right now for batch like: https://pulpito.ceph.com/yuriw-2021-03-23_21:23:36-rados-wip-yuri7-testing-2021-03-23-1015-nautilus-distro-basic-smithi/5990713/
we need to go to one of the run's log and then from there to a folder above to access scrape.log:
http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-23_21:23:36-rados-wip-yuri7-testing-2021-03-23-1015-nautilus-distro-basic-smithi/

we would need to :

add a button like in screenshot named scrape with address to http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-23_21:23:36-rados-wip-yuri7-testing-2021-03-23-1015-nautilus-distro-basic-smithi/scrape.log baased url

pulpito failed to run

Hi,

I followed the instructions to set up pulpito, everything looks good until I executed 'python run.py', the following error messages were outputted.

python run.py

Traceback (most recent call last):
File "run.py", line 7, in
import prod
ImportError: No module named prod

I tried to google the issue but still get no answer, any idea on this issue?

Thanks,
Jevon

No schema supplied

[root@yujiang-teuthology ~]# curl http://192.168.100.146:8081
<html>
     <head>
      <title>Pecan - Application Error</title>
     <body>
      <header>
        <h1>
          An error occurred!
        </h1>
      </header>
      <div id="error-content">
        <p>
          
<div class="traceback">
  <b>To disable this interface, set </b>
  <a target="window"
  href="https://pecan.readthedocs.io/en/latest/deployment.html#disabling-debug-mode">
    <pre>conf.app.debug = False</pre>
  </a>
</div>

          Pecan offers support for interactive debugging by installing the <a href="https://pypi.python.org/pypi/backlash" target="window">backlash</a> package:
          <br />
          <b><pre>pip install backlash</pre></b>
          ...and reloading this page.
        </p>
        <h2>Traceback</h2>
        <div id="traceback">
          <pre>Traceback (most recent call last):
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/pecan/middleware/debug.py", line 78, in __call__
    return self.app(environ, start_response)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/pecan/middleware/errordocument.py", line 75, in __call__
    app_iter = self.app(environ, replacement_start_response)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/pecan/core.py", line 840, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/home/pulpito/github/pulpito/pulpito/controllers/root.py", line 56, in index
    latest_runs = requests.get(uri).json()
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    prep = self.prepare_request(req)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/requests/models.py", line 293, in prepare
    self.prepare_url(url, params)
  File "/home/pulpito/github/pulpito/virtualenv/lib/python2.7/site-packages/requests/models.py", line 353, in prepare_url
    raise MissingSchema(error)
MissingSchema: Invalid URL '/runs/?page=1': No schema supplied. Perhaps you meant http:///runs/?page=1?
</pre>
        </div>
        <h2>WSGI Environment</h2>
        <div id="environ">
          <pre>{'ACTUAL_SERVER_PROTOCOL': 'HTTP/1.1',
 'HTTP_ACCEPT': '*/*',
 'HTTP_HOST': '192.168.100.146:8081',
 'HTTP_USER_AGENT': 'curl/7.29.0',
 'PATH_INFO': '/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '192.168.100.146',
 'REMOTE_PORT': '55218',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'simpleapp',
 'SERVER_PORT': '8081',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'CherryPy/3.8.0 Server',
 'pecan.locals': {'request': <Request at 0x7f9b0c0b1b50 GET http://192.168.100.146:8081/>,
                  'response': <Response at 0x7f9b0c0b1cd0 200 OK>},
 'pecan.recursive.script_name': '',
 'webob._parsed_query_vars': (GET([]), ''),
 'webob.adhoc_attrs': {'context': {'filters': {}},
                       'pecan': {'content_type': 'text/html',
                                 'extension': None,
                                 'routing_path': u'/'}},
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f9b1cdfe1e0>,
 'wsgi.input': <cherrypy.wsgiserver.wsgiserver2.KnownLengthRFile object at 0x7f9b0f15f790>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}</pre>
        </div>
      </div>
     </body>
    </html>

pulpito failed

Hi,
I followed the instructions to set up paddles and pulpito. I executed 'python run.py', the following error messages were outputted.
python run.py

ValueError('No JSON object could be decoded',)
Traceback (most recent call last):
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1353, in communicate
req.respond()
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 868, in respond
self.server.gateway(self).respond()
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2267, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2477, in call
return app(environ, start_response)
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in call
return self.application(environ, start_response)
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/pecan/middleware/errordocument.py", line 75, in call
app_iter = self.app(environ, replacement_start_response)
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/pecan/core.py", line 840, in call
return super(Pecan, self).call(environ, start_response)
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/pecan/core.py", line 683, in call
self.invoke_controller(controller, args, kwargs, state)
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
result = controller(*args, **kwargs)
File "/opt/pulpito-master/pulpito/controllers/root.py", line 56, in index
latest_runs = requests.get(uri).json()
File "/opt/pulpito-master/virtualenv/local/lib/python2.7/site-packages/requests/models.py", line 805, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")

I executed "curl paddles website" got the response as below
{"help": {"docs": "https://github.com/ceph/paddles", "extensions": {"Chrome": "https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc", "Firefox": "https://addons.mozilla.org/en-US/firefox/addon/jsonview/"}}, "last_job": " ago", "last_run": " ago"}[

I tried to search the issue but still get no answer, any idea on this issue?

Thanks,
nancy

`KeyError` when there are no jobs

I am getting errors when there are no jobs available for a given 'run' when I hit this url (might work later):

http://pulpito.front.sepia.ceph.com/teuthology-2013-11-14_19:01:22-ceph-deploy-cuttlefish-testing-basic-plana/

The complete traceback:

Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/pulpito/local/lib/python2.7/site-packages/pecan/middleware/debug.py", line 289, in __call__
    return self.app(environ, start_response)
  File "/home/ubuntu/.virtualenvs/pulpito/local/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/home/ubuntu/.virtualenvs/pulpito/local/lib/python2.7/site-packages/pecan/middleware/errordocument.py", line 75, in __call__
    app_iter = self.app(environ, replacement_start_response)
  File "/home/ubuntu/.virtualenvs/pulpito/local/lib/python2.7/site-packages/pecan/core.py", line 567, in __call__
    self.handle_request(req, resp)
  File "/home/ubuntu/.virtualenvs/pulpito/local/lib/python2.7/site-packages/pecan/core.py", line 505, in handle_request
    result = controller(*args, **kwargs)
  File "/home/ubuntu/pulpito/pulpito/controllers/root.py", line 128, in index
    run = self.run or self.get_run()
  File "/home/ubuntu/pulpito/pulpito/controllers/root.py", line 119, in get_run
    for job in run['jobs']:
KeyError: 'jobs'

We should consider turning debug off as well.

Nav bar at the top

Latest Runs
Runs against:

  • branch
  • suite
  • date (calendar)
  • same branch & suite as current run
    ...

Polish up the branch dropdown

16:20:29 yanfali zackc: when you select by suite it does this long drop down, if the  
                 window height isn't super tall it won't scroll. You can probably work  
                 around this by explicitly setting the height of the screen to something
                 longer than the view port  
16:20:38 yanfali zackc: re: pulpito  
16:33:29 zackc as you can see it's not the most polished tool ever created  
16:33:37 yanfali zackc: it's nice  
16:34:17 yanfali zackc: you might also want to move the search box inside the drop down 
                 to the top, actually since that list is so small I'm not sure what the 
                 use case is  
16:36:13 zackc yanfali: try typing 'w' into it  
16:36:46 yanfali oh that's confusing…  
16:36:49 zackc it's a typeahead populated with a prefetched list of branches... the ones
               in the dropdown are the ones we know are going to be around  
16:36:54 yanfali maybe move it to conventional place?  
16:36:58 zackc yeah...  
16:37:15 yanfali like right hand side of navbar  
16:38:03 zackc that doesn't sound less confusing  
16:38:07 yanfali maybe add a spyglass icon instead as well to make it explicit because  
                 placeholder text disappears as you start typing?  
16:38:54 yanfali the usage is also odd  
16:39:03 yanfali return doesn't make it work, only clicking?  
16:39:21 yanfali ah it's just slow  
16:39:31 yanfali you might need a spinner  

async API requests

Because this was a proof of concept when I started it, I decided it was easier to just do the request to the API in the controller, before even being able to return something.

This is far from ideal to have as a permanent solution and although there is some complexity in doing ajax calls, we are already dealing with a nice API that behaves correctly (usually a big part of the problem).

The biggest advantage I see, for now, is x20 (maybe a bit more in some cases) improvement in response times.

At some point, when we decide to have more information in the page we can take advantage of that work by re-using the same data without having to talk to the controller and re-loading what already exists in the DOM. This is similar to Github where you have tabs in a pull request but clicking them do not really reload the page at all.

This ticket might be big, so it would make sense to split up if the changes take too long.

The idea is to make the controller return the most basic things it needs to render the page (like title and page headers) along with what AJAX call will need to make the request, like the resource URL for example.

We have JQuery already, this is how a simple AJAX call would look like (note that you can use Mako variables inside there!):

$.ajax({
    type: 'GET',
    contentType: "application/json; charset=utf-8",
    url: '${api_resource_url()}',
    success: function(response){
        // Use the data in the response to fill in the DIVs
    },
    error: function(){
      // Show a nice message with a default error message or
      // re-use what came back from the API
    },
});

For that to work, we need to enable CORS in the server, I already created a CORS Pecan hook (it lives in hooks/cors.py) and it just needs to be added in the app config in the hooks section.

Make runs tables more usable

For example, instead of using Run.name, use:

user | scheduled | posted/started | branch | suite | machine_type

user and machine_type need to be added as columns to paddles:
ceph/paddles#36

hide/disable `Expand` button when there is nothing to expand

When all tests are passing, there is no need to show the expand button.

Since there is nothing to expand it appears like something is not working (e.g. nothing happens when clicked). It could be either removed or simply shown as "disabled".

Bootstrap has some CSS settings that allow you to show a button but make it look disabled by changing the colors of the button and not allowing clicking on it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.