dasch-swiss/sipi

Simple Image Presentation Interface

sipi's Issues

Error when importing tiffs

After some images that were imported via route convert_from_binaries

terminate called after throwing an instance of 'Sipi::JpegError'
  what():  Application transferred too few scanlines
Aborted (core dumped)

Find thread safe alternative for Lua requests module

The problem occurs when executing the preflight request (Lua function): each time it is called, the Lua module requests is loaded (require "requests"). This module or one of its dependencies does not seem to be thread safe.

Either we find an other Lua module to make a GET HTTP requests with cookie support, or we have to implement that ourselves in C.

We could try first these: https://luarocks.org/search?q=http

Change 'luarocks' system tree path to something local to sipi

After discussing it with @lrosenth , the current usage of the system path /usr/local/luarocks for the luarocks system tree is not intended. It should be changed to something local to the sipi directory, e.g., sipi/local/luarocks

Eliminate extra status code

When Sipi returns JSON in response to an HTTP request, it returns two status codes:

1. The standard HTTP status code.
2. Another status code inside the JSON.

This is confusing and error-prone, and seems redundant. There are cases where Sipi returns HTTP status 200 (OK), but the JSON contains a status code indicating an error: see dasch-swiss/dsp-api#266 .

Can we get rid of the error code inside the JSON, and just use HTTP status codes?

Required Lua modules should be installed during the build process

The script sipi.init-knora.lua requires the Lua modules requests and json, but these are not installed by default. Since they are used in code that's part of Sipi, they should be installed by Sipi's build process.

openssl check does not work in cmake

I forgot to install openssl-develon CentOS 7 but cmake did not complain. Then during the build, it failed (header files were missing).

Isn't cmake supposed to find out whether openssl-devel is missing?

Docker

I would like to add a docker build definition, so that it can be easy to spin up a SIPI server. This would also be used in Knora to automatically create the whole infrastructure for testing.

Data directory /tmp/webapi_tmp/ is not created

After installing Sipi and trying to run it with Knora, this error occurs:

Knora API answered with an error:

500 Server Error: Internal Server Error
{"status":4,"error":"org.knora.webapi.FileWriteException: Data directory /tmp/webapi_tmp/ does not exist on server”}

Either this directory should be created by the installation process, or the README should tell the user to create it before running Sipi.

cc @sofiag

Sourceforge is unreliable

We're sorry -- the Sourceforge site is currently in Disaster Recovery mode, and currently
requires the use of javascript to function. Please check back later.

[  8%] Performing download step (download, verify and extract) for 'project_xz'
-- downloading...
     src='http://tukaani.org/xz/xz-5.2.1.tar.gz'
     dst='/Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/xz-5.2.1.tar.gz'
     timeout='none'
-- [download 100% complete]
-- downloading... done
-- verifying file...
     file='/Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/xz-5.2.1.tar.gz'
-- verifying file... warning: did not verify file - no URL_HASH specified?
-- extracting...
     src='/Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/xz-5.2.1.tar.gz'
     dst='/Users/benjamingeer/git/Sipi/lib/xz-5.2.1'
-- extracting... [tar xfz]
CMake Error: Problem with archive_read_open_file(): Unrecognized archive format
CMake Error: Problem extracting tar: /Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/xz-5.2.1.tar.gz
-- extracting... [error clean up]
CMake Error at /Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/project_xz-stamp/extract-project_xz.cmake:33 (message):
  error: extract of
  '/Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/xz-5.2.1.tar.gz'
  failed


make[2]: *** [ext/xz/project_xz-prefix/src/project_xz-stamp/project_xz-download] Error 1
make[1]: *** [ext/xz/CMakeFiles/project_xz.dir/all] Error 2
make: *** [all] Error 2
iml-wall-e:build benjamingeer$ cat /Users/benjamingeer/git/Sipi/build/ext/xz/project_xz-prefix/src/xz-5.2.1.tar.gz
<html><head>
<title>SourceForge</title>
<!-- <script src="/js/jquery.com/jquery-1.11.0.min.js"></script> -->
<script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
<script src="//sourceforge.net/js/mirrors.js"></script>
<script src="/js/sf.js"></script>
<script>
var DR_loc = DR_parse_hash_url();
if (DR_loc) {
    DR_sf_main(DR_loc);
} else {
    window.location.href = 'http://sourceforge.net/home.html';
}
</script>
</head><body>
<noscript>
We're sorry -- the Sourceforge site is currently in Disaster Recovery mode, and currently requires
the use of javascript to function.  Please check back later.
</noscript>
</body></html>

Let's stop depending on Sourceforge. Can we mirror all these libraries on our own server?

Clang Compiler Version

Do we really need Clang Version >= 7.3?

The build ran through under OSX with Clang 6.0: https://travis-ci.org/dhlab-basel/Sipi/jobs/129059086

Build stops, says file already exists (on Mac OS X)

I did this as instructed in the README:

cd build
rm -rf * ../lib ../local
cmake ..
make

And the build stopped with this:

Performing patch step for 'project_kakadu'
The next patch would create the file make/Makefile-MAC-x86-64-gcc,
which already exists!  Assume -R? [n] 

'gettext' - missing dependency in documentation

Problem

The build process fails with the following error if gettext is not installed:

libtool: link: g++ -o ../bin/.libs/exiv2 exiv2.o actions.o utils.o  -ldl -L/usr/src/sipi/local/lib -L/usr/src/sipi/lib/exiv2-0.25/xmpsdk/src ./.libs/libexiv2.so -Wl,-rpath -Wl,/usr/src/sipi/local/lib
rm -f bs.gmo && : -c --statistics -o bs.gmo bs.po
mv: cannot stat 't-bs.gmo': No such file or directory
Makefile:128: recipe for target 'bs.gmo' failed
make[5]: *** [bs.gmo] Error 1
Makefile:158: recipe for target 'stamp-po' failed
make[4]: *** [stamp-po] Error 2
Makefile:60: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/project_exiv2.dir/build.make:109: recipe for target 'project_exiv2-prefix/src/project_exiv2-stamp/project_exiv2-build' failed
make[2]: *** [project_exiv2-prefix/src/project_exiv2-stamp/project_exiv2-build] Error 2
CMakeFiles/Makefile2:91: recipe for target 'CMakeFiles/project_exiv2.dir/all' failed
make[1]: *** [CMakeFiles/project_exiv2.dir/all] Error 2
Makefile:117: recipe for target 'all' failed
make: *** [all] Error 2

This issue is fixed in Exiv2 0.26 (we are downloading and building Exiv2 0.25): http://dev.exiv2.org/issues/1099

Solution

Gettext needs to be installed beforehand, e.g. apt-get install gettext

Segmentation Fault when importing tif Images

This happens when tifs are imported through Knora.

Starting program: /home/tobi/gitreps/Sipi_github/local/bin/sipi -config config/sipi.knora-config.lua
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Sipi Version 1.0 Beta (build 2016-06-30 09:08)
[New Thread 0x7ffff2d6e700 (LWP 19824)]
[New Thread 0x7ffff256d700 (LWP 19939)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff256d700 (LWP 19939)]
0x0000000000632ac8 in Exiv2::getUShort(unsigned char const*, Exiv2::ByteOrder) ()
(gdb) backtrace
#0  0x0000000000632ac8 in Exiv2::getUShort(unsigned char const*, Exiv2::ByteOrder) ()
#1  0x00000000005e754b in Exiv2::ValueType<unsigned short>::read(unsigned char const*, long, Exiv2::ByteOrder) ()
#2  0x0000000000559e87 in Sipi::SipiExif::addKeyVal (this=0x7fffe4018150, tag=34855, groupName="Photo", vals_p=0x7fffe40096d0, len=1095761921)
    at /home/tobi/gitreps/Sipi_github/src/metadata/SipiExif.cpp:259
#3  0x00000000005766fb in Sipi::SipiIOTiff::readExif (this=0xe2a970, img=0x7fffe4017790, tif=0x7fffe4017850, exif_offset=2644)
    at /home/tobi/gitreps/Sipi_github/src/formats/SipiIOTiff.cpp:1117
#4  0x0000000000572d96 in Sipi::SipiIOTiff::read (this=0xe2a970, img=0x7fffe4017790, filepath="/tmp/webapi_tmp/tmp_7753901786558647407.bin", region=0x0, size=0x0, 
    force_bps_8=false) at /home/tobi/gitreps/Sipi_github/src/formats/SipiIOTiff.cpp:564
#5  0x000000000056518f in Sipi::SipiImage::read (this=0x7fffe4017790, filepath="/tmp/webapi_tmp/tmp_7753901786558647407.bin", region=0x0, size=0x0, force_bps_8=false)
    at /home/tobi/gitreps/Sipi_github/src/SipiImage.cpp:222
#6  0x00000000005ac923 in Sipi::SImage_new (L=0x7fffe40056c8) at /home/tobi/gitreps/Sipi_github/src/SipiLua.cpp:313
#7  0x000000000092ada0 in luaD_precall ()
#8  0x0000000000935d0d in luaV_execute ()
#9  0x000000000092b157 in luaD_call ()
#10 0x000000000092a71c in luaD_rawrunprotected ()
#11 0x000000000092b40d in luaD_pcall ()
#12 0x0000000000928bd1 in lua_pcallk ()
#13 0x00000000008f44d8 in shttps::LuaServer::executeChunk (this=0x7ffff2569de0, 
    luastr="--\n-- Copyright © 2016 Lukas Rosenthaler, Andrea Bianco, Benjamin Geer,\n-- Ivan Subotic, Tobias Schweizer, André Kilchenmann, and André Fatton.\n-- This file is part of Sipi.\n-- Sipi is free softwar"...) at /home/tobi/gitreps/Sipi_github/lib/shttps/LuaServer.cpp:2300
#14 0x00000000008fa490 in shttps::ScriptHandler (conn=..., lua=..., user_data=0x7fffffffd2c0, hd=0xe5d868) at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:131
#15 0x00000000009001b9 in shttps::Server::processRequest (this=0x7fffffffd2c0, sock=5, ins=0x7ffff256cda0, os=0x7ffff256cc90, peer_ip="127.0.0.1", peer_port=47346, 
    secure=false) at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:855
#16 0x00000000008fe580 in shttps::process_request (arg=0xe5e1a0) at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:524
#17 0x00007ffff79c0184 in start_thread (arg=0x7ffff256d700) at pthread_create.c:312
#18 0x00007ffff666a37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Build fails on Mac OS X (duplicate symbol)

[100%] Linking CXX executable sipi
ld: warning: directory not found for option '-L/Users/benjamingeer/git/Sipi/local/lib64'
duplicate symbol _USWebCoatedSWOP_icc_len in:
    CMakeFiles/sipi.dir/metadata/SipiIcc.cpp.o
    CMakeFiles/sipi.dir/SipiImage.cpp.o
duplicate symbol _USWebCoatedSWOP_icc in:
    CMakeFiles/sipi.dir/metadata/SipiIcc.cpp.o
    CMakeFiles/sipi.dir/SipiImage.cpp.o
ld: 2 duplicate symbols for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [sipi] Error 1
make[1]: *** [CMakeFiles/sipi.dir/all] Error 2
make: *** [all] Error 2

I downloaded the files AdobeRGB1998.icc and USWebCoatedSWOP.icc from the Adobe web site and generated the files AdobeRGB1998_icc.h and USWebCoatedSWOP_icc.h as instructed in the README. The generated header files are not identical to the ones in the old Sipi repository on Gitlab.

Segmentation fault

description

I know that it doesn't say much, yet:

sipi dies on a Segmentation fault: 11 at the first request to upload a file.

$ ./build/sipi -config config/sipi.knora-config.lua

Sipi Version 1.0 Beta (build 2016-11-03 11:00)
Segmentation fault: 11

the log doesn't say more.

Let me know if I can provide more info.

context

ljaouen@SSP28034:~/Soft/Code/github/Sipi$ git branch ; git log -1
  bugfixes
* develop
  master
  openssl
  wip/fix_Mac_compilation_error
commit 839b8f3e36187d6168906c3ea774d6248ac9a72b
Merge: eb84449 dfe9935
Author: Tobias Schweizer <[email protected]>
Date:   Wed Oct 26 14:48:25 2016 +0200

    Merge pull request #73 from dhlab-basel/wip/knora-init

    fix (adapt sipi.init-knora-test.lua) add new error handling functions

Sipi freshly build.

running as my user with the knora-config file.

Allow a project to define what is a "restricted view" permission for a StillImageRepresentation

This issue is created following a discussion we had with Ben.

A "restricted view" permission for a knora-base:StillImageRepresentation may not be the same for every project and it may be a good thing to allow projects to define what is it they intend to display with this permission, e.g.:

• a blank page or a default image;
• an image with a watermark;
• no image at all, the user with a 'restricted view permission" will only be able to see the properties of a knora-base:StillImageRepresentation according to his permissions.

Sipi terminates unexpectedly

After a failed image request (no answer), Sipi terminates after another request.

GDB output:

...
[New Thread 0x7fffb9ffb700 (LWP 4479)]
[New Thread 0x7fffb97fa700 (LWP 4480)]
[New Thread 0x7fffb8ff9700 (LWP 4481)]
[New Thread 0x7fffb87f8700 (LWP 4482)]
[Thread 0x7fffb9ffb700 (LWP 4479) exited]
[Thread 0x7fffba7fc700 (LWP 4478) exited]
[Thread 0x7fffb97fa700 (LWP 4480) exited]
[Thread 0x7fffb8ff9700 (LWP 4481) exited]
[New Thread 0x7fffb7ff7700 (LWP 4483)]
[New Thread 0x7fffb77f6700 (LWP 4484)]
[Thread 0x7fffb87f8700 (LWP 4482) exited]
[New Thread 0x7fffb6ff5700 (LWP 4485)]
[Thread 0x7fffb77f6700 (LWP 4484) exited]
[New Thread 0x7fffb67f4700 (LWP 4486)]
[Thread 0x7fffb7ff7700 (LWP 4483) exited]
[New Thread 0x7fffb5ff3700 (LWP 4487)]
[New Thread 0x7fffb57f2700 (LWP 4488)]
[New Thread 0x7fffb4ff1700 (LWP 4489)]
no exact match h=368 ny=734
no exact match h=368 ny=734
no exact match h=368 ny=734
[Thread 0x7fffd1ffb700 (LWP 4454) exited]
[Thread 0x7fffb5ff3700 (LWP 4487) exited]
[Thread 0x7fffb4ff1700 (LWP 4489) exited]
[Thread 0x7fffb6ff5700 (LWP 4485) exited]
[Thread 0x7fffb67f4700 (LWP 4486) exited]

Program received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fffb57f2700 (LWP 4488)]
0x00007ffff79c735d in write () at ../sysdeps/unix/syscall-template.S:81
81  ../sysdeps/unix/syscall-template.S: No such file or directory.

GDB up:

#1  0x00000000009050b3 in shttps::SockStream::overflow (this=0x7fffb57ef790, ch=121)
    at /home/tobi/gitreps/Sipi_github/lib/shttps/SockStream.cpp:79
79              ssize_t tmp_n = write(sock, out_buf + nn, n - nn);

GDB backtrace

#0  0x00007ffff79c735d in write () at ../sysdeps/unix/syscall-template.S:81
#1  0x00000000009050b3 in shttps::SockStream::overflow (this=0x7fffb57ef790, ch=121)
    at /home/tobi/gitreps/Sipi_github/lib/shttps/SockStream.cpp:79
#2  0x00007ffff7249fe9 in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff723e1b3 in std::ostream::write(char const*, long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00000000008de0af in shttps::Connection::sendAndFlush (this=0x7fffb57efb90, buffer=0x7fffc0031b60, n=65536)
    at /home/tobi/gitreps/Sipi_github/lib/shttps/Connection.cpp:1076
#5  0x0000000000578ad6 in Sipi::empty_output_buffer (cinfo=0x7fffb57e6920)
    at /home/tobi/gitreps/Sipi_github/formats/SipiIOJpeg.cpp:65
#6  0x000000000071c2e7 in dump_buffer_e (entropy=entropy@entry=0x7fffc0022d60) at jchuff.c:289
#7  0x000000000071d010 in emit_bits_e (size=1, code=<optimized out>, entropy=0x7fffc0022d60) at jchuff.c:377
#8  emit_buffered_bits (entropy=entropy@entry=0x7fffc0022d60, bufstart=0x7fffc00a03d8 "\001\001\001", 
    bufstart@entry=0x7fffc00a03d1 "", nbits=nbits@entry=10) at jchuff.c:452
#9  0x000000000071e677 in emit_buffered_bits (nbits=10, bufstart=<optimized out>, entropy=0x7fffc0022d60) at jchuff.c:448
#10 encode_mcu_AC_refine (cinfo=0x7fffb57e6920, MCU_data=<optimized out>) at jchuff.c:876
#11 0x000000000071a987 in compress_output (cinfo=0x7fffb57e6920, input_buf=<optimized out>) at jccoefct.c:386
#12 0x0000000000709bb6 in jpeg_finish_compress (cinfo=0x7fffb57e6920) at jcapimin.c:179
#13 0x000000000057ad86 in Sipi::SipiIOJpeg::write (this=0xe0ea10, img=0x7fffb57e7740, filepath="HTTP", quality=80)
    at /home/tobi/gitreps/Sipi_github/formats/SipiIOJpeg.cpp:630
#14 0x000000000055f407 in Sipi::SipiImage::write (this=0x7fffb57e7740, ftype="jpg", filepath="HTTP", quality=-1)
    at /home/tobi/gitreps/Sipi_github/SipiImage.cpp:265
#15 0x000000000059367d in Sipi::process_get_request (conobj=..., luaserver=..., user_data=0x7fffffffd350, dummy=0x0)
    at /home/tobi/gitreps/Sipi_github/SipiHttpServer.cpp:919
#16 0x00000000008f3455 in shttps::Server::processRequest (this=0x7fffffffd350, sock=5)
    at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:613
#17 0x00000000008f230f in shttps::process_request (arg=0xe1f370) at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:425
#18 0x00007ffff79c0182 in start_thread (arg=0x7fffb57f2700) at pthread_create.c:312
---Type <return> to continue, or q <return> to quit---
#19 0x00007ffff6caa47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Sipi log file:

...
[2016-05-09 15:51:14.753] [shttps-logger] [debug] Nothing found in cache, reading and transforming file...
[2016-05-09 15:51:14.753] [shttps-logger] [debug] get_size: img_w=2033 img_h=2835 w=508 h=709 reduce=2 reduce only=false
[2016-05-09 15:51:14.792] [shttps-logger] [debug] Socket connection: timeout or socket closed from main
[2016-05-09 15:51:14.792] [shttps-logger] [debug] Socket was not closed, closing...
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket connection: timeout or socket closed from main
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket connection: timeout or socket closed from main
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket connection: timeout or socket closed from main
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket was not closed, closing...
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket was not closed, closing...
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket was not closed, closing...
[2016-05-09 15:51:14.793] [shttps-logger] [debug] Socket connection: timeout or socket closed from main
[2016-05-09 15:51:14.794] [shttps-logger] [debug] Socket was not closed, closing...
[2016-05-09 15:51:14.893] [shttps-logger] [debug] Writing new cache file './cache/cache_XXXXnXI8Ch'
[2016-05-09 15:51:14.900] [shttps-logger] [debug] Before writing JPG...

Absolute Paths in Lua Scripts

Use absolute paths in Lua scripts to refer to directories and files

-> test the routes using the test scripts

Error when serving images

When requesting a lot of images, I get the following error:

*** Error in `build/sipi': corrupted double-linked list: 0x00007fc5c80511b0 ***
Aborted (core dumped)

The images themselves seem to be fine, i.e. then can be requested separately without any problem.

missing dependencies in documentation

Failure in handling wrong command line options

Passing a wrong option causes a segmentation fault:

andrea@Ubuntu-MacPro:~/Src/sipi$ build/sipi -c sipi.config.lua 
Segmentation fault (core dumped)
andrea@Ubuntu-MacPro:~/Src/sipi$ build/sipi -c 3
Segmentation fault (core dumped)
andrea@Ubuntu-MacPro:~/Src/sipi$ build/sipi -c w
Segmentation fault (core dumped)
andrea@Ubuntu-MacPro:~/Src/sipi$ build/sipi -a w
Segmentation fault (core dumped)

Improve Cookie handling in Preflight Request

Look for sid= in cookie, do not assume that the cookie starts with it (get index of sid=)

Sipi should retunrn a non 200 HTTP status code if there was a problem

This returns a 200 status code which does not make sense:

-- if check returns false, the user's input is invalid
if not check then
    result = {
        status = 1,
        message = "Mimetypes and/or file extension are inconsistent."
    }

    jsonstr = server.table_to_json(result)

    server.print(jsonstr)

    return
end

Improve Preview URL Handling

Improve creation of preview URL in make_thumbnail.lua

Do not use hard coded strings, but us settings in config file (Sipi domain):

preview_path = "http://localhost:1024/thumbs/" .. tmpname .. "_THUMB.jpg" .. "/full/full/0/default.jpg",

This means we need a constant in the config file representing the SIpi domain.

Add Delete File Route

Add a Lua script that deletes a file (thumb an full quality).

dasch-swiss/dsp-api#67

Make sure that the request comes directly from Knora!

Lua Test Route luaexe/test2

The path to the test image Leaves8.tif served by luaexe/test2 is not set correctly when imgroot is set to ./images because it resides in ./test_server/images.

This only works if imgroot is set to ./test_server/images in the config file.

config.imgroot .. "/Leaves8.tif"

project png cannot be download on branch openssl

It says it cannot extract the tarball's name form the URL:

CMake Error at /usr/share/cmake-2.8/Modules/ExternalProject.cmake:1401 (message):
  Could not extract tarball filename from url:

    http://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.xz/download
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/ExternalProject.cmake:1805 (_ep_add_download_command)
  ext/png/CMakeLists.txt:8 (ExternalProject_Add)

it works with this cmake conf for ext/png:

cmake_minimum_required(VERSION 2.8.10)
project(libpng_builder C CXX)
include(ExternalProject)

#
# get png
#
ExternalProject_Add(project_png
    INSTALL_DIR ${COMMON_LOCAL}
    URL http://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download
    SOURCE_DIR ${COMMON_LIB}/libpng-1.6.20
    CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${COMMON_LOCAL}
)
ExternalProject_Get_Property(project_png install_dir)
if(MAKE_SHARED_SIPI)
    add_library(png SHARED IMPORTED GLOBAL)
    set_property(TARGET png PROPERTY IMPORTED_LOCATION ${install_dir}/lib/libpng${CMAKE_SHARED_LIBRARY_SUFFIX})
else()
    add_library(png STATIC IMPORTED GLOBAL)
    set_property(TARGET png PROPERTY IMPORTED_LOCATION ${install_dir}/lib/libpng${CMAKE_STATIC_LIBRARY_SUFFIX})
endif()
add_dependencies(png project_png)

Continuous Integration

If it is ok, I would like to add support for Travis-CI so that each pushed commit/pull-request automatically starts an automated build. I don't know if there are any tests, but we could then run them also.

Report parsing errors for Lua scripts

When a Lua scripts cannot be parsed correctly, Sipi should give a clear error message on startup.

Add other Tests (failure in serving file, not supported extension,etc.)

Sipi tests should cover some additional situations:

• the file extension requested is not supported,
• the file was not created (404 http status)
• ???

Add Google C++ Test Framework

As per our discussion from today, I'm opening an issue to track the integration.

Here are some links, especially the second one describes exactly how to do it with CMake.

• Google C++ Test Framework: https://github.com/google/googletest
• Example integration in cmakeable project: https://github.com/snikulov/google-test-examples

How to 'cmake clean'

How can I clean up my project directory? Running make clean doesn't remove everything. If the build directory where the only place into which stuff gets downloaded/installed, than I could simply trash it.

Delete temporary Files after some Time

For Knora, we created routes and Lua scripts that allow to create temporary files for preview in the SALSAH GUI. If the final image is created, they are deleted. But if the user aborts the operation, they just remain there.

Therefore, add a routine that deletes tmp files that are older than a few hours.

Still can't download libpng from SourceForge

The build process gets an HTTP 404 error, but it keeps retrying.

-- downloading...
       src='https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download'
       dst='/Users/benjamingeer/git/Sipi/build/ext/png/project_png-prefix/src/libpng-latest.tar.xz'
       timeout='none'
error: downloading 'https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download' failed
  status_code: 22
  status_string: "HTTP response code said error"
  log:
  --- LOG BEGIN ---
    Trying 216.34.181.60...
Connected to sourceforge.net (216.34.181.60) port 443 (#0)
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Server certificate: *.sourceforge.net
Server certificate: GeoTrust SSL CA - G3
Server certificate: GeoTrust Global CA
GET /projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download HTTP/1.1
Host: sourceforge.net
User-Agent: curl/7.43.0
Accept: */*

The requested URL returned error: 404 Not Found
Closing connection 0

  --- LOG END ---
-- Retrying...
-- downloading...
       src='https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download'
       dst='/Users/benjamingeer/git/Sipi/build/ext/png/project_png-prefix/src/libpng-latest.tar.xz'
       timeout='none'
error: downloading 'https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download' failed
  status_code: 22
  status_string: "HTTP response code said error"
  log:
  --- LOG BEGIN ---
    Trying 216.34.181.60...
Connected to sourceforge.net (216.34.181.60) port 443 (#0)
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Server certificate: *.sourceforge.net
Server certificate: GeoTrust SSL CA - G3
Server certificate: GeoTrust Global CA
GET /projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download HTTP/1.1
Host: sourceforge.net
User-Agent: curl/7.43.0
Accept: */*

The requested URL returned error: 404 Not Found
Closing connection 0

  --- LOG END ---
-- Retry after 5 seconds (attempt #2) ...
-- downloading...
       src='https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download'
       dst='/Users/benjamingeer/git/Sipi/build/ext/png/project_png-prefix/src/libpng-latest.tar.xz'
       timeout='none'
error: downloading 'https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download' failed
  status_code: 22
  status_string: "HTTP response code said error"
  log:
  --- LOG BEGIN ---
    Trying 216.34.181.60...
Connected to sourceforge.net (216.34.181.60) port 443 (#0)
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Server certificate: *.sourceforge.net
Server certificate: GeoTrust SSL CA - G3
Server certificate: GeoTrust Global CA
GET /projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download HTTP/1.1
Host: sourceforge.net
User-Agent: curl/7.43.0
Accept: */*

The requested URL returned error: 404 Not Found
Closing connection 0

  --- LOG END ---
-- Retry after 5 seconds (attempt #3) ...
-- downloading...
       src='https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download'
       dst='/Users/benjamingeer/git/Sipi/build/ext/png/project_png-prefix/src/libpng-latest.tar.xz'
       timeout='none'
error: downloading 'https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download' failed
  status_code: 22
  status_string: "HTTP response code said error"
  log:
  --- LOG BEGIN ---
    Trying 216.34.181.60...
Connected to sourceforge.net (216.34.181.60) port 443 (#0)
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Server certificate: *.sourceforge.net
Server certificate: GeoTrust SSL CA - G3
Server certificate: GeoTrust Global CA
GET /projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download HTTP/1.1
Host: sourceforge.net
User-Agent: curl/7.43.0
Accept: */*

The requested URL returned error: 404 Not Found
Closing connection 0

  --- LOG END ---
-- Retry after 15 seconds (attempt #4) ...
-- downloading...
       src='https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download'
       dst='/Users/benjamingeer/git/Sipi/build/ext/png/project_png-prefix/src/libpng-latest.tar.xz'
       timeout='none'
error: downloading 'https://sourceforge.net/projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download' failed
  status_code: 22
  status_string: "HTTP response code said error"
  log:
  --- LOG BEGIN ---
    Trying 216.34.181.60...
Connected to sourceforge.net (216.34.181.60) port 443 (#0)
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Server certificate: *.sourceforge.net
Server certificate: GeoTrust SSL CA - G3
Server certificate: GeoTrust Global CA
GET /projects/libpng/files/libpng16/1.6.23/libpng-1.6.23.tar.gz/download HTTP/1.1
Host: sourceforge.net
User-Agent: curl/7.43.0
Accept: */*

The requested URL returned error: 404 Not Found
Closing connection 0

  --- LOG END ---
-- Retry after 60 seconds (attempt #5) ...

warning message: "Couldnt switch tu user !"

description

When starting Sipi, I get the warning message:

[2016-11-03 14:00:17.440] [shttps-logger] [warning] Couldnt switch tu user ! You must start SIPI as root!

I can't read anything related on the project's page.

I tried to run it as root, but still had the same error.

context

$ git branch ; git log -1
  bugfixes
* develop
  master
  openssl
  wip/fix_Mac_compilation_error
commit 839b8f3e36187d6168906c3ea774d6248ac9a72b
Merge: eb84449 dfe9935
Author: Tobias Schweizer <[email protected]>
Date:   Wed Oct 26 14:48:25 2016 +0200

    Merge pull request #73 from dhlab-basel/wip/knora-init

Sipi Server crashes when executing Test Script

When executing test_server/knora_convert_from_binaries.sh, Sipi server crashes with output Segmentation fault: 11

Make specific status codes for errors

in:

• https://github.com/dhlab-basel/Sipi/blob/develop/scripts/convert_from_binaries.lua
• https://github.com/dhlab-basel/Sipi/blob/develop/scripts/convert_from_file.lua
• https://github.com/dhlab-basel/Sipi/blob/develop/scripts/make_thumbnail.lua

Can we delete old unused branches?

Looking at the github repository I saw that branch https://github.com/dhlab-basel/Sipi/tree/ssl is behind of 148 commits from develop.

@lrosenth Can we delete it?

Log file: what do these messages mean?

When using Sipi with Knora, I get the following messages in my log file:

[2016-10-26 08:48:02.980] [shttps-logger] [warning] Couldnt switch tu user ! You must start SIPI as root!
[2016-10-26 08:49:58.502] [shttps-logger] [error] Browser unexpectedly closed connection
[2016-10-26 08:49:58.502] [shttps-logger] [warning] Error shutdown socket! Reason: Transport endpoint is not connected

Why would I have to start Sipi with a root user? I should be able to run it with my local user that owns the whole Sipi dir.

The other two messages can be explained by an aborted browser request. Does this lead to a problem? This will occur very often I guess since we cannot control if the client aborts the connection at some point.

Sometimes Sipi also returns an internal server error (HTTP status code 500) when loading all the thumbnails of a manuscript (about 400 files). This could be related to these messages:

[2016-10-26 08:42:49.336] [shttps-logger] [error] Internal Server Error!Not a JPEG file: starts with 0xff 0xe0
[2016-10-26 08:43:44.189] [shttps-logger] [error] Blocking poll failed at line: 836 ERROR: Interrupted system call

Sometimes I get this:

[2016-10-26 09:01:07.571] [shttps-logger] [error] Internal Server Error!Empty input file

On a separate request, Sipi serves the jpg without problems. The jpeg was also created by Sipi itself when importing it using Knora.

Build fails on Mac OS X

I am trying to build Sipi on branch develop using Mac OS X:

/bin/sh: /Users/tobi/gitreps/Sipi-github/patches/Makefile-MAC-x86-all-gcc.patch: No such file or directory
make[2]: *** [ext/kakadu/project_kakadu-prefix/src/project_kakadu-stamp/project_kakadu-update] Error 1
make[1]: *** [ext/kakadu/CMakeFiles/project_kakadu.dir/all] Error 2
make: *** [all] Error 2

Seems to be a kakadu issue. I am using v7_8-01382N.

Tests in Bash Script

As a first step in building an efficient and working test environment, a set of bash script are going to be developed. Requirements:

• they shall be all be started by a single main script;
• they shall return 0 if everything went ok;
• they shall return 1 or any other number, but 0, if there is a fail;

I will start a branch and develop the first version in it
@subotic @lrosenth @tobiasschweizer

Making a IIIF Request Providing dimension(s) for size is very slow on Mac

When requesting a IIIF URL providing exact dimension(s), it sometimes takes SIPI very long to respond to the request:

http://localhost:1024/images/Leaves8.tif/full/83,/0/default.jpg

Manipulating the width can lead to much faster results. So I guess it could have something to do with size calculation and/or rounding. I experience these problems on Mac only. On my Ubuntu, it is fast.

README not clear about Adobe ICC Color profile

The README says I have to download the Adobe ICC Color profile and install it manually, but then (under General) it seems to say that the build framework does this automatically so I don't have to do it. On the develop branch, it's not downloaded automatically.

Which compiler should be used on Mac OS X?

I have both clang and gcc installed, but the build framework is using gcc. Is this correct?

build fails due to missing files

The version cloned from github can't build due to missing files:
vendor/v7_8-01382N.zip.ZIP.
and
'openssl/bio.h' file not found

include <openssl/bio.h>

used in shttps/jwt.c

Cannot download libpng-1.6.21 during Build Process

[ 45%] Performing download step (download, verify and extract) for 'project_png'
-- downloading...
     src='http://sourceforge.net/projects/libpng/files/libpng16/1.6.21/libpng-1.6.21.tar.gz/download'
     dst='/Users/tobi/gitreps/Sipi-github/build/ext/png/project_png-prefix/src/libpng-1.6.21.tar.gz'
     timeout='none'
-- [download 0% complete]
CMake Error at /Users/tobi/gitreps/Sipi-github/build/ext/png/project_png-prefix/src/project_png-stamp/download-project_png.cmake:21 (message):
  error: downloading
  'http://sourceforge.net/projects/libpng/files/libpng16/1.6.21/libpng-1.6.21.tar.gz/download'
  failed

    status_code: 22
    status_string: "HTTP response code said error"
    log:   Trying 216.34.181.60...

Maybe the version has to be changed to 1.6.22 or 1.6.23, see https://sourceforge.net/projects/libpng/files/libpng16/

SIPI hangs with ab stress test

This is an OS X problem.... (may be also on linux)

_From what I found out this seems to be a MacOSx problem. See the following thread for
information
http://stackoverflow.com/questions/1216267/ab-program-freezes-after-lots-of-requests-why

This is from the above thread:

On Mac OS X the default ephemeral port range is 49152 to 65535, for a total of 16384
ports. You can check this with the sysctl command:

$ sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535

Changing this configuration to start from 32768 helped
$ sudo sysctl -w net.inet.ip.portrange.first=32768
net.inet.ip.portrange.first: 49152 -> 32768

Another configuration that I changed was for default timeout
Set the default timeout to 1000ms like so:

$ sudo sysctl -w net.inet.tcp.msl=1000
net.inet.tcp.msl: 15000 -> 1000

After changing above configuration following are the results

Server Software:
Server Hostname: localhost
Server Port: 4000

Document Path: /
Document Length: 12 bytes

Concurrency Level: 1
Time taken for tests: 17.005 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 14800000 bytes
HTML transferred: 1200000 bytes
Requests per second: 5880.58 #/sec
Time per request: 0.170 ms
Time per request: 0.170 [ms](mean, across all concurrent requests)
Transfer rate: 849.93 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 12
Processing: 0 0 0.0 0 1
Waiting: 0 0 0.0 0 1
Total: 0 0 0.1 0 12

Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 12 (longest request)_

Why is an integer in Lua converted to a floating point number in JSON?

Lua script:

    result = {
        status = 1,
        message = "Mimetypes and/or file extension are inconsistent."
    }

    jsonstr = server.table_to_json(result)

    server.print(jsonstr)

In JSON, status is 1.0, but it should be just1.

This is returned by SIPI:

 {
    "message": "Mimetypes and/or file extension are inconsistent.",
   "status": 1.0
}

SIPI Server crashes when serving jp2 file

The following URL was requested from Sipi: localhost:1024/knora/incunabula_0000003496.jp2/full/610,804/0/default.jpg. It is a JP2000 file from the incunabula project.

When serving the image, Sipi crashes (Ubuntu).

It creates an empty file in the cache.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff256d700 (LWP 17430)]
__memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:36
36  ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
(gdb) backtrace
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:36
#1  0x000000000058642f in Sipi::SipiIOJpeg::write (this=0xe33a10, img=0x7ffff25637b0, filepath="HTTP", quality=80)
    at /home/tobi/gitreps/Sipi_github/src/formats/SipiIOJpeg.cpp:749
#2  0x000000000056a07d in Sipi::SipiImage::write (this=0x7ffff25637b0, ftype="jpg", filepath="HTTP", quality=-1)
    at /home/tobi/gitreps/Sipi_github/src/SipiImage.cpp:265
#3  0x000000000059fcb7 in Sipi::process_get_request (conobj=..., luaserver=..., user_data=0x7fffffffd330, dummy=0x0)
    at /home/tobi/gitreps/Sipi_github/src/SipiHttpServer.cpp:955
#4  0x0000000000908607 in shttps::Server::processRequest (this=0x7fffffffd330, sock=7, ins=0x7ffff256c910, os=0x7ffff256c800, 
    peer_ip="127.0.0.1", peer_port=57739, secure=false) at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:865
#5  0x000000000090680e in shttps::process_request (arg=0xe6b4e0) at /home/tobi/gitreps/Sipi_github/lib/shttps/Server.cpp:530
#6  0x00007ffff79c0184 in start_thread (arg=0x7ffff256d700) at pthread_create.c:312
#7  0x00007ffff666a37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Add in the readme - ipv4

Add in the readme file the information about the only supported version of the IP: 4.