ioda-net / geo-api3 Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 0.0 229.95 MB

Backend from geoadmin/mf-chsdi3

License: Other

Python 4.72% CSS 0.11% Makefile 0.05% HTML 0.47% JavaScript 94.45% Mako 0.11% Shell 0.09%

geo-api3's People

Contributors

Stargazers

Watchers

geo-api3's Issues

Consider to use jinja2 to render templates

This would avoid the need for gulp and nunjucks
jinja2 and nunjucks are almost completely compatible

Test with https://pypi.python.org/pypi/jinja2-cli/0.4.3. Current blocking points:

no support for toml
command line parse is a little buggy

Do we still need mo files in the API?

Everything seems to be handled by the frontend now.

Add scripts to launch specific tests with nose

See: https://github.com/tigerfoot/geo2015/blob/master/doc/nose.md

Print measurement : quality of label

Make label more readable : (print was made at 150dpi)

If all the dependancies are packaged (and I don't doubt we will succeed to do this), all that change in the API is located with the chsid folder. Which means we can easily rely on git to deploy the application. I simply propose to tag the commit we want to release with the current timestamp (or date formatted as iso). We could then use the two commands below to deploy or revert to the previous version as needed.

Get lastest version: git fetch && git checkout $(git tag | sort -nr | head -n 2) && sudo systemctl restart httpd
Revert to previous version: git fetch && git checkout $(git tag | sort -nr | head -n 2 | tail -n 1) && sudo systemclt restart httpd

Print Commune Label - fix sql

Durant une impression postgresql logue des erreurs - Voir correction du script/view sql
@verogis dit
J'ai eu un souci ce dimanche quand j'ai essayé d'imprimer -> voir le pglog :

FROM userdata.communes
jui 26 12:15:32 dumbo postgres[4340]: [2-4] WHERE
ST_Contains(userdata.communes.the_geom,
'SRID=21781;POINT(584578.698435136 232370.987534912)')
jui 26 12:16:44 dumbo postgres[4460]: [2-1] 2015-07-26 12:16:44 CEST
sit_dev sit_api3 ERROR: Exception in LWGEOM2GEOS: curved geometry not
supporte

Toujours la même erreurs lorsque j'essaye d'imprimer (rien quand je me
balade dessus)

CEST sit_dev sit_api3 ERROR: Exception in LWGEOM2GEOS: curved
geometry not supported.
jui 27 13:57:39 dumbo postgres[12448]: [2-2] 2015-07-27 13:57:39 CEST
sit_dev sit_api3 STATEMENT: SELECT userdata.communes.gid AS
userdata_communes_gid, userdata.communes.nom AS userdata_communes_nom,
ST_AsEWKB(userdata.communes.the_geom) AS userdata_communes_the_geom
jui 27 13:57:39 dumbo postgres[12448]: [2-3] FROM userdata.communes
jui 27 13:57:39 dumbo postgres[12448]: [2-4] WHERE
ST_Contains(userdata.communes.the_geom, 'SRID=21781;POINT(582409.2171616
230338.95201312)')

Mais en fait, je l'ai même quand je n'ai pas la couche commune activée.
Du coup éclair de génie -> c'est pas lié à la demande d'avoir le nom de
la commune à l'impression ?

Un call a print ne montre aucune activitée sur tomcat ???

Remove unuseful code

Service
Catalog
Layer config

api search : doesn't return searched number if too much results

Typically in cities portal where search for hydrants and others similar searches are active there's a case where returned results doesn't contain what's looked for due to a too big numbers of results

More specifically in erlach : searching an hydrants with number 57 work perfect (there's only one street address with this result and also only one parcel, so hydrant 57 is shown)

curl 'http://api/rest/services/erlach/SearchServer?searchText=57&lang=de&type=locations_preview' -H 'Host: api' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: http://erlach/?topic=05_alimentationeau&lang=de&bgLayer=voidLayer&layers=PARCELLE_NUMERO,BATIMENT_NUMERO,BATIMENTS,PARCELLE,ODEL_TEXTE,COUVERTUREDUSOL_TEXTE,CDS,ODEL,PFP,PFA,PIECE_INSTALLATION,INSTMESURE_DETAIL,RESERVOIR_DETAIL,TRONCON,VANNE,HYDRANT,TRONCON_LABEL,HYDRANT_LABEL&catalogNodes=&X=210090.50&Y=573892.00&zoom=6' -H 'Origin: http://erlach' -H 'Connection: keep-alive'

But if you want to see hydrant 13 you never get its results due to the too big number in address and parcels.

curl 'http://api/rest/services/erlach/SearchServer?searchText=13&lang=de&type=locations_preview' -H 'Host: api' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: http://erlach/?topic=05_alimentationeau&lang=de&bgLayer=voidLayer&layers=PARCELLE_NUMERO,BATIMENT_NUMERO,BATIMENTS,PARCELLE,ODEL_TEXTE,COUVERTUREDUSOL_TEXTE,CDS,ODEL,PFP,PFA,PIECE_INSTALLATION,INSTMESURE_DETAIL,RESERVOIR_DETAIL,TRONCON,VANNE,HYDRANT,TRONCON_LABEL,HYDRANT_LABEL&catalogNodes=&X=210084.54&Y=574456.30&zoom=9' -H 'Origin: http://erlach' -H 'Connection: keep-alive'

Questions :

are we abusing locations search ?
what could make the result better ?
Be sure to have keyword search

Use config.toml in test_sphinx

Not possible yet due to a bug in our toml parser for python. See uiri/toml#40

Create database in initializedb.py

A database named bod_${vars:db_staging} with a schema named re3 must exist for the script to complete. It should be able to create them.

The schema can be created with: engine.execute(CreateSchema('re3')). The name of the database can be fetched by db_name = settings['sqlalchemy.bod.url'].split('/')[-1]

Still have to find how to create the database. I tested:

engine.execute('CREATE DATABASE {}'.format(db_name))
conn = engine.connexion(); conn.execute('CREATE DATABASE {}'.format(db_name))

Fix get features after schema update

Fetch a box2d from a search on location to allow the interface to go there

SELECT commune_gid, commune_ofs, commune_nom, commune_arrondissement, 
       commune_ofs_arr, box2d(the_geom)
  FROM communes;

reload features not enough if a DDL change happen in db

If a DDL (data structure change) occurs in the database, running feature reload is not enough to get the full new structure.
It look like sqlalchemy keep a cache of the structure somewhere.
Calling restart-service apache work as expected.

Not too urgent

Replacement for Height, profile.json, profile.csv

We are looking for a replacement from reading raster files and use directly the dom/dtm (mnt/mns) database.

Check with jbr availablity of data
Create postgresql/postgis queries (bench them)
Create our new routes
Analyse rename impact of database
Update access to db

chsdi/static/images/legends/ carry more than 128 MB of specific data

There's 2047 files totalising 128MB of legend data, which are specific to swisstopo/admin.ch
Those need to be purged.
Question : how to clean up .git at the same time.

special columns + proxy protocols

We need to handle some special column with a proxy inside the api to return the result of an external request (only the api host can access those information)

Here the php snippet code how it was handle previously inside cartoweb3

    /**
     * Returns Protocol/PDF file name
     *
     * Format is set in configuration file, key fileName.
     * @return string
     */
    protected function generateFileName() {

        $Name = I18n::gt($this->Id);

        if (!is_null($this->getConfig()->fileName)) {
            $format = $this->getConfig()->fileName;
        } else {
            $format = '[protocol]-[date,dMY_H-i-s].pdf';
        }
        $fileName = $format;
        $Name = str_replace(array(' ', ',', ';', ':'),
                                 array('_', '', '', ''),
                                 $Name);
        $Name = $this->Proto.'_'.$Name;
        $fileName = str_replace('[protocol]', $Name, $fileName);
// remove for php 5.3        ereg('(.*)\[date,(.*)\](.*)', $fileName, $match);
        preg_match('/(.*)\[date,(.*)\](\..*)/',$fileName,$match);
        $fileName = $match[1] . date($match[2]) . $match[3];

        return $fileName;
    }

     * Computes export
     * @return ExportOutput
     * @see ExportPlugin::getExportResult
     */
    protected function getExport() {

        $this->getExportResult($this->getConfiguration());
        // Default value 
        $contents = '';
        $host = '';
        $directory = '';

        switch ($this->Proto){
            case "pfp":
                if (!is_null($this->getConfig()->pfpHost)) {
                    $host = $this->getConfig()->pfpHost;
                } 
                if (!is_null($this->getConfig()->pfpDirectory)) {            
                    $directory = $this->getConfig()->pfpDirectory;
                }
                break;

            case "geo":
                if (!is_null($this->getConfig()->geoHost)) {
                    $host = $this->getConfig()->geoHost;
                } 
                if (!is_null($this->getConfig()->geoDirectory)) {            
                    $directory = $this->getConfig()->geoDirectory;
                }
                break;

            case "pdc":
                if (!is_null($this->getConfig()->pdcHost)) {
                    $host = $this->getConfig()->pdcHost;
                } 
                if (!is_null($this->getConfig()->pdcDirectory)) {            
                    $directory = $this->getConfig()->pdcDirectory;
                }
                break;                
        }
        if ( $host == '' || $directory = '' || $this->Id == ''){
            throw new CartoclientException ('No valid configuration found for Proto = '. $this->Proto . ' / ID = '. $this->Id . ' getConfig = '. $this->getConfig()->geoDirectory);
        }
        //else{
        //  throw new CartoclientException ('Valid configuration found for Proto = '. $this->Proto . ' / ID = '. $this->Id . ' url = '. $host.$directory.$this->Id );
        //}
        /**
         *  Open the remote application server.
         */
        $ctx = stream_context_create(array('http' => array('timeout' => 15)));        
        $contents = file_get_contents ($host.$directory.$this->Id,0,$ctx);
        if ($contents === FALSE)
        {
            throw new CartoclientException ('Plugin exportProtocol can\'t load the remote protocol\'s PDF.' . $host.$directory.$this->Id);
        }

        $output = new ExportOutput();
        $output->setContents($contents);
        return $output;
    }

    /**
     * @see ExportPlugin::output()
     */
    public function output() {
        /**
         * FIXME ! If it's not a pdf we need to change this ...
         */
        /**
         * TODO : Put all output inside a smarty tpl
         */
        $pdfBuffer = $this->getExport()->getContents();
        $startP = substr($pdfBuffer,0,5);
        if ( $startP != '%PDF-' ){
            //Not a PDF go away
            throw new CartoclientException ('Plugin exportProtocol can\'t load the remote protocol\'s '.$this->Proto.' PDF for :' . $this->Id);
            return '';
        }

        /**
         * fixed : 2009-02-19 Trouble with download in certain case
         */

        header("Pragma: public");
        header("Expires: 0");
        header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
        header("Cache-Control: private",false);
        header("Content-Type: application/force-download");
        header("Content-Type: application/octet-stream");
        header("Content-Type: application/download");
        header("Content-Description: ".$this->fileName);            
        header("Content-Disposition: attachment; filename=\"".$this->fileName."\";");
        header("Content-Transfer-Encoding: binary");
        header("Content-Length: ".strlen($pdfBuffer));
        flush(); // this doesn't really matter.
        print( $pdfBuffer );
        return '';       
    }

Also each portal which was using this plugins get a similar config.ini file with this type of content

; Configuration pour l'export Protocol 
; pfp & geo supported 
;  
charsetUtf8 =  true 

; Not used anymore, the ClientTable Plugin contain the needed role
; allowedRoles =  "EXPORTPFP,EXPORTGEO,EXPORTPDC" 

fileName =  "N16_[protocol]_[date,Y-m-d_His].pdf" 

; Url for getting pfp pdf protocol with a N°
pfpHost =  "http://pfp.disney.interne/pdf/get/pnum/" 

; Url for getting geo pdf protocol with a N°
geoHost =  "http://geo.disney.interne/pdf/get/pnum/" 

; Url for getting pdc pdf protocol with a N°
pdcHost =  "http://pdc.disney.interne/pdf/get/cid/"

Related to the feature views (still to be done) we will have a protocol colum that we can format as we want actually.
Actually the call is made with
mode=protocol&project=n16&exportprotocol_type=geo&exportprotocol_id=IVC6

Beware of : protocol_id could be anything most of the id are varchar, and should be url_decode some of them containing special char, or space etc.

Extra bonus : We should check how to be safe from external request

communes

Make a specific view in api3 schema, with st_curve2line ()

Rewrite `escape_sphinx_syntax`

In lib.helpers

For each test that raise a 400 error, we should check that the message is correct

make serve crash Traceback build venv

after a make cleanall, make serve doesn't create correctly the venv (openSUSE 13.1)

Traceback (most recent call last):
File "setup.py", line 3, in
from setuptools import setup, find_packages
ImportError: No module named 'setuptools'
make: *** [venv] Error 1

Makefile : prod need ajustment

Review the use of
.PHONY: release
realease:
git tag $(shell date +"%Y-%m-%d-%H-%M-%S")
git push
git push --tags

.PHONY: prod
prod:
git pull && git checkout $(shell git tag | sort -nr | head -n 1) &&
sudo systemctl restart apache2

.PHONY: revert
revert:
git pull && git checkout $(shell git tag | sort -nr | head -n 2 | tail -n 1) &&
sudo systemctl restart apache2

Enable translations for layer search

chsdi.lib.raster requires cleanup

Use proper index in search

Each geoportal will have its index for search. To select the proper one, we can either use the Referer header, eg http://geoadmin.local/?X=226485.04&Y=588470.80&zoom=7&lang=en&topic=cwdev_geojb&bgLayer=COUVERTUREDUSOL&catalogNodes=2, the Origin header eg http://geoadmin.local/ or the topic id present in the request.

Since only the topic id allow us to directly access the search from the API, I will implement it this way.

Launch test for ows_checker

Currently, they have a syntax error.

search keywords : to be checked

Adjust list of supported keyword in search api and searchd following the customers lists

Mots clés - GEOJB
fr,de
parcelle,grundstück
rue,Strasse
place,Platz
egid,egid

Mots clés - Erlach
fr,de
parcelle,grundstück
rue,Strasse
place,Platz
egid,egid
hydrant,hydrant
cabine TV,TV-Kabine

Mots clés - N16
km,km
pf,fp
géologie,geologie (sondierung)
surveillance,überwachung

SI POSSIBLE EN MINUSCULE ET MAJUSCULE POUR LA PREMIERE LETTRE

Once done, give back for documentation all the keywords we actually handle

Decide what to do of /admin/kml

Swisstopo has recently added a /admin/kml page to view the last 50 KML drawings. I added it as part of the update of the API.

Should we keep it?
How to improve it: currently it uses layers from Swisstopo to display the KML so depending on the host, we may or may not see the background

@tigerfoot ?

Make translate to clean

There's a target without any utiliy

.PHONY: translate
translate: venv
./bin/translate.sh

Search with bbox is broken

Sphinx doesn't seem to take into account the GeoAnchor even if the view have a latitude and a longitude:

DROP VIEW search.sorted_buildings;

CREATE OR REPLACE VIEW search.sorted_buildings AS 
 SELECT b.gid,
    b.ofs,
    b.ofs_arr,
    b.localisation,
    b.numero,
    c.nom AS commune,
    b.code,
    b.nom,
    b.egid,
    (c.nom::text || ', '::text) || b.adresse::text AS adresse,
    box2d(b.the_geom) AS geom_st_box2d,
    st_x(st_centroid(st_curvetoline(b.the_geom))) AS y,
    st_y(st_centroid(st_curvetoline(b.the_geom))) AS x,
    st_y(st_centroid(ST_Transform(st_curvetoline(b.the_geom), 4326))) as lat,
    st_x(st_centroid(ST_Transform(st_curvetoline(b.the_geom), 4326))) as lon,
    row_number() OVER (ORDER BY b.localisation, rpad(lpad("substring"(b.numero::text, '[0-9]+'::text), 6, '0'::text), 8, '__'::text) || "substring"(b.numero::text, '.$'::text)) AS weight
   FROM batimenttotal b
     LEFT JOIN commune_unique c ON c.ofs = b.ofs AND c.ofs_arr = b.ofs_arr
  ORDER BY b.ofs, b.ofs_arr, b.localisation, rpad(lpad("substring"(b.numero::text, '[0-9]+'::text), 6, '0'::text), 8, '__'::text) || "substring"(b.numero::text, '.$'::text);

ALTER TABLE search.sorted_buildings
  OWNER TO sit_dev;
GRANT ALL ON TABLE search.sorted_buildings TO sit_dev;
GRANT SELECT, REFERENCES, TRIGGER ON TABLE search.sorted_buildings TO sit_mapserver;
GRANT SELECT, REFERENCES, TRIGGER ON TABLE search.sorted_buildings TO sphinx;

-- View: search.communes

DROP VIEW search.communes;

CREATE OR REPLACE VIEW search.communes AS 
 SELECT communes.gid,
    communes.nom,
    communes.ofs,
    communes.ofs_arr,
    box2d(communes.the_geom) AS geom_st_box2d,
    st_x(st_centroid(st_curvetoline(communes.the_geom))) AS y,
    st_y(st_centroid(st_curvetoline(communes.the_geom))) AS x,
    st_y(st_centroid(ST_Transform(st_curvetoline(communes.the_geom), 4326))) as lat,
    st_x(st_centroid(ST_Transform(st_curvetoline(communes.the_geom), 4326))) as lon,
    row_number() OVER (ORDER BY communes.nom) AS weight
   FROM communes
  ORDER BY communes.nom;

ALTER TABLE search.communes
  OWNER TO sit_dev;
GRANT ALL ON TABLE search.communes TO sit_dev;
GRANT SELECT, REFERENCES, TRIGGER ON TABLE search.communes TO sit_mapserver;
GRANT SELECT, REFERENCES, TRIGGER ON TABLE search.communes TO sphinx;

Generic templates for features

We can make them generic if we use translations to correctly display the key (instead of the name of the field in postgre). The only problem left is: the HTML will be generated from a python dict. So we cannot guaranty the display order (even if it is unlikely that they are displayed in a different order than the one the db defines).

Making the API generate a JSON and display in the frontend won't solve the issue: order are not guaranteed in in JS object.

If it is not a problem, we are good.

Make deploy.py compatible with python 2

Add an option in deploy.py to clean project

Where are our template mako

Since last merge some mako templates have disappear.
Time to find a now place for them?

Make release

Typo in tag
.PHONY: release
realease:
git tag $(shell date +"%Y-%m-%d-%H-%M-%S")
git push
git push --tags

features_reload : to be checked

Have a look if the reload cleans the list of register_features

tests : check configuration vars

if the shortener allowed domains doesn't contain . it will failed
qrencode test should use [template.debug] like the shortener test

Correctly choose the background layers in initializedb.py

Clean database schéma

Clean the bod databases:

No more german
Simplify

Make the way the app handle debug headers more robust

If pyramid_debug_headers is empty or ends with a column, the API crashes

Buildings without id

Some buildings from batimenttotal don't have id and that make getFeature fails. Use the request below to find one:

SELECT code, id, commune, ofs, surface, surfacegruda, adresse, localisationgruda, 
       rue, nom, numerogruda, lieudit, lieuditgruda, parcellegruda, 
       typegruda, edidgruda, egidgruda, id_gruda, idgruda, type, gid, 
       adressegruda, beid, arrondissement, ofs_arr, portal, the_geom
  FROM userdata.batimenttotal
  WHERE ST_DWITHIN(userdata.batimenttotal.the_geom, ST_GeomFromText('POLYGON((579586.96 227135.05, 579586.96 227307.55, 579822.46 227307.55, 579822.46 227135.05, 579586.96 227135.05))', 21781), 2.5) 
 LIMIT 201;

ioda-net / geo-api3 Goto Github PK

geo-api3's People

Contributors

Stargazers

Watchers

geo-api3's Issues

Recommend Projects

Recommend Topics

Recommend Org