Greenplum Database - Massively Parallel PostgreSQL for Analytics. An open-source massively parallel data platform for analytics, machine learning and AI.
Home Page: http://greenplum.org
License: Apache License 2.0
I got this from a query from the regression suite, after running installcheck-good:
regression=# explain select cn,sum(qty) from sale group by cube(prc,cn) order by 1, 2;
WARNING: invalid attnum 6 for rangetable entry rollup (parse_relation.c:2073)
...
This code changes the bufsize to 16M to reduce number of calls to 'curl_easy_perform' hence it will improve WET performance.
diff --git a/src/backend/access/external/url.c b/src/backend/access/external/url.c
index a81e687..1aaa6ba 100644
--- a/src/backend/access/external/url.c
+++ b/src/backend/access/external/url.c
@@ -2295,7 +2295,9 @@ static size_t curl_fwrite(char *buf, int nbytes, URL_FILE* file, CopyState pstat
*/
if(!curl->out.ptr)
{
- const int bufsize = 64 * 1024 * sizeof(char);
+ const int bufsize = 16 * 1024 * 1024 * sizeof(char); /* Enlarge the bufsize to reduce call to curl_easy_perform,
+ hence WET performance should be improved.
+ The size itself is magic */
MemoryContext oldcontext = CurrentMemoryContext;
MemoryContextSwitchTo(CurTransactionContext); /* TODO: is there a better cxt to use? */
But gpfdist's buffer should also be enlarged with options -m 33554432
\d onefield
Append-Only Columnar Table "public.onefield"
Column | Type | Modifiers
--------+-------------------+-----------
fld | character varying |
Checksum: t
Distributed randomly
SELECT max(length(fld)) from onefield;
max
-----
440
(1 row)
Time: 6134.606 ms
\d w16m_ext_1
External table "public.w16m_ext_1"
Column | Type | Modifiers
--------+-------------------+-----------
fld | character varying |
Type: writable
Encoding: UTF8
Format type: text
Format options: delimiter 'off' null '\N' escape '\'
External location: gpfdist://p_etl01:16665/a16m_1.txt
INSERT INTO w16m_ext_1 SELECT * from onefield;
INSERT 0 171602816
Time: 168113.381 ms
COPY onefield TO '/data/testing/onefield_copy';
COPY 171602816
Time: 70774.670 ms
truncate w16m;
TRUNCATE TABLE
Time: 137.081 ms
INSERT INTO w16m SELECT string_agg(fld, E'\n')
FROM onefield
GROUP BY ctid::int8>>18, gp_segment_id ;
INSERT 0 1344
Time: 80662.342 ms
SELECT max(length(fld)) from w16m;
max
----------
19412359
(1 row)
Time: 7475.346 ms
\d w16m_ext
External table "public.w16m_ext"
Column | Type | Modifiers
--------+-------------------+-----------
fld | character varying |
Type: writable
Encoding: UTF8
Format type: text
Format options: delimiter 'off' null '\N' escape '\'
External location: gpfdist://p_etl01:16665/a16m.txt
INSERT INTO w16m_ext SELECT * from w16m;
INSERT 0 1344
Time: 74387.568 ms
-rw------- 1 gpadmin gpadmin 29G Dec 24 14:19 a16m_1.txt
-rw------- 1 gpadmin gpadmin 29G Dec 24 14:07 a16m.txt
-rw-r--r-- 1 gpadmin gpadmin 29G Dec 24 14:08 onefield_copy
Enlarge datasize of each 'POST request' invoked by 'curl_fwrite' will dramatically improve WET performance . From 168113.381 ms reduced to 74387.568 ms.
HI,
in my test environment ,run tpc-h query 2 timeout. this is the test machine and gp configure.
HARDware
24 CORE
192G mem
FUSINO IO ssd
GREENPLUm:
4.3.6
1 master
12 segments
master parameter
port=1921 ##port = 5432 # sets the database listener port for
max_connections = 48 # inserted by initdb
shared_buffers = 1024MB # inserted by initdb
max_prepared_transactions = 250 # can be 0 or more
max_fsm_pages = 1638400 # inserted by initdb
optimizer_analyze_root_partition = on # stats collection on root partitions
gp_autostats_mode=on_no_stats # none, on_no_stats, on_change. see documentation for semantics.
gp_autostats_on_change_threshold=2147483647 # [0..INT_MAX]. see documentation for semantics.
log_autostats=off # print additional autostats information
datestyle = 'iso, mdy' # inserted by initdb
lc_messages = 'C' # inserted by initdb
lc_monetary = 'C' # inserted by initdb
lc_numeric = 'C' # inserted by initdb
lc_time = 'C' # inserted by initdb
gp_resqueue_memory_policy = 'eager_free' # memory request based queueing.
max_appendonly_tables = 10000 # Maximum number of append only tables that can
gp_interconnect_type=udpifc
gp_connections_per_thread = 64
gp_segment_connect_timeout = 600s
gp_vmem_protect_limit = 8192 #Virtual memory limit (in MB).
gp_backup_directIO = off # enable\disable dump with direct IO
gp_backup_directIO_read_chunk_mb = 20 # Size of read Chunk buffer in directIO dump (in MB)
checkpoint_segments=64
log_min_error_statement=error
log_error_verbosity=verbose
log_statement=all
log_duration=on
enable_nestloop=off
work_mem=256MB
segment parameter
listen_addresses='' ##listen_addresses = '' # what IP address(es) to listen on;
port=40000 ##port = 5432 # sets the database listener port for
max_connections = 144 # inserted by initdb
shared_buffers = 1024MB # inserted by initdb
max_prepared_transactions = 250 # can be 0 or more
max_fsm_pages = 1638400 # inserted by initdb
optimizer_analyze_root_partition = on # stats collection on root partitions
gp_autostats_mode=on_no_stats # none, on_no_stats, on_change. see documentation for semantics.
gp_autostats_on_change_threshold=2147483647 # [0..INT_MAX]. see documentation for semantics.
log_autostats=off # print additional autostats information
datestyle = 'iso, mdy' # inserted by initdb
lc_messages = 'C' # inserted by initdb
lc_monetary = 'C' # inserted by initdb
lc_numeric = 'C' # inserted by initdb
lc_time = 'C' # inserted by initdb
gp_resqueue_memory_policy = 'eager_free' # memory request based queueing.
max_appendonly_tables = 10000 # Maximum number of append only tables that can
gp_interconnect_type=udpifc
gp_connections_per_thread = 64
gp_segment_connect_timeout = 600s
gp_vmem_protect_limit = 8192 #Virtual memory limit (in MB).
gp_backup_directIO = off # enable\disable dump with direct IO
gp_backup_directIO_read_chunk_mb = 20 # Size of read Chunk buffer in directIO dump (in MB)
checkpoint_segments=64
log_min_error_statement=error
log_error_verbosity=verbose
log_statement=all
log_duration=on
enable_nestloop=off
work_mem=256MB
os centos 6 x64
tpch use https://github.com/digoal/pg_tpch
test data use http://www.tpc.org/tpch/tools_download/dbgen-download-request.asp generate.
1 GB data.
test case
http://blog.163.com/digoal@126/blog/static/16387704020151019111930303/
atexit() should be part of stdlib ("man atexit" confirms) but on Fedora 22 configure complains of not finding atexit() or on_exit(). System in question has glibc, -headers, -devel v 2.21.
Could there be a configure option or an environment variable that needs setting?
in the server_ips file, there's an entry for an IP:
10.10.2.147
Running the gpinitsystem command:
gpinitsystem -c gpinitsystem_config -h server_ips
20151119:19:33:57:013585 gpinitsystem:ip-10-10-2-75:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 250
20151119:19:33:57:013585 gpinitsystem:ip-10-10-2-75:gpadmin-[INFO]:-Checking configuration parameters, Completed
20151119:19:33:57:013585 gpinitsystem:ip-10-10-2-75:gpadmin-[INFO]:-Commencing multi-home checks, please wait...
Script Exiting!07:gpinitsystem:ip-10-10-2-75:gpadmin-[FATAL]:-Unknown host 10.10.2.147
The problem is that a host lookup fails for an IP (because the host has already been looked up). Not all machines have resolvable hostnames, but they do have valid IPs.
Please modify gpinitsystem to accept IP addresses in addition to hostnames.
Supposed the first argument is given a VAR node, 'contains_nonconstant_walker' would return FALSE. It's obviously wrong that VAR doesn't contain anything non-constant,isn't it?
Tracking installing Greenplum on SmartOS.
CPPFLAGS='-I/opt/local/include' LDFLAGS='-L/opt/local/lib' ./configure --with-openssl --with-ldap --with-libcurl --prefix=/home/admin/gpsql
make
make installFunction 'InitPMI' needs to check the legitimacy of 'partitionOid' parameter.
Otherwise, the following test failed.
mycluster=# drop table t2;
DROP TABLE
mycluster=# create table t1(tc1 int, tc2 int) distributed by (tc1);
CREATE TABLE
mycluster=# insert into t1 values(1,1);
INSERT 0 1
mycluster=# select pg_partition_oid(1, t1.*) from t1;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>
The stack is as follows:
Program received signal SIGSEGV, Segmentation fault.
0x0000000000a4b919 in selectPartitionMulti (partnode=0x0, values=0x304b310, isnull=0x304b330 "", tupdesc=0x7fd306bfe188, accessMethods=0x302c230) at cdbpartition.c:9018
9018 for (int i = 0; i < candidatePartNode->part->parnatts; i++)
(gdb) bt
#0 0x0000000000a4b919 in selectPartitionMulti (partnode=0x0, values=0x304b310, isnull=0x304b330 "", tupdesc=0x7fd306bfe188, accessMethods=0x302c230) at cdbpartition.c:9018
#1 0x00000000007b90c0 in AddRecordPMI (pmi=0x302a2a0, record=0x2ff91b0) at planpartition.c:1131
#2 0x00000000007b946b in pg_partition_oid_transfn (fcinfo=0x7fff2e9d0ed0) at planpartition.c:1256
#3 0x00000000006a934a in invoke_agg_trans_func (transfn=0x2feb740, numargs=2, transValue=0, noTransvalue=0x2fefe49 "\001", transValueIsNull=0x2fefe48 "\001\001", transtypeByVal=1 '\001',
transtypeLen=4, fcinfo=0x7fff2e9d0ed0, funcctx=0x2fe9ec8, tuplecontext=0x2fc2720, mem_manager=0x2fea190) at nodeAgg.c:478
#4 0x00000000006a91e7 in advance_transition_function (aggstate=0x2fe9ec8, peraggstate=0x2feb708, pergroupstate=0x2fefe40, fcinfo=0x7fff2e9d0ed0, mem_manager=0x2fea190) at nodeAgg.c:399
#5 0x00000000006a96d5 in advance_aggregates (aggstate=0x2fe9ec8, pergroup=0x2fefe40, mem_manager=0x2fea190) at nodeAgg.c:625
#6 0x00000000006aa7dc in agg_retrieve_direct (aggstate=0x2fe9ec8) at nodeAgg.c:1325
#7 0x00000000006aa26f in ExecAgg (node=0x2fe9ec8) at nodeAgg.c:1100
#8 0x00000000006924db in ExecProcNode (node=0x2fe9ec8) at execProcnode.c:979
#9 0x000000000068c4c8 in ExecutePlan (estate=0x2fe9940, planstate=0x2fe9ec8, operation=CMD_SELECT, numberTuples=0, direction=ForwardScanDirection, dest=0x2fb83f8) at execMain.c:2600
#10 0x00000000006897b2 in ExecutorRun (queryDesc=0x2fc8a80, direction=ForwardScanDirection, count=0) at execMain.c:911
#11 0x00000000008502c4 in PortalRunSelect (portal=0x2fc6940, forward=1 '\001', count=0, dest=0x2fb83f8) at pquery.c:1258
#12 0x000000000084ff07 in PortalRun (portal=0x2fc6940, count=9223372036854775807, isTopLevel=1 '\001', dest=0x2fb83f8, altdest=0x2fb83f8, completionTag=0x7fff2e9d19a0 "") at pquery.c:1082
#13 0x00000000008488b6 in exec_simple_query (query_string=0x2f91960 "select pg_partition_oid(16526, t2.*) from t2;", seqServerHost=0x0, seqServerPort=-1) at postgres.c:1731
#14 0x000000000084cef4 in PostgresMain (argc=1, argv=0x2e5a710, dbname=0x2e5a638 "mycluster", username=0x2e5a620 "gpadmin") at postgres.c:4654
#15 0x00000000007efd9f in BackendRun (port=0x2e5fc60) at postmaster.c:6704
#16 0x00000000007ef4a9 in BackendStartup (port=0x2e5fc60) at postmaster.c:6391
#17 0x00000000007e8a73 in ServerLoop () at postmaster.c:2450
#18 0x00000000007e74c5 in PostmasterMain (argc=18, argv=0x2e35050) at postmaster.c:1526
#19 0x000000000071896f in main (argc=18, argv=0x2e35050) at main.c:206
yum install pygresql won't fix the problem.
Adding an empty init.py in $GPHOME/lib/python/pygresql seems to fix the problem.
When I run gpstart -m, PGOPTIONS='-c gp_session_role=utility' psql and update gp_segment_configuration set mode='s' where dbid=3, it arise "Permission denied".
The code I used is commit(ee17fd1).
initGpmonPktFuncs is a jump table, whose size and order of entries is supposed to match the plan-node types in nodes/nodes.h. But it is missing the entry for TableFunctionScan (between FunctionScan and ValueScan), along with the entries for PlanNode types after T_Repeat.
sendInitGpmonPkts uses the expression [nodeTag(node) - T_Plan_Start] as an index into the jump table, so about half the plan types (those after T_FunctionScan) will jump to the wrong init function. Even worse, plans at T_Repeat and after will jump off the end of the table.
There is (supposedly) a compile-time check that the number of entries in the table matches T_Plan_End - T_Plan_Start. But that doesn't guarantee order correctness, nor does it appear to be working as intended.
But it looks like this whole ball of wax is dead code. sendInitGpmonPkts calls itself, but nothing seems to call it.
So... either the dead code should be removed (including the initGpmonPktForXXX functions in each executor/nodeType.c file), or an entry for TableFunctionScan should be added to the initGpmonPktFuncs jump table
It might be nice to have a Dockerfile/image to help people easily try out Greenplum.
Is this expected? :
~/gpdb]$ ./configure --enable-orca
checking build system type... x86_64-apple-darwin14.5.0
checking host system type... x86_64-apple-darwin14.5.0
checking which template to use... darwin
checking whether to build with 64-bit integer date/time support... yes
checking for default port number... 5432
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for __get_cpuid... yes
checking for __cpuid... no
checking for _mm_crc32_u8 and _mm_crc32_u32 with CFLAGS=... no
checking for _mm_crc32_u8 and _mm_crc32_u32 with CFLAGS=-msse4.2... yes
checking which CRC-32C implementation to use... SSE 4.2 with runtime check
checking if gcc supports -Wendif-labels... yes
checking if gcc supports -Wformat-security... yes
checking if gcc supports -fno-strict-aliasing... yes
checking if gcc supports -fwrapv... yes
checking if gcc supports -fexcess-precision=standard... yes
checking if gcc supports -fno-aggressive-loop-optimizations... no
checking if gcc supports -Wno-unused-but-set-variable... yes
checking if gcc supports -Wno-address... yes
checking whether the C compiler still works... yes
checking how to run the C preprocessor... gcc -E
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
enable_orca
checking whether to build with snmp... no
checking whether to build with connectemc... no
no
no
checking allow thread-safe client libraries... yes
checking whether to build with Tcl... no
checking whether to build Perl modules... no
checking whether to build Python modules... no
checking whether to build with GSSAPI support... no
checking whether to build with Kerberos 5 support... no
checking whether to build with PAM support... no
checking whether to build with LDAP support... no
checking whether to build with Bonjour support... no
checking whether to build with OpenSSL support... no
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
configure: using CPPFLAGS=
configure: using LDFLAGS=
checking for ld used by GCC... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking for ranlib... ranlib
checking for strip... strip
checking whether it is possible to strip libraries... no
checking for ar... ar
checking for tar... /usr/bin/tar
checking whether ln -s works... yes
checking for gawk... gawk
checking for a thread-safe mkdir -p... config/install-sh -c -d
checking for bison... /usr/bin/bison
configure: using bison (GNU Bison) 2.3
checking for flex... /usr/bin/flex
configure: using flex 2.5.35 Apple(flex-31)
checking for perl... /usr/bin/perl
configure: using perl 5.18.2
checking for main in -lm... yes
checking for library containing setproctitle... no
checking for library containing dlopen... none required
checking for library containing socket... none required
checking for library containing shl_load... no
checking for library containing getopt_long... none required
checking for library containing crypt... none required
checking for library containing fdatasync... none required
checking for library containing gethostbyname_r... no
checking for library containing shmget... none required
checking for library containing readline... -ledit
checking for inflate in -lz... yes
checking for clock_gettime in -lrt... no
configure: WARNING: Realtime library not found
checking for curl-config... /usr/bin/curl-config
checking for curl >= 7.19.0... yes
checking CURL_CFLAGS...
checking CURL_LIBS... -lcurl
checking whether CURLOPT_MAIL_FROM is declared... yes
checking for library containing BZ2_bzDecompress... -lbz2
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking gpopt/init.h usability... no
checking gpopt/init.h presence... no
checking for gpopt/init.h... no
configure: error: gpopt/init.h GPOPT header file is required for Greenplum Query Optimizer (orca)
test boolean ... gpdiff: ./expected/boolean.out: No such file or directory
Did we missed expected result in git?
Due to gp_dynamic_partition_pruning setted on/off, different result sets were gained.
A case as follow:
mycluster=# drop table t1,t2;
DROP TABLE
mycluster=# drop sequence cs1;
DROP SEQUENCE
mycluster=# create table t1(tc1 int, tc2 int) distributed by (tc1) partition by range(tc1) (partition p1 end(10), partition p2 end(20));
NOTICE: CREATE TABLE will create partition "t1_1_prt_p1" for table "t1"
NOTICE: CREATE TABLE will create partition "t1_1_prt_p2" for table "t1"
CREATE TABLE
mycluster=# create table t2(tc1 int, tc2 int) distributed randomly;
CREATE TABLE
mycluster=# insert into t1 values(2,2);
INSERT 0 1
mycluster=# insert into t2 values(2,2);
INSERT 0 1
mycluster=# create sequence cs1;
CREATE SEQUENCE
mycluster=# select nextval('cs1');
nextval
---------
1
(1 row)
mycluster=# set gp_dynamic_partition_pruning=on;
SET
mycluster=# explain select * from t1, t2 where t1.tc1 = t2.tc1 and t2.tc1 = nextval('cs1');
QUERY PLAN
----------------------------------------------------------------------------------------------------
Gather Motion 4:1 (slice3; segments: 4) (cost=2.08..13.94 rows=10 width=16)
-> Hash Join (cost=2.08..13.94 rows=3 width=16)
Hash Cond: public.t1.tc1 = public.t2.tc1
InitPlan (slice4)
-> Aggregate (cost=1.02..1.03 rows=1 width=32)
-> Gather Motion 4:1 (slice2; segments: 4) (cost=0.00..1.01 rows=1 width=8)
-> Seq Scan on t2 (cost=0.00..1.01 rows=1 width=8)
Filter: tc1 = nextval('cs1'::regclass)
-> Append (cost=0.00..9.61 rows=216 width=8)
-> Result (cost=0.00..9.61 rows=216 width=8)
One-Time Filter: 41376::oid = ANY ($0)
-> Seq Scan on t1_1_prt_p1 t1 (cost=0.00..9.61 rows=216 width=8)
-> Result (cost=0.00..0.00 rows=1 width=8)
One-Time Filter: 41403::oid = ANY ($0)
-> Seq Scan on t1_1_prt_p2 t1 (cost=0.00..0.00 rows=1 width=8)
-> Hash (cost=1.03..1.03 rows=1 width=8)
-> Redistribute Motion 4:4 (slice1; segments: 4) (cost=0.00..1.03 rows=1 width=8)
Hash Key: public.t2.tc1
-> Seq Scan on t2 (cost=0.00..1.01 rows=1 width=8)
Filter: tc1 = nextval('cs1'::regclass)
Settings: gp_dynamic_partition_pruning=on
(21 rows)
mycluster=# select * from t1, t2 where t1.tc1 = t2.tc1 and t2.tc1 = nextval('cs1');
tc1 | tc2 | tc1 | tc2
-----+-----+-----+-----
(0 rows)
mycluster=# drop sequence cs1;
DROP SEQUENCE
mycluster=# create sequence cs1;
CREATE SEQUENCE
mycluster=# select nextval('cs1');
nextval
---------
1
(1 row)
mycluster=# set gp_dynamic_partition_pruning=off;
SET
mycluster=# explain select * from t1, t2 where t1.tc1 = t2.tc1 and t2.tc1 = nextval('cs1');
QUERY PLAN
----------------------------------------------------------------------------------------------------
Gather Motion 4:1 (slice2; segments: 4) (cost=1.05..12.91 rows=10 width=16)
-> Hash Join (cost=1.05..12.91 rows=3 width=16)
Hash Cond: public.t1.tc1 = t2.tc1
-> Append (cost=0.00..9.61 rows=216 width=8)
-> Seq Scan on t1_1_prt_p1 t1 (cost=0.00..9.61 rows=216 width=8)
-> Seq Scan on t1_1_prt_p2 t1 (cost=0.00..0.00 rows=1 width=8)
-> Hash (cost=1.03..1.03 rows=1 width=8)
-> Redistribute Motion 4:4 (slice1; segments: 4) (cost=0.00..1.03 rows=1 width=8)
Hash Key: t2.tc1
-> Seq Scan on t2 (cost=0.00..1.01 rows=1 width=8)
Filter: tc1 = nextval('cs1'::regclass)
Settings: gp_dynamic_partition_pruning=off
(12 rows)
mycluster=# select * from t1, t2 where t1.tc1 = t2.tc1 and t2.tc1 = nextval('cs1');
tc1 | tc2 | tc1 | tc2
-----+-----+-----+-----
2 | 2 | 2 | 2
(1 row)
In my opinion, parameters for performance optimization couldn't bring impact on the accuracy of results.Maybe, some extra restrictions at 'MatchPartitionJoinPattern' or somewhere else are need.
Here is that, i think, righttree can't contain volatile function 'nextval'.
Hello Guys, I was playing with greenplum and I have seen that git repository is getting updated on hourly basis which is amazing. I have encountered some bugs which are already reported due to the merge of upstream patches.
To overcome this issue I have used the binaries provided from pivotal, but the licence there is not apache2.
So my question to the community:
Where I can find stable releases?
To pivotal:
I know that some parts of greenplum still commercial such as command centre, but what about the database server binaries such as greenplum-db-4.3.6.2-build-1-RHEL5-x86_64 and so on
○ → cat regression.diffs
*** ./expected/mapred.out 2015-11-16 18:03:18.068985485 +0800
--- ./results/mapred.out 2015-11-16 18:03:18.096984211 +0800
***************
*** 97,104 ****
$$ language plpythonu NO SQL;
SELECT python_version() FROM env GROUP BY python_version;
python_version
! -----------------------
! (2, 6, 2, 'final', 0)
(1 row)
--
--- 98,105 ----
$$ language plpythonu NO SQL;
SELECT python_version() FROM env GROUP BY python_version;
python_version
! -----------------------------------------------------------------------------
! sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0)
(1 row)
--
======================================================================
Hi,
Some of the segment instances are failing to start when cd gpAux/gpdemo,make cluster were executed
20151110:00:51:23:031307 gpstart:centos7:kotti-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
............
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-Process results...
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-DBID:2 FAILED host:'centos7' datadir:'/home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0' with reason:'Start failed; check segment logfile. "peer shut down connection before response was fully received Retrying no 1 failure: OtherTransitionInProgress failure: OtherTransitionInProgress"'
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:- Successful segment starts = 5
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Failed segment starts = 1 <<<<<<<<
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-Successfully started 5 of 6 segment instances <<<<<<<<
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Segment instance startup failures reported
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Failed start 1 of 6 segment instances <<<<<<<<
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Review /home/kotti/gpAdminLogs/gpstart_20151110.log
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-For more details on segment startup failure(s)
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Run gpstate -s to review current segment instance status
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-Starting Master instance centos7 directory /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1
20151110:00:51:37:031307 gpstart:centos7:kotti-[INFO]:-Command pg_ctl reports Master centos7 instance active
20151110:00:53:16:031307 gpstart:centos7:kotti-[WARNING]:-FATAL: DTM initialization: failure during startup recovery, retry failed, check segment status (cdbtm.c:1603)
20151110:00:53:16:031307 gpstart:centos7:kotti-[INFO]:-No standby master configured. skipping...
20151110:00:53:16:031307 gpstart:centos7:kotti-[WARNING]:-Number of segments which failed to start: 1
20151110:00:53:19:013057 gpinitsystem:centos7:kotti-[WARN]:
20151110:00:53:19:013057 gpinitsystem:centos7:kotti-[WARN]:-Failed to start Greenplum instance; review gpstart output to
/home/kotti/gpAdminLogs/gpstart_20151110.log has the following details :
20151110:00:51:20:031307 gpstart:centos7:kotti-[INFO]:-Starting gpstart with args: -a -d /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-
1
20151110:00:51:20:031307 gpstart:centos7:kotti-[INFO]:-Gathering information and validating the environment...
20151110:00:51:20:031307 gpstart:centos7:kotti-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.99.00 build dev'
20151110:00:51:20:031307 gpstart:centos7:kotti-[INFO]:-Greenplum Catalog Version: '201310150'
20151110:00:51:20:031307 gpstart:centos7:kotti-[INFO]:-Starting Master instance in admin mode
20151110:00:51:21:031307 gpstart:centos7:kotti-[INFO]:-Obtaining Greenplum Master catalog information
20151110:00:51:21:031307 gpstart:centos7:kotti-[INFO]:-Obtaining Segment details from master...
20151110:00:51:22:031307 gpstart:centos7:kotti-[INFO]:-Setting new master era
20151110:00:51:22:031307 gpstart:centos7:kotti-[INFO]:-Master Started...
20151110:00:51:22:031307 gpstart:centos7:kotti-[INFO]:-Shutting down master
20151110:00:51:23:031307 gpstart:centos7:kotti-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-Process results...
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-DBID:2 FAILED host:'centos7' datadir:'/home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/dbfast1/demo
DataDir0' with reason:'Start failed; check segment logfile. "peer shut down connection before response was fully received Retrying no 1 failure: OtherTr
ansitionInProgress failure: OtherTransitionInProgress"'
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:- Successful segment starts = 5
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Failed segment starts = 1 <<<<<<<<
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-Successfully started 5 of 6 segment instances <<<<<<<<
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Segment instance startup failures reported
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Failed start 1 of 6 segment instances <<<<<<<<
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Review /home/kotti/gpAdminLogs/gpstart_20151110.log
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-For more details on segment startup failure(s)
20151110:00:51:35:031307 gpstart:centos7:kotti-[WARNING]:-Run gpstate -s to review current segment instance status
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-----------------------------------------------------
20151110:00:51:35:031307 gpstart:centos7:kotti-[INFO]:-Starting Master instance centos7 directory /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/qddir/dem
oDataDir-1
20151110:00:51:37:031307 gpstart:centos7:kotti-[INFO]:-Command pg_ctl reports Master centos7 instance active
20151110:00:53:16:031307 gpstart:centos7:kotti-[WARNING]:-FATAL: DTM initialization: failure during startup recovery, retry failed, check segment status (
cdbtm.c:1603)
20151110:00:53:16:031307 gpstart:centos7:kotti-[INFO]:-No standby master configured. skipping...
20151110:00:53:16:031307 gpstart:centos7:kotti-[WARNING]:-Number of segments which failed to start: 1
pg_log from the failed segment seems to have the following details :
2015-11-10 00:53:24.004474 CST,,,p15205,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","mirror transition, primary address(port) 'centos7(25438)' mirror address(port) 'centos7(25441)'",,,,,"mirroring role 'primary role' mirroring state 'sync' segment state 'not initialized' process name(pid) 'filerep main process(15205)' filerep state 'not initialized' ",,0,,"cdbfilerep.c",3472,
2015-11-10 00:53:24.035837 CST,,,p15210,th0,,,,0,,,seg-1,,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1 0x8c65a3 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x163
2 0x7f346dc88130 libpthread.so.0 + 0x6dc88130
3 0x978893 postgres FileRep_CalculateCrc + 0x143
4 0x97ca9b postgres + 0x97ca9b
5 0x97d4e6 postgres FileRepPrimary_MirrorOpen + 0x386
6 0x9baf5e postgres + 0x9baf5e
7 0x9bb895 postgres MirroredBufferPool_Open + 0x185
8 0x7cc531 postgres + 0x7cc531
9 0x7cdb90 postgres mdnblocks + 0xf0
10 0x7d0470 postgres smgrnblocks + 0x10
11 0x4ae9ee postgres heap_beginscan + 0x7e
12 0x4b7c89 postgres systable_beginscan + 0x79
13 0x8d4610 postgres FindMyDatabase + 0x60
14 0x97b679 postgres + 0x97b679
15 0x97bec5 postgres FileRepSubProcess_Main + 0x295
16 0x9756a6 postgres + 0x9756a6
17 0x97aff3 postgres FileRep_Main + 0xba3
18 0x53c8a8 postgres AuxiliaryProcessMain + 0x508
19 0x775980 postgres + 0x775980
20 0x780b0c postgres StartFilerepProcesses + 0x3c
21 0x7834a7 postgres doRequestedPrimaryMirrorModeTransitions + 0x8b7
22 0x77d412 postgres + 0x77d412
23 0x77f829 postgres PostmasterMain + 0x789
24 0x485fab postgres main + 0x3bb
25 0x7f346d5d7af5 libc.so.6 __libc_start_main + 0xf5
26 0x4860c9 postgres + 0x4860c9
"
2015-11-10 00:53:24.050735 CST,,,p15209,th0,,,,0,,,seg-1,,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1 0x8c65a3 postgres StandardHandlerForSigillSigsegvSigbus_OnMainThread + 0x163
2 0x7f346dc88130 libpthread.so.0 + 0x6dc88130
3 0x978893 postgres FileRep_CalculateCrc + 0x143
4 0x97ca9b postgres + 0x97ca9b
5 0x97d4e6 postgres FileRepPrimary_MirrorOpen + 0x386
6 0x9baf5e postgres + 0x9baf5e
7 0x9bb895 postgres MirroredBufferPool_Open + 0x185
8 0x7cc531 postgres + 0x7cc531
9 0x7cdb90 postgres mdnblocks + 0xf0
10 0x7d0470 postgres smgrnblocks + 0x10
11 0x4ae9ee postgres heap_beginscan + 0x7e
12 0x4b7c89 postgres systable_beginscan + 0x79
13 0x8d4610 postgres FindMyDatabase + 0x60
14 0x97b679 postgres + 0x97b679
15 0x97bf05 postgres FileRepSubProcess_Main + 0x2d5
16 0x9756a6 postgres + 0x9756a6
17 0x97afe1 postgres FileRep_Main + 0xb91
18 0x53c8a8 postgres AuxiliaryProcessMain + 0x508
19 0x775980 postgres + 0x775980
20 0x780b0c postgres StartFilerepProcesses + 0x3c
21 0x7834a7 postgres doRequestedPrimaryMirrorModeTransitions + 0x8b7
22 0x77d412 postgres + 0x77d412
23 0x77f829 postgres PostmasterMain + 0x789
24 0x485fab postgres main + 0x3bb
25 0x7f346d5d7af5 libc.so.6 __libc_start_main + 0xf5
26 0x4860c9 postgres + 0x4860c9
"
2015-11-10 00:53:24.050809 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",5563,
2015-11-10 00:53:24.093364 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","filerep main process (PID 15205) exited with exit code 2",,,,,,,0,,"postmaster.c",5854,
2015-11-10 00:53:24.093395 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","BeginResetOfPostmasterAfterChildrenAreShutDown: counter 234",,,,,,,0,,"postmaster.c",2177,
2015-11-10 00:53:24.093405 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","gp_session_id high-water mark is 0",,,,,,,0,,"postmaster.c",2203,
2015-11-10 00:53:24.158949 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","resetting shared memory",,,,,,,0,,"postmaster.c",4249,
2015-11-10 00:53:24.158994 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","temporary files using default filespace",,,,,,,0,,"primary_mirror_mode.c",2569,
2015-11-10 00:53:24.159003 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","transaction files using default pg_system filespace",,,,,,,0,,"primary_mirror_mode.c",2629,
2015-11-10 00:53:24.237936 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","PrimaryMirrorMode: Processing postmaster reset with recent mode of 4",,,,,,,0,,"primary_mirror_mode.c",1124,
2015-11-10 00:53:24.237973 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","PrimaryMirrorMode: Processing postmaster reset to non-fault state",,,,,,,0,,"primary_mirror_mode.c",1141,
2015-11-10 00:53:24.237987 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","removing all temporary files",,,,,,,0,,"fd.c",1883,
2015-11-10 00:53:24.237994 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","BeginResetOfPostmasterAfterChildrenAreShutDown: should restart peer",,,,,,,0,,"postmaster.c",2226,
2015-11-10 00:53:24.238000 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","all server processes terminated; requested filerep peer reset",,,,,,,0,,"postmaster.c",2229,
2015-11-10 00:53:24.238006 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","peer reset process pid is 15236",,,,,,,0,,"postmaster.c",2231,
2015-11-10 00:53:24.419306 CST,,,p31434,th1857087552,,,,0,,,seg-1,,,,,"LOG","00000","received immediate shutdown request",,,,,,,0,,"postmaster.c",4409,
2015-11-10 00:53:24.421245 CST,,,p15236,th1857087552,,,,0,,,seg-1,,,,,"WARNING","01000","during reset, unable to contact primary/mirror peer to coordinate reset; will transition to fault state. Error code 16 and message 'failure: interrupted","'",,,,,,0,,"cdbfilerepresetpeerprocess.c",158,
Please advise if I am missing something here.
Current analyze.c in GPDB is different from PostgreSQL version.
Here are few differences observed:
PGSQL has:
acquire_inherited_sample_rows
acquire_sample_rows
analyze_rel
compare_mcvs
compare_rows
compare_scalars
compute_distinct_stats
compute_index_stats
compute_scalar_stats
compute_trivial_stats
...
Where the GPDB has:
analyzeComputeAttributeStatistics
analyzeComputeAverageWidth
analyzeComputeHistogram
...
analyzeNullCount
...
gp_statistics_estimate_reltuples_relpages_ao_cs
...
spiCallback_getEachResultComplumnAsArray
...
This issue is created to make the analyze.c matching with upstream.
when building with --enable-orca :
In file included from CConfigParamMapping.cpp:19:0:
../../../../src/include/gpopt/config/CConfigParamMapping.h:18:23: fatal error: gpos/base.h: No such file or directory
#include "gpos/base.h"
I have attached a script to prove ORCA doesn't support GiST index:
test_query_plan.sql.zip
test=# CREATE INDEX rtree_table_o ON o_table USING gist(geom);
CREATE INDEX
Time: 24.335 ms
test=# CREATE INDEX rtree_table_h ON h_table USING gist(geom);
CREATE INDEX
Time: 8.377 mstest=# EXPLAIN SELECT count(*) FROM o_table AS o, h_table AS h WHERE ST_DWITHIN(o.geom, h.geom, 0.0005);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
-------------------------------
Aggregate (cost=6394.99..6395.00 rows=1 width=8)
-> Gather Motion 2:1 (slice2; segments: 2) (cost=6394.94..6394.98 rows=1 width=8)
-> Aggregate (cost=6394.94..6394.95 rows=1 width=8)
-> Nested Loop (cost=0.00..6394.90 rows=7 width=0)
Join Filter: h.geom && st_expand(o.geom, 0.0005::double precision) AND _st_dwithin(o.geom, h.g
eom, 0.0005::double precision)
-> Broadcast Motion 2:2 (slice1; segments: 2) (cost=0.00..14.00 rows=300 width=32)
-> Seq Scan on h_table h (cost=0.00..5.00 rows=150 width=32)
-> Index Scan using rtree_table_o on o_table o (cost=0.00..10.61 rows=1 width=32)
Index Cond: o.geom && st_expand(h.geom, 0.0005::double precision)
Settings: optimizer=off
Optimizer status: legacy query optimizer
(11 rows)test=# ANALYZE o_table;
ANALYZE
Time: 57.722 ms
test=# ANALYZE h_table;
ANALYZE
Time: 22.876 ms
test=# EXPLAIN SELECT count(*) FROM o_table AS o, h_table AS h WHERE ST_DWITHIN(o.geom, h.geom, 0.0005);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
-------------------------------
Aggregate (cost=7112.09..7112.10 rows=1 width=8)
-> Gather Motion 2:1 (slice2; segments: 2) (cost=7112.04..7112.08 rows=1 width=8)
-> Aggregate (cost=7112.04..7112.05 rows=1 width=8)
-> Nested Loop (cost=0.00..7112.00 rows=7 width=0)
Join Filter: o.geom && st_expand(h.geom, 0.0005::double precision) AND _st_dwithin(o.geom, h.g
eom, 0.0005::double precision)
-> Broadcast Motion 2:2 (slice1; segments: 2) (cost=0.00..422.00 rows=10000 width=32)
-> Seq Scan on o_table o (cost=0.00..122.00 rows=5000 width=32)
-> Index Scan using rtree_table_h on h_table h (cost=0.00..0.31 rows=1 width=32)
Index Cond: h.geom && st_expand(o.geom, 0.0005::double precision)
Settings: optimizer=off
Optimizer status: legacy query optimizer
(11 rows)test=# SET OPTIMIZER = ON;
SET
Time: 9.593 ms
test=# SHOW OPTIMIZER;
optimizer
-----------
on
(1 row)
Time: 0.317 ms
test=# EXPLAIN SELECT count(*) FROM o_table AS o, h_table AS h WHERE ST_DWITHIN(o.geom, h.geom, 0.0005);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------
----------
Aggregate (cost=0.00..1479062.58 rows=1 width=8)
-> Gather Motion 2:1 (slice2; segments: 2) (cost=0.00..1479062.58 rows=1 width=8)
-> Aggregate (cost=0.00..1479062.58 rows=1 width=8)
-> Nested Loop (cost=0.00..1479062.58 rows=404544 width=1)
Join Filter: o_table.geom && st_expand(h_table.geom, 0.0005::double precision) AND h_table.geo
m && st_expand(o_table.geom, 0.0005::double precision) AND _st_dwithin(o_table.geom, h_table.geom, 0.0005::double p
recision)
-> Broadcast Motion 2:2 (slice1; segments: 2) (cost=0.00..431.27 rows=300 width=32)
-> Table Scan on h_table (cost=0.00..431.01 rows=150 width=32)
-> Table Scan on o_table (cost=0.00..431.18 rows=5000 width=32)
Settings: optimizer=on
Optimizer status: PQO version 1.597
(10 rows)
Time: 62.469 ms| Index | Tuples returned | Time consumed (ms) | |
|---|---|---|---|
| Legacy query optimizer | Index BIG table | 19586 | 245.160 |
| Index SMALL table | 19586 | 101633.243 | |
| enable ORCA | Index both | 19586 | 1818594.344 |
I read this function FileRepPrimary_ResyncBufferPoolIncrementalWrite() and find this code may be not correct.
As far as i know that before excute incwrite we got a series of CTL info, and i think the CTL end lsn should be equal or bigger than page lsn, because we collect CTL information from xlog records which write ahead data.
So it's hard to understand this code, the condition is not consistent with the ereport info.
if (XLByteLE(result->entries[ii].lsn_end, PageGetLSN(page))) { if (! XLByteEQ(PageGetLSN(page), result->entries[ii].lsn_end)) { ereport(LOG, (errmsg("Resynchonize buffer pool relation '%u/%u/%u' block '%d' has page lsn less than CT lsn, " "lsn end change tracking '%s(%u/%u)' lsn page '%s(%u/%u)' " "number of blocks '%d'", smgr_relation->smgr_rnode.spcNode, smgr_relation->smgr_rnode.dbNode, smgr_relation->smgr_rnode.relNode, result->entries[ii].block_num, XLogLocationToString(&loc), loc.xlogid, loc.xrecoff, XLogLocationToString(&loc1), loc1.xlogid, loc1.xrecoff, numBlocks), FileRep_errcontext())); } smgrwrite(smgr_relation, result->entries[ii].block_num, (char *)BufferGetBlock(buf), FALSE); }
Maybe i think, the condition should be like this:
if (XLByteLE(PageGetLSN(page), result->entries[ii].lsn_end))
I wanted to use gphdfs extension as dependency in my MapReduce job converting from Avro to GPWritable. I searched public Maven repositories and I couldn't find it. Can you please publish this library to a public Maven repository?
Hi,
It appears make cluster is failing on centos7, below is the standard o/p
______ _____ ______ _______ _______ _____
| ____ |_____] | \ |______ | | | | |
|_____| | |_____/ |______ | | | |_____|
This is a demo of the Greenplum Database system. We will create
a cluster installation with master and 6 segment instances
(3 primary & 3 mirror).
GPHOME ................. : /home/kotti/projects/gpdb
MASTER_DATA_DIRECTORY .. : /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1
MASTER PORT (PGPORT) ... : 15432
SEGMENT PORTS .......... : 25432 25433 25434 25435 25436 25437 25438 25439 25440 25441 25442 25443
NOTE(s):
* The DB ports identified above must be available for use.
* An environment file gpdemo-env.sh has been created for your use.
executing:
20151107:15:48:03:003610 gpinitsystem:centos7:kotti-[INFO]:-Checking configuration parameters, please wait...
20151107:15:48:03:003610 gpinitsystem:centos7:kotti-[INFO]:-Reading Greenplum configuration file clusterConfigFile
20151107:15:48:03:003610 gpinitsystem:centos7:kotti-[INFO]:-Locale has not been set in clusterConfigFile, will set to default value
/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
20151107:15:48:03:003610 gpinitsystem:centos7:kotti-[INFO]:-Locale set to en_US.utf8
/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-No DATABASE_NAME set, will exit following template1 updates
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 25
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-Detected a single host GPDB array build, reducing value of BATCH_DEFAULT from 60 to 4
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-Checking configuration parameters, Completed
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-Commencing multi-home checks, please wait...
.
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-Configuring build for standard array
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-Commencing multi-home checks, Completed
20151107:15:48:04:003610 gpinitsystem:centos7:kotti-[INFO]:-Building primary segment instance array, please wait...
...
20151107:15:48:05:003610 gpinitsystem:centos7:kotti-[INFO]:-Building group mirror array type , please wait...
...
20151107:15:48:05:003610 gpinitsystem:centos7:kotti-[INFO]:-Checking Master host
20151107:15:48:05:003610 gpinitsystem:centos7:kotti-[INFO]:-Checking new segment hosts, please wait...
/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
......
20151107:15:48:06:003610 gpinitsystem:centos7:kotti-[INFO]:-Checking new segment hosts, Completed
20151107:15:48:07:003610 gpinitsystem:centos7:kotti-[INFO]:-Building the Master instance database, please wait...
20151107:15:48:14:003610 gpinitsystem:centos7:kotti-[INFO]:-Starting the Master in admin mode
20151107:15:48:56:003610 gpinitsystem:centos7:kotti-[INFO]:-Commencing parallel build of primary segment instances
20151107:15:48:56:003610 gpinitsystem:centos7:kotti-[INFO]:-Spawning parallel processes batch [1], please wait...
...
20151107:15:48:56:003610 gpinitsystem:centos7:kotti-[INFO]:-Waiting for parallel processes batch [1], please wait...
.........................
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:------------------------------------------------
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:-Parallel process exit status
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:------------------------------------------------
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:-Total processes marked as completed = 3
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:-Total processes marked as killed = 0
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:-Total processes marked as failed = 0
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:------------------------------------------------
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:-Commencing parallel build of mirror segment instances
20151107:15:49:22:003610 gpinitsystem:centos7:kotti-[INFO]:-Spawning parallel processes batch [1], please wait...
...
20151107:15:49:23:003610 gpinitsystem:centos7:kotti-[INFO]:-Waiting for parallel processes batch [1], please wait...
..
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:------------------------------------------------
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-Parallel process exit status
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:------------------------------------------------
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-Total processes marked as completed = 0
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-Total processes marked as killed = 0
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[WARN]:-Total processes marked as failed = 3 <<<<<
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:------------------------------------------------
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[FATAL]:-Errors generated from parallel processes
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-Dumped contents of status file to the log file
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-Building composite backout file
20151107:15:49:25:gpinitsystem:centos7:kotti-[FATAL]:-Failures detected, see log file /home/kotti/gpAdminLogs/gpinitsystem_20151107.log for more detail Script Exiting!
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[WARN]:-Script has left Greenplum Database in an incomplete state
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[WARN]:-Run command /bin/bash /home/kotti/gpAdminLogs/backout_gpinitsystem_kotti_20151107_154803 to remove these changes
20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-Start Function BACKOUT_COMMAND
#20151107:15:49:25:003610 gpinitsystem:centos7:kotti-[INFO]:-End Function BACKOUT_COMMAND
OPTIMIZER STATE
make: [cluster] Error 2 (ignored)
The /home/kotti/gpAdminLogs/gpinitsystem_20151107.log has the following details :
20151107:14:37:13:008746 gpcreateseg.sh:centos7:kotti-[INFO]:-Commencing remote /home/kotti/projects/gpdb/gpAux/gpdemo/lalshell centos7 export GPHOME=/home/kotti/projects/gpdb; . /h
ome/kotti/projects/gpdb/greenplum_path.sh; /home/kotti/projects/gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -x postmaster.pid /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs
/dbfast3/demoDataDir2 [centos7]:/home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[FATAL]:-Unexpected EOF on RemotePysync output stream
20151107:14:37:14:008746 gpcreateseg.sh:centos7:kotti-[FATAL]:- Command export GPHOME=/home/kotti/projects/gpdb; . /home/kotti/projects/gpdb/greenplum_path.sh; /home/kotti/projects/
gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -x postmaster.pid /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2 [centos7]:/home/kotti/projects/gpdb/gpAu
x/gpdemo/datadirs/dbfast_mirror3/demoDataDir2 on centos7 failed with error status 3
20151107:14:37:14:008746 gpcreateseg.sh:centos7:kotti-[INFO]:-End Function RUN_COMMAND_REMOTE
20151107:14:37:14:008746 gpcreateseg.sh:centos7:kotti-[FATAL][2]:-Failed remote copy of segment data directory from centos7 to centos7
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[FATAL]:-Unexpected EOF on RemotePysync output stream
20151107:14:37:19:008577 gpcreateseg.sh:centos7:kotti-[FATAL]:- Command exportGPHOME=/home/kotti/projects/gpdb; . /home/kotti/projects/gpdb/greenplum_path.sh; /home/kotti/projects/
gpdb/bin/lib/pysync.py -x pg_log -x postgresql.conf -x postmaster.pid /home/kotti/projects/gpdb/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0 [centos7]:/home/kotti/projects/gpdb/gpAu
x/gpdemo/datadirs/dbfast_mirror1/demoDataDir0 on centos7 failed with error status 3
20151107:14:37:19:008577 gpcreateseg.sh:centos7:kotti-[INFO]:-End Function RUN_COMMAND_REMOTE
20151107:14:37:19:008577 gpcreateseg.sh:centos7:kotti-[FATAL][0]:-Failed remote copy of segment data directory from centos7 to centos7
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[FATAL]:-Unexpected EOF on RemotePysync output stream
Please advise....
Currently used by gpinitsystem and gpcreateseg.sh and defined in the function "ED_PG_CONF" (defined in gpMgmt/bin/lib/gp_bash_functions.sh), replace the use of "ed" (line-oriented text editor) with a more readily available package (example: sed).
If "ed" is not available, one will see somewhat messages similar to "[WARN]:-Failed to replace #port in */postgresql.conf". One needs to hunt around to actually see that "ed" is missing from the runtime environment.
I have a set of servers I'd like to provision. The master-to-be can ssh into the segments-to-be without a password because mykey.pem is specified in the ~/.ssh/config file (with the line IdentityFile /home/me/.ssh/mykey.pem)
Because gpseginstall and gpssh-exkeys skip the SSH config, they prompt for a password in order to exchange keys and/or log in.
It would make things easier if there was a command-line argument to specify the key file to use when running gpseginstall, gpssh-exkeys, and the like (perhaps -i).
So clearly the README has some missing steps when trying to compile and start the database for the first time. After doing a successful 'configure', 'make' and 'make install', the steps go straight to the 'gpdemo' part, but that doesn't work, it says that it cannot connect to the database. After digging around, I found that 'gpstart' is how you start the database engine, but even after setting MASTER_DATA_DIRECTORY to a random folder, it complains about missing files. What step is missing? How do you create a starting database and all related files so we can start the engine?
platform:ubuntu(32bit) 15.04 (Linux ubuntu 3.19.0-28-generic #30-Ubuntu SMP Mon Aug 31 15:52:10 UTC 2015 i686 i686 i686 GNU/Linux)
gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2
./configure --enable-debug --enable-cassert
______ _____ ______ _______ _______ _____
| ____ |_____] | \ |______ | | | | |
|_____| | |_____/ |______ | | | |_____|
This is a demo of the Greenplum Database system. We will create
a cluster installation with master and 6 segment instances
(3 primary & 3 mirror).
GPHOME ................. : /usr/local/pgsql
MASTER_DATA_DIRECTORY .. : /home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1
MASTER PORT (PGPORT) ... : 15432
SEGMENT PORTS .......... : 25432 25433 25434 25435 25436 25437 25438 25439 25440 25441 25442 25443
NOTE(s):
* The DB ports identified above must be available for use.
* An environment file gpdemo-env.sh has been created for your use.
executing:
20151111:10:03:56:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Checking configuration parameters, please wait...
20151111:10:03:56:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Reading Greenplum configuration file clusterConfigFile
20151111:10:03:56:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Locale has not been set in clusterConfigFile, will set to default value
20151111:10:03:56:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Locale set to en_US.utf8
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-No DATABASE_NAME set, will exit following template1 updates
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 25
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Detected a single host GPDB array build, reducing value of BATCH_DEFAULT from 60 to 4
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Checking configuration parameters, Completed
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Commencing multi-home checks, please wait...
.
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Configuring build for standard array
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Commencing multi-home checks, Completed
20151111:10:03:57:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Building primary segment instance array, please wait...
...
20151111:10:03:58:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Building group mirror array type , please wait...
...
20151111:10:03:59:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Checking Master host
20151111:10:03:59:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Checking new segment hosts, please wait...
......
20151111:10:04:01:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Checking new segment hosts, Completed
20151111:10:04:01:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Building the Master instance database, please wait...
#20151111:10:04:09:gpinitsystem:ubuntu:gpadmin-[FATAL]:- Command /usr/local/pgsql/bin/initdb -E UNICODE -D /home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 --locale=en_US.utf8 --max_connections=25 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1.initdb failed with error status 1, see log file /home/gpadmin/gpAdminLogs/gpinitsystem_20151111.log for more detail Script Exiting!
gpinitsystem_20151111.log:
20151111:10:04:01:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Building the Master instance database, please wait...
20151111:10:04:01:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Initializing Master Postgres instance /home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1
20151111:10:04:01:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Commencing local /usr/local/pgsql/bin/initdb -E UNICODE -D /home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 --locale=en_US.utf8 --max_connections=25 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1.initdb
The files belonging to this database system will be owned by user "gpadmin".
This user must also own the server process.
The database cluster will be initialized with locale en_US.utf8.
creating directory /home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 ... ok
creating subdirectories ... ok
selecting default max_connections ... initdb: error 35584 from: "/usr/local/pgsql/bin/postgres" --boot -x0 -F -c max_connections=25 -c shared_buffers=4000 -c max_fsm_pages=200000 < "/dev/null" > "/home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1.initdb" 2>&1
initdb: removing data directory "/home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1"
2015-11-11 02:04:02.026597 GMT,,,p7131,th-1219598592,,,,0,,,seg-1,,,,,"WARNING","01000","""fsync"": can not be set by the user and will be ignored.",,,,,,,,"set_config_option","guc.c",9622,
Segmentation fault (core dumped)
20151111:10:04:09:002381 gpinitsystem:ubuntu:gpadmin-[INFO]:-Start Function ERROR_EXIT
20151111:10:04:09:gpinitsystem:ubuntu:gpadmin-[FATAL]:- Command /usr/local/pgsql/bin/initdb -E UNICODE -D /home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 --locale=en_US.utf8 --max_connections=25 --shared_buffers=128000kB --is_filerep_mirrored=no --backend_output=/home/gpadmin/projects/gpdb_working/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1.initdb failed with error status 1, see log file /home/gpadmin/gpAdminLogs/gpinitsystem_20151111.log for more detail Script Exiting!
gdb postgres core
Did you ever see this on 32 bit linux?Seems like the CMPXCHG access illegal address
Hello
I understand that appendonly tables are optimized for inserting. But since DELETE and UPDATE commands are supported in 4.3, I assume they are supposed to work properly. But they don't, at least DELETE doesn't. Here is an example:
dwh=# CREATE TABLE bar (a int, b text) WITH (appendonly=true) DISTRIBUTED BY (a);
CREATE TABLE
dwh=# INSERT INTO bar SELECT a, a::text FROM generate_series(1,1000000) a;
INSERT 0 1000000
dwh=# SELECT count(*) FROM bar;
count
---------
1000000
(1 row)
dwh=# DELETE FROM bar;
DELETE 1000000
dwh=# SELECT count(*) FROM bar;
count
--------
868928
(1 row)
dwh=# DELETE FROM bar;
DELETE 868928
dwh=# SELECT count(*) FROM bar;
count
--------
868928
(1 row)
And now the completely crazy thing:
dwh=# DELETE FROM bar WHERE a = 500;
DELETE 1
dwh=# DELETE FROM bar WHERE a = 501;
DELETE 1
dwh=# DELETE FROM bar WHERE a = 502;
DELETE 1
dwh=# SELECT count(*) FROM bar;
count
--------
967229
(1 row)
I've tried this on two different installations of GP: same result with only difference in the number of rows that are deleted and then "restored". Am I doing anything wrong?
When I try to compile with the --enable-orca option I receive the following error:
In file included from CConfigParamMapping.cpp:19:0:
../../.././src/include/gpopt/config/CConfigParamMapping.h:18:23: fatal error: gpos/base.h: No such file or directory
A find of the gpdb source directory shows doesn't return a gpos directory.
It's not clear what versions of the Python modules (lockfile, paramiko, etc) are best for use with gpdb.
Please include a requirements.txt file to make it easy to install all of the python dependencies/extras in one shot with pip.
As an open source project, setup a build only continuous integration (CI) process on travis-ci.org.
The same issue is reported and fixed in https://issues.apache.org/jira/browse/HAWQ-63, but not fixed in gpdb.
gpadmin=# drop table public.test123;
DROP TABLE
gpadmin=# create table public.test123 (
gpadmin(# col1 text,
gpadmin(# col2 text,
gpadmin(# col3 int
gpadmin(# )
gpadmin-# with ( appendonly = true, orientation = column, compresslevel = 4, compresstype=zlib)
gpadmin-# distributed by (col1);
CREATE TABLE
gpadmin=# \dt test123
List of relations
Schema | Name | Type | Owner | Storage
--------+---------+-------+---------+----------------------
public | test123 | table | gpadmin | append only columnar
(1 row)
gpadmin=#
gpadmin=# SELECT c.relname, a.compresstype, a.compresslevel, a.blocksize, a.checksum
gpadmin-# FROM pg_catalog.pg_appendonly a, pg_catalog.pg_class c
gpadmin-# WHERE c.oid = a.relid AND c.oid = 'public.test123'::regclass;
relname | compresstype | compresslevel | blocksize | checksum
---------+--------------+---------------+-----------+----------
test123 | zlib | 4 | 32768 | t
(1 row)
gpadmin=# alter table public.test123 set distributed by (col1);
ALTER TABLE
gpadmin=#
gpadmin=# SELECT c.relname, a.compresstype, a.compresslevel, a.blocksize, a.checksum
gpadmin-# FROM pg_catalog.pg_appendonly a, pg_catalog.pg_class c
gpadmin-# WHERE c.oid = a.relid AND c.oid = 'public.test123'::regclass;
relname | compresstype | compresslevel | blocksize | checksum
---------+--------------+---------------+-----------+----------
test123 | | 0 | 32768 | t
(1 row)
gpadmin=#
gpadmin=# \dt test123
List of relations
Schema | Name | Type | Owner | Storage
--------+---------+-------+---------+----------------------
public | test123 | table | gpadmin | append only columnar
(1 row)
Can you add real time Streaming Query and Complex Event Processing capabilities like in: https://www.pipelinedb.com/, http://www.sqlstream.com/, etc.
commit c9a0fa4
Merge: 11c6425 5a2a527
Author: Heikki Linnakangas [email protected]
Date: Thu Nov 26 19:13:48 2015 +0200
Merge commit '5a2a527bb500a7665f4aba26574f2494c5fc0469'
Noth much interesting here, but I wanted to stop before the next upstream
commit, which is a mechanical Copyright year bump.
Conflicts:
contrib/btree_gist/btree_gist.sql.in
doc/FAQ
doc/FAQ_DEV
doc/FAQ_japanese
doc/bug.template
doc/src/FAQ/FAQ.html
doc/src/FAQ/FAQ_DEV.html
doc/src/FAQ/FAQ_japanese.html
doc/src/sgml/ref/pg_ctl-ref.sgml
doc/src/sgml/ref/postgres-ref.sgml
doc/src/sgml/ref/reindex.sgml
doc/src/sgml/release.sgml
src/backend/access/gin/ginvacuum.c
src/backend/access/heap/tuptoaster.c
src/backend/main/main.c
src/backend/postmaster/postmaster.c
src/backend/tcop/postgres.c
src/backend/utils/adt/float.c
src/backend/utils/adt/regexp.c
src/bin/pg_ctl/pg_ctl.c
src/bin/pg_dump/dumputils.c
src/include/catalog/pg_type.h
src/include/pg_config.h.win32
src/include/utils/builtins.h
src/interfaces/ecpg/compatlib/Makefile
src/interfaces/ecpg/ecpglib/Makefile
src/interfaces/ecpg/pgtypeslib/Makefile
src/interfaces/ecpg/preproc/Makefile
src/interfaces/libpq/Makefile
src/interfaces/libpq/libpq.rc.in
src/port/win32ver.rc
src/test/regress/pg_regress.c
src/tools/RELEASE_CHANGES
src/tools/copyright
src/tools/msvc/Project.pm
when i use gpinitsystem init a greenplum database cluster, it's error:
20151127:09:27:36:033798 gpinitsystem:digoal:gp-[INFO]:----------------------------------------
20151127:09:27:36:033798 gpinitsystem:digoal:gp-[INFO]:-Greenplum Primary Segment Configuration
20151127:09:27:36:033798 gpinitsystem:digoal:gp-[INFO]:----------------------------------------
20151127:09:27:36:033798 gpinitsystem:digoal:gp-[INFO]:-digoal /data01/gpdata/gpseg0 40000 2 0
20151127:09:27:36:033798 gpinitsystem:digoal:gp-[INFO]:-digoal /data01/gpdata/gpseg1 40001 3 1
20151127:09:27:36:033798 gpinitsystem:digoal:gp-[INFO]:-digoal /data01/gpdata/gpseg2 40002 4 2
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-End Function DISPLAY_CONFIG
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-Start Function ARRAY_REORDER
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-End Function ARRAY_REORDER
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-Start Function CREATE_QD_DB
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-Building the Master instance database, please wait...
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-Initializing Master Postgres instance /data01/gpdata/gpseg-1
20151127:09:27:37:033798 gpinitsystem:digoal:gp-[INFO]:-Commencing local /opt/gpdb/bin/initdb -E UTF8 -D /data01/gpdata/gpseg-1 --locale=C --max_connections=48 --shared_buffers=256MB --is_filerep_mirrored=no --backend_output=/data01/gpdata/gpseg-1.initdb
The files belonging to this database system will be owned by user "gp".
This user must also own the server process.
The database cluster will be initialized with locale C.
creating directory /data01/gpdata/gpseg-1 ... ok
creating subdirectories ... ok
selecting default max_connections ... 48
selecting default shared_buffers/max_fsm_pages ... 256MB/409600
creating configuration files ... ok
creating template1 database in /data01/gpdata/gpseg-1/base/1 ... 2015-11-27 14:27:37.822588 GMT,,,p38488,th1225431104,,,,0,,,seg-1,,,,,"WARNING","01000","""fsync"": can not be set by the user and will be ignored.",,,,,,,,"set_config_option","guc.c",4325,
2015-11-27 09:27:38.536971 EST,,,p38488,th1225431104,,,,0,con1,,seg-1,,,x1,sx1,"FATAL","XX000","shared_relation flag for ""pg_auth_time_constraint"" does not match IsSharedRelation(6070) (relcache.c:2833)",,,,,,,,"RelationBuildLocalRelation","relcache.c",2833,1 0x8bd8a8 postgres errstart (??:?)
2 0x8bf64b postgres elog_finish (??:?)
3 0x89de0f postgres RelationBuildLocalRelation (??:?)
4 0x549fe7 postgres heap_create (??:?)
5 0x54b70f postgres heap_create_with_catalog (??:?)
6 0x53a67d postgres boot_yyparse (??:?)
7 0x53bf6e postgres AuxiliaryProcessMain (??:?)
8 0x485e52 postgres main (??:?)
9 0x7f6d47b53af5 libc.so.6 __libc_start_main (??:0)
10 0x485f19 postgres <symbol not found> (??:?)
child process exited with exit code 1
initdb: removing data directory "/data01/gpdata/gpseg-1"
2015-11-27 14:27:37.509257 GMT,,,p38487,th1885685824,,,,0,,,seg-1,,,,,"WARNING","01000","""fsync"": can not be set by the user and will be ignored.",,,,,,,,"set_config_option","guc.c",4325,
20151127:09:27:38:033798 gpinitsystem:digoal:gp-[INFO]:-Start Function ERROR_EXIT
20151127:09:27:38:gpinitsystem:digoal:gp-[FATAL]:- Command /opt/gpdb/bin/initdb -E UTF8 -D /data01/gpdata/gpseg-1 --locale=C --max_connections=48 --shared_buffers=256MB --is_filerep_mirrored=no --backend_output=/data01/gpdata/gpseg-1.initdb failed with error status 1, see log file /home/gp/gpAdminLogs/gpinitsystem_20151127.log for more detail Script Exiting!
In GPDB
create type xyz as (a int, b int);
select row(1,2)::xyz;
create or replace function return_xyz() returns xyz as $$ begin
return row(1,2)::xyz;
end $$ language plpgsql;Above PLPGSQL throws out an error:
ERROR: RETURN must specify a record or row variable in function returning tuple at or near "row"
While in PG, it works well:
test=# create or replace function return_xyz() returns xyz as $$ begin
test$# return row(1,2)::xyz;
test$# end $$ language plpgsql;
CREATE FUNCTIONI'm using the following configure on a CentOS 7 machine:
./configure --with-perl --with-python --enable-dtrace --enable-snmp --with-openssl --with-ldap --with-krb5 --prefix=/usr/local/gpdb
The configure runs without error or warning. When I attempt to run a make, it cranks away for a while then eventually errors out:
"/usr/share/perl5/ExtUtils/xsubpp": No file or directory
Running locate on my machine I find xsubpp in a different location
/usr/share/perl5/vendor_perl/ExtUtils/xsubpp
Setting up a symlink fixes the make error.
ln -s /usr/share/perl5/vendor_perl/ExtUtils/xsubpp /usr/share/perl5/ExtUtils/xsubpp
When I use gpinitsystem with mirror config, there is only 1 mirror segment start successfully out of 9.
When I use gpinitsystem without mirror config, then use "gpaddmirrors -i ~/mirror_config", and the result is the same.
the mirror_config:
filespaceOrder=
mirror0=0:iad-gp-03:41000:42000:43000:/mnt/data2/mirror/gpseg0
mirror1=1:iad-gp-04:41000:42000:43000:/mnt/data2/mirror/gpseg1
mirror2=2:iad-gp-05:41000:42000:43000:/mnt/data2/mirror/gpseg2
mirror3=3:iad-gp-06:41000:42000:43000:/mnt/data2/mirror/gpseg3
mirror4=4:iad-gp-07:41000:42000:43000:/mnt/data2/mirror/gpseg4
mirror5=5:iad-gp-08:41000:42000:43000:/mnt/data2/mirror/gpseg5
mirror6=6:iad-gp-09:41000:42000:43000:/mnt/data2/mirror/gpseg6
mirror7=7:iad-gp-10:41000:42000:43000:/mnt/data2/mirror/gpseg7
mirror8=8:iad-gp-02:41000:42000:43000:/mnt/data2/mirror/gpseg8
The code I used is commit(ee17fd1).
Add two options to gptransfer:
--where='' To allow filtering on the select from the table
--append Used in conjunction with the prior option to enable data migration that is not a full table copy, this will remove the drop or truncate requirement.
The default installation directory (configure prefix) is currently set to the that used by PostgreSQL (/usr/local/pgsql). Although related, PostgreSQL and Greenplum Database are different products, I suggest changing the Greenplum Database default installation directory to /usr/local/gpsql.
There is also a related issue with the default directory (/usr/local/pgsql) as this will change the default installation directory for gp_toolkit.sql. Consequently, it will not be loaded by "gpinitsystem" (function LOAD_GP_TOOLKIT). The src/Makefile.globals.in dictates an alternate installation directory "/share" instead of "/share/postgresql" if the installation directory is pgsql or postgresql is used.
This issue will be more prevalent in the OSS context as the default prefix will cause the loading of gp_toolkit.sql to fail.
WORKAROUND: Use an installation directory other then one containing the strings "pgsql" or "postgres" by specifying the configure "--prefix" option.
this version
commit 7065077b4fef5acf41e67b5240771db021d52156
Author: Shang Shujie <[email protected]>
Date: Wed Dec 16 14:53:34 2015 +0800
fix bug: parquet can't write null out, with test case
when i use gpinitsystem init greenplum database.
ERROR.
[postgres@digoal ~]$ grep "^[a-Z]" gpinitsystem_config
ARRAY_NAME="EMC Greenplum DW"
SEG_PREFIX=gpseg
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/data01/gpdata /data01/gpdata)
MASTER_HOSTNAME=digoal.com
MASTER_DIRECTORY=/data01/gpdata
MASTER_PORT=1999
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=16
ENCODING=UTF-8
MACHINE_LIST_FILE=/home/postgres/host
[postgres@digoal ~]$ cat host
digoal.com
[postgres@digoal ~]$ ssh digoal.com date
Mon Dec 21 22:27:05 CST 2015
[postgres@digoal ~]$ gpinitsystem -c ./gpinitsystem_config --locale=C --max_connections=32 --shared_buffers=256MB --su_password=digoal -B 1
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Checking new segment hosts, Completed
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Greenplum Database Creation Parameters
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:---------------------------------------
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master Configuration
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:---------------------------------------
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master instance name = EMC Greenplum DW
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master hostname = digoal.com
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master port = 1999
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master instance dir = /data01/gpdata/gpseg-1
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master LOCALE = C
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Greenplum segment prefix = gpseg
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master Database =
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master connections = 32
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master buffers = 256MB
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Segment connections = 96
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Segment buffers = 256MB
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Checkpoint segments = 16
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Encoding = UTF-8
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Postgres param file = Off
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Initdb to be used = /opt/gpdb/bin/initdb
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-GP_LIBRARY_PATH is = /opt/gpdb/lib
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Ulimit check = Passed
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Array host connect type = Single hostname per node
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master IP address [1] = ::1
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master IP address [2] = 192.168.0.157
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Master IP address [3] = fe80::e2ca:94ff:fed5:c894
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Standby Master = Not Configured
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Primary segment # = 2
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Total Database segments = 2
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Trusted shell = ssh
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Number segment hosts = 1
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Mirroring config = OFF
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:----------------------------------------
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-Greenplum Primary Segment Configuration
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:----------------------------------------
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-digoal.com /data01/gpdata/gpseg0 40000 2 0
20151221:22:22:01:015135 gpinitsystem:digoal:postgres-[INFO]:-digoal.com /data01/gpdata/gpseg1 40001 3 1
Continue with Greenplum creation Yy/Nn>
y
20151221:22:22:02:015135 gpinitsystem:digoal:postgres-[INFO]:-Building the Master instance database, please wait...
20151221:22:22:17:015135 gpinitsystem:digoal:postgres-[INFO]:-Starting the Master in admin mode
20151221:22:22:31:gpinitsystem:digoal:postgres-[FATAL]:-Failed to complete obtain psql count Master gp_segment_configuration Script Exiting!
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[WARN]:-Script has left Greenplum Database in an incomplete state
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[WARN]:-Run command /bin/bash /home/postgres/gpAdminLogs/backout_gpinitsystem_postgres_20151221_222157 to remove these changes
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function BACKOUT_COMMAND
LOG file:
20151221:22:22:18:015135 gpinitsystem:digoal:postgres-[INFO]:-Completed starting the Master in admin mode
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function PING_HOST
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-digoal.com contact established
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function PING_HOST
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function UPDATE_GPCONFIG
tail: gpinitsystem_20151221.log: file truncated
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function ERROR_CHK
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Successfully completed obtain psql count Master gp_segment_configuration
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function ERROR_CHK
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Adding -1 on digoal.com /data01/gpdata/gpseg-1 to system configuration table
INSERT 0 1
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function ERROR_CHK
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Successfully completed add -1 on digoal.com to Master gp_segment_configuration
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function ERROR_CHK
20151221:22:22:25:015135 gpinitsystem:digoal:postgres-[INFO]:-Adding -1 on digoal.com /data01/gpdata/gpseg-1 to Master gp_filespace_entry
INSERT 0 1
20151221:22:22:26:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function ERROR_CHK
20151221:22:22:26:015135 gpinitsystem:digoal:postgres-[INFO]:-Successfully completed add -1 on digoal.com /data01/gpdata/gpseg-1 to Master pg_filespace_entry
20151221:22:22:26:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function ERROR_CHK
20151221:22:22:26:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function UPDATE_GPCONFIG
20151221:22:22:26:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function LOAD_QE_SYSTEM_DATA
20151221:22:22:26:015135 gpinitsystem:digoal:postgres-[INFO]:-Adding segment digoal.com to Master system tables
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function PING_HOST
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-digoal.com contact established
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function PING_HOST
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function UPDATE_GPCONFIG
tail: gpinitsystem_20151221.log: file truncated
psql: FATAL: semctl(17530881, 14, SETVAL, 0) failed: Invalid argument (pg_sema.c:151)
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function ERROR_CHK
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function ERROR_CHK
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function ERROR_EXIT
20151221:22:22:31:gpinitsystem:digoal:postgres-[FATAL]:-Failed to complete obtain psql count Master gp_segment_configuration Script Exiting!
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[WARN]:-Script has left Greenplum Database in an incomplete state
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[WARN]:-Run command /bin/bash /home/postgres/gpAdminLogs/backout_gpinitsystem_postgres_20151221_222157 to remove these changes
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-Start Function BACKOUT_COMMAND
20151221:22:22:31:015135 gpinitsystem:digoal:postgres-[INFO]:-End Function BACKOUT_COMMAND
code in bin/gpinitsystem
U_DB=$DEFAULTDB
CHK_COUNT=`env PGOPTIONS="-c gp_session_role=utility" $PSQL -p $MASTER_PORT -d "$U_DB" -A -t -c "SELECT count(*) FROM $GP_CONFIG_TBL WHERE content=${U_CONTENT} AND preferred_role='${U_ROLE}';" 2>$LOG_FILE` >> $LOG_FILE 2>&1
ERROR_CHK $? "obtain psql count Master $GP_CONFIG_TBL" 2
if [ $CHK_COUNT -eq 0 ]; then
LOG_MSG "[INFO]:-Adding $U_CONTENT on $U_HOSTNAME $U_DIR to system configuration table"
env PGOPTIONS="-c gp_session_role=utility" $PSQL -p $MASTER_PORT -d "$U_DB" -c "INSERT INTO $GP_CONFIG_TBL (dbid, content, role, preferred_role, mode, status, hostname, address, port, replication_port) VALUES (${U_DBID}, ${U_CONTENT}, '${U_ROLE}', '${U_ROLE}', 's', 'u', '${U_HOSTNAME}', '${U_ADDRESS}', ${U_PORT}, ${U_REPLICATION_PORT});" >> $LOG_FILE 2>&1
ERROR_CHK $? "add $U_CONTENT on $U_HOSTNAME to Master gp_segment_configuration" 2
OS X 10.10.5
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.5.0
./configure
make
results in
strlcpy.c:45:1: error: expected parameter declarator
strlcpy(char *dst, const char *src, size_t siz)
^
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
^
/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
^
strlcpy.c:45:1: error: expected ')'
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
^
/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
^
strlcpy.c:45:1: note: to match this '('
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
/usr/include/secure/_common.h:39:53: note: expanded from macro '__darwin_obsz'
^
strlcpy.c:45:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
strlcpy(char *dst, const char *src, size_t siz)
^
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
/usr/include/secure/_common.h:39:31: note: expanded from macro '__darwin_obsz'
^
strlcpy.c:45:1: error: conflicting types for '__builtin___strlcpy_chk'
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
strlcpy.c:45:1: note: '__builtin___strlcpy_chk' is a builtin with type 'unsigned long (char *, const char *,
unsigned long, unsigned long)'
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
strlcpy.c:45:1: error: definition of builtin function '__builtin___strlcpy_chk'
strlcpy(char *dst, const char *src, size_t siz)
^
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
__builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
^
2 warnings and 4 errors generated.
Upstream Postgres Money is 8 bytes. Greenplum uses 4 bytes. If running TPCH using MONEY data type, you will see negative amount of sum (overflow).
This is same issue as in HAWQ-19. Caleb points out the upstream,
postgres/postgres@74a4019#diff-7ddd78dd264f8cf2a62cf572baf90b84
Hi there,
I have two questions regarding lock related code:
The first question is: in function ProcSleep, after the process has been waken up, either with lock granted or deadlock detected, it would re-acquire the lock table's partition LWLock.
The code episode is here:
/*
* Re-acquire the lock table's partition lock. We have to do this to hold
* off cancel/die interrupts before we can mess with lockAwaited (else we
* might have a missed or duplicated locallock update).
*/
LWLockAcquire(partitionLock, LW_EXCLUSIVE);
/*
* We no longer want LockErrorCleanup to do anything.
*/
lockAwaited = NULL;
/*
* If we got the lock, be sure to remember it in the locallock table.
*/
if (MyProc->waitStatus == STATUS_OK)
GrantAwaitedLock();
/*
* We don't have to do anything else, because the awaker did all the
* necessary update of the lock table and MyProc.
*/
return MyProc->waitStatus;
Questions are:
(1) The comment says that "we might have a missed or duplicated locallock update", in what cases would we hit this if without holding the LWLock?
(2) The comment says "we have to do this to hold off cancel/die interrupts", then: why using LWLockAcquire instead of HOLD_INTERRUPTS directly?
From the handler of SIGINT and SIGTERM, seems nothing serious would be processed here(especially LockWaitCancel), since ImmediateInterruptOK is false, and no CHECK_FOR_INTERRUPS is called before releasing this LWLock. Why should we hold off cancel/die interrupts here?
(3) Before releasing this LWLock, the only share memory access is MyProc->waitStatus; since the process has been granted the lock or removed from the lock's waiting list because of deadlock, is it possible some other processes would access this field? if not, then why we need LWLock here? what does this lock protect?
The second question is:
The PROCLOCKTAG struct is like:
typedef struct PROCLOCKTAG
{
/* NB: we assume this struct contains no padding! */
LOCK *myLock; /* link to per-lockable-object information */
PGPROC *myProc; /* link to PGPROC of owning backend */
} PROCLOCKTAG;
The question is: why we can use pointer here in the struct, since the PROCLOCKTAG would reside in share memory, and the LOCK and PGPROC structs are also in share memory.
Consider this case: one process wants to acquire a lock, so it builds a PROCLOCKTAG and inserts it into LockMethodProcLockHash using the pointer of a LOCK, which is the mapping from a LOCK struct in share memory to the local process; then if another process is iterating the LockMethodProcLockHash, and it has its own mapping from a LOCK struct in share memory to its own address space; because the address space of these two processes are independent, there is chance that their LOCK pointer is exactly same, while the LOCK objects referenced in the share memory are actually different, then this would lead to unexpected results.
There is a comment saying that:
We can use pointers here because the PROCLOCKTAG need only be unique for the lifespan of the PROCLOCK, and it will never outlive the lock or the proc.
Hmm, I don't understand what this comment means actually, and it cannot explain the above case obviously.
Apologize if I missed anything here, and really appreciate if someone can help me on these two questions. Thanks
On RHEL7.1, make installcheck-good failed for below reason:
*** ./expected/percentile.out 2015-11-16 16:24:38.907191380 +0800
--- ./results/percentile.out 2015-11-16 16:24:38.952189622 +0800
***************
*** 964,971 ****
reset gp_idf_deduplicate;
select pg_get_viewdef('percv');
pg_get_viewdef
! -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
! SELECT percentile_cont(float8(0.4)) WITHIN GROUP (ORDER BY float8((perct.a / 10))) AS "percentile_cont", median(float8(perct.a)) AS "median", percentile_disc(float8(0.51)) WITHIN GROUP (ORDER BY float8(perct.a) DESC) AS "percentile_disc" FROM perct GROUP BY perct.b ORDER BY perct.b;
(1 row)
select pg_get_viewdef('percv2');
--- 964,971 ----
reset gp_idf_deduplicate;
select pg_get_viewdef('percv');
pg_get_viewdef
! -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
! SELECT percentile_cont(float8(0.4)) WITHIN GROUP ORDER BY float8((perct.a / 10)) AS "percentile_cont", median(float8(perct.a)) AS "median", percentile_disc(float8(0.51)) WITHIN GROUP ORDER BY float8(perct.a) DESC AS "percentile_disc" FROM perct GROUP BY perct.b ORDER BY perct.b;
(1 row)
select pg_get_viewdef('percv2');
======================================================================
We got an extension library with GIN index, which was however disabled by GP. So I compared the code of GIST index and GIN index, and found out some differences in HA. Then I add some code (mainly adding MIRROREDLOCK_BUFMGR_LOCK before ReadBuffer and MIRROREDLOCK_BUFMGR_UNLOCK in the end )and enabled creation of GIN. After some tests., it turned out to be running well so far. Can anybody inform me of the reason behind disabling GIN and any further detailed information of MPP-9329? Also, would you please share with me a test case of MPP-9329?
With git master:
postgres=# set gp_default_storage_options ='checksum=off';
ERROR: problem in alloc set PortalHeapMemory: detected write past chunk end in block 0x1749b40, chunk 0x174a198 (portalmem.c:461) (aset.c:1912) (seg0 localhost:40000 pid=20938) (cdbdisp.c:1326)
ERROR: problem in alloc set PortalHeapMemory: detected write past chunk end in block 0x20523b0, chunk 0x2052a08 (portalmem.c:461) (aset.c:1912)
Using gpload in a Windows environment with a 64-bit Python interpreter produces the following error:
> gpload.py -f gpload-test.yaml
gpload was unable to import The PyGreSQL Python module (pg.py) - DLL load failed with error code 193
Using Python 2.5.4 32-bit seems to correct the issue.
commit b80a01e
Author: Heikki Linnakangas [email protected]
Date: Thu Nov 5 00:20:11 2015 +0200
Remove useless 'initdb' bugbuster test
./configure --prefix=/opt/gpdb --without-libcurl
make error
gmake[3]: Leaving directory /opt/soft_bak/gpdb/src/backend/cdb' g++ -O3 -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-but-set-variable -Wno-address -L../../src/port -L../../src/port -Wl,--as-needed -Wl,-rpath,'/opt/gpdb/lib',--enable-new-dtags -Wl,-E access/SUBSYS.o bootstrap/SUBSYS.o catalog/SUBSYS.o parser/SUBSYS.o commands/SUBSYS.o executor/SUBSYS.o foreign/SUBSYS.o fts/SUBSYS.o lib/SUBSYS.o libpq/SUBSYS.o gp_libpq_fe/SUBSYS.o main/SUBSYS.o nodes/SUBSYS.o optimizer/SUBSYS.o port/SUBSYS.o postmaster/SUBSYS.o regex/SUBSYS.o replication/SUBSYS.o rewrite/SUBSYS.o storage/SUBSYS.o tcop/SUBSYS.o utils/SUBSYS.o ../../src/timezone/SUBSYS.o cdb/SUBSYS.o ../../src/port/libpgport_srv.a -lbz2 -lrt -lz -lcrypt -ldl -lm -lpthread -o postgres utils/SUBSYS.o:(.data+0xaab0): undefined reference toreadable_external_table_timeout'
collect2: ld returned 1 exit status
gmake[2]: *** [postgres] Error 1
gmake[2]: Leaving directory /opt/soft_bak/gpdb/src/backend' gmake[1]: *** [all] Error 2 gmake[1]: Leaving directory/opt/soft_bak/gpdb/src'
gmake: *** [all] Error 2
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.