Wednesday, October 7, 2015

Tip:: PPAS 9.4 and Global Temporary Table

Customers who moved/migrated their database from Oracle to PPAS frequently ask for Global Temporary Table in PPAS.

Currently, PPAS doesn't support Global Temporary tables. However, there is a way user can achieve this functionality in PPAS.

Before we continue with the implementation, lets first understand characteristics of Global Temporary Table. Following are the important characteristics of Global Temporary Table.
1. Global Temporary Table gives predefined structure for storing data.
2. It's an unlogged table which means any activity on this table will not be logged.
3. The data in a global temporary table are private, such that data inserted by a session can only be accessed by that session.

Based on the above characteristics of Global Temporary Table AKA GTT, we can define similar kind of work by using the following method:
1. Create UNLOGGED TABLE in PPAS, which activity won't be logged.
2. Create Row Level Security in such a way that session should be able to see their information (based on PID).
3. Create a process which can cleanup data from GTT based on pids which are not active in database.

Lets see how we can implement it in Adavanced Server.

1. Create an UNLOGGED table with all columns required and extra column of Pid.

CREATE UNLOGGED TABLE test_global_temporary_table
(id numeric, col text, pid bigint default pg_backend_pid());

2. Create a function to restrict the visibility of data.

CREATE OR REPLACE FUNCTION verify_pid_context (
    p_schema       TEXT,
    p_object       TEXT
   predicate TEXT;
    IF ( current_setting('is_superuser') = 'on')
      predicate = 'true';
      predicate := format('pid = %s',pg_backend_pid());
    END IF;
    RETURN predicate;

3. Apply the security policy based on above function.

  v_object_schema VARCHAR2(30)   := 'public';
  v_object_name VARCHAR2(30)     := 'test_global_temporary_table';
  v_policy_name VARCHAR2(30)     := 'secure_by_pid';
  v_function_schema VARCHAR2(30) := 'public';
  v_policy_function VARCHAR2(30) := 'verify_pid_context';
  v_statement_types VARCHAR2(30) := 'INSERT,UPDATE,DELETE,SELECT';
  v_update_check BOOLEAN         := TRUE;
  v_enable BOOLEAN               := TRUE;
  DBMS_RLS.ADD_POLICY( v_object_schema,

4. Create UPDATABLE view which can hide pid column. All sessions will be using this view as GTT.
   CREATE OR REPLACE VIEW test_global_temporary AS SELECT id, col FROM test_global_temporary_table;

5. Create a backend job, which can cleanup Table based on stale/old sessions.
For job, user/developer can do following:
   a. use superuser and execute DELETE command on table:
       DELETE FROM test_global_temporary WHERE pid NOT in (SELECT pid FROM pg_stat_activity);
   b. To Schedule above DELETE command, user can use one of the following:
        i. Crontab
       ii. Or PPAS DBMS_SCHEDULE  Package.

6. GRANT ALL privileges to database user who can access Global Temporary Table.

    GRANT ALL on test_global_temporary TO testdbuser;
    GRANT ALL on test_global_temporary_table To testdbuser;

Now, lets try above implementation of Global Temporary Table.

Open two sessions as a normal user (testdbuser) as given below:

[vibhorkumar@localhost ~]$ psql -U testdbuser edb
psql.bin (
Type "help" for help.

edb=> select pg_backend_pid();
(1 row)

edb=> select pg_backend_pid();
(1 row)

Now from both session insert some records:
From first session:

edb=> INSERT INTO test_global_temporary VALUES(1,'FROM pid 32722');
edb=> INSERT INTO test_global_temporary VALUES(2,'FROM pid 32722');
edb=> INSERT INTO test_global_temporary VALUES(3,'FROM pid 32722');

From Second session:

edb=> INSERT INTO test_global_temporary VALUES(1,'FROM pid 32729');
edb=> INSERT INTO test_global_temporary VALUES(2,'FROM pid 32729');
edb=> INSERT INTO test_global_temporary VALUES(3,'FROM pid 32729');

From First Session:
edb=> SELECT * FROM test_global_temporary;
 id |      col       
  1 | FROM pid 32722
  2 | FROM pid 32722
  3 | FROM pid 32722
(3 rows)

From Second Session:
edb=> SELECT * FROm test_global_temporary;
 id |      col       
  1 | FROM pid 32729
  2 | FROM pid 32729
  3 | FROM pid 32729
(3 rows)

which shows that unlogged table with right RLS policy and backend job, can be a potential solution for Global Temporary Tables.

Thursday, August 6, 2015

Postgres And Transparent Data Encryption (TDE)

Security has always been a great concern of Enterprises. Especially, if you have crucial information stored in the database, you would always prefer to have high security around it. Over the years, technologies have evolved and provided better solutions around it.

If you have very sensitive information, people try to keep this information encrypted so, that in case, somebody gets access of the system, then they cannot view this information, if they are not authorized.

For managing sensitive information, Enterprises use multiple methods:
1. Encrypting specific information.

If you are PPAS users, you would like to use DBMS_CRYPTO package which provides a way of encrypting sensitive information in databases.

For more information, please refer following link:

For PostgreSQL, users can use pgcrypto module.

2. Transparent Data Encryption (TDE) is another method employed by both Microsoft and Oracle to encrypt database files. TDE offers encryption at file level. This method solves the problem of protecting data at rest i.e. encrypting databases both on the hard drive and consequently on backup media. Enterprises typically employ TDE to solve compliance issues such as PCI DSS.

Postgres Plus, currently doesn't have inbuilt TDE, however, if Enterprises looking for encryption at the database file level, they can use one of the following methods for protecting data at rest:

1.  Full Disk Encryption:
Full disk or partition encryption is one of the best ways of protecting your data. This method not only protects each file, however, also protects the temporary storage that may contain parts of these files.  Full disk encryption protects all of your files and then you do not have to worry about selecting what you want to protect and possibly missing a file.

RHEL (Red Hat) supports Linux Unified Key Setup-on-disk-format (or LUKS). LUKS bulk encrypts Hard Drive partition.

For more information on LUKS, please refer following link:

2. File system-level encryption:
   File system-level encryption often called file/directory encryption. In this method individual files or directories are encrypted by the file system itself. 
There is stackable cryptographic file system encryption available which user can utilize in their environment.

File system level Encryption gives following advantages:
1. Flexible file-based key management, so that each file can be and usually is encrypted with a separate encryption key.

2. Individual management of encrypted files e.g. Incremental backups of the individual changed files even in encrypted form, rather than backup of the entire encrypted volume.

3. Access control can be enforced through the use of public-key cryptography, and the fact that cryptographic keys are only held in memory while the file that is decrypted by them is held open.

Stackable cryptographic file system encryption can be use for Postgres for Transparent Data Encryption.

In this blog, I will discuss using mount ecrpytfs as it requires less overhead in setup (LUKS requires a new disk to be configured and formatted before storing data on it. "mount ecrpytfs" works with existing directories and data).

If Enterprises want to give the control to DBAs for TDE, they can use/define few sudo rules for DBAs to execute commands for encryption.

Following is a method, which they can use:

1)    Ask system admin to create sudo rules to allow DBA to execute encryption for data directory for Postgres Plus. One common way to do this is using the “mount ecryptfs” command in Linux operating systems.
2)  If user needs to encrypt the /ppas94/data directory, they can use following command:

sudo mount -t ecryptfs /ppas94/data /ppas94/data
           More information can be found in the documentation from RHEL:

User can also specify encryption key type (passphrase, openssl), cipher (aes, des3_ede...) key byte size, and other options with above commands.

Example is given below:

# mount -t ecryptfs /home /home -o ecryptfs_unlink_sigs \
 ecryptfs_key_bytes=16 ecryptfs_cipher=aes ecryptfs_sig=c7fed37c0a341e19

Centos 7 and RHEL 7, by default doesn’t come with ecrpytfs therefore, user can also use encfs command.

For more information on encfs, please refer following link:

Following are the steps to use encfs to encrypt the data directory.
1.     Create a data directory using following command, as enterprisedb user.

   mkdir /var/lib/ppas/9.4/encrypted_data
   chmod 700 /var/lib/ppas/9.4/encrypted_data

2.     Use following encfs command to encrypt the data directory.

         encfs /var/lib/ppas-9.4/encrypted_data/ /var/lib/ppas-9.4/data

Snapshot of above command is given below:

encfs /var/lib/ppas/9.4/encrypted_data /var/lib/ppas/9.4/data
The directory "/var/lib/ppas/9.4/data" does not exist. Should it be created? (y,n) y
Creating new encrypted volume.
Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.
?> p

Paranoia configuration selected.

Configuration finished.  The filesystem to be created has
the following properties:
Filesystem cipher: "ssl/aes", version 3:0:2
Filename encoding: "nameio/block", version 3:0:1
Key Size: 256 bits
Block Size: 1024 bytes, including 8 byte MAC header
Each file contains 8 byte header with unique IV data.
Filenames encoded using IV chaining mode.
File data IV is chained to filename IV.
File holes passed through to ciphertext.

-------------------------- WARNING --------------------------
The external initialization-vector chaining option has been
enabled.  This option disables the use of hard links on the
filesystem. Without hard links, some programs may not work.
The programs 'mutt' and 'procmail' are known to fail.  For
more information, please see the encfs mailing list.
If you would like to choose another configuration setting,
please press CTRL-C now to abort and start over.

Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism.  However, the password can be changed
later using encfsctl.

New Encfs Password:
Verify Encfs Password:

3.     After encrypting, data directory, users also need to modify the postgresql- service script to include proper command in it for password. For that either, they can use sshpass or they can write their own program which can pass the password for mounting directory.

As you can see, achieving Transparent Data Encryption Postgres is very easy.

Wednesday, December 10, 2014

Dynamic RLS implementation in PPAS 9.3

In the course of my work at EnterpriseDB, migrating Oracle databases to EnterpriseDB's Postgres Plus Advanced Server is a common task. However, now and then we encounter unique situations. While working on a migration project recently, we encountered a new use case for RLS (Row level Security).

The customer had a centralized database where it stored a huge number of transactions. These transactions are performed by different business units located in different parts of the world. There are certain types of transactions that should not be visible even if they are being queried by the same company. That is where RLS comes in. With RSL, specific transactions, or kinds of transactions, that can remain visible are mapped back to an attribute in the table.

The customer needed the application to authenticate users and set the context for which records in the database become visible for a specific session. In its deployment of Oracle, the customer had used the functions/procedure in the Oracle package DBMS_SESSION. In the application, the customer used DBMS_SESSION.SET_CONTEXT to set the context. And for the Row Level Security, the customer was using the DBMS_SESSION.SYS_CONTEXT to implement security around the transactions. 

Postgres Plus Advanced Server has a DBMS_SESSION package that is compatible with Oracle. However, it does not currently offer users the capability of setting the user defined context and implementing RLS based on those context. Given others may experience similar situations , as our customer, I wanted to provide the procedures and functions that users could deploy.

SET_CONTEXT procedure.

The definition of this procedure is given below:
CREATE OR REPLACE PROCEDURE set_context(namespace TEXT, 
                                        attribute TEXT, 
                                        val       TEXT)
    EXECUTE IMMEDIATE format('SET %s.%s TO %s',namespace, attribute,val);

Using this procedure, users can set their own context at session level.

The following is a function to help view the context in session, which is set using the above procedure.
                                        parameter TEXT,
                                        len       BIGINT DEFAULT 8)
    return_val TEXT;
    EXECUTE IMMEDIATE format('SHOW %s.%s',namespace,parameter) INTO return_val;
    RETURN substr(return_val,1,len);
The following is an example of how we can implement row level security based on the above procedure and functions:

1. Create a table which will have attribute context_check to map the context set by procedure:
CREATE TABLE test_rls(id numeric, col text, context_check text);
INSERT INTO test_rls SELECT id, 'First_check','aaa' FROM generate_series(1,10) foo(id);
INSERT INTO test_rls SELECT id, 'First_check','bbb' FROM generate_series(1,10) foo(id);
INSERT INTO test_rls SELECT id, 'First_check','ddd' FROM generate_series(1,10) foo(id);
2. Now create a function to check the application context. Below is one function:
CREATE OR REPLACE FUNCTION verify_user_context (
    p_schema       TEXT,
    p_object       TEXT
   predicate TEXT;
    predicate := format('context_check = public.usys_context(''%s''::text,''%s''::text, 8)','CONTEXT','APP_PREDICATE');
    RETURN predicate;
3. Now Apply Security Policy using Policy Functions shown below:
    v_object_schema         VARCHAR2(30) := 'public';
    v_object_name           VARCHAR2(30) := 'test_rls';
    v_policy_name           VARCHAR2(30) := 'secure_data';
    v_function_schema       VARCHAR2(30) := 'public';
    v_policy_function       VARCHAR2(30) := 'verify_user_context';
    v_statement_types       VARCHAR2(30) := 'INSERT,UPDATE,DELETE,SELECT';
    v_update_check          BOOLEAN      := TRUE;
    v_enable                BOOLEAN      := TRUE;
Now we are set to test this implementation. Connect to one session and try the following:

1. Set the context using procedure SET_CONTEXT as given below:

EDB-SPL Procedure successfully completed
2. Verify in the same session to determine if we have set the Context properly:
(1 row)
3. Since in session, we have Context set as ddd, there in this session, we should be able to see rows respective to set contexts:
beta=# SELECT * FROM test_rls ;
 id |     col     | context_check 
  1 | First_check | ddd
  2 | First_check | ddd
  3 | First_check | ddd
  4 | First_check | ddd
  5 | First_check | ddd
  6 | First_check | ddd
  7 | First_check | ddd
  8 | First_check | ddd
  9 | First_check | ddd
 10 | First_check | ddd
(10 rows)
As you can see, the DBMS_RLS package in Postgres Plus Advanced Service can help in implementing Row Level Security based on Application Context.

Monday, December 1, 2014

Compiling PLV8 with Postgres Plus Advanced Server

PLV8 is a programming language that lets users write stored procedures and triggers in JavaScript and store them in their Postgres database. This allows application programmers to write a lot of their server-side programming in the same language they use to build their web client applications.  Fewer languages to learn usually means fewer mistakes and faster time to completion.  The extensive language support is one of many reasons why Postgres’ use across the world is increasing lately.  The recent addition of document data support with JSON and JSONB data types in PostgreSQL, and in Postgres Plus Advanced Server from EnterpriseDB, is the main reason for the increasing interest in the PL/V8 language extension.

Below are the steps you need to compile PLV8 with Postgres Plus Advanced Server 9.3/9.4.

To get started, here are the prerequisites:
1. A supported version of PostgreSQL or Postgres Plus Advanced Server, such as versions 9.1 and higher.
2. V8 version 3.14.5
3. g++ version 4.5.1

If you want to know more about V8, you can visit the following wiki page:

It’s important to note that when compiling PLV8 with Postgres Plus Advanced Server 9.3 or the upcoming 9.4, you will get the following two types of error messages:

The first error:
[root@localhost plv8js]# make

sed -e 's/^#undef PLV8_VERSION/#define PLV8_VERSION "1.5.0-dev1"/' > plv8_config.h

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8.o In function ‘void _PG_init()’: error: invalid conversion from ‘void (*)(XactEvent, void*)’ to ‘void (*)(XactEvent, void*, bool)’ error:   initializing argument 1 of ‘void RegisterXactCallback(void (*)(XactEvent, void*, bool), void*)’

make: *** [plv8.o] Error 1

The above error message is a result of a different signature of typedef void (*XactCallback) in the Advanced Server transaction system.

To fix the above issue, the user can replace the following in
static void plv8_xact_cb(XactEvent event, void *arg);

static void plv8_xact_cb(XactEvent event, void *arg, bool spl_context);

The second error:
After making the above changes, you may get the following error after trying to compile the source code using the “make” command:
[root@localhost plv8js]# make

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8.o warning: ‘void plv8_xact_cb(XactEvent, void*, bool)’ used but never defined warning: ‘void plv8_xact_cb(XactEvent, void*)’ defined but not used

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8_type.o

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8_func.o

In file included from plv8_param.h:11,


/usr/ppas-9.4/include/server/nodes/params.h:77: error: expected ‘,’ or ‘...’ before ‘typeid’

make: *** [plv8_func.o] Error 1

The above is mainly due to the use of typeid in params.h; typeid is the reserved keyword of C++ compiler.

To fix this issue, make the following changes in plv8.h
extern "C" {

#include "postgres.h"

#include "access/htup.h"

#include "fmgr.h"

#include "mb/pg_wchar.h"

#include "utils/tuplestore.h"

#include "windowapi.h"


#define typeid __typeid

extern "C" {

#include "postgres.h"

#include "access/htup.h"

#include "fmgr.h"

#include "mb/pg_wchar.h"

#include "utils/tuplestore.h"

#include "windowapi.h"


#undef typeid

In plv8_param.h, change the following:
extern "C" {

#include "postgres.h"


 * Variable SPI parameter is since 9.0.  Avoid include files in prior versions,

 * as they contain C++ keywords.


#include "nodes/params.h"

#if PG_VERSION_NUM >= 90000

#include "parser/parse_node.h"

#endif // PG_VERSION_NUM >= 90000

} // extern "C"

#define typeid __typeid

extern "C" {

#include "postgres.h"


 * Variable SPI parameter is since 9.0.  Avoid including files in prior versions,

 * as they contain C++ keywords.


#include "nodes/params.h"

#if PG_VERSION_NUM >= 90000

#include "parser/parse_node.h"

#endif // PG_VERSION_NUM >= 90000

} // extern "C"

#undef typeid

In, replace following:
extern "C" {

#include "catalog/pg_type.h"

#include "utils/builtins.h"

#include "utils/lsyscache.h"

} // extern "C"

#define typeid __typeid

extern "C" {

#include "catalog/pg_type.h"

#include "utils/builtins.h"

#include "utils/lsyscache.h"

} // extern "C"

#undef typeid

After making the above changes, you will be able to compile PLV8 with Advanced Server as shown below:
[root@localhost plv8js]# make

sed -e 's/^#undef PLV8_VERSION/#define PLV8_VERSION "1.5.0-dev1"/' > plv8_config.h

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8.o

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8_type.o

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8_func.o

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o plv8_param.o

echo "extern const unsigned char coffee_script_binary_data[] = {" >

(od -txC -v coffee-script.js | \

 sed -e "s/^[0-9]*//" -e s"/ \([0-9a-f][0-9a-f]\)/0x\1,/g" -e"\$d" ) >>

echo "0x00};" >>

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o coffee-script.o

echo "extern const unsigned char livescript_binary_data[] = {" >

(od -txC -v livescript.js | \

 sed -e "s/^[0-9]*//" -e s"/ \([0-9a-f][0-9a-f]\)/0x\1,/g" -e"\$d" ) >>

echo "0x00};" >>

g++ -Wall -O2 -DV8_USE_UNSAFE_HANDLES  -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -fPIC -c -o livescript.o

g++ -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fpic -shared -o plv8.o plv8_type.o plv8_func.o plv8_param.o coffee-script.o livescript.o -L/usr/ppas-9.4/lib -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/usr/ppas-9.4/lib',--enable-new-dtags  -lv8

sed -e 's/@PLV8_VERSION@/1.5.0-dev1/g' plv8.control.common | g++ -E -P -DLANG_plv8 - > plv8.control

sed -e 's/@LANG_NAME@/plv8/g' plv8.sql.common | g++ -E -P -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -DLANG_plv8 - > plv8--1.5.0-dev1.sql

sed -e 's/@PLV8_VERSION@/1.5.0-dev1/g' plv8.control.common | g++ -E -P -DLANG_plcoffee - > plcoffee.control

sed -e 's/@LANG_NAME@/plcoffee/g' plv8.sql.common | g++ -E -P -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -DLANG_plcoffee - > plcoffee--1.5.0-dev1.sql

sed -e 's/@PLV8_VERSION@/1.5.0-dev1/g' plv8.control.common | g++ -E -P -DLANG_plls - > plls.control

sed -e 's/@LANG_NAME@/plls/g' plv8.sql.common | g++ -E -P -I. -I./ -I/usr/ppas-9.4/include/server -I/usr/ppas-9.4/include/internal -I/usr/include/et -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include -DLANG_plls - > plls--1.5.0-dev1.sql

/bin/mkdir -p '/usr/ppas-9.4/lib'

/bin/mkdir -p '/usr/ppas-9.4/share/extension'

/bin/mkdir -p '/usr/ppas-9.4/share/extension'

/usr/bin/install -c -m 755 '/usr/ppas-9.4/lib/'

/usr/bin/install -c -m 644 plv8.control '/usr/ppas-9.4/share/extension/'

/usr/bin/install -c -m 644 plv8.control plv8--1.5.0-dev1.sql plcoffee.control plcoffee--1.5.0-dev1.sql plls.control plls--1.5.0-dev1.sql '/usr/ppas-9.4/share/extension/'

After compiling PLV8, you now can install the PLV8 language in Advanced Server using the following command:




To test your installed PLV8, here is some sample code:
beta=# DO $$ PLV8.elog(NOTICE, 'this', 'is', 'inline', 'code') $$ LANGUAGE PLV8;

NOTICE:  this is inline code


beta=# CREATE TYPE rec AS (i integer, t text);


beta=# CREATE FUNCTION set_of_records() RETURNS SETOF rec AS

beta-# $$

beta$#     // PLV8.return_next() stores records in an internal tuplestore,

beta$#     // and return all of them at the end of function.

beta$#     PLV8.return_next( { "i": 1, "t": "a" } );

beta$#     PLV8.return_next( { "i": 2, "t": "b" } );


beta$#     // You can also return records with an array of JSON.

beta$#     return [ { "i": 3, "t": "c" }, { "i": 4, "t": "d" } ];

beta$# $$



beta=# SELECT * FROM set_of_records();

 i | t


 1 | a

 2 | b

 3 | c

 4 | d

(4 rows)

In case you need a patched version of PLV8, use the following git repository: PLV8_ppas

To use this, execute the following command:
 git clone PLV8_ppas

cd PLV8_ppas


make install

To test the compiled PLV8, you can use the following command:
[root@localhost plv8js]# make installcheck

/usr/ppas-9.4/lib/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=./ --psqldir='/usr/ppas-9.4/bin'    --dbname=contrib_regression init-extension plv8 inline json startup_pre startup varparam json_conv window dialect

(using postmaster on Unix socket, port 5444)

============== dropping database "contrib_regression" ==============


============== creating database "contrib_regression" ==============



============== running regression test queries        ==============

test init-extension           ... ok

test plv8                     ... ok

test inline                   ... ok

test json                     ... ok

test startup_pre              ... ok

test startup                  ... ok

test varparam                 ... ok

test json_conv                ... ok

test window                   ... ok

test dialect                  ... ok


 All 10 tests passed.


Tuesday, November 18, 2014

Meet BART – A New Tool for Backup And Recovery Management

EnterpriseDB recently launched a new tool for backup and recovery – named simply EDB Backup and Recovery Tool, or BART. This tool makes the DBA’s life easier by simplifying the tasks for managing their Postgres physical backup and recovery tasks, whether they are PostgreSQL or Postgres Plus Advanced Server deployments.
BART has the following advantages over custom scripts for managing backups:
1. It’s stable and it uses the tool pg_basebackup to take a physical backup. This tool has been well defined and is well-supported by the PostgreSQL community.
2. It catalogs all of the backups users are taking, which is important in terms of:
    i. Listing the type of backups used
   ii. Listing the status of those backups with server information.
3. BART also provides functionality to restore backups, with all required archived WAL files. So automation around this tool will make DBAs’ lives easier for restore and recovery.
4. BART provides an option to validate your backup by using checksum. This is useful for confirming you took a valid backup and it is not corrupted at disk level.
5. BART provides an option to define your retention policy around the backups you are keeping.
Given all of the above advantages, I decided to give this new tool a try and share some tips. To get started, you need the following prerequisites:
1. BART currently requires a Linux 64 bit platform, CentOS 6.x or RHEL 6.x
2. Need to have password-less, direct SSH access to the target machine where you want to restore backups as well as the database servers you want backed up
3. Install the Postgres Plus Advanced Server or PostgreSQL binaries for pg_basebackup
Yum or rpm
To install this tool, you have two options that I will explore below:
1. Yum command
2. Rpm command.
Using the yum command:
To perform a yum command installation, BART users can ask EDB for credentials to the EnterpriseDB yum repository and configure the their local yum repository as follows:
echo "[tools]
name=EnterpriseDB Tools
gpgcheck=0" >/etc/yum.repos.d/edbtools.repo
After creating the yum repo, the user can execute the following command to install BART:
yum install edb-bart
If the user doesn't want to install the EDB Backup and Recovery Tool using the yum command, then the user can download a free standing rpm using the link below from EDB’s website:
and then enter the rpm install command as follows:
rpm -ivh edb-bart-1.0.1-1.rhel6.x86_64.rpm
After installing BART using the above commands, the user can see the binaries in the directory:/usr/edb-bart-1.0/bin and a sample BART configuration file in /usr/edb-bart-1.0/etc
That’s a very easy installation.
For more information on configuring BART Host and Database Host, the following are some documents that will help:
1. pg_basebackup configuration for PostgreSQL:
2. For direct password less ssh configuration user can refer following link
After the installation of the BART binaries, the user also has to create a BART configuration file.
The following is a sample configuration file for BART:
bart-host= enterprisedb@
backup_path = /opt/backup
pg_basebackup_path = /usr/ppas-9.4/bin/pg_basebackup
logfile = /tmp/bart.log

host =
port = 5432
user = postgres
description = Postgres server

host =
port = 5444
user = enterprisedb
description = PPAS 94 server
Global Configuration Settings
Content under the [BART] tag are called global configuration settings. Under this tag are the following:
1. bart-host: the IP address of the host on which BART is installed. The value for this parameter must be specified in the form: bart_user@bart_host_address, where bart_user is the operating system user account on the BART host that is used to run BART and owns the BART backup catalog directory. bart_host_address is the IP address of the BART host.
2. backup_path: specifies the file system parent directory where all BART database server base backups and archived WAL files are stored. This parameter is required.
3. pg_basebackup_path: specifies the path to the pg_basebackup program of the Postgres database server installed on the BART host.
4. log file: specifies the path to the BART log file. This parameter is optional. If no path to a log file is specified after logfile =, or if the parameter is commented out, BART does not create a log file.
The remaining part of configuration file is self-explanatory. The TAG: [PG]/[PPAS94] part is content for servers which the user wants to back up.
Pg_basebackup Settings
After performing the above configuration on the Backup Server, the user has to do set following settings on the servers that they want to back up. Below are the settings for enabling backup using pg_basebackup.
The user has to set a few parameters in PostgreSQL postgresql.conf file, which he wants to backup:
1. wal_level parameter to archive or hot_standby.
2. archive_mode=on
3. archive_command setting.
4. max_wal_senders to 1 or more than one, since pg_basebackup uses the replication protocol to copy data directory.
For more information on each setting please refer to the following:
1. wal_level:
2. archive_mode and archive_command:
3. max_wal_senders:
With the above settings, the user then needs to update the pg_hba.conf file for the replication connection.
Note: The above settings are for pg_basebackup to take backups using replication protocols. In case users need more information about pg_basebackup and settings, please use the above mentioned link
How BART Works
Now, since we have configured both servers, let’s have a look how BART works.
The following command executes a backup:
 bart -c bart.cfg BACKUP -s ppas94
And below is the output:
[bart@localhost ~]$ bart -c bart.cfg BACKUP -s ppas94

INFO:  creating backup for server 'ppas94'
INFO:  backup identifier: '1413852137762'
6394456/6394456 kB (100%), 1/1 tablespace

INFO:  backup checksum: 7f49ea9653511308710c174f22ec765d
INFO:  backup completed successfully
[bart@localhost ~]$ 
That was an easy way to take a backup. The DBA can also create a job to execute the above command to take backups.
If the user wants to list the backup using BART, the user can use the option SHOW-BACKUPS:
[bart@localhost ~]$ bart -c bart.cfg SHOW-BACKUPS -s ppas94
 Server Name   Backup ID       Backup Time           Backup Size  
 ppas94        1413852137762   2014-10-20 17:43:41   6244.59 MB   
This is useful for knowing what backups a user has available for recovery. The above command gives important information:
1. Backup ID: It’s a unique ID for the physical backup
2. Backup Time: Time when backup was taken
3. Backup Size: Size of backup
This information is useful when a user wants to plan for recovery using backup. This way, the user can also plan for disk size.
Sometimes a user wants to verify their backup state. VERIFY-CHKSUM option is useful in this case:
[bart@localhost ~]$ bart -c bart.cfg VERIFY-CHKSUM -s ppas94 -i 1413852137762
 Server Name   Backup ID       Verify  
 ppas94        1413852137762   OK
I have to say, after putting EDB BART through its paces, I think DBAs will enjoy having such a great tool for making Backup Management easy.
In my next post, I will blog about the Recovery process.