Misc in PostgresSQL

Sequence

A serial-type column in a table, e.g.

create table os (
  id                            serial,
  ...

it is actually defined as

postgres=> \d os
                         Table "public.os"
 Column | Type | Modifiers
---------+-------------+-------------------------------------------------
 id      | integer     | not null default nextval('os_id_seq'::regclass)
 ...

That is, a sequence named <table>_<column>_seq is created implicitly.

Sequence is guaranteed to advance every time nextval() is called, even the update is canceled by rollback. So “holes” might be left. It won’t skip existing values either (for example, suppose id is a unique key, and a value is inserted before the sequence reaches that value, a unique violation will occur when the sequence advanced to the value). The best practice is never set serial directly. In some special cases, such as restore/copy data, always call setval() after the data are copied.

Here are sequence functions

Function Return Type Description
currval(regclass) bigint Return value most recently obtained with nextval for specified sequence
lastval() bigint Return value most recently obtained with nextval for any sequence
nextval(regclass) bigint Advance sequence and return new value
setval(regclass, bigint) bigint Set sequence’s current value
setval(regclass, bigint, boolean) bigint Set sequence’s current value and is_called flag

See details at PostgreSQL Document 9.16. Sequence Manipulation Functions

 

Password File

In a trusted environment, accessing database with password can be eliminated with saved password file. On Posix systems, it is $HOME/.pgpass, on Windows, it is %APPDATA%\postgresql\pgpass.conf.

The file contains lines of following format:

hostname:port:database:username:password

except password, the first 4 fields can be *, which matches anything.

The permission of .pgpass should be stricter than 0600, otherwise the file will be ignored. However the permission of pgpass.conf is not checked on Windows.

Reference: PostgreSQL Manual: 31.15. The Password File

Errors and Messages

RAISE statement is used to report message and raise error in SQL/PLSQL

RAISE [ level ] 'format' [, expression [, ... ]] [ USING option = expression [, ... ] ];
RAISE [ level ] condition_name [ USING option = expression [, ... ] ];
RAISE [ level ] SQLSTATE 'sqlstate' [ USING option = expression [, ... ] ];
RAISE [ level ] USING option = expression [, ... ];
RAISE ;

level

  • DEBUG
  • LOG
  • NOTICE
  • WARNING
  • EXCEPTION (default, which aborts the current transaction))

‘format’

Must be a string literal (not an expression), followed by optional arguments to be inserted into the message, where % is replaced by the string representation of arguments.

USING option

  • MESSAGE
  • DETAIL
  • HINT
  • ERRCODE
  • COLUMN, CONSTRAINT, DATATYPE, TABLE, SCHEMA

Common Error Codes

  • 02000 – no_data_found
  • 22000 – data_exception
  • 23000 – integrity_constraint_violation
  • 40000 – transaction_rollback
  • 42000 – syntax_error_or_access_rule_violation

Note:

  • although the name of 02000 is “no_data” in all versions of manual, the actual is no_data_found

Please check the full error code list

Detect No Data in PL/SQL

Example 1 (not reliable)

declare
    pid int;
begin
    select id into pid
    from table_foo
    where name='foo';
    -- not reliable, e.g. the value of id is null
    if pid is null then
        raise no_data_found
        using detail=format('name %s is not found', 'foo');
    end if;
end;

Example 2

begin
    update build
    set state='start'
    where name='foo';
    if not found then
        raise no_data_found
        using detail=format('name %s is not found', 'foo');
    end if;
end;

Trapping Errors in PL/SQL

[ <> ]
[ DECLARE
      DECLARATIONS ]
BEGIN
    statements
EXCEPTION
    WHEN condition [ OR condition ... ] THEN
        handler_statements
   [WHEN condition [ OR condition ...] THEN
        handler_statements
    ...]
END;

Here is an example that traps error for SELECT … INTO STRICT var. INTO STRICT means there is EXACTLY one row retrieved

BEGIN
    SELECT * INTO STRICT myrec FROM emp WHERE empname = myname;
EXCEPTION
    WHEN NO_DATA_FOUND THEN
        RAISE EXCEPTION 'employee % not found', myname;
    WHEN TOO_MANY_ROWS THEN
        RAISE EXCEPTION 'employee % not unique', myname;
END;

Here are some examples of condition

WHEN division_by_zero THEN ...
WHEN SQLSTATE '22012' THEN ...

Advertisements

Parse Command Line arguments

Bash

#!/bin/bash
OPTIND=1           # reset in case getopts has been used previously
# default settings
output=""
verbose=0
while getopts "h?vf:" opt; do
    case "$opt" in
    h|\?)  show_help; exit 1;;
    v) verbose=1;;
    f) output=$OPTARG;;
    esac
done

shift $((OPTIND-1))
[ "$1" = "--" ] && shift;
echo "verbose=$verbose, output=$output, leftover: $@"

Reference:

Perl

#! /usr/bin/env perl

use strict;
use warnings;
use Getopt::Std;

sub usage {
    print "usage:\n".
          "$0 -hvf other_list\n".
          "  -h     help\n".
          "  -v     verbose\".
          "  -f file output file\n".
          "\n";
    exit 1;
}

my %opts;
my $rc = getopts('hvf:', \%opts);
usage() if ($rc || $opts{h});
print "options:\n";
print "$_: $opts{$_}\n" foreach (sort keys %opts);
print "leftovers: @ARGV\n";

Reference:

Python

#! /usr/bin/env python

import argparse

parser = argparse.ArgumentParser()
parser.ad_argument('-f', '--output_file', type=str,
                     default='my_output', help='output file name')
parser.add_argument('-v', '--verbose', action='store_true', help='verbose mode')
args = parser.parse_args()
if args.verbose:
    print 'verbose mode'
with open(args.output_file, 'w') as f:
    # do something
    pass

Reference:

C/C++

under construction

Set Hostname on Linux

Types of hostnames

The hostname can be configured as follows

  1. Static host name assigned by sysadmin. For example, “server1”, “wwwbox2”, or “server42.cyberciti.biz”.
  2. Transient/dynamic host name assigned by DHCP or mDNS server at run time.
  3. Pretty host name assigned by sysadmin/end-users and it is a free-form UTF8 host name for presentation to the user. For example, “Vivek’s netbook”.

hostnamectl command

Let us see how to use the hostnamectl command.

How do I see the host names?

$ hostnamectl
## OR ##
$ hostnamectl status

Sample outputs:

   Static hostname: centos-7-rc
         Icon name: computer
           Chassis: n/a
        Machine ID: b5470b10ccfd49ed8e4a3b0e953a53c3
           Boot ID: f79de79e2dac4670bddfe528e826b61f
    Virtualization: oracle
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-229.1.2.el7.x86_64
      Architecture: x86_64

How do I set the host name?

The syntax is:

# hostnamectl set-hostname Your-New-Host-Name-Here
# hostnamectl set-hostname "Your New Host Name Here" --pretty
# hostnamectl set-hostname Your-New-Host-Name-Here --static
# hostnamectl set-hostname Your-New-Host-Name-Here --transient

To set host name to “R2-D2”, enter:

# hostnamectl set-hostname R2-D2

To set static host name to “server1.cyberciti.biz”, enter:

# hostnamectl set-hostname server1.cyberciti.biz --static

To set pretty host name to “Senator Padmé Amidala’s Laptop”, enter:

# hostnamectl set-hostname "Senator Padmé Amidala's Laptop" --pretty

To verify new settings, enter:

# hostnamectl status

Sample outputs:

   Static hostname: server1.cyberciti.biz
   Pretty hostname: Senator Padmé Amidala's Laptop
Transient hostname: r2-d2
         Icon name: computer
           Chassis: n/a
        Machine ID: b5470b10ccfd49ed8e4a3b0e953a53c3
           Boot ID: f79de79e2dac4670bddfe528e826b61f
    Virtualization: oracle
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-229.1.2.el7.x86_64
      Architecture: x86_64

How do I delete a particular host name?

The syntax is:

# hostnamectl set-hostname ""
# hostnamectl set-hostname "" --static
# hostnamectl set-hostname "" --pretty

How do I change host name remotely?

Use any one of the following syntax:

# ssh root@server-ip-here hostnamectl set-hostname server1

OR set server1 as host name on a remote server called 192.168.1.42 using ssh:

# hostnamectl set-hostname server1 -H root@192.168.1.42

 

 

 

Reference

 

Note:

  • hostnamectl exists on RHEL/Centos7, Debian8 (validated)

 

Some topics in SSH

Login SSH without typing password every time

  1. create public and private keys on local-host (debian)
    jason@debian$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/jason/.ssh/id_rsa):[Enter key]
    Enter passphrase (empty for no passphrase): [Press enter key]
    Enter same passphrase again: [Pess enter key]
    Your identification has been saved in /home/jason/.ssh/id_rsa.
    Your public key has been saved in /home/jason/.ssh/id_rsa.pub.
    The key fingerprint is:
    33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9 jason@debian
    

    This command creates two RSA key files, id_rsa for private key, and id_rsa.pub for public key, in the directory $HOME/.ssh.

  2. copy the public key file to remote-host (fedora) using ssh-copy-id
    jason@debian$ ssh-copy-id -i ~/.ssh/id_rsa.pub fedora
    jason@fedora's password:
    jason@fedora$
    

    Now try logging into the remote machine (fedora)

    jason@debina$ ssh fedora
    Last login: Thu Feb 16 11:45:20 2017 from debian
    jason@fedora$
    

    Notice that ssh does not ask for password. Check the authorized_key to make sure we haven’t added extra keys that you weren’t expecting.

    jason@fedora$ cat .ssh/authorized_keys
    

ref – 3 Steps to Perform SSH Login Without Password Using ssh-keygen & ssh-copy-id
In case ssh-copy-id is not available (for example, on mingw), as an alternated solution, you can manually copy the public key file (id_rsa.pub) to the remote host, and add the content as a record into $HOME/.ssh/authorized_keys. The file is a text file, so if it doesn’t exist, you can create it with a text editor. Don’t forget to change the mode to 644, otherwise sshd won’t use it.

Define host aliases in SSH config

Following lines in ~/.ssh/config define two aliases for the host staging.example.com

host foo bar # aliases separated by whites
  hostname staging.example.com

ref – Defining host aliases in your SSH config

Setup PostgreSQL on Linux

Install from source

The current latest release version is 9.5.4.

# get and install postgres from source
$ cd $YOUR_WORKING_PATH
$ curl https://ftp.postgresql.org/pub/source/v9.5.4/postgresql-9.5.4.tar.gz > postgresql-9.5.4.tar.gz
$ tar xzf postgresql-9.5.4.tar.gz
$ cd postgresql-9.5.4
$ ./configure # default --prefix=/usr/local/pgsql
$ make

$ sudo make install
$ sudo adduser postgres
$ sudo passwd postgres
$ sudo mkdir /usr/local/pgsql/data
$ sudo chown postgres /usr/local/pgsql/data

$ su - postgres
$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data # initialize database cluster

Configure Access from network

By default, postgres allows connection only from localhost. To allow clients connect over network, host entries need to be added in /usr/local/pgsql/data/pg_hba.conf, such as

host all all 192.168.0.0/16 md5

allows all users from 192.168.x.x to access all database, using md5 to hash their password. See more details in the manual 18.3 Connection and Authentication

Auto start when system boots

Although 17.3. Starting the Database Server in the manual gives the solution for old-style /etc/rc.d, modern linux distributions use systemd, here is the solution

add postgresql.service

[Unit]
Description=PostgreSQL database server
After=network.target
[Service]
Type=forking
User=postgres
Group=postgres

# Where to send early-startup messages from the server (before the logging
# options of postgresql.conf take effect)
# This is normally controlled by the global default set by systemd
# StandardOutput=syslog
# Disable OOM kill on the postmaster
OOMScoreAdjust=-1000

# ... but allow it still to be effective for child processes
# (note that these settings are ignored by Postgres releases before 9.5)
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0

# Maximum number of seconds pg_ctl will wait for postgres to start.  Note that
# PGSTARTTIMEOUT should be less than TimeoutSec value.
Environment=PGSTARTTIMEOUT=270
Environment=PGDATA=/usr/local/pgsql/data

ExecStart=/usr/local/pgsql/bin/pg_ctl start -o "-h '*'" -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT}
ExecStop=/usr/local/pgsql/bin/pg_ctl stop -D ${PGDATA} -s -m fast
ExecReload=/usr/local/pgsql/bin/pg_ctl reload -D ${PGDATA} -s

# Give a reasonable amount of time for the server to start up/shut down.
# Ideally, the timeout for starting PostgreSQL server should be handled more
# nicely by pg_ctl in ExecStart, so keep its timeout smaller than this value.
TimeoutSec=300

[Install]
WantedBy=multi-user.target

as /usr/lib/systemd/system/postgresql.service, then run following commands

sudo systemctl daemon-reload
sudo systemctl enable postgresql
sudo systemctl start postgresql

After that you can connect the postgresql cluster as

/usr/local/pgsql/bin/psql -U postgres -h hostname
postgres=# create role myaccount login password='my passowrd';
postgres=# create database mydb;
postgres=# alter database mydb owner to myaccount;

 

Reference

 

Perl Tricks – each %hash

Perl has three ways to iterate a hash:

  • keys %hash iterates keys
  • values %hash iterates values
  • each %hash iterates the pair (key, value)

Perl contains an internal iterator for such kind of iteration. So far so good, as long as you finish the iteration every time. What if you break the iteration before it finishes, and start a new iteration later? According to perl doc, the iterator is reset only when

  • end of iteration
  • keys or values is called

so the script might not work as you expect in the following pattern:

while (my ($k, $v) = each %hash) {
      # do some work
      last if ($k == $somekey || $v == $somevalue);
}
# sometime later
while (my ($k, $v) = each %hash) {  # iterator is not reset if the previous while not finish
         # some work
}

The conclusion – don’t use each if you plan to break the iteration, use keys instead.

 

Capture signals in perl

Here’s a sample perl script

#! /usr/bin/perl
=pod

This program demonstrate the usage of %SIG to capture signals in perl.

An alternative solution is using pragma sigtrap.
    use sigtrap qw(handler hdl_int INT QUIT);
The disadvantage of sigtrap is that you can't save and restore
the original handlers.

=cut

use strict;
use warnings;
use Time::HiRes qw( usleep) ;

package ST;             # scope tracer
sub new {
    my $class = shift;
    my $name = shift;
    my $self = {
        name => $name,
    };
    print "++++++++++++++++enter $self->{name}\n";
    bless $self, $class;
    return $self;
}
sub DESTROY {
    my $self = shift;
    print "----------------leave $self->{name}\n";
}



package main;

my %orgSig = ();        # save original handlers if restoring is wished

sub listAllSignals {
    print "supported signals\n";
    foreach (sort keys %SIG) {
        unless (/NUM\d+/) {
            my $v = $SIG{$_} || '';
            print "  $_=$v\n";
        }
    }
}

sub longtask {
    # emulating a task that runs for a while.
    my $title = shift || 'some long-time task';
    my $loop = shift || 5;
    my $interval = shift || 1;
    my $v = new ST($title);
    foreach (1..$loop) {
        print "      $title: working ...$_\n";
        usleep($interval * 1000000);
    }
}

sub foo {
    my $s = ST->new('foo');
    my $greet = shift;

    print "$greet from foo\n";
    longtask('foo body', 10, 0.5);
}

sub hdl_int {
# signal is blocked and appended in the queue
# DURING the signal handler is running,
# so restoring original handler inside the handler
# IS GENERARLLY NOT a good idea unless that is expected behavior
    my $s = ST->new('hdl_int');
    longtask('hdl_int body', 5, 0.2);
#   $SIG{'INT'} = $orgSig{'INT'};
}

$orgSig{$_} = $SIG{$_} foreach (keys %SIG);
listAllSignals();

# these two signals are used for "normal" exit
$SIG{'INT'} = \&hdl_int;
$SIG{TERM} = sub { print "I captured TERM\n"; };

# There two can't be captured
#   For sure KILL (9) can't be captured
#   According to doc, QUIT is not able to be captured either,
#   however the interesting fact is that it's captured on debian 8
$SIG{QUIT} = sub { print "I captured QUIT\n"; };
$SIG{KILL} = sub { print "I captured KILL\n"; };


print("hello world, my pid=$$\n");
sleep(3);
foo("Greeting");
print("The end\n");

Here is the output when Control-C is pressed twice during the running

jasonz@jzdebian$ perl sig.pl
supported signals
  ABRT=
  ALRM=
  BUS=
  CHLD=
  CLD=
  CONT=
  FPE=IGNORE
  HUP=
  ILL=
  INT=
  IO=
  IOT=
  KILL=
  PIPE=
  POLL=
  PROF=
  PWR=
  QUIT=
  RTMAX=
  RTMIN=
  SEGV=
  STKFLT=
  STOP=
  SYS=
  TERM=
  TRAP=
  TSTP=
  TTIN=
  TTOU=
  UNUSED=
  URG=
  USR1=
  USR2=
  VTALRM=
  WINCH=
  XCPU=
  XFSZ=
hello world, my pid=47074
^C++++++++++++++++enter hdl_int
++++++++++++++++enter hdl_int body
      hdl_int body: working ...1
      hdl_int body: working ...2
      hdl_int body: working ...3
^C      hdl_int body: working ...4
      hdl_int body: working ...5
----------------leave hdl_int body
----------------leave hdl_int
++++++++++++++++enter hdl_int
++++++++++++++++enter hdl_int body
      hdl_int body: working ...1
      hdl_int body: working ...2
      hdl_int body: working ...3
      hdl_int body: working ...4
      hdl_int body: working ...5
----------------leave hdl_int body
----------------leave hdl_int
++++++++++++++++enter foo
Greeting from foo
++++++++++++++++enter foo body
      foo body: working ...1
      foo body: working ...2
      foo body: working ...3
      foo body: working ...4
      foo body: working ...5
      foo body: working ...6
      foo body: working ...7
      foo body: working ...8
      foo body: working ...9
      foo body: working ...10
----------------leave foo body
----------------leave foo
The end
jasonz@jzdebian$ 

The interesting fact is that QUIT is captured although the linux document declares that it is not capture-able. Here is the snippet of the output

  XCPU=
  XFSZ=
hello world, my pid=47344
++++++++++++++++enter foo
Greeting from foo
++++++++++++++++enter foo body
      foo body: working ...1
      foo body: working ...2
      foo body: working ...3
I captured QUIT
      foo body: working ...4
      foo body: working ...5
      foo body: working ...6
      foo body: working ...7
      foo body: working ...8
      foo body: working ...9
      foo body: working ...10
----------------leave foo body
----------------leave foo
The end
jasonz@jzdebian$ 

when a QUIT signal is sent

jasonz@jzdebian$ kill -QUIT 47344
jasonz@jzdebian$ 

Extended Usage – capture warn() and dir()

This is extremely convenient to log all those information

sub WARN_handler {
    my ($signal) = @_;
    log("WARN: $signal");
}
sub DIE_handler {
    my ($signal) = @_;
    log("DIE: $signal");
}
sub log {
    my (@array) = @_;
    open(LOGFILE, ">>my.log");
    print LOGFILE (@array);
    close(LOGFILE);
}

$SIG{__WARN__} = 'WARN_handler';
$SIG{__DIE__} = 'DIE_handler';
chdir('/printer') or warn($!);
chdir('/printer') or die($!);