Netezza Puredata Backup failure with Error: NZ-02465: Stream unresponsive after 300 seconds, operation aborted. Check concurrency limits on server.

Netezza Puredata Backup failure with Error: NZ-02465: Stream unresponsive after 300 seconds, operation aborted. Check concurrency limits on server.

I have seen similar kind issue in Puredata system 7.2.0.5

Basically, Netezza backup processes check that number of streams requested is supported by the backup tool. If Yes,  then only backup process starts.

If the check fails due to connection timeouts, the nzbackup process exits with the following

Error Message(NZ-02465): Stream unresponsive after 300 seconds, operation aborted. Check concurrency limits on server.

Cause:- It may  happens due to Media server which is associated with storage taking more than 5 min  to acquire the necessary mounts, If its taking more than 5 min(default 300sec).

Solution:-

Change the timeout to 600 seconds ( from the default of 300 seconds)

Method:-1

$ nzsystem pause

Are you sure you want to pause the system (y|n)? [n] y

$ nzsystem showRegistry | grep bnrStreamInitTimeoutSec

host.bnrStreamInitTimeoutSec      = 300

$ nzsystem set -arg host.bnrStreamInitTimeoutSec=600

Are you sure you want to change the system configuration (y|n)? [n] y

$ nzsystem resume

check Status

$nzstate

Method 2:-

A. make changes in  /nz/data/config/system.cfg by adding the below parameter:

host.bnrStreamInitTimeoutSec=600

B. Restart the NPS System  using the "nzsystem restart" command.

Frequently Asked Questions: - IBM PureData System for Analytics Netezza

Frequently Asked Questions: - IBM PureData System for Analytics Netezza

What components of a PureData System for Analytics appliance are available for Upgrade?

There are three main components of a system that can be upgraded: NPS, FDT, and HPF.

What is NPS?

NPS is the Netezza Performance Server, the main high performance database server for the system.

What is HPF?

HPF is the Host Platform. The HPF software contains kernel updates, firmware, and other special software used by the PureData System host servers. The HPF upgrade and downgrade process includes extensive error-checking, backups, and logging.

What is FDT?

FDT is the Firmware Diagnostics and Tools and contains the firmware updates for the hardware components of the PureData System for Analytics.

Where can I find the minimum versions of HPF and FDT required for the NPS I wish to upgrade to?

see "fixcentral" in the Related information section

Product Group: Information Management

Select from Information Management: IBM Netezza NPS Software and Clients

Install Version: desired NPS version

Platform: all

From here you will need to download the file called nz-npsdoc-vVersion.tar.gz. The pdf with this information is titled "Netezza Software Upgrade."

What should I download if having an NPS only upgrade performed?

We will need the NPS package from:

see "fixcentral" in the Related information section

Product Group: Information Management

Select from Information Management: IBM Netezza NPS Software and Clients

Install Version: desired NPS version

Platform: all

The package should be titled nz-nps-vVersion.tar.gz please download and place in the /nzscratch/upgrade directory

on the server.

see "fixcentral" in the Related information section

Product Group: Information Management

Select from Information Management: IBM Netezza Tools

Install Version: desired SWTools version(Typically match the NPS version)

Platform: all

Select the desired SWTools package and download the file called IBM_Netezza_Software_Support_Tools-vVersion.tar.gz

see "fixcentral" in the Related information section

Product Group: Information Management

Select from Information Management: IBM Netezza Tools

Install Version: latest HWTools version

Platform: all

Select the latest HWTools package and download the file called nz-hwsupport-tf-VVersion.tar.gz

What should I download if having a full upgrade performed?

The NPS packages listed above are needed, as well as the following HPF and FDT packages.

For HPF:

see "fixcentral" in the Related information section

Product Group: Information Management

Select from Information Management: IBM Netezza Platform

Install Version: desired HPF version

Platform: all

Select the desired HPF package and download the file called nz-hpf-vVersion.tar.gz.

For FDT:

see "fixcentral" in the Related information section

Product Group: Information Management

Select from Information Management: IBM Netezza Platform

Install Version: desired FDT version

Platform: all

Select the desired FDT package and download the file called nz-fdt-vVersion.tar.gz

What information about the system should I provide when opening the upgrade PMR?

Output to the following commands as nz user:

cat /nzlocal/scripts/version.txt

cat /nzlocal/scripts/MODEL

head -2 /opt/Netezza/FW/PSeries/version.txt

head -2 /opt/nz/fdt/version.txt

cat /etc/redhat-release

nzrev

nzstats

As root user:

dmidecode -t 1

Also please provide the address of the server to send the IBM on-site technician to be on-site for the duration of the upgrades, if needed.

Does my upgrade require an IBM technician to be on-site?

We require an IBM technician to be on-site for all FDT upgrades to versions FDT 3.0.5.4 and lower. The IBM on-site technician is there to perform the Host firmware updates on the physical system. Upgrades to FDT 4.1 and higher do not require an IBM SSR on-site as the hosts are able to be updated remotely.

When can I schedule upgrades?

We are available 24/7 for upgrade activity however there are a few restrictions for providing a tentative schedule:

For scheduling a full upgrade for a weekdays please provide at least 1 weeks notice. For scheduling on the weekends please contact your local Support to speak to them about lead times needed for upgrades. For NPS only upgrades please provide at least 48 hours notice for a tentative schedule. While we will do our best to meet customer requests we may, on occasion, need to look for other dates and times to carry out the requested upgrades. The more notice we can have the more likely we can accommodate your provided schedule.

Is a pre-upgrade check required?

Yes, we require a pre-upgrade check to be performed on the system 3-7 days prior to the scheduled upgrade date. The pre-upgrade checks are non-intrusive and can be run while the system is online. It is important that the pre-upgrade check is performed a maximum of 7 days prior to the upgrade to minimize the possibility of issues arising between pre-check and upgrade.

Do upgrades require root access?

Yes, all upgrades and pre-upgrade checks require root access to the system. Please have root passwords ready for scheduled pre-upgrade checks and upgrades.

How will the remote technician access my system?

We have two ways to access systems to perform upgrades. The first is Webex where we will send an invitation to the customer and they will share their desktop to connect to the system. Second method of access is if the customer has previously set up remote access we will be able to access those systems remotely to perform the upgrades.

Do I need to resolve hardware issues prior to upgrading?

Yes we require that all hardware issues be resolved before performing the upgrades, whether it is disk replacements, SPU repair, etc.

Can I schedule an upgrade with another activity such as Hardware replacement?

No, full upgrades typically take 5-8 hours to complete, scheduling any hardware activity for the same outage can cause delays in the upgrade and creates more risk that issues may arise during the upgrade activity.

What components of Firmware get updated in FDT?

The firmware components of FDT are as follows: Hosts, Remote Power Controllers, Spa Management Modules, Spa Gigabit switches, Spa SAS switches, Rack Management switches, Rack Fabric switches, Spa power supplies, Blades, HBA, Storage Enclosures, and Storage Media.

Where can I find what is new in each version of FDT?

You can extract the files from the nz-fdt-vVersion.tar.gz file to get the release notes and what fw levels were updated in this version of FDT.

What is sys_rev_check?

Sys_rev_check is a script that checks the current firmware levels of the system and compares them to the levels of firmware that the FDT expects. This tells us which components need to be updated during the FDT upgrades.

How often should I upgrade?

You should upgrade your system HPF and FDT once a year. NPS can also be updated to the latest at this time or customer can choose not to update NPS as well if satisfied with current version.

How long do upgrades take?

If the system meets the minimum requirements(HPF and FDT) for the desired NPS release then we can perform an NPS only upgrade which takes approximately 1 hour to complete. If it does not meet the minimum requirements for that release of NPS then we will have to perform a full upgrade(NPS, HPF, FDT) which takes roughly 8 -12 hours to complete. On average a 8 hours outage window is needed, yet for a system with 48 SPUs or higher, an outage window of at least 12 hours is required.

The rough time estimates on these are:

NPS: 1 hour

HPF: 1 hour

FDT: 6 -10 hours

Here is the breakdown of the approximate time for each component of FDT

AMM - 1-3 hrs

Blades - 2-6 hrs

Storage Media - 30min - 1 hr.

Host Bios Firmware - 2 (1 hr for each host)- can take upto 3 hrs(1.5 hrs per host) sometimes

HBA - 15 min

Please note that the upgrade may vary depending on the system. There are several factors that influence this such as

- The current and the desired levels of NPS/FDT/HPF

- Was the system recently upgraded ?

- Size of the system- sometimes blade updates may take longer to complete on big systems

- No hardware or platform issues are encountered during the course of the upgrade. We do check the system for any issues prior to the upgrade but this does not guarantee that no issues will be encountered during the course of the upgrade.

The time frame provided above are approximates based on a typical upgrade. There is no guarantee that the upgrade will not take longer than anticipated. If any components fail or firmware upgrades do not take, the support engineer needs to perform active troubleshooting which includes restarts and this adds time to upgrade and hence the outage.

If currently using INZA(IBM Netezza Analytics), do I need to install a new version of INZA?

Please check the pre-requisites section of the IBM_Netezza_Analytics_Administrators_Guide.pdf to confirm that your current version of INZA is compatible with the desired version of NPS. If the INZA requires an upgrade as well please open a ticket with our Database Technical Support team to have a compatible version of INZA installed on your system.

Installation of the SQL Extensions or COMMON_DB Toolkit on IBM Netezza

Installation of the SQL Extensions or COMMON_DB Toolkit on IBM Netezza

Login as nz user

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ pwd

/export/home/nz/SQLExtn/

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ ls -lrt

total 4568

-rwxrwxr-x 1 nz nz 4.7M Feb 23 11:04 sql-ext-7.2.0.5.tar.gz*

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ tar -zxvf sql-ext-7.2.0.5.tar.gz*

Netezza_sql_toolkit.pdf

sqlext.package.tar.z

SQL_Toolkit_Readme.txt

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ ls

Netezza_sql_toolkit.pdf  sql-ext-7.2.0.5.tar.gz  sqlext.package.tar.z  SQL_Toolkit_Readme.txt

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ tar -zxf sql-ext-7.2.0.5.tar.gz

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ tar -zxf sqlext.package.tar.z

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ ls

libnetcrypto-1.6.P-1.tar.gz  Netezza_sql_toolkit.pdf  sqlext.package.tar.z

libnetxml-7.2.0.5.tar.gz sql-ext-7.2.0.5.tar.gz     SQL_Toolkit_Readme.txt

And again to get to the libnet libraries.

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ tar -zxf libnetcrypto-7.2.0.5.tar.gz

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ tar -zxf libnetxml-7.2.0.5.tar.gz

[nz@edw-ntz-0a1-lq SQLExtn]$ ls

libnetcrypto/                libnetxml/                Netezza_sql_toolkit.pdf  sqlext.package.tar    SQL_Toolkit_Readme.txt

libnetcrypto-7.2.0.5.tar.gz  libnetxml-7.2.0.5.tar.gz  sql-ext-7.2.0.5.tar.gz*  sqlext.package.tar.z

[nz@nz@edw-ntz-0a1-lq SQLExtn]$ cd libnetcrypto/7.2.0.5/

[nz@nz@edw-ntz-0a1-lq 7.2.0.5/]$ ls

install*    libbotan_host.so*  libnetcrypto.o_spu10  licenses/       netfpe.o_x86*  RELEASE_NOTES

install.pm  libbotan_spu.so*   libnetcrypto.o_x86*   netfpe.o_spu10  README         sqlext-version.txt

You can ask the installer for help.

[nz@nz@edw-ntz-0a1-lq 7.2.0.5]$ ./install -h

Usage: install [options]

Options:

    -db|-D   :  Specify the database to install the toolkit

    -user|-U   :  Specify the Userid used to connect to the database

                          (default: admin)

    -pw|-W   :  Specify the Password used to connect to the database

    -fenced            :  Install the library in fenced mode on release 6.x

                          (default: unfenced)

    -uninstall|-R      :  Uninstall the library

    -verbose|-V        :  Verbose messages during installation / uninstallation

    -schema|-S         :  Specify the Schema Name to install the toolkit

           

Here’s our database.

[nz@clvs1001 sqlext]$ nz_get_database_name SQLEXTN

SQLEXTN

Now to install libnetcrypto.

[nz@nz@edw-ntz-0a1-lq 7.2.0.5]$./install -d SQLEXTN

Installing libnetcrypto to default schema of database SQLEXTN in unfenced mode: .................................................. Done

And Now libnetxml

[nz@clvs1001 7.2.0.5]$ cd ../../libnetxml/7.2.0.5/

[nz@clvs1001 7.2.0.5]$ ./install -d SQLEXTN

Installing libnetxml to default schema of database SQLEXTN in unfenced mode: ............................................ Done

Always  Remember to update any existing synonyms and create new ones for database’s which are going to use the functions.

select distinct 'create synonym '||function||' for SQLEXTN.'||owner||'.'||function||';' from _v_function where builtin = 'f';

select distinct 'create synonym '||aggregate||' for SQLEXTN.'||owner||'.'||aggregate||';' from _v_aggregate where builtin = 'f';

And to confirm the version.

[nz@edw-ntz-0a1-lq 7.2.0.5]$ nzsql -d SQLEXTN

Welcome to nzsql, the IBM Netezza SQL interactive terminal.

Type:  \h for help with SQL commands

       \? for help on internal slash commands

       \g or terminate with semicolon to execute query

       \q to quit

SQLEXTN.ADMIN(ADMIN)=> SELECT regexp_version();

                                    REGEXP_VERSION

--------------------------------------------------------------------------------------

 IBM Netezza SQL Extensions XML / Regular Expression Library Version 7.2.0.5 Build ()

(1 row)

SQLEXTN.ADMIN(ADMIN)=>

Configure History Database in IBM Netezza

Configure History Database in IBM Netezza

Step:-1 Create History User                                          

-----------------------------------------                                                           

nzsql -c "Create user HISTUSR with password 'HISTUSR';"  

               

Step: - 2 Grant privileges   to user HISTUSR                                                      

-----------------                                                      

nzsql -c "grant create database to HISTUSR;"                           

nzsql -c "grant list on user to HISTUSR ;"                              

Step: 3 create the history database                                            

---------------------------                                            

nzhistcreatedb -d HISTDB -t query -o HISTUSR -pw HISTUSR -u HISTUSR  -v 3    

                                              

Step: 3 create the history configuration                                      

---------------------------------                                      

nzsql -d SYSTEM                                                         

CREATE HISTORY CONFIGURATION HISTORY_ENABLE_V3                         

HISTTYPE QUERY                                                         

NPS LOCALHOST                                                          

DATABASE HISTDB                                                         

USER  HISTUSR                                                          

PASSWORD 'HISTUSR'                                                     

COLLECT QUERY , PLAN , TABLE , COLUMN , SERVICE , STATE                

LOADINTERVAL 5                                                         

LOADMINTHRESHOLD 4                                                     

LOADMAXTHRESHOLD 20                                                     

STORAGELIMIT 51200                                                     

LOADRETRY 1                                                            

ENABLEHIST TRUE                                                        

ENABLESYSTEM TRUE                                                      

VERSION 3 ;                                                            

                                           

Step: 4 to start the collection of history data                                 

---------------------------------------                                

nzsql -c "set history configuration HISTORY_ENABLE_V3;"                

Step 5. Restart NPS to place to changes.

---------------------------------------------

nzstop/nzstart at linux prompt

                           

Keep Checking status on using nzstate.

Troubleshooting

----------------------------------------------

     

IF the password of the "HISTUSR" changes, please note that we need to   Update the new password to the history configuration also.             

     

nzsql -c "CREATE HISTORY CONFIGURATION hist_disabled HISTTYPE NONE;"   

nzsql -c "SET HISTORY CONFIGURATION hist_disabled;"                    

     

nzstop/nzstart at linux prompt                                         

     

Alter the change password in query history                             

     

nzsql -c "ALTER HISTORY CONFIGURATION HISTORY_ENABLE_V3 USER HISTUSR PASSWORD 'HISTUSR'"                                                     

     

Set query history     

                                                

nzsql -c "set history configuration HISTORY_ENABLE_V3;"                

     

nzstop/nzstart at linux prompt                                          

     

====================================================================   

     

Below are command to check the snippet of outputs   for history database.                                    

     

nzsql -c "set history configuration HIST_ALL;"

nzsql -c "show history configuration all"  

LDAP Troubleshooting in IBM Netezza Pure Data

LDAP Troubleshooting in IBM Netezza 

Troubleshooting 1:-

LDAP User Timeout Error or slow connection time or bad performance of LDAP User from remote machine. From Netezza database.

Error from client when doing SQL:-

ExecuteReader requires an open and available Connection. The connection's current state is closed.

With Active directory setups you may need to add the following to /etc/pam_ldap.conf

tls_request never         - For  self signed certificate

referrals no                   - fix for LDAP time out issues

Troubleshooting continued ….

 If the users are able to login into the database without error then the next step is to finish the setup.   Run the below command.

cp /etc/pam_ldap.conf /nz/data/config/pam_ldap.conf

Note: The ownership of /nz/data/config/pam_ldap,conf must be nz:nz. If you copied file by root user then issue this command

chown nz:nz /nz/data/config/pam_ldap.conf

Troubleshooting 2:-

To do a proper troubleshoot ldap issues some information which you used to setup LDAP in Netezza.

As root user issue the following commands to Check connectivity to with below command and make sure this should work.

#ldap server nc -vz ldapservername.or.ip 389

If the command does not succeed then there is a connectivity issue or ldap is not running on the port indicated.

Information that needs to be collected

cat /etc/redhat-release

rpm -qa | grep openldap

egrep -v '^(#|$)' /etc/pam_ldap.conf

Put the output of all three commands into a service request.  openldap-clients will need to be installed for further PD of the issue.

With ldapsearch installed you can check the binddn access using the information collected earlier..

1. Server : adserver.company.com

2. Port : 389

3. BaseDN: dc=users,dc=company,dc=com

4. BindDN: cn=Manager,dc=Accounts,dc=company,dc=com

5. BindPassword: password

6. attrname: sAMAccountName

Example

ldapsearch -h 10.192.83.13 -b "DC=domestic,DC=guitarcenter,DC=com" "(&(memberOf=CN=GCEDW-NETEZZA-GRP,OU=Netezza, OU=Security Groups - Global, OU=Corporate, DC=domestic, DC=guitarcenter,DC=com))" -D "CN=svcNTZ-LDAPS-prod,OU=LDAPRefAccounts ,OU=Service Accounts, OU=Special,DC=domestic,DC=guitarcenter,DC=com" -w ‘password’ |grep -i samaccountname

Any errors returned will need to be fixed.

Issues normally encountered are

Password locked (AD code 532 PASSWORD_EXPIRED) Have AD / LDAP team unlock the password

User not found (AD code 525 USER NOT FOUND)    binddn name incorrect

Troubleshooting 3:-

Database connection Test Failed or refused from Netezza Database

LDAP authentication is configured on Netezza server any user is having any issue like.

error=LDAP  authentication failed for user 'MANISH.JAISWAL'

You can check log file for error and troubleshooting:-

[nz@edw-ntz-0a1-lq ~]$ cat /nz/kit/log/postgres/pg.log|grep  MANISH.JAISWAL

2016-03-29 15:40:01.052431 PDT [31921]  DEBUG:  connection failed: host=127.0.0.1 user=MANISH.JAISWAL database=QMFISTAGE error=LAP  authentication failed for user 'MANISH.JAISWAL'

User can do the below mistake:-

Please ask them to use him username is upper case (MANIHS.JAISWAL)

1.       Not using LDAP id still using old account. You may check his account is exist or not.

Nzsql > \du user_name

2.       They may be not using LDAP password (desktop password)

3.       May be they don’t have access to those database, Please check access using below command.

$nz_ddl_user MANISH.JAISWAL

Above command will which database they are having which kind of access, if they want more access or account creation then