Tuesday 15 January 2019

Fundamentals of Python programming


Fundamentals of Python programming:
Following below are the fundamental constructs of Python programming:
  • Python Data types
  • Python Operators
  • Conditional Statements
  • Looping Constructs
  • Python functions
i) Python Data types:
Python programming language is the dynamically typed. Hence explicit type define not required for the variables, methods etc.,

List of data types:
  • Numerical data types – int, float, complex
  • String data type - str
  • Data structure data types – list, set, dict and tuple

Numerical data types:



String data type:
Strings are identified as a set of characters represented in the quotations either in (‘ ‘ , “ “, “”” “””, ‘’’ ‘’’).



Data structure data types:
Following are the data structure data types:
  • list, 
  • set,
  • dict and 
  • tuple

List:
List is an ordered sequence of items.

Note:
All the items in a list do not need to be of the same type.

Declaration:
Separated by commas are enclosed within brackets [ ].

Slicing operator [ ]:
To extract an item or a range of items from a list. Index starts form 0 in Python.


Example:


Python Set:
Set is an unordered collection of unique items.

Declaration:
Set is defined by values separated by comma inside curly braces { }. 

Note:

  • Items in a set are not ordered and not indexed so slicing operator won’t work with sets
  • We can perform set operations like union, intersection on two sets. 
  • Set have unique values which in turn eliminate duplicates.



Example:


Python Dictionary:
Dictionary is an un-ordered collection of key-value pairs.

Declaration:
In Python, dictionaries are defined within braces {} with each item being a pair in the form key:value. Key and value can be of any type.

Note:
To retrieve value of dictionary can be done through only the key.


Example:



Python Tuple:
Tuple is an ordered sequence of items same like list.

Note:
The only difference is that tuples are immutable. Tuples once created cannot be modified, it returns an exception,
Tuples are used to write and protect data, are usually faster than list as it cannot change dynamically.

Declaration:
Separated by commas within parenthesis or braces,

Slicing operator [ ]:
For tuples, using slicing operator, only retrieving the values can be allowed, but not allow the value change.


Example:



ii) Python operators:

Please follow the blog post for   Python operators


iii) Python conditional statements:

Please follow the blog post for Python conditional statements


iv) Looping constructs:




Example:

v) Python functions: source
A function is a block of code which only runs when it is called. We can pass data known as parameters into a function. functions can return data as a result and provides a re usability of code.

There are three types of functions:

  • User defined functions
  • Pre-defined functions
  • Lambda functions

User-defined functions:
Let's explore about user defined functions:
In Python, user defined function are created using the def keyword and called by using function name followed by parenthesis.

Syntax and Example:





Function parameters/arguments:
There are various ways using function arguments based on the use cases:

  • Default function arguments
  • Variable length arguments
  • Keyword arguments

Default function arguments:
Default function argument values indicates, the function will take that value if no argument value is passed during function call and overrides the value if any explicit values for the argument. The default value is assigned by using assignment(=) operator.

Syntax and Example:







Variable length arguments:

Variable length arguments is used to pass a variable number of arguments to a function. It is used to pass a non-keyworded, variable-length argument list with help of special syntax *args.

Syntax and Example:





Keyword arguments:

Keyword arguments in python is used to pass a keyworded, variable-length argument list. We use the name kwargs with the special syntax **kwargs.

Syntax and Example:






Lambda(Anonymous functions):
Lambda function can have any number of arguments but only one expression as a return statement and these functions are mostly used on map, filter, reduce functions etc.,

Note:

Lambda functions are syntactically restricted to a single expression.

Syntax:
#Python lambda functions - Syntax
'''
lambda args: return statement

'''

Thanks and Please follow the blog for further updates.

Wednesday 13 June 2018

Apache Nifi Installation on Ubuntu

Apache Nifi Installation on Ubuntu:
Step 1: Download from Apache Nifi website and extract Nifi package in desired directory

Apache Nifi website link to download : https://nifi.apache.org/download.html

There could be two distributions:

  1. ends with tar.gz - for Linux
  2. ends with zip - for Windows

Screenshot for reference:



Extract the distribution:

Command: tar -xvf /home/mano/Hadoop_setup/nifi-1.6.0-bin.tar.gz

Screenshot for reference:


Step 2: Configuration

NiFi provides several different configuration options which can be configured on nifi.properties file.

At present, i'm just making change to nifi.ui.banner.text property.





Step 3: Starting Apache Nifi:
On the terminal window,navigate to the Nifi directory and run the following below commands:

  • bin/nifi.sh run - Lauches the applicaion run in  the foreground and exit by pressing Ctrl-c.
  • bin/nifi.sh start - Lauches the application run the background.
  • bin/nifi.sh status - To check the application status
  • bin/nifi.sh stop - To shutdown the application

i)bin/nifi.sh run:



ii) bin/nifi.sh start:

iii) bin/nifi.sh status:

iv)bin/nifi.sh stop:



Step 4: Apache Nifi Web User Interface: 
After Apache Nifi Started, Web User Interface (UI) to create and monitor our dataflow.


To use Apache Nifi, open a web browser and navigate to http://localhost:8080/nifi




Friday 1 June 2018

Steps to install Apache Spark on Ubuntu

Steps to install Apache Spark on Ubuntu

Step 1: Download Apache Spark distribution

Use the link to download the spark distribution ==> http://spark.apache.org/downloads.html






Download from terminal using below command:

wget http://www-eu.apache.org/dist/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz



Step 2: Untar the Spark distribution

tar xzf spark-1.6.1-bin-hadoop2.6.tgz

 
Step 3: Setup the environment variable:

set SPARK_HOME=/usr/local/spark

Follow below steps to set spark environment variables in .bashrc file.  

nano .bashrc


 

Step 4: Launch Spark shell /pyspark context:

scala API command line: 
run spark-shell to enter into scala context.

Python API command line:

run pyspark to enter into python context.


R API command line:

run sparkR to enter into R context.

 Step 5: Spark UI:
Enter the below url in the browser to check spark execution or DAG    information to debug etc.

URL ==> http://localhost:4040
 
 Done, it's great step to proceed further data processing using Apache Spark.


Fundamentals of Python programming

Fundamentals of Python programming: Following below are the fundamental constructs of Python programming: Python Data types Python...