1)About Apache Spark

Apache Spark:

Apache Spark is in-memory cluster computing technology that increases the processing speed of an application.

Spark uses Hadoop in one way for storage purpose only, Since Spark has its own cluster management computation,

Note:

• Designed for fast computation. 
• It is based on Hadoop MapReduce and it extends the MapReduce model to efficiently use it for more types of computations, which includes interactive queries and stream processing etc.,
• It reduces the management burden of maintaining separate tools.

Features of Apache Spark:

1. Speed
2. Supports multiple languages
3. Advanced Analytics
4. Runs Everywhere

1.Speed:

• Spark helps to run an application in Hadoop cluster, up to 100 times faster in memory, and 10 times faster when running on disk. 
• To achieve speed of processing, It stores the intermediate processing data in memory.

2.Supports multiple languages:

• Scala applications supports built in API's in Java, Scala, Python, R

3.Advanced Analytics:

• Combine SQL, streaming, and complex analytics  
• Spark not only supports ‘Map’ and ‘reduce’. It also supports SQL queries, Streaming data, Machine learning (ML), and Graph algorithms.

4.Runs Everywhere:

• Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3.

Components/Modules of Spark:
The following are the components/modules of Apache Spark:

1. Spark core
2. Spark SQL
3. Spark Streaming
4. MLlib (Machine Learning Library)
5. GraphX

1)Spark core:
Spark core provides In-Memory computing and referencing datasets in external storage systems.

2)Spark SQL:
Spark SQL is a component on top of Spark Core that introduces a new data abstraction called SchemaRDD.

Note:
Provides support for structured and semi-structured data.

3)Spark Streaming:

Spark Streaming on top of spark core to perform streaming analytics in batches of data.

4)MLlib (Machine Learning Library):

MLlib is a distributed machine learning framework above Spark

Note:

Spark MLlib is nine times as fast as the Hadoop disk-based version of Apache Mahout

5)GraphX:

GraphX is a distributed graph-processing framework on top of Spark

Spark Architecture Execution flow:
Spark Architecture Execution flow includes the following below

• Driver Program
• Cluster Manager
• Worker nodes
• Executor

Apache Spark - Cluster modes:
Below are the cluster modes in Apache Spark

• Localmode
• YARN
• Mesos
• Standalone

Please click the next to proceed further ==> Next

7)Python if.elif.else Statements

Python if.elif.else Statements

The if…elif…else statement is used in Python for decision making.

i)if Statement:
Syntax:

if test expression:
    statement(s)

if Statement Flowchart:

Example:

x = 10
if x > 0:
    print(x,"is positive numner")

ii)if...else Statement
Syntax:

if test expression:
    Body of if
else:
    Body of else

if...else Statement Flowchart:

Example:

x = 10
if x > 0:
    print(x,"is positive numner")
else:
    print(x,"is either zero or negative number")

iii)if...elif...else statement:

Syntax:

if test expression:
    Body of if
elif test expression:
    Body of elif
else:
    Body of else

if...elif...else Flowchart:

Example:

x = 10
if x > 0:
    print(x,"is positive numner")
elif x < 0:
    print(x,"is negative number")
else:
    print(x,"is zero")

6)Python Operators

Python Operators
Operators are special symbols in Python that are useful to compute values of operands

Type of operators in Python:

1. Arithmetic operators
2. Comparison (Relational) operators
3. Logical (Boolean) operators
4. Bitwise operators
5. Assignment operators
6. Special operators

1.Arithmetic operators:
Arithmetic operators are used to perform mathematical operations like addition, subtraction, multiplication etc.

Arithmetic operators table:

Operator	Meaning	Example
+	Add two operands or unary plus	x + y +2
-	Subtract right operand from the left or unary minus	x - y -2
*	Multiply two operands	x * y
/	Divide left operand by the right one (always results into float)	x / y
%	Modulus - remainder of the division of left operand by the right	x % y (remainder of x/y)
//	Floor division - division that results into whole number adjusted to the left in the number line	x // y
**	Exponent - left operand raised to the power of right	x**y (x to the power y)

Examples:

2.Comparison operators:

Comparison operators are used to compare values. It either returns True or False

Comparison operators table:

Operator	Meaning	Example
>	Greater that - True if left operand is greater than the right	x > y
<	Less that - True if left operand is less than the right	x < y
==	Equal to - True if both operands are equal	x == y
!=	Not equal to - True if operands are not equal	x != y
>=	Greater than or equal to - True if left operand is greater than or equal to the right	x >= y
<=	Less than or equal to - True if left operand is less than or equal to the right	x <= y

Examples:

3.Logical operators:

Logical operators are the and, or, not operators.

Logical operators table:

Operator	Meaning	Example
and	True if both the operands are true	x and y
or	True if either of the operands is true	x or y
not	True if operand is false (complements the operand)	not x

Examples:

4.Bitwise operators:

Bitwise operators act on operands as if they were string of binary digits.

Bitwise operators table:

Operator	Description	Example
& Binary AND	Operator copies a bit to the result if it exists in both operands	(a & b) (means 0000 1100)
| Binary OR	It copies a bit if it exists in either operand.	(a | b) = 61 (means 0011 1101)
^ Binary XOR	It copies the bit if it is set in one operand but not both.	(a ^ b) = 49 (means 0011 0001)
~ Binary Ones Complement	It is unary and has the effect of 'flipping' bits.	(~a ) = -61 (means 1100 0011 in 2's complement form due to a signed binary number.
<< Binary Left Shift	The left operands value is moved left by the number of bits specified by the right operand.	a << 2 = 240 (means 1111 0000)
>> Binary Right Shift	The left operands value is moved right by the number of bits specified by the right operand.	a >> 2 = 15 (means 0000 1111)

Examples:

5.Assignment operators:
Assignment operators are used to assign values to variables.

Assignment operators table:

Operator	Example	Equivatent to
=	x = 5	x = 5
+=	x += 5	x = x + 5
-=	x -= 5	x = x - 5
*=	x *= 5	x = x * 5
/=	x /= 5	x = x / 5
%=	x %= 5	x = x % 5
//=	x //= 5	x = x // 5
**=	x **= 5	x = x ** 5
&=	x &= 5	x = x & 5
|=	x |= 5	x = x | 5
^=	x ^= 5	x = x ^ 5
>>=	x >>= 5	x = x >> 5
<<=	x <<= 5	x = x << 5

6.Special operators:
They are two types of special operators:

1. Identity operators
2. Membership operators

i)Identity operators:
Identity operators used to check if two values (or variables) are located on the same part of the memory.

Identity operators Table:

Operator	Meaning	Example
is	True if the operands are identical (refer to the same object)	x is True
is not	True if the operands are not identical (do not refer to the same object)	x is not True

ii)Membership operators:
in and not in are the membership operators in Python. They are used to test whether a value or variable is found in a sequence (string, list, tuple, set and dictionary).

Membership operators Table:

Operator	Meaning	Example
in	True if value/variable is found in the sequence	5 in x
not in	True if value/variable is not found in the sequence	5 not in x

Example:

9)Tuples in scala

Tuples in scala:
Tuple is the collection class in Scala which can hold multiple values with same or different types together.

Syntax:
tuples are declared within braces "()"

Note:
tuple can hold up-to 22 values/elements, since the scala compiler is smart enough, so dont need to specify the lengh explicitly.

tuple1,tuple2.....tuple22

Useful at below situations:

• if you want a common reference
• Ex: student details like name,age, etc.,
• Performance wise is better than list

Example:
Student details with three elements

Note:

.getClass function is used to return data type of variable

Access values of Tuple in Scala:
To access values inside the tuple is through position of elements _1,_2,......_22.

Syntax:

<<tuple_variable>>._1
<<tuple_variable>>._2
<<tuple_variable>>._3...etc

Example:
val studentname = studentinfo._1

Assign values of tuples to multiple variables in one line:
We can also assign the tuple values to multiple variables at once

Syntax:

val (var1,var2,...) = tupleholder

Example:

8)Pattern Matching in Scala

Pattern Matching in Scala:
Pattern matching is a mechanism for checking a value against a pattern.  and much similar to java switch statement.

Syntax:
A match expression with the match keyword and at least one case clause.

varible_name match
{
case value => rep_value
.....
}

Simple Example:

Note:

Wild card _ - which means it will match anything.

Pattern matching to store the result in varible:

let's we try to hold the return match result into variable.

Example:

Pattern matching with or condtion:
passing or (|) filter in the case conditon

Example:

7)Range in Scala

Range in Scala:
Selective range of many types such as numbers or characters in sequence that can be used in looping constructs.

Types of Ranges:

1. Number Ranges
2. Alphabetical Ranges

1.Number Ranges:
Selective range of numbers using to,until, and by kewords.

Example:

i)Selective range using to, until keyword:

Note:

• to keyword to include the the last upper bound
• until keyword to exclude the last upper bound

ii)Range numbers 1 to 10 with increment of 3:

Note:

by keyword to increment by the number as opposed to the default value of 1.

2.Alphabetical Ranges:

Selective range of alphabets using to,until, and by kewords.

Example:

i)Selective range using to, until keyword:

Note:

• to keyword to include the the last upper bound character
• until keyword to exclude the last upper bound character

ii)Range characters 'A' to 'Z' with specified increment of 3:

Note:

by keyword to increment by the character as opposed to the default value of 1.

iii)Convert Scala Range into collections:
Sometimes we may need to convert the Range into some other data structure such as List, Set, Sequence or an Array.


Below, which uses methods called toList, toArray, toSet and toSeq functions to convert Range

Example:

Note:

• mkString() function -  create a string representation of each collection type. The .mkString() function takes in a delimiter.
• toList, toArray, toSet and toSeq - to convert Range

6)Scala programming constructs: for,while and do-while loop

for loop:
Repeatation of execution until the specific cycles.

1)Iterating 1 to 5 number in scala using to keyword:

Example:

Note:
keyword to which meant that iteration number 5 is included.

2)Iterating 1 to 5 number in scala using until keyword:

Example:

Note:

keyword until  which meant that iteration number 5 is not included.

3)if conditions in for loop as well:

if clause to add filtering support as you iterate through your items.

Example:

4)if conditions in for loop and return the result back using the yield keyword:


using the {} in our for loop to make our expressions more explicit.

Example:

5)for loop usage in 2-Dimensional array:

In scala, to work with 2-Dimensional arrays. There is a Scala Array class and the appropriate ofDim() function, pass in the type of String in square brackets [String] and then specify R by C array in the function parameter.

Syntax:
Declaration:

val varible_name = Array.ofDim[String](R,C)

Initialization:

varible_name[R1][C1]=valuevarible_name[R1][C2]=valuevarible_name[R2][C1]=valuevarible_name[R2][C2]=value

Then as usual, loop through it for each occurrence.

Example:

While loop:
Executes the block of code, only if the condition is true.

Example:

do while loop:
do {} will be ran at least once regardless of the condition within the while() clause.

Example: