Python for Spark Tutorial – Method and Function in Python

In the last tutorial, we’ve learned about the different control structures in Python. Now that we know control structures and basics of Python programming, let’s have a look into how to encapsulate code into functions. Therefore, we will have a look at the method and function in Python.

Defining a Function in Python

Due to the overall differences with C-like languages, also the method definition is slightly different. However, it is very similar to the control structures layout. Let’s have a look

def functionname(args):
      FUNCTION BLOCK
      return value

A function in Python is always defined with “def”. After that, a function name is provided. Values passed to the function are then in parentheses. Due to the dynamic aspects of Python, it doesn’t know any dedicated type definitions. This means that values are passed by their name to the function. After the “:”, the function block starts. Everything within the function block needs to be indented. Python can also return values by adding the “return” statement. The following function adds one to a number:

def myiter(n):
    return n + 1
val = myiter(33)
val

Output:

34

As you would expect, it is also possible to add different levels by indenting control structures and alike. The following function creates an array from 0-4 (in the range of 5) and calls a function that iterates over each item in the array.

def printall(vals):
    for val in vals:
        print(val)
values = range(5)
printall(values)

Output:

0
1
2
3
4

Lambda Expressions

A very cool feature of most modern programming languages is the availability of Lambda expressions. With this, it is possible to significantly reduce the code complexity by writing easy functions in one-liners. For instance, the first sample of the iterator could also be written as a lambda expression. Basically, a lambda expression is a function defined in-line to be called on each item in a list or an array. It is very useful for data manipulations. Basically, a Lambda expression is introduced with the following statement:

lambda variable: STATEMENT

In this case, variable is one or more variable(s) to work with in the following statement. For instance, if it is the previous sample, it would mean a number. Thus we would only use one variable. If it would be a dictionary, it can also be just one (and the key/values are available as methods) or you would provide both, for instance as x, y. A sample lambda expression matching with the previous one is this:

v = map(lambda x: x + 1, values)
printall(v)

Output:

1
2
3
4
5

In the above statement, we used the “map” function out of python that is capable of calling a lambda function on the specified iterable (array in our case). We re-used the existing array specified in the statement above. Each item of the array is now changed in its value by one. As you can see, it is very easy to work with lambda expressions in Python and they are very useful to keep your code simple and clean.

In the next tutorial, we will have a look at how to encapsulate methods and functions into classes and packages. We will also have a look at inheritance.

If you are not yet familiar with Spark, have a look at the Spark Tutorial i created here. Also, I will create more tutorials on Python and Machine Learning in the future, so make sure to check back often to the Big Data & Data Science tutorial overview. I hope you liked this tutorial. If you have any suggestions and what to improve, please feel free to get in touch with me! If you want to learn more about Python, I also recommend you the official page.