Python for Spark Tutorial – Statistics and Mathematics in Python

One of the reasons why Python is so popular for Data Science is that Python has a very rich set of functionality for Mathematics and Statistics. In this tutorial, I will show the very basic functions; however, you might be very disappointed, since they are really basic. When we talk about real data science, you might rather consider learning scikit learn, pytorch or Spark ML. However, today’s tutorial will focus on the elements of it, before moving on to the more complex tutorials.

Basic Mathematics in Python from the math Library

The math-library in Python provides a great number of most of the relevant functionality you might want to use in Python when working with numbers. The following samples provide some overview on them:

import math
vone = 1.2367
print(math.ceil(vone))

First, we import “math” from the standard library and then we create some values. The first function we use is ceiling. In the following sample, we calculate the greatest common denominator between two numbers.

math.gcd(44,77)

Other functions are logarithmic, power, cosinus and many more. Some of them are displayed in the following sample:

math.log(5)
math.pow(2,3)
math.cos(4)
math.pi

Basic statistics in Python from the statistics library

The standard library offers some elementary statistical functions. We will first import the library and then calculate the mean of 5 values:

from statistics import *
values = [1,2,3,4,5]
mean(values)

Some other possible functions are:

median(values)
stdev(values)
variance(values)

Have a look at those two libraries – there is quite a lot to explore.

What’s next?

Now, the tutorial series for Python is over. You should now be fit to using pyspark. If you are not yet familiar with Spark, have a look at the Spark Tutorial i created here. Also, I will create more tutorials on Python and Machine Learning in the future, so make sure to check back often to the Big Data & Data Science tutorial overview. I hope you liked this tutorial. If you have any suggestions and what to improve, please feel free to get in touch with me! If you want to learn more about Python, I also recommend you the official page.