PySpark ETL with PostgreSQL - Case Study

Описание

This pyspark tutorial explains how to load data from csv and write to PostgreSQL database, do simple ETL manipulation like removing duplicates and null values, and ranking . Join dataframes using columns and saving these data to postgresql. writing from dataframe to database reading postgresql data using spark session. Joining data from 2 different sources i.e. from csv and Postgresql. Uses jupyter notebooks and spark during sesion

It is having the following dataflow 1) reads / extracts data from csv students and marksheet, then does Transformation like replacing null values, removing duplicates and does ranking saves data to postgresql and reads from it to dataframe joins data from 2 different sources using dataframe again writes it back to postgresql.
#pyspark tutorial
#pyspark ETL
#postgresql
#spark
#csv
#Spark ETL job
#read csv data from spark
#write data to postgresql
#read data from postgresql

Contact info:
Name: Constantine Stanley
Phone: +91 9895942514
Email: vivarem@gmail.com
twitter: @vivarem