pyspark-session-2015-04-10.txt

First session on PySpark

mparsian@Mahmouds-MacBook-2:~/spark-1.3.0/bin# cat zfox_data.txt
crazy red fox ran fast
red fox jumped very very high
red fox is very crazy
red fox ran very fast
mparsian@Mahmouds-MacBook-2:~/spark-1.3.0/bin# 

mparsian@Mahmouds-MacBook-2:~/spark-1.3.0/bin# ./pyspark
Python 2.6.9 (unknown, Sep  9 2014, 15:05:12) 
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.3.0
      /_/

Using Python version 2.6.9 (unknown, Sep  9 2014 15:05:12)
SparkContext available as sc, SQLContext available as sqlCtx.
>>> 
>>> sc
<pyspark.context.SparkContext object at 0x1097ac410>
>>> 
>>> lines = sc.textFile("zfox_data.txt")
>>> 
>>> lines.collect()
[u'crazy red fox ran fast', u'red fox jumped very very high', u'red fox is very crazy', u'red fox ran very fast']
>>> 
>>> lines.count()
4
>>> 
>>> words = lines.flatMap(lambda x: x.split(' '))
>>> 
>>> words.collect()
[u'crazy', u'red', u'fox', u'ran', u'fast', u'red', u'fox', u'jumped', u'very', u'very', u'high', u'red', u'fox', u'is', u'very', u'crazy', u'red', u'fox', u'ran', u'very', u'fast']
>>> 
>>> words.count()
21
>>> 
>>> ones = words.map(lambda x: (x, 1))
>>> 
>>> ones.collect()
[(u'crazy', 1), (u'red', 1), (u'fox', 1), (u'ran', 1), (u'fast', 1), (u'red', 1), (u'fox', 1), (u'jumped', 1), (u'very', 1), (u'very', 1), (u'high', 1), (u'red', 1), (u'fox', 1), (u'is', 1), (u'very', 1), (u'crazy', 1), (u'red', 1), (u'fox', 1), (u'ran', 1), (u'very', 1), (u'fast', 1)]
>>> 
>>> counts = ones.reduceByKey(lambda x, y: x + y)
>>> 
>>> 
>>> counts.collect()
[(u'crazy', 2), (u'ran', 2), (u'is', 1), (u'fox', 4), (u'fast', 2), (u'high', 1), (u'very', 4), (u'red', 4), (u'jumped', 1)]
>>>