Thursday, 8 May 2014

Git

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

  • Created by same people who developed LINUX
  • The most popular implementation of version control today
  • Everything is stored in local repositories on your computer
  • Operated from the command line
GitHub
Github.Com is a web based hosting service for software development projects which uses the Git version control system.

Pushing and pulling:
Basic Git commands

Configure username and email using Git Bash

$git config --global user.name "your name"
$git config --global user.email "your email"

Type the following to confirm your changes

$git config --list

Close Git Bash with following command

$ exit

Initialize repository in an existing directory

$ git init localpath

Cloning and existing repository

$git clone url

Adding

$git add . (add new files)
$git add -u (update tracking for files that changed names or were deleted)
$git add -A (does both of the previous)

You should do this before committing.

Committing

$git commit -m "message" ("message" is useful description of what you did)

git commit -am "save arezzo files"( This command will add and commit file. git commit -a -m "message" - both do the same thing)

Pushing
$git push

Push an existing repository
$git remote add origin https://github.com/manaliajudiya/R.git
$git push -u origin master

Checking the status of your file
$git status

Branches
Sometimes you are working on a project with a version being used by many people. You may not want to edit that version. 
So you can create a branch with the command
$git checkout -b branchname

To see what branch you are on
$git branch

To switch back to master branch type
$git checkout master

Pull requests
If you fork someone's repository or have multiple branches you will both be working separately. In this case, to merge you changes, you need to send a pull request. This is a feature of github.

$git pull  (fetch files from repository, you need to have all files on your computer otherwise there will be error "Updates were rejected. Remote contains work that you do not have locally.")

List files in local git repo
$git rev-parse --show-toplevel

To change top level
$cd ~/dirname

Show current working directory
$git ls-files --directory

Change working directory
$git --git-dir git/dirname status

Monday, 7 April 2014

What do data scientists do?

In general terms, Data Scientists gets data and convert it in to information/predicts results.

Data Scientists collects data from real world(generally BigData from internet), process that data and convert it in to dataset that can be analyzed. They analyze this dataset based on statistical models or machine learning and create results/reports which can be useful for data driven products(even for general public).

Step by step process they do with data :
  • Define the question
  • Define the ideal dataset
  • Determine what data you can access
  • Obtain the data
    • Reading data - Excel, XML,JSON, Web,..
    • Merging data
  • Clean the data
    • Reshaping data
    • Summarizing data
  • Exploratory data analysis
    • Graphs
    • Plotting systems
    • Clustering
  • Statistical prediction/modeling
    • Extracting generalization information from data
  • Interpret results
  • Challenge results
  • Synthesize/Write up results
  • Create reproducible code
    • Completely reproduce all the documents such that you can communicate it with other people
  • Distribute results to other people

Why R?

R programming language has become the single most important tool for computational statistics, visualization and data science.
  • It is free
  • It has comprehensive set of packages
    • Data Access
    • Data Cleaning
    • Analysis
    • Data reporting
  • It has one of the best development environment - R Studio (in terms of data science)
  • It has amazing ecosystem of developers - large collection of packages developed and published by larger R community
  • Packages are easy to install and "nicely play together"


Sunday, 6 April 2014

Difference between Pig and Hive

Apache Pig and Hive are two projects that layer on top of Hadoop, and provide a higher-level language for using Hadoop's MapReduce library. 

Pig
  • Apache Pig provides a scripting language for describing operations like reading, filtering, transforming, joining, and writing data -- exactly the operations that MapReduce was originally designed for. 
  • Rather than expressing these operations in thousands of lines of Java code that uses MapReduce directly, Pig lets users express them in a language not unlike a bash or perl script. Pig is excellent for prototyping and rapidly developing MapReduce-based jobs, as opposed to coding MapReduce jobs in Java itself. 
If Pig is "scripting for Hadoop", then Hive is "SQL queries for Hadoop".

Hive
  • Apache Hive offers an even more specific and higher-level language, for querying data by running Hadoop jobs, rather than directly scripting step-by-step the operation of several MapReduce jobs on Hadoop. 
  • The language is, by design, extremely SQL-like. Hive is still intended as a tool for long-running batch-oriented queries over massive data; it's not "real-time" in any sense. 
  • Hive is an excellent tool for analysts and business development types who are accustomed to SQL-like queries and Business Intelligence systems; it will let them easily leverage your shiny new Hadoop cluster to perform ad-hoc queries or generate report data across data stored in storage systems mentioned above.