Monday, December 12, 2016

Data Analysis Tools Week 1

Something Significant

A new look at data

I decided to pick some data that would allow me to do ANOVA. For this project I used the addhealth data and codebook to look at drinking habits and school retention. My original question did not allow me to do this week's assignment

My first model

I compared number of drinks of alcohol that respondents drank each time they drank. (H1TO16) and if they had ever been retained a grade in school (H1ED5).

Massaging the data

First, I converted all the legitimate skips in H1TO16 to "0" since it meant they had not drank alcohol in the last 12 months. I also converted the codes for "Skip", "Refused" or "Don't know" to NA. I also changed the "Skip", "Refused" or "Don't know" in H1ED5 to NA.

The code

  1. model1 = smf.ols(formula='H1TO16 ~ (H1ED5)', data=data)  
  2. results1=model1.fit()  
  3. print(results1.summary())  

The result

This test shows a statistically significant difference in the number of drinks a person had consumed at each sitting and if that person had failed or been retained a grade. The (p) is .00000614. The null hypothesis can be rejected.

Digging a little more

I was interested to know if any particular level of failing a grade had an impact on drinking. I created a new column based on which grade the respondent had reported being held back. Grades 1-6 were recoded as 1 for Elementary, 7 and 8 were recoded as 2 for Middle School, 9-11 were recoded 3 for High School. I then ran the test again again received the following result.

This was statistically significant in comparing grade levels (p) = 0.000538.

I ran a Post Hoc test to determine which level(s) would be significantly different in variance.
  1. comparegrades = multi.MultiComparison(sub1['H1TO16'], sub1['GRADERETAINED'])  
  2. result = comparegrades.tukeyhsd()  
  3. print(result.summary()) 

This resulted in the following
Multiple Comparison of Means - Tukey HSD,FWER=0.05
============================================
group1 group2 meandiff  lower  upper  reject
--------------------------------------------
 1.0    2.0    0.0715  -1.3398 1.4828 False 
 1.0    3.0    2.2715   0.8967 3.6463  True 
 2.0    3.0     2.2     0.3485 4.0515  True 
--------------------------------------------

This result shows that variance High School retention is statistically significant and the null hypothesis can be rejected.

The whole enchilada

If you want to see the whole batch of code I used - here it is.
  1. import numpy as np  
  2. import pandas as pd  
  3. import statsmodels.formula.api as smf  
  4. import statsmodels.stats.multicomp as multi  
  5.   
  6. filename ='addhealth_pds.csv'  
  7. data = pd.read_csv(filename, low_memory=False)  
  8.   
  9.   
  10. data['H1TO16']=data['H1TO16'].replace(96, np.nan)  
  11. data['H1TO16']=data['H1TO16'].replace(97, 0)  
  12. data['H1TO16']=data['H1TO16'].replace(98, np.nan)  
  13. data['H1TO16']=data['H1TO16'].replace(99, np.nan)  
  14. data['H1ED5']=data['H1ED5'].replace(6, np.nan)  
  15. data['H1ED5']=data['H1ED5'].replace(8, np.nan)  
  16. data['H1TO16'] = pd.to_numeric(data['H1TO16'])  
  17.   
  18. model1 = smf.ols(formula='H1TO16 ~ (H1ED5)', data=data)  
  19. results1=model1.fit()  
  20. print(results1.summary())  
  21.   
  22. def retained (row):  
  23.       
  24.     if row['H1ED6A'] == 1:  
  25.         return 1  
  26.     if row['H1ED6B'] == 1:  
  27.         return 1  
  28.     if row['H1ED6C'] == 1:  
  29.         return 1  
  30.     if row['H1ED6D'] == 1:  
  31.         return 1  
  32.     if row['H1ED6E'] == 1:  
  33.         return 1  
  34.     if row['H1ED6F'] == 1:  
  35.         return 1  
  36.     if row['H1ED6G'] == 1:  
  37.         return 2  
  38.     if row['H1ED6H'] == 1:  
  39.         return 2  
  40.     if row['H1ED6I'] == 1:  
  41.         return 3  
  42.     if row['H1ED6J'] == 1:  
  43.         return 3  
  44.     if row['H1ED6K'] == 1:  
  45.         return 3  
  46.                       
  47. data['GRADERETAINED'] = data.apply(lambda row: retained (row), axis = 1)  
  48. data['GRADERETAINED'] = data['GRADERETAINED'].astype('category')  
  49.   
  50.   
  51.   
  52. #sub1 = data[data['H1ED5'] == 1].dropna()  
  53. sub1 =data[['H1TO16', 'GRADERETAINED']].dropna()  
  54.   
  55. grademean = sub1.groupby('GRADERETAINED').mean()  
  56. print(grademean)  
  57.   
  58. model2 = smf.ols(formula='H1TO16 ~ (GRADERETAINED)', data=sub1)  
  59. results2=model2.fit()  
  60. print(results2.summary())  
  61.   
  62. comparegrades = multi.MultiComparison(sub1['H1TO16'], sub1['GRADERETAINED'])  
  63. result = comparegrades.tukeyhsd()  
  64. print(result.summary())  

Late addition of graphs


This is a graph of boxplots with means shown. The code for the graphs is below
I'm sure my way of getting the data into lists, then array is rather inelegant. Any comments to make it run smoother are welcome.

Code for boxplots

  1. import numpy as np  
  2. import pandas as pd  
  3. import statsmodels.formula.api as smf  
  4. import statsmodels.stats.multicomp as multi  
  5. import seaborn  
  6. import matplotlib.pyplot as plt  
  7.   
  8. filename ='addhealth_pds.csv'  
  9. data = pd.read_csv(filename, low_memory=False)  
  10.   
  11.   
  12. data['H1TO16']=data['H1TO16'].replace(96, np.nan)  
  13. data['H1TO16']=data['H1TO16'].replace(97, 0)  
  14. data['H1TO16']=data['H1TO16'].replace(98, np.nan)  
  15. data['H1TO16']=data['H1TO16'].replace(99, np.nan)  
  16. data['H1ED5']=data['H1ED5'].replace(6, np.nan)  
  17. data['H1ED5']=data['H1ED5'].replace(8, np.nan)  
  18. data['H1TO16'] = pd.to_numeric(data['H1TO16'])  
  19.   
  20. data["NUMRETAINED"] = (0 + data['H1ED6A'] + data['H1ED6B'] + data['H1ED6C'] +  
  21.  data['H1ED6D'] + data['H1ED6E'] + data['H1ED6F'] + data['H1ED6G'] +   
  22.  data['H1ED6H'] + data['H1ED6I'] + data['H1ED6J'] + data['H1ED6K'])  
  23.   
  24. def retained (row):  
  25.     if row["NUMRETAINED"] > 3:  
  26.         return 4  
  27.     if row['H1ED6A'] == 1:  
  28.         return 1  
  29.     if row['H1ED6B'] == 1:  
  30.         return 1  
  31.     if row['H1ED6C'] == 1:  
  32.         return 1  
  33.     if row['H1ED6D'] == 1:  
  34.         return 1  
  35.     if row['H1ED6E'] == 1:  
  36.         return 1  
  37.     if row['H1ED6F'] == 1:  
  38.         return 1  
  39.     if row['H1ED6G'] == 1:  
  40.         return 2  
  41.     if row['H1ED6H'] == 1:  
  42.         return 2  
  43.     if row['H1ED6I'] == 1:  
  44.         return 3  
  45.     if row['H1ED6J'] == 1:  
  46.         return 3  
  47.     if row['H1ED6K'] == 1:  
  48.         return 3  
  49.     else:  
  50.         return 0  
  51.                       
  52. data['GRADERETAINED'] = data.apply(lambda row: retained (row), axis = 1)  
  53. data['GRADERETAINED'] = data['GRADERETAINED'].astype('category')  
  54.   
  55. sub1 = data[['H1TO16', 'GRADERETAINED']].dropna()  
  56. never, high, middle, elementary, multiple = [], [], [], [], []  
  57.   
  58. subnever = data[(data['GRADERETAINED'] == 0)].dropna()  
  59. never = subnever['H1TO16'].tolist()  
  60.   
  61. subelem = data[(data['GRADERETAINED'] == 1)].dropna()  
  62. elementary= subelem['H1TO16'].tolist()  
  63.   
  64. subms = data[(data['GRADERETAINED'] == 2)].dropna()  
  65. middle = subms['H1TO16'].tolist()  
  66.   
  67. subhigh = data[(data['GRADERETAINED'] == 3)].dropna()  
  68. high = subhigh['H1TO16'].tolist()  
  69.   
  70. submult = data[(data['GRADERETAINED'] == 4)].dropna()  
  71. multiple = submult['H1TO16'].tolist()  
  72.   
  73.   
  74. mydata = [never, elementary, middle, high, multiple]  
  75.   
  76.   
  77. plt.boxplot(mydata, 0, "", labels=('Never', 'Elem', 'MS', 'HS', 'Multi'),   
  78.             showmeans=True)  
  79. plt.xlabel("Grade-level retained")  
  80. plt.ylabel("Number of drinks per sitting")  
  81. plt.show()  
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)