Chi-Square - back to my old data

Objective

Run a chi-square test on categorical -> categorical data. For last week, I used different data since I wanted to check p-values on categorical -> quantitative data.

Preparing the data

The first category was easy. It was based on a question "Have you ever been held back or retained in a grade?" All I had to do was deal with skips, and "I don't know" answers which were turned into N/A.

The second category was a little more challenging. I selected three questions to determine violence. In the last 12 months: "How often have you fought?", "How often have you pulled a knife or gun on someone?", and "How often have you shot or stabbed someone?"

Since that category had answers other than "yes" or "no", I had to convert all affirmative answers to a 1 and  "valid skip" and "0" answers as "0" in a newly created a new variable "VIOLENT". Skips, refuse, and other non-answers were marked as NA.

Finally, I created a new dataframe without NA data.

I am using RETAINED as my independent variable and VIOLENT as my dependent variable

The graph

Crosstabs and Chi-Square - output from code

Crosstab view of categories "Have been retained" and "Has committed violence"

RETAINED   0.0  1.0
VIOLENT           
0         3575  770
1         1487  618


Percentage table of crosstab view above

RETAINED       0.0       1.0
VIOLENT                    
0         0.706243  0.554755
1         0.293757  0.445245


Chi-square value, p-value, expected counts
(113.01797080208391, 2.1382800168079119e-26, 1, array([[ 3409.98294574,   935.01705426],
       [ 1652.01705426,   452.98294574]]))

Conclusions

The null hypothesis can be rejected. There is a clear relationship between being retained a grade and the likelihood of committing violence. The P-value is extremely low 2.1 X 10^(-26) and the Chi-square was far greater than 3.84 (113.02) for a 2X2 table.

Expected values
Not retained and Not violent 3409 compared to 3575 actual
Not retained and violent 935 expected and 770 actual
Retained and Not violent expected 1652 with actual 1487
Retained and violent expected 452 with actual 618

The code

1. # -*- coding: utf-8 -*-    
2. """  
3. Created on Tue Dec 13 17:35:10 2016  
4.   
5. @author: broth  
6. """    
7.     
8. import numpy as np    
9. import pandas as pd    
10. import matplotlib.pyplot as plt    
11. import seaborn    
12. import scipy.stats as sps    
13.     
14. filename ='addhealth_pds.csv'    
15. data = pd.read_csv(filename, low_memory=False)    
16.     
17. data['H1ED5'] = pd.to_numeric(data['H1ED5']) #failed any grade    
18. pd.to_numeric(data["H1ED6A"])#failed 1ST grade    
19. pd.to_numeric(data["H1ED6B"]) #failed 2ND grade    
20. pd.to_numeric(data["H1ED6C"]) #failed 3RD grade    
21. pd.to_numeric(data["H1ED6D"]) #failed 4th grade    
22. pd.to_numeric(data["H1ED6E"])#failed 5th grade    
23. pd.to_numeric(data["H1ED6F"])#failed 6th grade    
24. pd.to_numeric(data["H1ED6G"]) #failed 7th grade    
25. pd.to_numeric(data["H1ED6H"]) #failed 8th grade    
26. pd.to_numeric(data["H1ED6I"]) #failed 9th grade    
27. pd.to_numeric(data["H1ED6J"]) #failed 10th grade    
28. pd.to_numeric(data["H1ED6K"]) #failed 11th grade    
29. pd.to_numeric(data["H1FV5"]) #got into fight    
30. pd.to_numeric(data["H1FV7"]) #pulled a knife or gun    
31. pd.to_numeric(data["H1FV8"]) #shot or stabbed someone    
32.     
33. data['H1ED5']=data['H1ED5'].replace(6, np.nan)    
34. data['H1ED5']=data['H1ED5'].replace(8, np.nan)    
35.     
36. data['H1FV5'] = data['H1FV5'].replace(6, np.nan)    
37. data['H1FV5'] = data['H1FV5'].replace(8, np.nan)    
38. data['H1FV5'] = data['H1FV5'].replace(9, np.nan)    
39.     
40. data['H1FV7'] = data['H1FV7'].replace(6, np.nan)    
41. data['H1FV7'] = data['H1FV7'].replace(8, np.nan)    
42. data['H1FV7'] = data['H1FV7'].replace(9, np.nan)    
43.     
44. data['H1FV8'] = data['H1FV8'].replace(6, np.nan)    
45. data['H1FV8'] = data['H1FV8'].replace(8, np.nan)    
46. data['H1FV8'] = data['H1FV8'].replace(9, np.nan)    
47.     
48. def violent (row):    
49.     if row["H1FV5"] >= 1 and row["H1FV5"] <= 2:    
50.         return 1    
51.     if row["H1FV7"] >= 1 and row["H1FV7"] <= 2:    
52.         return 1    
53.     if row["H1FV8"] >= 1 and row["H1FV8"] <= 2:    
54.         return 1    
55.     else:    
56.         return 0    
57.     
58. sub1=data[['H1ED5', 'H1FV5', 'H1FV7', 'H1FV8']].dropna()    
59. check=sub1['H1ED5'].value_counts()    
60.     
61. sub1['VIOLENT'] = sub1.apply(lambda row: violent (row), axis = 1)    
62. sub1['RETAINED']=sub1['H1ED5']    
63.     
64. seaborn.factorplot(x="RETAINED", y="VIOLENT", data =sub1, kind='bar')    
65. plt.xlabel("Has been held back or retained a grade in school")    
66. plt.ylabel("Percentage who have been violent in last 12 months")    
67.     
68. sub1['VIOLENT']=sub1['VIOLENT'].astype('category')    
69. sub1['RETAINED']=sub1['RETAINED'].astype('category')    
70.     
71. ct1 = pd.crosstab(sub1['VIOLENT'], sub1['RETAINED'])    
72. print("Crosstab view of categories \"Have been retained\" and \"Has committed \    
73. violence\"\n")    
74. print(ct1)    
75.     
76. colsum=ct1.sum(axis=0)    
77. colpct=ct1/colsum    
78. print("\n\nPercentage table of crosstab view above\n")    
79. print(colpct)    
80.     
81. print("\n\nChi-square value, p-value, expected counts")    
82. cs1=sps.chi2_contingency(ct1)    
83. print(cs1)