Neighbor tried to ask me out he is married and 55 years old. I am 27 and not interested?

I have a neighbor who is about 53 any who he does maintenance in my mothers home well one day he came over to my home and asked for my number and my mothers number I thought it was strange because although we see him and he is friendly he has never ever came over here.... I call my mom immediately and said mom the neighbor came over and asked for both of our numbers but my mom said last year... not last week... but last year I asked him to fix my refrigerator. today he fixed the fridge but after he fixed the fridge he called my cellphone and said how is the fridge working and I said it alright my wanted him to come back over because he left something unfinished and I told him over the phone that my mom wanted him to come back over and he told me i don't live across from you all anymore me and my wife broke up I wanted to ask you some questions how old are you i said 27 and he said I didn't know you were that old when I thought something is wrong with this conversation I said sir when my mom comes back I will have her call you ok, I tried to rush him off the phone because first off i am 27, secondly im not interested, 3rd he is my neighbor, 4th he is my neighbors husband... I called my mom and said what he told me my mom laughed and said she had to get off the phone because she was at the store and boy did I get upset and scared... but I got upset and scared because when he tried to hit on me over the phone it just took me back to history of older mean hitting on me and it didn't feel right... this dudes daughter was in high school with me, she has 4 kids at 26, I have no children and I have 3 degrees not trying to compare or contrast but she looks like something is wrong with her and her sisters look a but screwed up to.... his wife I believe is dating our other neighbors husband because he is always coming out of the wife's house when the neighbor who fixed our fridge leaves so I cant tell his wife because I don't know if she really cares maybe they have a swinging relationship but honestly I don't want him calling me anymore unless its about house work or saying hi. My mom said she saw him at a community college with a girl my age years ago but after he called me my day went down hill. what should I do?