Ya, at the beginning it can be REALLY hard. Especially if you have limited Japanese, or you aren't that bubbly idiot who will talk to anything moving or not... The longer you are in an area, the easier it becomes, but many ALTs who have limited "away from home" experience, or no international living experience (a one month trip does not count imo) get REALLY depressed really quickly.
Even in my town of 100,000 people, (really about 60k, but the town area is f'n massive) it was rough until I met a good group of Japanese people (after having improved my spoken Japanese) and started to live here, rather than visit.