I've been able to trace my Harts back to Henry (my second great grandfather). The family can be found all together in the 1841, 1851 census records. Henry is gone before the 1861 census and the family moves on after that.
This is a quick breakdown:
Rosetta (his wife) can be found the 1870 & 1880 censuses in NYC.
Lazarus Hart (b. 1829) I am not positive what happens to him. He may have gone to California.
David Hart (b. 1830 d 1907) moved to Victoria, British Columbia, Canada
Amelia Hart (b. 1832 d. 1909) marries a man named Emanuel and moves to San Francisco, California.
Julia Hart (b. 1834) can be found with her mother in NYC.
Sampson Hart (b. 1834) unknown what happens to him.
Samuel Hart (b. 1835 d. 1918) was my g-g-grandfather. He served in the Civil War and passes away in Boston, MA.
Henry Hart (b. 1838) last known residence is in Montreal (1901 census).
Maria Hart (b. 1840) can be found with her mother in NYC.
Sophia Hart (b. 1842) can be found with her mother in NYC.
Emanuel Hart (b. 1848 d. 1910) passes away in Toronto, Canada.
I have lots more info. Just let me know who you are interested in. I am interested if anyone knows who the parents of Henry Hart or his wife Rosetta Cardoza is. Also present in all of the censuses is Rosetta's sister, Abigail.