This is a case where tattling on your neighbors is not only appropriate, but necessary.
First up, there's the immediate matter of their pet's health and well-being. "It's incredibly important that concerned citizens report suspected animal cruelty or neglect," says Howard Lawrence, vice president of the ASPCA Anti-Cruelty Group. "By doing so, you are speaking up for those who can't."
In NYC, the best avenue for reporting a problem is 311, Lawrence tells us. "It is also important to consider your own personal safety," he adds, "[and] most cities and towns allow you to make anonymous complaints to law enforcement. For example, in New York City, calls to 311 can remain anonymous."
As far as your apartment building goes, your landlord or building manager would surely be concerned about how your neighbors are handling their pet in the backyard and other common areas. From what you describe, their poor treatment and training of their pet impacts both the enjoyment and safety of other residents, and you may want to give management a heads-up so they can approach your neighbor. (Though keep in mind that in most cases it's not feasible to submit complaints anonymously to the management, and it's possible your neighbors could find out through them—or through their powers of deduction—that it's you who called them out.)
You may also want to check in with a local animal shelter, animal welfare organization, or animal control agency for advice about your situation, and potential resources that could be helpful to your neighbors and their neglected pup. For further resources, start out with the ASPCA, the Humane Society, and Animal Care Centers of NYC.
"Knowing what resources are available in your community to help pet owners could be one of your best assets in combating animal cruelty and neglect," Lawrence adds. "For example, in New York City the ASPCA Cruelty Intervention Advocacy Program (CIA) works in the areas of animal hoarding, emergency veterinary care for underserved communities and resources for domestic violence survivors to improve conditions for the animals and humans involved."
If the situation is more extreme however, you should take your concerns straight to the police. "If you witness a crime against an animal in progress you should call 911 immediately and wait for the police to arrive," says Lawrence.