If your goal is to enable the fax machine with an Answering machine then your answer mode should be set to TAM (Telephone Answering Machine). This mode will allow the answering machine to pickup all calls after your RTA (Rings to Answer) threshold has been met. When the printer hears the anwering machine pickup and fax tones are heard then the printer will take over the phone call and respond to the fax call.
To get TAM working you need to configure your RTA to be higher than the RTA on the answering machine. For example if your answering machine is set to 4 RTA then your printer should be set to 5 RTA.
Play around with your fax settings a little and see if you can get everything working. If not then your printer may need to be reset as it got hung up on a bad fax message. Resetting the printer will clear the printers memory and allow it to respond to fax calls correctly again.
Worst case scenario you can always eliminate your phyisical fax machine and go with a Online Fax vendor. Online Fax allows you manage all your incoming/outgoing fax messages from your email. Online Fax would also allow you to plug your phone cable directly into your answering machine without having to go through your printer at all, making your enviornment simpier to manage.
More info with Online Fax: