It's possible to use trademark law to obtain exclusive rights for your cat (as one feline has proven). However, there are some things to keep in mind.
The USPTO will only register a book title if it is part of a book series (for example, Pete the Cat for children's books). So you won't be able to federally register your cat's name for books until you've got a second book. You can, however, register in a different class.
What are classes? When you apply for a trademark, you seek exclusive trademark rights for a class (or classes) of goods or services. For example, if your cat were featured in a series of books, you would register in Class 16 (Paper Goods). If your cat was a superstar on Instagram and earning money from live performances, you would register in Class 41 (Educational and Entertainment Services). Your registrations are restricted to those classes of goods or services that you are currently selling or that you plan to offer (if you are registering on an intent-to-use basis). For each class of goods or services that you register, you must pay a separate registration fee and you must meet the basic trademark requirements.
What about copyright? Copyright provides the exclusive right to reproduce or modify a creative work such as music, art, photographs, books, and movies. You'd use copyright law to protect the text, illustrations and photos. Keep in mind that even though it's your cat in the photo, you don't own the copyright unless you took the picture or paid for the rights. (The photographer usually owns the copyright.) A combination of copyright and trademark will augment your cat's intellectual property rights so you can concentrate on earning the big bucks.
P.S. What about your cat's right of publicity? Humans can stop others from using their likeness to endorse products under the right of publicity, but animals don't have a right of publicity as we explained in a related question.