There are two modes of tracker connection: Load Balancing and High Availability.
Load balancing mode forces each agent to distribute their requests to all tracker servers they know of as even as it is possible. Distribution is done on a job level. For example, we have 3 jobs (A, B, C) and 2 tracker servers available. Agent will connect to tracker #1 for jobs A and C and tracker #2 for job B. The algorithm of choosing tracker is pseudo-random and gives always same results on all Agents if they have same list of trackers. Providing different lists of trackers when Agents use trackers in load balancing will result in inability to connect. This is the default mode.
High Availability mode forces each agent to connect to all trackers it can reach for each job to get maximum information of all peers. You can configure any number of tracker servers for each Agent in HA mode, although ensure that they intersect (and able to connect) to at least one common tracker server. See here how to enable HA mode.